├── .gitignore
├── .project
├── .pydevproject
├── README
├── README.md
├── assembly_compilation_ARM
└── main_original.s
├── assembly_compilation_x86_gcc
└── main_original.s
├── configurationFiles
├── configuration-template-explanation.xml
├── configuration_ARM_IPC.xml
├── configuration_likwidPowerMeter.xml
├── configuration_x86_IPC.xml
└── measurement
│ ├── IPC.xml
│ └── measurementLikwid.xml
└── src
├── Algorithm.py
├── Fitness
├── DefaultFitness.py
└── SimplicityTempFitness.py
├── Individual.py
├── Instruction.py
├── Measurement
├── Measurement.py
├── MeasurementIPC.py
└── MeasurementLikwidPower.py
├── Operand.py
├── Population.py
├── __init__.py
└── parseGeneticResults.py
/.gitignore:
--------------------------------------------------------------------------------
1 |
2 | *.pyc
3 |
--------------------------------------------------------------------------------
/.project:
--------------------------------------------------------------------------------
1 |
2 |
3 | GeST
4 |
5 |
6 |
7 |
8 |
9 | org.python.pydev.PyDevBuilder
10 |
11 |
12 |
13 |
14 |
15 | org.python.pydev.pythonNature
16 |
17 |
18 |
--------------------------------------------------------------------------------
/.pydevproject:
--------------------------------------------------------------------------------
1 |
2 |
3 |
4 | /${PROJECT_DIR_NAME}/src
5 |
6 | python 3.6
7 | Default
8 |
9 |
--------------------------------------------------------------------------------
/README:
--------------------------------------------------------------------------------
1 | Copyright 2019 ARM Ltd. and University of Cyprus
2 | Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction,
3 | including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software,
4 | and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
5 | The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
6 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
7 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
8 | TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
9 |
10 |
11 | AUTOMATIC FRAMEWORK FOR STRESS TEST GENERATION (GeST Generating Stress-Tests)
12 |
13 | GENERAL DESCRIPTION
14 |
15 | This framework is based on Genetic algorithms (GA) and it has been succesfully been used to develop power viruses (stres-tests that maximize power consumption),
16 | dI/dt viruses (stress-tests that maximize voltage noise) and IPC stress tests (maximize instruction per cycle throughput) on various CPUs such as ARM cortex-A15, cortex-A7,
17 | cortex-A57,cortex-A53, cortex-A72, AMD Athlon II X4 645, X-Gene2, X-Gene3
18 |
19 | GA optimize for the target metric using bio-inspired operators such as crossover (exchange of genes), mutation, selection of fittest individual for breeding (for crossover).
20 | GA typically begins with a population of individuals (randomly generated) and evolves this population by measuring the fitness of the individuals and applying the aforementioned operators.
21 |
22 | The framework produces assembly instructions sequences that are measured for the metric of interest (with a fitness function). The goal of the framework is to generate the fittest instruction sequence
23 | (in the GA terminology the fittest individual).
24 | For instance if the goal is maximum CPU power consumption the framework attempts to generate instruction sequences that maximize the power consumption.
25 |
26 | The framework can be extended by the user to stress other components such as DRAM, Last level cache (Uncore) etc. Also it can be extended to support more complicate fitness functions that take in account
27 | multiple measurement metrics (power, pef counters etc) and instinsic instruction sequence characteristics such as instruction mix breakdown.
28 |
29 | The framework also offers an easy interface to the user to add custom measurement procedures.
30 | Some template measurements are included in the framework such as maximizing max power using IPMI readings and maximizing IPC through perf counters on a ssh reachable machine.
31 | But for instance if a user wants to maximize on a bare metal machine power using a specific power plug meter, he can write his own measurement function and seemlesly integrate it in the framework (read DEFINING CUSTOM MEASUREMENTS - CUSTOM FITNESS CLASSES and FUNCTIONS section).
32 |
33 | The framework is written in Python 3 and xml is used for configuration files
34 |
35 | REQUIREMENTS
36 |
37 | The framework shoud work without any issues on Linux enviroment, but it has been tested only on Windows only with Eclipse IDE for development and running experiments.
38 | The following enviroment is recommended
39 |
40 | 1) Windows OS
41 | 2) LiClipse IDE
42 | 3) Python 3.6 is recommended but probably any Python 3 would work (NOTE this is not compatible with python 2!!)
43 | 4) NI-visa drivers must be installed if going to use measurement instruments such oscilloscopes and spectrum analyzer for measurements (tested with NI-VISA 5)
44 | 5) Paramiko python libary and its dependencies must be installed for enabling ssh communication (execute from shell the command: pip install paramiko #pip is the python package manager ). Very useful coordinating execution on the target machine.
45 | 6) Python PYVISA libary if going to use measurement instruments such oscilloscopes and spectrum analyzer for measurements
46 |
47 | INSTALLATION INSTRUCTIONS on WINDOWS
48 |
49 | 1) Download liclipse for python development https://www.liclipse.com/
50 | 2) Download python 3 https://www.python.org/downloads/
51 | 3) Locate pip binary. For python 3.6 should be found in C:\Users\UserName\AppData\Local\Programs\Python\Python36-32\python.exe
52 | From a windows terminal run
53 | pip install paramiko
54 | pip install -U pyvisa #necessary if measurements with spectrum analyzers and oscilloscopes
55 |
56 |
57 | RUN INSTRUCTIONS ON ECLIPSE/Liclipse
58 |
59 | 1) Import the project as Python project
60 | 2) Set correctly the configuration file (look section below)
61 | 3) Right click on the project then debug-as. In the arguments section specify the full path to the configuration file you will use.
62 |
63 | SET CORRECTLY THE CONFIGURATION FILE
64 | This section explains how to set correctly the configuration file and we will explain this with a particular example.
65 | Let's say you want to optimize for maximum power consumption on an x86 CPU using likwid power meter.
66 | In this case you can use the following provided class and files: configuration_likwidPowerMeter.xml, measurementLikwid.xml and MeasurementLikwidPower.py.
67 | Below are some steps you need to follow to set correctly the configuration files in relation to your environemnt. Similar steps should be followed for the IPC optimization (configuration files included in the release)
68 | any any other optimization you wish to perform.
69 |
70 | 1) Open the configuration_likwidPowerMeter.xml
71 | 2) Set the dirToSaveResults attribute to the path where you want the results to be saved.
72 | 3) Set the compilationDir to the path where the assembly template compilation is located (the director where the main_original.s is located)
73 | 4) Make sure the main_original.s has a #loop_code string which point to GeST where to print the individual. Add to the main_original.s any register and memory initilization you wish to have.
74 | 5) Make sure that the measurementClass attribute is set to MeasurementLikwidPower, the measurementClassConfFile is set to measurementLikwid and the fitnessClass to DefaultFitness
75 | 6) On the target machine (the machine for which the virus is generated) create a temporary directory which will be used by the GA for compiling the individuals and placing temporary the
76 | individual's source code and binary.
77 | 7) Open the measurement xml configuration file, in this particular example the file you need to open is the measurement/measurementLikwid.xml
78 | 8) Set the targetRunDir with the path where GA temporary directory is located (the one created in step 6)
79 | 9) Set the targetHostname attribute value to the ip or hostname with which the target machine is accessible through ssh
80 | 10) Set the targetSSHusername and targetSSHpassword attributes. Make sure that with the username account you can run sudo commands without password.
81 | To achieve this make sure your account belongs to sudo group and if e.g. the target runs Ubuntu add this line "%sudo ALL=(ALL:ALL) NOPASSWD:ALL " to the /etc/sudoers file.
82 | 11) Set the core ids that will be used in the optimization (that will run the indiviudal binary)
83 | 12) Set the time_to_measure each individual. For power optimization with likwid we had good results with 5 seconds but ofcourse you are free to play with this variable.
84 |
85 | MORE DETAILS ON HOW TO USE GEST
86 |
87 | Detailed explanation of the main configuration file options is found in configuration-template-explanation.xml located in ./configurationFiles.
88 | Also we recommend to read the ISPASS 2019 paper "GeST an automatic framework for generating CPU stress-tests" https://ieeexplore.ieee.org/document/8695639 which is a good reading to understand the framework and how to use it.
89 |
90 | To get more information on how to use GeST and what capabilities it offers please continue reading.
91 |
92 | GeST gives you the ability:
93 | a) To play with some algorithm parameters e.g. mutation, crossover,loop sizes, population size etc
94 | b) To specify explicity what instructions or even instruction sequences you want to use in the optimization process.
95 | c) Specify operands and which instructions will use those operands which eventually give you the ability to force or not force dependencies between instructions
96 | d) You can specify fixed code segments which will be present in all the individuals (an individual is an instruction sequence generated) e.g. some check for not touching illegal memory, register and memory initialization
97 | e) Make a run starting with fixed instruction mix breakdwon
98 | f) Ability to save a run and continue it after
99 | g) Different measurement procedures by defining your own manual class that inherits from the Measurement/Mesurement.py class.
100 | You don't need to change anything else in the code just specify the custom measurement class name in the configuration file.
101 | h) Different fitness functions by defining your own manual class that inherits from the Fitness/DefaultFitness.py class
102 | You don't need to change anything else in the code just specify the custom measurement class name in the configuration file.
103 |
104 | CONFIGURATION FILES LOCATION
105 | The main configuarion files should be placed in ./configurationFiles folder. Configuration related to custom measurement classes should be placed in ./configurationFiles/measurement directory
106 |
107 | ASSEMBLY COMPILATION DIRECTORY
108 | The dirs with prefix assembly_compilation are the dirs which contain all the necessary files for the generation of the binary that is going to be run on the target machine.
109 | You must specify which dir you are going to use for the compilation in the configuration file.
110 | The assembly compilation dir in the most simple form contains one source file that will be compiled, namely the main_original.s
111 | The main_orginal.s must contain a line which contains the string “#loop_code”. This string will be replaced by the generated by framework code sequence.
112 | Any memory, register initialization or in general common fixed code accross all individuals can be placed on the main_original.s file
113 |
114 | FRAMEWORK OUTPUT RESULTS
115 | The dirToSaveResults parameters which is set in the configuration.xml file specifies the directory in which results – outputs of the framework run will be saved.
116 | In this directory you can find the generated code sequences which will be save in ascii format in .txt files. The name format of each file goes like this populationNumber_individualsIdNumber_measurement1_measurement2_etc.
117 | You can specify as many measurements as you want. Also a dir which is named after the date/time of the start of the run is created e.g. (13-12-10-15-06 means the run started on 13th of December at 15-06). This dir will contain each population saved in .pkl file and the rand state (also in .pkl format) at each population.
118 | This dir can be used as a seed dir in case you want to continue an unfinished run. So basically each run leaves a seed dir behind. (Note to start a run from an existing SeedDir set the seedDir parameter in the configuration file)
119 | In the seed dir the assembly compilation file is copied. All the compilations and modifications of the code happen on that copied assembly compilation file which is located in the seed dir.
120 | In the seed dir also is copied the configuration.xml and measurement.xml file and the src of the framework which was used for that run.
121 | The population pkl files can be used for parsing results. After measuring a population the whole population is saved in a .pkl file.
122 | The populations are saved in ascending order like this 1.pkl, 2.pkl 3.pkl and etc. The parseGeneticResults.py file is an example of how to parse the population files and get statistics about average and best of each generation as well as the average and best individual’s instruction mix for each population.
123 | To use the script just pass the seed dir’s absolute path as argument.
124 |
125 | DEFINING CUSTOM MEASUREMENTS - CUSTOM FITNESS CLASSES and FUNCTIONS
126 | Framework allows definining custom measurement classes and fitness functions.
127 | The measurement procedure is a a very environment specific procedure. A GA stress-test generation framework must support IPC, thermal, power, etc. optimizations.
128 | Moreover, for instance, to optimize for power consumption various power measurment tools might be used such as IPMI, RAPL etc.
129 | Due to so many alternatives a framework that supports all these alternatives is impossible to create.
130 | Hence, we provide a simple interface to the user to write it's own measurement procedures and use them with this framework. Templates of measurement classes can be found in ./Measurement directory and
131 | the representative configuration files in ./configurationFiles/measurement directory. To add your own measurement class create a class that inherits from Measurement.py and place it ./Measurement directory.
132 | You have to overwrite the measure function and most likely you
133 | will also need to overwite the init function to read any custom parameters required by your class. The custom parameters need to be specified in a configuration file placed in ./configurationFiles/measurement.
134 | You don't need to touch any other framework's code to use your measurement class. Just add in the main configuration file the name of the class and the name of the classe's configuration file (just the name not the full path) in the
135 | measurementClass and measurementClassConfFile parameters.
136 | This is achieved with Python's capability to dynamically load a class (without specifing it during source compilation).
137 |
138 | You can also modify the fitness function with the same principles as for the Measurement class.
139 | You can do this by inheriting from the DefaultFitness class and overriding the getFitness function. The fitness function takes as parameter the individual after is being measurement. The individual object comes with
140 | its instruction sequence and its measurements. The default getFitness function just returns the first measurement as the fitness value, e.g. it can be the average power while running the individual.
141 | You have to place the custom fitness class in the ./Fitness directory. To
142 | use your custom fitness class specify the name in the fitnessClass parameter in the configuration file.
143 | For now Fitness classes does not come with a configuration file like measurement classes.
144 | If needed this might be changed in the future.
145 |
--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
1 | # GeST
2 |
3 | GeST (Generating Stress-Tests) is a framework for automatic hardware stress-test generation.
4 |
5 | A scientific paper about GeST is published in ISPASS 2019 https://ieeexplore.ieee.org/document/8695639
6 |
7 | Hadjilambrou, Z., Das, S., Whatmough, P.N., Bull, D. and Sazeides, Y., 2019, April. GeST: An Automatic Framework For Generating CPU Stress-Tests. In 2019 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS) (pp. 1-10). IEEE.
8 |
--------------------------------------------------------------------------------
/assembly_compilation_ARM/main_original.s:
--------------------------------------------------------------------------------
1 | /*
2 | Copyright 2019 ARM Ltd. and University of Cyprus
3 | Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction,
4 | including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software,
5 | and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
6 | The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
7 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
8 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
9 | TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
10 | */
11 | .data
12 |
13 | msg:
14 | .ascii "\n"
15 | len = . - msg
16 | .text
17 | .align 2
18 | .global main
19 | .type main, %function
20 |
21 |
22 |
23 | main:
24 |
25 | #reg init
26 |
27 | LDR x0,=0xAAAAAAAA
28 | LDR x1,=0x55555555
29 | LDR x2,=0x00000000
30 | LDR x3,=0x00000000
31 | LDR x4,=0x55555555
32 | LDR x5,=0x33333333
33 | LDR x6,=0xFFFFFFFF
34 | LDR x7,=0xFFFFFFFE
35 | LDR x8,=0x00000001
36 | LDR x9,=0xCCCCCCCC
37 | LDR x10,=0xAAAAAAAA
38 | LDR x11,=0x55555555
39 | LDR x12,=0x00000000
40 | LDR x13,=0x00000000
41 | LDR x14,=0x55555555
42 | LDR x15,=0x33333333
43 | LDR x16,=0xFFFFFFFF
44 | LDR x17,=0xFFFFFFFE
45 | LDR x18,=0x00000001
46 | LDR x19,=0xCCCCCCCC
47 | LDR x20,=0xAAAAAAAA
48 | LDR x21,=0x55555555
49 | LDR x22,=0x00000000
50 | LDR x23,=0x00000000
51 | LDR x24,=0x55555555
52 | LDR x25,=0x33333333
53 | LDR x26,=0xFFFFFFFF
54 | LDR x27,=0xFFFFFFFE
55 | LDR x28,=0x00000001
56 | LDR x29,=0xCCCCCCCC
57 | LDR x30,=0xCCCCCCCC
58 | LDR D0,=0xAAAAAAAAAAAAAAAA
59 | LDR D1,=0xFFFFFFFFFFFFFFFF
60 | LDR D2,=0x5555555555555555
61 | LDR D3,=0x3333333333333333
62 | LDR D4,=0xCCCCCCCCCCCCCCCC
63 | LDR D5,=0x0000000100000001
64 | LDR D6,=0xFFFFFFFEFFFFFFFE
65 | LDR D7,=0x0000000000000000
66 | LDR D8,=0xAAAAAAAA55555555
67 | LDR D9,=0x55555555AAAAAAAA
68 | LDR D10,=0xAAAAAAAAAAAAAAAA
69 | LDR D11,=0xFFFFFFFFFFFFFFFF
70 | LDR D12,=0x5555555555555555
71 | LDR D13,=0x3333333333333333
72 | LDR D14,=0xCCCCCCCCCCCCCCCC
73 | LDR D15,=0x0000000100000001
74 | LDR D16,=0xFFFFFFFEFFFFFFFE
75 | LDR D17,=0x0000000000000000
76 | LDR D18,=0xAAAAAAAA55555555
77 | LDR D19,=0x55555555AAAAAAAA
78 | LDR D20,=0xAAAAAAAAAAAAAAAA
79 | LDR D21,=0xFFFFFFFFFFFFFFFF
80 | LDR D22,=0x5555555555555555
81 | LDR D23,=0x3333333333333333
82 | LDR D24,=0xCCCCCCCCCCCCCCCC
83 | LDR D25,=0x0000000100000001
84 | LDR D26,=0xFFFFFFFEFFFFFFFE
85 | LDR D27,=0x0000000000000000
86 | LDR D28,=0xAAAAAAAA55555555
87 | LDR D29,=0x55555555AAAAAAAA
88 | LDR D30,=0x55555555AAAAAAAA
89 |
90 |
91 |
92 | LDR x12,=0x200
93 | #0x200 is 512 bytes
94 | SUB x12,sp,x12
95 | MOV x10,x12
96 | MOV x11,10
97 | #x12 will hold the end address
98 |
99 | LDR x28, =0x3B9ACA00/*1 billion iterations. x28 to be used for iteration counter if running for fixed iterations*/
100 |
101 | Start:
102 |
103 |
104 | #loop_code
105 |
106 |
107 |
108 |
109 | b Start
110 |
111 | #SUB x28,x28,1 #uncomment for running for fixed iterations
112 | #CBNZ x28,Start
113 |
114 | ret
115 | .size main, .-main
116 | .ident "GCC: (APM-8.0.10-le) 4.9.3 20150218 (prerelease)"
117 | .section .note.GNU-stack,"",%progbits
118 |
--------------------------------------------------------------------------------
/assembly_compilation_x86_gcc/main_original.s:
--------------------------------------------------------------------------------
1 | /*
2 | Copyright 2019 ARM Ltd. and University of Cyprus
3 | Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction,
4 | including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software,
5 | and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
6 | The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
7 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
8 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
9 | TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
10 | */
11 |
12 | .file "main.s"
13 | .data
14 | .align 32
15 | .simdvalue:
16 | .long 0xaaaaaaaa
17 | .long 0x55555555
18 | .long 0x33333333
19 | .long 0xcccccccc
20 | .long 0xaaaaaaaa
21 | .long 0x55555555
22 | .long 0x33333333
23 | .long 0xcccccccc
24 | .text
25 | .globl main
26 | main:
27 | .LFB0:
28 | .cfi_startproc
29 | pushq %rbp
30 | .cfi_def_cfa_offset 8
31 | .cfi_offset 5, -8
32 | movl %esp, %ebp
33 | .cfi_def_cfa_register 5
34 |
35 |
36 |
37 | #reg init
38 |
39 |
40 | mov $0x55555555, %rax
41 | mov $0x33333333, %rbx
42 | mov $0x22222222, %rdx
43 | mov $0x44444444, %rsi
44 | mov $0x77777777, %rdi
45 |
46 | fldpi
47 | fldpi
48 | fldpi
49 | fldpi
50 | fldpi
51 | fldpi
52 | fldpi
53 |
54 | vmovdqa .simdvalue(%rip), %ymm0
55 | vmovdqa .simdvalue(%rip), %ymm1
56 | vmovdqa .simdvalue(%rip), %ymm2
57 | vmovdqa .simdvalue(%rip), %ymm3
58 | vmovdqa .simdvalue(%rip), %ymm4
59 | vmovdqa .simdvalue(%rip), %ymm5
60 | vmovdqa .simdvalue(%rip), %ymm6
61 | vmovdqa .simdvalue(%rip), %ymm7
62 | vmovdqa .simdvalue(%rip), %ymm8
63 | vmovdqa .simdvalue(%rip), %ymm9
64 | vmovdqa .simdvalue(%rip), %ymm10
65 | vmovdqa .simdvalue(%rip), %ymm11
66 | vmovdqa .simdvalue(%rip), %ymm12
67 | vmovdqa .simdvalue(%rip), %ymm13
68 | vmovdqa .simdvalue(%rip), %ymm14
69 | vmovdqa .simdvalue(%rip), %ymm15
70 |
71 | mov $50000000, %rcx #leave for i--
72 |
73 | #subq $304, %rsp
74 |
75 | .L2:
76 |
77 | #loop_code
78 |
79 |
80 | #sub $1,%rcx #remove this and below comment for fixed iterations
81 | #cmp $0, %rcx
82 | jmp .L2
83 |
84 | leave
85 | .cfi_restore 5
86 | .cfi_def_cfa 4, 4
87 | ret
88 |
89 | .cfi_endproc
90 | .LFE0:
91 | .ident "GCC: (GNU) 6.4.0"
92 |
--------------------------------------------------------------------------------
/configurationFiles/configuration-template-explanation.xml:
--------------------------------------------------------------------------------
1 |
2 |
12 |
13 |
14 |
15 |
16 |
17 |
18 |
19 |
20 |
21 |
22 |
24 |
25 |
26 |
27 |
28 |
29 |
30 |
31 |
32 |
33 |
34 |
35 |
36 |
37 |
38 |
39 |
41 |
42 |
43 |
44 |
45 |
46 |
47 |
48 |
49 |
50 |
55 |
56 |
59 |
60 |
61 |
62 |
63 |
69 |
70 |
71 |
72 |
73 |
74 |
75 |
76 |
77 |
78 |
82 |
83 |
87 |
88 |
92 |
93 |
97 |
98 |
102 |
103 |
107 |
108 |
109 |
110 |
111 |
112 |
113 |
115 |
116 |
144 |
145 |
150 |
151 |
152 |
157 |
158 |
159 |
164 |
165 |
166 |
171 |
172 |
173 |
174 |
180 |
181 |
182 |
188 |
189 |
190 |
196 |
197 |
198 |
204 |
205 |
206 |
211 |
212 |
217 |
218 |
219 |
220 |
221 |
226 |
227 |
228 |
233 |
234 |
235 |
240 |
241 |
242 |
247 |
248 |
249 |
254 |
255 |
256 |
257 |
262 |
263 |
264 |
269 |
270 |
271 |
272 |
279 |
280 |
281 |
282 |
289 |
290 |
291 |
292 |
299 |
300 |
301 |
308 |
309 |
310 |
317 |
318 |
319 |
320 |
327 |
328 |
329 |
330 |
331 |
332 |
333 |
334 |
335 |
337 |
338 |
359 |
360 |
361 |
362 |
371 |
372 |
373 |
374 |
375 |
384 |
385 |
386 |
387 |
396 |
397 |
398 |
407 |
408 |
409 |
418 |
419 |
420 |
429 |
430 |
431 |
440 |
441 |
442 |
443 |
444 |
453 |
454 |
455 |
456 |
457 |
468 |
469 |
470 |
480 |
481 |
482 |
483 |
494 |
495 |
496 |
506 |
507 |
508 |
509 |
518 |
519 |
520 |
521 |
522 |
530 |
531 |
532 |
540 |
541 |
542 |
543 |
544 |
553 |
554 |
555 |
556 |
565 |
566 |
567 |
568 |
577 |
578 |
579 |
580 |
589 |
590 |
591 |
600 |
601 |
602 |
603 |
604 |
--------------------------------------------------------------------------------
/configurationFiles/configuration_ARM_IPC.xml:
--------------------------------------------------------------------------------
1 |
11 |
12 |
13 |
14 |
15 |
16 |
17 |
18 |
19 |
20 |
21 |
23 |
24 |
25 |
26 |
27 |
28 |
29 |
30 |
33 |
34 |
37 |
38 |
41 |
42 |
45 |
48 |
49 |
52 |
53 |
56 |
57 |
58 |
61 |
62 |
66 |
67 |
68 |
69 |
70 |
71 |
72 |
73 |
74 |
79 |
80 |
83 |
84 |
85 |
86 |
87 |
90 |
94 |
95 |
96 |
97 |
98 |
100 |
101 |
102 |
103 |
104 |
105 |
106 |
107 |
108 |
112 |
113 |
117 |
118 |
122 |
123 |
127 |
128 |
132 |
133 |
137 |
138 |
139 |
140 |
141 |
142 |
143 |
144 |
146 |
147 |
175 |
176 |
177 |
182 |
183 |
184 |
189 |
190 |
191 |
196 |
197 |
198 |
203 |
204 |
205 |
206 |
221 |
222 |
223 |
229 |
230 |
231 |
237 |
238 |
239 |
245 |
246 |
247 |
253 |
254 |
255 |
260 |
261 |
266 |
267 |
268 |
269 |
270 |
275 |
276 |
277 |
282 |
283 |
284 |
289 |
290 |
291 |
296 |
297 |
298 |
303 |
304 |
305 |
306 |
311 |
312 |
313 |
318 |
319 |
320 |
321 |
328 |
329 |
330 |
331 |
338 |
339 |
340 |
341 |
348 |
349 |
350 |
357 |
358 |
359 |
366 |
367 |
368 |
369 |
376 |
377 |
378 |
386 |
387 |
388 |
389 |
390 |
398 |
399 |
400 |
401 |
402 |
403 |
404 |
405 |
407 |
408 |
429 |
430 |
431 |
432 |
433 |
441 |
442 |
453 |
454 |
471 |
472 |
479 |
480 |
487 |
488 |
489 |
490 |
499 |
500 |
501 |
510 |
511 |
512 |
513 |
522 |
523 |
524 |
533 |
534 |
535 |
544 |
545 |
546 |
555 |
556 |
557 |
566 |
567 |
568 |
569 |
570 |
579 |
580 |
581 |
582 |
583 |
594 |
595 |
596 |
606 |
607 |
608 |
609 |
620 |
621 |
622 |
632 |
633 |
634 |
635 |
636 |
644 |
645 |
646 |
654 |
655 |
656 |
665 |
666 |
667 |
676 |
677 |
678 |
687 |
688 |
689 |
698 |
699 |
700 |
710 |
711 |
712 |
720 |
721 |
722 |
730 |
731 |
732 |
741 |
742 |
743 |
744 |
745 |
753 |
754 |
755 |
764 |
765 |
766 |
775 |
776 |
777 |
778 |
787 |
788 |
789 |
790 |
799 |
800 |
801 |
802 |
811 |
812 |
813 |
814 |
815 |
816 |
--------------------------------------------------------------------------------
/configurationFiles/configuration_likwidPowerMeter.xml:
--------------------------------------------------------------------------------
1 |
2 |
3 |
13 |
14 |
15 |
16 |
17 |
18 |
19 |
20 |
21 |
22 |
23 |
24 |
26 |
27 |
28 |
29 |
30 |
31 |
32 |
33 |
36 |
37 |
40 |
41 |
44 |
45 |
48 |
51 |
52 |
55 |
56 |
59 |
60 |
61 |
64 |
65 |
69 |
70 |
71 |
72 |
75 |
78 |
79 |
82 |
83 |
84 |
85 |
86 |
89 |
93 |
94 |
95 |
96 |
97 |
98 |
100 |
101 |
102 |
103 |
104 |
105 |
106 |
107 |
108 |
109 |
113 |
114 |
118 |
119 |
123 |
124 |
128 |
129 |
133 |
134 |
138 |
139 |
140 |
141 |
142 |
143 |
144 |
145 |
147 |
148 |
176 |
177 |
182 |
183 |
184 |
189 |
190 |
191 |
196 |
197 |
198 |
199 |
206 |
207 |
208 |
215 |
216 |
217 |
224 |
225 |
226 |
227 |
228 |
229 |
230 |
231 |
233 |
234 |
255 |
256 |
257 |
258 |
267 |
268 |
277 |
278 |
288 |
289 |
297 |
298 |
299 |
300 |
309 |
310 |
311 |
320 |
321 |
322 |
331 |
332 |
333 |
341 |
342 |
343 |
344 |
345 |
346 |
354 |
355 |
356 |
364 |
365 |
366 |
374 |
375 |
376 |
384 |
385 |
386 |
387 |
395 |
396 |
397 |
405 |
406 |
407 |
408 |
416 |
417 |
418 |
419 |
428 |
429 |
430 |
431 |
440 |
441 |
442 |
443 |
444 |
445 |
461 |
462 |
479 |
480 |
481 |
482 |
491 |
492 |
493 |
502 |
503 |
504 |
513 |
514 |
515 |
524 |
525 |
526 |
535 |
536 |
537 |
538 |
594 |
616 |
626 |
627 |
628 |
629 |
731 |
732 |
733 |
734 |
735 |
736 |
--------------------------------------------------------------------------------
/configurationFiles/configuration_x86_IPC.xml:
--------------------------------------------------------------------------------
1 |
2 |
12 |
13 |
14 |
15 |
16 |
17 |
18 |
19 |
20 |
21 |
22 |
23 |
25 |
26 |
27 |
28 |
29 |
30 |
31 |
32 |
35 |
36 |
39 |
40 |
43 |
44 |
47 |
50 |
51 |
54 |
55 |
58 |
59 |
60 |
63 |
64 |
66 |
67 |
68 |
69 |
70 |
71 |
72 |
73 |
74 |
75 |
78 |
81 |
82 |
85 |
86 |
87 |
88 |
89 |
92 |
96 |
97 |
98 |
99 |
100 |
101 |
103 |
104 |
105 |
106 |
107 |
108 |
109 |
110 |
111 |
115 |
116 |
120 |
121 |
125 |
126 |
130 |
131 |
135 |
136 |
140 |
141 |
142 |
143 |
144 |
145 |
146 |
147 |
149 |
150 |
178 |
179 |
184 |
185 |
186 |
191 |
192 |
193 |
194 |
201 |
202 |
203 |
210 |
211 |
212 |
219 |
220 |
221 |
222 |
223 |
224 |
225 |
226 |
228 |
229 |
250 |
251 |
252 |
253 |
254 |
262 |
263 |
264 |
272 |
273 |
274 |
282 |
283 |
284 |
292 |
293 |
294 |
302 |
303 |
304 |
305 |
306 |
307 |
315 |
316 |
317 |
325 |
326 |
327 |
335 |
336 |
337 |
345 |
346 |
347 |
348 |
349 |
350 |
358 |
359 |
360 |
368 |
369 |
370 |
371 |
379 |
380 |
381 |
382 |
391 |
392 |
393 |
394 |
403 |
404 |
405 |
406 |
407 |
408 |
414 |
415 |
416 |
422 |
423 |
424 |
430 |
431 |
432 |
438 |
439 |
440 |
441 |
442 |
450 |
451 |
452 |
460 |
461 |
462 |
470 |
471 |
472 |
480 |
481 |
482 |
490 |
491 |
492 |
500 |
501 |
502 |
510 |
511 |
512 |
520 |
521 |
522 |
530 |
531 |
532 |
540 |
541 |
542 |
543 |
544 |
545 |
546 |
--------------------------------------------------------------------------------
/configurationFiles/measurement/IPC.xml:
--------------------------------------------------------------------------------
1 |
2 |
3 |
13 |
14 |
15 |
16 |
19 |
20 |
23 |
24 |
27 |
28 |
31 |
32 |
33 |
36 |
37 |
38 |
39 |
40 |
41 |
--------------------------------------------------------------------------------
/configurationFiles/measurement/measurementLikwid.xml:
--------------------------------------------------------------------------------
1 |
2 |
3 |
13 |
14 |
15 |
16 |
19 |
20 |
23 |
24 |
27 |
28 |
31 |
32 |
33 |
36 |
37 |
38 |
39 |
40 |
41 |
42 |
--------------------------------------------------------------------------------
/src/Algorithm.py:
--------------------------------------------------------------------------------
1 | '''
2 | Copyright 2019 ARM Ltd. and University of Cyprus
3 | Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction,
4 | including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software,
5 | and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
6 | The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
7 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
8 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
9 | TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
10 | '''
11 |
12 | import subprocess
13 | from xml.dom import minidom
14 | from Instruction import Instruction
15 | from Operand import Operand
16 | from Population import Population
17 | from Individual import Individual
18 | import math
19 | import fileinput
20 | from random import Random
21 | import os
22 | import shutil
23 | import sys
24 | from threading import Timer
25 | from threading import Thread
26 | import time
27 | import pickle;
28 | import re;
29 | import atexit;
30 | import datetime;
31 | from paramiko import SSHClient, client
32 | import paramiko
33 | import socket
34 | import platform
35 | import visa
36 | from statistics import stdev
37 | from multiprocessing.pool import ThreadPool
38 | from multiprocessing import TimeoutError
39 | import importlib
40 |
41 | #import serial;
42 |
43 | class Algorithm(object):
44 |
45 |
46 | UNIFORM_CROSSOVER="0"
47 | ONEPOINT_CROSSOVER="1"
48 | WHEEL_SELECTION="1";
49 | TOURNAMENT_SELECTION="0";
50 |
51 |
52 | def __init__(self,configurationFile,rand=Random()): #reads configuration file and initializes the first population
53 | '''general initialization'''
54 | self.general_initialization(configurationFile, rand)
55 | '''specific initializations'''
56 | self.__instructions_operands_init__()
57 | print("End of inputs\n");
58 |
59 | def general_initialization(self,configurationFile,rand):
60 | '''general algorithm and run parameters'''
61 | self.xmldoc = minidom.parse(configurationFile)
62 | self.intitializeAlgorithmAndRunParameters(configurationFile, rand)
63 | '''create results and saved state directories'''
64 | self.setupDirs(configurationFile)
65 | '''print general and run parameters'''
66 | self.printGeneralInputs();
67 |
68 |
69 | def intitializeAlgorithmAndRunParameters(self,configurationFile,rand):
70 |
71 | ##genetic algorithm parameters
72 | self.populationSize = self.xmldoc.getElementsByTagName('population_size')[0].attributes['value'].value;
73 | self.mutationRate = self.xmldoc.getElementsByTagName('mutation_rate')[0].attributes['value'].value;
74 | self.crossoverType = self.xmldoc.getElementsByTagName('crossover_type')[0].attributes['value'].value;
75 | self.crossoverRate = self.xmldoc.getElementsByTagName('crossover_rate')[0].attributes['value'].value;
76 | self.uniformRate = self.xmldoc.getElementsByTagName('uniform_rate')[0].attributes['value'].value;
77 | self.ellitism = self.xmldoc.getElementsByTagName('ellitism')[0].attributes['value'].value;
78 | self.selectionMethod=self.xmldoc.getElementsByTagName('selectionMethod')[0].attributes['value'].value;#0 wheel,1 tournament
79 | self.tournamentSize = self.xmldoc.getElementsByTagName('tournament_size')[0].attributes['value'].value;
80 | self.populationsToRun= int(self.xmldoc.getElementsByTagName('populations_to_run')[0].attributes['value'].value);
81 |
82 |
83 | ##compilation and measurement parameters
84 | self.fitnessClassName = self.xmldoc.getElementsByTagName('fitnessClass')[0].attributes['value'].value;
85 |
86 | module= importlib.import_module("Fitness."+self.fitnessClassName)
87 | self.fitnessClass = getattr(module,self.fitnessClassName)
88 | self.fitness = self.fitnessClass()
89 |
90 | self.measurementClassName = self.xmldoc.getElementsByTagName('measurementClass')[0].attributes['value'].value;
91 | self.measurementClassConfFile = self.xmldoc.getElementsByTagName('measurementClassConfFile')[0].attributes['value'].value;
92 |
93 |
94 | module= importlib.import_module("Measurement."+self.measurementClassName)
95 | self.measurementClass = getattr(module,self.measurementClassName)
96 | if self.measurementClassConfFile[-4:]!=".xml":
97 | self.measurementClassConfFile=self.measurementClassConfFile+".xml"
98 | self.measurement=self.measurementClass("./configurationFiles/measurement/"+self.measurementClassConfFile)
99 | self.measurement.init()
100 |
101 | self.dirToSaveResults = self.xmldoc.getElementsByTagName('dirToSaveResults')[0].attributes['value'].value;
102 | self.seedDir=self.xmldoc.getElementsByTagName('seedDir')[0].attributes['value'].value;
103 | self.compilationDir = self.xmldoc.getElementsByTagName('compilationDir')[0].attributes['value'].value;
104 |
105 | ##make sure that dirs end with /
106 | self.dirToSaveResults=self.__fixDirEnd__(self.dirToSaveResults)
107 | self.seedDir=self.__fixDirEnd__(self.seedDir)
108 | self.compilationDir=self.__fixDirEnd__(self.compilationDir)
109 |
110 | ##some other variable initialization
111 |
112 | try:
113 | self.saveWholeSource=self.xmldoc.getElementsByTagName('save_whole_source')[0].attributes['value'].value;
114 | except:
115 | self.saveWholeSource=1
116 |
117 | self.population=Population();
118 | self.rand=rand;
119 | self.populationsExamined=1;
120 | self.bestIndividualUntilNow=None;
121 | self.waitCounter=0;
122 | self.populationsTested=0
123 |
124 | def __fixDirEnd__(self,dir):
125 | if dir=="":
126 | return dir
127 | if dir[-1:]!="/":
128 | dir=dir+"/"
129 | return dir
130 |
131 | def setupDirs(self,configurationFile):
132 | '''create the results dirs'''
133 | if(not os.path.exists(self.dirToSaveResults)):
134 | os.mkdir(self.dirToSaveResults);
135 | self.timeStart=datetime.datetime.now().strftime("%y-%m-%d-%H-%M");
136 | self.savedStateDir=self.dirToSaveResults+self.timeStart+"/"
137 | if(not os.path.exists(self.savedStateDir)):
138 | os.mkdir(self.savedStateDir);#create a dir in results dir named after start time that will be used to save state like population and rand state
139 | atexit.register(self.saveRandstate); #register a function which will save rand state at exit
140 |
141 | ''' save configuration src in the results dir '''
142 | if os.path.exists(self.savedStateDir+"configuration.xml"):
143 | os.remove(self.savedStateDir+"configuration.xml");
144 | shutil.copy(configurationFile,self.savedStateDir+"configuration.xml" )
145 |
146 | if os.path.exists(self.savedStateDir+"src"):
147 | shutil.rmtree(self.savedStateDir+"src", ignore_errors=True)
148 | shutil.copytree("./src",self.savedStateDir+"src" )
149 |
150 | ''' save measurement configuration '''
151 | if os.path.exists(self.savedStateDir+"measurement.xml"):
152 | os.remove(self.savedStateDir+"measurement.xml", ignore_errors=True)
153 | shutil.copy("./configurationFiles/measurement/"+self.measurementClassConfFile,self.savedStateDir+"measurement.xml")
154 |
155 | '''save assembly compilation in the results dir'''
156 | if os.path.exists(self.savedStateDir+"assembly_compilation"):
157 | shutil.rmtree(self.savedStateDir+"assembly_compilation", ignore_errors=True)
158 | shutil.copytree(self.compilationDir,self.savedStateDir+"assembly_compilation" )
159 | if os.path.exists(self.savedStateDir+"assembly_compilation/main.s"):
160 | os.remove(self.savedStateDir+"assembly_compilation/main.s")#remove this because they belong to previous run and this can get confusing
161 | self.compilationDir=self.savedStateDir+"assembly_compilation/"
162 | print("New compilationDir is "+self.compilationDir)
163 |
164 |
165 | def __instructions_operands_init__(self):
166 | self.loopSize = self.xmldoc.getElementsByTagName('loopSize')[0].attributes['value'].value;
167 | print ("loop Size: " + self.loopSize);
168 |
169 | self.percentage_clue=self.xmldoc.getElementsByTagName('instruction_percentage_clue')[0].attributes['value'].value;
170 | print("Percentage clue? :"+self.percentage_clue);
171 | self.instruction_types={};#a dictionary of instruction type and the amount of each instruction in loop.. Note useful only when percentage clue is True
172 | self.instructions={}; #a dictionay that has for keeps an array of instructions for every instruction type
173 | self.allInstructionArray=[]; #An array which hold all instructions
174 | self.operands={};
175 | self.toggleInstructionsList={};
176 |
177 | itemList = self.xmldoc.getElementsByTagName("instruction_type");
178 | for instruction_type in itemList:
179 | name=instruction_type.attributes["id"].value;
180 | perc= instruction_type.attributes["perc"].value;
181 | self.instruction_types[name]= int (float(perc) * float(self.loopSize)); '''calculate how many of these instructions will be in the loop'''
182 | if(self.percentage_clue=="True"):
183 | print("amount per instruction type in the loop:")
184 | print (self.instruction_types);
185 | sum=0;
186 | for value in list(self.instruction_types.values()):
187 | sum+=value;
188 | self.loopSize=sum;
189 | print("actual loop size is "+str(self.loopSize));
190 |
191 | itemList = self.xmldoc.getElementsByTagName("operand");
192 | print("Available operands\n")
193 | for operandDesc in itemList:
194 | ins_type=operandDesc.attributes["type"].value;
195 | if (ins_type=="immediate" or ins_type=="constant" or ins_type=="automatically_incremented_operand"):
196 | anOperand = Operand(id=operandDesc.attributes["id"].value,type=operandDesc.attributes["type"].value,values=[],min=operandDesc.attributes["min"].value,max=operandDesc.attributes["max"].value,stride=operandDesc.attributes["stride"].value,toggleable=operandDesc.attributes["toggle"].value);
197 | #elif ins_type=="branch_label":
198 | # anOperand = BranchLabel(id=operandDesc.attributes["id"].value,ins_type=operandDesc.attributes["type"].value,values=operandDesc.attributes["values"].value.split(),min=operandDesc.attributes["min"].value,max=operandDesc.attributes["max"].value,stride=operandDesc.attributes["stride"].value,toggleable=operandDesc.attributes["toggle"].value);
199 | else:
200 | anOperand = Operand(id=operandDesc.attributes["id"].value,type=operandDesc.attributes["type"].value,values=operandDesc.attributes["values"].value.split(),toggleable=operandDesc.attributes["toggle"].value);
201 |
202 | print("id "+anOperand.id.__str__());
203 | print ("values ");
204 | print(anOperand.values);
205 | print("max "+ anOperand.max.__str__())
206 | print ("min " + anOperand.min.__str__())
207 | print("stride "+anOperand.stride.__str__()+"\n")
208 | self.operands[anOperand.id] = anOperand;
209 | print("End of available operands\n")
210 |
211 |
212 | itemList = self.xmldoc.getElementsByTagName("instruction");
213 | print("Available instructions\n")
214 | for instructionDesc in itemList:
215 | name=instructionDesc.attributes["name"].value;
216 | ins_type=instructionDesc.attributes["type"].value;
217 | numOfOperands=instructionDesc.attributes["num_of_operands"].value;
218 | if "format" in instructionDesc.attributes:
219 | anInstruction = Instruction(name,ins_type,numOfOperands,format=instructionDesc.attributes["format"].value,toggleable=instructionDesc.attributes["toggle"].value);
220 | else:
221 | print("Instruction "+name+"doesnt have format specified.. All instructions must have format... Exitting");
222 | sys.exit();
223 | #anInstruction = Instruction(name,ins_type,numOfOperands,toggleable=instructionDesc.attributes["toggle"].value);
224 |
225 | if(instructionDesc.attributes["toggle"].value=="True"):
226 | self.toggleInstructionsList[instructionDesc.attributes["name"].value]=1;
227 |
228 | operands=[];#TODO fix this source of bugs.. It's irritating when the num of operands does not match the operands specified in the xml file. and in general is stupid to how the operands specified and the number of operands at the same time.. this redundancy leads to bugs
229 | for i in range(1,int(anInstruction.numOfOperands)+1):
230 | operands.append(self.operands[(instructionDesc.attributes["operand"+i.__str__()].value)].copy());
231 | anInstruction.setOperands(operands);
232 | print(anInstruction); ##for debugging
233 | self.instructions.setdefault(anInstruction.ins_type,[]).append(anInstruction);
234 | #print (self.instructions);
235 | print("End of available instructions\n");
236 |
237 | for array in list(self.instructions.values()):
238 | for ins in array:
239 | self.allInstructionArray.append(ins);
240 | print("register initialization");
241 | #print(self.registerInitStr);
242 | #sys.exit()
243 |
244 |
245 | def printGeneralInputs(self):
246 | print("Debug Inputs");
247 | print ("Population size: " +self.populationSize);
248 | print ("Mutation Rate: "+self.mutationRate);
249 | print ("Crossover Rate: "+self.crossoverRate);
250 | if(self.crossoverType==Algorithm.ONEPOINT_CROSSOVER):
251 | print("Crossover Type: one point crossover")
252 | elif(self.crossoverType==Algorithm.UNIFORM_CROSSOVER):
253 | print("Crossover Type: uniform crossover")
254 | print ("Uniform Rate: "+self.uniformRate);
255 | print ("Ellitism: "+self.ellitism);
256 | if(self.selectionMethod==Algorithm.TOURNAMENT_SELECTION):
257 | print("Selection Method: Tournament")
258 | print ("Tournament selection size: "+self.tournamentSize);
259 | elif(self.selectionMethod==Algorithm.WHEEL_SELECTION):
260 | print("Selection Method: Wheel Selection")
261 | #print ("Termination Conditions are: ");
262 | #print("If for "+self.populationsToRun+" populations there is no more than "+self.fitnessThreshold+" improvement stop")
263 | print ("ResultsDir: "+self.dirToSaveResults);
264 | print ("compilationDir: "+self.compilationDir);
265 |
266 |
267 | @staticmethod
268 | def returnRunType(configurationFile):
269 | xmldoc = minidom.parse(configurationFile)
270 | run_type = xmldoc.getElementsByTagName('run_type')[0].attributes['value'].value;
271 | if(run_type==Algorithm.PROFILE_RUN):
272 | return Algorithm.PROFILE_RUN;
273 | else:
274 | return Algorithm.INSTRUCTION_RUN
275 |
276 | def saveRandstate(self,postfix=""):
277 | output=open(self.savedStateDir +"rand_state"+postfix+".pkl","wb");
278 | pickle.dump(self.rand.getstate(),output);
279 | output.close();
280 |
281 | def loadRandstate(self):
282 | latest=1;
283 | if(not os.path.exists(self.seedDir+"rand_state.pkl")):
284 | for root, dirs, filenames in os.walk(self.seedDir):
285 | for f in filenames:
286 | if("rand_state" in f):
287 | num=int(f.replace("rand_state","").replace(".pkl",""));
288 | if(num>latest):
289 | latest=num;
290 | stateToLoad="rand_state"+str(latest)+".pkl";
291 | input=open(self.seedDir+stateToLoad,"rb");
292 | self.rand.setstate(pickle.load(input));
293 | input.close();
294 | else:
295 | input=open(self.seedDir+"rand_state.pkl","rb");
296 | self.rand.setstate(pickle.load(input));
297 | input.close();
298 |
299 | '''def __initializeRegisterValues__ (self):#DEPRECATED DON"T USE THIS ANYMORE AS IT MAY LEAD TO BUGS ##remember this functions expects a clean original copy of main.s
300 | for line in fileinput.input(self.compilationDir+"/main.s", inplace=1): #TODO keep in mind that it works with and without /
301 | print(line,end="");
302 | if "reg init" in line:
303 | print(self.registerInitStr);
304 | fileinput.close();
305 |
306 | def __initializeMemory__ (self,core=1): #DEPRECATED DON"T USE THIS ANYMORE AS IT MAY LEAD TO BUGS ##remember this functions expects a clean original copy of main.s
307 | for line in fileinput.input(self.compilationDir+"/startup.s", inplace=1):
308 | if "MemoryInit"+str(core) in line:
309 | print (line,end="");
310 | print(self.memoryInitArray[core-1]);
311 | else:
312 | print(line,end="");
313 | fileinput.close();'''
314 |
315 | def createInitialPopulation (self):
316 | individuals=[];
317 | if self.seedDir == "": #random initialization
318 | for i in range(int(self.populationSize)):
319 | individuals.append(self.__randomlyCreateIndividual__());
320 | self.population=Population(individuals);
321 | else: #initial population based on existing individuals.. Useful for continuing runs that where stopped
322 | newerPop=0; ##find which is the newest population
323 | for root, dirs, filenames in os.walk(self.seedDir):
324 | for f in filenames:
325 | if((".pkl" in f) and ("rand" not in f)):
326 | tokens=re.split('[_.]',f);
327 | popNum=int(tokens[0]);
328 | if (popNum>newerPop):
329 | newerPop=popNum;
330 | input = open(self.seedDir+str(newerPop)+'.pkl',mode="rb");
331 | self.population=Population.unpickle(input);
332 | input.close();
333 |
334 | maxId=0;##track the maxId
335 | for indiv in self.population.individuals:
336 | if(indiv.myId>maxId):
337 | maxId=indiv.myId;
338 |
339 | Individual.id=maxId; #the new indiv will start from maxId
340 | self.bestIndividualUntilNow=self.population.getFittest(); #set the best individual seen until now
341 | self.loopSize=self.bestIndividualUntilNow.getInstructions().__len__(); #ensure that loop Size is correct
342 | self.populationsExamined=newerPop+1;#population will start from the newer population
343 | self.loadRandstate(); #load the previous rand state before evolving pop
344 | self.evolvePopulation(); #immediately evolve population
345 |
346 |
347 |
348 | return;
349 |
350 | def measurePopulation (self):
351 | for individual in self.population.individuals:
352 | ##NOTE Due to fluctuations in measurements is desirable to measure the best individual again
353 | while True: #measure until measurement succesful...
354 | try:
355 | measurements=self.__measureIndividual__(individual);
356 | measurement=measurements[0];
357 | measurement_str="%.6f" % float(measurement);
358 | break;
359 | except (ValueError,IOError):
360 | continue;
361 |
362 | individual.setMeasurementsVector(measurements);
363 | fitnessArray=self.fitness.getFitness(individual)
364 | fitnessValue=fitnessArray[0]
365 | individual.setFitness(fitnessValue)
366 |
367 | measurementStr=""
368 |
369 | for measurement in fitnessArray:
370 | measurement_str=("%.6f" % float(measurement)).replace(".","DOT").strip()+"_"
371 | measurementStr=measurementStr+measurement_str
372 |
373 | if int(self.saveWholeSource)==1:
374 | fpath=""
375 | if(individual.belongsToInitialSeed()):
376 | fpath=self.dirToSaveResults+str(individual.generation)+"_"+str(individual.myId)+"_"+measurementStr+"0_0" +".txt"
377 | else:
378 | fpath=self.dirToSaveResults+str(individual.generation)+"_"+str(individual.myId)+"_"+measurementStr+str(individual.parents[0].myId)+"_"+str(individual.parents[1].myId) +".txt"
379 | shutil.copy(self.compilationDir+"/main.s",fpath)
380 | else:
381 | if(individual.belongsToInitialSeed()):
382 | f = open(self.dirToSaveResults+str(individual.generation)+"_"+str(individual.myId)+"_"+measurementStr+"0_0" +".txt",mode="w")
383 | else:
384 | f = open(self.dirToSaveResults+str(individual.generation)+"_"+str(individual.myId)+"_"+measurementStr+str(individual.parents[0].myId)+"_"+str(individual.parents[1].myId) +".txt",mode="w")
385 | f.write(individual.__str__());
386 | f.close();
387 |
388 |
389 | individual.clearParents();#TODO this is just a cheap hack I do for now in order to avoid the recursive grow in length of .pkl files. This is only okay given that I don't actually need anymore that parents.. fon now is okay
390 | ##save a file describing the population so it can be loaded later. useful in case of you want to start a run based on the state of previous runs
391 | output = open(self.savedStateDir+str(self.populationsExamined)+'.pkl','wb');
392 | self.population.pickle(output);
393 | output.close();
394 |
395 | ##also save the rand_state
396 | self.saveRandstate(postfix=str(self.populationsExamined));
397 | self.populationsExamined=self.populationsExamined+1;
398 | self.populationsTested=self.populationsTested+1
399 |
400 | def __measureIndividual__(self,individual):
401 | #####before each individual bring back the original copy of main and startup
402 | self.__bring_back_code_template__();
403 | ####initialize register values in main.s
404 | #self.__initializeRegisterValues__();
405 | ####dump memory initialization in startup.s for each core
406 | #for core in range(1,int(int(self.cores)+1)):
407 | # self.__initializeMemory__(core);
408 | ##apply toggling on operands
409 | for key in list(self.toggleInstructionsList.keys()):
410 | self.toggleInstructionsList[key]=1; #first initialize dictionay
411 | for ins in individual.getInstructions():
412 | if(ins.toggleable=="True"):
413 | if(int(self.toggleInstructionsList[ins.name])%2==1):
414 | ins.toggle(0);
415 | else:
416 | ins.toggle(1);
417 | self.toggleInstructionsList[ins.name]+=1;
418 | #dump the individual into mail loop of main.s
419 | for line in fileinput.input(self.compilationDir+"/main.s", inplace=1):
420 | if "loop_code" in line:
421 | print(individual);
422 | else:
423 | print(line,end="");
424 | fileinput.close();
425 | '''at last do the actual measurement'''
426 | measurements= self.__doTheMeasurement__()
427 | return measurements;
428 |
429 | '''def __saveIndiv__(self,individual):
430 | #####before each individual bring back the original copy of main and startup
431 | self.__bring_back_code_template__();
432 | ####initialize register values in main.s
433 | #self.__initializeRegisterValues__();
434 | ####dump memory initialization in startup.s for each core
435 | #for core in range(1,int(int(self.cores)+1)):
436 | # self.__initializeMemory__(core);
437 | ##apply toggling on operands
438 | for key in list(self.toggleInstructionsList.keys()):
439 | self.toggleInstructionsList[key]=1; #first initialize dictionay
440 | for ins in individual.getInstructions():
441 | if(ins.toggleable=="True"):
442 | if(int(self.toggleInstructionsList[ins.name])%2==1):
443 | ins.toggle(0);
444 | else:
445 | ins.toggle(1);
446 | self.toggleInstructionsList[ins.name]+=1;
447 | #dump the individual into mail loop of main.s
448 | for line in fileinput.input(self.compilationDir+"/main.s", inplace=1):
449 | if "loop_code" in line:
450 | print(individual);
451 | else:
452 | print(line,end="");
453 | fileinput.close();'''
454 |
455 |
456 | def __bring_back_code_template__(self):
457 | #####before each individual bring back the original copy of main and startup
458 | if(os.path.exists(self.compilationDir +"main.s")):
459 | os.remove(self.compilationDir +"main.s")
460 | shutil.copy(self.compilationDir +"main_original.s", self.compilationDir +"main.s")
461 |
462 |
463 | if(os.path.exists(self.compilationDir +"startup.s")):
464 | os.remove(self.compilationDir +"startup.s")
465 | #shutil.copy(self.compilationDir +"startup_original.s", self.compilationDir +"startup.s")
466 |
467 |
468 | def __doTheMeasurement__(self):
469 | self.measurement.setSourceFilePath(self.compilationDir+"/main.s")
470 | return self.measurement.measure()
471 |
472 | def __randomlyCreateIndividual__ (self):
473 | instruction_sequence=[]
474 | if self.percentage_clue=="True":
475 | for ins_type in self.instruction_types.keys():##for each instruction type
476 | for i in range(self.instruction_types[ins_type]): ##create as many instructions as written in the hash
477 | instructions=self.instructions[ins_type];
478 | instruction_to_copy=instructions[self.rand.randint(0,instructions.__len__()-1)]; #choose random one instruction from the available types
479 | instruction=instruction_to_copy.copy();
480 | instruction.mutateOperands(self.rand); ##initialize randomy the instruction operands
481 | instruction_sequence.append(instruction);
482 | #print(instruction);
483 | self.rand.shuffle(instruction_sequence);
484 | else:
485 | for i in range(int(self.loopSize)):
486 | instruction=self.rand.choice(self.allInstructionArray).copy();
487 | instruction.mutateOperands(self.rand);
488 | instruction_sequence.append(instruction);
489 | newIndividual = Individual (instruction_sequence,self.populationsExamined)
490 | return newIndividual;
491 |
492 | def areWeDone(self):
493 |
494 | '''if self.populationsToRun>0:
495 | if self.populationsToRun==self.populationsTested:
496 | self.__saveIndiv__(self.bestIndividualUntilNow)
497 | return True
498 | current_population_best=self.population.getFittest();
499 |
500 | if(self.bestIndividualUntilNow is None): #only for the first time
501 | self.bestIndividualUntilNow=current_population_best;
502 | self.waitCounter=0;
503 | return False;
504 |
505 | if float(self.best_pop_target)>0 and current_population_best.getFitness()>=float(self.best_pop_target):
506 | #SAVE BEST INDIV SOURCE
507 | self.__saveIndiv__(current_population_best)
508 | return True
509 |
510 | if float(self.avg_pop_target)>0 and self.population.getAvgFitness()>=float(self.avg_pop_target):
511 | return True
512 |
513 |
514 | if (float(current_population_best.getFitness()) < float(self.bestIndividualUntilNow.getFitness())):
515 | self.waitCounter=self.waitCounter+1;
516 | else:
517 | improvement = float(current_population_best.getFitness()) / self.bestIndividualUntilNow.getFitness();
518 | if (improvement - 1) < float(self.fitnessThreshold):
519 | self.waitCounter=self.waitCounter+1;
520 | else:
521 | self.waitCounter=0;
522 | self.bestIndividualUntilNow=current_population_best;'''
523 | self.waitCounter=self.waitCounter+1
524 | if int(self.waitCounter) == int(self.populationsToRun):
525 | return True;
526 | else:
527 | return False;
528 |
529 | def __roulletteWheelSelection__ (self):
530 | individuals=self.population.individuals;
531 | turn=self.rand.randint(0,individuals[individuals.__len__()-1].cumulativeFitness);
532 | for indiv in self.population.individuals:
533 | if(int(indiv.cumulativeFitness)>=turn):
534 | return indiv; #return the first indiv that is equal or bigger to the random generated value
535 |
536 | def __tournamentSelection__(self):
537 | tournamentIndiv=[];
538 | for j in range(0,int(self.tournamentSize)):
539 | tournamentIndiv.append(self.population.pickRandomlyAnIndividual(self.rand));
540 | tournamentPop=Population(tournamentIndiv);
541 | return tournamentPop.getFittest();
542 |
543 | def evolvePopulation (self):
544 | #individuals=[0]*int(self.populationSize);
545 | individuals=[];
546 | individuals2=[];
547 | self.bestIndividualUntilNow=self.population.getFittest()
548 | if self.ellitism=="true": #TODO make the choice to keep more individuals from previous populations TODO FIX THE true
549 | individuals.append(self.bestIndividualUntilNow);
550 | self.bestIndividualUntilNow.generation+=1; #For the next measurement the promoted individuals will be recorded as individuals of the next population.. Is just for avoiding confusions when processing the results
551 | childsCreated=1;
552 | else:
553 | childsCreated=0;
554 | if(self.selectionMethod==Algorithm.WHEEL_SELECTION):
555 | self.population.keepHalfBest();##sightly algorithm change apply roullete on best 50%
556 | self.population.setCumulativeFitness();
557 |
558 | while childsCreatedcrossover_point):
627 | #do crossover
628 | loop_code1.append(individual2.getInstruction(i).copy());
629 | loop_code2.append(individual1.getInstruction(i).copy());
630 | else: #keep instruction as it is
631 | loop_code1.append(individual1.getInstruction(i).copy());
632 | loop_code2.append(individual2.getInstruction(i).copy());
633 | children=[];
634 | children.append(Individual(sequence=loop_code1,generation=self.populationsExamined));
635 | children.append(Individual(sequence=loop_code2,generation=self.populationsExamined));
636 | children[0].setParents(individual1,individual2); ##the first parent is the code that remains the same.. the second parent is the code that came after crossover
637 | children[1].setParents(individual2,individual1);
638 | return children;
639 |
640 | #def getFittest(self):
641 | # return self.bestIndividualUntilNow;
642 |
643 |
--------------------------------------------------------------------------------
/src/Fitness/DefaultFitness.py:
--------------------------------------------------------------------------------
1 | '''
2 | Copyright 2019 ARM Ltd. and University of Cyprus
3 | Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction,
4 | including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software,
5 | and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
6 | The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
7 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
8 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
9 | TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
10 | '''
11 |
12 | class DefaultFitness(object):
13 | '''
14 | classdocs
15 | '''
16 |
17 | '''
18 | Constructor
19 | '''
20 | def __init__(self):
21 |
22 |
23 | '''Override getFitness to specify other fitness functions
24 | should be called after individual measurements are set (in other words the individual object instance at this point should carry the measurements with it)
25 | The getFitness function can be very sophisticated
26 | e.g. take in account instruction count instruction mix, various metrics like perf counters power etc
27 | the getfitness should return an array where the first the first item MUST be the finess value, followed by any other useful information the user would like to see on file name of the indiviudal
28 | such as specific components ued to calculate the fitness and the individual measurements (like in the below example)
29 | To use the user defined fitness function specify the class name in the fitnessClass parameter in the configuration file
30 | '''
31 |
32 | def getFitness(self,individual):
33 | fitness=individual.getMeasurements()[0]
34 | toReturn=[]
35 | toReturn.append(fitness)
36 | for value in individual.getMeasurements():
37 | toReturn.append(value)
38 | return toReturn
--------------------------------------------------------------------------------
/src/Fitness/SimplicityTempFitness.py:
--------------------------------------------------------------------------------
1 | '''
2 | Copyright 2019 ARM Ltd. and University of Cyprus
3 | Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction,
4 | including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software,
5 | and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
6 | The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
7 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
8 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
9 | TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
10 | '''
11 |
12 | class SimplicityTempFitness(object): # a fitness function for maximizing both temperature and instruction stream simplicity
13 | '''
14 | classdocs
15 | '''
16 |
17 | '''
18 | Constructor
19 | '''
20 | def __init__(self):
21 | self.MAX_TEMP=100
22 | self.BASELINE_TEMP=40
23 | self.MAX_TEMP_SCORE=self.MAX_TEMP-self.BASELINE_TEMP
24 | self.percentages=[0.5,0.5]
25 |
26 |
27 | '''Override this function to specify other fitness functions
28 | should be called after individual measurements are set (in other words the individual object instance at this point should carry the measurements with it)
29 | The getFitness function can be very sophisticated
30 | e.g. take in account instruction count instruction mix, various metrics like perf counters power etc
31 | the getfitness should return an array where the first the first item MUST be the finess value, followed by any other useful information the user would like to see on file name of the indiviudal
32 | such as specific components ued to calculate the fitness and the individual measurements (like in the below example)
33 | To use the user defined fitness function specify the class name in the fitnessClass parameter in the configuration file
34 | '''
35 |
36 | def getFitness(self,individual):
37 | insHash={}
38 | instructions=individual.getInstructions()
39 | totalInstructions=len(instructions)
40 |
41 | for ins in instructions:
42 | insHash[ins.name]=1
43 | uniqueIns=len(insHash.keys())
44 |
45 | tempScore=(individual.getMeasurements()[0]-self.BASELINE_TEMP) / self.MAX_TEMP_SCORE
46 | tempScoreWeighted=tempScore*self.percentages[0]
47 |
48 | instructionsScore=(totalInstructions - uniqueIns)/totalInstructions
49 | instructionsScoreWeighted=instructionsScore*self.percentages[1]
50 |
51 | fitness=tempScoreWeighted+instructionsScoreWeighted
52 |
53 | toReturn=[]
54 | toReturn.append(fitness)
55 | toReturn.append(tempScoreWeighted)
56 | toReturn.append(instructionsScoreWeighted)
57 | for value in individual.getMeasurements():
58 | toReturn.append(value)
59 | return toReturn
--------------------------------------------------------------------------------
/src/Individual.py:
--------------------------------------------------------------------------------
1 | '''
2 | Copyright 2019 ARM Ltd. and University of Cyprus
3 | Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction,
4 | including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software,
5 | and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
6 | The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
7 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
8 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
9 | TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
10 | '''
11 | import pickle;
12 | import copy;
13 |
14 | class Individual(object):
15 | '''
16 | classdocs
17 | '''
18 | id=0;
19 |
20 | def __init__(self,sequence=[],generation=0): #Note This code assumes that instruction operands are initiated (mutated) from before
21 | Individual.id+=1;
22 | self.myId=Individual.id;
23 | self.sequence=sequence;
24 | self.fitness=0.0;
25 | #self.secondaryMeasurement=0.0;
26 | self.measurements=[]
27 | self.branchLabels={};
28 | self.cumulativeFitness=0.0; #for wheel selection
29 | self.parents=[];
30 | self.generation=generation;
31 | self.fixUnconditionalBranchLabels();
32 | return;
33 |
34 | def addInstruction(self,anInstruction):
35 | self.sequence.append(anInstruction);
36 |
37 | def getInstruction(self,index):
38 | return self.sequence[index];
39 |
40 | def getInstructions(self):
41 | return self.sequence;
42 |
43 | def fixUnconditionalBranchLabels(self):
44 | automatically_incremented={};
45 | for ins in self.sequence:
46 | for operand in ins.getOperands():
47 | if(operand.type=="automatically_incremented_operand"):
48 | if(operand.id in automatically_incremented.keys()):
49 | automatically_incremented[operand.id]+=int(operand.stride);
50 | else:
51 | automatically_incremented[operand.id]=int(operand.min);
52 | operand.currentValue=automatically_incremented[operand.id];
53 |
54 | def setMeasurementsVector(self,measurements):#by convention the first item of the vector is the fitness value
55 | self.measurements=measurements
56 |
57 | def setFitness(self,fitness):
58 | self.fitness=fitness
59 |
60 | '''def setMeasurementsVector(self,measurements):
61 | self.measurements=measurements
62 | return;'''
63 |
64 | def setCumulativeFitness(self,fitness): #for wheel selection
65 | self.cumulativeFitness=fitness;
66 |
67 | def getFitness(self):
68 | try:
69 | return self.fitness; #by convention the first item of the vector is the fitness value
70 | except:
71 | return self.measurement #to support continuing runs from legacy population pkls
72 | #def getSecondaryMeasurement(self):
73 | # return self.secondaryMeasurement;
74 | def getMeasurements(self):
75 | return self.measurements
76 |
77 | def setParents(self,par1,par2): ##the first parent is the code that remains the same.. the second parent is the code that came after crossover
78 | self.parents.append(par1);
79 | self.parents.append(par2);
80 |
81 | def clearParents(self):
82 | self.parents=None;
83 |
84 | def belongsToInitialSeed(self):
85 | if self.parents:
86 | return False;
87 | else:
88 | return True;
89 |
90 | def __str__(self):
91 | index=0;
92 | output=""
93 | for ins in self.sequence:
94 | output+=str("\t"+ins.__str__()+"\n");
95 | if str(index) in self.branchLabels.keys():
96 | output+=str(self.branchLabels[str(index)]+"\n");
97 | index+=1;
98 | return output;
99 |
100 | def __cmp__(self,other):
101 | if self.getFitness() < other.getFitness():
102 | return -1
103 | elif self.getFitness()> other.getFitness():
104 | return 1
105 | else:
106 | return 0
107 |
108 | def pickle(self,filename):
109 | pickle.dump(self, filename);
110 |
111 | @staticmethod
112 | def unpickle(filename):
113 | return pickle.load(filename);
114 |
115 | def copy(self):
116 | return copy.deepcopy(self);
117 |
--------------------------------------------------------------------------------
/src/Instruction.py:
--------------------------------------------------------------------------------
1 | '''
2 | Copyright 2019 ARM Ltd. and University of Cyprus
3 | Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction,
4 | including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software,
5 | and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
6 | The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
7 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
8 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
9 | TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
10 | '''
11 | import copy
12 | import sys;
13 |
14 |
15 | class Instruction(object):
16 | '''
17 | classdocs
18 | '''
19 |
20 |
21 | def __init__(self,name,ins_type,numOfOperands,operands=[],format="op1,op2,op3",toggleable="False"):
22 | '''
23 | Constructor
24 | '''
25 | self.name = name;
26 | self.ins_type = ins_type;
27 | self.operands = operands;
28 | self.format=format;
29 | self.format=self.format.replace("\\n","\n"); #replace double backslashes with one backslash to make sure atomic sequences work. for some reason the xml parsing by default ads a backslash to escape characters
30 | self.format=self.format.replace("\\t","\t");
31 | self.numOfOperands = numOfOperands;
32 | self.toggleable=toggleable;
33 |
34 | def copy(self):
35 | return copy.deepcopy(self);
36 |
37 |
38 | def setOperands (self,operands):
39 | self.operands=operands;
40 |
41 | def getOperand(self,index=0):
42 | if (index>=self.operands.__len__() or index <0):
43 | print("error index out of bounds")
44 | sys.exit;
45 | return self.operands[index];
46 |
47 | def getOperands(self):
48 | return self.operands;
49 |
50 | def toggle(self,value_index): #It's the algorithm class job to provide the right indexes for toggling
51 | for op in self.operands:
52 | if(op.toggleable=="True"):
53 | op.setCurrentValueByIndex(value_index);
54 |
55 |
56 | def mutateOperands (self,rand):##TODO maybe give an option of changing particular operands
57 | for op in self.operands:
58 | op.mutate(rand);
59 |
60 | ''' def toggle(self): #TODO think if you really need this
61 | '''
62 |
63 |
64 | def __str__ (self):
65 |
66 | if( (str(self.numOfOperands).strip() == "0") or (self.operands[0].currentValue!= "") ):
67 | representation=self.format
68 | for i in range(0,self.operands.__len__()):
69 | toReplace="op"+str(i+1);
70 | representation=representation.replace(toReplace.__str__(),str(self.operands[i].__str__()));
71 |
72 | #representation=self.name + " " + representation
73 | return representation
74 | else:
75 | representation="name "+self.name + "\ntype "+self.ins_type +"\nformat "+self.format+"\nnumOfOperands "+self.numOfOperands+"\n";
76 | for i in range(int(self.numOfOperands)):
77 | representation+=("\t"+str(self.operands[i])+"\n");
78 | return representation;
79 |
80 |
--------------------------------------------------------------------------------
/src/Measurement/Measurement.py:
--------------------------------------------------------------------------------
1 | '''
2 | Copyright 2019 ARM Ltd. and University of Cyprus
3 | Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction,
4 | including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software,
5 | and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
6 | The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
7 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
8 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
9 | TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
10 | '''
11 |
12 | from abc import ABC, abstractmethod
13 | from xml.dom import minidom
14 | from paramiko import SSHClient, client
15 | import paramiko
16 | import socket
17 | import platform
18 | import visa
19 | import os
20 |
21 | class Measurement(ABC):
22 | '''
23 | classdocs
24 | '''
25 |
26 | def __init__(self, confFile):
27 | '''
28 | Constructor
29 | '''
30 | self.confFile=confFile
31 | self.xmldoc = minidom.parse(confFile)
32 |
33 | #most of the below are expected to be initialized in init function (should be called after constructor)
34 | self.targetRunDir= None
35 | self.targetHostname= None
36 | self.targetSSHusername= None
37 | self.targetSSHpassword = None
38 | self.coresToUse=None
39 | self.sourceFilePath=None #to be set in setSourceFilePath funtion
40 | super().__init__() #abstract class init
41 |
42 | def init(self): #should be called after constructor.. this can be overridden by child measurement classes to add new or use other configuration parameters..
43 |
44 | self.targetRunDir= self.tryGetStringValue('targetRunDir')
45 | self.targetHostname= self.tryGetStringValue('targetHostname')
46 | self.targetSSHusername= self.tryGetStringValue('targetSSHusername')
47 | self.targetSSHpassword = self.tryGetStringValue('targetSSHpassword')
48 | coresToUseString=self.tryGetStringValue('coresToUse')
49 | self.coresToUse=[]
50 | for core in coresToUseString.split(" "):
51 | self.coresToUse.append(int(core))
52 |
53 | def setSourceFilePath(self,sourceFilePath): #should be called before measurement or in the begining of the GA run if the source file path doesn't changes
54 | self.sourceFilePath=sourceFilePath
55 |
56 | ##helper functions to make clearer the code.. on exception the code doesn't terminate immediately but it does produce a warning message..
57 | ##This is the case because sometimes this might be desirable based on the functionality.. For instance bare metal runs won't use the ssh parameters
58 | def tryGetStringValue(self,key):
59 | try:
60 | value=self.xmldoc.getElementsByTagName(key)[0].attributes['value'].value;
61 | return value
62 | except:
63 | print("Warning failed to read "+str(key))
64 |
65 | def tryGetIntValue(self,key):
66 | try:
67 | value=int(self.xmldoc.getElementsByTagName(key)[0].attributes['value'].value);
68 | return value
69 | except:
70 | print("Warning failed to read "+str(key))
71 |
72 | def tryGetFloatValue(self,key):
73 | try:
74 | value=float(self.xmldoc.getElementsByTagName(key)[0].attributes['value'].value);
75 | return value
76 | except:
77 | print("Warning failed to read "+str(key))
78 |
79 | #this function should return an array of results.. at least one item should be returned.. the defaultFitness.py class (that calculates indidual's fitness) assumes by convention that the first array
80 | #item is the fitness value
81 | @abstractmethod
82 | def measure(self):
83 | pass
84 |
85 | ## utility function for executing commands over ssh connection.. very common functionality
86 | def executeSSHcommand(self,command,continousAttempt=True,max_tries=10):
87 | tries=0
88 | while True:
89 | try:
90 | ssh = SSHClient()
91 | ssh.set_missing_host_key_policy(client.AutoAddPolicy())
92 | ssh.connect(self.targetHostname, username=self.targetSSHusername, password=self.targetSSHpassword)
93 | stdin,stdout,stderr =ssh.exec_command(command)
94 | lines=[]
95 | for line in stdout.readlines():
96 | lines.append(line)
97 | ssh.close()
98 | return lines
99 |
100 | except:
101 | if continousAttempt and tries/dev/null;"
31 | execution_command="cd "+self.targetRunDir + " ; ./individual & perf stat -e instructions,cycles -o tmp -p $! sleep "+str(self.timeToMeasure) +" ; pkill individual &> /dev/null;"
32 | output_command="cd "+self.targetRunDir + " ; cat tmp | grep insn | tr ',' '.' | awk '{print $4}'; rm main.s; rm individual; rm tmp; ";
33 | super().executeSSHcommand(compilation_command)
34 | super().executeSSHcommand(execution_command)
35 | stdout=super().executeSSHcommand(output_command)
36 |
37 | ipc=0
38 |
39 | for line in stdout:
40 | #print ("line is "+line)
41 | try:
42 | test=float(line)
43 | ipc=test
44 | except ValueError:
45 | print ("Exception line not ipc")
46 |
47 |
48 | measurements=[];
49 | measurements.append(ipc);
50 | return measurements;
51 | #return ipc;
52 |
--------------------------------------------------------------------------------
/src/Measurement/MeasurementLikwidPower.py:
--------------------------------------------------------------------------------
1 | '''
2 | Copyright 2019 ARM Ltd. and University of Cyprus
3 | Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction,
4 | including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software,
5 | and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
6 | The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
7 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
8 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
9 | TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
10 | '''
11 |
12 | from Measurement.Measurement import Measurement
13 |
14 | class MeasurementLikwidPower(Measurement):
15 | '''
16 | classdocs
17 | '''
18 |
19 | def __init__(self,confFile):
20 | super().__init__(confFile)
21 |
22 |
23 | def init(self):
24 | super().init()
25 | self.timeToMeasure = self.tryGetIntValue('time_to_measure')
26 |
27 | #the below code is tested to work with likwid 4.3 and 3.1 versions
28 |
29 | def measure(self):
30 |
31 | super().copyFileOverFTP()
32 | compilation_command="cd "+self.targetRunDir + " ; gcc main.s -o individual &>/dev/null;"
33 | execution_command="cd "+self.targetRunDir+" ; "
34 | for core in self.coresToUse:
35 | execution_command+="taskset -c "+str(core)+" ./individual &>/dev/null & "
36 | execution_command+=" sudo likwid-powermeter -s "+str(self.timeToMeasure) +"s > tmp ; pkill individual &> /dev/null;" #make sure that msr module is loaded (modprobe msr) and sudo without password is enabled
37 | output_command="cd "+self.targetRunDir + " ; cat tmp | grep Watt | head -n 1 | awk '{print $3}'; rm main.s; rm individual; rm tmp; "; #this grabs the package power
38 | super().executeSSHcommand(compilation_command)
39 | super().executeSSHcommand(execution_command)
40 | stdout=super().executeSSHcommand(output_command)
41 |
42 |
43 | for line in stdout:
44 | try:
45 | test=float(line)
46 | power_meas=test
47 | except ValueError:
48 | print ("Exception line not power")
49 |
50 |
51 | measurements=[];
52 | measurements.append(power_meas);
53 |
54 | return measurements;
55 |
--------------------------------------------------------------------------------
/src/Operand.py:
--------------------------------------------------------------------------------
1 | '''
2 | Copyright 2019 ARM Ltd. and University of Cyprus
3 | Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction,
4 | including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software,
5 | and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
6 | The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
7 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
8 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
9 | TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
10 | '''
11 | import copy
12 | import sys
13 | class Operand(object):
14 | '''
15 | classdocs
16 | '''
17 |
18 |
19 | def __init__(self,id,type,values=[],min=0,max=4294967296,stride=1,toggleable="False"):
20 | '''Constructor takes as parameters the values an operand can take either in array or in bounds format'''
21 | self.values = values;
22 | self.min = min;
23 | self.max = max;
24 | self.stride = stride;
25 | self.currentValue="";
26 | self.id=id;
27 | self.type=type;
28 | self.toggleable=toggleable
29 | if( (int(min)>int(max)) or (int(stride)<=0)):
30 | print("Watch out you should always put a stride above 0 otherwise an infinitive loop will be caused and min should be less or equal than max");
31 | sys.exit();
32 | if not self.values: #if possible values not set.. manually create them
33 | if(self.type!="automatically_incremented_operand"): #some types are automatically implemented e.g. branch labels
34 | i=int(self.min);
35 | while(i<=int(self.max)):
36 | self.values.append(i);
37 | i+=int(self.stride);
38 | #self.togleValue=self.values[0];
39 |
40 | def copy(self):
41 | return copy.deepcopy(self);
42 |
43 | def mutate(self,rand):
44 | '''Basicaly sets as current value a random value from the acceptable range'''
45 | if(self.type=="automatically_incremented_operand"):
46 | self.currentValue=self.min;
47 | else:
48 | self.currentValue=rand.choice(self.values);
49 |
50 |
51 | def getValue(self):
52 | return self.currentValue;
53 |
54 | def setCurrentValueByIndex(self,index):
55 | self.currentValue=self.values[index];
56 |
57 | '''def toggle(self):
58 | if(self.toggleable=="True"):
59 | if(self.togleValue==self.values[0]):
60 | self.togleValue=self.values[1];
61 | else:
62 | self.togleValue=self.values[0];
63 |
64 | def setToggleValueAsCurrent(self):
65 | if(self.toggleable=="True"):
66 | self.currentValue=self.togleValue;'''
67 |
68 | def __str__(self):
69 | if(self.currentValue != ""):
70 | return str(self.currentValue);
71 | else:
72 | return str(self.id);
73 |
74 |
--------------------------------------------------------------------------------
/src/Population.py:
--------------------------------------------------------------------------------
1 | '''
2 | Copyright 2019 ARM Ltd. and University of Cyprus
3 | Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction,
4 | including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software,
5 | and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
6 | The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
7 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
8 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
9 | TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
10 | '''
11 |
12 | import operator
13 | import pickle
14 |
15 | class Population(object):
16 | '''
17 | classdocs
18 | '''
19 |
20 |
21 | def __init__(self,individuals=[]):
22 | self.individuals=individuals;
23 |
24 |
25 | def getIndividual(self,index):
26 | return self.individuals[index];
27 |
28 | def getFittest(self):
29 | best_value=float(self.individuals[0].getFitness());
30 | best_indiv=self.individuals[0];
31 | for i in range (self.individuals.__len__()):
32 | if float(self.individuals[i].getFitness()) > best_value:
33 | best_value=float(self.individuals[i].getFitness())
34 | best_indiv=self.individuals[i];
35 | return best_indiv;
36 |
37 | def getAvgFitness(self):
38 | sum=0
39 | for indiv in self.individuals:
40 | sum+=float(indiv.getFitness())
41 | avg=sum/self.individuals.__len__()
42 | return avg
43 | def getSize (self):
44 | return self.individuals.__len__()
45 |
46 | def pickRandomlyAnIndividual(self,rand):
47 | return rand.choice(self.individuals);
48 |
49 | def setCumulativeFitness(self): #for roulette wheel selection
50 | sum=0.0;
51 | self.individuals[0].setCumulativeFitness(int(self.individuals[0].getFitness()*1000000));
52 | for i in range(1,self.individuals.__len__()):
53 | fitness=int(self.individuals[i].getFitness()*1000000);
54 | self.individuals[i].setCumulativeFitness(self.individuals[i-1].cumulativeFitness+fitness);
55 |
56 | def sortByFitessToWeakest(self):
57 | self.individuals.sort(key=operator.attrgetter('fitness'),reverse=True)
58 |
59 | def sortByWeakestToFitess(self):
60 | self.individuals.sort(key=operator.attrgetter('fitness'))
61 | #sorted(self.individuals,key=lambda obj: obj.fitness, reverse=True);
62 |
63 | def saveIndividual (self,index,individual):
64 | self.individuals[index]=individual;
65 |
66 | def __str__(self):
67 | output="";
68 | for code in self.individuals:
69 | output+=str(code.__str__()+"\n");
70 | return output;
71 |
72 | def keepHalfBest(self):
73 | self.sortByFitessToWeakest();
74 | half=int(self.individuals.__len__()/2);
75 | newList=[]
76 | for i in range(0,half):
77 | newList.append(self.individuals[i]);
78 | self.individuals=newList;
79 |
80 |
81 | def pickle(self,filename):
82 | pickle.dump(self, filename);
83 |
84 | @staticmethod
85 | def unpickle(filename):
86 | return pickle.load(filename);
--------------------------------------------------------------------------------
/src/__init__.py:
--------------------------------------------------------------------------------
1 | '''
2 | Copyright 2019 ARM Ltd. and University of Cyprus
3 | Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction,
4 | including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software,
5 | and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
6 | The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
7 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
8 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
9 | TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
10 | '''
11 | import Instruction
12 | import Operand
13 | from random import Random
14 | from Algorithm import Algorithm
15 | from Individual import Individual
16 | import sys
17 |
18 |
19 | '''1st argument full path to configuration file.'''
20 |
21 | rand = Random (); #to use a fixed seed give an int argument
22 | algorithm = Algorithm (sys.argv[1],rand)
23 | algorithm.createInitialPopulation();
24 | algorithm.measurePopulation();
25 | while algorithm.areWeDone()!= True:
26 | algorithm.evolvePopulation()
27 | algorithm.measurePopulation()
28 | fittest=algorithm.getFittest();
29 | print ("Fittest individual is:\n " + fittest.__str__() + "with measurement equal to "+str(fittest.getMeasurement()));
30 |
31 |
--------------------------------------------------------------------------------
/src/parseGeneticResults.py:
--------------------------------------------------------------------------------
1 | '''
2 | Copyright 2019 ARM Ltd. and University of Cyprus
3 | Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction,
4 | including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software,
5 | and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
6 | The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
7 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
8 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
9 | TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
10 | '''
11 | import os
12 | import sys
13 | import pickle
14 | import pprint;
15 | from Population import Population
16 | from Individual import Individual
17 | from Instruction import Instruction
18 | from Operand import Operand
19 | import re;
20 |
21 |
22 | path = sys.argv[1]
23 | files=[]
24 | for root, dirs, filenames in os.walk(path): #takes as input the dir with the saved state
25 | for f in filenames:
26 | if((".pkl" in f) and ("rand" not in f)):
27 | files.append(f);
28 |
29 | files.sort(key=lambda x: int(x.split('.')[0]));
30 | pop=Population([]);
31 | allValues="";
32 | allKeys="";
33 | columns=[];
34 | theBest=[];
35 | print("best and average of each generation");
36 | print("generation best average");
37 | insHash={};
38 |
39 | for f in files:
40 | input=open(path+f,"rb");
41 | pop=pickle.load(input);
42 | input.close();
43 | columns.append(f.split('.')[0]);
44 | best=pop.getFittest();
45 | theBest.append(best);
46 | sum=0.0;
47 | count=0;
48 | for indiv in pop.individuals:
49 | sum+=float(indiv.getFitness());
50 | count+=1;
51 | for ins in indiv.sequence:
52 |
53 | if(ins.name in insHash.keys()):
54 | insHash[ins.name]+=1;
55 | else:
56 | insHash[ins.name]=1;
57 | sorted(insHash,key=lambda key: insHash[key]);
58 | #print(insHash);
59 | allKeys+=list(insHash.keys()).__str__()+"\n";
60 | allValues+=insHash.values().__str__()+"\n";
61 | average=sum/count;
62 | for key in list(insHash.keys()): #clear the hash for the next population
63 | insHash[key]=0;
64 |
65 | print(str(columns[-1])+" "+str(round(float(best.getFitness()),6))+" "+str(round(float(average),6)) );
66 | #print (allKeys);
67 | print("end of generation best average");
68 |
69 | values=re.sub("[A-Za-z]", "", allValues);
70 | values=re.sub("[\[\]\(\),_]", "", values);
71 | values=values.strip(' \n');
72 | data=[];
73 | totalSize=pop.individuals[0].getInstructions().__len__()*pop.getSize();
74 | #print (values);
75 | for row in values.split("\n"):
76 | #print(row);
77 | data.append([float(float(s)/float(totalSize)) for s in row.split()])
78 | #data.append([int(s) for s in row.split()])
79 | #data=pprint.pformat (data);
80 |
81 |
82 | rows=re.sub("[\[,\'\]]", "", allKeys);
83 | rows=re.sub("[\{\}\':,]", "", rows);
84 | rows=rows.strip(' \n');
85 |
86 | for column in rows.split("\n"):
87 | #print(row);
88 | rows=[str(s) for s in column.split()]
89 |
90 |
91 | print("Instruction Mix per generation");
92 |
93 | print (" "+' '.join(rows));
94 |
95 | for i in range(columns.__len__()):
96 | print (columns[i],end=" ");
97 | for j in range(rows.__len__()):
98 | print(round(data[i][j],2),end=" ");
99 | print("");
100 |
101 | #print(values);
102 |
103 | print("Instruction Mix for best of each generation");
104 |
105 | print (" "+' '.join(rows));
106 |
107 |
108 | loopSize=theBest[0].getInstructions().__len__()
109 |
110 | i=1;
111 | for indiv in theBest:
112 | for key in list(insHash.keys()): #clear the hash for the next individual
113 | insHash[key]=0;
114 | for ins in indiv.sequence:
115 | if(ins.name in insHash.keys()):
116 | insHash[ins.name]+=1;
117 | else:
118 | insHash[ins.name]=1;
119 | sorted(insHash,key=lambda key: insHash[key]);
120 | print(str(i),end=" ");
121 | i+=1;
122 | for key in list(insHash.keys()):
123 | print(round(float(float(insHash[key])/loopSize),2),end=" ");
124 | print("");
125 |
126 | print("Type Mix for best of each generation");
127 |
128 | typeHash={};
129 |
130 | i=1;
131 | for indiv in theBest:
132 | for key in list(typeHash.keys()): #clear the hash for the next individual
133 | typeHash[key]=0;
134 | for ins in indiv.sequence:
135 | try:
136 | if(ins.type in typeHash.keys()):
137 | typeHash[ins.type]+=1;
138 | else:
139 | typeHash[ins.type]=1;
140 | except: #in legacy pkls the attribute was type instead of ins_type
141 | if(ins.ins_type in typeHash.keys()):
142 | typeHash[ins.ins_type]+=1;
143 | else:
144 | typeHash[ins.ins_type]=1;
145 |
146 | sorted(typeHash,key=lambda key: typeHash[key]);
147 | print(str(i),end=" ");
148 | i+=1;
149 | for key in list(typeHash.keys()):
150 | print(round(float(float(typeHash[key])/loopSize),2),end=" ");
151 | print("");
152 | for key in typeHash.keys():
153 | print(key,end=" ")
154 |
155 | sys.exit();
156 |
157 |
--------------------------------------------------------------------------------