├── .gitignore ├── .project ├── .pydevproject ├── README ├── README.md ├── assembly_compilation_ARM └── main_original.s ├── assembly_compilation_x86_gcc └── main_original.s ├── configurationFiles ├── configuration-template-explanation.xml ├── configuration_ARM_IPC.xml ├── configuration_likwidPowerMeter.xml ├── configuration_x86_IPC.xml └── measurement │ ├── IPC.xml │ └── measurementLikwid.xml └── src ├── Algorithm.py ├── Fitness ├── DefaultFitness.py └── SimplicityTempFitness.py ├── Individual.py ├── Instruction.py ├── Measurement ├── Measurement.py ├── MeasurementIPC.py └── MeasurementLikwidPower.py ├── Operand.py ├── Population.py ├── __init__.py └── parseGeneticResults.py /.gitignore: -------------------------------------------------------------------------------- 1 | 2 | *.pyc 3 | -------------------------------------------------------------------------------- /.project: -------------------------------------------------------------------------------- 1 | 2 | 3 | GeST 4 | 5 | 6 | 7 | 8 | 9 | org.python.pydev.PyDevBuilder 10 | 11 | 12 | 13 | 14 | 15 | org.python.pydev.pythonNature 16 | 17 | 18 | -------------------------------------------------------------------------------- /.pydevproject: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | /${PROJECT_DIR_NAME}/src 5 | 6 | python 3.6 7 | Default 8 | 9 | -------------------------------------------------------------------------------- /README: -------------------------------------------------------------------------------- 1 | Copyright 2019 ARM Ltd. and University of Cyprus 2 | Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, 3 | including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, 4 | and to permit persons to whom the Software is furnished to do so, subject to the following conditions: 5 | The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. 6 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 7 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, 8 | TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 9 | 10 | 11 | AUTOMATIC FRAMEWORK FOR STRESS TEST GENERATION (GeST Generating Stress-Tests) 12 | 13 | GENERAL DESCRIPTION 14 | 15 | This framework is based on Genetic algorithms (GA) and it has been succesfully been used to develop power viruses (stres-tests that maximize power consumption), 16 | dI/dt viruses (stress-tests that maximize voltage noise) and IPC stress tests (maximize instruction per cycle throughput) on various CPUs such as ARM cortex-A15, cortex-A7, 17 | cortex-A57,cortex-A53, cortex-A72, AMD Athlon II X4 645, X-Gene2, X-Gene3 18 | 19 | GA optimize for the target metric using bio-inspired operators such as crossover (exchange of genes), mutation, selection of fittest individual for breeding (for crossover). 20 | GA typically begins with a population of individuals (randomly generated) and evolves this population by measuring the fitness of the individuals and applying the aforementioned operators. 21 | 22 | The framework produces assembly instructions sequences that are measured for the metric of interest (with a fitness function). The goal of the framework is to generate the fittest instruction sequence 23 | (in the GA terminology the fittest individual). 24 | For instance if the goal is maximum CPU power consumption the framework attempts to generate instruction sequences that maximize the power consumption. 25 | 26 | The framework can be extended by the user to stress other components such as DRAM, Last level cache (Uncore) etc. Also it can be extended to support more complicate fitness functions that take in account 27 | multiple measurement metrics (power, pef counters etc) and instinsic instruction sequence characteristics such as instruction mix breakdown. 28 | 29 | The framework also offers an easy interface to the user to add custom measurement procedures. 30 | Some template measurements are included in the framework such as maximizing max power using IPMI readings and maximizing IPC through perf counters on a ssh reachable machine. 31 | But for instance if a user wants to maximize on a bare metal machine power using a specific power plug meter, he can write his own measurement function and seemlesly integrate it in the framework (read DEFINING CUSTOM MEASUREMENTS - CUSTOM FITNESS CLASSES and FUNCTIONS section). 32 | 33 | The framework is written in Python 3 and xml is used for configuration files 34 | 35 | REQUIREMENTS 36 | 37 | The framework shoud work without any issues on Linux enviroment, but it has been tested only on Windows only with Eclipse IDE for development and running experiments. 38 | The following enviroment is recommended 39 | 40 | 1) Windows OS 41 | 2) LiClipse IDE 42 | 3) Python 3.6 is recommended but probably any Python 3 would work (NOTE this is not compatible with python 2!!) 43 | 4) NI-visa drivers must be installed if going to use measurement instruments such oscilloscopes and spectrum analyzer for measurements (tested with NI-VISA 5) 44 | 5) Paramiko python libary and its dependencies must be installed for enabling ssh communication (execute from shell the command: pip install paramiko #pip is the python package manager ). Very useful coordinating execution on the target machine. 45 | 6) Python PYVISA libary if going to use measurement instruments such oscilloscopes and spectrum analyzer for measurements 46 | 47 | INSTALLATION INSTRUCTIONS on WINDOWS 48 | 49 | 1) Download liclipse for python development https://www.liclipse.com/ 50 | 2) Download python 3 https://www.python.org/downloads/ 51 | 3) Locate pip binary. For python 3.6 should be found in C:\Users\UserName\AppData\Local\Programs\Python\Python36-32\python.exe 52 | From a windows terminal run 53 | pip install paramiko 54 | pip install -U pyvisa #necessary if measurements with spectrum analyzers and oscilloscopes 55 | 56 | 57 | RUN INSTRUCTIONS ON ECLIPSE/Liclipse 58 | 59 | 1) Import the project as Python project 60 | 2) Set correctly the configuration file (look section below) 61 | 3) Right click on the project then debug-as. In the arguments section specify the full path to the configuration file you will use. 62 | 63 | SET CORRECTLY THE CONFIGURATION FILE 64 | This section explains how to set correctly the configuration file and we will explain this with a particular example. 65 | Let's say you want to optimize for maximum power consumption on an x86 CPU using likwid power meter. 66 | In this case you can use the following provided class and files: configuration_likwidPowerMeter.xml, measurementLikwid.xml and MeasurementLikwidPower.py. 67 | Below are some steps you need to follow to set correctly the configuration files in relation to your environemnt. Similar steps should be followed for the IPC optimization (configuration files included in the release) 68 | any any other optimization you wish to perform. 69 | 70 | 1) Open the configuration_likwidPowerMeter.xml 71 | 2) Set the dirToSaveResults attribute to the path where you want the results to be saved. 72 | 3) Set the compilationDir to the path where the assembly template compilation is located (the director where the main_original.s is located) 73 | 4) Make sure the main_original.s has a #loop_code string which point to GeST where to print the individual. Add to the main_original.s any register and memory initilization you wish to have. 74 | 5) Make sure that the measurementClass attribute is set to MeasurementLikwidPower, the measurementClassConfFile is set to measurementLikwid and the fitnessClass to DefaultFitness 75 | 6) On the target machine (the machine for which the virus is generated) create a temporary directory which will be used by the GA for compiling the individuals and placing temporary the 76 | individual's source code and binary. 77 | 7) Open the measurement xml configuration file, in this particular example the file you need to open is the measurement/measurementLikwid.xml 78 | 8) Set the targetRunDir with the path where GA temporary directory is located (the one created in step 6) 79 | 9) Set the targetHostname attribute value to the ip or hostname with which the target machine is accessible through ssh 80 | 10) Set the targetSSHusername and targetSSHpassword attributes. Make sure that with the username account you can run sudo commands without password. 81 | To achieve this make sure your account belongs to sudo group and if e.g. the target runs Ubuntu add this line "%sudo ALL=(ALL:ALL) NOPASSWD:ALL " to the /etc/sudoers file. 82 | 11) Set the core ids that will be used in the optimization (that will run the indiviudal binary) 83 | 12) Set the time_to_measure each individual. For power optimization with likwid we had good results with 5 seconds but ofcourse you are free to play with this variable. 84 | 85 | MORE DETAILS ON HOW TO USE GEST 86 | 87 | Detailed explanation of the main configuration file options is found in configuration-template-explanation.xml located in ./configurationFiles. 88 | Also we recommend to read the ISPASS 2019 paper "GeST an automatic framework for generating CPU stress-tests" https://ieeexplore.ieee.org/document/8695639 which is a good reading to understand the framework and how to use it. 89 | 90 | To get more information on how to use GeST and what capabilities it offers please continue reading. 91 | 92 | GeST gives you the ability: 93 | a) To play with some algorithm parameters e.g. mutation, crossover,loop sizes, population size etc 94 | b) To specify explicity what instructions or even instruction sequences you want to use in the optimization process. 95 | c) Specify operands and which instructions will use those operands which eventually give you the ability to force or not force dependencies between instructions 96 | d) You can specify fixed code segments which will be present in all the individuals (an individual is an instruction sequence generated) e.g. some check for not touching illegal memory, register and memory initialization 97 | e) Make a run starting with fixed instruction mix breakdwon 98 | f) Ability to save a run and continue it after 99 | g) Different measurement procedures by defining your own manual class that inherits from the Measurement/Mesurement.py class. 100 | You don't need to change anything else in the code just specify the custom measurement class name in the configuration file. 101 | h) Different fitness functions by defining your own manual class that inherits from the Fitness/DefaultFitness.py class 102 | You don't need to change anything else in the code just specify the custom measurement class name in the configuration file. 103 | 104 | CONFIGURATION FILES LOCATION 105 | The main configuarion files should be placed in ./configurationFiles folder. Configuration related to custom measurement classes should be placed in ./configurationFiles/measurement directory 106 | 107 | ASSEMBLY COMPILATION DIRECTORY 108 | The dirs with prefix assembly_compilation are the dirs which contain all the necessary files for the generation of the binary that is going to be run on the target machine. 109 | You must specify which dir you are going to use for the compilation in the configuration file. 110 | The assembly compilation dir in the most simple form contains one source file that will be compiled, namely the main_original.s 111 | The main_orginal.s must contain a line which contains the string “#loop_code”. This string will be replaced by the generated by framework code sequence. 112 | Any memory, register initialization or in general common fixed code accross all individuals can be placed on the main_original.s file 113 | 114 | FRAMEWORK OUTPUT RESULTS 115 | The dirToSaveResults parameters which is set in the configuration.xml file specifies the directory in which results – outputs of the framework run will be saved. 116 | In this directory you can find the generated code sequences which will be save in ascii format in .txt files. The name format of each file goes like this populationNumber_individualsIdNumber_measurement1_measurement2_etc. 117 | You can specify as many measurements as you want. Also a dir which is named after the date/time of the start of the run is created e.g. (13-12-10-15-06 means the run started on 13th of December at 15-06). This dir will contain each population saved in .pkl file and the rand state (also in .pkl format) at each population. 118 | This dir can be used as a seed dir in case you want to continue an unfinished run. So basically each run leaves a seed dir behind. (Note to start a run from an existing SeedDir set the seedDir parameter in the configuration file) 119 | In the seed dir the assembly compilation file is copied. All the compilations and modifications of the code happen on that copied assembly compilation file which is located in the seed dir. 120 | In the seed dir also is copied the configuration.xml and measurement.xml file and the src of the framework which was used for that run. 121 | The population pkl files can be used for parsing results. After measuring a population the whole population is saved in a .pkl file. 122 | The populations are saved in ascending order like this 1.pkl, 2.pkl 3.pkl and etc. The parseGeneticResults.py file is an example of how to parse the population files and get statistics about average and best of each generation as well as the average and best individual’s instruction mix for each population. 123 | To use the script just pass the seed dir’s absolute path as argument. 124 | 125 | DEFINING CUSTOM MEASUREMENTS - CUSTOM FITNESS CLASSES and FUNCTIONS 126 | Framework allows definining custom measurement classes and fitness functions. 127 | The measurement procedure is a a very environment specific procedure. A GA stress-test generation framework must support IPC, thermal, power, etc. optimizations. 128 | Moreover, for instance, to optimize for power consumption various power measurment tools might be used such as IPMI, RAPL etc. 129 | Due to so many alternatives a framework that supports all these alternatives is impossible to create. 130 | Hence, we provide a simple interface to the user to write it's own measurement procedures and use them with this framework. Templates of measurement classes can be found in ./Measurement directory and 131 | the representative configuration files in ./configurationFiles/measurement directory. To add your own measurement class create a class that inherits from Measurement.py and place it ./Measurement directory. 132 | You have to overwrite the measure function and most likely you 133 | will also need to overwite the init function to read any custom parameters required by your class. The custom parameters need to be specified in a configuration file placed in ./configurationFiles/measurement. 134 | You don't need to touch any other framework's code to use your measurement class. Just add in the main configuration file the name of the class and the name of the classe's configuration file (just the name not the full path) in the 135 | measurementClass and measurementClassConfFile parameters. 136 | This is achieved with Python's capability to dynamically load a class (without specifing it during source compilation). 137 | 138 | You can also modify the fitness function with the same principles as for the Measurement class. 139 | You can do this by inheriting from the DefaultFitness class and overriding the getFitness function. The fitness function takes as parameter the individual after is being measurement. The individual object comes with 140 | its instruction sequence and its measurements. The default getFitness function just returns the first measurement as the fitness value, e.g. it can be the average power while running the individual. 141 | You have to place the custom fitness class in the ./Fitness directory. To 142 | use your custom fitness class specify the name in the fitnessClass parameter in the configuration file. 143 | For now Fitness classes does not come with a configuration file like measurement classes. 144 | If needed this might be changed in the future. 145 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # GeST 2 | 3 | GeST (Generating Stress-Tests) is a framework for automatic hardware stress-test generation. 4 | 5 | A scientific paper about GeST is published in ISPASS 2019 https://ieeexplore.ieee.org/document/8695639 6 | 7 | Hadjilambrou, Z., Das, S., Whatmough, P.N., Bull, D. and Sazeides, Y., 2019, April. GeST: An Automatic Framework For Generating CPU Stress-Tests. In 2019 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS) (pp. 1-10). IEEE. 8 | -------------------------------------------------------------------------------- /assembly_compilation_ARM/main_original.s: -------------------------------------------------------------------------------- 1 | /* 2 | Copyright 2019 ARM Ltd. and University of Cyprus 3 | Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, 4 | including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, 5 | and to permit persons to whom the Software is furnished to do so, subject to the following conditions: 6 | The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. 7 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 8 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, 9 | TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 10 | */ 11 | .data 12 | 13 | msg: 14 | .ascii "\n" 15 | len = . - msg 16 | .text 17 | .align 2 18 | .global main 19 | .type main, %function 20 | 21 | 22 | 23 | main: 24 | 25 | #reg init 26 | 27 | LDR x0,=0xAAAAAAAA 28 | LDR x1,=0x55555555 29 | LDR x2,=0x00000000 30 | LDR x3,=0x00000000 31 | LDR x4,=0x55555555 32 | LDR x5,=0x33333333 33 | LDR x6,=0xFFFFFFFF 34 | LDR x7,=0xFFFFFFFE 35 | LDR x8,=0x00000001 36 | LDR x9,=0xCCCCCCCC 37 | LDR x10,=0xAAAAAAAA 38 | LDR x11,=0x55555555 39 | LDR x12,=0x00000000 40 | LDR x13,=0x00000000 41 | LDR x14,=0x55555555 42 | LDR x15,=0x33333333 43 | LDR x16,=0xFFFFFFFF 44 | LDR x17,=0xFFFFFFFE 45 | LDR x18,=0x00000001 46 | LDR x19,=0xCCCCCCCC 47 | LDR x20,=0xAAAAAAAA 48 | LDR x21,=0x55555555 49 | LDR x22,=0x00000000 50 | LDR x23,=0x00000000 51 | LDR x24,=0x55555555 52 | LDR x25,=0x33333333 53 | LDR x26,=0xFFFFFFFF 54 | LDR x27,=0xFFFFFFFE 55 | LDR x28,=0x00000001 56 | LDR x29,=0xCCCCCCCC 57 | LDR x30,=0xCCCCCCCC 58 | LDR D0,=0xAAAAAAAAAAAAAAAA 59 | LDR D1,=0xFFFFFFFFFFFFFFFF 60 | LDR D2,=0x5555555555555555 61 | LDR D3,=0x3333333333333333 62 | LDR D4,=0xCCCCCCCCCCCCCCCC 63 | LDR D5,=0x0000000100000001 64 | LDR D6,=0xFFFFFFFEFFFFFFFE 65 | LDR D7,=0x0000000000000000 66 | LDR D8,=0xAAAAAAAA55555555 67 | LDR D9,=0x55555555AAAAAAAA 68 | LDR D10,=0xAAAAAAAAAAAAAAAA 69 | LDR D11,=0xFFFFFFFFFFFFFFFF 70 | LDR D12,=0x5555555555555555 71 | LDR D13,=0x3333333333333333 72 | LDR D14,=0xCCCCCCCCCCCCCCCC 73 | LDR D15,=0x0000000100000001 74 | LDR D16,=0xFFFFFFFEFFFFFFFE 75 | LDR D17,=0x0000000000000000 76 | LDR D18,=0xAAAAAAAA55555555 77 | LDR D19,=0x55555555AAAAAAAA 78 | LDR D20,=0xAAAAAAAAAAAAAAAA 79 | LDR D21,=0xFFFFFFFFFFFFFFFF 80 | LDR D22,=0x5555555555555555 81 | LDR D23,=0x3333333333333333 82 | LDR D24,=0xCCCCCCCCCCCCCCCC 83 | LDR D25,=0x0000000100000001 84 | LDR D26,=0xFFFFFFFEFFFFFFFE 85 | LDR D27,=0x0000000000000000 86 | LDR D28,=0xAAAAAAAA55555555 87 | LDR D29,=0x55555555AAAAAAAA 88 | LDR D30,=0x55555555AAAAAAAA 89 | 90 | 91 | 92 | LDR x12,=0x200 93 | #0x200 is 512 bytes 94 | SUB x12,sp,x12 95 | MOV x10,x12 96 | MOV x11,10 97 | #x12 will hold the end address 98 | 99 | LDR x28, =0x3B9ACA00/*1 billion iterations. x28 to be used for iteration counter if running for fixed iterations*/ 100 | 101 | Start: 102 | 103 | 104 | #loop_code 105 | 106 | 107 | 108 | 109 | b Start 110 | 111 | #SUB x28,x28,1 #uncomment for running for fixed iterations 112 | #CBNZ x28,Start 113 | 114 | ret 115 | .size main, .-main 116 | .ident "GCC: (APM-8.0.10-le) 4.9.3 20150218 (prerelease)" 117 | .section .note.GNU-stack,"",%progbits 118 | -------------------------------------------------------------------------------- /assembly_compilation_x86_gcc/main_original.s: -------------------------------------------------------------------------------- 1 | /* 2 | Copyright 2019 ARM Ltd. and University of Cyprus 3 | Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, 4 | including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, 5 | and to permit persons to whom the Software is furnished to do so, subject to the following conditions: 6 | The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. 7 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 8 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, 9 | TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 10 | */ 11 | 12 | .file "main.s" 13 | .data 14 | .align 32 15 | .simdvalue: 16 | .long 0xaaaaaaaa 17 | .long 0x55555555 18 | .long 0x33333333 19 | .long 0xcccccccc 20 | .long 0xaaaaaaaa 21 | .long 0x55555555 22 | .long 0x33333333 23 | .long 0xcccccccc 24 | .text 25 | .globl main 26 | main: 27 | .LFB0: 28 | .cfi_startproc 29 | pushq %rbp 30 | .cfi_def_cfa_offset 8 31 | .cfi_offset 5, -8 32 | movl %esp, %ebp 33 | .cfi_def_cfa_register 5 34 | 35 | 36 | 37 | #reg init 38 | 39 | 40 | mov $0x55555555, %rax 41 | mov $0x33333333, %rbx 42 | mov $0x22222222, %rdx 43 | mov $0x44444444, %rsi 44 | mov $0x77777777, %rdi 45 | 46 | fldpi 47 | fldpi 48 | fldpi 49 | fldpi 50 | fldpi 51 | fldpi 52 | fldpi 53 | 54 | vmovdqa .simdvalue(%rip), %ymm0 55 | vmovdqa .simdvalue(%rip), %ymm1 56 | vmovdqa .simdvalue(%rip), %ymm2 57 | vmovdqa .simdvalue(%rip), %ymm3 58 | vmovdqa .simdvalue(%rip), %ymm4 59 | vmovdqa .simdvalue(%rip), %ymm5 60 | vmovdqa .simdvalue(%rip), %ymm6 61 | vmovdqa .simdvalue(%rip), %ymm7 62 | vmovdqa .simdvalue(%rip), %ymm8 63 | vmovdqa .simdvalue(%rip), %ymm9 64 | vmovdqa .simdvalue(%rip), %ymm10 65 | vmovdqa .simdvalue(%rip), %ymm11 66 | vmovdqa .simdvalue(%rip), %ymm12 67 | vmovdqa .simdvalue(%rip), %ymm13 68 | vmovdqa .simdvalue(%rip), %ymm14 69 | vmovdqa .simdvalue(%rip), %ymm15 70 | 71 | mov $50000000, %rcx #leave for i-- 72 | 73 | #subq $304, %rsp 74 | 75 | .L2: 76 | 77 | #loop_code 78 | 79 | 80 | #sub $1,%rcx #remove this and below comment for fixed iterations 81 | #cmp $0, %rcx 82 | jmp .L2 83 | 84 | leave 85 | .cfi_restore 5 86 | .cfi_def_cfa 4, 4 87 | ret 88 | 89 | .cfi_endproc 90 | .LFE0: 91 | .ident "GCC: (GNU) 6.4.0" 92 | -------------------------------------------------------------------------------- /configurationFiles/configuration-template-explanation.xml: -------------------------------------------------------------------------------- 1 | 2 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 41 | 42 | 43 | 44 | 45 | 46 | 47 | 48 | 49 | 50 | 55 | 56 | 59 | 60 | 61 | 62 | 63 | 69 | 70 | 71 | 72 | 73 | 74 | 75 | 76 | 77 | 78 | 82 | 83 | 87 | 88 | 92 | 93 | 97 | 98 | 102 | 103 | 107 | 108 | 109 | 110 | 111 | 112 | 113 | 115 | 116 | 144 | 145 | 150 | 151 | 152 | 157 | 158 | 159 | 164 | 165 | 166 | 171 | 172 | 173 | 174 | 180 | 181 | 182 | 188 | 189 | 190 | 196 | 197 | 198 | 204 | 205 | 206 | 211 | 212 | 217 | 218 | 219 | 220 | 221 | 226 | 227 | 228 | 233 | 234 | 235 | 240 | 241 | 242 | 247 | 248 | 249 | 254 | 255 | 256 | 257 | 262 | 263 | 264 | 269 | 270 | 271 | 272 | 279 | 280 | 281 | 282 | 289 | 290 | 291 | 292 | 299 | 300 | 301 | 308 | 309 | 310 | 317 | 318 | 319 | 320 | 327 | 328 | 329 | 330 | 331 | 332 | 333 | 334 | 335 | 337 | 338 | 359 | 360 | 361 | 362 | 371 | 372 | 373 | 374 | 375 | 384 | 385 | 386 | 387 | 396 | 397 | 398 | 407 | 408 | 409 | 418 | 419 | 420 | 429 | 430 | 431 | 440 | 441 | 442 | 443 | 444 | 453 | 454 | 455 | 456 | 457 | 468 | 469 | 470 | 480 | 481 | 482 | 483 | 494 | 495 | 496 | 506 | 507 | 508 | 509 | 518 | 519 | 520 | 521 | 522 | 530 | 531 | 532 | 540 | 541 | 542 | 543 | 544 | 553 | 554 | 555 | 556 | 565 | 566 | 567 | 568 | 577 | 578 | 579 | 580 | 589 | 590 | 591 | 600 | 601 | 602 | 603 | 604 | -------------------------------------------------------------------------------- /configurationFiles/configuration_ARM_IPC.xml: -------------------------------------------------------------------------------- 1 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 33 | 34 | 37 | 38 | 41 | 42 | 45 | 48 | 49 | 52 | 53 | 56 | 57 | 58 | 61 | 62 | 66 | 67 | 68 | 69 | 70 | 71 | 72 | 73 | 74 | 79 | 80 | 83 | 84 | 85 | 86 | 87 | 90 | 94 | 95 | 96 | 97 | 98 | 100 | 101 | 102 | 103 | 104 | 105 | 106 | 107 | 108 | 112 | 113 | 117 | 118 | 122 | 123 | 127 | 128 | 132 | 133 | 137 | 138 | 139 | 140 | 141 | 142 | 143 | 144 | 146 | 147 | 175 | 176 | 177 | 182 | 183 | 184 | 189 | 190 | 191 | 196 | 197 | 198 | 203 | 204 | 205 | 206 | 221 | 222 | 223 | 229 | 230 | 231 | 237 | 238 | 239 | 245 | 246 | 247 | 253 | 254 | 255 | 260 | 261 | 266 | 267 | 268 | 269 | 270 | 275 | 276 | 277 | 282 | 283 | 284 | 289 | 290 | 291 | 296 | 297 | 298 | 303 | 304 | 305 | 306 | 311 | 312 | 313 | 318 | 319 | 320 | 321 | 328 | 329 | 330 | 331 | 338 | 339 | 340 | 341 | 348 | 349 | 350 | 357 | 358 | 359 | 366 | 367 | 368 | 369 | 376 | 377 | 378 | 386 | 387 | 388 | 389 | 390 | 398 | 399 | 400 | 401 | 402 | 403 | 404 | 405 | 407 | 408 | 429 | 430 | 431 | 432 | 433 | 441 | 442 | 453 | 454 | 471 | 472 | 479 | 480 | 487 | 488 | 489 | 490 | 499 | 500 | 501 | 510 | 511 | 512 | 513 | 522 | 523 | 524 | 533 | 534 | 535 | 544 | 545 | 546 | 555 | 556 | 557 | 566 | 567 | 568 | 569 | 570 | 579 | 580 | 581 | 582 | 583 | 594 | 595 | 596 | 606 | 607 | 608 | 609 | 620 | 621 | 622 | 632 | 633 | 634 | 635 | 636 | 644 | 645 | 646 | 654 | 655 | 656 | 665 | 666 | 667 | 676 | 677 | 678 | 687 | 688 | 689 | 698 | 699 | 700 | 710 | 711 | 712 | 720 | 721 | 722 | 730 | 731 | 732 | 741 | 742 | 743 | 744 | 745 | 753 | 754 | 755 | 764 | 765 | 766 | 775 | 776 | 777 | 778 | 787 | 788 | 789 | 790 | 799 | 800 | 801 | 802 | 811 | 812 | 813 | 814 | 815 | 816 | -------------------------------------------------------------------------------- /configurationFiles/configuration_likwidPowerMeter.xml: -------------------------------------------------------------------------------- 1 | 2 | 3 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 36 | 37 | 40 | 41 | 44 | 45 | 48 | 51 | 52 | 55 | 56 | 59 | 60 | 61 | 64 | 65 | 69 | 70 | 71 | 72 | 75 | 78 | 79 | 82 | 83 | 84 | 85 | 86 | 89 | 93 | 94 | 95 | 96 | 97 | 98 | 100 | 101 | 102 | 103 | 104 | 105 | 106 | 107 | 108 | 109 | 113 | 114 | 118 | 119 | 123 | 124 | 128 | 129 | 133 | 134 | 138 | 139 | 140 | 141 | 142 | 143 | 144 | 145 | 147 | 148 | 176 | 177 | 182 | 183 | 184 | 189 | 190 | 191 | 196 | 197 | 198 | 199 | 206 | 207 | 208 | 215 | 216 | 217 | 224 | 225 | 226 | 227 | 228 | 229 | 230 | 231 | 233 | 234 | 255 | 256 | 257 | 258 | 267 | 268 | 277 | 278 | 288 | 289 | 297 | 298 | 299 | 300 | 309 | 310 | 311 | 320 | 321 | 322 | 331 | 332 | 333 | 341 | 342 | 343 | 344 | 345 | 346 | 354 | 355 | 356 | 364 | 365 | 366 | 374 | 375 | 376 | 384 | 385 | 386 | 387 | 395 | 396 | 397 | 405 | 406 | 407 | 408 | 416 | 417 | 418 | 419 | 428 | 429 | 430 | 431 | 440 | 441 | 442 | 443 | 444 | 445 | 461 | 462 | 479 | 480 | 481 | 482 | 491 | 492 | 493 | 502 | 503 | 504 | 513 | 514 | 515 | 524 | 525 | 526 | 535 | 536 | 537 | 538 | 594 | 616 | 626 | 627 | 628 | 629 | 731 | 732 | 733 | 734 | 735 | 736 | -------------------------------------------------------------------------------- /configurationFiles/configuration_x86_IPC.xml: -------------------------------------------------------------------------------- 1 | 2 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 35 | 36 | 39 | 40 | 43 | 44 | 47 | 50 | 51 | 54 | 55 | 58 | 59 | 60 | 63 | 64 | 66 | 67 | 68 | 69 | 70 | 71 | 72 | 73 | 74 | 75 | 78 | 81 | 82 | 85 | 86 | 87 | 88 | 89 | 92 | 96 | 97 | 98 | 99 | 100 | 101 | 103 | 104 | 105 | 106 | 107 | 108 | 109 | 110 | 111 | 115 | 116 | 120 | 121 | 125 | 126 | 130 | 131 | 135 | 136 | 140 | 141 | 142 | 143 | 144 | 145 | 146 | 147 | 149 | 150 | 178 | 179 | 184 | 185 | 186 | 191 | 192 | 193 | 194 | 201 | 202 | 203 | 210 | 211 | 212 | 219 | 220 | 221 | 222 | 223 | 224 | 225 | 226 | 228 | 229 | 250 | 251 | 252 | 253 | 254 | 262 | 263 | 264 | 272 | 273 | 274 | 282 | 283 | 284 | 292 | 293 | 294 | 302 | 303 | 304 | 305 | 306 | 307 | 315 | 316 | 317 | 325 | 326 | 327 | 335 | 336 | 337 | 345 | 346 | 347 | 348 | 349 | 350 | 358 | 359 | 360 | 368 | 369 | 370 | 371 | 379 | 380 | 381 | 382 | 391 | 392 | 393 | 394 | 403 | 404 | 405 | 406 | 407 | 408 | 414 | 415 | 416 | 422 | 423 | 424 | 430 | 431 | 432 | 438 | 439 | 440 | 441 | 442 | 450 | 451 | 452 | 460 | 461 | 462 | 470 | 471 | 472 | 480 | 481 | 482 | 490 | 491 | 492 | 500 | 501 | 502 | 510 | 511 | 512 | 520 | 521 | 522 | 530 | 531 | 532 | 540 | 541 | 542 | 543 | 544 | 545 | 546 | -------------------------------------------------------------------------------- /configurationFiles/measurement/IPC.xml: -------------------------------------------------------------------------------- 1 | 2 | 3 | 13 | 14 | 15 | 16 | 19 | 20 | 23 | 24 | 27 | 28 | 31 | 32 | 33 | 36 | 37 | 38 | 39 | 40 | 41 | -------------------------------------------------------------------------------- /configurationFiles/measurement/measurementLikwid.xml: -------------------------------------------------------------------------------- 1 | 2 | 3 | 13 | 14 | 15 | 16 | 19 | 20 | 23 | 24 | 27 | 28 | 31 | 32 | 33 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | -------------------------------------------------------------------------------- /src/Algorithm.py: -------------------------------------------------------------------------------- 1 | ''' 2 | Copyright 2019 ARM Ltd. and University of Cyprus 3 | Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, 4 | including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, 5 | and to permit persons to whom the Software is furnished to do so, subject to the following conditions: 6 | The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. 7 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 8 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, 9 | TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 10 | ''' 11 | 12 | import subprocess 13 | from xml.dom import minidom 14 | from Instruction import Instruction 15 | from Operand import Operand 16 | from Population import Population 17 | from Individual import Individual 18 | import math 19 | import fileinput 20 | from random import Random 21 | import os 22 | import shutil 23 | import sys 24 | from threading import Timer 25 | from threading import Thread 26 | import time 27 | import pickle; 28 | import re; 29 | import atexit; 30 | import datetime; 31 | from paramiko import SSHClient, client 32 | import paramiko 33 | import socket 34 | import platform 35 | import visa 36 | from statistics import stdev 37 | from multiprocessing.pool import ThreadPool 38 | from multiprocessing import TimeoutError 39 | import importlib 40 | 41 | #import serial; 42 | 43 | class Algorithm(object): 44 | 45 | 46 | UNIFORM_CROSSOVER="0" 47 | ONEPOINT_CROSSOVER="1" 48 | WHEEL_SELECTION="1"; 49 | TOURNAMENT_SELECTION="0"; 50 | 51 | 52 | def __init__(self,configurationFile,rand=Random()): #reads configuration file and initializes the first population 53 | '''general initialization''' 54 | self.general_initialization(configurationFile, rand) 55 | '''specific initializations''' 56 | self.__instructions_operands_init__() 57 | print("End of inputs\n"); 58 | 59 | def general_initialization(self,configurationFile,rand): 60 | '''general algorithm and run parameters''' 61 | self.xmldoc = minidom.parse(configurationFile) 62 | self.intitializeAlgorithmAndRunParameters(configurationFile, rand) 63 | '''create results and saved state directories''' 64 | self.setupDirs(configurationFile) 65 | '''print general and run parameters''' 66 | self.printGeneralInputs(); 67 | 68 | 69 | def intitializeAlgorithmAndRunParameters(self,configurationFile,rand): 70 | 71 | ##genetic algorithm parameters 72 | self.populationSize = self.xmldoc.getElementsByTagName('population_size')[0].attributes['value'].value; 73 | self.mutationRate = self.xmldoc.getElementsByTagName('mutation_rate')[0].attributes['value'].value; 74 | self.crossoverType = self.xmldoc.getElementsByTagName('crossover_type')[0].attributes['value'].value; 75 | self.crossoverRate = self.xmldoc.getElementsByTagName('crossover_rate')[0].attributes['value'].value; 76 | self.uniformRate = self.xmldoc.getElementsByTagName('uniform_rate')[0].attributes['value'].value; 77 | self.ellitism = self.xmldoc.getElementsByTagName('ellitism')[0].attributes['value'].value; 78 | self.selectionMethod=self.xmldoc.getElementsByTagName('selectionMethod')[0].attributes['value'].value;#0 wheel,1 tournament 79 | self.tournamentSize = self.xmldoc.getElementsByTagName('tournament_size')[0].attributes['value'].value; 80 | self.populationsToRun= int(self.xmldoc.getElementsByTagName('populations_to_run')[0].attributes['value'].value); 81 | 82 | 83 | ##compilation and measurement parameters 84 | self.fitnessClassName = self.xmldoc.getElementsByTagName('fitnessClass')[0].attributes['value'].value; 85 | 86 | module= importlib.import_module("Fitness."+self.fitnessClassName) 87 | self.fitnessClass = getattr(module,self.fitnessClassName) 88 | self.fitness = self.fitnessClass() 89 | 90 | self.measurementClassName = self.xmldoc.getElementsByTagName('measurementClass')[0].attributes['value'].value; 91 | self.measurementClassConfFile = self.xmldoc.getElementsByTagName('measurementClassConfFile')[0].attributes['value'].value; 92 | 93 | 94 | module= importlib.import_module("Measurement."+self.measurementClassName) 95 | self.measurementClass = getattr(module,self.measurementClassName) 96 | if self.measurementClassConfFile[-4:]!=".xml": 97 | self.measurementClassConfFile=self.measurementClassConfFile+".xml" 98 | self.measurement=self.measurementClass("./configurationFiles/measurement/"+self.measurementClassConfFile) 99 | self.measurement.init() 100 | 101 | self.dirToSaveResults = self.xmldoc.getElementsByTagName('dirToSaveResults')[0].attributes['value'].value; 102 | self.seedDir=self.xmldoc.getElementsByTagName('seedDir')[0].attributes['value'].value; 103 | self.compilationDir = self.xmldoc.getElementsByTagName('compilationDir')[0].attributes['value'].value; 104 | 105 | ##make sure that dirs end with / 106 | self.dirToSaveResults=self.__fixDirEnd__(self.dirToSaveResults) 107 | self.seedDir=self.__fixDirEnd__(self.seedDir) 108 | self.compilationDir=self.__fixDirEnd__(self.compilationDir) 109 | 110 | ##some other variable initialization 111 | 112 | try: 113 | self.saveWholeSource=self.xmldoc.getElementsByTagName('save_whole_source')[0].attributes['value'].value; 114 | except: 115 | self.saveWholeSource=1 116 | 117 | self.population=Population(); 118 | self.rand=rand; 119 | self.populationsExamined=1; 120 | self.bestIndividualUntilNow=None; 121 | self.waitCounter=0; 122 | self.populationsTested=0 123 | 124 | def __fixDirEnd__(self,dir): 125 | if dir=="": 126 | return dir 127 | if dir[-1:]!="/": 128 | dir=dir+"/" 129 | return dir 130 | 131 | def setupDirs(self,configurationFile): 132 | '''create the results dirs''' 133 | if(not os.path.exists(self.dirToSaveResults)): 134 | os.mkdir(self.dirToSaveResults); 135 | self.timeStart=datetime.datetime.now().strftime("%y-%m-%d-%H-%M"); 136 | self.savedStateDir=self.dirToSaveResults+self.timeStart+"/" 137 | if(not os.path.exists(self.savedStateDir)): 138 | os.mkdir(self.savedStateDir);#create a dir in results dir named after start time that will be used to save state like population and rand state 139 | atexit.register(self.saveRandstate); #register a function which will save rand state at exit 140 | 141 | ''' save configuration src in the results dir ''' 142 | if os.path.exists(self.savedStateDir+"configuration.xml"): 143 | os.remove(self.savedStateDir+"configuration.xml"); 144 | shutil.copy(configurationFile,self.savedStateDir+"configuration.xml" ) 145 | 146 | if os.path.exists(self.savedStateDir+"src"): 147 | shutil.rmtree(self.savedStateDir+"src", ignore_errors=True) 148 | shutil.copytree("./src",self.savedStateDir+"src" ) 149 | 150 | ''' save measurement configuration ''' 151 | if os.path.exists(self.savedStateDir+"measurement.xml"): 152 | os.remove(self.savedStateDir+"measurement.xml", ignore_errors=True) 153 | shutil.copy("./configurationFiles/measurement/"+self.measurementClassConfFile,self.savedStateDir+"measurement.xml") 154 | 155 | '''save assembly compilation in the results dir''' 156 | if os.path.exists(self.savedStateDir+"assembly_compilation"): 157 | shutil.rmtree(self.savedStateDir+"assembly_compilation", ignore_errors=True) 158 | shutil.copytree(self.compilationDir,self.savedStateDir+"assembly_compilation" ) 159 | if os.path.exists(self.savedStateDir+"assembly_compilation/main.s"): 160 | os.remove(self.savedStateDir+"assembly_compilation/main.s")#remove this because they belong to previous run and this can get confusing 161 | self.compilationDir=self.savedStateDir+"assembly_compilation/" 162 | print("New compilationDir is "+self.compilationDir) 163 | 164 | 165 | def __instructions_operands_init__(self): 166 | self.loopSize = self.xmldoc.getElementsByTagName('loopSize')[0].attributes['value'].value; 167 | print ("loop Size: " + self.loopSize); 168 | 169 | self.percentage_clue=self.xmldoc.getElementsByTagName('instruction_percentage_clue')[0].attributes['value'].value; 170 | print("Percentage clue? :"+self.percentage_clue); 171 | self.instruction_types={};#a dictionary of instruction type and the amount of each instruction in loop.. Note useful only when percentage clue is True 172 | self.instructions={}; #a dictionay that has for keeps an array of instructions for every instruction type 173 | self.allInstructionArray=[]; #An array which hold all instructions 174 | self.operands={}; 175 | self.toggleInstructionsList={}; 176 | 177 | itemList = self.xmldoc.getElementsByTagName("instruction_type"); 178 | for instruction_type in itemList: 179 | name=instruction_type.attributes["id"].value; 180 | perc= instruction_type.attributes["perc"].value; 181 | self.instruction_types[name]= int (float(perc) * float(self.loopSize)); '''calculate how many of these instructions will be in the loop''' 182 | if(self.percentage_clue=="True"): 183 | print("amount per instruction type in the loop:") 184 | print (self.instruction_types); 185 | sum=0; 186 | for value in list(self.instruction_types.values()): 187 | sum+=value; 188 | self.loopSize=sum; 189 | print("actual loop size is "+str(self.loopSize)); 190 | 191 | itemList = self.xmldoc.getElementsByTagName("operand"); 192 | print("Available operands\n") 193 | for operandDesc in itemList: 194 | ins_type=operandDesc.attributes["type"].value; 195 | if (ins_type=="immediate" or ins_type=="constant" or ins_type=="automatically_incremented_operand"): 196 | anOperand = Operand(id=operandDesc.attributes["id"].value,type=operandDesc.attributes["type"].value,values=[],min=operandDesc.attributes["min"].value,max=operandDesc.attributes["max"].value,stride=operandDesc.attributes["stride"].value,toggleable=operandDesc.attributes["toggle"].value); 197 | #elif ins_type=="branch_label": 198 | # anOperand = BranchLabel(id=operandDesc.attributes["id"].value,ins_type=operandDesc.attributes["type"].value,values=operandDesc.attributes["values"].value.split(),min=operandDesc.attributes["min"].value,max=operandDesc.attributes["max"].value,stride=operandDesc.attributes["stride"].value,toggleable=operandDesc.attributes["toggle"].value); 199 | else: 200 | anOperand = Operand(id=operandDesc.attributes["id"].value,type=operandDesc.attributes["type"].value,values=operandDesc.attributes["values"].value.split(),toggleable=operandDesc.attributes["toggle"].value); 201 | 202 | print("id "+anOperand.id.__str__()); 203 | print ("values "); 204 | print(anOperand.values); 205 | print("max "+ anOperand.max.__str__()) 206 | print ("min " + anOperand.min.__str__()) 207 | print("stride "+anOperand.stride.__str__()+"\n") 208 | self.operands[anOperand.id] = anOperand; 209 | print("End of available operands\n") 210 | 211 | 212 | itemList = self.xmldoc.getElementsByTagName("instruction"); 213 | print("Available instructions\n") 214 | for instructionDesc in itemList: 215 | name=instructionDesc.attributes["name"].value; 216 | ins_type=instructionDesc.attributes["type"].value; 217 | numOfOperands=instructionDesc.attributes["num_of_operands"].value; 218 | if "format" in instructionDesc.attributes: 219 | anInstruction = Instruction(name,ins_type,numOfOperands,format=instructionDesc.attributes["format"].value,toggleable=instructionDesc.attributes["toggle"].value); 220 | else: 221 | print("Instruction "+name+"doesnt have format specified.. All instructions must have format... Exitting"); 222 | sys.exit(); 223 | #anInstruction = Instruction(name,ins_type,numOfOperands,toggleable=instructionDesc.attributes["toggle"].value); 224 | 225 | if(instructionDesc.attributes["toggle"].value=="True"): 226 | self.toggleInstructionsList[instructionDesc.attributes["name"].value]=1; 227 | 228 | operands=[];#TODO fix this source of bugs.. It's irritating when the num of operands does not match the operands specified in the xml file. and in general is stupid to how the operands specified and the number of operands at the same time.. this redundancy leads to bugs 229 | for i in range(1,int(anInstruction.numOfOperands)+1): 230 | operands.append(self.operands[(instructionDesc.attributes["operand"+i.__str__()].value)].copy()); 231 | anInstruction.setOperands(operands); 232 | print(anInstruction); ##for debugging 233 | self.instructions.setdefault(anInstruction.ins_type,[]).append(anInstruction); 234 | #print (self.instructions); 235 | print("End of available instructions\n"); 236 | 237 | for array in list(self.instructions.values()): 238 | for ins in array: 239 | self.allInstructionArray.append(ins); 240 | print("register initialization"); 241 | #print(self.registerInitStr); 242 | #sys.exit() 243 | 244 | 245 | def printGeneralInputs(self): 246 | print("Debug Inputs"); 247 | print ("Population size: " +self.populationSize); 248 | print ("Mutation Rate: "+self.mutationRate); 249 | print ("Crossover Rate: "+self.crossoverRate); 250 | if(self.crossoverType==Algorithm.ONEPOINT_CROSSOVER): 251 | print("Crossover Type: one point crossover") 252 | elif(self.crossoverType==Algorithm.UNIFORM_CROSSOVER): 253 | print("Crossover Type: uniform crossover") 254 | print ("Uniform Rate: "+self.uniformRate); 255 | print ("Ellitism: "+self.ellitism); 256 | if(self.selectionMethod==Algorithm.TOURNAMENT_SELECTION): 257 | print("Selection Method: Tournament") 258 | print ("Tournament selection size: "+self.tournamentSize); 259 | elif(self.selectionMethod==Algorithm.WHEEL_SELECTION): 260 | print("Selection Method: Wheel Selection") 261 | #print ("Termination Conditions are: "); 262 | #print("If for "+self.populationsToRun+" populations there is no more than "+self.fitnessThreshold+" improvement stop") 263 | print ("ResultsDir: "+self.dirToSaveResults); 264 | print ("compilationDir: "+self.compilationDir); 265 | 266 | 267 | @staticmethod 268 | def returnRunType(configurationFile): 269 | xmldoc = minidom.parse(configurationFile) 270 | run_type = xmldoc.getElementsByTagName('run_type')[0].attributes['value'].value; 271 | if(run_type==Algorithm.PROFILE_RUN): 272 | return Algorithm.PROFILE_RUN; 273 | else: 274 | return Algorithm.INSTRUCTION_RUN 275 | 276 | def saveRandstate(self,postfix=""): 277 | output=open(self.savedStateDir +"rand_state"+postfix+".pkl","wb"); 278 | pickle.dump(self.rand.getstate(),output); 279 | output.close(); 280 | 281 | def loadRandstate(self): 282 | latest=1; 283 | if(not os.path.exists(self.seedDir+"rand_state.pkl")): 284 | for root, dirs, filenames in os.walk(self.seedDir): 285 | for f in filenames: 286 | if("rand_state" in f): 287 | num=int(f.replace("rand_state","").replace(".pkl","")); 288 | if(num>latest): 289 | latest=num; 290 | stateToLoad="rand_state"+str(latest)+".pkl"; 291 | input=open(self.seedDir+stateToLoad,"rb"); 292 | self.rand.setstate(pickle.load(input)); 293 | input.close(); 294 | else: 295 | input=open(self.seedDir+"rand_state.pkl","rb"); 296 | self.rand.setstate(pickle.load(input)); 297 | input.close(); 298 | 299 | '''def __initializeRegisterValues__ (self):#DEPRECATED DON"T USE THIS ANYMORE AS IT MAY LEAD TO BUGS ##remember this functions expects a clean original copy of main.s 300 | for line in fileinput.input(self.compilationDir+"/main.s", inplace=1): #TODO keep in mind that it works with and without / 301 | print(line,end=""); 302 | if "reg init" in line: 303 | print(self.registerInitStr); 304 | fileinput.close(); 305 | 306 | def __initializeMemory__ (self,core=1): #DEPRECATED DON"T USE THIS ANYMORE AS IT MAY LEAD TO BUGS ##remember this functions expects a clean original copy of main.s 307 | for line in fileinput.input(self.compilationDir+"/startup.s", inplace=1): 308 | if "MemoryInit"+str(core) in line: 309 | print (line,end=""); 310 | print(self.memoryInitArray[core-1]); 311 | else: 312 | print(line,end=""); 313 | fileinput.close();''' 314 | 315 | def createInitialPopulation (self): 316 | individuals=[]; 317 | if self.seedDir == "": #random initialization 318 | for i in range(int(self.populationSize)): 319 | individuals.append(self.__randomlyCreateIndividual__()); 320 | self.population=Population(individuals); 321 | else: #initial population based on existing individuals.. Useful for continuing runs that where stopped 322 | newerPop=0; ##find which is the newest population 323 | for root, dirs, filenames in os.walk(self.seedDir): 324 | for f in filenames: 325 | if((".pkl" in f) and ("rand" not in f)): 326 | tokens=re.split('[_.]',f); 327 | popNum=int(tokens[0]); 328 | if (popNum>newerPop): 329 | newerPop=popNum; 330 | input = open(self.seedDir+str(newerPop)+'.pkl',mode="rb"); 331 | self.population=Population.unpickle(input); 332 | input.close(); 333 | 334 | maxId=0;##track the maxId 335 | for indiv in self.population.individuals: 336 | if(indiv.myId>maxId): 337 | maxId=indiv.myId; 338 | 339 | Individual.id=maxId; #the new indiv will start from maxId 340 | self.bestIndividualUntilNow=self.population.getFittest(); #set the best individual seen until now 341 | self.loopSize=self.bestIndividualUntilNow.getInstructions().__len__(); #ensure that loop Size is correct 342 | self.populationsExamined=newerPop+1;#population will start from the newer population 343 | self.loadRandstate(); #load the previous rand state before evolving pop 344 | self.evolvePopulation(); #immediately evolve population 345 | 346 | 347 | 348 | return; 349 | 350 | def measurePopulation (self): 351 | for individual in self.population.individuals: 352 | ##NOTE Due to fluctuations in measurements is desirable to measure the best individual again 353 | while True: #measure until measurement succesful... 354 | try: 355 | measurements=self.__measureIndividual__(individual); 356 | measurement=measurements[0]; 357 | measurement_str="%.6f" % float(measurement); 358 | break; 359 | except (ValueError,IOError): 360 | continue; 361 | 362 | individual.setMeasurementsVector(measurements); 363 | fitnessArray=self.fitness.getFitness(individual) 364 | fitnessValue=fitnessArray[0] 365 | individual.setFitness(fitnessValue) 366 | 367 | measurementStr="" 368 | 369 | for measurement in fitnessArray: 370 | measurement_str=("%.6f" % float(measurement)).replace(".","DOT").strip()+"_" 371 | measurementStr=measurementStr+measurement_str 372 | 373 | if int(self.saveWholeSource)==1: 374 | fpath="" 375 | if(individual.belongsToInitialSeed()): 376 | fpath=self.dirToSaveResults+str(individual.generation)+"_"+str(individual.myId)+"_"+measurementStr+"0_0" +".txt" 377 | else: 378 | fpath=self.dirToSaveResults+str(individual.generation)+"_"+str(individual.myId)+"_"+measurementStr+str(individual.parents[0].myId)+"_"+str(individual.parents[1].myId) +".txt" 379 | shutil.copy(self.compilationDir+"/main.s",fpath) 380 | else: 381 | if(individual.belongsToInitialSeed()): 382 | f = open(self.dirToSaveResults+str(individual.generation)+"_"+str(individual.myId)+"_"+measurementStr+"0_0" +".txt",mode="w") 383 | else: 384 | f = open(self.dirToSaveResults+str(individual.generation)+"_"+str(individual.myId)+"_"+measurementStr+str(individual.parents[0].myId)+"_"+str(individual.parents[1].myId) +".txt",mode="w") 385 | f.write(individual.__str__()); 386 | f.close(); 387 | 388 | 389 | individual.clearParents();#TODO this is just a cheap hack I do for now in order to avoid the recursive grow in length of .pkl files. This is only okay given that I don't actually need anymore that parents.. fon now is okay 390 | ##save a file describing the population so it can be loaded later. useful in case of you want to start a run based on the state of previous runs 391 | output = open(self.savedStateDir+str(self.populationsExamined)+'.pkl','wb'); 392 | self.population.pickle(output); 393 | output.close(); 394 | 395 | ##also save the rand_state 396 | self.saveRandstate(postfix=str(self.populationsExamined)); 397 | self.populationsExamined=self.populationsExamined+1; 398 | self.populationsTested=self.populationsTested+1 399 | 400 | def __measureIndividual__(self,individual): 401 | #####before each individual bring back the original copy of main and startup 402 | self.__bring_back_code_template__(); 403 | ####initialize register values in main.s 404 | #self.__initializeRegisterValues__(); 405 | ####dump memory initialization in startup.s for each core 406 | #for core in range(1,int(int(self.cores)+1)): 407 | # self.__initializeMemory__(core); 408 | ##apply toggling on operands 409 | for key in list(self.toggleInstructionsList.keys()): 410 | self.toggleInstructionsList[key]=1; #first initialize dictionay 411 | for ins in individual.getInstructions(): 412 | if(ins.toggleable=="True"): 413 | if(int(self.toggleInstructionsList[ins.name])%2==1): 414 | ins.toggle(0); 415 | else: 416 | ins.toggle(1); 417 | self.toggleInstructionsList[ins.name]+=1; 418 | #dump the individual into mail loop of main.s 419 | for line in fileinput.input(self.compilationDir+"/main.s", inplace=1): 420 | if "loop_code" in line: 421 | print(individual); 422 | else: 423 | print(line,end=""); 424 | fileinput.close(); 425 | '''at last do the actual measurement''' 426 | measurements= self.__doTheMeasurement__() 427 | return measurements; 428 | 429 | '''def __saveIndiv__(self,individual): 430 | #####before each individual bring back the original copy of main and startup 431 | self.__bring_back_code_template__(); 432 | ####initialize register values in main.s 433 | #self.__initializeRegisterValues__(); 434 | ####dump memory initialization in startup.s for each core 435 | #for core in range(1,int(int(self.cores)+1)): 436 | # self.__initializeMemory__(core); 437 | ##apply toggling on operands 438 | for key in list(self.toggleInstructionsList.keys()): 439 | self.toggleInstructionsList[key]=1; #first initialize dictionay 440 | for ins in individual.getInstructions(): 441 | if(ins.toggleable=="True"): 442 | if(int(self.toggleInstructionsList[ins.name])%2==1): 443 | ins.toggle(0); 444 | else: 445 | ins.toggle(1); 446 | self.toggleInstructionsList[ins.name]+=1; 447 | #dump the individual into mail loop of main.s 448 | for line in fileinput.input(self.compilationDir+"/main.s", inplace=1): 449 | if "loop_code" in line: 450 | print(individual); 451 | else: 452 | print(line,end=""); 453 | fileinput.close();''' 454 | 455 | 456 | def __bring_back_code_template__(self): 457 | #####before each individual bring back the original copy of main and startup 458 | if(os.path.exists(self.compilationDir +"main.s")): 459 | os.remove(self.compilationDir +"main.s") 460 | shutil.copy(self.compilationDir +"main_original.s", self.compilationDir +"main.s") 461 | 462 | 463 | if(os.path.exists(self.compilationDir +"startup.s")): 464 | os.remove(self.compilationDir +"startup.s") 465 | #shutil.copy(self.compilationDir +"startup_original.s", self.compilationDir +"startup.s") 466 | 467 | 468 | def __doTheMeasurement__(self): 469 | self.measurement.setSourceFilePath(self.compilationDir+"/main.s") 470 | return self.measurement.measure() 471 | 472 | def __randomlyCreateIndividual__ (self): 473 | instruction_sequence=[] 474 | if self.percentage_clue=="True": 475 | for ins_type in self.instruction_types.keys():##for each instruction type 476 | for i in range(self.instruction_types[ins_type]): ##create as many instructions as written in the hash 477 | instructions=self.instructions[ins_type]; 478 | instruction_to_copy=instructions[self.rand.randint(0,instructions.__len__()-1)]; #choose random one instruction from the available types 479 | instruction=instruction_to_copy.copy(); 480 | instruction.mutateOperands(self.rand); ##initialize randomy the instruction operands 481 | instruction_sequence.append(instruction); 482 | #print(instruction); 483 | self.rand.shuffle(instruction_sequence); 484 | else: 485 | for i in range(int(self.loopSize)): 486 | instruction=self.rand.choice(self.allInstructionArray).copy(); 487 | instruction.mutateOperands(self.rand); 488 | instruction_sequence.append(instruction); 489 | newIndividual = Individual (instruction_sequence,self.populationsExamined) 490 | return newIndividual; 491 | 492 | def areWeDone(self): 493 | 494 | '''if self.populationsToRun>0: 495 | if self.populationsToRun==self.populationsTested: 496 | self.__saveIndiv__(self.bestIndividualUntilNow) 497 | return True 498 | current_population_best=self.population.getFittest(); 499 | 500 | if(self.bestIndividualUntilNow is None): #only for the first time 501 | self.bestIndividualUntilNow=current_population_best; 502 | self.waitCounter=0; 503 | return False; 504 | 505 | if float(self.best_pop_target)>0 and current_population_best.getFitness()>=float(self.best_pop_target): 506 | #SAVE BEST INDIV SOURCE 507 | self.__saveIndiv__(current_population_best) 508 | return True 509 | 510 | if float(self.avg_pop_target)>0 and self.population.getAvgFitness()>=float(self.avg_pop_target): 511 | return True 512 | 513 | 514 | if (float(current_population_best.getFitness()) < float(self.bestIndividualUntilNow.getFitness())): 515 | self.waitCounter=self.waitCounter+1; 516 | else: 517 | improvement = float(current_population_best.getFitness()) / self.bestIndividualUntilNow.getFitness(); 518 | if (improvement - 1) < float(self.fitnessThreshold): 519 | self.waitCounter=self.waitCounter+1; 520 | else: 521 | self.waitCounter=0; 522 | self.bestIndividualUntilNow=current_population_best;''' 523 | self.waitCounter=self.waitCounter+1 524 | if int(self.waitCounter) == int(self.populationsToRun): 525 | return True; 526 | else: 527 | return False; 528 | 529 | def __roulletteWheelSelection__ (self): 530 | individuals=self.population.individuals; 531 | turn=self.rand.randint(0,individuals[individuals.__len__()-1].cumulativeFitness); 532 | for indiv in self.population.individuals: 533 | if(int(indiv.cumulativeFitness)>=turn): 534 | return indiv; #return the first indiv that is equal or bigger to the random generated value 535 | 536 | def __tournamentSelection__(self): 537 | tournamentIndiv=[]; 538 | for j in range(0,int(self.tournamentSize)): 539 | tournamentIndiv.append(self.population.pickRandomlyAnIndividual(self.rand)); 540 | tournamentPop=Population(tournamentIndiv); 541 | return tournamentPop.getFittest(); 542 | 543 | def evolvePopulation (self): 544 | #individuals=[0]*int(self.populationSize); 545 | individuals=[]; 546 | individuals2=[]; 547 | self.bestIndividualUntilNow=self.population.getFittest() 548 | if self.ellitism=="true": #TODO make the choice to keep more individuals from previous populations TODO FIX THE true 549 | individuals.append(self.bestIndividualUntilNow); 550 | self.bestIndividualUntilNow.generation+=1; #For the next measurement the promoted individuals will be recorded as individuals of the next population.. Is just for avoiding confusions when processing the results 551 | childsCreated=1; 552 | else: 553 | childsCreated=0; 554 | if(self.selectionMethod==Algorithm.WHEEL_SELECTION): 555 | self.population.keepHalfBest();##sightly algorithm change apply roullete on best 50% 556 | self.population.setCumulativeFitness(); 557 | 558 | while childsCreatedcrossover_point): 627 | #do crossover 628 | loop_code1.append(individual2.getInstruction(i).copy()); 629 | loop_code2.append(individual1.getInstruction(i).copy()); 630 | else: #keep instruction as it is 631 | loop_code1.append(individual1.getInstruction(i).copy()); 632 | loop_code2.append(individual2.getInstruction(i).copy()); 633 | children=[]; 634 | children.append(Individual(sequence=loop_code1,generation=self.populationsExamined)); 635 | children.append(Individual(sequence=loop_code2,generation=self.populationsExamined)); 636 | children[0].setParents(individual1,individual2); ##the first parent is the code that remains the same.. the second parent is the code that came after crossover 637 | children[1].setParents(individual2,individual1); 638 | return children; 639 | 640 | #def getFittest(self): 641 | # return self.bestIndividualUntilNow; 642 | 643 | -------------------------------------------------------------------------------- /src/Fitness/DefaultFitness.py: -------------------------------------------------------------------------------- 1 | ''' 2 | Copyright 2019 ARM Ltd. and University of Cyprus 3 | Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, 4 | including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, 5 | and to permit persons to whom the Software is furnished to do so, subject to the following conditions: 6 | The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. 7 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 8 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, 9 | TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 10 | ''' 11 | 12 | class DefaultFitness(object): 13 | ''' 14 | classdocs 15 | ''' 16 | 17 | ''' 18 | Constructor 19 | ''' 20 | def __init__(self): 21 | 22 | 23 | '''Override getFitness to specify other fitness functions 24 | should be called after individual measurements are set (in other words the individual object instance at this point should carry the measurements with it) 25 | The getFitness function can be very sophisticated 26 | e.g. take in account instruction count instruction mix, various metrics like perf counters power etc 27 | the getfitness should return an array where the first the first item MUST be the finess value, followed by any other useful information the user would like to see on file name of the indiviudal 28 | such as specific components ued to calculate the fitness and the individual measurements (like in the below example) 29 | To use the user defined fitness function specify the class name in the fitnessClass parameter in the configuration file 30 | ''' 31 | 32 | def getFitness(self,individual): 33 | fitness=individual.getMeasurements()[0] 34 | toReturn=[] 35 | toReturn.append(fitness) 36 | for value in individual.getMeasurements(): 37 | toReturn.append(value) 38 | return toReturn -------------------------------------------------------------------------------- /src/Fitness/SimplicityTempFitness.py: -------------------------------------------------------------------------------- 1 | ''' 2 | Copyright 2019 ARM Ltd. and University of Cyprus 3 | Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, 4 | including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, 5 | and to permit persons to whom the Software is furnished to do so, subject to the following conditions: 6 | The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. 7 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 8 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, 9 | TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 10 | ''' 11 | 12 | class SimplicityTempFitness(object): # a fitness function for maximizing both temperature and instruction stream simplicity 13 | ''' 14 | classdocs 15 | ''' 16 | 17 | ''' 18 | Constructor 19 | ''' 20 | def __init__(self): 21 | self.MAX_TEMP=100 22 | self.BASELINE_TEMP=40 23 | self.MAX_TEMP_SCORE=self.MAX_TEMP-self.BASELINE_TEMP 24 | self.percentages=[0.5,0.5] 25 | 26 | 27 | '''Override this function to specify other fitness functions 28 | should be called after individual measurements are set (in other words the individual object instance at this point should carry the measurements with it) 29 | The getFitness function can be very sophisticated 30 | e.g. take in account instruction count instruction mix, various metrics like perf counters power etc 31 | the getfitness should return an array where the first the first item MUST be the finess value, followed by any other useful information the user would like to see on file name of the indiviudal 32 | such as specific components ued to calculate the fitness and the individual measurements (like in the below example) 33 | To use the user defined fitness function specify the class name in the fitnessClass parameter in the configuration file 34 | ''' 35 | 36 | def getFitness(self,individual): 37 | insHash={} 38 | instructions=individual.getInstructions() 39 | totalInstructions=len(instructions) 40 | 41 | for ins in instructions: 42 | insHash[ins.name]=1 43 | uniqueIns=len(insHash.keys()) 44 | 45 | tempScore=(individual.getMeasurements()[0]-self.BASELINE_TEMP) / self.MAX_TEMP_SCORE 46 | tempScoreWeighted=tempScore*self.percentages[0] 47 | 48 | instructionsScore=(totalInstructions - uniqueIns)/totalInstructions 49 | instructionsScoreWeighted=instructionsScore*self.percentages[1] 50 | 51 | fitness=tempScoreWeighted+instructionsScoreWeighted 52 | 53 | toReturn=[] 54 | toReturn.append(fitness) 55 | toReturn.append(tempScoreWeighted) 56 | toReturn.append(instructionsScoreWeighted) 57 | for value in individual.getMeasurements(): 58 | toReturn.append(value) 59 | return toReturn -------------------------------------------------------------------------------- /src/Individual.py: -------------------------------------------------------------------------------- 1 | ''' 2 | Copyright 2019 ARM Ltd. and University of Cyprus 3 | Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, 4 | including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, 5 | and to permit persons to whom the Software is furnished to do so, subject to the following conditions: 6 | The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. 7 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 8 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, 9 | TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 10 | ''' 11 | import pickle; 12 | import copy; 13 | 14 | class Individual(object): 15 | ''' 16 | classdocs 17 | ''' 18 | id=0; 19 | 20 | def __init__(self,sequence=[],generation=0): #Note This code assumes that instruction operands are initiated (mutated) from before 21 | Individual.id+=1; 22 | self.myId=Individual.id; 23 | self.sequence=sequence; 24 | self.fitness=0.0; 25 | #self.secondaryMeasurement=0.0; 26 | self.measurements=[] 27 | self.branchLabels={}; 28 | self.cumulativeFitness=0.0; #for wheel selection 29 | self.parents=[]; 30 | self.generation=generation; 31 | self.fixUnconditionalBranchLabels(); 32 | return; 33 | 34 | def addInstruction(self,anInstruction): 35 | self.sequence.append(anInstruction); 36 | 37 | def getInstruction(self,index): 38 | return self.sequence[index]; 39 | 40 | def getInstructions(self): 41 | return self.sequence; 42 | 43 | def fixUnconditionalBranchLabels(self): 44 | automatically_incremented={}; 45 | for ins in self.sequence: 46 | for operand in ins.getOperands(): 47 | if(operand.type=="automatically_incremented_operand"): 48 | if(operand.id in automatically_incremented.keys()): 49 | automatically_incremented[operand.id]+=int(operand.stride); 50 | else: 51 | automatically_incremented[operand.id]=int(operand.min); 52 | operand.currentValue=automatically_incremented[operand.id]; 53 | 54 | def setMeasurementsVector(self,measurements):#by convention the first item of the vector is the fitness value 55 | self.measurements=measurements 56 | 57 | def setFitness(self,fitness): 58 | self.fitness=fitness 59 | 60 | '''def setMeasurementsVector(self,measurements): 61 | self.measurements=measurements 62 | return;''' 63 | 64 | def setCumulativeFitness(self,fitness): #for wheel selection 65 | self.cumulativeFitness=fitness; 66 | 67 | def getFitness(self): 68 | try: 69 | return self.fitness; #by convention the first item of the vector is the fitness value 70 | except: 71 | return self.measurement #to support continuing runs from legacy population pkls 72 | #def getSecondaryMeasurement(self): 73 | # return self.secondaryMeasurement; 74 | def getMeasurements(self): 75 | return self.measurements 76 | 77 | def setParents(self,par1,par2): ##the first parent is the code that remains the same.. the second parent is the code that came after crossover 78 | self.parents.append(par1); 79 | self.parents.append(par2); 80 | 81 | def clearParents(self): 82 | self.parents=None; 83 | 84 | def belongsToInitialSeed(self): 85 | if self.parents: 86 | return False; 87 | else: 88 | return True; 89 | 90 | def __str__(self): 91 | index=0; 92 | output="" 93 | for ins in self.sequence: 94 | output+=str("\t"+ins.__str__()+"\n"); 95 | if str(index) in self.branchLabels.keys(): 96 | output+=str(self.branchLabels[str(index)]+"\n"); 97 | index+=1; 98 | return output; 99 | 100 | def __cmp__(self,other): 101 | if self.getFitness() < other.getFitness(): 102 | return -1 103 | elif self.getFitness()> other.getFitness(): 104 | return 1 105 | else: 106 | return 0 107 | 108 | def pickle(self,filename): 109 | pickle.dump(self, filename); 110 | 111 | @staticmethod 112 | def unpickle(filename): 113 | return pickle.load(filename); 114 | 115 | def copy(self): 116 | return copy.deepcopy(self); 117 | -------------------------------------------------------------------------------- /src/Instruction.py: -------------------------------------------------------------------------------- 1 | ''' 2 | Copyright 2019 ARM Ltd. and University of Cyprus 3 | Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, 4 | including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, 5 | and to permit persons to whom the Software is furnished to do so, subject to the following conditions: 6 | The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. 7 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 8 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, 9 | TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 10 | ''' 11 | import copy 12 | import sys; 13 | 14 | 15 | class Instruction(object): 16 | ''' 17 | classdocs 18 | ''' 19 | 20 | 21 | def __init__(self,name,ins_type,numOfOperands,operands=[],format="op1,op2,op3",toggleable="False"): 22 | ''' 23 | Constructor 24 | ''' 25 | self.name = name; 26 | self.ins_type = ins_type; 27 | self.operands = operands; 28 | self.format=format; 29 | self.format=self.format.replace("\\n","\n"); #replace double backslashes with one backslash to make sure atomic sequences work. for some reason the xml parsing by default ads a backslash to escape characters 30 | self.format=self.format.replace("\\t","\t"); 31 | self.numOfOperands = numOfOperands; 32 | self.toggleable=toggleable; 33 | 34 | def copy(self): 35 | return copy.deepcopy(self); 36 | 37 | 38 | def setOperands (self,operands): 39 | self.operands=operands; 40 | 41 | def getOperand(self,index=0): 42 | if (index>=self.operands.__len__() or index <0): 43 | print("error index out of bounds") 44 | sys.exit; 45 | return self.operands[index]; 46 | 47 | def getOperands(self): 48 | return self.operands; 49 | 50 | def toggle(self,value_index): #It's the algorithm class job to provide the right indexes for toggling 51 | for op in self.operands: 52 | if(op.toggleable=="True"): 53 | op.setCurrentValueByIndex(value_index); 54 | 55 | 56 | def mutateOperands (self,rand):##TODO maybe give an option of changing particular operands 57 | for op in self.operands: 58 | op.mutate(rand); 59 | 60 | ''' def toggle(self): #TODO think if you really need this 61 | ''' 62 | 63 | 64 | def __str__ (self): 65 | 66 | if( (str(self.numOfOperands).strip() == "0") or (self.operands[0].currentValue!= "") ): 67 | representation=self.format 68 | for i in range(0,self.operands.__len__()): 69 | toReplace="op"+str(i+1); 70 | representation=representation.replace(toReplace.__str__(),str(self.operands[i].__str__())); 71 | 72 | #representation=self.name + " " + representation 73 | return representation 74 | else: 75 | representation="name "+self.name + "\ntype "+self.ins_type +"\nformat "+self.format+"\nnumOfOperands "+self.numOfOperands+"\n"; 76 | for i in range(int(self.numOfOperands)): 77 | representation+=("\t"+str(self.operands[i])+"\n"); 78 | return representation; 79 | 80 | -------------------------------------------------------------------------------- /src/Measurement/Measurement.py: -------------------------------------------------------------------------------- 1 | ''' 2 | Copyright 2019 ARM Ltd. and University of Cyprus 3 | Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, 4 | including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, 5 | and to permit persons to whom the Software is furnished to do so, subject to the following conditions: 6 | The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. 7 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 8 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, 9 | TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 10 | ''' 11 | 12 | from abc import ABC, abstractmethod 13 | from xml.dom import minidom 14 | from paramiko import SSHClient, client 15 | import paramiko 16 | import socket 17 | import platform 18 | import visa 19 | import os 20 | 21 | class Measurement(ABC): 22 | ''' 23 | classdocs 24 | ''' 25 | 26 | def __init__(self, confFile): 27 | ''' 28 | Constructor 29 | ''' 30 | self.confFile=confFile 31 | self.xmldoc = minidom.parse(confFile) 32 | 33 | #most of the below are expected to be initialized in init function (should be called after constructor) 34 | self.targetRunDir= None 35 | self.targetHostname= None 36 | self.targetSSHusername= None 37 | self.targetSSHpassword = None 38 | self.coresToUse=None 39 | self.sourceFilePath=None #to be set in setSourceFilePath funtion 40 | super().__init__() #abstract class init 41 | 42 | def init(self): #should be called after constructor.. this can be overridden by child measurement classes to add new or use other configuration parameters.. 43 | 44 | self.targetRunDir= self.tryGetStringValue('targetRunDir') 45 | self.targetHostname= self.tryGetStringValue('targetHostname') 46 | self.targetSSHusername= self.tryGetStringValue('targetSSHusername') 47 | self.targetSSHpassword = self.tryGetStringValue('targetSSHpassword') 48 | coresToUseString=self.tryGetStringValue('coresToUse') 49 | self.coresToUse=[] 50 | for core in coresToUseString.split(" "): 51 | self.coresToUse.append(int(core)) 52 | 53 | def setSourceFilePath(self,sourceFilePath): #should be called before measurement or in the begining of the GA run if the source file path doesn't changes 54 | self.sourceFilePath=sourceFilePath 55 | 56 | ##helper functions to make clearer the code.. on exception the code doesn't terminate immediately but it does produce a warning message.. 57 | ##This is the case because sometimes this might be desirable based on the functionality.. For instance bare metal runs won't use the ssh parameters 58 | def tryGetStringValue(self,key): 59 | try: 60 | value=self.xmldoc.getElementsByTagName(key)[0].attributes['value'].value; 61 | return value 62 | except: 63 | print("Warning failed to read "+str(key)) 64 | 65 | def tryGetIntValue(self,key): 66 | try: 67 | value=int(self.xmldoc.getElementsByTagName(key)[0].attributes['value'].value); 68 | return value 69 | except: 70 | print("Warning failed to read "+str(key)) 71 | 72 | def tryGetFloatValue(self,key): 73 | try: 74 | value=float(self.xmldoc.getElementsByTagName(key)[0].attributes['value'].value); 75 | return value 76 | except: 77 | print("Warning failed to read "+str(key)) 78 | 79 | #this function should return an array of results.. at least one item should be returned.. the defaultFitness.py class (that calculates indidual's fitness) assumes by convention that the first array 80 | #item is the fitness value 81 | @abstractmethod 82 | def measure(self): 83 | pass 84 | 85 | ## utility function for executing commands over ssh connection.. very common functionality 86 | def executeSSHcommand(self,command,continousAttempt=True,max_tries=10): 87 | tries=0 88 | while True: 89 | try: 90 | ssh = SSHClient() 91 | ssh.set_missing_host_key_policy(client.AutoAddPolicy()) 92 | ssh.connect(self.targetHostname, username=self.targetSSHusername, password=self.targetSSHpassword) 93 | stdin,stdout,stderr =ssh.exec_command(command) 94 | lines=[] 95 | for line in stdout.readlines(): 96 | lines.append(line) 97 | ssh.close() 98 | return lines 99 | 100 | except: 101 | if continousAttempt and tries/dev/null;" 31 | execution_command="cd "+self.targetRunDir + " ; ./individual & perf stat -e instructions,cycles -o tmp -p $! sleep "+str(self.timeToMeasure) +" ; pkill individual &> /dev/null;" 32 | output_command="cd "+self.targetRunDir + " ; cat tmp | grep insn | tr ',' '.' | awk '{print $4}'; rm main.s; rm individual; rm tmp; "; 33 | super().executeSSHcommand(compilation_command) 34 | super().executeSSHcommand(execution_command) 35 | stdout=super().executeSSHcommand(output_command) 36 | 37 | ipc=0 38 | 39 | for line in stdout: 40 | #print ("line is "+line) 41 | try: 42 | test=float(line) 43 | ipc=test 44 | except ValueError: 45 | print ("Exception line not ipc") 46 | 47 | 48 | measurements=[]; 49 | measurements.append(ipc); 50 | return measurements; 51 | #return ipc; 52 | -------------------------------------------------------------------------------- /src/Measurement/MeasurementLikwidPower.py: -------------------------------------------------------------------------------- 1 | ''' 2 | Copyright 2019 ARM Ltd. and University of Cyprus 3 | Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, 4 | including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, 5 | and to permit persons to whom the Software is furnished to do so, subject to the following conditions: 6 | The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. 7 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 8 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, 9 | TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 10 | ''' 11 | 12 | from Measurement.Measurement import Measurement 13 | 14 | class MeasurementLikwidPower(Measurement): 15 | ''' 16 | classdocs 17 | ''' 18 | 19 | def __init__(self,confFile): 20 | super().__init__(confFile) 21 | 22 | 23 | def init(self): 24 | super().init() 25 | self.timeToMeasure = self.tryGetIntValue('time_to_measure') 26 | 27 | #the below code is tested to work with likwid 4.3 and 3.1 versions 28 | 29 | def measure(self): 30 | 31 | super().copyFileOverFTP() 32 | compilation_command="cd "+self.targetRunDir + " ; gcc main.s -o individual &>/dev/null;" 33 | execution_command="cd "+self.targetRunDir+" ; " 34 | for core in self.coresToUse: 35 | execution_command+="taskset -c "+str(core)+" ./individual &>/dev/null & " 36 | execution_command+=" sudo likwid-powermeter -s "+str(self.timeToMeasure) +"s > tmp ; pkill individual &> /dev/null;" #make sure that msr module is loaded (modprobe msr) and sudo without password is enabled 37 | output_command="cd "+self.targetRunDir + " ; cat tmp | grep Watt | head -n 1 | awk '{print $3}'; rm main.s; rm individual; rm tmp; "; #this grabs the package power 38 | super().executeSSHcommand(compilation_command) 39 | super().executeSSHcommand(execution_command) 40 | stdout=super().executeSSHcommand(output_command) 41 | 42 | 43 | for line in stdout: 44 | try: 45 | test=float(line) 46 | power_meas=test 47 | except ValueError: 48 | print ("Exception line not power") 49 | 50 | 51 | measurements=[]; 52 | measurements.append(power_meas); 53 | 54 | return measurements; 55 | -------------------------------------------------------------------------------- /src/Operand.py: -------------------------------------------------------------------------------- 1 | ''' 2 | Copyright 2019 ARM Ltd. and University of Cyprus 3 | Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, 4 | including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, 5 | and to permit persons to whom the Software is furnished to do so, subject to the following conditions: 6 | The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. 7 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 8 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, 9 | TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 10 | ''' 11 | import copy 12 | import sys 13 | class Operand(object): 14 | ''' 15 | classdocs 16 | ''' 17 | 18 | 19 | def __init__(self,id,type,values=[],min=0,max=4294967296,stride=1,toggleable="False"): 20 | '''Constructor takes as parameters the values an operand can take either in array or in bounds format''' 21 | self.values = values; 22 | self.min = min; 23 | self.max = max; 24 | self.stride = stride; 25 | self.currentValue=""; 26 | self.id=id; 27 | self.type=type; 28 | self.toggleable=toggleable 29 | if( (int(min)>int(max)) or (int(stride)<=0)): 30 | print("Watch out you should always put a stride above 0 otherwise an infinitive loop will be caused and min should be less or equal than max"); 31 | sys.exit(); 32 | if not self.values: #if possible values not set.. manually create them 33 | if(self.type!="automatically_incremented_operand"): #some types are automatically implemented e.g. branch labels 34 | i=int(self.min); 35 | while(i<=int(self.max)): 36 | self.values.append(i); 37 | i+=int(self.stride); 38 | #self.togleValue=self.values[0]; 39 | 40 | def copy(self): 41 | return copy.deepcopy(self); 42 | 43 | def mutate(self,rand): 44 | '''Basicaly sets as current value a random value from the acceptable range''' 45 | if(self.type=="automatically_incremented_operand"): 46 | self.currentValue=self.min; 47 | else: 48 | self.currentValue=rand.choice(self.values); 49 | 50 | 51 | def getValue(self): 52 | return self.currentValue; 53 | 54 | def setCurrentValueByIndex(self,index): 55 | self.currentValue=self.values[index]; 56 | 57 | '''def toggle(self): 58 | if(self.toggleable=="True"): 59 | if(self.togleValue==self.values[0]): 60 | self.togleValue=self.values[1]; 61 | else: 62 | self.togleValue=self.values[0]; 63 | 64 | def setToggleValueAsCurrent(self): 65 | if(self.toggleable=="True"): 66 | self.currentValue=self.togleValue;''' 67 | 68 | def __str__(self): 69 | if(self.currentValue != ""): 70 | return str(self.currentValue); 71 | else: 72 | return str(self.id); 73 | 74 | -------------------------------------------------------------------------------- /src/Population.py: -------------------------------------------------------------------------------- 1 | ''' 2 | Copyright 2019 ARM Ltd. and University of Cyprus 3 | Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, 4 | including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, 5 | and to permit persons to whom the Software is furnished to do so, subject to the following conditions: 6 | The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. 7 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 8 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, 9 | TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 10 | ''' 11 | 12 | import operator 13 | import pickle 14 | 15 | class Population(object): 16 | ''' 17 | classdocs 18 | ''' 19 | 20 | 21 | def __init__(self,individuals=[]): 22 | self.individuals=individuals; 23 | 24 | 25 | def getIndividual(self,index): 26 | return self.individuals[index]; 27 | 28 | def getFittest(self): 29 | best_value=float(self.individuals[0].getFitness()); 30 | best_indiv=self.individuals[0]; 31 | for i in range (self.individuals.__len__()): 32 | if float(self.individuals[i].getFitness()) > best_value: 33 | best_value=float(self.individuals[i].getFitness()) 34 | best_indiv=self.individuals[i]; 35 | return best_indiv; 36 | 37 | def getAvgFitness(self): 38 | sum=0 39 | for indiv in self.individuals: 40 | sum+=float(indiv.getFitness()) 41 | avg=sum/self.individuals.__len__() 42 | return avg 43 | def getSize (self): 44 | return self.individuals.__len__() 45 | 46 | def pickRandomlyAnIndividual(self,rand): 47 | return rand.choice(self.individuals); 48 | 49 | def setCumulativeFitness(self): #for roulette wheel selection 50 | sum=0.0; 51 | self.individuals[0].setCumulativeFitness(int(self.individuals[0].getFitness()*1000000)); 52 | for i in range(1,self.individuals.__len__()): 53 | fitness=int(self.individuals[i].getFitness()*1000000); 54 | self.individuals[i].setCumulativeFitness(self.individuals[i-1].cumulativeFitness+fitness); 55 | 56 | def sortByFitessToWeakest(self): 57 | self.individuals.sort(key=operator.attrgetter('fitness'),reverse=True) 58 | 59 | def sortByWeakestToFitess(self): 60 | self.individuals.sort(key=operator.attrgetter('fitness')) 61 | #sorted(self.individuals,key=lambda obj: obj.fitness, reverse=True); 62 | 63 | def saveIndividual (self,index,individual): 64 | self.individuals[index]=individual; 65 | 66 | def __str__(self): 67 | output=""; 68 | for code in self.individuals: 69 | output+=str(code.__str__()+"\n"); 70 | return output; 71 | 72 | def keepHalfBest(self): 73 | self.sortByFitessToWeakest(); 74 | half=int(self.individuals.__len__()/2); 75 | newList=[] 76 | for i in range(0,half): 77 | newList.append(self.individuals[i]); 78 | self.individuals=newList; 79 | 80 | 81 | def pickle(self,filename): 82 | pickle.dump(self, filename); 83 | 84 | @staticmethod 85 | def unpickle(filename): 86 | return pickle.load(filename); -------------------------------------------------------------------------------- /src/__init__.py: -------------------------------------------------------------------------------- 1 | ''' 2 | Copyright 2019 ARM Ltd. and University of Cyprus 3 | Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, 4 | including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, 5 | and to permit persons to whom the Software is furnished to do so, subject to the following conditions: 6 | The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. 7 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 8 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, 9 | TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 10 | ''' 11 | import Instruction 12 | import Operand 13 | from random import Random 14 | from Algorithm import Algorithm 15 | from Individual import Individual 16 | import sys 17 | 18 | 19 | '''1st argument full path to configuration file.''' 20 | 21 | rand = Random (); #to use a fixed seed give an int argument 22 | algorithm = Algorithm (sys.argv[1],rand) 23 | algorithm.createInitialPopulation(); 24 | algorithm.measurePopulation(); 25 | while algorithm.areWeDone()!= True: 26 | algorithm.evolvePopulation() 27 | algorithm.measurePopulation() 28 | fittest=algorithm.getFittest(); 29 | print ("Fittest individual is:\n " + fittest.__str__() + "with measurement equal to "+str(fittest.getMeasurement())); 30 | 31 | -------------------------------------------------------------------------------- /src/parseGeneticResults.py: -------------------------------------------------------------------------------- 1 | ''' 2 | Copyright 2019 ARM Ltd. and University of Cyprus 3 | Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, 4 | including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, 5 | and to permit persons to whom the Software is furnished to do so, subject to the following conditions: 6 | The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. 7 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 8 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, 9 | TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 10 | ''' 11 | import os 12 | import sys 13 | import pickle 14 | import pprint; 15 | from Population import Population 16 | from Individual import Individual 17 | from Instruction import Instruction 18 | from Operand import Operand 19 | import re; 20 | 21 | 22 | path = sys.argv[1] 23 | files=[] 24 | for root, dirs, filenames in os.walk(path): #takes as input the dir with the saved state 25 | for f in filenames: 26 | if((".pkl" in f) and ("rand" not in f)): 27 | files.append(f); 28 | 29 | files.sort(key=lambda x: int(x.split('.')[0])); 30 | pop=Population([]); 31 | allValues=""; 32 | allKeys=""; 33 | columns=[]; 34 | theBest=[]; 35 | print("best and average of each generation"); 36 | print("generation best average"); 37 | insHash={}; 38 | 39 | for f in files: 40 | input=open(path+f,"rb"); 41 | pop=pickle.load(input); 42 | input.close(); 43 | columns.append(f.split('.')[0]); 44 | best=pop.getFittest(); 45 | theBest.append(best); 46 | sum=0.0; 47 | count=0; 48 | for indiv in pop.individuals: 49 | sum+=float(indiv.getFitness()); 50 | count+=1; 51 | for ins in indiv.sequence: 52 | 53 | if(ins.name in insHash.keys()): 54 | insHash[ins.name]+=1; 55 | else: 56 | insHash[ins.name]=1; 57 | sorted(insHash,key=lambda key: insHash[key]); 58 | #print(insHash); 59 | allKeys+=list(insHash.keys()).__str__()+"\n"; 60 | allValues+=insHash.values().__str__()+"\n"; 61 | average=sum/count; 62 | for key in list(insHash.keys()): #clear the hash for the next population 63 | insHash[key]=0; 64 | 65 | print(str(columns[-1])+" "+str(round(float(best.getFitness()),6))+" "+str(round(float(average),6)) ); 66 | #print (allKeys); 67 | print("end of generation best average"); 68 | 69 | values=re.sub("[A-Za-z]", "", allValues); 70 | values=re.sub("[\[\]\(\),_]", "", values); 71 | values=values.strip(' \n'); 72 | data=[]; 73 | totalSize=pop.individuals[0].getInstructions().__len__()*pop.getSize(); 74 | #print (values); 75 | for row in values.split("\n"): 76 | #print(row); 77 | data.append([float(float(s)/float(totalSize)) for s in row.split()]) 78 | #data.append([int(s) for s in row.split()]) 79 | #data=pprint.pformat (data); 80 | 81 | 82 | rows=re.sub("[\[,\'\]]", "", allKeys); 83 | rows=re.sub("[\{\}\':,]", "", rows); 84 | rows=rows.strip(' \n'); 85 | 86 | for column in rows.split("\n"): 87 | #print(row); 88 | rows=[str(s) for s in column.split()] 89 | 90 | 91 | print("Instruction Mix per generation"); 92 | 93 | print (" "+' '.join(rows)); 94 | 95 | for i in range(columns.__len__()): 96 | print (columns[i],end=" "); 97 | for j in range(rows.__len__()): 98 | print(round(data[i][j],2),end=" "); 99 | print(""); 100 | 101 | #print(values); 102 | 103 | print("Instruction Mix for best of each generation"); 104 | 105 | print (" "+' '.join(rows)); 106 | 107 | 108 | loopSize=theBest[0].getInstructions().__len__() 109 | 110 | i=1; 111 | for indiv in theBest: 112 | for key in list(insHash.keys()): #clear the hash for the next individual 113 | insHash[key]=0; 114 | for ins in indiv.sequence: 115 | if(ins.name in insHash.keys()): 116 | insHash[ins.name]+=1; 117 | else: 118 | insHash[ins.name]=1; 119 | sorted(insHash,key=lambda key: insHash[key]); 120 | print(str(i),end=" "); 121 | i+=1; 122 | for key in list(insHash.keys()): 123 | print(round(float(float(insHash[key])/loopSize),2),end=" "); 124 | print(""); 125 | 126 | print("Type Mix for best of each generation"); 127 | 128 | typeHash={}; 129 | 130 | i=1; 131 | for indiv in theBest: 132 | for key in list(typeHash.keys()): #clear the hash for the next individual 133 | typeHash[key]=0; 134 | for ins in indiv.sequence: 135 | try: 136 | if(ins.type in typeHash.keys()): 137 | typeHash[ins.type]+=1; 138 | else: 139 | typeHash[ins.type]=1; 140 | except: #in legacy pkls the attribute was type instead of ins_type 141 | if(ins.ins_type in typeHash.keys()): 142 | typeHash[ins.ins_type]+=1; 143 | else: 144 | typeHash[ins.ins_type]=1; 145 | 146 | sorted(typeHash,key=lambda key: typeHash[key]); 147 | print(str(i),end=" "); 148 | i+=1; 149 | for key in list(typeHash.keys()): 150 | print(round(float(float(typeHash[key])/loopSize),2),end=" "); 151 | print(""); 152 | for key in typeHash.keys(): 153 | print(key,end=" ") 154 | 155 | sys.exit(); 156 | 157 | --------------------------------------------------------------------------------