├── code ├── __init__.py ├── run.sh ├── example.py ├── Tests.py ├── full_analysis.py ├── notebooks │ └── skimming axelrod players.ipynb └── initialExploring.py ├── LaTeX └── tex │ ├── main.pdf │ ├── img │ ├── vcs │ │ ├── PR-open.png │ │ ├── commit-log.png │ │ ├── scope-discussion.png │ │ └── feature-discussion.png │ ├── flows │ │ ├── ga_cycle.pdf │ │ └── custom_ga_cycle.pdf │ ├── universitylogo.eps │ ├── descriptive │ │ ├── cor_plot.pdf │ │ ├── k_means.pdf │ │ ├── reg_tree.pdf │ │ ├── best_score_hist.pdf │ │ ├── sequence_scatter.pdf │ │ ├── sequence_plot_score_pt1.pdf │ │ ├── sequence_plot_score_pt2.pdf │ │ ├── sequence_scatter_colour.pdf │ │ ├── sequence_plot_alphabetical_pt1.pdf │ │ └── sequence_plot_alphabetical_pt2.pdf │ ├── dist_matrix │ │ ├── dist_cor.pdf │ │ ├── dist_cos.pdf │ │ ├── dist_ham.pdf │ │ ├── dist_jac.pdf │ │ └── dist_ham_binned.pdf │ ├── examples │ │ ├── cosinesimilarity.png │ │ └── tit_for_tat_LUD.pdf │ ├── plots │ │ ├── MUT_POT_bs_v_gen_all.pdf │ │ ├── NEW_GEN_bs_v_gen_all.pdf │ │ ├── GENS_max_bs_v_gens_all.pdf │ │ ├── INIT_POP_bs_v_gens_all.pdf │ │ ├── MUT_FREQ_bs_v_gen_all.pdf │ │ ├── MUT_POT_bs_diff_v_pot_all.pdf │ │ ├── NEW_MUT_FREQ_bs_v_gen_all.pdf │ │ ├── NEW_MUT_POT_bs_v_gen_all.pdf │ │ ├── MUT_FREQ_bs_diff_v_freq_all.pdf │ │ ├── GENS_mean_bs_diff_v_gens_all.pdf │ │ ├── NEW_INIT_POP_bs_v_gen_all_old.pdf │ │ ├── INIT_POP_mean_bs_diff_v_init_pop_all.pdf │ │ └── NEW_INIT_POP_bs_v_gen_non_performers.pdf │ └── universitylogo-eps-converted-to.pdf │ ├── code_snippets │ ├── classWrappingFunction.py │ ├── analysisExample.py │ ├── dev-examples │ │ ├── example-test.py │ │ └── jupyterCells.py │ ├── oldCrossover.py │ ├── grudgerTotalities.py │ ├── newCrossover.py │ ├── mutateFromDojo.py │ ├── tournCode.py │ ├── newCrossoverEX.py │ ├── appendix │ │ ├── reg_tree.py │ │ ├── runGeneticAlgo.py │ │ ├── dist_matrix.py │ │ └── AnalysisRun.py │ ├── populationChecker.py │ ├── generationChecker.py │ ├── mutationFrequencyChecker.py │ ├── mutationPotencyChecker.py │ └── initialPopulationCode.py │ ├── wordcount.bat │ ├── FrontCover.tex │ ├── main.tex │ ├── chapters │ ├── 07-conclusions.tex │ ├── 02-background.tex │ ├── 03-literature.tex │ ├── 04-code.tex │ ├── 01-introduction.tex │ ├── 06-results.tex │ └── 05-implementation.tex │ └── bibliography.bib ├── Final Presentation.pdf ├── .travis.yml ├── .gitignore ├── requirements.txt └── README.md /code/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /LaTeX/tex/main.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GitToby/FinalYearProject/HEAD/LaTeX/tex/main.pdf -------------------------------------------------------------------------------- /Final Presentation.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GitToby/FinalYearProject/HEAD/Final Presentation.pdf -------------------------------------------------------------------------------- /LaTeX/tex/img/vcs/PR-open.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GitToby/FinalYearProject/HEAD/LaTeX/tex/img/vcs/PR-open.png -------------------------------------------------------------------------------- /LaTeX/tex/img/flows/ga_cycle.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GitToby/FinalYearProject/HEAD/LaTeX/tex/img/flows/ga_cycle.pdf -------------------------------------------------------------------------------- /LaTeX/tex/img/universitylogo.eps: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GitToby/FinalYearProject/HEAD/LaTeX/tex/img/universitylogo.eps -------------------------------------------------------------------------------- /LaTeX/tex/img/vcs/commit-log.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GitToby/FinalYearProject/HEAD/LaTeX/tex/img/vcs/commit-log.png -------------------------------------------------------------------------------- /LaTeX/tex/img/descriptive/cor_plot.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GitToby/FinalYearProject/HEAD/LaTeX/tex/img/descriptive/cor_plot.pdf -------------------------------------------------------------------------------- /LaTeX/tex/img/descriptive/k_means.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GitToby/FinalYearProject/HEAD/LaTeX/tex/img/descriptive/k_means.pdf -------------------------------------------------------------------------------- /LaTeX/tex/img/descriptive/reg_tree.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GitToby/FinalYearProject/HEAD/LaTeX/tex/img/descriptive/reg_tree.pdf -------------------------------------------------------------------------------- /LaTeX/tex/img/dist_matrix/dist_cor.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GitToby/FinalYearProject/HEAD/LaTeX/tex/img/dist_matrix/dist_cor.pdf -------------------------------------------------------------------------------- /LaTeX/tex/img/dist_matrix/dist_cos.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GitToby/FinalYearProject/HEAD/LaTeX/tex/img/dist_matrix/dist_cos.pdf -------------------------------------------------------------------------------- /LaTeX/tex/img/dist_matrix/dist_ham.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GitToby/FinalYearProject/HEAD/LaTeX/tex/img/dist_matrix/dist_ham.pdf -------------------------------------------------------------------------------- /LaTeX/tex/img/dist_matrix/dist_jac.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GitToby/FinalYearProject/HEAD/LaTeX/tex/img/dist_matrix/dist_jac.pdf -------------------------------------------------------------------------------- /LaTeX/tex/img/flows/custom_ga_cycle.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GitToby/FinalYearProject/HEAD/LaTeX/tex/img/flows/custom_ga_cycle.pdf -------------------------------------------------------------------------------- /LaTeX/tex/img/vcs/scope-discussion.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GitToby/FinalYearProject/HEAD/LaTeX/tex/img/vcs/scope-discussion.png -------------------------------------------------------------------------------- /LaTeX/tex/img/vcs/feature-discussion.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GitToby/FinalYearProject/HEAD/LaTeX/tex/img/vcs/feature-discussion.png -------------------------------------------------------------------------------- /LaTeX/tex/img/examples/cosinesimilarity.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GitToby/FinalYearProject/HEAD/LaTeX/tex/img/examples/cosinesimilarity.png -------------------------------------------------------------------------------- /LaTeX/tex/img/examples/tit_for_tat_LUD.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GitToby/FinalYearProject/HEAD/LaTeX/tex/img/examples/tit_for_tat_LUD.pdf -------------------------------------------------------------------------------- /LaTeX/tex/img/plots/MUT_POT_bs_v_gen_all.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GitToby/FinalYearProject/HEAD/LaTeX/tex/img/plots/MUT_POT_bs_v_gen_all.pdf -------------------------------------------------------------------------------- /LaTeX/tex/img/plots/NEW_GEN_bs_v_gen_all.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GitToby/FinalYearProject/HEAD/LaTeX/tex/img/plots/NEW_GEN_bs_v_gen_all.pdf -------------------------------------------------------------------------------- /LaTeX/tex/img/descriptive/best_score_hist.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GitToby/FinalYearProject/HEAD/LaTeX/tex/img/descriptive/best_score_hist.pdf -------------------------------------------------------------------------------- /LaTeX/tex/img/descriptive/sequence_scatter.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GitToby/FinalYearProject/HEAD/LaTeX/tex/img/descriptive/sequence_scatter.pdf -------------------------------------------------------------------------------- /LaTeX/tex/img/dist_matrix/dist_ham_binned.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GitToby/FinalYearProject/HEAD/LaTeX/tex/img/dist_matrix/dist_ham_binned.pdf -------------------------------------------------------------------------------- /LaTeX/tex/img/plots/GENS_max_bs_v_gens_all.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GitToby/FinalYearProject/HEAD/LaTeX/tex/img/plots/GENS_max_bs_v_gens_all.pdf -------------------------------------------------------------------------------- /LaTeX/tex/img/plots/INIT_POP_bs_v_gens_all.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GitToby/FinalYearProject/HEAD/LaTeX/tex/img/plots/INIT_POP_bs_v_gens_all.pdf -------------------------------------------------------------------------------- /LaTeX/tex/img/plots/MUT_FREQ_bs_v_gen_all.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GitToby/FinalYearProject/HEAD/LaTeX/tex/img/plots/MUT_FREQ_bs_v_gen_all.pdf -------------------------------------------------------------------------------- /LaTeX/tex/img/plots/MUT_POT_bs_diff_v_pot_all.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GitToby/FinalYearProject/HEAD/LaTeX/tex/img/plots/MUT_POT_bs_diff_v_pot_all.pdf -------------------------------------------------------------------------------- /LaTeX/tex/img/plots/NEW_MUT_FREQ_bs_v_gen_all.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GitToby/FinalYearProject/HEAD/LaTeX/tex/img/plots/NEW_MUT_FREQ_bs_v_gen_all.pdf -------------------------------------------------------------------------------- /LaTeX/tex/img/plots/NEW_MUT_POT_bs_v_gen_all.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GitToby/FinalYearProject/HEAD/LaTeX/tex/img/plots/NEW_MUT_POT_bs_v_gen_all.pdf -------------------------------------------------------------------------------- /LaTeX/tex/img/universitylogo-eps-converted-to.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GitToby/FinalYearProject/HEAD/LaTeX/tex/img/universitylogo-eps-converted-to.pdf -------------------------------------------------------------------------------- /LaTeX/tex/img/plots/MUT_FREQ_bs_diff_v_freq_all.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GitToby/FinalYearProject/HEAD/LaTeX/tex/img/plots/MUT_FREQ_bs_diff_v_freq_all.pdf -------------------------------------------------------------------------------- /LaTeX/tex/img/descriptive/sequence_plot_score_pt1.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GitToby/FinalYearProject/HEAD/LaTeX/tex/img/descriptive/sequence_plot_score_pt1.pdf -------------------------------------------------------------------------------- /LaTeX/tex/img/descriptive/sequence_plot_score_pt2.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GitToby/FinalYearProject/HEAD/LaTeX/tex/img/descriptive/sequence_plot_score_pt2.pdf -------------------------------------------------------------------------------- /LaTeX/tex/img/descriptive/sequence_scatter_colour.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GitToby/FinalYearProject/HEAD/LaTeX/tex/img/descriptive/sequence_scatter_colour.pdf -------------------------------------------------------------------------------- /LaTeX/tex/img/plots/GENS_mean_bs_diff_v_gens_all.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GitToby/FinalYearProject/HEAD/LaTeX/tex/img/plots/GENS_mean_bs_diff_v_gens_all.pdf -------------------------------------------------------------------------------- /LaTeX/tex/img/plots/NEW_INIT_POP_bs_v_gen_all_old.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GitToby/FinalYearProject/HEAD/LaTeX/tex/img/plots/NEW_INIT_POP_bs_v_gen_all_old.pdf -------------------------------------------------------------------------------- /LaTeX/tex/img/descriptive/sequence_plot_alphabetical_pt1.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GitToby/FinalYearProject/HEAD/LaTeX/tex/img/descriptive/sequence_plot_alphabetical_pt1.pdf -------------------------------------------------------------------------------- /LaTeX/tex/img/descriptive/sequence_plot_alphabetical_pt2.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GitToby/FinalYearProject/HEAD/LaTeX/tex/img/descriptive/sequence_plot_alphabetical_pt2.pdf -------------------------------------------------------------------------------- /LaTeX/tex/img/plots/INIT_POP_mean_bs_diff_v_init_pop_all.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GitToby/FinalYearProject/HEAD/LaTeX/tex/img/plots/INIT_POP_mean_bs_diff_v_init_pop_all.pdf -------------------------------------------------------------------------------- /LaTeX/tex/img/plots/NEW_INIT_POP_bs_v_gen_non_performers.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/GitToby/FinalYearProject/HEAD/LaTeX/tex/img/plots/NEW_INIT_POP_bs_v_gen_non_performers.pdf -------------------------------------------------------------------------------- /code/run.sh: -------------------------------------------------------------------------------- 1 | # Run the full analysis script in parallel using gnu-parallel 2 | NUM_STRATEGIES=221 3 | MAX_INDEX=$(($NUM_STRATEGIES - 1)) 4 | parallel --jobs 20 "python full_analysis.py {1}" ::: $(seq 0 $MAX_INDEX) 5 | -------------------------------------------------------------------------------- /.travis.yml: -------------------------------------------------------------------------------- 1 | language: python 2 | 3 | python: 4 | - "3.6" 5 | 6 | #install the modules needed 7 | install: 8 | - pip install -r requirements.txt 9 | 10 | #find and run_one unit tests 11 | script: 12 | - python code/Tests.py -v 13 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | 2 | \.idea/ 3 | code/__pycache__/ 4 | code/output/** 5 | LaTeX/tex/\.vscode/ 6 | LaTeX/tex/__pycache__/ 7 | LaTeX/tex/_minted-main/ 8 | LaTeX/tex/__latexindent_temp.tex 9 | LaTeX/tex/chapters/__latexindent_temp.tex 10 | *.aux 11 | *.gz 12 | -------------------------------------------------------------------------------- /LaTeX/tex/code_snippets/classWrappingFunction.py: -------------------------------------------------------------------------------- 1 | def getSeededPlayer(player_class): 2 | class NewClass(player_class): 3 | def __init__(self, seed=0): 4 | axl.seed(seed) 5 | super().__init__() 6 | 7 | return NewClass 8 | -------------------------------------------------------------------------------- /code/example.py: -------------------------------------------------------------------------------- 1 | from code import full_analysis as fa 2 | 3 | 4 | import axelrod as axl 5 | 6 | run = fa.NewAnalysisRun(mutation_frequency=0.33) 7 | run.save_file_prefix = "example-" 8 | 9 | run.add_opponent(axl.TitForTat()) 10 | run.add_opponent(axl.Random()) 11 | run.add_opponent(axl.Grudger()) 12 | 13 | run.start() 14 | -------------------------------------------------------------------------------- /LaTeX/tex/code_snippets/analysisExample.py: -------------------------------------------------------------------------------- 1 | from full_analysis import NewAnalysisRun 2 | import axelrod as axl 3 | 4 | run = NewAnalysisRun(mutation_frequency=0.33) 5 | run.save_file_prefix = "example-" 6 | 7 | run.add_opponent(axl.TitForTat()) 8 | run.add_opponent(axl.Random()) 9 | run.add_opponent(axl.Grudger()) 10 | 11 | run.start() 12 | -------------------------------------------------------------------------------- /LaTeX/tex/code_snippets/dev-examples/example-test.py: -------------------------------------------------------------------------------- 1 | def test_creation_seqLen(self): 2 | axl.seed(0) 3 | test_length = 10 4 | self.instance = CyclerParams(sequence_length=test_length) 5 | self.assertEqual(self.instance.sequence, [D, C, C, D, C, C, C, C, C, C]) 6 | self.assertEqual(self.instance.sequence_length, test_length) 7 | self.assertEqual(len(self.instance.sequence), test_length) -------------------------------------------------------------------------------- /LaTeX/tex/wordcount.bat: -------------------------------------------------------------------------------- 1 | @echo Writing chapter wordcount to wordcount.html ... 2 | @echo off 3 | REM this need to be chaged to where your wordcount perd dist is and your tex folders 4 | perl C:\Users\toby\Downloads\TeXcount_3_1\texcount.pl -v -html C:\dev\projects\uni\FinalYearProject\LaTeX\tex\chapters\0*.tex > wordcount.html 5 | REM to run with appendix also, remove the 0* and replace with * 6 | @echo Done. -------------------------------------------------------------------------------- /LaTeX/tex/code_snippets/oldCrossover.py: -------------------------------------------------------------------------------- 1 | def crossover_old(self, other_cycler,in_seed=0): 2 | seq1 = self.sequence 3 | seq2 = other_cycler.sequence 4 | 5 | if not in_seed == 0: 6 | # only seed for when we explicitly give it a seed 7 | random.seed(in_seed) 8 | 9 | midpoint = int(random.randint(0, len(seq1)) / 2) 10 | new_seq = seq1[:midpoint] + seq2[midpoint:] 11 | return CyclerParams(sequence=new_seq) 12 | 13 | 14 | 15 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | Axelrod>=4.0.0 2 | -e git+https://github.com/GitToby/axelrod-dojo.git@bf9d9af5f5701ea2ef46e3b6c740a80ffbcdf4e0#egg=axelrod_dojo 3 | cloudpickle>=0.5.2 4 | cycler>=0.10.0 5 | dask>=0.17.1 6 | kiwisolver>=1.0.1 7 | matplotlib>=2.2.0 8 | numpy>=1.14.2 9 | pandas>=0.22.0 10 | prompt-toolkit>=1.0.15 11 | pyparsing>=2.2.0 12 | pyswarm>=0.6 13 | python-dateutil>=2.7.0 14 | pytz>=2018.3 15 | scipy>=1.0.0 16 | six>=1.11.0 17 | toolz>=0.9.0 18 | tqdm>=4.19.7 19 | wcwidth>=0.1.7 -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Final Year Project 2 | 3 | This is the repository for my 20 credit project during my final year at cardiff university. 4 | The project involves researching the best responses to strategies in the Iterated Prisoners Dilemma. 5 | 6 | 7 | My code is split into 2 sections, code and report. 8 | The code section contains examples and notebooks used for research. 9 | The Latex section contains the final report source and compiled pdf. 10 | 11 | The data has been open sourced and can be found here: 12 | https://zenodo.org/record/1261701 13 | -------------------------------------------------------------------------------- /LaTeX/tex/code_snippets/dev-examples/jupyterCells.py: -------------------------------------------------------------------------------- 1 | 2 | # -------- CELL 1 -------- 3 | # The ! means 'run this as a bash script' 4 | ! pip install axelrod 5 | ! pip install axelrod-dojo 6 | ! wget https://raw.githubusercontent.com/GitToby/FinalYearProject/master/code/full_analysis.py 7 | 8 | # -------- CELL 2 -------- 9 | import axelrod as axl 10 | import full_analysis as fa 11 | 12 | run = fa.NewAnalysisRun(population_size=40) 13 | 14 | run.add_opponent(axl.TitForTat()) 15 | run.add_opponent(axl.Random()) 16 | run.add_opponent(axl.Grudger()) 17 | 18 | run.start() -------------------------------------------------------------------------------- /LaTeX/tex/code_snippets/grudgerTotalities.py: -------------------------------------------------------------------------------- 1 | players = (axl.Grudger(), axl.Cycler("C")) 2 | match = axl.Match(players, 200) 3 | match.play() 4 | print("final scores:", match.final_score()) 5 | print("per turn:", match.final_score_per_turn()) 6 | 7 | # >final scores: (600, 600) 8 | # >per turn: (3.0, 3.0) 9 | 10 | players = (axl.Grudger(), axl.Cycler("D")) 11 | match = axl.Match(players, 200) 12 | match.play() 13 | print("final scores:", match.final_score()) 14 | print("per turn:", match.final_score_per_turn()) 15 | 16 | # >final scores: (199, 204) 17 | # >per turn: (0.995, 1.02) 18 | -------------------------------------------------------------------------------- /LaTeX/tex/code_snippets/newCrossover.py: -------------------------------------------------------------------------------- 1 | def crossover(self, other_cycler): 2 | # 10 crossover points: 3 | step_size = int(len(self.get_sequence()) / 10) 4 | # empty starting seq 5 | new_seq = [] 6 | seq1 = self.get_sequence() 7 | seq2 = other_cycler.get_sequence() 8 | i = 0 9 | j = i + step_size 10 | while j <= len(seq1) - step_size: 11 | new_seq = new_seq + seq1[i:j] 12 | new_seq = new_seq + seq2[i + step_size:j + step_size] 13 | i += 2 * +step_size 14 | j += 2 * +step_size 15 | return CyclerParams(sequence=new_seq) 16 | -------------------------------------------------------------------------------- /LaTeX/tex/code_snippets/mutateFromDojo.py: -------------------------------------------------------------------------------- 1 | def mutate(self): 2 | # if the mutation occurs 3 | if random.rand() <= self.mutation_probability: 4 | mutated_sequence = self.get_sequence() 5 | for _ in range(self.mutation_potency): 6 | index_to_change = random.randint(0, len(mutated_sequence)) 7 | # Mutation - change a single gene 8 | if mutated_sequence[index_to_change] == C: 9 | mutated_sequence[index_to_change] = D 10 | else: 11 | mutated_sequence[index_to_change] = C 12 | self.sequence = mutated_sequence 13 | -------------------------------------------------------------------------------- /LaTeX/tex/code_snippets/tournCode.py: -------------------------------------------------------------------------------- 1 | players = [axl.DefectorHunter(), axl.SolutionB1(), 2 | axl.Willing(), axl.TrickyDefector(), 3 | axl.Cycler("C"+"D"*199)] 4 | tournament = axl.Tournament(players,turns=200) 5 | results = tournament.play() 6 | 7 | >>> Index, Rank, Name, Median score, Coop rating, Wins 8 | >>> 0, 0, Cycler: C1,199, 4.9625, 0.005, 4.0 9 | >>> 1, 1, Tricky Defector, 2.23625, 0.2475, 1.5 10 | >>> 2, 2, SolutionB1, 1.7675, 0.71775, 2.0 11 | >>> 3, 3, Defector Hunter, 1.508125, 0.995, 0.0 12 | >>> 4, 4, Willing, 1.5, 0.849, 0.5 13 | 14 | -------------------------------------------------------------------------------- /LaTeX/tex/code_snippets/newCrossoverEX.py: -------------------------------------------------------------------------------- 1 | seq1 = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] 2 | seq2 = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] 3 | step_size = int(len(seq1) / 10) 4 | i = 0 5 | j = i + step_size 6 | new_seq = [] 7 | while j <= len(seq1) - step_size: 8 | new_seq = new_seq + seq1[i:j] 9 | new_seq = new_seq + seq2[i + step_size:j + step_size] 10 | i += 2 * +step_size 11 | j += 2 * +step_size 12 | print(seq1) 13 | print(seq2) 14 | print(new_seq) 15 | 16 | # >[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] 17 | # >[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] 18 | # >[1, 1, 0, 0, 1, 1, 0, 0, 1, 1, 0, 0, 1, 1, 0, 0, 1, 1, 0, 0] 19 | -------------------------------------------------------------------------------- /LaTeX/tex/FrontCover.tex: -------------------------------------------------------------------------------- 1 | % !TEX root = ./main.tex 2 | \begin{titlepage} 3 | \begin{center} 4 | \vspace*{2cm} 5 | 6 | \textbf{The Iterated Prisoner's Dilemma} 7 | 8 | \vspace{0.5cm} 9 | Generating Solution Sequences for Strategies Using Genetic Improvement Algorithms 10 | 11 | \vspace{1.5cm} 12 | 13 | \textbf{Toby Devlin} 14 | \vspace{1.5cm} 15 | 16 | \includegraphics[width=0.5\textwidth]{img/universitylogo.eps} 17 | 18 | \vfill 19 | 20 | 21 | B.Sc. Final Year Dissertation\\ 22 | 23 | \vspace{0.5cm} 24 | Cardiff School of Mathematics 25 | 26 | \vspace{1.5cm} 27 | 28 | 29 | 30 | \end{center} 31 | \end{titlepage} -------------------------------------------------------------------------------- /LaTeX/tex/code_snippets/appendix/reg_tree.py: -------------------------------------------------------------------------------- 1 | # We use mean absolut error, as we want to penalise the magnitude of the error linearly 2 | # (rmse would how how the variance in our scores is reduced move by move) 3 | regressor_tree = tree.DecisionTreeRegressor(criterion='mae',max_depth=4) 4 | #Rerun this for # of cs in a sequence. 5 | regressor_tree.fit(df_vectors[move_cols],df_vectors["best_score"],) 6 | 7 | # This will export the graph to a .dot file, use `dot -Tpng .\tree.dot -o tree.png` in cmd to convert to png 8 | # dot_data_clasif = tree.export_graphviz(regressor_tree, out_file='tree.dot') 9 | dot_data_reg = tree.export_graphviz(regressor_tree, out_file='reg_tree.dot') 10 | # Run on CMD line 11 | !dot -Tpng .\reg_tree.dot -o reg_tree.png 12 | !dot -Tpdf .\reg_tree.dot -o reg_tree.pdf 13 | 14 | i = Image.open('./reg_tree.png') 15 | plt.figure(figsize=(20,20)) 16 | plt.imshow(i) -------------------------------------------------------------------------------- /code/Tests.py: -------------------------------------------------------------------------------- 1 | import unittest 2 | from code import full_analysis as fa 3 | import axelrod as axl 4 | 5 | 6 | class MyTestCase(unittest.TestCase): 7 | def test_working(self): 8 | self.assertEqual(True, True) 9 | 10 | def test_false_case(self): 11 | self.assertNotEqual("abcd", "ABCD") 12 | 13 | def test_true_case(self): 14 | self.assertEqual(2 * 4, 8) 15 | 16 | def test_example(self): 17 | run = fa.NewAnalysisRun() 18 | run.save_file_prefix = "example-" 19 | 20 | run.add_opponent(axl.TitForTat()) 21 | run.add_opponent(axl.Random()) 22 | run.add_opponent(axl.Grudger()) 23 | 24 | # must have 12 opponents; 10 Randoms and the other 2 25 | self.assertEqual(len(run.opponent_list), 12) 26 | run.start() 27 | 28 | import glob 29 | all_files = glob.glob("./output/*.csv") 30 | # assert all files were created 31 | self.assertEqual(len(all_files), 12) 32 | 33 | 34 | if __name__ == '__main__': 35 | unittest.main() 36 | -------------------------------------------------------------------------------- /LaTeX/tex/main.tex: -------------------------------------------------------------------------------- 1 | \documentclass{report} 2 | 3 | % settings 4 | \setlength{\parindent}{0cm} 5 | \setlength{\parskip}{1em} 6 | 7 | \usepackage{amssymb} 8 | \usepackage{amsmath} 9 | \usepackage{amsthm} 10 | \usepackage{minted} 11 | \usepackage{graphicx} 12 | \usepackage{booktabs} 13 | \usepackage[export]{adjustbox} 14 | \usepackage[margin=1.5cm, includefoot, footskip=30pt]{geometry} 15 | \usepackage{appendix} 16 | \usepackage{multicol} 17 | \usepackage{rotating} 18 | \usepackage{tikz} 19 | \usepackage{tkz-graph} 20 | \usetikzlibrary{positioning} 21 | % \usepackage{hyperref} 22 | 23 | \graphicspath{} 24 | \includeonly{ 25 | FrontCover, 26 | chapters/01-introduction, 27 | chapters/02-background, 28 | chapters/03-literature, 29 | chapters/04-code, 30 | chapters/05-implementation, 31 | chapters/06-results, 32 | chapters/07-conclusions, 33 | chapters/appendix 34 | } 35 | 36 | \begin{document} 37 | 38 | \include{FrontCover} 39 | 40 | \tableofcontents 41 | 42 | \include{chapters/01-introduction} 43 | \include{chapters/02-background} 44 | \include{chapters/03-literature} 45 | \include{chapters/04-code} 46 | \include{chapters/05-implementation} 47 | \include{chapters/06-results} 48 | \include{chapters/07-conclusions} 49 | 50 | \include{chapters/appendix} 51 | 52 | \listoffigures 53 | \listoftables 54 | \bibliography{bibliography} 55 | \bibliographystyle{unsrt} 56 | 57 | \end{document} -------------------------------------------------------------------------------- /LaTeX/tex/code_snippets/populationChecker.py: -------------------------------------------------------------------------------- 1 | def populationChecker(opponent): 2 | # make a nice file name 3 | file_name = "data/" + str(opponent) \ 4 | .replace(" ", "_") \ 5 | .replace(":", "_") \ 6 | .lower() + "_pop.csv" 7 | 8 | # if the file exists don't run_one, it takes forever, make sure it exists 9 | if not os.path.isfile(file_name): 10 | df_main = pd.DataFrame(data=None, columns=col_names) 11 | for pop_size in populations: 12 | start_time = time.clock() 13 | pop_run = runGeneticAlgo(opponent, 14 | population_size=pop_size, 15 | number_of_game_turns=200, 16 | cycle_length=200, 17 | generations=150, 18 | mutation_probability=0.1, 19 | reset_file=True) 20 | end_time = time.clock() 21 | tmp_df = pd.read_csv(pop_run[0], names=col_names) 22 | tmp_df["population"] = pop_size 23 | tmp_df["time_taken"] = end_time - start_time 24 | df_main = df_main.append(tmp_df, ignore_index=True) 25 | df_main.to_csv(file_name) 26 | print("List Complete:", file_name) 27 | return df_main 28 | else: 29 | print("file already exists, no calcs to do.") 30 | file_df = pd.read_csv(file_name) 31 | # remove first column 32 | file_df = file_df[list(file_df)[1:]] 33 | return file_df 34 | -------------------------------------------------------------------------------- /LaTeX/tex/code_snippets/generationChecker.py: -------------------------------------------------------------------------------- 1 | def generationSizeChecker(opponent): 2 | file_name = "data/" + str(opponent) \ 3 | .replace(" ", "_") \ 4 | .replace(":", "_"). \ 5 | lower() + "_generation.csv" 6 | if not os.path.isfile(file_name): 7 | df_main = pd.DataFrame(data=None, columns=col_names) 8 | for gens in generation_list: 9 | start_time = time.clock() 10 | pop_run = runGeneticAlgo(opponent, 11 | population_size=150, 12 | number_of_game_turns=200, 13 | cycle_length=200, 14 | generations=gens, 15 | mutation_probability=0.1, 16 | reset_file=True) 17 | end_time = time.clock() 18 | tmp_df = pd.read_csv(pop_run[0], names=col_names) 19 | tmp_df["generations"] = gens 20 | tmp_df["time_taken"] = end_time - start_time 21 | tmp_df["opponent"] = str(opponent) 22 | tmp_df["best_score_diff"] = np.append([0], np.diff(tmp_df["best_score"])) 23 | df_main = df_main.append(tmp_df, ignore_index=True) 24 | df_main.to_csv(file_name) 25 | print("List Complete:", file_name) 26 | return df_main 27 | else: 28 | print("file ", file_name, " already exists, no calcs to do.") 29 | file_df = pd.read_csv(file_name) 30 | # remove first column 31 | file_df = file_df[list(file_df)[1:]] 32 | return file_df 33 | -------------------------------------------------------------------------------- /LaTeX/tex/code_snippets/mutationFrequencyChecker.py: -------------------------------------------------------------------------------- 1 | def mutationFrequencyChecker(opponent): 2 | file_name = "data/" + str(opponent).replace(" ", "_") \ 3 | .replace(":", "_") \ 4 | .lower() + "_mutation_frequency.csv" 5 | if not os.path.isfile(file_name): 6 | df_main = pd.DataFrame(data=None, columns=col_names) 7 | for freq in mutation_frequency_list: 8 | start_time = time.clock() 9 | pot_run = runGeneticAlgo(opponent, 10 | population_size=150, 11 | number_of_game_turns=200, 12 | cycle_length=200, 13 | generations=250, 14 | mutation_probability=freq, 15 | mutation_potency=1, 16 | reset_file=True) 17 | end_time = time.clock() 18 | tmp_df = pd.read_csv(pot_run[0], names=col_names) 19 | tmp_df["mutation_frequency"] = freq 20 | tmp_df["time_taken"] = end_time - start_time 21 | tmp_df["opponent"] = str(opponent) 22 | tmp_df["best_score_diff"] = np.append([0], np.diff(tmp_df["best_score"])) 23 | df_main = df_main.append(tmp_df, ignore_index=True) 24 | df_main.to_csv(file_name) 25 | print("List Complete:", file_name) 26 | return df_main 27 | else: 28 | print("file ", file_name, " already exists, no calcs to do.") 29 | file_df = pd.read_csv(file_name) 30 | # remove first column 31 | file_df = file_df[list(file_df)[1:]] 32 | return file_df 33 | -------------------------------------------------------------------------------- /LaTeX/tex/code_snippets/mutationPotencyChecker.py: -------------------------------------------------------------------------------- 1 | def mutationPotencyChecker(opponent): 2 | file_name = "data/" + str(opponent) \ 3 | .replace(" ", "_") \ 4 | .replace(":", "_") \ 5 | .lower() + "_mutation_potency.csv" 6 | if not os.path.isfile(file_name): 7 | df_main = pd.DataFrame(data=None, columns=col_names) 8 | for potency in mutatuon_potency_list: 9 | start_time = time.clock() 10 | pot_run = runGeneticAlgo(opponent, 11 | population_size=150, 12 | number_of_game_turns=200, 13 | cycle_length=200, 14 | generations=250, 15 | mutation_probability=0.1, 16 | mutation_potency=potency, 17 | reset_file=True) 18 | end_time = time.clock() 19 | tmp_df = pd.read_csv(pot_run[0], names=col_names) 20 | tmp_df["mutation_potency"] = potency 21 | tmp_df["time_taken"] = end_time - start_time 22 | tmp_df["opponent"] = str(opponent) 23 | tmp_df["best_score_diff"] = np.append([0], np.diff(tmp_df["best_score"])) 24 | df_main = df_main.append(tmp_df, ignore_index=True) 25 | df_main.to_csv(file_name) 26 | print("List Complete:", file_name) 27 | return df_main 28 | else: 29 | print("file ", file_name, " already exists, no calcs to do.") 30 | file_df = pd.read_csv(file_name) 31 | # remove first column 32 | file_df = file_df[list(file_df)[1:]] 33 | return file_df 34 | -------------------------------------------------------------------------------- /LaTeX/tex/code_snippets/appendix/runGeneticAlgo.py: -------------------------------------------------------------------------------- 1 | def runGeneticAlgo(opponent, population_size=150, number_of_game_turns=200, cycle_length=200, generations=250, 2 | mutation_probability=0.1, mutation_potency=1, reset_file=True): 3 | cycler_class = axl_dojo.CyclerParams 4 | cycler_objective = axl_dojo.prepare_objective(name="score", turns=number_of_game_turns, repetitions=1) 5 | cycler_kwargs = { 6 | "sequence_length": cycle_length, 7 | "mutation_probability":mutation_probability, 8 | "mutation_potency":mutation_potency 9 | } 10 | 11 | output_file_name = "data/" + str(opponent).replace(" ","_") + ".csv" 12 | try: 13 | if reset_file and os.path.isfile(output_file_name): 14 | os.remove(output_file_name) 15 | finally: 16 | print(str(opponent), 17 | "|| pop size:", population_size, 18 | "\tturns:", number_of_game_turns, 19 | "\tcycle len:", cycle_length, 20 | "\tgens:", generations, 21 | "\tmut. rate:",mutation_probability, 22 | "\t, potency:",mutation_potency) 23 | 24 | axl.seed(1) 25 | 26 | population = axl_dojo.Population(params_class=cycler_class, 27 | params_kwargs=cycler_kwargs, 28 | size=population_size, 29 | objective=cycler_objective, 30 | processes=0, 31 | output_filename=output_file_name, 32 | opponents=[opponent], 33 | print_output=False) 34 | population.run(generations) 35 | print("\tAnalysis Complete:",output_file_name) 36 | # Store the file name and opponent name as a tuple 37 | return output_file_name, str(opponent) -------------------------------------------------------------------------------- /LaTeX/tex/code_snippets/initialPopulationCode.py: -------------------------------------------------------------------------------- 1 | def getCyclerParamsPrePop2(pop_size=200, mutation_prop=0.1, muation_pot=1): 2 | pop = [] 3 | if pop_size < 164: 4 | print("population must be 164+,") 5 | return 6 | 7 | # Totalities & Handshakes 8 | handshake_leng = 5 9 | for start in itertools.product("CD", repeat=handshake_leng): 10 | pop.append(axl_dojo.CyclerParams(list(start) + [C] * (200 - handshake_leng))) 11 | pop.append(axl_dojo.CyclerParams(list(start) + [D] * (200 - handshake_leng))) 12 | 13 | # 50-50 14 | pop.append(axl_dojo.CyclerParams([C] * 100 + [D] * 100)) 15 | pop.append(axl_dojo.CyclerParams([D] * 100 + [C] * 100)) 16 | 17 | # Single Change 18 | for i in range(1, 11): 19 | pop.append(axl_dojo.CyclerParams([C] * i + [D] * (200 - i))) 20 | pop.append(axl_dojo.CyclerParams([D] * i + [C] * (200 - i))) 21 | 22 | for i in range(1, 11): 23 | pop.append(axl_dojo.CyclerParams([C] * (200 - i) + [D] * i)) 24 | pop.append(axl_dojo.CyclerParams([D] * (200 - i) + [C] * i)) 25 | 26 | # Matching Tails 27 | for i in range(1, 6): 28 | for j in range(1, 6): 29 | pop.append(axl_dojo.CyclerParams([C] * i + [D] * (200 - (i + j)) + [C] * j)) 30 | pop.append(axl_dojo.CyclerParams([D] * i + [C] * (200 - (i + j)) + [D] * j)) 31 | 32 | # Alternating 33 | pop.append(axl_dojo.CyclerParams([C, D] * 100)) 34 | pop.append(axl_dojo.CyclerParams([D, C] * 100)) 35 | pop.append(axl_dojo.CyclerParams([C, C, D, D] * 50)) 36 | pop.append(axl_dojo.CyclerParams([D, D, C, C] * 50)) 37 | pop.append(axl_dojo.CyclerParams([C, C, C, C, D, D, D, D] * 25)) 38 | pop.append(axl_dojo.CyclerParams([D, D, D, D, C, C, C, C] * 25)) 39 | pop.append(axl_dojo.CyclerParams([C, C, C, C, C, D, D, D, D, D] * 20)) 40 | pop.append(axl_dojo.CyclerParams([D, D, D, D, D, C, C, C, C, C] * 20)) 41 | 42 | seq_len = 200 43 | while len(pop) < pop_size: 44 | random_moves = list(map(axl.Action, np.random.randint(0, 1 + 1, (seq_len, 1)))) 45 | pop.append(axl_dojo.CyclerParams(random_moves)) 46 | 47 | return pop 48 | -------------------------------------------------------------------------------- /LaTeX/tex/code_snippets/appendix/dist_matrix.py: -------------------------------------------------------------------------------- 1 | from sklearn import metrics 2 | 3 | 4 | def CD_map_to_int(x): 5 | # Returns D:0 C:1 6 | return 68 - ord(x) 7 | 8 | 9 | # Itertuples is a list of (index,col1,col2,..) for each row. We sort them by score first 10 | df_vectors = pd.DataFrame() 11 | df_vectors_strings = pd.DataFrame() 12 | df_generation_max = df_generation_max.sort_values('opponent_name') 13 | df_generation_max = df_generation_max.sort_values('best_score') 14 | for tup in df_generation_max.itertuples(): 15 | # index vals mean the indexing starts at 1 so +1 to these: 16 | #(0, 'generation'), (1, 'score_mean'), (2, 'score_median'), (3, 'score_pop_var'), (4, 'score_range'), (5, 'best_score'), (6, 'best_sequence'), (7, 'opponent_name'), (8, 'seed'), (9, 'analysis_name'), (10, 'base_player'), (11, 'stochastic'), (12, 'memory_depth'), (13, 'makes_use_of'), (14, 'score_bin'), (15, 'start_move'), (16, 'num_blocks'), (17, 'mean_block_len'), (18, 'median_block_len') 17 | seq_str = tup[7] 18 | opponent_name = str(tup[8]) 19 | best_score = tup[6] 20 | bins = tup[15] 21 | # Mapping to integers 0 &1 22 | df_vectors[opponent_name] = list( 23 | map(CD_map_to_int, seq_str)) + [best_score] + [seq_str] + [bins] 24 | 25 | # No Mapping Cs & Ds 26 | df_vectors_strings[opponent_name] = list( 27 | seq_str) + [best_score] + [seq_str] + [bins] 28 | 29 | 30 | move_cols = ['move ' + str(x+1) for x in range(200)] 31 | # Transposing and labeling moves as were forming it in a rotated way. 32 | # (Its easier to just make 2 dfs than map one to the other) 33 | df_vectors = df_vectors.transpose() 34 | df_vectors.columns = move_cols + ["best_score", "best_sequence", "score_bins"] 35 | 36 | df_vectors_strings = df_vectors_strings.transpose() 37 | df_vectors_strings.columns = move_cols + \ 38 | ["best_score", "best_sequence", "score_bins"] 39 | 40 | # Manhatten Distance == Hamming Distance in this example 41 | # Intresting examples: cosine(??), hamming, jaccard sim 42 | dist_array_ham = metrics.pairwise.pairwise_distances( 43 | df_vectors[move_cols], metric='hamming') 44 | # add labels 45 | dist_array_ham = pd.DataFrame( 46 | dist_array_ham, index=df_vectors.index, columns=df_vectors.index) 47 | 48 | dist_array_cos = metrics.pairwise.pairwise_distances( 49 | df_vectors[move_cols], metric='cosine') 50 | dist_array_cos = pd.DataFrame( 51 | dist_array_cos, index=df_vectors.index, columns=df_vectors.index) 52 | 53 | dist_array_jac = metrics.pairwise.pairwise_distances( 54 | df_vectors[move_cols], metric='jaccard') 55 | dist_array_cos = pd.DataFrame( 56 | dist_array_cos, index=df_vectors.index, columns=df_vectors.index) 57 | 58 | dist_array_other = metrics.pairwise.pairwise_distances( 59 | df_vectors[move_cols], metric='correlation') 60 | dist_array_cos = pd.DataFrame( 61 | dist_array_cos, index=df_vectors.index, columns=df_vectors.index) 62 | -------------------------------------------------------------------------------- /code/full_analysis.py: -------------------------------------------------------------------------------- 1 | import itertools 2 | 3 | import axelrod as axl 4 | import axelrod_dojo as axl_dojo 5 | import numpy as np 6 | import os 7 | 8 | C, D = axl.Action 9 | 10 | 11 | class NewAnalysisRun: 12 | # default options 13 | opponent_list = [] 14 | output_files = {} 15 | save_directory = "output/" 16 | save_file_prefix = "" 17 | save_file_suffix = "" 18 | global_seed = 0 19 | overwrite_files = True 20 | stochastic_seeds = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] 21 | 22 | def __init__(self, sequence_length=20, 23 | population_size=25, 24 | generation_length=20, 25 | mutation_frequency=0.1, 26 | mutation_potency=1): 27 | self.sequence_length = sequence_length 28 | self.population_size = population_size 29 | self.generation_length = generation_length 30 | self.mutation_frequency = mutation_frequency 31 | self.mutation_potency = mutation_potency 32 | 33 | def get_pre_made_pop(self, pop_size: int): 34 | pop = [] 35 | 36 | # Totalities & Handshakes 37 | handshake_leng = 5 38 | for start in itertools.product("CD", repeat=handshake_leng): 39 | pop.append(axl_dojo.CyclerParams(list(start) + [C] * (200 - handshake_leng))) 40 | pop.append(axl_dojo.CyclerParams(list(start) + [D] * (200 - handshake_leng))) 41 | 42 | # 50-50 43 | pop.append(axl_dojo.CyclerParams([C] * 100 + [D] * 100)) 44 | pop.append(axl_dojo.CyclerParams([D] * 100 + [C] * 100)) 45 | 46 | # Single Change 47 | for i in range(1, 11): 48 | pop.append(axl_dojo.CyclerParams([C] * i + [D] * (200 - i))) 49 | pop.append(axl_dojo.CyclerParams([D] * i + [C] * (200 - i))) 50 | 51 | for i in range(1, 11): 52 | pop.append(axl_dojo.CyclerParams([C] * (200 - i) + [D] * i)) 53 | pop.append(axl_dojo.CyclerParams([D] * (200 - i) + [C] * i)) 54 | 55 | # Matching Tails 56 | for i in range(1, 6): 57 | for j in range(1, 6): 58 | pop.append(axl_dojo.CyclerParams([C] * i + [D] * (200 - (i + j)) + [C] * j)) 59 | pop.append(axl_dojo.CyclerParams([D] * i + [C] * (200 - (i + j)) + [D] * j)) 60 | 61 | # Alternating 62 | pop.append(axl_dojo.CyclerParams([C, D] * 100)) 63 | pop.append(axl_dojo.CyclerParams([D, C] * 100)) 64 | pop.append(axl_dojo.CyclerParams([C, C, D, D] * 50)) 65 | pop.append(axl_dojo.CyclerParams([D, D, C, C] * 50)) 66 | pop.append(axl_dojo.CyclerParams([C, C, C, C, D, D, D, D] * 25)) 67 | pop.append(axl_dojo.CyclerParams([D, D, D, D, C, C, C, C] * 25)) 68 | pop.append(axl_dojo.CyclerParams([C, C, C, C, C, D, D, D, D, D] * 20)) 69 | pop.append(axl_dojo.CyclerParams([D, D, D, D, D, C, C, C, C, C] * 20)) 70 | 71 | # Random Filler 72 | while len(pop) < pop_size: 73 | random_moves = list(map(axl.Action, np.random.randint(0, 1 + 1, (self.sequence_length, 1)))) 74 | pop.append(axl_dojo.CyclerParams(random_moves)) 75 | 76 | return pop 77 | 78 | @staticmethod 79 | def _get_seeded_player_class(player_class): 80 | class NewClass(player_class): 81 | def __init__(self, seed=0): 82 | my_seed = seed # for pickling 83 | axl.seed(my_seed) 84 | super().__init__() 85 | 86 | return NewClass 87 | 88 | def _get_file_name(self, opponent: axl.Player): 89 | return self.save_directory \ 90 | + self.save_file_prefix \ 91 | + str(opponent).replace(" ", "_").replace(":", "@").replace("\\", "(") \ 92 | + self.save_file_suffix \ 93 | + ".csv" 94 | 95 | def add_opponent(self, opponent: axl.Player): 96 | # Stochastic players need re-seeding 97 | if opponent.classifier['stochastic']: 98 | for seed in self.stochastic_seeds: 99 | self.opponent_list.append(self._get_seeded_player_class(type(opponent))(seed)) 100 | # Otherwise its fine to just add them 101 | else: 102 | self.opponent_list.append(opponent) 103 | 104 | def clear_opponent_list(self): 105 | self.opponent_list = [] 106 | 107 | def set_opponent_list(self, new_list: list): 108 | self.opponent_list = list() 109 | self.opponent_list = new_list 110 | 111 | def set_save_directory(self, new_directory: str): 112 | if new_directory[0] == "/": 113 | new_directory = new_directory[1:] 114 | 115 | self.save_directory = new_directory.split("/")[0] + "/" 116 | 117 | def start(self): 118 | print("-------- RUNNING ANALYSIS --------") 119 | print("SEQUENCE_LENGTH:", self.sequence_length) 120 | print("POPULATION_SIZE:", self.population_size) 121 | print("GENERATION_LENGTH:", self.generation_length) 122 | print("MUTATION_FREQUENCY:", self.mutation_frequency) 123 | print("MUTATION_POTENCY:", self.mutation_potency) 124 | print() 125 | print("Save directory is ", "\'" + self.save_directory + "\'") 126 | print("Global seed is set to", self.global_seed) 127 | if self.save_file_prefix == "": 128 | print("No file prefix was given") 129 | if self.save_file_suffix == "": 130 | print("No file suffix was given") 131 | print("--------------------------") 132 | print() 133 | print("There are", len(self.opponent_list), "opponents to analyse") 134 | 135 | if not os.path.exists(self.save_directory): 136 | os.makedirs(self.save_directory) 137 | 138 | if self.overwrite_files: 139 | print("Overwriting files during run") 140 | 141 | cycler_objective = axl_dojo.prepare_objective(name="score", turns=self.sequence_length, repetitions=1) 142 | cycler_kwargs = { 143 | "sequence_length": self.sequence_length, 144 | "mutation_probability": self.mutation_frequency, 145 | "mutation_potency": self.mutation_potency 146 | } 147 | 148 | i = 1 149 | print("Starting analysis...") 150 | print() 151 | for opponent in self.opponent_list: 152 | print(i, "of", len(self.opponent_list), "| Analysing player:", str(opponent), "...") 153 | 154 | if self.overwrite_files: 155 | opponent_file = self._get_file_name(opponent) 156 | try: 157 | os.remove(opponent_file) 158 | print("\tremoved file: " + opponent_file) 159 | except FileNotFoundError: 160 | print("\tcould not remove file: " + opponent_file) 161 | 162 | population = axl_dojo.Population(params_class=axl_dojo.CyclerParams, 163 | params_kwargs=cycler_kwargs, 164 | size=self.population_size, 165 | processes=1, 166 | population=self.get_pre_made_pop(self.population_size), 167 | objective=cycler_objective, 168 | output_filename=self._get_file_name(opponent), 169 | opponents=[opponent]) 170 | 171 | population.run(self.generation_length, print_output=False) 172 | print("{:.2f}% Done.\tSaved to:".format((100 * i) / len(self.opponent_list)), 173 | self._get_file_name(opponent)) 174 | self.output_files[str(opponent)] = self._get_file_name(opponent) 175 | i += 1 176 | 177 | -------------------------------------------------------------------------------- /LaTeX/tex/code_snippets/appendix/AnalysisRun.py: -------------------------------------------------------------------------------- 1 | class NewAnalysisRun: 2 | # default options 3 | opponent_list = [] 4 | output_files = {} 5 | save_directory = "output/" 6 | save_file_prefix = "" 7 | save_file_suffix = "" 8 | global_seed = 0 9 | overwrite_files = True 10 | stochastic_seeds = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] 11 | 12 | def __init__(self, sequence_length=20, 13 | population_size=25, 14 | generation_length=20, 15 | mutation_frequency=0.1, 16 | mutation_potency=1): 17 | self.sequence_length = sequence_length 18 | self.population_size = population_size 19 | self.generation_length = generation_length 20 | self.mutation_frequency = mutation_frequency 21 | self.mutation_potency = mutation_potency 22 | 23 | def get_pre_made_pop(self, pop_size: int): 24 | pop = [] 25 | 26 | # Totalities & Handshakes 27 | handshake_leng = 5 28 | for start in itertools.product("CD", repeat=handshake_leng): 29 | pop.append(axl_dojo.CyclerParams( 30 | list(start) + [C] * (200 - handshake_leng))) 31 | pop.append(axl_dojo.CyclerParams( 32 | list(start) + [D] * (200 - handshake_leng))) 33 | 34 | # 50-50 35 | pop.append(axl_dojo.CyclerParams([C] * 100 + [D] * 100)) 36 | pop.append(axl_dojo.CyclerParams([D] * 100 + [C] * 100)) 37 | 38 | # Single Change 39 | for i in range(1, 11): 40 | pop.append(axl_dojo.CyclerParams([C] * i + [D] * (200 - i))) 41 | pop.append(axl_dojo.CyclerParams([D] * i + [C] * (200 - i))) 42 | 43 | for i in range(1, 11): 44 | pop.append(axl_dojo.CyclerParams([C] * (200 - i) + [D] * i)) 45 | pop.append(axl_dojo.CyclerParams([D] * (200 - i) + [C] * i)) 46 | 47 | # Matching Tails 48 | for i in range(1, 6): 49 | for j in range(1, 6): 50 | pop.append(axl_dojo.CyclerParams( 51 | [C] * i + [D] * (200 - (i + j)) + [C] * j)) 52 | pop.append(axl_dojo.CyclerParams( 53 | [D] * i + [C] * (200 - (i + j)) + [D] * j)) 54 | 55 | # Alternating 56 | pop.append(axl_dojo.CyclerParams([C, D] * 100)) 57 | pop.append(axl_dojo.CyclerParams([D, C] * 100)) 58 | pop.append(axl_dojo.CyclerParams([C, C, D, D] * 50)) 59 | pop.append(axl_dojo.CyclerParams([D, D, C, C] * 50)) 60 | pop.append(axl_dojo.CyclerParams([C, C, C, C, D, D, D, D] * 25)) 61 | pop.append(axl_dojo.CyclerParams([D, D, D, D, C, C, C, C] * 25)) 62 | pop.append(axl_dojo.CyclerParams([C, C, C, C, C, D, D, D, D, D] * 20)) 63 | pop.append(axl_dojo.CyclerParams([D, D, D, D, D, C, C, C, C, C] * 20)) 64 | 65 | # Random Filler 66 | while len(pop) < pop_size: 67 | random_moves = list(map(axl.Action, np.random.randint( 68 | 0, 1 + 1, (self.sequence_length, 1)))) 69 | pop.append(axl_dojo.CyclerParams(random_moves)) 70 | 71 | return pop 72 | 73 | @staticmethod 74 | def _get_seeded_player_class(player_class): 75 | class NewClass(player_class): 76 | def __init__(self, seed=0): 77 | my_seed = seed # for pickling 78 | axl.seed(my_seed) 79 | super().__init__() 80 | 81 | return NewClass 82 | 83 | def _get_file_name(self, opponent: axl.Player): 84 | return self.save_directory \ 85 | + self.save_file_prefix \ 86 | + str(opponent).replace(" ", "_").replace(":", "@").replace("\\", "(") \ 87 | + self.save_file_suffix \ 88 | + ".csv" 89 | 90 | def add_opponent(self, opponent: axl.Player): 91 | # Stochastic players need re-seeding 92 | if opponent.classifier['stochastic']: 93 | for seed in self.stochastic_seeds: 94 | self.opponent_list.append( 95 | self._get_seeded_player_class(type(opponent))(seed)) 96 | # Otherwise its fine to just add them 97 | else: 98 | self.opponent_list.append(opponent) 99 | 100 | def clear_opponent_list(self): 101 | self.opponent_list = [] 102 | 103 | def set_opponent_list(self, new_list: list): 104 | self.opponent_list = list() 105 | self.opponent_list = new_list 106 | 107 | def set_save_directory(self, new_directory: str): 108 | if new_directory[0] == "/": 109 | new_directory = new_directory[1:] 110 | 111 | self.save_directory = new_directory.split("/")[0] + "/" 112 | 113 | def start(self): 114 | print("-------- RUNNING ANALYSIS --------") 115 | print("SEQUENCE_LENGTH:", self.sequence_length) 116 | print("POPULATION_SIZE:", self.population_size) 117 | print("GENERATION_LENGTH:", self.generation_length) 118 | print("MUTATION_FREQUENCY:", self.mutation_frequency) 119 | print("MUTATION_POTENCY:", self.mutation_potency) 120 | print() 121 | print("Save directory is ", "\'" + self.save_directory + "\'") 122 | print("Global seed is set to", self.global_seed) 123 | if self.save_file_prefix == "": 124 | print("No file prefix was given") 125 | if self.save_file_suffix == "": 126 | print("No file suffix was given") 127 | print("--------------------------") 128 | print() 129 | print("There are", len(self.opponent_list), "opponents to analyse") 130 | 131 | if not os.path.exists(self.save_directory): 132 | os.makedirs(self.save_directory) 133 | 134 | if self.overwrite_files: 135 | print("Overwriting files during run") 136 | 137 | cycler_objective = axl_dojo.prepare_objective( 138 | name="score", turns=self.sequence_length, repetitions=1) 139 | cycler_kwargs = { 140 | "sequence_length": self.sequence_length, 141 | "mutation_probability": self.mutation_frequency, 142 | "mutation_potency": self.mutation_potency 143 | } 144 | 145 | i = 1 146 | print("Starting analysis...") 147 | print() 148 | for opponent in self.opponent_list: 149 | print(i, "of", len(self.opponent_list), 150 | "| Analysing player:", str(opponent), "...") 151 | 152 | if self.overwrite_files: 153 | opponent_file = self._get_file_name(opponent) 154 | try: 155 | os.remove(opponent_file) 156 | print("\tremoved file: " + opponent_file) 157 | except FileNotFoundError: 158 | print("\tcould not remove file: " + opponent_file) 159 | 160 | population = axl_dojo.Population(params_class=axl_dojo.CyclerParams, 161 | params_kwargs=cycler_kwargs, 162 | size=self.population_size, 163 | processes=1, 164 | population=self.get_pre_made_pop( 165 | self.population_size), 166 | objective=cycler_objective, 167 | output_filename=self._get_file_name( 168 | opponent), 169 | opponents=[opponent]) 170 | 171 | population.run(self.generation_length, print_output=False) 172 | print("{:.2f}% Done.\tSaved to:".format((100 * i) / len(self.opponent_list)), 173 | self._get_file_name(opponent)) 174 | self.output_files[str(opponent)] = self._get_file_name(opponent) 175 | i += 1 176 | -------------------------------------------------------------------------------- /LaTeX/tex/chapters/07-conclusions.tex: -------------------------------------------------------------------------------- 1 | % !TEX root = ../main.tex 2 | \chapter{Conclusions}\label{ch:conclusions} 3 | This report has looked at the concept, structure and generation of solution sequences to strategies within the IPD game. 4 | It has identified a successful method of generating these solutions using an evolutionary algorithm and executed analysis to find these solutions to the majority of opponents listed in the Python Axelrod Library. 5 | After these solutions were generated analysis was performed into how we could group opponents with similar solutions and considered any patterns that existed within the data we produced. 6 | 7 | Each section of this Chapter reflects on a substantive piece of work giving useful content and/or leading to productive discussion. 8 | Section~\ref{sec:conclusion_approach} looks at the first period of the project; researching relevant topics, writing code, creating relevant tools and building a rapport with supervisors. 9 | % TODO make this sound right ^ 10 | Section~\ref{sec:conclusion_execution} looks at the execution of the algorithm and how well the problem described in Section~\ref{sec:briefOverview} was solved. 11 | Section~\ref{sec:conclusion_application} covers where the work completed in this report can be applied in practice. 12 | Finally Section~\ref{sec:follow_up} identifies areas that could be followed up in further work by myself or others. 13 | 14 | \section{Reflection of Approach}\label{sec:conclusion_approach} 15 | Undertaking this project was the first piece of collaborative work I'd undertaken at Cardiff University and, as such, meant learning new skill to apply to what I would be doing. 16 | My project supervisors were incredibly helpful and inclusive towards questions I had and suggestions to work on. 17 | As with any project there were small teething problems with the scope of work I was expected to complete and the time frame it was expected to be completed in. 18 | These were worked out quickly and the work was then accelerated with regular meetings to review progress and set goals. 19 | The level of supervision allowed me to stay focused on the goal of the project while also peruse areas of the project I personally found interesting. 20 | 21 | The initial part of the project was focused on background to the PD, ML, previous research and work that was to be used further on in the project. 22 | Learning Git and the rest of the content in Chapter~\ref{ch:developingthecodebase} took less time than expected, this lead to an extended period of analysing the algorithm leading to a successful result overall. 23 | This time could have also been used to properly scope and implement a larger expansion of the Axelrod Dojo codebase; I could have picked up more issues that had been previously put forward by the owners on GitHub, increasing this projects contribution to open source research software. 24 | 25 | % Overall the work that was completed regarding code, research and developing topics and tools used was well rounded and ... 26 | 27 | \section{Summary of Analysis Execution}\label{sec:conclusion_execution} 28 | During the analysis we were finding a sequence that will manipulate our opponent into providing us the most number of cooperation moves we can subsequently defect against without retaliation. 29 | The genetic algorithm was just one method of approaching this problem. 30 | While executing the genetic algorithm there could be some extra implantations which could improve runtime performance and the results overall. 31 | \begin{itemize} 32 | \item {We know that the best score per turn we can achieve is a 5.0; adding a check at the end of every generation for the top scoring result for this score per turn may have allowed us to complete the search faster. 33 | The LRT strategies, which took over 900 hours of computation, may have found the best result with one of the initial population but were not added to this report due to unneeded extra computation.} 34 | \item {Suboptimal results were given for 57 strategies, as mentioned in Section~\ref{sec:distance_matracies}. 35 | By definition we know we could continue the search until a result with a defection move on turn 200 occurs before terminating the number of generations} 36 | \item {Improving the genetic algorithm further could have improved runtimes and potentially shown better results. 37 | This could have been done using more sophisticated crossover and mutation methods or the use of multi population models~\cite{whitley2012genetic}.} 38 | \end{itemize} 39 | 40 | Once the data had been collected the analysis was purely descriptive. 41 | The extra data, not included in the output, was limited to some classifiers provided in the Axelrod library. 42 | From observation these didn't add much to the analysis other than the stochastic variable for each opponent. 43 | If any models for predicting solution sequences are to be built, much more data (other than their explicit class structure) would be required from each opponent to identify it from a pool of others. 44 | During my research I found no other content I could use as predictor variables; Section~\ref{sec:follow_up} discusses potential data that could be used to predict sequences in a game environment. 45 | 46 | \section{Applications of results}\label{sec:conclusion_application} 47 | The application of these results can be leveraged clearly in an IPD instance. 48 | As described in Section~\ref{sec:follow_up} and~\ref{sec:results_conclusion} we can `solve' tournaments to win them. 49 | As long as we can identify the opponent (if they are stochastic we would also need to know the seed) in a game we can play our solution sequence to get an optimal result. 50 | However this information typically is not provided; Section~\ref{sec:follow_up} looks at problems with identifying opponents. 51 | 52 | Outside of a purely IPD setting the largest take away is understanding that the algorithm was finding methods of manipulating the opponent for the highest cooperation moves we can defect against without encoring a penalty. 53 | Along with the fact the solution sequence with the largest number of opponents was $C199,1$, representing the idea that, for your own highest benefit, you should be cooperative up until your opponent cannot react any more at which point you should defect. 54 | 55 | \section{Potential follow up work}\label{sec:follow_up} 56 | This report has backed up the idea that there is no existence of one universal strategy that will beat every opponent. 57 | Theoretically, if a method of identifying an opponent without affecting the games scores could be created, a lookup to the results of this report could be introduced and the solution sequence could be played for the remainder of the game. 58 | There are 2 flaws to creating a strategy with this approach: 59 | \begin{enumerate} 60 | \item {As of writing this report there are 231 strategies listed in the Axelrod library. 61 | For simplicity we can create this `perfect' strategy for only non stochastic opponents\footnote{Using only these we will know the exact solution every time}, all 138 that we analysed. 62 | Of these there are 43 solution sequences that contain only non stochastic opponents and another 9 that contain both stochastic and non stochastic, meaning we have a total of 52 sequences to select from as we start a game. 63 | From here we have to predict, with the minimal amount of moves in the game, which sequence to play in order to beat our opponent. 64 | This however leads us to the second problem:} 65 | 66 | \item {Creating enough variance in order to identify the solution sequence to play may ruin our chances to score well. 67 | For example, lets take Collective Strategy as an opponent. 68 | When we start our game we wont know were playing Collective Strategy and we will most likely miss the $CD$ handshake, in turn we will have a constant defector to play against for the rest of the game. 69 | This will lead to us not being able to play $C1,1,197,1$ and getting the $3.0$, we would have to play $D$ to stop our losses.} 70 | \end{enumerate} 71 | 72 | Because of these reasons, in order to beat any opponent we have to first find a way of identify that opponent without harming our opportunity to play our solution sequence. 73 | Further work could be done to investigate a method of predicting which opponent we are playing without impact on the current game being played. 74 | This could involve statistical data generated within a match, a package of information players are allowed to analyse before the game or some technique to play each strategy against one another to identify the player the same way some LRT strategies do. 75 | 76 | This work also opens up the observation that there is a lack of meta-data about each strategy in the Axelrod Library. 77 | The library does a good job of creating a central location for computational IPD tournaments, however the information for separating and identifying strategies, other than their explicit class structure, relies on 5 classifier variables. 78 | Expanding on data regarding each strategy, whether it be analyticity or statistically, may allow more sophisticated algorithms to be applied to research on the complexities of how individual IPD strategies are linked. 79 | 80 | Finally the results found in this report have weak correlations and patterns with regard to the areas of analysis undertaken. 81 | Other areas of mathematics, such as pattern recognition or chaos theory, may be able to provide more clarity in the solution sequence data. -------------------------------------------------------------------------------- /LaTeX/tex/chapters/02-background.tex: -------------------------------------------------------------------------------- 1 | % !TEX root = ../main.tex 2 | 3 | \chapter{Task Methodology}\label{ch:taskBackground} 4 | This chapter will cover sections on how each part of the project will been carried out. 5 | Useful notation is given in Section~\ref{sec:notation} which will allow us to describe a given sequence or set of sequences in a compact form. 6 | We will also look at our specific instance of genetic algorithm that exists in the Axelrod Dojo~\cite{dojoV008} codebase. 7 | Lastly Section~\ref{sec:solutionForm} looks at the what results we will expect from the analysis and the potential problems we want to mitigate. 8 | 9 | \section{Notation}\label{sec:notation} 10 | As described in Chapter 1 this work focuses on identifying best responses against the set of opponents in the Python Axelrod Library. The best responses are in a sequence format with a length of 200. In this Section we introduce a notation which will allow us to represent such sequences in a concise format. 11 | 12 | Let $S$ be a sequence where \(S\in\{C, D\}^L\) and \(C\), \(D\) represents a cooperation, defection respectively. 13 | \(L=200\) is used throughout this report. 14 | Using the binary nature of the sequence elements, we can split up sequences into blocks of consecutive move elements of the same type. 15 | We will use \(B_i\) to denote the block after \(i\) changes of move type from the explicitly stated starting move type. 16 | \begin{itemize} 17 | \item Every move in a block is of the same type, $C$ or $D$; 18 | the type is implicit based on whether \(i\) is even or not and what the starting move type was. 19 | \item We can use the notation \(|B_i|\) to denote the length of the \(i\)th block (number of moves within the block) in the sequence.\(|B_i| \in \mathbb{Z}\) 20 | \end{itemize} 21 | 22 | This means we can write a sequence as a series of blocks: 23 | \[S= B_1 B_2,\ldots,B_n\] 24 | A sequence can also be represented shorthand by specifying the starting move and the length of subsequent blocks: 25 | \[S = C\ |B_1|,|B_2|,\ldots,|B_n|\ \quad \Rightarrow \quad S=\overbrace{C\ldots C}^{|B_1|}\overbrace{D\ldots D}^{|B_2|}\ldots\overbrace{(C|D)\ldots (C|D)}^{|B_n|} \] 26 | We can also construct sequences from repetitions of a sequence of blocks when it makes sense: 27 | \[C\ \big{(}|B_1|,|B_2|,\ \ldots,\ |B_m|\big{)}^{k}\ \quad \Rightarrow \quad \big{(}\small{\overbrace{C\ldots C}^{|B_1|}\overbrace{D\ldots D}^{|B_2|}\ldots\overbrace{D\ldots D}^{|B_m|}}\big{)}^{k-times}\] 28 | The two notations can be combined to add starting and ending blocks to a repeating sequence (shown in the examples). 29 | 30 | It is also possible to define sets of sequences by adding variables to parameters of the sequence. 31 | Appropriate selection of parameters mean the length of sequences should not grow. 32 | \[ \{Ci,l-i\} \quad i\in [a,b] \Rightarrow \{\underbrace{C\ldots C}_{a}\overbrace{D\ldots D}^{l-a},\ \underbrace{C\ldots C}_{a+1}\overbrace{D\ldots D}^{l-(a+1)},\ldots ,\ \underbrace{C\ldots C}_{b}\overbrace{D\ldots D}^{l-b}\} \] 33 | For long sequences where there is no recognisable pattern it is typically easier to describe the solution. 34 | Otherwise we use the notation to describe a sequence. 35 | When we look at solutions there too long to write the sequence directly, in which case we may abbreviate in the following style: $C1,5,2,3,5,6,...=CDDDDDCCDDDCCCCCDDDDDD...$. 36 | 37 | Examples: 38 | \begin{align} 39 | C1,4,3,2 &= CDDDDCCCDD\\ 40 | D(1,1)^{5} &= DCDCDCDCDC\\ 41 | C1,(2,1)^{2},2,1 &= CDDCDDCDDC\\ 42 | \{D[i,5-i]\} \quad i\in [2,4] &= \{DDCCC,\ DDDCC,\ DDDCC\}\\ 43 | \end{align} 44 | 45 | To avoid confusion between a sequence for a given opponent, $S_{O_i}$, and the score for a given sequence we define the score function $f(S_{O_i})$ as follows. 46 | $$f(S_{O_i}):S_{O_i} \rightarrow [0,5]$$ 47 | This score function represents playing the sequence against the given opponent and calculating the score per turn over 200 turns. 48 | In this report the score will not be given its own notation other than the result of the function $f$. 49 | 50 | \section{Customizing an Algorithm}\label{sec:buildingTheAlgorithm} 51 | In order to generate a solution sequence we have to train against each Axelrod opponent. 52 | We will expand on the GA described in Section~\ref{subsec:geneticAlgorithms} to produce our own. 53 | Our implementation of a GA has the following steps: 54 | \begin{enumerate} 55 | \item Start with a predefined population, supplemented with randomly generated member until to size. 56 | \item Each member plays the given opponent with their sequence and returns with the average score per turn. 57 | \item Members of the population are ranked by this average score per turn and the highest scoring 25\% will be kept for the next round. 58 | The remaining 75\% are killed off. 59 | \item The remaining population will then be copied and these copies are mutated to create unique sequences before being merged back in to the main population. 60 | \item The remaining 50\% difference is then made up of mutated results of crossovers from members of the current population or random new members, depending on a random selection algorithm.\footnote{This algorithm had a bug which would change the size of the population in the first generation. 61 | This was fixed after Section~\ref{sec:conclusionOfApproach} was written. 62 | See: https://github.com/Axelrod-Python/axelrod-dojo/issues/43} 63 | \item A generation has now concluded. 64 | Repeat from step 2 until the desired number of generations are finished and a final best sequence is returned. 65 | \end{enumerate} 66 | 67 | \begin{figure}[ht] 68 | \includegraphics[width=1.0\textwidth, center]{./img/flows/custom_ga_cycle} 69 | \caption{High Level Genetic Algorithm Cycle. Extension of Figure~\ref{fig:genericGACycle}}\label{fig:customGAcycle} 70 | \end{figure} 71 | 72 | Figure~\ref{fig:customGAcycle} shows a flow diagram of our cycle. 73 | This is the algorithm we will use in Chapter~\ref{ch:implementation} when analysing the algorithms parameters. 74 | The model means we can create a population of Cycler players, described in Section~\ref{sec:strategiesOfInterest}, and input a sequence of length 200 as a parameter to set off our genetic algorithm. 75 | The subsequent inputs for the populations Cycler players will be created using the genetic mutation and crossover techniques, see Section~\ref{subsec:geneticAlgorithms} for details. 76 | 77 | The looping will be the basis of creating the optimal strategy for each other opponent. 78 | Each step is defined in the Axelrod Dojo \mintinline{python}{Population} and \mintinline{python}{CyclerParams} classes. 79 | Rather than store all the functionality in one place we are able split up aspects of the flow to allow for flexibility in what type of population can be used. 80 | 81 | \section{Solution Form}\label{sec:solutionForm} 82 | In this research our goal will be to use the algorithm, described in Subsection~\ref{sec:buildingTheAlgorithm}, to produce an arbitrary sequence for an opponent, \(S_o\). 83 | This sequence will represent the moves we should play against the opponent to get our largest potential score per turn for a single game of 200 turns. 84 | 85 | This investigation focuses on sequences that will allow us to maximise our score overall, rather than just beating any given opponent. 86 | An analogy of this concept is a team playing a football tournament, but instead of a knockout competition our team is placed in the standings based off the total goals they have scored across the tournament. 87 | More real world applications of these results are discussed in Chapter~\ref{ch:conclusions}. 88 | 89 | Each solution sequence is uniquely generated for each opponent. 90 | If there are two similar opponents, say Grudger and Collective Strategy, these will be independently analysed and individual solutions will be generated. 91 | 92 | If needed, we may re-run analysis on some opponents if there are bugs that arise in the code. 93 | In this case older results will be overwritten and previous analysis discarded. 94 | 95 | When working with stochastic players, we will be seeding the random numbers they use in order to determine the best sequence. 96 | Each stochastic player will be considered under a variety of seeds. 97 | The motive to allow this to happen is described in Section~\ref{sec:stochasticOpponents}. Non-stochastic opponents will be seeded in the code, however there is no change in their behaviour because of this. 98 | 99 | \section{Conclusion} 100 | Here we have looked at the background required to answer a specific question, build a relevant model and understand what were looking for. 101 | The concepts themselves are extensions to the basic models brought up in Chapter~\ref{ch:intro} and should allow us to provide relevant results in a uniform manner. 102 | 103 | We have also looked at what problems may arise from certain areas of solving the task, such as understanding what a best response for a stochastic opponents could be or how to represent a complex solution in a result. 104 | Because of this premeditation of the problem and its components Chapters~\ref{ch:implementation} and~\ref{ch:results} will be more concise. 105 | 106 | After we conclude the runtime of the GA and obtain a solution we will have an output sequence for each opponent. 107 | These solution sequences can then be compared on how similar they are to each other; 108 | what type of opponents and their corresponding `best scores' are and whether there are patterns to how to group strategies, Chapter~\ref{ch:results} contains further details. -------------------------------------------------------------------------------- /code/notebooks/skimming axelrod players.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": { 7 | "collapsed": true 8 | }, 9 | "outputs": [], 10 | "source": [ 11 | "import axelrod as axl" 12 | ] 13 | }, 14 | { 15 | "cell_type": "code", 16 | "execution_count": 26, 17 | "metadata": { 18 | "collapsed": false 19 | }, 20 | "outputs": [], 21 | "source": [ 22 | "y = [x for x in axl.strategies]\n", 23 | "y = sorted(y, key=lambda x: x.name)" 24 | ] 25 | }, 26 | { 27 | "cell_type": "code", 28 | "execution_count": 31, 29 | "metadata": { 30 | "collapsed": false 31 | }, 32 | "outputs": [ 33 | { 34 | "name": "stdout", 35 | "output_type": "stream", 36 | "text": [ 37 | "\\item $\\phi$\n", 38 | "\\item $\\pi$\n", 39 | "\\item $e$\n", 40 | "\\item ALLCorALLD*\n", 41 | "\\item Adaptive\n", 42 | "\\item Adaptive Pavlov 2006\n", 43 | "\\item Adaptive Pavlov 2011\n", 44 | "\\item Adaptive Tit For Tat\n", 45 | "\\item Aggravater\n", 46 | "\\item Alexei\n", 47 | "\\item Alternator\n", 48 | "\\item Alternator Hunter\n", 49 | "\\item Anti Tit For Tat\n", 50 | "\\item AntiCycler\n", 51 | "\\item Appeaser\n", 52 | "\\item Arrogant QLearner*\n", 53 | "\\item Average Copier*\n", 54 | "\\item BackStabber\n", 55 | "\\item Better and Better*\n", 56 | "\\item Bully\n", 57 | "\\item Calculator*\n", 58 | "\\item Cautious QLearner*\n", 59 | "\\item Champion*\n", 60 | "\\item CollectiveStrategy\n", 61 | "\\item Contrite Tit For Tat\n", 62 | "\\item Cooperator\n", 63 | "\\item Cooperator Hunter\n", 64 | "\\item Cycle Hunter\n", 65 | "\\item Cycler CCCCCD\n", 66 | "\\item Cycler CCCD\n", 67 | "\\item Cycler CCCDCD\n", 68 | "\\item Cycler CCD\n", 69 | "\\item Cycler DC\n", 70 | "\\item Cycler DDC\n", 71 | "\\item DBS (LRT)\n", 72 | "\\item Davis\n", 73 | "\\item Defector\n", 74 | "\\item Defector Hunter\n", 75 | "\\item Desperate*\n", 76 | "\\item DoubleCrosser\n", 77 | "\\item DoubleResurrection\n", 78 | "\\item Doubler\n", 79 | "\\item Dynamic Two Tits For Tat*\n", 80 | "\\item EasyGo\n", 81 | "\\item Eatherley*\n", 82 | "\\item EugineNier\n", 83 | "\\item Eventual Cycle Hunter\n", 84 | "\\item Evolved ANN\n", 85 | "\\item Evolved ANN 5\n", 86 | "\\item Evolved ANN 5 Noise 05\n", 87 | "\\item Evolved FSM 16\n", 88 | "\\item Evolved FSM 16 Noise 05\n", 89 | "\\item Evolved FSM 4\n", 90 | "\\item Evolved HMM 5*\n", 91 | "\\item EvolvedLookerUp1\\_1\\_1\n", 92 | "\\item EvolvedLookerUp2\\_2\\_2\n", 93 | "\\item Feld*\n", 94 | "\\item Firm But Fair*\n", 95 | "\\item Fool Me Forever\n", 96 | "\\item Fool Me Once\n", 97 | "\\item Forgetful Fool Me Once*\n", 98 | "\\item Forgetful Grudger\n", 99 | "\\item Forgiver\n", 100 | "\\item Forgiving Tit For Tat\n", 101 | "\\item Fortress3\n", 102 | "\\item Fortress4\n", 103 | "\\item GTFT*\n", 104 | "\\item General Soft Grudger\n", 105 | "\\item Gladstein\n", 106 | "\\item Go By Majority\n", 107 | "\\item Go By Majority 10\n", 108 | "\\item Go By Majority 20\n", 109 | "\\item Go By Majority 40\n", 110 | "\\item Go By Majority 5\n", 111 | "\\item Gradual\n", 112 | "\\item Gradual Killer\n", 113 | "\\item Grofman*\n", 114 | "\\item Grudger\n", 115 | "\\item GrudgerAlternator\n", 116 | "\\item Grumpy\n", 117 | "\\item Handshake\n", 118 | "\\item Hard Go By Majority\n", 119 | "\\item Hard Go By Majority 10\n", 120 | "\\item Hard Go By Majority 20\n", 121 | "\\item Hard Go By Majority 40\n", 122 | "\\item Hard Go By Majority 5\n", 123 | "\\item Hard Prober\n", 124 | "\\item Hard Tit For 2 Tats\n", 125 | "\\item Hard Tit For Tat\n", 126 | "\\item Hesitant QLearner*\n", 127 | "\\item Hopeless*\n", 128 | "\\item Inverse*\n", 129 | "\\item Inverse Punisher\n", 130 | "\\item Joss*\n", 131 | "\\item Knowledgeable Worse and Worse*\n", 132 | "\\item Level Punisher\n", 133 | "\\item Limited Retaliate\n", 134 | "\\item Limited Retaliate 2\n", 135 | "\\item Limited Retaliate 3\n", 136 | "\\item MEM2\n", 137 | "\\item Math Constant Hunter\n", 138 | "\\item Meta Hunter\n", 139 | "\\item Meta Hunter Aggressive\n", 140 | "\\item Meta Majority* (LRT)\n", 141 | "\\item Meta Majority Finite Memory* (LRT)\n", 142 | "\\item Meta Majority Long Memory* (LRT)\n", 143 | "\\item Meta Majority Memory One* (LRT)\n", 144 | "\\item Meta Minority* (LRT)\n", 145 | "\\item Meta Mixer* (LRT)\n", 146 | "\\item Meta Winner* (LRT)\n", 147 | "\\item Meta Winner Deterministic* (LRT)\n", 148 | "\\item Meta Winner Ensemble* (LRT)\n", 149 | "\\item Meta Winner Finite Memory* (LRT)\n", 150 | "\\item Meta Winner Long Memory* (LRT)\n", 151 | "\\item Meta Winner Memory One* (LRT)\n", 152 | "\\item Meta Winner Stochastic* (LRT)\n", 153 | "\\item Michaelos*\n", 154 | "\\item N Tit(s) For M Tat(s)\n", 155 | "\\item NMWE Deterministic* (LRT)\n", 156 | "\\item NMWE Finite Memory* (LRT)\n", 157 | "\\item NMWE Long Memory* (LRT)\n", 158 | "\\item NMWE Memory One* (LRT)\n", 159 | "\\item NMWE Stochastic* (LRT)\n", 160 | "\\item Naive Prober*\n", 161 | "\\item Negation*\n", 162 | "\\item Nice Average Copier*\n", 163 | "\\item Nice Meta Winner* (LRT)\n", 164 | "\\item Nice Meta Winner Ensemble* (LRT)\n", 165 | "\\item Nydegger\n", 166 | "\\item Omega TFT\n", 167 | "\\item Once Bitten\n", 168 | "\\item Opposite Grudger\n", 169 | "\\item PSO Gambler 1\\_1\\_1*\n", 170 | "\\item PSO Gambler 2\\_2\\_2*\n", 171 | "\\item PSO Gambler 2\\_2\\_2 Noise 05*\n", 172 | "\\item PSO Gambler Mem1*\n", 173 | "\\item Predator\n", 174 | "\\item Prober\n", 175 | "\\item Prober 2\n", 176 | "\\item Prober 3\n", 177 | "\\item Prober 4\n", 178 | "\\item Pun1\n", 179 | "\\item Punisher\n", 180 | "\\item Raider\n", 181 | "\\item Random*\n", 182 | "\\item Random Hunter\n", 183 | "\\item Remorseful Prober*\n", 184 | "\\item Resurrection\n", 185 | "\\item Retaliate\n", 186 | "\\item Retaliate 2\n", 187 | "\\item Retaliate 3\n", 188 | "\\item Revised Downing\n", 189 | "\\item Ripoff\n", 190 | "\\item Risky QLearner*\n", 191 | "\\item SelfSteem*\n", 192 | "\\item ShortMem\n", 193 | "\\item Shubik\n", 194 | "\\item Slow Tit For Two Tats 2\n", 195 | "\\item Sneaky Tit For Tat\n", 196 | "\\item Soft Grudger\n", 197 | "\\item Soft Joss*\n", 198 | "\\item SolutionB1\n", 199 | "\\item SolutionB5\n", 200 | "\\item Spiteful Tit For Tat\n", 201 | "\\item Stalker*\n", 202 | "\\item Stein and Rapoport\n", 203 | "\\item Stochastic Cooperator*\n", 204 | "\\item Stochastic WSLS*\n", 205 | "\\item Suspicious Tit For Tat\n", 206 | "\\item TF1\n", 207 | "\\item TF2\n", 208 | "\\item TF3\n", 209 | "\\item Tester\n", 210 | "\\item ThueMorse\n", 211 | "\\item ThueMorseInverse\n", 212 | "\\item Thumper\n", 213 | "\\item Tideman and Chieruzzi\n", 214 | "\\item Tit For 2 Tats\n", 215 | "\\item Tit For Tat\n", 216 | "\\item Tranquilizer*\n", 217 | "\\item Tricky Cooperator\n", 218 | "\\item Tricky Defector\n", 219 | "\\item Tullock*\n", 220 | "\\item Two Tits For Tat\n", 221 | "\\item VeryBad\n", 222 | "\\item Willing*\n", 223 | "\\item Win-Shift Lose-Stay\n", 224 | "\\item Win-Stay Lose-Shift\n", 225 | "\\item Winner12\n", 226 | "\\item Winner21\n", 227 | "\\item Worse and Worse*\n", 228 | "\\item Worse and Worse 2*\n", 229 | "\\item Worse and Worse 3*\n", 230 | "\\item ZD-Extort-2*\n", 231 | "\\item ZD-Extort-2 v2*\n", 232 | "\\item ZD-Extort-4*\n", 233 | "\\item ZD-Extort3*\n", 234 | "\\item ZD-Extortion*\n", 235 | "\\item ZD-GEN-2*\n", 236 | "\\item ZD-GTFT-2*\n", 237 | "\\item ZD-Mem2*\n", 238 | "\\item ZD-Mischief*\n", 239 | "\\item ZD-SET-2*\n", 240 | "\n" 241 | ] 242 | } 243 | ], 244 | "source": [ 245 | "s=\"\"\n", 246 | "for player in y:\n", 247 | " extra_str = \"\"\n", 248 | " if player.classifier[\"stochastic\"]:\n", 249 | " extra_str += \"*\"\n", 250 | " \n", 251 | " if player.classifier[\"long_run_time\"]:\n", 252 | " extra_str += \" (LRT)\"\n", 253 | " \n", 254 | " if player.classifier[\"inspects_source\"]:\n", 255 | " extra_str += \" (ISo)\"\n", 256 | " \n", 257 | " if player.classifier[\"manipulates_source\"]:\n", 258 | " extra_str += \" (MSo)\"\n", 259 | " \n", 260 | " if player.classifier[\"manipulates_state\"]:\n", 261 | " extra_str += \" (St)\" \n", 262 | " \n", 263 | " add_str = \"\\item \" + player.name + extra_str + \"\\n\"\n", 264 | " add_str=add_str.replace(\"_\",\"\\_\")\n", 265 | " s+= add_str\n", 266 | " \n", 267 | "print(s)" 268 | ] 269 | }, 270 | { 271 | "cell_type": "code", 272 | "execution_count": 17, 273 | "metadata": { 274 | "collapsed": false 275 | }, 276 | "outputs": [ 277 | { 278 | "data": { 279 | "text/plain": [ 280 | "{'inspects_source': False,\n", 281 | " 'long_run_time': False,\n", 282 | " 'makes_use_of': set(),\n", 283 | " 'manipulates_source': False,\n", 284 | " 'manipulates_state': False,\n", 285 | " 'memory_depth': 1,\n", 286 | " 'stochastic': True}" 287 | ] 288 | }, 289 | "execution_count": 17, 290 | "metadata": {}, 291 | "output_type": "execute_result" 292 | } 293 | ], 294 | "source": [ 295 | "y[4].classifier" 296 | ] 297 | } 298 | ], 299 | "metadata": { 300 | "kernelspec": { 301 | "display_name": "Python 3", 302 | "language": "python", 303 | "name": "python3" 304 | }, 305 | "language_info": { 306 | "codemirror_mode": { 307 | "name": "ipython", 308 | "version": 3 309 | }, 310 | "file_extension": ".py", 311 | "mimetype": "text/x-python", 312 | "name": "python", 313 | "nbconvert_exporter": "python", 314 | "pygments_lexer": "ipython3", 315 | "version": "3.6.0" 316 | } 317 | }, 318 | "nbformat": 4, 319 | "nbformat_minor": 2 320 | } 321 | -------------------------------------------------------------------------------- /LaTeX/tex/chapters/03-literature.tex: -------------------------------------------------------------------------------- 1 | % !TEX root = ../main.tex 2 | 3 | \chapter{Literature Review}\label{ch:literature} 4 | In this chapter we will look at previous works in areas relevant to this report. 5 | We will look at how these pieces of work developed standards in the areas we are interested in and the history of studying them with computer models. 6 | 7 | Section~\ref{sec:strategiesOfInterest} describes interesting opponents we will look at in more detail during Chapter~\ref{ch:implementation} to provide our algorithm a variety of opponents to be developed against. 8 | 9 | \section{Background}\label{sec:background} 10 | The PD and its a large area of repeated games in GT and has applications in the real world. 11 | This is due to the game being a good example of strategies that give a cooperative benefit to repeated games. 12 | The PD was first formally presented by Albert W. Tucker~\cite{cambell2016thesis, gass2005annotated} and the iterated version was made famous by Robert Axelrods work in the 1980s~\cite{axelrod1980effective}. 13 | It has been used to describe actions of people and governments in situations stretching from warfare~\cite{tooby1988war,aumann1992handbook} and finance~\cite{cable1997finance} to politics~\cite{snidal1985Politics} and relationships~\cite{low2015sex}. 14 | 15 | The Prisoners dilemma became a large research area in the combination of mathematics and computer science after Robert Axelrod published his work named `Effective Choice In The Prisoners Dilemma'~\cite{axelrod1980effective}. 16 | In it he makes an introduction to how tournaments are run and the properties of successful results. 17 | His method of experimenting became the standard for handling the IDP problem. 18 | This was also the first example of computer tournaments, for which he asked for strategies as computer programs which had inputs of the history for both players and resulted in an output move for that next turn. 19 | After the tournament is complete he describes what successful strategies had in common; 20 | It turns out the majority of successful strategies, including the winner, Tit For Tat, have properties of niceness and forgiveness. 21 | Niceness is the property of not defecting to start and forgiveness is the property of forgetting previous defections in a timely manner. 22 | This allowed them to thrive in the tournament and have overall scores that rose above strategies without niceness or forgiveness. 23 | 24 | Since Axelrods' original tournament there have been many research papers on what makes a successful strategy for a specific objective. 25 | For example William H Press and Freeman J Dyson,~\cite{press2012iterated}, looked at how to it is possible to deterministically set an opponents score and Shashi Mittal and Kalyanmoy Deb have looked at satisfying a range of different objectives at once~\cite{mittal2009optimal}. 26 | These specific objectives are the core part of applying GT and the PD in the real word~\cite{rehmeyer2012climateNegotiations,osang2013environmental,Schneier2012doping}. 27 | Objectives are the wrapper for which we can work with real world scenarios, for example in a the cold war it was not the goal to get as many missiles as possible to pass an oppositions defence but to not allow any missiles through your own defence; i.e.minimising your opponents score. 28 | In a football tournament where winning is the number or goals your team scores it does not matter how many goals you get in so long as you score the most overall. 29 | This leads to observing that the IPD describes that a players world is better if their opponent is cooperative. 30 | There are also some very useful applications levering these ideas, such as modelling rent splitting or work assignments~\cite{goldman2015spliddit}; all of which were programmed and deployed to a website for use online at~\cite{spliddit}. 31 | 32 | \section{Strategy Structures}\label{sec:stratergyStructures} 33 | A strategy is a way of defining how to play the IPD. 34 | It is a way of identifying which move to play next based on some parameters; for example the strategy of all $C$ moves is valid, as is a wildly complicated description of what to do based on the last 20 moves of an opponent. 35 | Some examples, further looked into in Section~\ref{sec:strategiesOfInterest}, include creating strategies that extort others to create overall scores it desires or strategies with handshakes for self identification~\cite{knight2017evolution}. 36 | 37 | Individual strategies can be defined in multiple ways~\cite{harper2017reinforcement}. 38 | Each method of representing a strategies has its benefits and drawbacks, typically due to fundamental properties of the strategy (like being stochastic) or that it is more concise to write in a certain way. 39 | Ways of structuring a strategy are shown below: 40 | 41 | \begin{itemize} 42 | \item LookerUpper, for example figure~\ref{fig:tit_for_tat_LUD}. 43 | \item Gambler, for example the Stochastic strategies; examples can be found in~\cite{press2012iterated}. 44 | \item Neural Networks. 45 | \item Finite State Machines, for example figure~\ref{fig:tit_for_tat_FSD}. 46 | \item Hidden Markov Models. 47 | \item Explicit Move List, for example the solutions given in Appendix~\ref{apndx:solutionGroups}. 48 | \item Mixtures. 49 | \end{itemize} 50 | 51 | \subsection{Equivalent Strategies}\label{subsec:equivalentStrategies} 52 | Looking at certain opponents there are occasions some strategies that look indistinguishable from others; 53 | for example two stochastic opponents can can be incredibly similar, but identifying which one is which from their play history could be impossible. 54 | One of the ways we are able to identify strategies is the process of fingerprinting~\cite{Ashlock2004,Ashlock2008,cambell2016thesis}. 55 | However this can be inconclusive and less accurate than desired. 56 | An effective method of identifying equivalent strategies is to write them down in the form of the other. 57 | For example if we can write any strategy in one of the forms given in figures~\ref{fig:tit_for_tat_FSD},~\ref{fig:tit_for_tat_LUD} we know its an equivalent to Tit For Tat. 58 | 59 | The work being completed in this paper is another method of identifying strategies; identifying the best response to them may lead to observations about how and why different strategies act in similar manners. 60 | This wont lead to show strategy equivalence, but it will show how solution equivalence. 61 | 62 | \begin{figure}[ht] 63 | \centering 64 | \begin{minipage}{0.48\textwidth} 65 | \centering 66 | \tikzset{EdgeStyle/.append style = {->} , 67 | LabelStyle/.style = {rectangle, rounded corners, draw, fill = black!5}} 68 | \begin{tikzpicture}[ 69 | startnode/.style={rectangle, rounded corners, draw, fill = blue!5}, 70 | roundnode/.style={circle, draw=black!60, fill=green!10, very thick, minimum size=7mm} 71 | ] 72 | %Nodes 73 | \node[roundnode](initial_state){1}; 74 | \node[startnode](start_node)[left=of initial_state]{Start}; 75 | %Lines 76 | \Edge[label = $C$](start_node)(initial_state) 77 | \Loop[dist = 3cm, dir = NO, label = $C/C$](initial_state.north) 78 | \Loop[dist = 3cm, dir = SO, label = $D/D$](initial_state.south) 79 | \end{tikzpicture} 80 | \caption{Finite State diagram of strategy Tit For Tat}\label{fig:tit_for_tat_FSD} 81 | \end{minipage}\hfill 82 | \begin{minipage}{0.48\textwidth} 83 | \centering 84 | \includegraphics[width=0.6\textwidth, center]{./img/examples/tit_for_tat_LUD.pdf} 85 | \caption{Look Up diagram of strategy Tit For Tat}\label{fig:tit_for_tat_LUD} 86 | \end{minipage} 87 | \end{figure} 88 | 89 | \section{Strategies Of Interest}\label{sec:strategiesOfInterest} 90 | \textbf{Tit For Tat} was the winner of the original axelrod tournament in 1980~\cite{axelrod1980effective}. It s a very basic opponent who has strong forgiveness (it will forgive a defection after 1 move) and generosity (it will start with a cooperation) which is thought to give its well overall score in tournaments. 91 | 92 | \textbf{Alternator} is a `dumb' opponent, i.e. no strategic method at all. All it will do is alternate between cooperation and defection until the game ends. 93 | Playing against an Alternator effectively is to defect the whole game; identifying an alternator to play this sequence of defections without backlash can be difficult. 94 | If we are playing a similar starting strategy such as Collective Strategy defecting in the first two moves will cause us to sore badly overall. 95 | 96 | \textbf{Grudger} can be considered as the most unforgiving strategy that exists. 97 | Starting by cooperating, if you defect even once then the Grudger will defect until the end of the game. 98 | In line with Tit For Tat, our best score will come from not `upsetting' the opponent until the last move where it cant retaliate. 99 | 100 | \textbf{Random} is the most basic stochastic opponent, and like Alternator and Cycler it is `dumb'. 101 | With a probability $p$ of cooperating and $1-p$ of defecting, we can just defect the whole time to beat this opponent; picking up bonus points on its cooperation moves. 102 | 103 | \textbf{EvolvedFSM16} is a 16 state Finite State Machine (FSM) that has been trained with an evolutionary algorithm. It has been optimised for tournament matches, the definition can be found in the axelrod documentation~\cite{axelrodproject}. 104 | 105 | \textbf{CollectiveStrategy} was defined in~\cite{li2009strategy}. 106 | ‘It always cooperates in the first move and defects in the second move. 107 | If the opponent also cooperates in the first move and defects in the second move, CS will cooperate until the opponent defects. Otherwise, CS will always defect.’. 108 | 109 | \textbf{ZDExtortion} is first mentioned in the paper \textit{Iterated Prisoner’s Dilemma contains strategies that dominate any evolutionary opponent}~\cite{press2012iterated}. 110 | In it the authors discuss methods of extorting score from another player. We use this player an example of a difficult to beat player. 111 | 112 | \textbf{Cycler} is a strategy which cycles and repeats a sequence until the end of the match. 113 | For example \mintinline{python}{cycler("CCD")} will play C$2,1,2,1,2,\ldots$ for as long as necessary. 114 | The Cycler strategy will be directly used in the GA by using the current solution, of length 200, as the parameter to play a specific absolute sequence during a generations scoring round. 115 | This cycler class will contain our solution sequence and the sequence can be extracted for mutation \& crossover as needed. 116 | 117 | \section{Genetic Algorithms} 118 | This work will focus on GAs, which are a specific form of ML. 119 | Advanced techniques of ML can be combined and used together\footnote{An example includes techniques for teaching and developing static algorithms. Building a video game AI where the core goal of the AI is fine tuned using a GA in an development environment, then only the trained algorithm and not the training is implemented into the game~\cite{bakkes2009rapid}.}. 120 | The field of mathematics research is one which has plenty of examples of GAs in action; 121 | for example the same techniques as we will using is used in~\cite{chu1997genetic} to solve the Generalised Assignment Problem. 122 | In it the authors discuss how to solve an NP-complete problem by exploring solutions in the problem space using a GA. 123 | This is similar to what the result of this report will contain; 124 | we will be searching the problem space (Every possible sequence permutation of $C$ and $D$ with length 200) to create solutions to the problem described in Chapter~\ref{ch:taskBackground}. 125 | 126 | GAs are not limited to mathematics, in~\cite{hauptpractical} there are discussions around the applications of GAs in art and music. 127 | The goals of a GA have to be defined beforehand and can be as abstract as we like, both art and music are abstract and would require careful consideration for what the algorithms fitness function would do. 128 | Because of their flexibility GAs are used in a wide range of application in many areas including antenna design~\cite{lohn2004evolutionary} and creating cost effective networks~\cite{savic1997genetic}. 129 | 130 | \section{Conclusion} 131 | In this chapter we have looked at previous work in areas of the IPD and GAs. 132 | Importantly we have looked at structures of strategies and some various examples that would be good to peruse when looking into constructing an effective GA. 133 | We have also outlined how strategies can be linked together through certain properties, such as their structure. 134 | As we continue looking at solutions for certain strategies we will be constructing another grouping of strategies in their solution sequences. 135 | This may lead to finding similarities to opponents who seem to have vastly different in structure. 136 | -------------------------------------------------------------------------------- /LaTeX/tex/chapters/04-code.tex: -------------------------------------------------------------------------------- 1 | \chapter{Developing The Codebase}\label{ch:developingthecodebase} 2 | This chapter will discuss the development practices encountered during the project and a walkthrough of setting up a research environment. 3 | Details on the set up and reasoning behind using specific development environments will be given along with a look at the open source contributions made. 4 | A brief look at version control will also show how modern code is developed to be sustainable and reproducible. 5 | 6 | \section{Codebase Contributions} 7 | Throughout the project I had split time between writing my own code and expanding the Axelrod Dojo codebase. 8 | In modern software development there is a commitment in the developer community to track and reuse as much code as possible, typically using a version control system such as Git or Subversion. 9 | The majority of open source community development is conducted on GitHub\cite{GitHub} using Git. 10 | The Axelrod libraries, along with most other libraries I used, are primarily hosted here. 11 | 12 | When writing code in a professional environment there is a predetermined scope that all parties agree upon before the work commences. 13 | However research development is more organic; 14 | final products that are created when conducting research for papers are highly personalised and typically cannot be used in other areas. 15 | This leads to more flexible products that operate more as platforms or tools for further research. 16 | The Axelrod, Axelrod Dojo and other libraries mentioned in table~\ref{table:functionalLibrares} exist as platforms because of this and the open source drive for reusable code. 17 | My final code will only be used for researching the IPD in my projects specific direction, as I worked on the project I had to extend the platforms in order to handle what my project needed them to do, subsection~\ref{ssec:versioncontrol} looks into how this was completed in my project. 18 | 19 | \subsection{Version Control}\label{ssec:versioncontrol} 20 | During the project I created a `fork' of the Axelrod Dojo in order to add content to the open source community. 21 | This resulted in using Git to create a new `branch' on which to write my code before asking the owners of the repository to pull my work back into the core product using a `pull request'(PR). 22 | The PR opened for my code\footnote{https://github.com/Axelrod-Python/axelrod-dojo/pull/45} resulted in changes to 457 lines of code and 14 files, adding classes and fixing bug that had been previously flagged. 23 | Figure~\ref{fig:PR-open} shows the initial scope of the PR and Figure~\ref{fig:commit-log} shows a section of the commits made before merging the branch. 24 | 25 | \begin{figure}[ht] 26 | \centering 27 | \begin{minipage}{0.48\textwidth} 28 | \centering 29 | \includegraphics[width=1.0\textwidth, center]{./img/vcs/PR-Open.png} 30 | \caption{Description and commits for PR on Github}\label{fig:PR-open} 31 | \end{minipage}\hfill 32 | \begin{minipage}{0.48\textwidth} 33 | \centering 34 | \includegraphics[width=1.0\textwidth, center,keepaspectratio]{./img/vcs/commit-log.png} 35 | \caption{Tail of code commit log as shown on Github}\label{fig:commit-log} 36 | \end{minipage} 37 | \end{figure} 38 | 39 | After a request is opened there is a period of reviews by the owners, this is to ensure code quality and scope coverage. 40 | As my work progressed I continued to add to the PR which lead to more and more requests being added for features that fell outside the scope of the PR. 41 | Eventually we decided to create another fork of the PR that contained specialist code for my project that would not benefit the codebase as a whole. 42 | Figures~\ref{fig:PR-discussion} \&~\ref{fig:PR-scope-discussion} show examples of requests and discussions around features and code quality. 43 | Because of this review process the overall quality of the codebase can be kept high and the owners can decide on how their platforms are developed. 44 | 45 | \begin{figure}[ht] 46 | \centering 47 | \begin{minipage}{0.48\textwidth} 48 | \centering 49 | \includegraphics[width=1.0\textwidth, center,keepaspectratio]{./img/vcs/feature-discussion.png} 50 | \caption{Feature discussions in PR on GitHub}\label{fig:PR-discussion} 51 | \end{minipage}\hfill 52 | \begin{minipage}{0.48\textwidth} 53 | \centering 54 | \includegraphics[width=1.0\textwidth, center,keepaspectratio]{./img/vcs/scope-discussion.png} 55 | \caption{Scope Discussion in PR on GitHub}\label{fig:PR-scope-discussion} 56 | \end{minipage} 57 | \end{figure} 58 | 59 | \subsection{Testing}\label{ssec:testing} 60 | Code testing and version control go hand in hand. 61 | When new releases of code are made it is important to ensure that the new changes dont break previous functionality. 62 | This is kept in check by the presence of continues integrations (CI) and test environments. 63 | As of this project the CI for the Axelrod Dojo keeps the libraries tests running on a linux environment and the output fed to the Github page. 64 | During my development I had to implement tests for any functionality I wrote to ensure the library still worked as intended. 65 | Following the mixed practice of test driven design (TDD) and behaviour driven design (BDD) the code written to extend the Axelrod Dojo had tests to cover examples of what happens in a production environment~\cite{hong2015top, prlic2012ten, sandve2013ten}. 66 | Snippet~\ref{code:test-example} shows an example of a test. 67 | 68 | \begin{figure}[ht] 69 | \inputminted{python}{code_snippets/dev-examples/example-test.py} 70 | \caption{An Example of a test in the Axelrod Dojo}\label{code:test-example} 71 | \end{figure} 72 | 73 | \section{Libraries And External Modules} 74 | \paragraph{Main Research Libraries} 75 | \begin{itemize} 76 | \item \textbf{Axelrod} --- Used for the core of the prisoners dilemma and iterated prisoners dilemma functionality code~\cite{axelrodproject}. 77 | \item \textbf{Axelrod Dojo} --- Applied machine learning techniques that revolve around generating solutions to questions relating to the Axelrod library~\cite{dojoV008}. 78 | \end{itemize} 79 | 80 | \paragraph{Functional libraries} 81 | Table~\ref{table:functionalLibrares} shows the external functional libraries used, while Table~\ref{table:builtinmodules} shows the internal python built in modules that where leveraged during development. 82 | These are libraries which are not involved in the core functionality of the IPD\@. 83 | \begin{table*}[ht] 84 | \centering 85 | \begin{tabular}{ccc} 86 | \toprule 87 | Library & Reason & Reference\\ 88 | \midrule 89 | \textbf{matplotlib pyplot} & For plotting graphs and images with data & \cite{hunter2007matplotlib}\\ 90 | \textbf{pandas} & For data manipulation. & \cite{PandasGithub,Mckinney2010pandas}\\ 91 | \textbf{numpy} & For reducing complexity of numerical calculations.& \cite{oliphant2006numpy}\\ 92 | \textbf{SciKit Learn} & For existing ML tools. & \cite{pedregosa2011scikit}\\ 93 | \bottomrule 94 | \end{tabular} 95 | \caption{Functional Python libraries for analysis}\label{table:functionalLibrares} 96 | \end{table*} 97 | \begin{table*}[ht] 98 | \centering 99 | \begin{tabular}{cc} 100 | \toprule 101 | Module & Reason\\ 102 | \midrule 103 | \textbf{os} & For operating system functionality.\\ 104 | \textbf{time} & For time calculations.\\ 105 | \textbf{itertools} & For easier iterations over data structures.\\ 106 | \bottomrule 107 | \end{tabular} 108 | \caption{Internal built in python modules used}\label{table:builtinmodules} 109 | \end{table*} 110 | 111 | \section{Reproducing Analysis}\label{subsec:settingUpAResearchEnvironment} 112 | A mix of Jupyter Notebooks and integrated development environments\footnote{Pycharm Professional Edition and Microsoft VS Code.} were used to write and execute code. 113 | The main analysis was run using a factory class pattern, called \mintinline{python}{AnalysisRun} in the \mintinline{python}{full_analysis} module, show in Appendix Figure~\ref{apcode:AnalysisRun.py}. 114 | This class was used to wrap a query to the Axelrod Dojo functionality and subsequently to the Axelrod Library in such a way that was easy to control batch executions. 115 | The analysis itself was done using native multi-threading on a Linux OS to improve individual opponent analysis run times and the overall scalability of the project. 116 | 117 | The remainder of this tutorial will also assume you're working with a local development station; 118 | analysis on remote cloud instances of Jupyter Notebooks is possible but the set up is different. 119 | The steps for working on a cloud instance are described after the local set up. 120 | 121 | \paragraph{Installing Basic Libraries} 122 | Your first step should be to download and install the Anaconda distribution for your OS here: https://www.anaconda.com/download/. 123 | This will allow you to use the integrated c++ libraries python has to offer without needing to mess about too much. 124 | Anaconda also has them majority of Functional Libraries above and Jupyter Notebooks pre installed to make setting up much easier. 125 | From here, follow the instructions the install wizard has to add any environment variables to allow CMD/Bash access to binaries. 126 | 127 | Installing the Axelrod and Axelrod Dojo libraries uses the pip tool that already comes with Anaconda and should be ready to execute after the last step. 128 | Running `pip install axelrod' then `pip install axelrod-dojo' will install these. 129 | 130 | Once this is installed the \mintinline{python}{full\_analysis.py} file has to be downloaded from GitHub\footnote{https://github.com/GitToby/FinalYearProject}, it can be found in the code directory. 131 | This can just be copied and pasted if needed all were interested in is the class to generate a sequence for an opponent. 132 | 133 | \paragraph{Running a Test} 134 | Figure~\ref{code:analysisExample} is some sample code that will run an analysis with the following settings: 135 | \begin{itemize} 136 | \item Override the default mutation frequency of \(0.1\) to \(0.33\). 137 | \item Set the prefix for all the files to be `example-'. 138 | \item Analysing 3 opponents. (Random will automatically have multiple instances for different seeds.) 139 | \end{itemize} 140 | \begin{figure}[ht] 141 | \centering 142 | \inputminted{python}{code_snippets/analysisExample.py} 143 | \caption{Code to create a sequence result to optimise best score for 3 opponents}\label{code:analysisExample} 144 | \end{figure} 145 | 146 | After it has run the generated data output should be stored in the `./output' directory. 147 | If the code fails to run there may be issues with this directory being created. 148 | There should be multiple output files, each with one opponents evolution stages through the generations. 149 | 150 | \paragraph{Cloud Notebook Setup} 151 | If you want to use a Cloud service, such as Azure Notebooks or AWS Sagemaker, the set up is similar to the above just executed differently. 152 | Installing Anaconda is not needed, the environment has the required installs. 153 | Using pip to install the Axelrod libraries and download the full analysis script can be done in an integrated web terminal or directly in a notebook. 154 | Figure~\ref{code:jupyterExample} shows and example in azure, copy these in your top few cells of your jupyter notebook and it will work as required. 155 | 156 | \begin{figure}[ht] 157 | \inputminted{python}{code_snippets/dev-examples/jupyterCells.py} 158 | \caption{Cells for creating the jupyter instance of a research environment}\label{code:jupyterExample} 159 | \end{figure} 160 | 161 | \section{Conclusion} 162 | In this chapter we covered the contributions needed to the Axelrod Dojo to carry on the work described in this report. Moreover, we covered the procedure of implementing and contributing to an open source package. 163 | Of the learning that was complexity during the project, learning a new version control system was the hardest along with handling the organic growth of the project. 164 | While writing code there where sections that needed changing or removing, however there was no scope or predetermined set of outcomes causing my own analysis notebooks to swell and become unwieldy at times. 165 | These problems were solved by selectively isolating work and leveraging 3rd party libraries more. 166 | 167 | The final codebase contributions were a success and version 0.0.8 of the Axelrod Dojo has been released to the python installer package, `pip'~\cite{dojoV008}. 168 | Following the steps in Section~\ref{subsec:settingUpAResearchEnvironment} will allow a working environment to be set up on any platform, letting this experiment be repeated. 169 | Providing a reproducible experiment allows the open source and research communities independently verify any findings in this work to ensure a robust foundation for future work. 170 | All finalised notebooks and final code are published online on GitHub, online resources can be found in Appendix~\ref{apndx:resources}. -------------------------------------------------------------------------------- /LaTeX/tex/chapters/01-introduction.tex: -------------------------------------------------------------------------------- 1 | % !TEX root = ../main.tex 2 | 3 | \chapter{Introduction}\label{ch:intro} 4 | In this report we will be looking for best responses of strategies in the Iterated Prisoners Dilemma (IPD); 5 | we will try to identify the best overall score, and its corresponding set of moves, for a player on the other side of the game to the strategy. 6 | We will look into explicit sequences which should be played for $1$ vs $1$ games of length 200 to gain the highest score per turn upon the games conclusion. 7 | 8 | This task will be completed using reinforcement techniques, specifically genetic algorithms, to continuously improve our sequences during training. Once we have best responses, these sequences will then be compared and contrasted to understand how opponents could be grouped together even though they have no structure in common. 9 | 10 | This work assumes a basic knowledge of game theory and how to model a normal form game. 11 | Knowledge of the Prisoners Dilemma is helpful but not required, the IPD game itself is described in Section~\ref{sec:iteratedPrisonersDilemma} and literature is discussed in Chapter~\ref{ch:literature}. 12 | There is also the assumption the reader is familiar with basic algorithms and programming. 13 | 14 | \section{The Prisoners Dilemma and Its Iterated Form}\label{sec:iteratedPrisonersDilemma} 15 | % \begin{center} 16 | % \itshape~Two members of a criminal gang are arrested and imprisoned. 17 | % Each prisoner is in solitary confinement with no means of communicating with the other. 18 | % The prosecutors lack sufficient evidence to convict the pair on the principal charge. 19 | % They hope to get both sentenced to a year in prison on a lesser charge. Simultaneously, the prosecutors offer each prisoner a bargain. 20 | % Each prisoner is given the opportunity either to: betray the other by testifying that the other committed the crime, or to cooperate with the other by remaining silent. 21 | % \end{center} 22 | 23 | The Prisoners Dilemma (PD) is a normal form game in the space $\mathbb{R}^{{2\times 2}^2}$ with utility matrices and mixed strategies for the row and column player as follows: 24 | $$ 25 | A=\begin{pmatrix}R & S\\ T & P\end{pmatrix}\quad 26 | B=\begin{pmatrix}R & T\\ S & P\end{pmatrix} 27 | $$ 28 | $$ 29 | \sigma_r=\begin{pmatrix}C & D\\ \end{pmatrix}\quad 30 | \sigma_c=\begin{pmatrix}C & D\\ \end{pmatrix} 31 | $$ 32 | with the following utility constraints: 33 | $$T>R>P>S, \qquad 2R>T+S$$ 34 | These constraints mean that the defection action, $D$, dominates the cooperation action, $C$, for both player and that the largest payoff comes when both players cooperate. This payoff model has the following interpretation: 35 | \begin{itemize} 36 | \item $T$ --- Temptation, the utility for successfully tricking your opponent to cooperate while you defect. 37 | \item $R$ --- Reward, by both coopering with your respective opponents you both receive the reward. 38 | \item $P$ --- Punishment, if both you and your opponent defect you will both receive the punishment 39 | \item $S$ --- Sucker, the utility for being tricked by your opponent to cooperate while they defect; they get temptation and you get sucker. 40 | \end{itemize} 41 | Because of the way the game is put together we arrive at the dilemma. 42 | Do you cooperate and risk being taken advantage of by your opponent, first row or column. 43 | Or do you defect and hope your opponent cooperates to give you a bonus, second row or column. 44 | In this work we will be using the T, R, S \& P utilities originally introduced in axelrods 1980s work~\cite{axelrod1980effective}. 45 | $$ 46 | A=\begin{pmatrix}3 & 0 \\ 5 & 1\end{pmatrix}\quad 47 | B=\begin{pmatrix}3 & 5 \\ 0 & 1\end{pmatrix} 48 | $$ 49 | From a game theoretic point of view it can be seen that there exists a strongly dominant strategy for both the row and column player: defection. 50 | This however does not reflect on the iterated version of the game where repetition can allow for higher overall scores and players can take advantage of each other. 51 | 52 | We will be looking into the iterated form of the game which, simply put, is a series of single games played back to back. 53 | Players do not know what their opponent will play on any given turn, they only know what has been played on previous turns. 54 | Players can algorithmically, stochastically or using a combination of the two decide on their next move. 55 | The number of turns is often provided beforehand, but does not need to be~\cite{axelrod1980more}, and the overall score of the game is normalised to by the number of turns to allow for comparisons between games of different length. 56 | 57 | \section{Machine Learning \& Computer Intelligence}\label{sec:machineLearningAndcomputerIntelligence} 58 | Machine learning (ML) was most famously introduced by questions posed by Alan Turing in 1950~\cite{turing1950computing} asking `can machines think?', a question that has been refined and analysed to this day. 59 | The field of computer intelligence is rich in its complexities and has recently been making breakthroughs~\cite{knight2017alphaZeroMIT} on topics which would traditionally be considered `thinking'. 60 | Along with this, recently there has also been record levels of funding~\cite{chui2017artificial} put in to companies which operate in this field, producing results in areas that would usually seem `solved'. 61 | 62 | Computer intelligence breaks down into 2 general topics, supervised and unsupervised learning. 63 | Supervised learning is done with help from an external source; inferring a function from labelled examples that will be used to map unseen data to their potential labels. 64 | Unsupervised learning is the process of creating new data about a dataset which will be used to expand the understanding of existing data; for example finding a hidden structure or searching a solution space. 65 | Reinforcement learning is a technique that can be applied to both supervised and unsupervised learning. 66 | What it means is to repeat an algorithm using results from a previous run to improve parameters of its next run; genetic algorithms are a prime example of this. 67 | 68 | \subsection{Genetic Algorithms}\label{subsec:geneticAlgorithms} 69 | This report will cover a class of ML called genetic algorithms (GAs). 70 | GAs fall under a branch of ML called evolutionary algorithms. 71 | More generally, genetic algorithms are put into a class of unsupervised reinforcement learning algorithms. 72 | These represent techniques of using genetic algorithms for generating solutions to problems that typically revolve around heuristically improving members of a population who represent these solutions,\cite{horn1994niched,rahmat1999electromagnetic}. 73 | The concept of a genetic algorithm, and more generally an evolutionary algorithm, comes from nature; 74 | like nature we create a survival of the fittest selection~\cite{darwin2009origin} competition to evaluate a population then kill off the weakest members. 75 | After this cull we create offspring from the most successful population or introduce new members from a predefined source. 76 | This process is then repeated until we stop it, or forever in the case of nature. 77 | 78 | We say a genetic algorithm is structured in the following way. 79 | Given a population, \(P\), each with unique genes (the terms `genes' and `member properties' are interchangeable), and a number of generations, \(G\in \mathbb{N}\), the algorithm will create \(G\) cycles of scoring and potentially removing each of the members of the population, \(p_i \in P\). 80 | It does this by using a mapping from a member of the population to an ordered set, for example \(f(p_i)\mapsto \mathbb{R}\). 81 | This function, \(f\), is defined beforehand in a way which describes the goal of our investigation. 82 | Defining a cut-off or bottleneck \(b<|P|\), such that on conclusion of completing any cycle, the top \(b\) ranking members (or proportion of members) by score can be kept and the rest discarded. 83 | By doing this we are saving the more successful candidates allowing us to rebuild the population using a series of crossovers and mutations (and possibly introducing new members into the population) with the genes which were successful. 84 | 85 | \begin{itemize} 86 | \item Crossovers take in 2 members of the population and return a new member based on some parameters of the 2 `parents'. 87 | For example, our crossover takes the first half of a sequence from one and the second half from the other, merging them to form the third. 88 | \item Mutations allow a (possibly targeted\footnote{For example using intuition and targeting specific genes, or allowing another algorithm to improve the targeting of this meta function.}) change in a single member of the population. 89 | A mutation has 2 parameters, a potency \(M_p\in \mathbb{R}>0\) and a frequency \(M_f\in [0,1]\). 90 | \(M_p\) describes how strong the mutation is, the higher it is the larger change to the member occurs. 91 | \(M_f\) explains the percentage of how many members of the population are mutated. 92 | \end{itemize} 93 | 94 | Figure~\ref{fig:genericGACycle} shows a flow diagram of a generic genetic algorithm cycle. 95 | The specific algorithm we will be using is described in Subsection~\ref{sec:buildingTheAlgorithm}. 96 | 97 | \begin{figure}[ht] 98 | \centering 99 | \includegraphics[width=1.0\textwidth, center]{./img/flows/ga_cycle.pdf} 100 | \caption{Generic genetic algorithm cycle diagram}\label{fig:genericGACycle} 101 | \end{figure} 102 | 103 | Genetic algorithms are designed in way to avoid local maxima/minima when searching a solution space. 104 | The property of mutating a member of its population allows alterations to the members features; 105 | this in turn will lead to the change in that members fitness. 106 | If a member is `stuck' in a local maxima/minima with its fitness score, a mutation of a certain feature could potentially move their position in the solution space to better local maxima/minima. 107 | Along with mutation, the crossover property explicitly combines 2 members of the population to create a new member. 108 | Crossover adds more variance in the population by taking properties of both parent members and introducing the evaluation of their combination to the solution space. 109 | By doing this the algorithm can search multiple local maxima/minima at once with its population and start to identify the global maxima/minima as members become more optimized. 110 | 111 | \section{Brief Task Overview}\label{sec:briefOverview} 112 | This work will look at discovering which sequences provide the highest score in a game for each given opponent in the Iterated Prisoners Dilemma. 113 | For every game of length $n$ there are $2^n$ possible combinations of moves we can make against an opponent, we are trying to find the sequence which provides us with the highest score overall. 114 | 115 | Analysis will be focused on looking into just the single opponent use case, but the idea of designing a sequence for a tournament for a given number of opponents is a potential follow on to this work. 116 | Our task is as follows: 117 | 118 | Problem: 119 | \begin{center} 120 | \itshape~When playing a given Iterated Prisoners Dilemma strategy as an opponent, what is the best ordered sequence of moves to play in order for us to obtain the highest possible average score per move across the game? Are there any patterns in these solution sequences? 121 | \end{center} 122 | 123 | For example an opponent known as Tit For Tat, which cooperates on its first move and the copies your last move on subsequent moves, will have a solution sequence of moves that are all cooperation apart from the last. 124 | This is an easy example and a simple strategy to calculate a solution for due to its explicit rules. 125 | The ultimate goal of this investigation is to look into every strategy as defined in the open source Axelrod Library\cite{axelrodproject}, where over 200 strategies are implemented. 126 | These opponents are listed in Appendix Section~\ref{apndx:opponents} and definitions can be found in the Axelrod Library codebase. 127 | 128 | This problem can be reduced to searching the solution space of every single possible combination of $C$ and $D$ moves. 129 | With a game length of 200 however, this is an infeasibly large number of checks and so we will use the genetic algorithm to selectively improve potential sequences. 130 | Some examples of 5 move games against Tit For Tat are shown below. 131 | The optimal solution for Tit For Tat in this scenario is game 5 whereas all the others are sub optimal. 132 | This is the process we will be doing for every opponent in the context of a genetic algorithm. 133 | 134 | $$ 135 | \begin{matrix} 136 | \textbf{Game 1}& 1 & 2 & 3 & 4 & 5 & &\text{final score} & \text{per turn}\\ 137 | \text{Tit For Tat:} & C & D & D & D & D && 4 & 0.8\\ 138 | \text{Solution:} & D & D & D & D & D && 9 & 1.8 139 | \end{matrix} 140 | $$ 141 | $$ 142 | \begin{matrix} 143 | \textbf{Game 2}& 1 & 2 & 3 & 4 & 5 & &\text{final score} & \text{per turn}\\ 144 | \text{Tit For Tat:} & C & D & C & D & D && 7 & 1.4\\ 145 | \text{Solution:} & D & C & D & D & D && 12 & 2.4 146 | \end{matrix} 147 | $$ 148 | $$ 149 | \begin{matrix} 150 | \textbf{Game 3}& 1 & 2 & 3 & 4 & 5 & &\text{final score} & \text{per turn}\\ 151 | \text{Tit For Tat:} & C & D & C & D & C && 10 & 2.0\\ 152 | \text{Solution:} & D & C & D & C & D && 15 & 3.0 153 | \end{matrix} 154 | $$ 155 | $$ 156 | \begin{matrix} 157 | \textbf{Game 4}& 1 & 2 & 3 & 4 & 5 & &\text{final score} & \text{per turn}\\ 158 | \text{Tit For Tat:} & C & D & C & C & C && 11 & 2.2\\ 159 | \text{Solution:} & D & C & C & C & D && 16 & 3.2 160 | \end{matrix} 161 | $$ 162 | $$ 163 | \begin{matrix} 164 | \textbf{Game 5}& 1 & 2 & 3 & 4 & 5 & &\text{final score} & \text{per turn}\\ 165 | \text{Tit For Tat:} & C & C & C & C & C && 12 & 2.4\\ 166 | \text{Solution:} & C & C & C & C & D && 17 & 3.4 167 | \end{matrix} 168 | $$ 169 | $$ 170 | \begin{matrix} 171 | \textbf{Game 6}& 1 & 2 & 3 & 4 & 5 & &\text{final score} & \text{per turn}\\ 172 | \text{Tit For Tat:} & C & C & C & C & C && 15 & 3.0\\ 173 | \text{Solution:} & C & C & C & C & C && 15 & 3.0 174 | \end{matrix} 175 | $$ 176 | 177 | \section{Conclusion \& Structure of This Report} 178 | This chapter contained introduction to content that will be needed and expanded on moving forward. 179 | All the topics mentioned in this chapter are core themes to what the rest of this report focuses on. 180 | 181 | This report will contain the following chapters: 182 | \begin{itemize} 183 | \item Chapter~\ref{ch:taskBackground} describes methodology and concepts that will be needed to understand results in later chapters. 184 | \item Chapter~\ref{ch:literature} looks into previous work and literature behind the concepts we will be working with. 185 | \item Chapter~\ref{ch:developingthecodebase} looks into the programming and development that was undertaken to complete the work. 186 | \item Chapter~\ref{ch:implementation} looks at how the application of our genetic algorithm was improved before running the final analysis. 187 | \item Chapter~\ref{ch:results} is the chapter communicating the results of the final analysis. 188 | \item Chapter~\ref{ch:conclusions} describes the takeaways and possible applications of the work. 189 | \end{itemize} -------------------------------------------------------------------------------- /code/initialExploring.py: -------------------------------------------------------------------------------- 1 | import os 2 | import time 3 | 4 | import axelrod as axl 5 | import axelrod_dojo as axl_dojo 6 | import matplotlib.pyplot as plt 7 | import numpy as np 8 | import pandas as pd 9 | 10 | C, D = axl.Action 11 | 12 | interesting_opponents = [axl.TitForTat(), axl.Alternator(), axl.Grudger(), axl.Random(), axl.EvolvedFSM16(), 13 | axl.CollectiveStrategy()] 14 | col_names = ["generation", "mean_score", "standard_deviation", "best_score", "sequence"] 15 | 16 | 17 | def runGeneticAlgo(opponent, population_size=150, number_of_game_turns=200, cycle_length=200, generations=250, 18 | mutation_probability=0.1, mutation_potency=1, reset_file=True): 19 | cycler_class = axl_dojo.CyclerParams 20 | cycler_objective = axl_dojo.prepare_objective(name="score", turns=number_of_game_turns, repetitions=1) 21 | cycler_kwargs = { 22 | "sequence_length": cycle_length, 23 | "mutation_probability": mutation_probability, 24 | "mutation_potency": mutation_potency 25 | } 26 | 27 | output_file_name = "data/" + str(opponent).replace(" ", "_") + ".csv" 28 | try: 29 | if reset_file and os.path.isfile(output_file_name): 30 | os.remove(output_file_name) 31 | finally: 32 | print(str(opponent), 33 | "|| pop size:", population_size, 34 | "\tturns:", number_of_game_turns, 35 | "\tcycle len:", cycle_length, 36 | "\tgens:", generations, 37 | "\tmut. rate:", mutation_probability, 38 | "\t, potency:", mutation_potency) 39 | 40 | axl.seed(1) 41 | 42 | population = axl_dojo.Population(params_class=cycler_class, 43 | params_kwargs=cycler_kwargs, 44 | size=population_size, 45 | objective=cycler_objective, 46 | output_filename=output_file_name, 47 | opponents=[opponent], 48 | print_output=False) 49 | population.run(generations) 50 | print("\tAnalysis Complete:", output_file_name) 51 | # Store the file name and opponent name as a tuple 52 | return output_file_name, str(opponent) 53 | 54 | 55 | # ------------------------------POPULATION SIZE 56 | populations = [25, 50, 100, 150, 200, 250, 500] 57 | 58 | 59 | def populationChecker(opponent): 60 | # make a nice file name 61 | file_name = "data/" + str(opponent).replace(" ", "_").replace(":", "_").lower() + "_pop.csv" 62 | 63 | # if the file exists dont run_one, it takes forever, make sure it exists 64 | if not os.path.isfile(file_name): 65 | df_main = pd.DataFrame(data=None, columns=col_names) 66 | 67 | for pop_size in populations: 68 | start_time = time.clock() 69 | pop_run = runGeneticAlgo(opponent, 70 | population_size=pop_size, 71 | number_of_game_turns=200, 72 | cycle_length=200, 73 | generations=150, 74 | mutation_probability=0.1, 75 | reset_file=True) 76 | end_time = time.clock() 77 | tmp_df = pd.read_csv(pop_run[0], names=col_names) 78 | tmp_df["population"] = pop_size 79 | tmp_df["time_taken"] = end_time - start_time 80 | df_main = df_main.append(tmp_df, ignore_index=True) 81 | 82 | df_main.to_csv(file_name) 83 | print("List Complete:", file_name) 84 | return df_main 85 | else: 86 | print("file already exists, no calcs to do.") 87 | file_df = pd.read_csv(file_name) 88 | # remove first column 89 | file_df = file_df[list(file_df)[1:]] 90 | return file_df 91 | 92 | 93 | df_TitForTat_pop = populationChecker(axl.TitForTat()) 94 | df_Alternator_pop = populationChecker(axl.Alternator()) 95 | df_Grudger_pop = populationChecker(axl.Grudger()) 96 | df_Random_pop = populationChecker(axl.Random()) 97 | df_Evolved_pop = populationChecker(axl.EvolvedFSM16()) 98 | df_Collective_pop = populationChecker(axl.CollectiveStrategy()) 99 | 100 | mutation_pop_dict = {"Tit For Tat": df_TitForTat_pop, 101 | "Alternator": df_Alternator_pop, 102 | "Grudger": df_Grudger_pop, 103 | "Random": df_Random_pop, 104 | "Evolved FSM16": df_Evolved_pop, 105 | "Collective Stratergy": df_Collective_pop} 106 | 107 | # PLOT 108 | working_dict = mutation_pop_dict 109 | 110 | f, axs = plt.subplots(2, 3, sharex=False, figsize=(20, 25)) 111 | f.suptitle("Improvments on best score over 150 generations with different initial populations") 112 | 113 | a = 0 114 | for opponent in working_dict: 115 | axs[a % 2, a % 3].set_title(opponent) 116 | a += 1 117 | 118 | i = 0 119 | for opponet in working_dict: 120 | for lab, df in working_dict.get(opponet).groupby("population"): 121 | axs[i % 2, i % 3].plot(df["generation"], df["best_score"], label=lab) 122 | i += 1 123 | 124 | for j in range(len(working_dict)): 125 | axs[j % 2, j % 3].set(xlabel='Generations', ylabel='Best Score') 126 | axs[j % 2, j % 3].legend() 127 | 128 | f.savefig('plots/initial_pop/ini_pop score all opponents.png') 129 | 130 | # ------------------------------ GENERATION LENGTH 131 | generation_list = [50, 150, 250, 350, 450, 500] 132 | 133 | 134 | def generationSizeChecker(opponent): 135 | file_name = "data/" + str(opponent).replace(" ", "_").replace(":", "_").lower() + "_generation.csv" 136 | 137 | if not os.path.isfile(file_name): 138 | df_main = pd.DataFrame(data=None, columns=col_names) 139 | 140 | for gens in generation_list: 141 | start_time = time.clock() 142 | pop_run = runGeneticAlgo(opponent, 143 | population_size=150, 144 | number_of_game_turns=200, 145 | cycle_length=200, 146 | generations=gens, 147 | mutation_probability=0.1, 148 | reset_file=True) 149 | end_time = time.clock() 150 | tmp_df = pd.read_csv(pop_run[0], names=col_names) 151 | tmp_df["generations"] = gens 152 | tmp_df["time_taken"] = end_time - start_time 153 | tmp_df["opponent"] = str(opponent) 154 | tmp_df["best_score_diff"] = np.append([0], np.diff(tmp_df["best_score"])) 155 | df_main = df_main.append(tmp_df, ignore_index=True) 156 | 157 | df_main.to_csv(file_name) 158 | print("List Complete:", file_name) 159 | return df_main 160 | else: 161 | print("file ", file_name, " already exists, no calcs to do.") 162 | file_df = pd.read_csv(file_name) 163 | # remove first column 164 | file_df = file_df[list(file_df)[1:]] 165 | return file_df 166 | 167 | 168 | df_TitForTat_gen = generationSizeChecker(axl.TitForTat()) 169 | df_Alternator_gen = generationSizeChecker(axl.Alternator()) 170 | df_Grudger_gen = generationSizeChecker(axl.Grudger()) 171 | df_Random_gen = generationSizeChecker(axl.Random()) 172 | df_Evolved_gen = generationSizeChecker(axl.EvolvedFSM16()) 173 | df_Collective_gen = generationSizeChecker(axl.CollectiveStrategy()) 174 | 175 | mutation_gen_dict = {"Tit For Tat": df_TitForTat_gen, 176 | "Alternator": df_Alternator_gen, 177 | "Grudger": df_Grudger_gen, 178 | "Random": df_Random_gen, 179 | "Evolved FSM16": df_Evolved_gen, 180 | "Collective Stratergy": df_Collective_gen} 181 | 182 | # PLOT 183 | working_dict = mutation_gen_dict 184 | 185 | f, axs = plt.subplots(2, 3, sharex=False, sharey=True, figsize=(20, 20)) 186 | f.suptitle("Average increase of score vs # of generations") 187 | 188 | a = 0 189 | for opponent in working_dict: 190 | axs[a % 2, a % 3].set_title(opponent) 191 | a += 1 192 | 193 | i = 0 194 | for opponet in working_dict: 195 | for lab, df in working_dict.get(opponet).groupby("generations"): 196 | axs[i % 2, i % 3].scatter(lab, df["best_score_diff"].mean(), label="{:3.0f} generations".format(lab)) 197 | i += 1 198 | 199 | for j in range(len(working_dict)): 200 | axs[j % 2, j % 3].set(xlabel='Total generations', ylabel='Best Score') 201 | axs[j % 2, j % 3].legend() 202 | 203 | f.savefig('plots/gen_len/gen_len avg diff all opponents.png') 204 | 205 | # PLOT2 206 | working_dict = mutation_gen_dict 207 | 208 | f, axs = plt.subplots(2, 3, sharex=False, figsize=(20, 20)) 209 | f.suptitle("Max best score vs # of generations coloured by time") 210 | 211 | a = 0 212 | for opponent in working_dict: 213 | axs[a % 2, a % 3].set_title(opponent) 214 | a += 1 215 | 216 | i = 0 217 | for opponet in working_dict: 218 | for lab, df in working_dict.get(opponet).groupby("generations"): 219 | axs[i % 2, i % 3].scatter(lab, df["best_score"].max(), label="{:3.0f} generations".format(lab)) 220 | i += 1 221 | 222 | for j in range(len(working_dict)): 223 | axs[j % 2, j % 3].set(xlabel='Total generations', ylabel='Best Score') 224 | axs[j % 2, j % 3].legend() 225 | 226 | f.savefig('plots/gen_len/gen_len best score all opponents.png') 227 | 228 | # ------------------------------ MUTATION POTENCY 229 | mutation_potency_list = [1, 2, 3, 5, 10, 15, 20] 230 | 231 | 232 | def mutationPotencyChecker(opponent): 233 | file_name = "data/" + str(opponent).replace(" ", "_").replace(":", "_").lower() + "_mutation_potency.csv" 234 | 235 | if not os.path.isfile(file_name): 236 | df_main = pd.DataFrame(data=None, columns=col_names) 237 | 238 | for potency in mutation_potency_list: 239 | start_time = time.clock() 240 | pot_run = runGeneticAlgo(opponent, 241 | population_size=150, 242 | number_of_game_turns=200, 243 | cycle_length=200, 244 | generations=250, 245 | mutation_probability=0.1, 246 | mutation_potency=potency, 247 | reset_file=True) 248 | end_time = time.clock() 249 | tmp_df = pd.read_csv(pot_run[0], names=col_names) 250 | tmp_df["mutation_potency"] = potency 251 | tmp_df["time_taken"] = end_time - start_time 252 | tmp_df["opponent"] = str(opponent) 253 | tmp_df["best_score_diff"] = np.append([0], np.diff(tmp_df["best_score"])) 254 | df_main = df_main.append(tmp_df, ignore_index=True) 255 | 256 | df_main.to_csv(file_name) 257 | print("List Complete:", file_name) 258 | return df_main 259 | else: 260 | print("file ", file_name, " already exists, no calcs to do.") 261 | file_df = pd.read_csv(file_name) 262 | # remove first column 263 | file_df = file_df[list(file_df)[1:]] 264 | return file_df 265 | 266 | 267 | df_TitForTat_potency = mutationPotencyChecker(axl.TitForTat()) 268 | df_Alternator_potency = mutationPotencyChecker(axl.Alternator()) 269 | df_Grudger_potency = mutationPotencyChecker(axl.Grudger()) 270 | df_Random_potency = mutationPotencyChecker(axl.Random()) 271 | df_Evolved_potency = mutationPotencyChecker(axl.EvolvedFSM16()) 272 | df_Collective_potency = mutationPotencyChecker(axl.CollectiveStrategy()) 273 | 274 | mutation_potency_dict = {"Tit For Tat": df_TitForTat_potency, 275 | "Alternator": df_Alternator_potency, 276 | "Grudger": df_Grudger_potency, 277 | "Random": df_Random_potency, 278 | "Evolved FSM16": df_Evolved_potency, 279 | "Collective Stratergy": df_Collective_potency} 280 | 281 | # PLOT 282 | working_dict = mutation_potency_dict 283 | 284 | f, axs = plt.subplots(2, 3, sharex=False, figsize=(20, 25)) 285 | f.suptitle("") 286 | 287 | a = 0 288 | for opponent in working_dict: 289 | axs[a % 2, a % 3].set_title(opponent) 290 | a += 1 291 | 292 | i = 0 293 | for opponet in working_dict: 294 | for lab, df in working_dict.get(opponet).groupby("mutation_potency"): 295 | axs[i % 2, i % 3].plot(df["generation"], df["best_score"], label=lab) 296 | i += 1 297 | 298 | for j in range(len(working_dict)): 299 | axs[j % 2, j % 3].set(xlabel='Generations', ylabel='Best Score') 300 | axs[j % 2, j % 3].legend() 301 | 302 | f.savefig('plots/mutation_pot/mut_pot all opponents.png') 303 | 304 | # PLOT2 305 | working_dict = mutation_potency_dict 306 | 307 | f, axs = plt.subplots(2, 3, sharex=False, sharey=True, figsize=(20, 25)) 308 | f.suptitle("") 309 | 310 | a = 0 311 | for opponent in working_dict: 312 | axs[a % 2, a % 3].set_title(opponent) 313 | a += 1 314 | 315 | i = 0 316 | for opponet in working_dict: 317 | for lab, df in working_dict.get(opponet).groupby("mutation_potency"): 318 | axs[i % 2, i % 3].scatter(lab, df["best_score_diff"].mean(), label=lab) 319 | i += 1 320 | 321 | for j in range(len(working_dict)): 322 | axs[j % 2, j % 3].set(xlabel='Mutation Potency', ylabel='Best Score Diff') 323 | axs[j % 2, j % 3].axhline(0) 324 | axs[j % 2, j % 3].legend() 325 | 326 | f.savefig('plots/mutation_pot/mut_pot score diff all opponents.png') 327 | 328 | # ------------------------------ MUTATION FREQ 329 | mutation_frequency_list = [0.1, 0.15, 0.2, 0.25] 330 | 331 | 332 | def mutationFrequencyChecker(opponent): 333 | file_name = "data/" + str(opponent).replace(" ", "_").replace(":", "_").lower() + "_mutation_frequency.csv" 334 | 335 | if not os.path.isfile(file_name): 336 | df_main = pd.DataFrame(data=None, columns=col_names) 337 | 338 | for freq in mutation_frequency_list: 339 | start_time = time.clock() 340 | pot_run = runGeneticAlgo(opponent, 341 | population_size=150, 342 | number_of_game_turns=200, 343 | cycle_length=200, 344 | generations=250, 345 | mutation_probability=freq, 346 | mutation_potency=1, 347 | reset_file=True) 348 | end_time = time.clock() 349 | tmp_df = pd.read_csv(pot_run[0], names=col_names) 350 | tmp_df["mutation_frequency"] = freq 351 | tmp_df["time_taken"] = end_time - start_time 352 | tmp_df["opponent"] = str(opponent) 353 | tmp_df["best_score_diff"] = np.append([0], np.diff(tmp_df["best_score"])) 354 | df_main = df_main.append(tmp_df, ignore_index=True) 355 | 356 | df_main.to_csv(file_name) 357 | print("List Complete:", file_name) 358 | return df_main 359 | else: 360 | print("file ", file_name, " already exists, no calcs to do.") 361 | file_df = pd.read_csv(file_name) 362 | # remove first column 363 | file_df = file_df[list(file_df)[1:]] 364 | return file_df 365 | 366 | 367 | df_TitForTat_freq = mutationFrequencyChecker(axl.TitForTat()) 368 | df_Alternator_freq = mutationFrequencyChecker(axl.Alternator()) 369 | df_Grudger_freq = mutationFrequencyChecker(axl.Grudger()) 370 | df_Random_freq = mutationFrequencyChecker(axl.Random()) 371 | df_Evolved_freq = mutationFrequencyChecker(axl.EvolvedFSM16()) 372 | df_Collective_freq = mutationFrequencyChecker(axl.CollectiveStrategy()) 373 | 374 | mutation_freq_dict = {"Tit For Tat": df_TitForTat_freq, 375 | "Alternator": df_Alternator_freq, 376 | "Grudger": df_Grudger_freq, 377 | "Random": df_Random_freq, 378 | "Evolved FSM16": df_Evolved_freq, 379 | "Collective Stratergy": df_Collective_freq} 380 | 381 | # PLOT 382 | working_dict = mutation_freq_dict 383 | 384 | f, axs = plt.subplots(2, 3, sharex=False, figsize=(20, 30)) 385 | f.suptitle("") 386 | 387 | a = 0 388 | for opponent in working_dict: 389 | axs[a % 2, a % 3].set_title(opponent) 390 | a += 1 391 | 392 | i = 0 393 | for opponet in working_dict: 394 | for lab, df in working_dict.get(opponet).groupby("mutation_frequency"): 395 | axs[i % 2, i % 3].plot(df["generation"], df["best_score"], label=lab) 396 | i += 1 397 | 398 | for j in range(len(working_dict)): 399 | axs[j % 2, j % 3].set(xlabel='Generations', ylabel='Best Score') 400 | axs[j % 2, j % 3].legend() 401 | 402 | f.savefig('plots/mutation_freq/mut_freq all opponents.png') 403 | 404 | 405 | def populationChecker(opponent): 406 | # make a nice file name 407 | file_name = "data/" + str(opponent).replace(" ", "_").replace(":", "_").lower() + "_pop.csv" 408 | # if the file exists dont run_one, it takes forever, make sure it exists 409 | if not os.path.isfile(file_name): 410 | df_main = pd.DataFrame(data=None, columns=col_names) 411 | for pop_size in populations: 412 | start_time = time.clock() 413 | pop_run = runGeneticAlgo(opponent, 414 | population_size=pop_size, 415 | number_of_game_turns=200, 416 | cycle_length=200, 417 | generations=150, 418 | mutation_probability=0.1, 419 | reset_file=True) 420 | end_time = time.clock() 421 | tmp_df = pd.read_csv(pop_run[0], names=col_names) 422 | tmp_df["population"] = pop_size 423 | tmp_df["time_taken"] = end_time - start_time 424 | df_main = df_main.append(tmp_df, ignore_index=True) 425 | df_main.to_csv(file_name) 426 | print("List Complete:", file_name) 427 | return df_main 428 | else: 429 | print("file already exists, no calcs to do.") 430 | file_df = pd.read_csv(file_name) 431 | # remove first column 432 | file_df = file_df[list(file_df)[1:]] 433 | return file_df 434 | -------------------------------------------------------------------------------- /LaTeX/tex/bibliography.bib: -------------------------------------------------------------------------------- 1 | @article{axelrod1987evolution, 2 | title = {The evolution of strategies in the iterated prisoner’s dilemma}, 3 | author = {Axelrod, Robert and others}, 4 | journal = {The dynamics of norms}, 5 | pages = {1--16}, 6 | year = {1987}, 7 | } 8 | 9 | @article{knight2017evolution, 10 | title={Evolution Reinforces Cooperation with the Emergence of Self-Recognition Mechanisms: an empirical study of the Moran process for the iterated Prisoner's dilemma}, 11 | author={Knight, Vincent and Harper, Marc and Glynatsi, Nikoleta E and Campbell, Owen}, 12 | journal={arXiv preprint arXiv:1707.06920}, 13 | year={2017} 14 | } 15 | 16 | @article{chui2017artificial, 17 | title = {Artificial intelligence the next digital frontier?}, 18 | author = {Chui, Michael}, 19 | journal = {McKinsey and Company Global Institute}, 20 | pages = {47}, 21 | year = {2017} 22 | } 23 | 24 | @inproceedings{lohn2004evolutionary, 25 | title={Evolutionary design of an X-band antenna for NASA's space technology 5 mission}, 26 | author={Lohn, Jason D and Linden, Derek S and Hornby, Gregory S and Kraus, William F}, 27 | booktitle={Antennas and Propagation Society International Symposium, 2004. IEEE}, 28 | volume={3}, 29 | pages={2313--2316}, 30 | year={2004}, 31 | organization={IEEE} 32 | } 33 | 34 | @article{savic1997genetic, 35 | title={Genetic algorithms for least-cost design of water distribution networks}, 36 | author={Savic, Dragan A and Walters, Godfrey A}, 37 | journal={Journal of water resources planning and management}, 38 | volume={123}, 39 | number={2}, 40 | pages={67--77}, 41 | year={1997}, 42 | publisher={American Society of Civil Engineers} 43 | } 44 | 45 | @article{chu1997genetic, 46 | title = {A genetic algorithm for the generalised assignment problem}, 47 | author = {Chu, Paul C and Beasley, John E}, 48 | journal = {Computers \& Operations Research}, 49 | volume = {24}, 50 | number = {1}, 51 | pages = {17--23}, 52 | year = {1997}, 53 | publisher = {Elsevier}, 54 | url = {https://ac.els-cdn.com/S0305054896000329/1-s2.0-S0305054896000329-main.pdf?_tid=dd585c54-0bf6-11e8-8ce4-00000aab0f02&acdnat=1518001734_d7656dfe4825599757058dea0d683c56} 55 | } 56 | 57 | @article{bhanu1995adaptive, 58 | title = {Adaptive image segmentation using a genetic algorithm}, 59 | author = {Bhanu, Bir and Lee, Sungkee and Ming, John}, 60 | journal = {IEEE Transactions on systems, man, and cybernetics}, 61 | volume = {25}, 62 | number = {12}, 63 | pages = {1543--1567}, 64 | year = {1995}, 65 | publisher = {IEEE}, 66 | url = {http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=478444} 67 | } 68 | 69 | @book{mitchell1998introduction, 70 | title = {An introduction to genetic algorithms}, 71 | author = {Mitchell, Melanie}, 72 | year = {1998}, 73 | publisher = {MIT press} 74 | } 75 | 76 | @article{turing1950computing, 77 | title = {Computing machinery and intelligence}, 78 | author = {Turing, Alan M}, 79 | journal = {Mind}, 80 | volume = {59}, 81 | number = {236}, 82 | pages = {433--460}, 83 | year = {1950}, 84 | publisher = {JSTOR} 85 | } 86 | 87 | @article{knight2017alphaZeroMIT, 88 | author = {Knight, Will}, 89 | title = {Alpha Zero’s “Alien” Chess Shows the Power, and the Peculiarity, of AI}, 90 | journal = {MIT Technology Review}, 91 | year = {2017}, 92 | url = {https://www.technologyreview.com/s/609736/alpha-zeros-alien-chess-shows-the-power-and-the-peculiarity-of-ai/}, 93 | } 94 | @article{rehmeyer2012climateNegotiations, 95 | author = {Rehmyer, Julie}, 96 | title = {Game theory suggests current climate negotiations won’t avert catastrophe}, 97 | journal = {Science News}, 98 | year = {2012}, 99 | url = {https://www.sciencenews.org/article/game-theory-suggests-current-climate-negotiations-won%E2%80%99t-avert-catastrophe}, 100 | } 101 | 102 | @article{Schneier2012doping, 103 | author = {Schneier, Bruce}, 104 | title = {Lance Armstrong and the Prisoners' Dilemma of Doping in Professional Sports}, 105 | journal = {WIRED}, 106 | year = {2012}, 107 | url = {https://www.wired.com/2012/10/lance-armstrong-and-the-prisoners-dilemma-of-doping-in-professional-sports/}, 108 | } 109 | 110 | @article{Perone2013blog, 111 | author = {Perone, Christian S.}, 112 | title = {Machine Learning :: Cosine Similarity for Vector Space Models (Part III)}, 113 | year = {2013}, 114 | url = {http://blog.christianperone.com/2013/09/machine-learning-cosine-similarity-for-vector-space-models-part-iii}, 115 | } 116 | 117 | @article{bakkes2009rapid, 118 | title = {Rapid and reliable adaptation of video game AI}, 119 | author = {Bakkes, Sander and Spronck, Pieter and Van den Herik, Jaap}, 120 | journal = {IEEE Transactions on Computational Intelligence and AI in Games}, 121 | volume = {1}, 122 | number = {2}, 123 | pages = {93--104}, 124 | year = {2009}, 125 | publisher = {IEEE}, 126 | url = {http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.386.7181&rep=rep1&type=pdf} 127 | } 128 | 129 | @book{darwin2009origin, 130 | title = {The origin of species by means of natural selection: or, the preservation of favored races in the struggle for life}, 131 | author = {Darwin, Charles and Bynum, William F}, 132 | year = {2009}, 133 | publisher = {AL Burt} 134 | } 135 | 136 | @misc{axelrodAuthors_2016, 137 | title = {An Open Framework for the Reproducible Study of the Iterated Prisoner's Dilemma}, 138 | url = {http://doi.org/10.5334/jors.125}, 139 | journal = {Journal of Open Research Software}, 140 | publisher = {Ubiquity Press}, 141 | author = {Knight, Vincent and Campbell, Owen and Harper, Marc and Langner, Karol and Campbell, James and Campbell, Thomas and Carney, Alex and Chorley, Martin and Davidson-Pilon, Cameron and Glass, Kristian and et al.}, 142 | year = {2016}, 143 | month = {Aug} 144 | } 145 | 146 | @article{snidal1985Politics, 147 | ISSN = {00438871, 10863338}, 148 | URL = {http://www.jstor.org/stable/2010350}, 149 | abstract = {Game theory is elaborated as a theoretical approach to international politics by contrasting it with metaphorical and analogical uses of games. Because it embraces a diversity of models, game theory is especially useful for capturing the most important contextual features of the international system that affect prospects for international cooperation. Through a discussion of the relation among and extension of different game models, the versatility and scope of game-theoretic approaches to international relations are demonstrated. Special attention is paid to the empirical issues of international politics which are raised by game theory and are analyzed in other articles in this symposium.}, 150 | author = {Duncan Snidal}, 151 | journal = {World Politics}, 152 | number = {1}, 153 | pages = {25-57}, 154 | publisher = {Cambridge University Press}, 155 | title = {The Game Theory of International Politics}, 156 | volume = {38}, 157 | year = {1985} 158 | } 159 | 160 | 161 | @article{tooby1988war, 162 | title = {The evolution of war and its cognitive foundations}, 163 | author = {Tooby, John and Cosmides, Leda}, 164 | journal = {Institute for evolutionary studies technical report}, 165 | volume = {88}, 166 | number = {1}, 167 | pages = {1--15}, 168 | year = {1988} 169 | } 170 | 171 | @book{low2015sex, 172 | title = {Why sex matters: A Darwinian look at human behavior}, 173 | author = {Low, Bobbi S}, 174 | year = {2015}, 175 | pages = {178--180}, 176 | publisher = {Princeton University Press} 177 | } 178 | 179 | @book{aumann1992handbook, 180 | title = {Handbook of game theory with economic applications}, 181 | author = {Aumann, Robert J and Hart, Sergiu}, 182 | volume = {2}, 183 | year = {1992}, 184 | publisher = {Elsevier} 185 | } 186 | 187 | @incollection{yamagishi1984prisoner, 188 | title = {Prisoner’s dilemma networks: selection strategy versus action strategy}, 189 | author = {Yamagishi, Toshio and Hayashi, Nahoko and Jin, Nobuhito}, 190 | booktitle = {Social dilemmas and cooperation}, 191 | pages = {233--250}, 192 | year = {1984}, 193 | publisher = {Springer} 194 | } 195 | 196 | 197 | @article{cable1997finance, 198 | Abstract = {New business startups with venture capital backing depend on mutual cooperation between entrepreneurs and venture capitalists, but little is known about what makes these relationships work. The present article considers the implicit similarities between entrepreneur-venture capitalist relationships and the Prisoner's Dilemma framework, using this paradigm to develop a conceptual model of entrepreneurs' and venture capitalists' decisions to cooperate. The model is used to generate a number of testable propositions concerning long- term cooperation between entrepreneurs and venture capitalists. Implications of the model for researchers, entrepreneurs, and venture capitalists are discussed, and the paper concludes by examining implications of the entrepreneur-venture capitalist context for the traditional Prisoner's Dilemma framework. "We need each other. We're in the same lifeboat; once we've cut the mooring and written the check and are rowing out to sea with the guy (sic), we can eith}, 199 | Author = {Cable, Daniel M. and Shane, Scott}, 200 | ISSN = {03637425}, 201 | Journal = {Academy of Management Review}, 202 | Keywords = {Strategic alliances (Business), Venture capital, New business enterprises, Joint ventures, Entrepreneurship, Business planning, Startup costs, Decision making, Probability theory, Prisoner's dilemma game}, 203 | Number = {1}, 204 | Pages = {142 - 176}, 205 | Title = {A PRISONER'S DILEMMA APPROACH TO ENTREPRENEUR-VENTURE CAPITALIST RELATIONSHIPS.}, 206 | Volume = {22}, 207 | URL = {http://search.ebscohost.com/login.aspx?direct=true&db=bth&AN=9707180262&site=ehost-live&scope=site}, 208 | Year = {1997}, 209 | } 210 | 211 | @article{cambell2016thesis, 212 | Author = {Cambell, James}, 213 | Title = {THESIS}, 214 | Year = {2016}, 215 | } 216 | 217 | @article{Ashlock2008, 218 | author = {Ashlock, Daniel and Kim, Eon Youn}, 219 | doi = {10.1109/TEVC.2008.920675}, 220 | journal = {IEEE Transactions on Evolutionary Computation}, 221 | keywords = {Automatic analysis,Evolutionary computation,Feature selection,Game theory,Prisoner's dilemma}, 222 | number = {5}, 223 | pages = {647--659}, 224 | title = {Fingerprinting: Visualization and automatic analysis of prisoner's dilemma strategies}, 225 | volume = {12}, 226 | year = {2008} 227 | } 228 | 229 | @article{Ashlock2004, 230 | author = {Ashlock, Daniel and Kim, Eun Youn and Kurt, Warren}, 231 | number = {2}, 232 | pages = {397--410}, 233 | title = {Finite Rationality and Interpersonal Complexity in Repeated Games}, 234 | volume = {56}, 235 | year = {2004} 236 | } 237 | 238 | @article{harper2017reinforcement, 239 | title = {Reinforcement learning produces dominant strategies for the Iterated Prisoner’s Dilemma}, 240 | author = {Harper, Marc and Knight, Vincent and Jones, Martin and Koutsovoulos, Georgios and Glynatsi, Nikoleta E and Campbell, Owen}, 241 | journal = {PloS one}, 242 | volume = {12}, 243 | number = {12}, 244 | pages = {e0188046}, 245 | year = {2017}, 246 | publisher = {Public Library of Science} 247 | } 248 | 249 | @article{axelrod1980effective, 250 | title={Effective choice in the prisoner's dilemma}, 251 | author={Axelrod, Robert}, 252 | journal={Journal of conflict resolution}, 253 | volume={24}, 254 | number={1}, 255 | pages={3--25}, 256 | year={1980}, 257 | publisher={Sage Publications Sage CA: Los Angeles, CA}, 258 | url={http://www.jstor.org/stable/pdf/173932.pdf} 259 | } 260 | @article{axelrod1980more, 261 | title={More effective choice in the prisoner's dilemma}, 262 | author={Axelrod, Robert}, 263 | journal={Journal of Conflict Resolution}, 264 | volume={24}, 265 | number={3}, 266 | pages={379--403}, 267 | year={1980}, 268 | publisher={Sage Publications Sage CA: Los Angeles, CA} 269 | } 270 | 271 | 272 | @article{press2012iterated, 273 | title={Iterated Prisoner’s Dilemma contains strategies that dominate any evolutionary opponent}, 274 | author={Press, William H and Dyson, Freeman J}, 275 | journal={Proceedings of the National Academy of Sciences}, 276 | volume={109}, 277 | number={26}, 278 | year={2012}, 279 | publisher={National Acad Sciences}, 280 | url={http://www.pnas.org/content/pnas/109/26/10409.full.pdf} 281 | } 282 | 283 | @article{mittal2009optimal, 284 | title={Optimal strategies of the iterated prisoner's dilemma problem for multiple conflicting objectives}, 285 | author={Mittal, Shashi and Deb, Kalyanmoy}, 286 | journal={IEEE Transactions on Evolutionary Computation}, 287 | volume={13}, 288 | number={3}, 289 | pages={554--565}, 290 | year={2009}, 291 | publisher={IEEE}, 292 | url={https://dspace.mit.edu/openaccess-disseminate/1721.1/54742} 293 | } 294 | 295 | @article{goldman2015spliddit, 296 | title={Spliddit: Unleashing fair division algorithms}, 297 | author={Goldman, Jonathan and Procaccia, Ariel D}, 298 | journal={ACM SIGecom Exchanges}, 299 | volume={13}, 300 | number={2}, 301 | pages={41--46}, 302 | year={2015}, 303 | publisher={ACM}, 304 | url={http://www.jagoldman.com/img/thesis.pdf} 305 | } 306 | 307 | @article{choi2010survey, 308 | title={A survey of binary similarity and distance measures}, 309 | author={Choi, Seung-Seok and Cha, Sung-Hyuk and Tappert, Charles C}, 310 | publisher={Citeseer}, 311 | url={http://www.iiisci.org/journal/CV$/sci/pdfs/GS315JG.pdf} 312 | } 313 | 314 | @inproceedings{horn1994niched, 315 | title={A niched Pareto genetic algorithm for multiobjective optimization}, 316 | author={Horn, Jeffrey and Nafpliotis, Nicholas and Goldberg, David E}, 317 | booktitle={Evolutionary Computation, 1994. IEEE World Congress on Computational Intelligence., Proceedings of the First IEEE Conference on}, 318 | year={1994}, 319 | organization={Ieee}, 320 | url={http://dataaspirant.com/2014/09/19/supervised-and-unsupervised-learning/} 321 | } 322 | 323 | @article{rahmat1999electromagnetic, 324 | title={Electromagnetic optimization by genetic algorithms}, 325 | author={Rahmat-Samii, Yahya and Michielssen, Eric}, 326 | journal={Microwave Journal}, 327 | volume={42}, 328 | number={11}, 329 | pages={232--232}, 330 | year={1999}, 331 | publisher={Horizon House Publications, Inc.} 332 | } 333 | 334 | @book{gass2005annotated, 335 | title={An annotated timeline of operations research: An informal history}, 336 | author={Gass, Saul I and Assad, Arjang A}, 337 | volume={75}, 338 | pages={125}, 339 | year={2005}, 340 | publisher={Springer Science \& Business Media} 341 | } 342 | 343 | @article{osang2013environmental, 344 | title={Environmental regulation of polluting firms: Porter's hypothesis revisited}, 345 | author={Osang, Thomas and Nandy, Arundhati}, 346 | journal={Revista Brasileira de Economia de Empresas}, 347 | volume={3}, 348 | number={3}, 349 | year={2013}, 350 | url={http://faculty.smu.edu/tosang/pdf/regln0803.pdf} 351 | } 352 | @misc{spliddit, 353 | title = {spliddit.org}, 354 | howpublished = {http://www.spliddit.org}, 355 | url = {http://www.spliddit.org} 356 | } 357 | 358 | @book{tukey1977exploratory, 359 | title={Exploratory data analysis}, 360 | author={Tukey, John W}, 361 | volume={2}, 362 | year={1977}, 363 | publisher={Reading, Mass.} 364 | } 365 | 366 | @inproceedings{norouzi2012hamming, 367 | title={Hamming distance metric learning}, 368 | author={Norouzi, Mohammad and Fleet, David J and Salakhutdinov, Ruslan R}, 369 | booktitle={Advances in neural information processing systems}, 370 | pages={1061--1069}, 371 | year={2012} 372 | } 373 | 374 | @article{bora2014effect, 375 | title={Effect of different distance measures on the performance of K-means algorithm: an experimental study in Matlab}, 376 | author={Bora, Mr and Jyoti, Dibya and Gupta, Dr and Kumar, Anil}, 377 | journal={arXiv preprint arXiv:1405.7471}, 378 | year={2014} 379 | } 380 | 381 | @book{breiman2017classification, 382 | title={Classification and regression trees}, 383 | author={Breiman, Leo}, 384 | year={2017}, 385 | publisher={Routledge} 386 | } 387 | 388 | @article{li2009strategy, 389 | title={A strategy with novel evolutionary features for the iterated prisoner's dilemma}, 390 | author={Li, Jiawei and Kendall, Graham}, 391 | journal={Evolutionary computation}, 392 | volume={17}, 393 | number={2}, 394 | pages={257--274}, 395 | year={2009}, 396 | publisher={MIT Press} 397 | } 398 | 399 | @article{hong2015top, 400 | title={Top tips to make your research irreproducible}, 401 | author={Hong, Neil P Chue and Crick, Tom and Gent, Ian P and Kotthoff, Lars and Takeda, Kenji}, 402 | journal={arXiv preprint arXiv:1504.00062}, 403 | year={2015} 404 | } 405 | 406 | @article{prlic2012ten, 407 | title={Ten simple rules for the open development of scientific software}, 408 | author={Prli{\'c}, Andreas and Procter, James B}, 409 | journal={PLoS Computational Biology}, 410 | volume={8}, 411 | number={12}, 412 | pages={e1002802}, 413 | year={2012}, 414 | publisher={Public Library of Science} 415 | } 416 | 417 | @article{sandve2013ten, 418 | title={Ten simple rules for reproducible computational research}, 419 | author={Sandve, Geir Kjetil and Nekrutenko, Anton and Taylor, James and Hovig, Eivind}, 420 | journal={PLoS computational biology}, 421 | volume={9}, 422 | number={10}, 423 | pages={e1003285}, 424 | year={2013}, 425 | publisher={Public Library of Science} 426 | } 427 | 428 | @incollection{whitley2012genetic, 429 | title={Genetic algorithms—A survey of models and methods}, 430 | author={Whitley, Darrell and Sutton, Andrew M}, 431 | booktitle={Handbook of natural computing}, 432 | pages={628--687}, 433 | year={2012}, 434 | publisher={Springer} 435 | } 436 | 437 | @article{hauptpractical, 438 | title={PRACTICAL GENETIC ALGORITHMS}, 439 | author={Haupt, Randy L and Haupt, Sue Ellen} 440 | } 441 | 442 | # Code 443 | @misc{axelrodproject, 444 | author = {The Axelrod project developers}, 445 | title = {Axelrod: v4.0.0}, 446 | year = {2017}, 447 | doi = {10.5281/168358}, 448 | url = {http://dx.doi.org/10.5281/zenodo.168358} 449 | } 450 | 451 | @misc{GitHub, 452 | author = {Linus Torvald}, 453 | title = {Git}, 454 | publisher = {GitHub}, 455 | journal = {GitHub repository}, 456 | url = {https://github.com/git/git}, 457 | } 458 | 459 | @misc{PandasGithub, 460 | author = {Multiple}, 461 | title = {Pandas}, 462 | publisher = {GitHub}, 463 | journal = {GitHub repository}, 464 | url = {https://github.com/pandas-dev/pandas}, 465 | } 466 | @inproceedings{Mckinney2010pandas, 467 | title={Data structures for statistical computing in python}, 468 | author={McKinney, Wes and others}, 469 | booktitle={Proceedings of the 9th Python in Science Conference}, 470 | volume={445}, 471 | pages={51--56}, 472 | year={2010}, 473 | organization={Austin, TX} 474 | } 475 | 476 | @article{hunter2007matplotlib, 477 | title={Matplotlib: A 2D graphics environment}, 478 | author={Hunter, John D}, 479 | journal={Computing in science \& engineering}, 480 | volume={9}, 481 | number={3}, 482 | pages={90--95}, 483 | year={2007}, 484 | publisher={IEEE} 485 | } 486 | 487 | @book{oliphant2006numpy, 488 | title={A guide to NumPy}, 489 | author={Oliphant, Travis E}, 490 | volume={1}, 491 | year={2006}, 492 | publisher={Trelgol Publishing USA} 493 | } 494 | 495 | @article{pedregosa2011scikit, 496 | title={Scikit-learn: Machine learning in Python}, 497 | author={Pedregosa, Fabian and Varoquaux, Ga{\"e}l and Gramfort, Alexandre and Michel, Vincent and Thirion, Bertrand and Grisel, Olivier and Blondel, Mathieu and Prettenhofer, Peter and Weiss, Ron and Dubourg, Vincent and others}, 498 | journal={Journal of machine learning research}, 499 | volume={12}, 500 | number={Oct}, 501 | pages={2825--2830}, 502 | year={2011} 503 | } 504 | 505 | @misc{dojoV008, 506 | author = {Marc and 507 | Vince Knight and 508 | Martin Jones and 509 | T.J. Gaffney and 510 | Toby Devlin and 511 | Nikoleta and 512 | Georgios Koutsovoulos}, 513 | title = {Axelrod-Python/axelrod-dojo: v0.0.8}, 514 | month = mar, 515 | year = 2018, 516 | doi = {10.5281/zenodo.1199134}, 517 | url = {https://doi.org/10.5281/zenodo.1199134} 518 | } -------------------------------------------------------------------------------- /LaTeX/tex/chapters/06-results.tex: -------------------------------------------------------------------------------- 1 | % !TEX root = ../main.tex 2 | \chapter{Results and Discussion}\label{ch:results} 3 | This chapter will look into the results produced after running the analysis across all opponents. 4 | Details of the distribution of the results and basic output from the computation are given in section~\ref{sec:descriptive_data}. 5 | 6 | The analysis performed in this chapter contains data for opponents listed in Appendix~\ref{apndx:opponents}. 7 | Opponents without results due to being incomplete are all the Long Run Time (LRT) strategies. 8 | As these become available they will be added to the analysis. 9 | % TODO: add the extra data if it becomes available 10 | 11 | Each opponent was submitted to the compute engine using the standard analysis factory class \mintinline{python}{AnalysisRun}, shown in Figure~\ref{apcode:AnalysisRun.py}, using native OS multi-threading to improve individual opponent analysis run times and the overall scalability of the project. 12 | We ran the code over a period of days resulting in 200+ output files of data which could then be analysed. 13 | 14 | \section{Resulting Data}\label{sec:descriptive_data} 15 | The data we generated from the final analysis contained 760 of the 959 opponents we set out to analyse. 16 | Figure~\ref{table:data_dump} shows a raw data head from the output of the $\phi$ opponent. 17 | 18 | \begin{table*}[ht] 19 | \centering 20 | \begin{tabular}{ccccccccc} 21 | \toprule 22 | gen & score mean & score median & score var & score range & best score & best sequence & name & seed \\ 23 | \midrule 24 | 1 & 2.90966 & 3.0575 & 1.665496 & 4.985 & 5.0 & DDD\ldots & $\phi$ & 0 \\ 25 | 2 & 4.79004 & 4.9250 & 0.420476 & 2.275 & 5.0 & DDD\ldots & $\phi$ & 0\\ 26 | 3 & 4.79352 & 4.97 & 0.489173 & 2.185 & 5.0 & DDD\ldots & $\phi$ & 0\\ 27 | 4 & 4.81386 & 5 & 0.460776 & 2.2 & 5.0 & DDD\ldots & $\phi$ & 0\\ 28 | 5 & 4.81826 & 5 & 0.479307 & 2.18 & 5.0 & DDD\ldots & $\phi$ & 0\\ 29 | 6 & 4.83508 & 5 & 0.430063 & 2.03 & 5.0 & DDD\ldots & $\phi$ & 0\\ 30 | \ldots & \ldots & \ldots & \ldots & \ldots & \ldots & \ldots & \ldots & \ldots\\ 31 | \bottomrule 32 | \end{tabular} 33 | \caption{Raw data from \mintinline{python}{AnalysisRun.py} output file}\label{table:data_dump} 34 | \end{table*} 35 | 36 | The data collected was, for each of the strategies, merged and collated with metadata to form an analysis table which was then processed. 37 | This metadata included player classifiers as given in the Axelrod library, generated values from fields in Table~\ref{table:data_dump} and information generalizing stochastic players to their original base player before seeding. 38 | Table~\ref{table:meta_data} shows the first few rows of this information. 39 | 40 | \begin{table*}[ht] 41 | \centering 42 | \begin{tabular}{cccccccc} 43 | \toprule 44 | base & stochastic & memory depth & makes use of & score bin & start move & blocks & mean block length\\ 45 | \midrule 46 | $\phi$ & False & inf & \mintinline{python}{set()} & very high & $D$ & 1 & 200\\ 47 | 48 | $\pi$ & False & inf & \mintinline{python}{set()} & very high & $D$ & 1 & 200\\ 49 | 50 | $e$ & False & inf & \mintinline{python}{set()} & very high & $D$ & 1 & 200\\ 51 | 52 | ALLCorALLD & True & 1 & \mintinline{python}{set()} & very low/high\footnote{Stochastic opponents can have different score bins depending on the seed.} & $D$ & 1 & 200\\ 53 | 54 | Adaptive & False & inf & \mintinline{python}{'game'} & very high & $C$ & 2 & 100\\ 55 | 56 | \ldots & \ldots & \ldots & \ldots & \ldots & \ldots & \ldots & \ldots\\ 57 | \bottomrule 58 | \end{tabular} 59 | \caption{Metadata which was merged to Table~\ref{table:data_dump} during analysis; joined base on name.}\label{table:meta_data} 60 | \end{table*} 61 | 62 | When analysing the overall best sequences for each opponent we can pick out some interesting and descriptive data. 63 | The distribution on best scores are showing in Figure~\ref{fig:best_score_hist}, its kernel density estimate (KDE) \cite{tukey1977exploratory} exaggerates that fact we have a skew towards the higher scores with a fat tail on 4.5 to 5. 64 | This shows we can average a score higher than 3.0 against the majority of opponents per turn. 65 | From this we can infer that there is a way of outplaying many of the opponents in the Axelrod Library (i.e. we can perform better than a game where both players would have an average score of $3.0$ from mutual cooperations). 66 | If there is a way of identifying an opponent then we would be able to tip the scales in a tournament by scoring 3 or more every turn by playing the sequences shown. 67 | 68 | \begin{figure}[ht] 69 | \includegraphics[width=0.95\textwidth, center]{./img/descriptive/best_score_hist.pdf} 70 | \caption{A histogram showing the distribution of best scores with overlaid KDE}\label{fig:best_score_hist} 71 | \end{figure} 72 | 73 | Figure~\ref{fig:cor_plot} shows patterns related to how the number of blocks within a sequence changes depending on score, opponent type and start move. 74 | In the left plot, solution sequences starting with a defection lead to 3 groupings; low block number, between 50 \& 125 blocks and near 200. 75 | This pattern has little information that describes the relationship between the solutions, but there could be an underlying reason in the make-up of an opponent which describes the groupings. 76 | All 3 contain both stochastic and non stochastic and have a wide range of scores. 77 | Starting with a cooperation shows a trend for non stochastic opponents; ignoring the near totalities on the far right shows a linear trend from 0 to 200 blocks as the score increases. 78 | This trend describes that, for non stochastic opponents, there are complicated solutions that score us more than simple solutions for particular opponents. 79 | It shows evidence that if a non stochastic opponent is more complicated in its solution, we can do better than if it is simple; for example, the solution for Tit For Tat is $C199,1$ which scores 3.01 whereas Adaptive Pavlov 2011 has the alternator solution $C1,1,1,1,\ldots$ and scores 3.97. 80 | This does not hold for stochastic opponents as, for example, ZD Extort with seed 0 has the solution sequence $D4,5,1,9,1,41,1,42,1,90,$ and only scored a 1.355. 81 | 82 | By looking into the distributions of best scores and the make up of solutions we are showing 2 things. 83 | That most opponents can be outplayed; we are able to score higher than an average of 3 per turn over the course of the game and hence, by the definition of the prisoners dilemma, defiantly come out with a higher score than our opponent. 84 | And that solution sequences are varied and there is little correlation between the complexity of a solution and the best score we can achieve. 85 | 86 | \begin{figure}[ht] 87 | \includegraphics[width=0.95\textwidth, center]{./img/descriptive/cor_plot.pdf} 88 | \caption{A joint plot of best score vs number of blocks coloured by stochastic boolean}\label{fig:cor_plot} 89 | \end{figure} 90 | 91 | \section{Solution Distance Matrices}\label{sec:distance_matracies} 92 | The first avenue of analysis, after constructing descriptive data, was to look at the relationship best response sequences have with each other. 93 | A distance matrix shows how much sequence for one opponent differs from every other, if 2 sequences are similar with respect to the distance function then they will score lower than 2 sequences that are more distinct. 94 | 95 | In each matrix we order the opponents, $S_(O_i)$, by the score of their sequences\footnote{See section~\ref{sec:notation} for explanations of notation.} so that the order is now $f(S_{O_0}) \ge f(S_{O_1}) \ge \ldots \ge f(S_{O_n})$. but we will shorten this notation $S_{O_i}$ to $S_{i}$ for simplicity. 96 | The best response sequence for the $i$th opponent, $S_i$ vs the best response sequence for the $j$th, $S_j$ is scored using our distance function, for example $d(S_0,S_n)$ is the distance between the best and worst scoring sequences respectively. 97 | The Matrix itself will be symmetric down the diagonal, so looking across rows vs columns tends to make more logical sense; the top rows are the highest scoring best response sequences, and the lower rows the worse scoring sequences. 98 | 99 | There were two distance measures considered: Hamming Distance and Cosine Distance. 100 | 101 | \paragraph{Hamming Distance}\cite{norouzi2012hamming} 102 | $$d(S_i,S_j) = S_i \cdot S_j^T $$ 103 | This corresponds to 104 | $$ 1-\frac{\sum^n_{i,j=0}\delta_{ij}}{n}\text{ where } \delta_{ij} = \begin{cases} 105 | 1 & S_i=S_j \\ 106 | 0 & S_i\ne S_j 107 | \end{cases} $$ 108 | 109 | The Hamming Distance represents the count of the elements that differ in any two sequences. 110 | A Hamming Distance will thus correspond to the number of places we have to play different moves against opponents to get out best score. 111 | Figure~\ref{fig:dist_ham} shows the matrix generated by the code in Figure~\ref{apcode:dist_matrix.py}. 112 | 113 | By observing the graph row by row, we can build an idea of how similar each sequence is to the range of others. 114 | The top section of the plot shows the best scoring sequences vs other high scoring sequences on the left, and vs the worse scoring sequences on the right. 115 | At the very top left there crossing dark and light columns, suggesting that there are lots of very similar or dissimilar solutions at the high vs high end of the score level. 116 | As we move to the high vs low scores there are larger blocks of dark red representing high dissimilarity. 117 | This is shown again as we move to the bottom of the diagram. 118 | The large blocks or red covering the bottom 100 or so rows show that the lowest scoring opponents have very different sequences to the higher scorers, but as a group of low scoring vs low scoring they are quite similar. 119 | 120 | We can look into what distances we expect by trying to analyse how scored are formed. 121 | The recording of an average score means that we can estimate the ratio of move combinations that formed a score because we know, if we are the first player, the pay offs for the combinations: $(C,C)=3$, $(C,D)=0$, $(D,C)=5$, $(D,D)=1$. 122 | Thus we can assume scoring in the midrange ($3$ ish) means mostly cooperating, or a combination of $(C,D)$ and $(D,C)$, whereas at the high and low ends its more likely we are to be defecting. 123 | This would mean when looking at the high vs low section and low vs high section of the graph we would expect a high level of similarity; $d(S_i,S_j)\approx 0$. 124 | However this hypothesis is not supported; its clear there is a mix of distances because all 4 corners of the graph are not showing $d(S_i,S_j)\approx 0$. 125 | 126 | To understand why this disparity between expectation and result exists we can look at a visual representation of each of the strategies. 127 | Figure~\ref{fig:sequence_plot_score} shows what the solution sequence looks like move by move, sorted by best scoring at the top of the left image moving down to worst scoring on the bottom of the right. 128 | From this diagram it is clear that some of the predictions above were true; the very high and low scoring solutions are totalities of defections. 129 | There is however much more noise for sequences at the top of the scoreboard, resulting in the large distances between the scores. 130 | 131 | Figure~\ref{fig:sequence_plot_score} also shows some interesting results about picking up points in patterns. 132 | Some opponents (approximately half way down on right image) work using ratios, we can trick them for half the game and take advantage for the remainder. 133 | Others are much shorter term solution, alternating between $C$ and $D$ every other turn ($1/4$ down the left image). 134 | When scoring in the mid range (bottom of right, top of left) there seems to be patterns too; totalities of $C$ are to be expected but there are some results that seem to be tricking an opponent for the first number of moved before defecting for the rest of the game. 135 | At the higher score ranges there seems to be lots of noise, many of these opponents are rather complicated in their Strategy, the top 12 non totalities are listed below: 136 | 137 | One final point regarding what Figure~\ref{fig:sequence_plot_score} shows is the effectiveness of the genetic algorithm. 138 | From the theory of repeated games, an equilibria for the repeated game must end in an equilibria for the static game. 139 | Hence for one of our solution sequences to be optimal the result must end in a $D$. 140 | This is true for all but $57$ of the solutions, meaning there is room for possible improvement in more than one result. 141 | These sub optimal results were all stochastic. 142 | 143 | \paragraph{Cosine Distance}\cite{bora2014effect} 144 | The cosine of two vectors constructed by using the dot product formula as shown. 145 | In our interpretation each dimension represents a sequence element so we are working in $\mathbb{R}^{200}$ with every value taking a $C:1$ a $D:0$. 146 | Figure~\ref{fig:dist_cos} shows the distance matrix generated from the data files using code in Figure~\ref{apcode:dist_matrix.py} 147 | 148 | $$ d(S_i,S_j) = \cos(\theta) = \frac{{S_i} \cdot {S_j}}{|| {S_i} || \; || {S_j} ||} $$ 149 | 150 | The Matrix for Cosine and Hamming are almost exactly similar, the only difference being the value of the distance between sequences. 151 | This is due to the two measures being similar in their relationship of space~\cite{choi2010survey}. 152 | We can conclude that there is no extra information shown in this diagram. 153 | \begin{figure}[ht] 154 | \centering 155 | \begin{minipage}{0.48\textwidth} 156 | \centering 157 | \includegraphics[width=1.0\textwidth, center]{./img/dist_matrix/dist_ham.pdf} 158 | \caption{Distance Matrix for Hamming Distance}\label{fig:dist_ham} 159 | \end{minipage}\hfill 160 | \begin{minipage}{0.48\textwidth} 161 | \includegraphics[width=1.0\textwidth]{./img/dist_matrix/dist_cos.pdf} 162 | \caption{Distance Matrix for Cosine Distance}\label{fig:dist_cos} 163 | \end{minipage} 164 | \end{figure} 165 | 166 | \section{Solution Groups}\label{sec:solutionGroups} 167 | \begin{figure}[ht] 168 | \includegraphics[width=0.95\textwidth, center]{./img/descriptive/sequence_scatter_colour.pdf} 169 | \caption{Trends for opponents grouped by their best sequence. Dot size represents the number of opponents in the group.}\label{fig:sequence_scatter} 170 | \end{figure} 171 | 172 | If we want to group opponents together, the most obvious way is to look at which opponents have have the same best score sequence. 173 | Appendix~\ref{apndx:solutionGroups} has full details, and figure~\ref{fig:sequence_scatter} shows a plot of the trends. 174 | This figure shows an almost logarithmic/polynomial trend in score as the number of blocks in a sequence increases. 175 | Below this trend there are also a group of stochastic opponents who dont seem to follow the pattern and form a group below this. 176 | This could indicate that the trend only applies to specific strategies and that this grouping of stochastic opponents use more unforgiving approaches, such as the ZD Extort strategies. 177 | 178 | \begin{figure}[ht] 179 | \centering 180 | \begin{minipage}{0.48\textwidth} 181 | \centering 182 | \includegraphics[width=1.0\textwidth, center]{./img/descriptive/sequence_plot_score_pt1.pdf} 183 | \end{minipage}\hfill 184 | \begin{minipage}{0.48\textwidth} 185 | \includegraphics[width=1.0\textwidth, center]{./img/descriptive/sequence_plot_score_pt2.pdf} 186 | \end{minipage} 187 | \caption{Sequence Diagram, sorted by score; High: top left $\rightarrow$ bottom right: Low. $C$ is light and $D$ is Dark. 188 | \textit{NOTE: labels are not complete, approx 1 in 4 shown.}}\label{fig:sequence_plot_score} 189 | \end{figure} 190 | 191 | Figure~\ref{fig:sequence_plot_score} provides a visual representation of sequences so that identifying groups of equivalent or almost equivalent solution sequences is easy. 192 | Families of similar solutions can be picked out, for example the Qlearners at the bottom right of the figure all have chunky sections to their solution sequences and are all grouped near one another, even though the hamming distance between any two in different families could be vastly different. 193 | The bottom of the left figure shows many occurrences of the largest group, $C199,1$ with some that have similar strategies with larger tails. 194 | Grouping strategies by their solution sequence seems to be very restrictive for providing relationships between strategies and solutions. 195 | It may be beneficial to apply some binary pattern recognition to the solutions, the solutions all seem quite chaotic that could have patterns in other areas of mathematics. 196 | 197 | Of the initial sequences that were predicted to occur as solutions and set as our starting population, as described in Section~\ref{sec:alteringinitialpopulation}, 29 of them appeared as actual solution sequences. 198 | This is shown below. 199 | 200 | \begin{multicols}{3} 201 | \begin{itemize} 202 | \item C199,1 with 96 opponents 203 | \item C198,2 with 21 opponents 204 | \item C196,4 with 3 opponents 205 | \item C194,6 with 1 opponent 206 | \item C193,7 with 2 opponents 207 | \item C100,100 with 8 opponents 208 | \item C5,195 with 3 opponents 209 | \item C2,1,1,196 with 1 opponent 210 | \item C2,198 with 2 opponents 211 | \item C1,1,1,1,\ldots with 17 opponents 212 | \item C1,1,1,197 with 1 opponent 213 | \item C1,2,1,196 with 1 opponent 214 | \item C1,199 with 11 opponents 215 | \item D1,198,1 with 21 opponents 216 | \item D1,196,3 with 1 opponent 217 | \item D1,194,5 with 2 opponents 218 | \item D1,4,195 with 4 opponents 219 | \item D1,3,196 with 4 opponents 220 | \item D1,2,197 with 9 opponents 221 | \item D2,197,1 with 3 opponents 222 | \item D2,196,2 with 2 opponents 223 | \item D2,195,3 with 1 opponent 224 | \item D2,194,4 with 2 opponents 225 | \item D2,193,5 with 1 opponent 226 | \item D2,1,197 with 1 opponent 227 | \item D3,196,1 with 4 opponents 228 | \item D3,194,3 with 1 opponent 229 | \item D3,192,5 with 8 opponents 230 | \item D200 with 113 opponents 231 | \end{itemize} 232 | \end{multicols} 233 | 234 | 235 | \section{Clustering Analysis} 236 | Here we will look at ways of grouping opponents based on their best response sequences and what that means for scores and potential hidden relationships between strategies. 237 | These clustering algorithms are not meant for predictive purposes and are all forms of unsupervised learning; there is not enough metadata present in each strategy definition to use as features for supervised prediction of our best response sequences. 238 | These clustering algorithms come from the python library SciKit Learn~\cite{pedregosa2011scikit}. 239 | 240 | \subsection{K Means clustering~\cite{bora2014effect}}\label{ssec:k_means} 241 | Here we attempted to label k clusters depending on their parameters. 242 | Nothing much was identified from the k means clustering data analysis, typically clusters were found to strongly correlate with parameters such as number of blocks or mean block length. 243 | Figure~\ref{fig:k_means} shows the clustering for the most correlated variables in 3 dimensions. 244 | Its clear that these clusters are layered over the number of blocks of the solutions sequence. 245 | Section~\ref{sec:solutionGroups} looks in depth as to how opponents are distributed over the solution sequences. 246 | 247 | \begin{figure}[ht] 248 | \includegraphics[width=1.0\textwidth, center]{./img/descriptive/k_means.pdf} 249 | \caption{K means clustering with 2,3 and 4 clusters}\label{fig:k_means} 250 | \end{figure} 251 | 252 | \subsection{Regression Trees~\cite{breiman2017classification}} 253 | Regression trees are a way of looking at reducing the variance of a parameter through attributes of observed instances. 254 | SciKit Learn has an implementation of regression trees, the code used to generate the data in this subsection is shown in Appendix code~\ref{apcode:reg_tree.py}. 255 | 256 | Our case, we are looking at reducing the variance of the score by looking at which turns tend to cause the largest disparity in score. 257 | To do this the algorithm look at reducing the Mean Absolute Error (MAE) of the scores, which is, in effect, reducing the $L1$ norm (sometimes referred to as the $L_1$ loss of the algorithm) of the scores. 258 | 259 | $$\text{MAE} = \frac{1}{n}\sum_{i=0}^n |x_i-y_i|$$ 260 | 261 | Figure~\ref{fig:reg_tree} shows which moves cause the largest variance in resulting score per turn. 262 | As you move right on the tree (or move in the false direction on the tree) then playing cooperations at the moves listed. 263 | The diagram backs up that by playing $C$ moves consistently (furthest right leaf) we reach a score value of $3.01$, and with 325 samples and $MAE=0.205$ this is a clear way of doing well against a large number of opponents. 264 | If we move left on the diagram then the score value increases, which is to be expected; these are the best turns to defect and get a higher score as a result. 265 | 266 | The move that dictates the best change in score is move $164$ (starting from the 0th turn), if we defect on this move then the score value goes up but so does the $MAE$. 267 | Looking at leaf nodes provides an overview of solutions being grouped into minimal $MAE$ after considering 5 moves. 268 | These represent which moves are played on the same turn by multiple solutions, all who scored approximately the same. 269 | It is clear that there are some paths that, considering one move difference, have clusters of very different scores. 270 | For example, the 3rd and 4th groups from the left (with 19 and 4 members respectively) show that, on the turn $156$, you can defect and most likely get around a $3.2$ or cooperate and get half that score. 271 | This shows that even though some solutions are similar, the corresponding opponents are vastly different in operation and will punish at some points others would forgive. 272 | 273 | This tree does not have very many useful properties other than displaying the complexities in trying to predict a solution for any opponent. 274 | As discussed in Section~\ref{sec:solutionGroups} if an opponent does not sit in one of the major groups it becomes incredibly hard to observe some sort of pattern between them. 275 | One solution may be to look into combinations of initial handshakes that create the largest variation of responses in the population of opponents; from this we could map these results to a best response sequence to play for the remainder of the game. 276 | Section~\ref{sec:follow_up} looks into this in more detail. 277 | 278 | \begin{sidewaysfigure} 279 | \includegraphics[width=1.0\textwidth, center]{./img/descriptive/reg_tree.pdf} 280 | \centering 281 | \caption{A regression tree showing which moves introduce the largest absolute error in the best score. If $X[i]<=0.5$ is true, it means move $i$ is a Defection move. 282 | \textbf{TRUE or left} $\Rightarrow D$, \textbf{FALSE or right} $\Rightarrow C$} 283 | \label{fig:reg_tree} 284 | \end{sidewaysfigure} 285 | 286 | 287 | \section{Conclusion}\label{sec:results_conclusion} 288 | This chapter includes the most clear results from the analysis conducted on the data output. 289 | The results are showing signs of patterns between the complexities in the solution sequence and the strategies they correspond to. 290 | The goal, as described in Section~\ref{sec:briefOverview}, to identify solutions for every opponent has been satisfied. 291 | However there is no strong correlation or predictor identified to allow accurate prediction of a strategies solution to beat an opponent given only basic parameters of an opponent. 292 | 293 | The data produced by this report allows us to easily identify best responses to certain predetermined tournaments where all the solution information is known. 294 | For example, in a tournament against players Defector Hunter, Solution B1, Willing and Tricky Defector we can play $C1,199$ and easily come out on top. 295 | The code in Appendix Section~\ref{apcode:tournCode.py} produced the results in Table~\ref{table:table_test_opsponents}. 296 | It is clear that our solution sequence came out on top for this set of opponents, however if we played this in a different tournament we would score very low. 297 | If all players in the tournament have the same solution then selecting the absolute sequence to play is trivial, however if we have multiple solutions then the problem becomes one of identifying which opponent we are playing in any round. 298 | This problem is looked into in more detail in Section~\ref{sec:follow_up}. 299 | 300 | \begin{table*} 301 | \centering 302 | \begin{tabular}{ccccc} 303 | \toprule 304 | Rank & Name & Median score & Cooperation rating & Wins \\ 305 | \midrule 306 | 0 & Cycler: C1,199 & 4.9625 & 0.005 & 4.0 \\ 307 | 1 & Tricky Defector & 2.23625 & 0.2475 & 1.5 \\ 308 | 2 & SolutionB1 & 1.7675 & 0.71775 & 2.0 \\ 309 | 3 & Defector Hunter & 1.508125 & 0.995 & 0.0 \\ 310 | 4 & Willing & 1.5 & 0.849 & 0.5 \\ 311 | \bottomrule 312 | \end{tabular} 313 | \caption{Table of test opponents}\label{table:table_test_opsponents} 314 | \end{table*} 315 | 316 | 317 | We have looked at various methods of grouping and reducing the solutions to try and observe a clear reason for an opponent having a certain solution; yet within the data we have created there seems to be no clear sign of correlation. 318 | Each section of analysis has its own takeaways but there is no theme being displayed throughout the results. 319 | If the information here is to be assessed further it would be beneficial to gather more data on each opponent to work with. 320 | The lack of usable opponent meta-data means that the GA, currently, is the only reasonable method of identifying solution sequences. 321 | 322 | -------------------------------------------------------------------------------- /LaTeX/tex/chapters/05-implementation.tex: -------------------------------------------------------------------------------- 1 | % !TEX root = ../main.tex 2 | \chapter{Implementation Of Sequence Discovery}\label{ch:implementation} 3 | Genetic algorithms can have many variations depending on parameters and implementations of any attached methods. In this chapter, consideration will be given to the process of finding the optimal sequence of moves against another player. 4 | There will be various approaches used and a detailed analysis of the optimisation procedures and parameters will be described for each parameter. 5 | 6 | In this chapter the terms strategy and opponent are used interchangeably. 7 | An opponent has a set of rules for calculating its next move, this is known as its strategy and is unique. 8 | Also in this chapter the terms solution and sequence are interchange, often called the solution sequence. 9 | A solution is a sequence of $C$ and $D$ moves, as described in Section~\ref{sec:solutionForm}. 10 | 11 | \section{Background} 12 | Before conducting the bulk calculations for the set of opponents listed in Appendix Section~\ref{apndx:opponents} we will test values for the algorithms' parameters to see what values of parameters are best for finding solution sequences. 13 | We will run a series of tests against pre selected strategies and review the results of their best score per turn before running the full analysis. 14 | These test opponents have been selected because they are interpretable and have either calculable solution sequences or are interesting stochastic opponents. 15 | Table~\ref{table:table_test_opsponents} shows the opponents and their solution. 16 | Each one has a fundamentally different structure to how they work, because of this we can confirm that the genetic algorithm can select the optimal sequence solution for each structure. 17 | 18 | \begin{table*} 19 | \centering 20 | \begin{tabular}{ccc} 21 | \toprule 22 | Player & Optimal Sequence & Representation of game length $n$\\ 23 | \midrule 24 | axl.TitForTat()&\(CCC\ldots CD\)& $Cn-1,1$\\ 25 | axl.Alternator()&\(DDD\ldots DD\)&$Dn$\\ 26 | axl.Grudger()&\(CCC\ldots CD\)&$Cn-1,1$\\ 27 | axl.Random()&\(DDD\ldots DD\)&$Dn$\\ 28 | axl.EvolvedFSM16()& $CDC\ldots DD$ & $C1,1,n-2$\\ 29 | axl.CollectiveStrategy()&$CDC\ldots CD$&$C1,1,n-3,1$\\ 30 | axl.Champion()& Various\footnote{Champion is a stochastic opponent.} & $NA$\\ 31 | axl.ZDExtort()& Various\footnote{ZD Extort is a stochastic opponent.} & $NA$\\ 32 | \bottomrule 33 | \end{tabular} 34 | \caption{Table of test opponents}\label{table:table_test_opsponents} 35 | \end{table*} 36 | 37 | In this chapter sequences will be loosely described as `converged' if the best score of the population has reached a stable point considering the generations it has run. 38 | In order to find these settings we look at is how the best score in a population rises over the number generations after changing the parameter were observing. 39 | Once the best score per turn hits a maximum such that it doesn't appear to change no mater how many more generations are run this will be described as an optimal solution sequence. 40 | Note that for a solution to be unique we must be playing a non stochastic opponent, in the event of playing a stochastic opponent we will seed the random element of their behaviour for reproducibility. 41 | 42 | During the investigation we may find solutions that are not optimal, meaning that the algorithm will have found a solution that will do well against an opponent but hasn't found the best from the solution space. 43 | These sub optimal solutions are due to the occurrence of local maxima (due to an increasing fitness function) in the space of neighbouring solution sequences to members. 44 | Section ~\ref{subsec:geneticAlgorithms} discusses how GAs are designed to mitigate local maxima. 45 | 46 | Some of the questions we will hope to answer in this chapter include: 47 | \begin{itemize} 48 | \item If we have a larger initial population sample to start with, will we reach our maximum best score earlier? 49 | \item Will increasing the number of generations impact our ability to find an optimal solution? Is there an optimal number of generations to run the algorithm for such that we always find a solution sequence? 50 | \item If we make each sequence more likely to mutate generation to generation what will happen? 51 | What about increasing how potent our mutations are? 52 | \item How can we overcome the possibility of our algorithm finding a local maximum rather than the global maximum? 53 | \end{itemize} 54 | 55 | \section{Changing Initial Population Size}\label{sec:ChangingInitialPopulationSize} 56 | The initial population size is the number of starting sequences we use in our algorithms first generation. 57 | Once this generation concludes the population will go through the series of phases outlined in Figure~\ref{fig:customGAcycle}. 58 | This will alter the population to keep the fittest members to continue on to subsequent generations. 59 | We will look into population size because, during any generation the size of the population influences the range of scores that we can achieve against our opponent. 60 | The more unique members we have the more unique evaluations of the fitness function exist. 61 | Because of this we can reasonably assume the larger our population gets the chance of finding the solution sequence with the optimal score will increase. 62 | Hence with a larger population we should converge to the solution sequence in less generations. 63 | For this section we analysed a range of populations to understand how solutions are affected when we run our algorithm through the set of population sizes \(|P| \in [25,50,100,150,200,250,500]\). 64 | 65 | The code used for this these tests will be shown in Appendix Snippet~\ref{apcode:populationChecker.py} as an implementation how this analysis was conducted. 66 | It leverages the use of the function \mintinline{python}{runGeneticAlgo} show in Appendix Snippet~\ref{apcode:runGeneticAlgo.py}. 67 | This code will output data in the form of Table~\ref{table:popCheckerDataTable}. 68 | 69 | As a note on efficiency; increasing the size of our population will have an impact on computation time. 70 | Each generation must process the full population in a linear fashion causing a computation overhead of \(O(n)\). 71 | For an increase to be useful a in any time restricted scenario our algorithm would need to show a higher order benefit in our generations to convergence, or in an increase of average score per turn. 72 | However We are not working in a time restricted scenario, and so we should just select the best overall initial population size independent of computation overhead. 73 | 74 | \begin{table*} 75 | \centering 76 | \begin{tabular}{ccccccc} 77 | \toprule 78 | Best Score & Gen & Mean Score & Population & Sequence & Std Dev & Time Taken \\ 79 | \midrule 80 | 2.425 & 1 & 2.264 & \textbf{25.0} & DD\ldots & 0.067 & 6.646\\ 81 | 2.425 & 2 & 2.343 & \textbf{25.0} & DD\ldots & 0.046 & 6.646\\ 82 | 2.425 & 3 & 2.393 & \textbf{25.0} & DD\ldots & 0.038 & 6.646\\ 83 | \ldots & \ldots & \ldots & \ldots & \ldots & \ldots & \ldots \\ 84 | 2.830 & 102 & 2.782 & \textbf{100.0} & CC\ldots & 0.112 & 28.425\\ 85 | \ldots & \ldots & \ldots & \ldots & \ldots & \ldots & \ldots \\ 86 | 2.980 & 150 & 2.911 & \textbf{500.0} & CC\ldots & 0.158 & 152.684\\ 87 | \ldots & \ldots & \ldots & \ldots & \ldots & \ldots & \ldots \\ 88 | \bottomrule 89 | \end{tabular} 90 | \caption{Output data table}\label{table:popCheckerDataTable} 91 | \end{table*} 92 | 93 | After running the tests for the populations stated above we can group the data by population size and observations on how changing this parameter affects different opponents can be made. 94 | Figure~\ref{fig:INIT-POP-bs-v-gens-all} shows the best score in the population for each opponent as the generations increase for every population size. 95 | We see that the initial population size has a significant effect on finding better sequences. 96 | This figure shows if there is a larger initial population there is typically a higher best score shown once concluding all of the generations. 97 | It doesn't, however, ensure that we find the solution sequence, as is shown in the lack of long plateaus of best score across opponents. 98 | 99 | \begin{figure}[h] 100 | \includegraphics[width=0.8\textwidth, keepaspectratio, center]{./img/plots/INIT_POP_bs_v_gens_all.pdf} 101 | \caption{\textbf{Initial Population Size Analysis:} Best score per turn vs generation for different initial population sizes}\label{fig:INIT-POP-bs-v-gens-all} 102 | \end{figure} 103 | 104 | The improvements from the effect of population increase are non-linear from observation of Figure~\ref{fig:INIT-POP-mean-bs-v-init-pop-all}. 105 | This figure shows the change in mean best score across population in the final generation against the population size. 106 | The change in final best score for a population of 50 compared with a population of 250 is large in comparison to the same relative increase from 250 to 500. 107 | This may suggest there are more efficient approaches to improving our final best score after a certain initial population is reached. 108 | 109 | \begin{figure}[h] 110 | \includegraphics[width=0.8\textwidth, keepaspectratio, center]{./img/plots/INIT_POP_mean_bs_diff_v_init_pop_all.pdf} 111 | \caption{\textbf{Initial Population Size Analysis:} Scatter of mean best score vs different initial populations}\label{fig:INIT-POP-mean-bs-v-init-pop-all} 112 | \end{figure} 113 | 114 | None of these results have found a solution sequence (or at least we cant tell from the graph). 115 | It is clear that larger initial populations do, on a relative scale, much better than small ones. 116 | There are no large plateaus for the graph, so as we continue our research the initial population size will be increased to 150 to keep test computation times manageable. 117 | The actual parameter we will use in the full analysis will be given consideration in Section~\ref{sec:conclusionOfApproach}. 118 | 119 | \section{Generation Length Analysis}\label{sec:generationlengthanalysis} 120 | Another major component parameter of a genetic algorithm is the number of generations it will run for before outputting a final set of members with solution sequences. 121 | The number of generations has an influence on a number of different things within the algorithm: 122 | \begin{itemize} 123 | \item {The total combinations of features that the algorithm can evaluate. 124 | The more generations it runs for the more combinations of members we can evaluate a fitness for.} 125 | \item {The total number of low performers it removes in the population over the course of its run. 126 | Each generation removes a proportion of its lowest fitness members. 127 | By definition of the algorithm we have a non-decreasing sequence of fitness scores; removing more lower fitness members will lead to a population of better or equal scoring members. } 128 | \end{itemize} 129 | Generation size differs from other parameters in the fact this is purely performance based, there is no changes to the inner mechanics of the algorithm only the amount of time it will run for. 130 | A genetic algorithm with 1 generation is just a series of tests split into 2 results sets --- good performers and bad performers. 131 | As we extend the number of generations we want be more focused on what happens to measured quantities normalised by generations, rather than any absolute improvement. 132 | 133 | As a note on efficiency, the goal of finding the optimal solution sequence for each opponent would be solved by extending the generations to infinity, i.e exhaustively search the whole solution space. 134 | This, however, is not a feasible solution, and so this section looks in to the effect of increasing generations has on improving a solution sequence with respect to the generations its run. 135 | Here we will take generation lengths \(G \in [50,150,250,350,450,500]\) during these tests and a population size of 150. 136 | The code in Appendix Snippet~\ref{apcode:generationChecker.py} shows how the approach the generation analysis was undertaken. 137 | 138 | Figure~\ref{fig:GENS-mean-bs-diff-v-gens-all} shows how the mean best score across the population changes generation to generation normalized by generation. 139 | It shows that over 150 generations the best scores against Alternator across its population increased, on average, by 0.0035 per generation. 140 | From this we can observe how the number of generations has a declining effect the overall change in our mean best score per generation. 141 | This trend is to be expected, when we are close to a maximum it is more difficult to randomly select which element in the sequence needs changing to improve a members score. 142 | On this result we can conclude as we increase generations there is less and less benefit per generation. 143 | 144 | \begin{figure}[h] 145 | \includegraphics[width=0.8\textwidth, keepaspectratio, center]{./img/plots/GENS_mean_bs_diff_v_gens_all.pdf} 146 | \caption{\textbf{Generation Analysis:} Mean Best Score diff vs total generation lengths}\label{fig:GENS-mean-bs-diff-v-gens-all} 147 | \end{figure} 148 | 149 | Figure~\ref{fig:GENS-max-bs-v-gens-all} shows the maximum best score in a population once the analysis has concluded. 150 | For most of the opponents 250 generations seems reasonable to reach a solution sequence as shown in the Alternator and Tit For Tat. 151 | However, there are clear signs of local maximums occurring in the EvolvedFSM16 example. 152 | In Figure~\ref{fig:GENS-max-bs-v-gens-all} EvolvedFSM16 has reached a better sequence in 150 generations than 500\footnote{These are independent trials and have different sequences.}, meaning that increasing the generation length doesn't necessarily mean finding a global maximum. 153 | The complexities with local maximums during the generations lie with mutation rates and crossovers. 154 | We will cover this in Section~\ref{sec:mitigatingLocalMaximums} 155 | 156 | \begin{figure}[h] 157 | \includegraphics[width=0.8\textwidth, keepaspectratio, center]{./img/plots/GENS_max_bs_v_gens_all.pdf} 158 | \caption{\textbf{Generation Analysis:} Max best score vs total number of generations}\label{fig:GENS-max-bs-v-gens-all} 159 | \end{figure} 160 | 161 | It is clear that a higher number of generations is preferred to find a better solution sequence. 162 | As the number of generations increase each generation provides less of an improvement to the best scores. 163 | This is due to the probability of finding a better solution sequence decreasing as we continue to improve a population. 164 | The benefits of extending the generations are incredibly useful and due to the amount of computation time we can probably incorporate a high number of generations. 165 | However we may have better performance by altering another parameter of the algorithm. 166 | The actual parameter we will use in the full analysis will be given consideration in Section~\ref{sec:conclusionOfApproach}. 167 | 168 | \section{Changing Mutation Rate}\label{sec:changeingmutationrate} 169 | This section looks at changing the amount of mutations that occur in our population and the number elements within a sequence each mutation effects. 170 | The default settings are a mutation frequency, $M_f$, of 0.1, meaning for every 10 members of our population that continue into the next generation one of these has some elements in its sequence changed. 171 | And a mutation potency, $M_p$, of 1, meaning that every sequence that is mutated only has 1 element altered. 172 | \begin{itemize} 173 | \item Is it beneficial for more/less than 1 in 10 members to be mutated generation to generation? (higher frequent mutation) 174 | \item Is changing one or more actions of a members' sequence the best way of mutating a member? (higher potent mutation) 175 | \end{itemize} 176 | 177 | These are two separate questions; first we will look at increasing the potency of our mutation. 178 | Once we have found some information on how this effects our solution, we can look into the frequency of our mutations with the new potency as a permanent setting. 179 | 180 | As a not on efficiency, this approach allows for an \(O(1)\) factor of computation scaling. 181 | Changes in mutation are great candidates for an approach to reduce the number of generations to a solution sequence compared with other approaches. 182 | 183 | \subsection{Changing Mutation Potency}\label{subsec:changingMutationPotency} 184 | Changing the potency of the algorithm will provide small changes in a members features generation to generation. 185 | Increasing the potency also has the effect of increasing the Hamming distance\footnote{Hamming distance: \(d(s_1,s_2)\) = the number of differing positions between 2 sequences \(s_1\) and \(s_2\). 186 | For example: \(d(111,110) = d(CCC,CCD) = 1 \). This is covered in details in section~\ref{sec:distance_matracies}} between original and the mutated sequence. 187 | 188 | Increasing potency too much has the potential for an algorithm that is too unstable for convergence. 189 | We can imagine a sequence as a vector in 200 dimensional space then a mutation for the \(i^{th}\) sequence element is the same as changing the vector in its \(i^{th}\) dimension. 190 | Shortening this example to a vector in 3 dimensions (or a sequence of length 3) then a mutation is much more easily visualised. 191 | Mutation potency should be kept low to keep consecutively mutated sequences more similar, keeping results of the mutation within a small neighbourhood of the original. 192 | We will look into having mutation potencies \(M_p \in [1,2,3,5,10,15,20]\). 193 | 194 | Figure~\ref{fig:MUT-POT-bs-v-gen-all} shows the best score as the algorithm progresses through generations. 195 | There no clear benefit from increasing the mutation potency. 196 | For example, in the Collective Strategy plot, having 15 genes changed per mutation still does not improve our score as much as changing only 2 or 3. 197 | This may be down to chance of the random parameters used, however looking at more opponents than just Collective Strategy (EvolvedFSM16 for example) we find there is no clear benefit to increasing the mutation potency with respect to the overall best score against an opponent. 198 | 199 | \begin{figure}[h] 200 | \includegraphics[width=0.8\textwidth, keepaspectratio, center]{./img/plots/MUT_POT_bs_v_gen_all.pdf} 201 | \caption{\textbf{Mutation Potency Analysis:} Best score vs generation for different mutation potencies}\label{fig:MUT-POT-bs-v-gen-all} 202 | \end{figure} 203 | 204 | Figure~\ref{fig:MUT-POT-bs-diff-v-pot-all} shows the trend of average increase of final best score across the population per generation against the mutation potency. 205 | The increase in mean best score difference is not substantial and is most likely be down to chance in the areas it does change. 206 | We see there is no sign these variables are correlated and hence can assume this parameter has negligible effect on the outcome of our sequence. 207 | The actual parameter we will use in the full analysis will be given consideration in Section~\ref{sec:conclusionOfApproach}. 208 | 209 | \begin{figure}[h] 210 | \includegraphics[width=0.8\textwidth, keepaspectratio, center]{./img/plots/MUT_POT_bs_diff_v_pot_all.pdf} 211 | \caption{\textbf{Mutation Potency Analysis:} Average best score diff vs mutation potencies}\label{fig:MUT-POT-bs-diff-v-pot-all} 212 | \end{figure} 213 | 214 | \subsection{Changing Mutation Frequency}\label{subsec:changingMutationFrequency} 215 | In contrast to changing the mutation potency, increasing the frequency should allow us to generate more unique sequences generation to generation. 216 | We will look at what happens when we run the genetic algorithm on a set of mutation frequencies \(M_f \in [0.1,0.2,0.3,0.4,0.5]\). 217 | The code in Appendix Snippet~\ref{apcode:mutationFrequencyChecker.py} shows the code that completed this analysis. 218 | 219 | Figure~\ref{fig:MUT-FREQ-bs-v-gen-all} shows how the best score improved generation to generation for each opponent across different mutation frequencies. 220 | The results show there is little effect on the best score as we increase the mutation frequency. 221 | However there is an interesting result that can be seen on the Grudger and EvolvedFSM16 plots; the algorithm has found 3-4 clearly different solution sequences each with different scores. 222 | In the Grudger plot, we can see that the mutation frequencies of 0.15 and 0.2 produced higher scoring solutions than the other mutation frequencies. 223 | In the EvolvedFSM16 Plot we have the same result. 224 | There is no positive (or negative) correlation displayed in any of the opponents however. 225 | 226 | \begin{figure}[h] 227 | \includegraphics[width=0.8\textwidth, keepaspectratio, center]{./img/plots/MUT_FREQ_bs_v_gen_all.pdf} 228 | \caption{\textbf{Mutation Frequency Analysis:} Best score vs generation for different mutation frequencies}\label{fig:MUT-FREQ-bs-v-gen-all} 229 | \end{figure} 230 | 231 | 232 | % \begin{figure}[h] 233 | % \includegraphics[width=0.8\textwidth, keepaspectratio, center]{./img/plots/MUT_FREQ_bs_diff_v_freq_all.pdf} 234 | % \caption{\textbf{Mutation Frequency Analysis:} Average best score diff vs mutation frequencies}\label{fig:MUT-FREQ-bs-diff-v-freq-all} 235 | % \end{figure} 236 | 237 | \subsection{Conclusions of altering Mutation parameters} 238 | In the previous sections no improvements were found from altering any of the two parameters. 239 | The default of $M_f=0.1$ and $M_p=1$ will be used in further analysis, and the actual parameters we will use in the full analysis will be given consideration in Section~\ref{sec:conclusionOfApproach}. 240 | 241 | These sections did show, in a clear manner, that there were local maxima found. 242 | Section~\ref{subsec:changingMutationFrequency} showed clearly that sub optimal solutions for EvolvedFSM16 and Grudger were common. 243 | Section~\ref{sec:mitigatingLocalMaximums} will discuss approaches we can take to reduce the chances of this happening. 244 | 245 | \section{Mitigating Local Maximum Solutions}\label{sec:mitigatingLocalMaximums} 246 | Local maxima have clearly occurred when the algorithm calculates sequences for opponents Grudger and EvolvedFSM16. 247 | There may be other occurrences we have not observed. 248 | Each stochastic opponent is also incredibly difficult to understand whether a solution is actually found, this is discussed further ing Section~\ref{sec:stochasticOpponents} 249 | 250 | 251 | Figure~\ref{fig:MUT-FREQ-bs-v-gen-all} shows that there are clearly multiple distinct score plateaus reached for the Grudger opponent. 252 | The difference between Grudger and the other opponents considered is that the Grudger has a singularity where its behaviour changes. 253 | The change in behaviour is not uncommon, Tit For Tat, Collective Strategy and others work in a similar way and in all cases the algorithm has managed to identify this behaviour and adapt to overcome its negative effects. 254 | 255 | Grudger is an opponent which it is possible to attain a local maximum score and be `trapped' in this solution sequence. 256 | If we look at a random start sequence and then the end sequence after 250 generations we can see that the genetic algorithm is learning defect after some point and cooperate before. 257 | This is due to the fact a `good' solution will cooperate to the point of a defection and end in lots of defections after it has already defected counter Grudgers' harsh behaviour (see Section~\ref{sec:strategiesOfInterest} for the Grudger strategy). 258 | 259 | %Grudger best start: \(C:[6,8,2,1,2,1,6,1,2,2,4,2,1,1,2,10,1,1,1,3,1,3,1,1,1,1,2,2,1,1,1,2,3,5,2,2,2,1,1,2,1,1,4,1,2,1,1,1,2,3,1,1,2,2,1,7,3,2,1,2,3,1,1,5,1,1,1,1,1,1,1,1,1,1,4,1,1,1,1,4,2,1,1,3,1,1,2,3,4,1,1,3,2,4,2,1,2,3]\) 260 | Grudger best start: C$6,8,2,1,2,1,6,1,2,2,4,2,1,1,\ldots$ (No pattern is obvious.) 261 | 262 | Grudger best end: C$22,178$ 263 | 264 | We may be able to identify a way to mitigate this singularity effect by looking at the difference between 2 opponents whose strategy differs by their forgiveness parameter. 265 | Grudger and Tit For Tat differ in their strategies in two ways: 266 | \begin{itemize} 267 | \begin{item} 268 | Grudger never changes its mind. 269 | There is one change in behaviour for the entire game, unlike Tit For Tat. 270 | The algorithm then only has a single opportunity to observe this per population per generation meaning the behaviour is much less frequently encountered. 271 | \end{item} 272 | \begin{item} 273 | Grudger will not forgive and becomes a `dumb' only defect opponent. 274 | This means that the move the algorithm picks up the effect of a single defection in its sequence it starts playing a `dumb' opponent. 275 | A random start of Cs and Ds puts the likelihood of at least 1 defection occurring in the first 10 moves at above \(99.99\% \). 276 | This swap from `smart' to `dumb' will, most likely, always occur in the first 10 moves; 277 | the only way of extending the `smart' player is to add a cooperation to the end of the starting Cs. 278 | \end{item} 279 | \end{itemize} 280 | 281 | To investigate this we will play two totality games against grudger, one of all Cs and one of all Ds, shown in Snippet~\ref{code:gudgerTotalities}. 282 | These games show we should be converging on a totality of Cs rather than what its doing, finding the totality of Ds after a number of cooperations. 283 | This is probably because the algorithm initially limits its best score per turn once the first generation is complete and a cut-off defection has been established for each of the initial population. 284 | The crossover method between generations then doesn't provide enough of a mix up to allow the algorithm to escape the local minimum by switching a subsection with a sufficiently different, potentially better, subsection. 285 | Then when it comes to mutating, there is little any number of mutations can do to drastically change large sections of the solution sequence without knowing exactly where to target. 286 | 287 | \begin{figure}[h] 288 | \inputminted{python}{code_snippets/grudgerTotalities.py} 289 | \caption{Grudger matches against totalities}\label{code:gudgerTotalities} 290 | \end{figure} 291 | 292 | The example totalities, shown in Snippet~\ref{code:gudgerTotalities}, are edge cases and would be rarely encountered as a starting point in the random initial population. 293 | Because of this, the algorithm has to shuffle towards the potential benefit of using these totalities rather than start with analysing them. 294 | Our case against grudger requires the algorithm to attempt this shuffle towards a totality after encountering the grudging effect. 295 | This would then require the algorithm to select the first defection move and change it to a cooperation move all by chance. 296 | The likelihood of this occurring is incredibly small, and starting at common or uniform sequences would be more beneficial to searching to solution space, as posed in Section~\ref{sec:alteringinitialpopulation}. 297 | 298 | \subsection{Ineffective Approaches of Altering Crossover And Mutation}\label{subsec:ineffectiveApproachOfAlteringCrossoverAndMutation} 299 | The process of converging to Ds when building a solution against grudger then sheds light on the process the algorithm takes to find a solution. 300 | If the algorithm is to find the optimal solution sequence, it must take a crossover and mutation path which doesn't cut off better paths as the algorithm work our way towards good solutions. 301 | This is much easier said than put into practice due to the way the algorithm `cuts off paths'. 302 | The current technique is taking halves from 2 members and merging them, potentially removing successful relationships between elements in both the first and second halves. 303 | If we reverse this thinking and try to alter our crossover design and mutation rate such that, instead of `cutting off' a path by taking large sections of each member, it may be possible to `build' new ones using more, smaller, sections of the 2 parents. 304 | 305 | We can re-design the crossover to switch up smaller subsections of the solution sequence then allow the mutations to optimise these sub-sequences. 306 | The current design is shown in Figure~\ref{fig:oldCrossover}. 307 | We want to allow the crossover to have more of an impact on optimizing each section. 308 | i.e.go from: 309 | \[|oooooooooooooooooooo| \text{ and } |xxxxxxxxxxxxxxxxxxxx|= |ooooooooooxxxxxxxxxx|\] 310 | to the mixture: 311 | \[|oooooooooooooooooooo|\text{ and } |xxxxxxxxxxxxxxxxxxxx|= |ooxxooxxooxxooxxooxx|\] 312 | This will allow the mutation to alter the subsections in a more interlaced manner, hopefully overcoming the pitfalls of sparse mutations to escape local maximums. 313 | 314 | \begin{figure}[h] 315 | \inputminted{python}{code_snippets/oldCrossover.py} 316 | \caption{Old Crossover algorithm}\label{fig:oldCrossover} 317 | \end{figure} 318 | 319 | Our new crossover method is shown in Figure~\ref{fig:newCrossover}. 320 | As shown, the algorithm splits the two sequences into 10 section and the new sequence is formed from alternating sections. 321 | Figure~\ref{fig:newCrossoverEX} shows an example of the new crossover method. 322 | 323 | \begin{figure}[h] 324 | \inputminted{python}{code_snippets/newCrossover.py} 325 | \caption{New Crossover algorithm}\label{fig:newCrossover} 326 | \end{figure} 327 | 328 | \begin{figure}[h] 329 | \inputminted{python}{code_snippets/newCrossoverEX.py} 330 | \caption{Example of new crossover algorithm}\label{fig:newCrossoverEX} 331 | \end{figure} 332 | 333 | After testing at this new crossover algorithm with the default mutation (freq=.1 and pot=1), there was no noticeable improvement of mitigating local maxima from introducing this change of function. 334 | Each players average score per turn was within the original score per turn if the algorithms crossover method was unchanged. 335 | The problem of mitigating sub optimal solutions may be more efficiently solved using a predefined population. 336 | Section~\ref{sec:alteringinitialpopulation} discusses this in more detail. 337 | 338 | \section{Altering Initial Population}\label{sec:alteringinitialpopulation} 339 | When performing analysis of an opponent using a GA it can sometimes be useful to allocate the starting positions of our feature selection to create uncommon\footnote{For example of a randomly constructed sequence that is a totality of Cs is incredibly rare, with around a \(6.2^{-61}\) chance of occurring naturally with a random selection.} starting points. 340 | Altering the initial population allows selection of starting sequences that fit patterns known to have good results against certain strategies. 341 | For example totalities, with some head/tail moves, are usually very effective against some opponents; 342 | Tit For Tat, Random or Grudger for example. 343 | Alternating sequences are also good starting sequences for an initial population, allowing a more intelligently distributed set of starting points for the random mutation process. 344 | 345 | Adding a pre defined starting population can be visualised as placing balls on a lumpy 3d plane to try and find the deepest valley. 346 | Starting with an educated guess and distributing the starting positions evenly means we are less likely get all the balls stuck in one, sub optimal, valley. 347 | This section looks into benefits of using a pre defined population in the GA. 348 | 349 | \subsection{Constructing a population} 350 | We will start by creating a population of successful known starting members. 351 | When starting with these members we allow the entropy of the genetic algorithm to alter these sequences towards optimal solutions (assuming these are not already optimal). 352 | Deciding where to start our algorithm may mitigate potential sub-optimal solutions by reducing the distance between the starting sequences and optimal solutions. 353 | This list of starting members are stated below: 354 | 355 | Totalities 356 | \begin{itemize} 357 | \item \(C:[200]\) | 1 $\times$ 2 sequence 358 | \end{itemize} 359 | 360 | Single Change Sequences 361 | \begin{itemize} 362 | \item \(\{C:[i,200-i]\} \quad i\in [1,10]\) | 10 $\times$ 2 sequences 363 | \item \(\{C:[200-i,i]\} \quad i\in [1,10]\) | 10 $\times$ 2 sequences 364 | \end{itemize} 365 | 366 | Matching Tail Sequences 367 | \begin{itemize} 368 | \item \(\{C:[i,200-(i+j),j]\} \quad i,j \in [1,5]\}\) | 25 $\times$ 2 sequences 369 | \end{itemize} 370 | 371 | Alternating 372 | \begin{itemize} 373 | \item \(C:[\ (i,i)^{100/i}] \quad i \in \{1,2,4,5\}\) | 4 $\times$ 2 sequences 374 | \end{itemize} 375 | 376 | 3 Block Handshakes*\footnote{This was added after the first test, with different population sizes, with the new population} 377 | \begin{itemize} 378 | \item \(C:[\ i,j,k,200-(i+j+k)] \quad i,j,k \in \{0,1,2,3\}\) | 32 $\times$ 2 sequences 379 | \end{itemize} 380 | 381 | For each of these starting members, a sequence with a defection move to start but with the same pattern will also be added to the set of starting sequences. 382 | This in total gives 164 (after code~\ref{code:initialPopulationCode}) constructions of sequences, we will then make up the difference to the population limit using a set of random sequences. 383 | 384 | Now that the Initial population has been altered we can conduct the analysis in Sections~\ref{sec:ChangingInitialPopulationSize},~\ref{sec:generationlengthanalysis},~\ref{subsec:changingMutationPotency} and~\ref{sec:subsec:changingMutationFrequency} again. 385 | 386 | \subsection{Population Size}\label{subsec:populationSize} 387 | Figure~\ref{fig:NEW-INIT-POP-bs-v-gen-all} shows the best score generation to generation for each population size. 388 | We will have \(|P| \in [102,200,250,500]\) due to starting with the 102 initial members (*this was the initial size of pre set population, this increased to 164 after this test). 389 | 390 | \begin{figure}[h] 391 | \includegraphics[width=0.8\textwidth, keepaspectratio, center]{./img/plots/NEW_INIT_POP_bs_v_gen_all_old.pdf} 392 | \caption{Best Score vs generations for pre-set initial populations on top of random sequences}\label{fig:NEW-INIT-POP-bs-v-gen-all} 393 | \end{figure} 394 | 395 | Figure~\ref{fig:NEW-INIT-POP-bs-v-gen-all} appears to have a better results for Collective Strategy with an initial set population for the algorithm. 396 | Of the results that are not showing optimal solutions we can observe that the algorithm has score plateaus for all but Random, Collective Strategy, ZD Extort and Champion. 397 | This is probably due to 2 different reasons: 398 | 399 | \textbf{Random \& Stochastic Opponents}: This strategy should being beaten with a totality of D, the algorithm has in fact converged to almost this totality, but still has intermittent Cs. 400 | The reason for this is the scoring grade `score per turn' will reflect on the number of intermittent Cs in the Random players sequence. 401 | More Ds in the Random sequence will allow the solutions defections to score less. 402 | This leads to a solution sequence containing some random Cs not because they score better in some turns, but because these tests not a fair trial. 403 | The two sequences play against different Random opponents generation to generation without then being seeded. 404 | 405 | \textbf{Collective Strategy}: This strategy is a combination of a handshake and the Grudger strategy. 406 | If we look into the solution it would be seen to have found the handshake but then arrives on the Grudger effect later in the game. 407 | After this encounter the same problem as we had before, with grudger, the algorithms limits the damage by splitting solutions into Cs then Ds. 408 | Solving the collective strategy (and handshakes in general) may be simple; 409 | we just put in all the possible n move handshakes followed by totalities and then set to work on the 2nd part of the sequence. 410 | (*This means we increase our initial population from 102 members to 164) 411 | 412 | It is clear that having a larger population is good from the old analysis, but the initial population still has to be tweaked to improve its scores against certain handshake opponents. 413 | The Random and other stochastic opponents are a special cases that require more analysis to find the absolute optimal. 414 | The new initial population, now of size 164, will include all combinations of C \& D of length 4, followed by finishing on all Cs or Ds as shown in the Totalities \& handshakes section of Snippet~\ref{code:initialPopulationCode}. 415 | Further tests will also supplement this population with random members for \(|P| \in [164,200,250,500]\) 416 | 417 | \begin{figure}[h] 418 | \inputminted{python}{code_snippets/initialPopulationCode.py} 419 | \caption{Initial Population Code}\label{code:initialPopulationCode} 420 | \end{figure} 421 | 422 | Figure~\ref{fig:NEW-INIT-POP-bs-v-gen-non-performers} shows the best score generation to generation for each population size, this time for the previous strategies where an optimal solution was not found. 423 | After adding the extra members of the initial population we can see that now the handshake strategy (such as Collective Strategy) are solved much sooner. 424 | Also shown is the Random and ZD extort players, these are examples of Stochastic players which are still not finding optimal solutions due to the observation of seeding stochastic opponents. 425 | These classes of opponent are examined further in Section~\ref{sec:stochasticOpponents}. 426 | 427 | \begin{figure}[h] 428 | \includegraphics[width=0.8\textwidth, keepaspectratio, center]{./img/plots/NEW_INIT_POP_bs_v_gen_non_performers.pdf} 429 | \caption{\textbf{New Population:} Non optimal sequence players after changing initial population}\label{fig:NEW-INIT-POP-bs-v-gen-non-performers} 430 | \end{figure} 431 | 432 | It is clear the change of initial population is beneficial. 433 | The actual parameter we will use in the full analysis will be given consideration in Section~\ref{sec:conclusionOfApproach}, but the initial population will stay as an effective starting point for the GA. 434 | 435 | \subsection{Generation Length, Mutation Potency \& Mutation Frequency Analysis After Change of Initial Population}\label{subsec:starting_pop_analysis} 436 | Figure~\ref{fig:NEW-GEN-bs-v-gen-all} shows how the best score is affected while changing the number of generations the algorithm will run to \(G \in [50,150,250,350,450,500] \). 437 | This figure is also displaying with a population size of \(|P|=200\), 164 of which are pre defined. 438 | It shows no real improvement from increasing the generations, the actual parameter we will use in the full analysis will be given consideration based off conclusions made in Section~\ref{sec:generationlengthanalysis}. 439 | 440 | \begin{figure}[h] 441 | \includegraphics[width=0.8\textwidth, keepaspectratio, center]{./img/plots/NEW_GEN_bs_v_gen_all.pdf} 442 | \caption{\textbf{New Population:} Generation Length analysis for best score mean against generation length.}\label{fig:NEW-GEN-bs-v-gen-all} 443 | \end{figure} 444 | 445 | Figure~\ref{fig:NEW-MUT-POT-bs-v-gen-all} shows the best score generation to generation while changing the mutation potency. 446 | \(M_p \in [1,2,3,5,10,15,20] \) was used whilst also running with a population size of \(|P|=200\). 447 | The figure shows no real improvement from increasing this variable of the algorithm. 448 | The actual parameter we will use in the full analysis will be given consideration based off conclusions made in Section~\ref{subsec:changingMutationPotency}. 449 | 450 | \begin{figure}[h] 451 | \includegraphics[width=0.8\textwidth, keepaspectratio, center]{./img/plots/NEW_MUT_POT_bs_v_gen_all.pdf} 452 | \caption{\textbf{New Population:} Best score against generation for different mutation potency levels}\label{fig:NEW-MUT-POT-bs-v-gen-all} 453 | \end{figure} 454 | 455 | 456 | Figure~\ref{fig:NEW-MUT-FREQ-bs-gen-all} shows the best score generation to generation while changing the mutation frequency. 457 | \(M_f \in [0.1,0.15,0.2,0.25] \) was used whilst also running with a population size of \(|P|=200\). 458 | The figure shows no real improvement from increasing this variable of the algorithm. 459 | The actual parameter we will use in the full analysis will be given consideration based off conclusions made in Section~\ref{subsec:changingMutationFrequency}. 460 | 461 | \begin{figure}[h] 462 | \includegraphics[width=0.8\textwidth, keepaspectratio, center]{./img/plots/NEW_MUT_FREQ_bs_v_gen_all.pdf} 463 | \caption{\textbf{New Population:} Best score against generation for different mutation frequencies}\label{fig:NEW-MUT-FREQ-bs-gen-all} 464 | \end{figure} 465 | 466 | \subsubsection{Discussion}\label{subsubsec:discussion} 467 | The approach of adding a predefined set of members to the population before running the algorithm was incredibly successful. 468 | Most opponents have solution sequence equal to or very close to members in the starting population. 469 | This result will mean that when calculating the solution sequence for the main population the predefined set, generated by Snippet~\ref{code:initialPopulationCode}, will be used to shorten analysis times. 470 | 471 | Random, ZDExtort and Champion are still being shown that the algorithm is not finding the optimal solution for these opponents. 472 | This is due to the fact these player belong to the class of stochastic opponents. 473 | Section~\ref{sec:stochasticOpponents} will look into these in more detail, putting forward a possible approach to simplify finding of a solution sequence for these opponents. 474 | 475 | \section{Stochastic Opponents}\label{sec:stochasticOpponents} 476 | Stochastic opponents create a problem with respect to continuity of testing. 477 | The programming of the algorithm we are using generates a new opponent for every member every generation we run. 478 | This, essentially, leads to the algorithm playing a new version of the opponent every time, leaving very little opportunity to identify features of stochastic opponents which can be exploited. 479 | 480 | The concept of a stochastic opponent can be somewhat `overridden' by seeding the pseudo random number generator that creates the parameters that define what moves the strategy will take. 481 | Currently the only way of doing this is by seeing the whole Axelrod library upon initialising the opponent instance. 482 | Snippet~\ref{code:wrappingFunction} shows the code for wrapping any given opponent with the global seed command. 483 | 484 | \begin{figure}[h] 485 | \inputminted{python}{code_snippets/classWrappingFunction.py} 486 | \caption{A function for wrapping a player with a global seed function call}\label{code:wrappingFunction} 487 | \end{figure} 488 | 489 | When calculating the solution sequence for all of the opponents we will do so with seeded versions of the stochastic opponents. 490 | The parameters for stochastic opponents will be the default set by the creators and new instances will be made with the seed that is set for algorithm as a whole. 491 | The means we will be playing the same `version' of a stochastic opponent over and over, removing the layer of abstraction in its strategy. 492 | 493 | \section{Conclusion of approach}\label{sec:conclusionOfApproach} 494 | Each section of testing allowed the algorithm to become more specialised in solving the given task. 495 | The list at the end of this section shows which parameters we have chosen for the final analysis. 496 | 497 | After constructing the final analysis factory and running some environment checks we made the following changes to the code for ease of use: 498 | \begin{itemize} 499 | \item Number of generations changed: $300\rightarrow600$. 500 | \item Crossover algorithm, shown in Figure~\ref{fig:newCrossover}, was reverted to code given in Figure ~\ref{fig:oldCrossover} 501 | \end{itemize} 502 | 503 | These changes resulted in a streamlined piece of code we were able to run on the Cardiff University Mathematics department big compute machine. 504 | Reflecting on the execution and deployment of the code shows that we could have improved the analysis of short run time strategies against longer run time strategies and the reproducibility of the batch processing within the Axelrod Dojo Module. 505 | As far as each individuals analysis goes, I feel like the approach of adding predetermined sequences to an initial population was a huge improvement in compute time and overall predictions of best sequences. 506 | 507 | The algorithm parameters described in section~\ref{subsec:geneticAlgorithms} have the following parameters before running the final analysis: 508 | \begin{itemize} 509 | \item Initial Population, $P$, of 164 pre generated members + 86 random members for a total of 250. 510 | \item Generation length, $G$, of 600. 511 | \item Mutation Potency, $M_p$, of 1. 512 | \item Mutation Frequency, $M_f$, of 0.1. 513 | \item Bottleneck, $b=P/4$. 514 | \item Crossover method as that shown in Snippet~\ref{fig:oldCrossover}. 515 | \item Mutation method as that shown in Snippet~\ref{apcode:mutateFromDojo} 516 | \item Each opponent is wrapped in the seed wrapper shown in Snippet~\ref{code:wrappingFunction}, stochastic opponents will be run 10 times each with different seeds. 517 | \end{itemize} --------------------------------------------------------------------------------