├── README.md ├── aggregate.py ├── create_ranking.py ├── pagerank.py ├── rank_aggregators.py ├── test ├── data.csv ├── data10_10.csv ├── data20_40.csv └── data50_150.csv └── test_runs.txt /README.md: -------------------------------------------------------------------------------- 1 | rank_aggregation 2 | =============================================== 3 | 4 | This repository contains a number of simple rank aggregation algorithms and iterative algorithms that take as input a ranker and improves the ranking. 5 | 6 | Copyright (c) 2014, Sibel Adali 7 | 8 | All rights reserved. 9 | 10 | Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: 11 | 12 | • Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. 13 | • Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. 14 | 15 | Any publication resulting from the use of this work must cite the following publication : 16 | 17 | Sibel Adali, Brandeis Hill and Malik Magdon-Ismail. 18 | "Information vs. Robustness in Rank Aggregation: Models, Algorithms and a Statistical Framework for Evaluation", 19 | Journal of Digital Information Management (JDIM), special issue on Web information retrieval, 5(5), October, 2007. 20 | 21 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 22 | 23 | Module Description 24 | ------------------- 25 | 26 | All rankers are provided in file named rank_aggregators.py, except for pagerank method is given in a separate module. 27 | 28 | The module create_ranking.py can be used to create random test rankers. 29 | 30 | All rankers should be provided in a single input file which is: 31 | 32 | * comma separated 33 | * contain a header row 34 | * first column is the object ids, assumed to be integers 35 | * each column is a separate ranker 36 | * if an object is not ranked by a ranker, leave that value empty. 37 | 38 | Rankers can be total or partial, ranks can be consecutive or not, ties may exist. Currently, all ties are disregarded. There is no penalty for breaking ties. 39 | 40 | The performance score is given by Kendall-tau, implemented as a score between 1 and -1. See Wikipedia for details. 41 | 42 | Simple aggregators pagerank and indegree are based on a graph representation of the ranks, where the weights from object i to j represents the number of rankers that rank object j higher than object i. 43 | 44 | Iterative improvements algorithms are iterative greedy flip, igf, (flip a pair as long as improvements are made), iterative best flip, ibf, (flip a pair even when it does not improve for each possible pairs and try other greedy flips), and remove top k worst rankers, ir. 45 | 46 | IBF is described in the above paper. 47 | 48 | 49 | A wrapper to call rank aggregation from comamand line is also included. You can call a simple aggregator, followed by any number of iterative improvements on it. 50 | 51 | Please include 52 | 53 | To call the wrapper around the rank aggregation algorithms, use the following: 54 | 55 | Usage 56 | ------------- 57 | 58 | Call the wrapper function by: 59 | 60 | python aggregate.py inputfile aggregator 61 | 62 | Aggregator list: 63 | 64 | * in: indegree 65 | * pg alpha: pagerank with given alpha (float between 0-1, default 0.85) 66 | * rnd k: random with k tries 67 | 68 | Iterative algorithms (executed in the order given): 69 | 70 | * igf: iterative greedy flip 71 | * ibf k: iterative best flip (at most k (integer, default 1) rounds 72 | * ir k: iterative remove up to k (integer, default 1) rankers 73 | 74 | All parameters with default values must be explicitly provided when combined with other functions. 75 | 76 | Example: 77 | 78 | python aggregate.py test/data10_10.csv pg 0.85 ibf 79 | Pagerank, followed by ibf 80 | 81 | Example: 82 | 83 | python aggregate.py test/data10_10.csv in ir 5 ibf 84 | Indegree followed by iterative remove, followed by ibf 85 | 86 | 87 | -------------------------------------------------------------------------------- /aggregate.py: -------------------------------------------------------------------------------- 1 | """ This is a command line runner for rank aggregation. 2 | Author: Sibel Adali 3 | 4 | To learn about how to run, simply type: 5 | 6 | python aggregate.py 7 | 8 | """ 9 | 10 | import sys 11 | import rank_aggregators as r 12 | import time 13 | 14 | def print_menu(): 15 | print "Usage: python aggregate.py inputfile aggregator " 16 | print "Aggregator list:" 17 | print "\tin: indegree" 18 | print "\tpg alpha: pagerank with given alpha (float between 0-1, default 0.85)" 19 | print "\trnd k: random with k tries" 20 | print 21 | print "Iterative algorithms (executed in the order given)" 22 | print "\tigf: iterative greedy flip" 23 | print "\tibf k: iterative best flip (at most k (integer, default 1) rounds" 24 | print "\tir k: iterative remove up to k (integer, default 1) rankers" 25 | print 26 | print "Example: python aggregate.py pg 0.85 ibf" 27 | print "\tPagerank, followed by ibf" 28 | print "Example: python aggregate.py in ir 5 ibf" 29 | print "\tIndegree followed by iterative remove, followed by ibf" 30 | 31 | def print_error(msg): 32 | print 33 | print "*" * (len(msg)+10) 34 | print "ERROR >>>", msg 35 | print "*" * (len(msg)+10) 36 | print 37 | print_menu() 38 | print 39 | sys.exit() 40 | 41 | if __name__ == "__main__": 42 | start = time.time() 43 | if len(sys.argv) <3: 44 | print_menu() 45 | sys.exit() 46 | 47 | fname = sys.argv[1] 48 | try: 49 | (objects,ranker_names) = r.read_rankers(fname) 50 | except: 51 | print_error("Incorrect file provided, cannot read rankers") 52 | 53 | agg = sys.argv[2] 54 | arguments = sys.argv[3:] 55 | 56 | lastloc = 0 57 | if agg == 'pg': 58 | alpha = 0.85 59 | if len(arguments)>0: 60 | try: 61 | alpha = float(arguments[0]) 62 | lastloc = 1 63 | except: 64 | print_error("Incorrect alpha provided or alpha is omitted") 65 | ranker, score = r.pagerank_aggregator(objects, 0.000001, alpha) 66 | print "Pagerank algorithm, alpha =", alpha, ", score:", score 67 | 68 | elif agg == 'in': 69 | ranker, score = r.indegree_aggregator(objects) 70 | print "Indegree algorithm, score:", score 71 | 72 | elif agg == 'rnd': 73 | k=1 74 | if len(arguments)>0: 75 | try: 76 | k = int(arguments[0]) 77 | lastloc = 1 78 | except: 79 | print_error("An integer for the number of tries is required") 80 | print "Random rank algorithm with k =", k 81 | ranker, score = r.best_random_aggregator(objects, k) 82 | else: 83 | print_error("No valid rank aggregation algorithm found") 84 | 85 | arguments = arguments[lastloc:] 86 | 87 | while len(arguments) > 0: 88 | agg = arguments[0] 89 | lastloc = 1 90 | if agg == 'ibf': 91 | newranker, score = r.iterative_best_flip(objects, ranker) 92 | print "Iterative best flip, score:", score 93 | 94 | elif agg in ['igf','ir']: 95 | k = 1 96 | if len(arguments) > 1: 97 | try: 98 | k = int(arguments[1]) 99 | lastloc = 2 100 | except: 101 | print_error("An integer k value is needed for algorithm" + agg) 102 | if agg == 'igf': 103 | newranker, score, flipped = r.iterative_greedy_flip(objects, ranker, k) 104 | print "Iterative greedy flip with k =", k, "score:", score 105 | else: 106 | newranker, score, removed, newobjects = r.remove_top_k(objects, ranker_names, ranker, k) 107 | print "Iterative best removal with k =", k, "score:", score 108 | objects = newobjects 109 | line = "" 110 | for item in removed: 111 | line += item + ", " 112 | print "Removed rankers (in order):", line.strip().strip(",") 113 | ranker = newranker 114 | else: 115 | print_error("Unknown algorithm"+agg) 116 | 117 | arguments = arguments[lastloc:] 118 | 119 | print "Final score:", score 120 | print "Final ranker:" 121 | r.print_single_ranker(ranker) 122 | 123 | end = time.time() 124 | print 125 | print "Took %.2f seconds" % (end-start) 126 | print 127 | -------------------------------------------------------------------------------- /create_ranking.py: -------------------------------------------------------------------------------- 1 | """ 2 | Program to create a set of ranked lists of a given number of objects, 3 | to be used for testing rank aggregation code. 4 | Author: Sibel Adali 5 | 6 | Creates partial rankers between num_objects/2 to 3*num_objects/4 objects 7 | completely randomly. 8 | 9 | Usage python create_ranking outfilename numobjects numrankers 10 | 11 | Output is saved in an output file in comma separated format 12 | 13 | """ 14 | 15 | import random 16 | import sys 17 | 18 | def get_val(val): 19 | if val == None: 20 | return "" 21 | else: 22 | return str(val) 23 | 24 | if __name__ == "__main__": 25 | if len(sys.argv) < 4: 26 | print "Usage python create_ranking outfilename numobjects numrankers" 27 | sys.exit() 28 | 29 | foutname = sys.argv[1] 30 | 31 | num_objects = int(sys.argv[2]) ##numbered 1,2,... 32 | num_rankers = int(sys.argv[3]) 33 | 34 | 35 | ##init rankers 36 | rankers = {} 37 | obj = range(1, num_objects+1) 38 | for i in obj: 39 | rankers[i] = [] 40 | 41 | 42 | ##each ranker ranks about num_objects/2 to 3*num_objects/4 objects 43 | min = num_objects/2 44 | max = 3*num_objects/4 45 | ##create each ranker and append 46 | for i in range(num_rankers): 47 | num_ranked = random.randint(min, max) 48 | newranker = obj[:num_ranked] + [None]*(num_objects-num_ranked) 49 | random.shuffle(newranker) 50 | for i in range(len(newranker)): 51 | o = newranker[i] 52 | rankers[obj[i]].append(o) 53 | 54 | f = open(foutname, "w") 55 | 56 | line = "objects," 57 | for i in range(1,num_rankers+1): 58 | line += "ranker" + str(i) + "," 59 | f.write(line.strip(",") +"\n") 60 | 61 | for o in obj: 62 | line = str(o) + "," 63 | for val in rankers[o]: 64 | line += get_val(val) + "," 65 | f.write(line[:-1]+"\n") 66 | 67 | f.close() 68 | -------------------------------------------------------------------------------- /pagerank.py: -------------------------------------------------------------------------------- 1 | """ 2 | Pagerank utility to be used in aggregations 3 | Author: Sibel Adali 4 | 5 | Takes as input a graph, given in a dictionary of the form: 6 | 7 | links: key as nodes, a list of pairs of the form: [(outlink, weight),...] 8 | 9 | where weights are assumed to be normalized to add up to 1. 10 | 11 | """ 12 | 13 | def compute_scores(scores, links, jump_prob, alpha=0.85): 14 | new_scores = {} 15 | for item in links: 16 | new_scores[item] = (1-alpha) * jump_prob[item] 17 | for p in links: 18 | for (q,w) in links[p]: 19 | new_scores[q] += alpha*w*scores[p] 20 | return new_scores 21 | 22 | def compute_diff(score1,score2): 23 | diff = 0 24 | for item in score1: 25 | diff += abs( score1[item] - score2[item] ) 26 | return diff 27 | 28 | def pagerank(links, jump_prob={}, threshold=0.00001, alpha=0.85): 29 | num_nodes = float(len(links.keys())) 30 | 31 | if len(jump_prob.keys()) == 0: ##no random probability defined, use uniform 32 | for item in links.keys(): 33 | jump_prob[item] = 1/num_nodes 34 | 35 | scores = {} 36 | for item in links.keys(): 37 | scores[item] = jump_prob[item] 38 | 39 | iter = 0 40 | while (True): 41 | iter += 1 42 | new_scores = compute_scores(scores, links, jump_prob, alpha) 43 | diff = compute_diff(scores,new_scores) 44 | #print "Iteration %d (diff %.6f):" %(iter,diff) 45 | scores = new_scores 46 | if (diff < threshold) or (iter>100): 47 | break 48 | return scores 49 | -------------------------------------------------------------------------------- /rank_aggregators.py: -------------------------------------------------------------------------------- 1 | """ Simple rank aggregation program. 2 | Author: Sibel Adali 3 | 4 | Call using:: 5 | 6 | python rankers.py filename 7 | 8 | Input text file should be: 9 | * comma separated 10 | * contain a header row 11 | * first column is the object ids, assumed to be integers 12 | * each column is a separate ranker 13 | * if an object is not ranked by a ranker, leave that value empty. 14 | 15 | Data structures used: 16 | 17 | Objects is a dictionary where object names are key, and the ranks with 18 | respect to each ranker is a value (None if the ranker does not rank 19 | a given object) 20 | 21 | Example: Two rankers, 3 objects (a,b,c) 22 | {'a': [1,1,2], 'b':[2,None,1], 'c':[None,2,3]} 23 | 24 | A ranker is a single value version of this, for example 25 | {'a': 1, 'b': 2, 'c': 3} 26 | Rankers are assumed to be total, i.e. have a rank for each 27 | object. It is not necessary for ranks to be increasing order. Ties 28 | are possible, but are disregarded in processing. 29 | 30 | To Do: 31 | ----------- 32 | 1. Handle ties 33 | 34 | """ 35 | 36 | import sys 37 | import random 38 | import pagerank as pg 39 | import time 40 | import copy 41 | 42 | ################################################## 43 | ######### Input Output functions 44 | ################################################## 45 | def read_rankers(fname): 46 | f = open(fname) 47 | header = f.readline() 48 | ranker_names = header.strip().split(",") 49 | ranker_names = ranker_names[1:] 50 | objects = {} 51 | oid = 0 52 | for line in f: 53 | oid += 1 54 | m = line.strip().split(",") 55 | objects[oid] = m[1:] 56 | for i in range(len(objects[oid])): 57 | if len(objects[oid][i].strip()) == 0: 58 | objects[oid][i] = None 59 | else: 60 | objects[oid][i] = int(objects[oid][i]) 61 | return (objects, ranker_names) 62 | 63 | 64 | def print_rankers(objects, ranker_names): 65 | line = "Obj:\t" 66 | for name in ranker_names: 67 | line += name + "\t" 68 | print line 69 | for key in objects: 70 | line = "%d:\t" %key 71 | for val in objects[key]: 72 | if val == None: 73 | line += "None\t" 74 | else: 75 | line += "%d\t" %val 76 | print line 77 | 78 | def print_single_ranker(ranker): 79 | ranked = [] 80 | for key in ranker.keys(): 81 | ranked.append( (ranker[key], key) ) 82 | ranked.sort() ## low rank is good 83 | for (val, key) in ranked: 84 | print key, 85 | print 86 | 87 | ################################################## 88 | ######### Evaluation: Kendall tau 89 | ################################################## 90 | 91 | def kendall_tau(objects, cmp_ranker): 92 | agree = 0 93 | disagree = 0 94 | obj = objects.keys() 95 | n = len(obj) 96 | num_rankers = len(objects[obj[0]]) 97 | for i in range(n-1): 98 | for j in range(i+1,n): 99 | key1 = obj[i] 100 | key2 = obj[j] 101 | for ranker in range(len(objects[key1])): 102 | r11 = objects[key1][ranker] 103 | r12 = objects[key2][ranker] 104 | if r11 != None and r12 != None: 105 | r21 = cmp_ranker[key1] 106 | r22 = cmp_ranker[key2] 107 | if r21 != None and r22 != None: 108 | if r11 > r12: 109 | if r21 > r22: 110 | agree += 1 111 | else: 112 | disagree += 1 113 | elif r11 < r12: 114 | if r21 < r22: 115 | agree += 1 116 | else: 117 | disagree += 1 118 | 119 | return float(agree - disagree)/(0.5*n*(n-1)*num_rankers) 120 | 121 | 122 | def compare_two(objects, key1, key2): 123 | """Compares only a specific pair of objects for all the rankers. 124 | It is assumed that key1 is lower ranked than key2 in comparison. 125 | Hence, a flip is ordering key2 above key1. 126 | 127 | """ 128 | 129 | agree = 0 130 | disagree = 0 131 | for ranker in range(len(objects[key1])): 132 | r1 = objects[key1][ranker] 133 | r2 = objects[key2][ranker] 134 | if r1 != None and r2 != None: 135 | if r1 < r2: 136 | agree += 1 137 | elif r2 < r1: 138 | disagree += 1 139 | return agree, disagree 140 | 141 | def kendall_tau_partial(objects, ranker, oldscore, key1, key2): 142 | """Computes the change in kendall tau assuming the objects 143 | at key1 and key2 are being switched. It updates the old 144 | score and sends the new score 145 | 146 | """ 147 | 148 | if ranker[key1] > ranker[key2]: ##switch so that key1 is the smaller rank 149 | tmp = key1 150 | key1 = key2 151 | key2 = tmp 152 | 153 | 154 | a1,d1,a2,d2 = 0,0,0,0 155 | for obj in set(ranker.keys())-set([key1,key2]): 156 | if ranker[obj] > ranker[key1] and \ 157 | ranker[obj] < ranker[key2]: 158 | a,d = compare_two(objects, key1, obj) 159 | a1 += a 160 | d1 += d 161 | a,d = compare_two(objects, obj, key2) 162 | a2 += a 163 | d2 += d 164 | a3,d3 = compare_two(objects, key1, key2) 165 | 166 | n = len(objects.keys()) 167 | num_rankers = len(objects[key1]) 168 | multiplier = (0.5*n*(n-1)*num_rankers) 169 | newscore = (oldscore*multiplier + 2*(d1+d2+d3) - 2*(a1+a2+a3))/multiplier 170 | return newscore 171 | 172 | ################################################## 173 | ######### Util functions 174 | ################################################## 175 | 176 | def get_ranker(objlist): 177 | """ Convert a permutation of objects to a ranker dictionary """ 178 | ranker = {} 179 | for i in range(len(objlist)): 180 | key = objlist[i] 181 | ranker[key] = i+1 182 | return ranker 183 | 184 | def get_ranker_for_scores(scores): 185 | """ Convert a dictionary containing scores for each object to a ranker dictionary """ 186 | oscores = [] 187 | for key in scores: 188 | oscores.append( (scores[key], key) ) 189 | oscores.sort(reverse=True) 190 | objlist = [] 191 | for (score, key) in oscores: 192 | objlist.append(key) 193 | return( get_ranker(objlist) ) 194 | 195 | def num_higher(objects, key1, key2): 196 | """ Counts the number of times key2 is higher than key1 in rankers. """ 197 | count = 0 198 | for ranker in range(len(objects[key1])): 199 | x1 = objects[key1][ranker] 200 | x2 = objects[key2][ranker] 201 | if x1 != None and x2 != None and x1 > x2: 202 | count += 1 203 | return count 204 | 205 | def remove_ranker(objects, i): 206 | for key in objects: 207 | val = objects[key] 208 | objects[key] = val[:i] + val[i+1:] 209 | 210 | 211 | def switch(ranker, key1, key2): 212 | """ Switch the rankers given by the two keys """ 213 | r = ranker[key1] 214 | ranker[key1] = ranker[key2] 215 | ranker[key2] = r 216 | 217 | 218 | ################################################## 219 | ######### Rank aggregation functions 220 | ################################################## 221 | 222 | def best_random_aggregator(objects, tries, debug=False): 223 | """ Try random rankers given number of tries. Print debug 224 | info if debug is set to True. 225 | 226 | """ 227 | 228 | obj = objects.keys() 229 | ##initialize 230 | trial = obj[:] 231 | random.shuffle(trial) 232 | bestranker = get_ranker(trial) #best so far 233 | bestscore = kendall_tau(objects, bestranker) 234 | 235 | if debug: 236 | print bestranker, bestscore 237 | 238 | ## try random choices 239 | for i in range(tries): 240 | trial = obj[:] 241 | random.shuffle(trial) 242 | ranker = get_ranker(trial) 243 | score = kendall_tau(objects, ranker) 244 | if score > bestscore: 245 | bestscore = score 246 | bestranker = ranker 247 | if debug: 248 | print "changed rankers", bestranker, bestscore 249 | 250 | return bestranker, bestscore 251 | 252 | 253 | def pagerank_aggregator(objects, threshold, alpha): 254 | """Implements the pagerank aggregation for a given alpha and epsilon. 255 | Alpha is for the bias towards surf probability, non-random in this case. 256 | Epsilon controls the convergence threshold, a small number in practice. 257 | 258 | The program first constructs the graph given the rankers and then calls 259 | pagerank function repeatedly until the average change in pagerank scores 260 | is below epsion. Then, it converts the resulting scores into a ranking. 261 | 262 | """ 263 | 264 | ### Construct graph version of the rankers and update indegrees 265 | indegrees = {} 266 | total_indegrees = 0.0 267 | graph = {} 268 | for key in objects.keys(): 269 | graph[key] = [] 270 | indegrees[key] = 0.0 271 | 272 | for key1 in objects.keys(): 273 | for key2 in objects.keys(): 274 | if key1 != key2: 275 | count = num_higher(objects, key1, key2) 276 | if count > 0: 277 | graph[key1].append( (key2,float(count)) ) 278 | indegrees[key2] += count 279 | total_indegrees += count 280 | 281 | 282 | ##Normalize to add the outlinks to 1 283 | for key1 in graph.keys(): 284 | total = 0 285 | for (key2,val) in graph[key1]: 286 | total += val 287 | for i in range(len(graph[key1])): 288 | (key2, val) = graph[key1][i] 289 | graph[key1][i] = (key2, val/total) 290 | 291 | ##Normalize indegrees as well 292 | for key in indegrees.keys(): 293 | indegrees[key] /= total_indegrees 294 | 295 | 296 | ### Call page rank 297 | final_scores = pg.pagerank(graph, indegrees, threshold, alpha) 298 | 299 | #final_scores = pg.pagerank(graph, {}, threshold, alpha) 300 | 301 | ### Convert the final scores to a ranking 302 | ranker = get_ranker_for_scores(final_scores) 303 | 304 | rankscore = kendall_tau(objects, ranker) 305 | return ranker, rankscore 306 | 307 | 308 | def indegree_aggregator(objects): 309 | """Returns a simple indegree aggregation based on the number of rankers that rank the 310 | given object higher than the rest. 311 | """ 312 | 313 | ### Construct graph version of the rankers and update indegrees 314 | indegrees = {} 315 | 316 | for key in objects.keys(): 317 | indegrees[key] = 0.0 318 | 319 | for key1 in objects.keys(): 320 | for key2 in objects.keys(): 321 | if key1 != key2: 322 | count = num_higher(objects, key1, key2) 323 | if count > 0: 324 | indegrees[key2] += count 325 | 326 | ### Convert the indegree scores to a ranking 327 | ranker = get_ranker_for_scores(indegrees) 328 | 329 | rankscore = kendall_tau(objects, ranker) 330 | return ranker, rankscore 331 | 332 | ################################################## 333 | ######### Iterative flip algorithms 334 | ################################################## 335 | 336 | def iterative_greedy_flip(objects, inputranker, k=1): 337 | """ Flip a pair of objects in ranker until k total passes are 338 | done or no improvements are possible. 339 | 340 | """ 341 | 342 | allkeys = objects.keys() 343 | 344 | pairs = [] 345 | for i in range(len(allkeys)-1): 346 | for j in range(i+1,len(allkeys)): 347 | pairs.append( (allkeys[i], allkeys[j]) ) 348 | 349 | ranker = copy.deepcopy(inputranker) ##copy we will work with 350 | currentscore = kendall_tau(objects, ranker) 351 | total_flips = 0 352 | iter = 0 353 | while (iter < k): 354 | iter += 1 355 | flip_done = False 356 | random.shuffle(pairs) 357 | ##One pass, try all pairs in pairs and check if flipping 358 | ##the given pair of objects improves the error in ranker 359 | for i in range(len(pairs)): 360 | key1, key2 = pairs[i] 361 | ##switch key1 and key2 362 | newscore = kendall_tau_partial(objects, ranker, currentscore, key1, key2) 363 | switch(ranker, key1, key2) 364 | if newscore > currentscore: 365 | flip_done = True 366 | total_flips += 1 367 | currentscore = newscore 368 | else: ## reverse the switch 369 | switch(ranker, key1, key2) 370 | if not flip_done: 371 | break 372 | return ranker, currentscore, total_flips 373 | 374 | 375 | def iterative_best_flip(objects, inputranker): 376 | """Flip a pair of objects in ranker regardless of whether it improves, then perform 377 | all other possible flips if they improve performance and record the output. 378 | 379 | Continue for all possible pairs, and return the best performance 380 | over all such configurations. 381 | 382 | """ 383 | 384 | allkeys = objects.keys() 385 | 386 | pairs = [] 387 | for i in range(len(allkeys)-1): 388 | for j in range(i+1,len(allkeys)): 389 | pairs.append( (allkeys[i], allkeys[j]) ) 390 | random.shuffle(pairs) 391 | 392 | ranker = copy.deepcopy(inputranker) ##copy we will work with 393 | currentscore = kendall_tau(objects, ranker) 394 | total_flips = 0 395 | configs = [] 396 | iter = 0 397 | max_score = currentscore 398 | max_ranker = inputranker 399 | 400 | for i in range(len(pairs)): 401 | key1,key2 = pairs[i] ##current pair being flipped 402 | 403 | iter_ranker = copy.deepcopy(ranker) 404 | iter_score = kendall_tau_partial(objects, ranker, currentscore, key1, key2) 405 | switch(iter_ranker, key1, key2) 406 | 407 | ##One pass, try all pairs in pairs and check if flipping 408 | ##the given pair of objects improves the error in ranker 409 | for j in range(len(pairs)): 410 | if i == j: 411 | continue 412 | key1, key2 = pairs[j] 413 | ##switch key1 and key2 414 | newscore = kendall_tau_partial(objects, ranker, iter_score, key1, key2) 415 | switch(iter_ranker, key1, key2) 416 | if newscore > iter_score: 417 | iter_score = newscore 418 | else: ## reverse the switch 419 | switch(iter_ranker, key1, key2) 420 | if iter_score > max_score: 421 | max_score = iter_score 422 | max_ranker = iter_ranker 423 | 424 | return max_ranker, max_score 425 | 426 | 427 | ################################################## 428 | ######### Function to call aggregators while 429 | ######### removing rankers to manage errors 430 | ################################################## 431 | 432 | def remove_top_k(objects, ranker_names, nullranker, k): 433 | """ Removes up to k rankers until the error of using the 434 | input aggregator improves. 435 | 436 | """ 437 | 438 | nullscore = kendall_tau(objects, nullranker) ##initial score 439 | localobjects = copy.deepcopy(objects) ##must not change the original set 440 | names = ranker_names[:] ##local copy of ranker names 441 | 442 | iter = 0 443 | removed = [] 444 | 445 | while (iter 0: 465 | to_remove = performance[0][1] 466 | remove_ranker(localobjects, to_remove) 467 | 468 | ##get the improved new score and record performance improvement 469 | nullranker, nullscore = pagerank_aggregator(localobjects, 0.000001, 0.95) 470 | removed.append( names[to_remove] ) 471 | names = names[:to_remove]+names[to_remove+1:] 472 | else: 473 | break ##no further improvements 474 | 475 | return nullranker, nullscore, removed, localobjects 476 | 477 | ################################################## 478 | ######### Main body of the code 479 | ################################################## 480 | 481 | if __name__ == "__main__": 482 | debug = False 483 | if len(sys.argv)<2: 484 | print "Usage python rankers.py filename" 485 | else: 486 | 487 | start = time.time() 488 | fname = sys.argv[1] 489 | (objects,ranker_names) = read_rankers(fname) 490 | #print_rankers(objects, ranker_names) 491 | 492 | ranker, score = indegree_aggregator(objects) 493 | print "Indegree", score 494 | print_single_ranker(ranker) 495 | 496 | ranker, score = pagerank_aggregator(objects, 0.000001, 0.85) 497 | print "Pagerank:", score 498 | print_single_ranker(ranker) 499 | 500 | ### You can also try random aggregation, but not advisable for large data 501 | ### Unless you run many trials, this will not work well 502 | #ranker, score = best_random_aggregator(objects, 10, debug) 503 | #print "Best random:", score 504 | #print_single_ranker(ranker) 505 | 506 | ranker2, score, total_flips = iterative_greedy_flip(objects, ranker, 5) 507 | print "Iterative greedy flip k=5 using pagerank:", score 508 | print_single_ranker(ranker2) 509 | print "Total number of flips", total_flips 510 | 511 | ranker1, score, removed, newobjects = remove_top_k(objects, ranker_names, ranker, 5) 512 | print "Iterative remove with pagerank:", score 513 | print_single_ranker(ranker1) 514 | print "Removed", removed 515 | 516 | ranker3, score, total_flips = iterative_greedy_flip(newobjects, ranker1, 5) 517 | print "Iterative greedy flip after removal:", score 518 | print_single_ranker(ranker3) 519 | print "Total number of flips", total_flips 520 | 521 | ranker3, score = iterative_best_flip(objects, ranker) 522 | print "Iterative best flip using pagerank:", score 523 | print_single_ranker(ranker3) 524 | 525 | end = time.time() 526 | print "Took", end-start, "seconds" 527 | 528 | -------------------------------------------------------------------------------- /test/data.csv: -------------------------------------------------------------------------------- 1 | Objects,Ranker1,Ranker2,Ranker3,Ranker4 2 | 1,,,1,3 3 | 2,,1,2, 4 | 3,5,,7,4 5 | 4,1,2,, 6 | 5,,,,5 7 | 6,,3,4,1 8 | 7,2,4,3,2 9 | 8,,5,, 10 | 9,3,,5, 11 | 10,4,6,6, -------------------------------------------------------------------------------- /test/data10_10.csv: -------------------------------------------------------------------------------- 1 | objects,ranker1,ranker2,ranker3,ranker4,ranker5,ranker6,ranker7,ranker8,ranker9,ranker10 2 | 1,3,,,1,,4,,1,2, 3 | 2,2,1,5,6,7,,5,3,1,3 4 | 3,,,,7,,,1,,, 5 | 4,4,6,1,,4,1,,6,3,2 6 | 5,,,2,,5,5,,5,, 7 | 6,1,2,6,5,2,3,3,7,,4 8 | 7,,,4,3,6,6,,,6,6 9 | 8,5,3,,,,,2,,7,7 10 | 9,,4,3,2,1,2,4,4,5,5 11 | 10,,5,,4,3,,,2,4,1 12 | -------------------------------------------------------------------------------- /test/data20_40.csv: -------------------------------------------------------------------------------- 1 | objects,ranker1,ranker2,ranker3,ranker4,ranker5,ranker6,ranker7,ranker8,ranker9,ranker10,ranker11,ranker12,ranker13,ranker14,ranker15,ranker16,ranker17,ranker18,ranker19,ranker20,ranker21,ranker22,ranker23,ranker24,ranker25,ranker26,ranker27,ranker28,ranker29,ranker30,ranker31,ranker32,ranker33,ranker34,ranker35,ranker36,ranker37,ranker38,ranker39,ranker40 2 | 1,10,8,6,,9,,,1,5,,,11,,3,12,9,,,,,9,3,2,5,10,6,4,,,11,7,1,7,11,,,8,8,10, 3 | 2,8,2,,,6,6,,,7,9,1,6,,11,,2,,5,,,,,,1,7,,,,,,8,,5,10,10,,,,, 4 | 3,,,4,,,8,,2,,6,,15,7,6,11,,2,1,3,4,,7,5,11,8,3,13,7,,7,3,7,6,9,1,9,5,,,5 5 | 4,,6,10,,1,,7,,6,3,12,7,3,,,,3,,9,,,11,7,2,5,10,7,4,9,8,1,2,13,1,,5,,,6,4 6 | 5,,,2,5,3,,8,3,11,8,5,,2,,,,5,,5,10,12,5,,3,15,11,5,3,2,,,,,,7,7,9,7,, 7 | 6,,4,,,,,12,4,,,9,1,9,5,7,3,6,6,7,7,5,2,,6,11,2,8,12,10,,,,10,,8,1,,,8,7 8 | 7,7,13,,2,10,,11,,4,1,14,,,7,,10,7,,11,3,4,10,6,12,14,4,9,14,4,9,9,10,2,6,,10,10,4,5, 9 | 8,5,9,,8,,,,8,,4,8,12,,,3,13,,,4,9,3,,,15,,8,,1,11,4,,11,3,,5,11,1,,,3 10 | 9,,,,10,,,6,7,3,,13,9,10,,10,7,,,,,8,14,,,,,,11,5,,,,14,3,6,14,,3,, 11 | 10,11,7,,3,,,13,10,1,,,4,11,,,,,7,,,2,13,9,7,1,5,12,9,12,5,6,5,,7,,4,,,7, 12 | 11,,,7,1,7,10,4,,,7,11,,8,,,6,,3,,5,,1,1,,12,,10,,,1,,9,,,,3,4,,1, 13 | 12,12,,1,,,1,5,,12,5,4,14,5,,6,5,8,2,12,1,10,,,14,13,1,14,,,,2,6,,4,,,,5,3,6 14 | 13,,12,,6,11,7,,,,10,6,2,6,,,,,,2,,,,11,9,4,12,15,10,6,3,,3,4,5,,,2,10,, 15 | 14,4,11,8,,13,9,,,8,2,,,4,10,1,1,9,,,,,12,4,8,9,,2,,3,,,13,,,4,8,7,,2,9 16 | 15,,5,3,,2,5,10,,2,,,5,,,8,8,,,1,,13,,,,3,,,2,8,6,,,12,2,2,,6,6,12,10 17 | 16,2,1,9,9,5,,14,,9,,3,13,,1,5,,,10,10,,,,12,,,,,5,,,,12,11,8,,13,11,2,, 18 | 17,9,,5,7,12,2,9,,,,7,3,1,4,,,1,8,6,2,11,8,8,4,6,7,6,8,7,,4,14,9,,,6,,,4,2 19 | 18,6,,11,,8,4,1,9,,,15,8,,8,9,11,10,,,6,6,6,3,13,,9,11,,1,2,10,8,8,,11,,3,1,11,8 20 | 19,3,3,,11,4,,3,5,10,11,10,,,2,4,4,4,9,8,,7,4,13,10,2,13,1,6,,10,5,4,1,,3,2,,9,9, 21 | 20,1,10,,4,,3,2,6,,,2,10,,9,2,12,,4,,8,1,9,10,,,,3,13,,12,,,,12,9,12,,,,1 22 | -------------------------------------------------------------------------------- /test/data50_150.csv: -------------------------------------------------------------------------------- 1 | objects,ranker1,ranker2,ranker3,ranker4,ranker5,ranker6,ranker7,ranker8,ranker9,ranker10,ranker11,ranker12,ranker13,ranker14,ranker15,ranker16,ranker17,ranker18,ranker19,ranker20,ranker21,ranker22,ranker23,ranker24,ranker25,ranker26,ranker27,ranker28,ranker29,ranker30,ranker31,ranker32,ranker33,ranker34,ranker35,ranker36,ranker37,ranker38,ranker39,ranker40,ranker41,ranker42,ranker43,ranker44,ranker45,ranker46,ranker47,ranker48,ranker49,ranker50,ranker51,ranker52,ranker53,ranker54,ranker55,ranker56,ranker57,ranker58,ranker59,ranker60,ranker61,ranker62,ranker63,ranker64,ranker65,ranker66,ranker67,ranker68,ranker69,ranker70,ranker71,ranker72,ranker73,ranker74,ranker75,ranker76,ranker77,ranker78,ranker79,ranker80,ranker81,ranker82,ranker83,ranker84,ranker85,ranker86,ranker87,ranker88,ranker89,ranker90,ranker91,ranker92,ranker93,ranker94,ranker95,ranker96,ranker97,ranker98,ranker99,ranker100,ranker101,ranker102,ranker103,ranker104,ranker105,ranker106,ranker107,ranker108,ranker109,ranker110,ranker111,ranker112,ranker113,ranker114,ranker115,ranker116,ranker117,ranker118,ranker119,ranker120,ranker121,ranker122,ranker123,ranker124,ranker125,ranker126,ranker127,ranker128,ranker129,ranker130,ranker131,ranker132,ranker133,ranker134,ranker135,ranker136,ranker137,ranker138,ranker139,ranker140,ranker141,ranker142,ranker143,ranker144,ranker145,ranker146,ranker147,ranker148,ranker149,ranker150 2 | 1,32,,26,31,,,,,,6,26,11,,22,16,,3,,25,,,,13,30,34,,21,,,3,5,6,5,,9,7,10,24,,,,7,5,15,22,9,,26,11,25,19,31,33,,24,3,3,32,36,16,,,14,,4,24,9,,3,33,25,19,20,6,,,23,,16,,1,,30,,,29,33,10,17,8,19,28,11,28,,12,9,,29,13,21,5,17,15,7,19,21,,,,18,7,15,28,19,,,1,14,27,9,17,,27,,3,14,15,,,29,,,,11,,22,,24,,,23,3,3,,1,5,25,2, 3 | 2,6,24,32,18,,,9,,,33,,36,26,4,34,,5,20,10,,,25,4,7,22,22,,,8,,24,1,7,21,22,6,,6,,,32,18,27,4,24,23,9,9,,31,22,,,24,,2,,28,35,,29,,13,,19,12,3,,,4,,34,3,,11,,8,35,28,25,2,,,15,18,30,24,,,2,1,15,,,9,,23,,26,32,,6,22,,3,,,26,,32,19,,,,,26,5,,7,,26,19,8,4,22,32,,10,8,3,11,7,,36,32,27,17,,25,4,,3,18,20,,,,22,, 4 | 3,30,,27,21,30,10,16,26,29,14,11,15,9,25,,13,9,4,,,7,4,3,8,,10,,,,25,3,,,5,,17,33,,12,7,,,8,12,13,10,,30,4,16,9,25,19,20,13,24,2,26,,1,,16,,13,17,3,26,18,18,,19,9,26,25,6,,5,18,2,8,31,23,,23,21,,32,,7,14,,,,11,23,,,26,31,,,34,,,,18,5,30,,23,2,10,,22,21,14,1,7,,32,,20,23,,17,35,,13,,27,,20,7,20,,24,26,16,4,,27,27,13,18,18,,22,,,3 5 | 4,15,11,,,,,,,23,32,,35,,,9,,19,,,21,5,,,,33,32,26,13,21,,6,21,9,,,29,,,,,24,,28,30,4,,,6,2,2,,5,,,,19,10,15,,,24,,,29,10,,15,10,,,,,27,,23,16,17,14,26,3,,,9,36,1,4,,,23,11,,16,26,23,2,25,22,1,6,28,27,,3,,2,,6,33,4,8,27,,,9,9,15,,,20,,,,,,7,16,9,24,,2,9,,,5,19,3,,26,,8,7,21,,,5,,24,1,, 6 | 5,14,22,,32,,,4,24,11,,22,14,29,,12,,,,7,8,,8,,,23,9,11,24,11,,7,,12,,36,,9,22,4,22,11,,,21,,27,29,28,,1,,,21,,7,20,14,,,26,,,,,3,19,,,,1,35,31,,,5,,,36,31,9,9,27,26,22,,19,21,1,,21,6,,15,,15,29,,,,,22,3,11,,,1,27,,11,,30,2,,19,27,21,9,15,33,,11,9,,19,15,22,8,20,,33,13,18,15,26,,,9,30,,28,9,13,23,,12,,6,,20,17 7 | 6,,9,,30,2,18,,,21,,6,29,34,23,4,,6,18,,16,,10,12,3,27,14,1,9,,26,,,3,,28,11,,,,,,6,18,,11,,27,19,18,,23,3,,,4,,,,27,23,,17,25,,,33,,11,,11,26,1,,10,9,11,29,22,6,13,3,1,28,33,,18,,,8,6,14,22,6,22,6,11,,2,28,15,,31,15,,1,,33,25,,31,,,,31,,12,15,6,22,14,10,,,18,25,11,,,24,14,,8,12,37,15,11,,22,23,12,,28,34,19,2,,,,,22 8 | 7,,,1,,6,,,,,5,16,12,27,7,32,28,23,,3,,,19,,27,,,3,11,,,15,,,,13,2,,14,,12,,26,17,25,15,,15,24,30,23,,15,,22,,,,19,20,,11,26,,4,,,,27,32,,36,10,,20,14,,14,28,29,24,,18,17,14,5,,,,14,,7,,14,24,,10,4,,11,,,1,,14,19,6,19,13,,,26,,20,,,6,24,2,37,13,17,,13,1,,27,1,,,,22,,16,29,,33,,14,7,23,8,,,,32,31,8,13,5,7 9 | 8,13,4,5,15,,,12,19,,3,5,17,14,,30,,,,,,,,26,31,6,17,5,,,,17,4,20,4,8,15,,9,9,23,15,1,12,18,6,17,26,8,17,9,1,29,,,,16,,,37,10,,13,5,15,,,,,10,31,,,,,19,7,13,13,8,10,,,11,19,28,27,,,,,10,3,1,,22,17,1,,7,3,12,,29,,23,,18,,,,32,,,30,25,,6,13,31,31,7,11,,26,,20,16,,14,,5,,13,27,20,16,3,28,26,,,5,12,16,20,17,,9,15,35 10 | 9,,18,14,,12,20,,7,,15,20,22,36,,10,,22,24,,26,18,,1,14,16,,,15,14,,,15,,,24,,,3,10,,18,28,,,21,,28,,,,7,,6,,5,,,2,15,29,27,,29,12,22,11,,,15,,20,23,10,,13,5,9,,5,12,37,20,,27,,17,26,15,,1,31,,9,27,,15,3,13,,19,6,27,,,,5,1,11,27,27,10,,,,29,,,,1,36,,15,,,18,29,,22,2,6,10,,5,,8,,,3,17,14,15,17,,,28,12,10,12,22, 11 | 10,,,7,11,22,24,,15,5,13,2,,8,,35,,14,8,,17,,,18,6,24,,29,23,26,17,19,,,20,5,,,,,21,,,,19,2,32,6,17,3,17,15,,25,,23,5,26,3,32,8,31,15,16,16,13,,10,31,,19,,33,14,11,,,3,1,25,15,,,10,,22,28,5,20,,,,,5,,,5,31,12,,18,24,32,9,,,,,9,,7,4,13,,25,,19,19,28,15,2,6,,,,16,,,,,,12,1,,2,30,,,4,,16,12,14,32,26,8,,,30,18,14 12 | 11,,,33,,9,19,,16,25,31,21,31,,,,,34,16,22,9,25,,,,,12,,,,21,10,11,,,37,1,8,,17,,,27,,37,,8,19,12,8,,,9,5,,10,8,27,16,24,30,13,23,21,,,,31,8,14,23,24,,,12,,9,11,,15,,24,,,,23,6,13,14,16,27,21,,,2,,,17,27,17,23,,11,,10,,4,32,21,22,35,6,,12,23,,1,10,11,3,25,28,3,,,,6,,16,1,29,,24,,,,,21,13,2,31,,25,27,32,14,6,19,2,,32 13 | 12,31,,6,,14,,,10,12,22,1,,,17,5,10,8,25,21,23,23,24,22,32,,36,24,,22,,9,13,,9,18,23,17,7,,6,23,4,16,27,19,6,,16,24,,,2,29,10,21,13,,14,31,4,1,19,26,,,17,5,28,17,18,17,,,,,,20,27,,11,,,,,10,,17,24,3,,,,,12,,,,,30,,5,,,6,16,22,31,12,17,12,33,20,7,,13,10,,21,17,9,,22,,,26,,20,3,13,,28,25,19,32,,12,,25,8,20,10,6,14,12,34,,7,,4,26 14 | 13,1,3,,16,26,23,,35,,,29,4,23,18,,4,,,13,30,13,6,,25,,13,4,,15,5,,,,6,,,25,17,,4,,,15,22,20,21,22,36,29,,,,31,5,,4,,,4,,,9,3,18,25,7,,14,,2,3,11,13,,15,20,,12,27,1,,,5,8,,,2,23,20,3,,,3,26,4,,20,28,5,,,,,1,13,3,7,31,,,,23,8,24,23,35,,4,30,,27,18,3,,12,14,15,18,,16,7,,,9,12,,15,8,,,,15,25,22,24,,14,,,13 15 | 14,33,25,,14,25,26,,6,4,23,,37,15,,,,18,9,,22,11,9,5,18,,,,,18,,,23,,,20,,,,7,8,,20,,,,1,21,,21,,20,21,,18,,,,35,1,,20,,11,,6,29,,23,20,9,,8,12,,,14,27,15,30,16,,,12,20,6,,3,,22,28,,,,3,19,,,5,16,,,,,8,15,7,,,26,33,11,6,,,,31,26,20,,,21,,21,,32,26,23,5,3,9,24,27,,12,,4,8,18,6,22,,,31,15,,10,,,, 16 | 15,,,9,10,,3,14,,26,,,26,20,8,18,,,6,,29,10,3,,4,18,30,,27,,10,,9,,3,6,26,20,13,,,,,,17,,26,,18,9,4,18,,22,,,18,,,17,,30,22,20,,,10,12,9,26,24,,13,,,,2,,23,1,4,28,,,,,9,,2,24,,,13,16,,17,,30,25,,,13,17,2,21,28,27,35,5,,26,14,16,11,20,3,24,,,,,13,10,,12,30,36,,,,23,,23,25,,1,1,,19,,,,,10,25,19,5,,33,23, 17 | 16,22,27,,9,18,2,1,,10,29,,28,12,,,,25,29,2,7,15,,8,2,13,29,,,4,7,2,14,14,13,,8,,10,,13,27,23,26,,,16,,,32,,,,15,12,30,,17,,11,25,2,,9,23,27,25,,,,21,,28,19,,3,,22,32,11,,27,5,31,25,8,,22,,2,,9,25,13,34,,,5,4,15,5,26,8,,,18,,9,3,19,,16,,16,,10,3,,24,18,30,8,5,20,,33,,19,,,34,15,,,22,,34,,,30,,,26,5,9,13,3,18,32,,27 18 | 17,,15,,,,17,,13,24,7,24,30,,,25,1,17,,9,24,,23,,28,9,27,12,25,,,1,20,17,14,1,21,22,,18,15,30,,,23,9,33,5,5,,,,10,27,,25,15,,36,14,19,23,1,,,8,,23,6,25,,23,30,21,16,2,,6,19,33,,7,25,4,18,,,,7,,,23,23,25,,11,,12,3,,9,,,,25,17,2,,34,5,22,17,9,,7,,,,,9,,5,,,5,4,31,3,,7,,,29,,15,,19,,,20,,6,,8,,36,,12,3,17,2 19 | 18,18,,11,6,,,8,,,,15,,,6,6,22,26,11,,2,20,15,23,24,12,31,,,25,16,8,,23,18,32,,,,,,13,22,7,35,,25,,22,27,,,8,,2,1,10,,18,,,,21,,2,26,15,16,,19,32,2,27,28,,,,,21,21,,,28,22,,,,11,,19,,,,19,21,,13,,32,8,10,,7,12,,,,14,32,25,,,,,8,,32,14,,,33,31,,,11,,1,,21,26,8,14,,2,,14,30,7,21,19,17,,18,17,31,,,34,,,24 20 | 19,,20,,,10,,,28,31,4,10,2,,5,31,3,,,,31,,,9,13,19,19,14,4,7,,,,18,11,35,,32,,,26,3,13,,32,,,10,20,15,,,23,,,,26,22,11,34,6,18,,17,,28,23,22,,,22,,17,,8,16,21,26,2,12,,20,14,14,35,13,7,19,,,9,17,11,18,,,1,,34,32,1,17,20,7,,12,15,,24,,3,,,14,,17,,13,,36,17,19,4,,,29,,,,,13,,,27,24,18,15,20,23,22,3,24,,,5,,,13,28,,8 21 | 20,,2,31,23,8,5,,2,14,9,27,20,6,14,36,18,16,17,27,13,,5,,,25,21,23,19,,,,16,,,,3,18,2,22,,6,,4,9,,31,14,34,,21,,12,28,,19,,,33,3,,,7,,,,22,,13,,,8,6,22,14,26,13,24,,,7,13,21,16,,7,22,36,5,,23,20,7,,10,16,21,21,14,23,16,14,,4,,5,8,,1,23,19,,24,13,,18,,,,,6,29,26,24,7,13,,,8,,10,,15,17,4,,17,2,2,,19,1,32,20,2,26,,25,,16,20 22 | 21,11,,18,5,,11,25,8,7,,28,3,,,,23,20,28,24,4,,,6,15,,5,13,2,,27,25,5,24,,31,14,31,,5,,,11,,,18,,8,4,,,,,30,,11,,,4,,,,11,8,20,,21,33,5,4,29,4,,,17,17,,18,8,4,,15,6,,1,27,13,,12,,12,24,12,,19,7,14,,10,22,,23,,,2,,16,30,20,12,,,17,26,29,2,34,18,14,5,8,1,,7,14,2,23,,,11,15,23,3,18,33,25,10,24,12,,7,23,16,33,21,10,,,6,13, 23 | 22,21,10,,1,13,,,34,,30,,10,1,,17,17,,23,,11,,13,,9,29,8,22,7,24,1,26,,25,23,14,,26,21,6,,,2,21,,7,22,30,2,20,5,21,32,18,3,31,,28,22,6,27,5,14,,,,,4,25,6,6,,,9,21,,4,,29,24,,,29,29,4,20,,25,26,,15,13,5,,15,25,27,33,35,,33,,29,,,,,3,7,24,21,7,14,,16,,17,27,,26,19,14,,19,13,23,21,,6,,,,,10,28,6,18,,24,,,,4,4,,,,1,10,14,12 24 | 23,,21,10,4,3,15,,12,22,28,18,18,33,,33,15,32,27,,32,14,1,,,,,6,14,,13,,,6,,,22,,,,3,5,19,13,13,12,,17,,7,,3,17,23,,18,6,23,5,22,9,,10,,,,,28,21,21,30,21,,,3,18,,1,,9,,,19,,10,,1,,19,,24,25,,10,5,3,,14,11,2,6,9,,18,23,22,21,22,,9,,,,,11,,,25,8,4,15,,,,,28,15,5,25,,,,14,1,14,26,,10,,31,,14,22,30,,25,2,,21,9,6 25 | 24,5,,22,3,7,4,24,,3,,12,,,24,1,7,15,,18,5,,,14,33,,,32,,2,20,13,,,,16,32,7,16,2,,29,,25,,8,,2,21,,,,,1,,8,,15,34,,21,19,,4,,23,32,7,,,3,10,21,17,18,,15,12,24,23,5,33,15,8,,,2,1,18,1,,29,8,24,8,24,,26,33,24,26,,22,8,16,,24,,4,1,,25,3,25,6,,2,,,12,,,,,,20,13,,,23,,8,,30,7,3,,4,5,16,1,28,,,14,27,4,,18,,18 26 | 25,9,14,,24,,,,,,,,7,28,,15,12,28,,14,,1,14,27,17,17,20,8,,3,19,20,,,7,2,,11,,1,11,,,,6,,14,3,7,,18,,16,35,19,,,5,17,,,,,31,22,14,20,20,20,22,,,4,2,,,,,,,6,14,13,,29,,,35,25,,,30,,4,14,8,9,19,16,,20,,12,23,7,6,,8,28,,14,28,,19,4,,33,4,,29,22,23,,,23,14,,,,,32,2,26,24,,7,26,5,,,,,11,,,,24,30,7,,19 27 | 26,26,28,,,23,21,5,5,,,,27,32,1,,16,37,10,,,,27,24,,4,1,18,8,,11,12,,,19,,18,19,,16,,,16,,,,3,24,,,,13,,10,,,7,24,8,2,5,,,22,3,15,13,18,,,26,7,15,,28,,,,,20,27,12,2,20,,,25,,,10,,,,,18,13,22,7,19,20,22,,14,14,12,25,,12,29,15,20,15,4,29,27,6,30,23,12,6,18,,,25,6,,,21,12,19,,26,,11,13,,29,25,33,21,10,22,20,1,,6,15,,17,25, 28 | 27,20,13,17,28,,22,3,20,19,21,,34,4,,,,1,12,23,1,21,22,,,10,6,16,,,,11,,,,33,16,6,,24,,19,,29,14,25,2,,3,1,7,8,,7,,,14,,12,,17,14,5,,21,1,,,,27,8,33,,,,,,10,4,32,,32,,33,7,2,26,34,21,,,,20,,,18,,10,20,3,,,,1,5,4,,2,,,2,5,,27,10,1,4,11,,32,20,,,9,8,9,,17,4,12,28,18,4,,25,,22,14,,5,27,,33,,8,21,,15,36,28,11 29 | 28,,8,30,22,,14,23,9,13,26,25,,3,,23,19,,,,10,22,28,28,19,2,37,20,26,,,,2,,24,25,,4,,11,24,26,12,,2,,35,,,10,26,25,,20,6,,,16,,25,,,,,26,2,2,6,2,,27,28,12,,,,,30,3,,22,29,8,13,28,,14,23,22,,,,27,2,17,,4,28,7,14,8,18,,,27,29,,24,35,7,16,,,22,1,8,11,,23,8,16,,,6,,,4,,,18,,6,5,8,1,13,,27,29,,,17,7,6,29,,22,9,26,10, 30 | 29,,23,29,,17,,11,4,16,19,,,,11,2,,35,13,,12,17,7,,1,,26,,,,,,19,,,11,24,,,3,,14,,,8,,24,,31,,29,12,,,15,,1,25,27,,13,,27,18,7,,,25,17,29,,12,5,,13,,6,16,26,3,,35,30,,,,20,6,,,20,,29,,,,,15,21,,25,8,33,,24,9,,25,2,,,29,18,10,,16,,21,,35,23,,1,14,2,,34,6,,17,,,16,29,35,2,,12,,10,,,35,11,23,29,,,19,,31 31 | 30,27,5,,,,,17,3,,11,,32,,,,27,10,3,20,,6,2,,,,25,25,17,19,,,,21,,,,,27,,18,4,,14,20,16,5,,,13,11,,11,,,29,,21,,23,2,3,,,32,5,,11,,,,1,25,24,,20,,,6,10,,8,24,,26,14,24,,,,,32,,,,,2,,36,13,,1,,21,18,,23,,,,1,36,,17,,12,,2,,19,,2,,,28,,,,1,15,4,1,30,26,,,,,,,26,,30,16,,7,,29,,27,1 32 | 31,,26,36,,21,,13,11,,16,4,,11,10,24,,,,17,,,,,,,4,9,6,1,22,23,25,,,30,13,24,26,8,,,25,,,3,11,,33,23,3,,,32,,26,,18,,30,15,26,,19,,12,26,19,26,24,5,13,20,,,,,25,,,14,17,,,2,4,,27,,9,16,,,,20,,,32,31,19,35,,18,28,9,,,17,,8,30,,5,24,14,,13,,,,35,22,,,9,3,33,,,,31,,13,,,29,21,29,31,,24,2,,,,,23,26,29,19,33 33 | 32,12,6,19,26,20,,18,,27,24,23,6,,9,,26,11,22,19,33,,,,29,11,,17,,23,6,18,24,8,16,3,9,,,,,17,17,3,11,1,,34,,26,30,16,20,36,7,,22,11,30,28,31,6,,28,,,5,,,30,,,16,1,,21,19,21,25,,21,,7,19,34,3,,37,13,25,18,,,29,,1,28,,17,,,,24,5,11,27,,28,15,,28,12,,21,,,29,17,22,34,,12,23,,,19,10,,,,22,16,,,,23,6,6,7,,5,19,,,,37,,,27,8,29 34 | 33,16,17,13,8,5,6,6,33,,,14,8,17,13,,8,13,14,8,,,,15,,35,,7,,5,9,22,,,17,,,27,25,,1,20,10,19,36,,7,23,,,,17,27,12,13,20,,8,21,21,,8,,,17,,4,29,24,,14,5,2,,26,24,,,5,17,,16,,,24,26,,10,,11,25,2,,,13,20,8,,,,,,4,,3,20,10,34,19,21,9,31,,3,15,26,25,7,18,21,24,20,,16,,,30,12,,4,,17,28,9,34,22,8,,17,,29,13,,29,30,16,13,31,5,,34 35 | 34,,,,20,,,,,1,,,1,35,26,21,,27,,,20,,18,7,,15,7,,18,10,,,10,15,22,26,20,23,19,,9,10,21,11,16,,18,16,27,,,4,,9,9,28,,,,13,24,15,12,12,9,20,9,17,,7,16,6,22,7,9,1,,,31,,,36,26,6,16,19,16,7,4,,,4,17,,,10,23,,6,,17,,13,30,,30,17,10,,,,8,,28,,11,18,8,25,,34,,,5,,1,28,,,,17,4,19,20,19,27,9,23,10,27,6,20,,21,28,,27,28,24,, 36 | 35,7,,35,,19,25,2,22,28,20,,,13,,26,5,,2,4,19,4,26,20,35,28,35,31,10,16,12,,12,10,,,,28,18,,,1,15,,10,,29,32,32,12,19,,7,24,4,17,,,13,,,21,20,,33,,,14,32,,7,30,,4,23,,,15,,22,,6,,1,,,8,,3,21,13,26,1,8,16,,,25,37,,34,19,,16,,,13,29,23,,24,,,4,17,,23,16,16,23,7,,,1,20,11,19,11,,,19,,17,,23,21,13,1,,15,30,25,2,,17,9,18,,,3,28 37 | 36,10,,24,,,1,,23,,10,,24,5,,14,2,4,26,15,,,,17,26,3,,,,,15,,,,15,21,27,13,15,13,,21,29,,34,,,,29,,22,,26,,,12,,13,,16,,,,7,31,,16,2,3,12,13,18,,,4,,23,,20,,18,5,22,21,30,24,5,28,17,12,,5,2,12,4,14,3,8,9,,,,16,27,,26,12,,16,2,4,,19,23,13,15,27,29,,10,21,16,16,,24,,7,10,2,25,11,3,21,14,,,,31,27,28,,,,,7,15,7,27,,7,16 38 | 37,,12,16,,29,16,,30,30,2,8,,22,19,11,,33,19,1,14,,11,,20,,,19,1,,,4,17,22,1,34,25,5,11,20,27,31,,10,,,30,11,,25,8,,6,8,8,,21,7,25,5,7,17,,30,30,,27,27,,8,,27,18,,5,8,22,,11,,,,,3,21,16,,20,,15,,,9,,25,5,16,13,,4,27,10,9,,,,,,8,20,34,20,15,6,18,,20,,3,,37,15,,,10,5,,,,5,7,,22,,31,,20,11,6,,,21,12,7,,11,30,11,,,30 39 | 38,3,,15,29,24,7,21,27,,1,13,,21,,28,24,2,5,26,,3,,,,21,24,,21,6,4,,,13,,7,12,,1,25,28,8,8,,31,27,28,25,25,,,2,34,,17,,,12,,,,28,24,10,24,,1,,12,31,36,9,36,16,22,10,,,16,7,,19,12,34,5,15,3,30,6,6,,,19,22,,,7,27,22,,21,15,2,26,4,14,26,,6,,13,,26,,,24,,28,26,,5,,21,18,21,,2,,9,,1,,,23,3,28,5,,36,29,18,5,24,24,,1,,3,4,,10 40 | 39,4,,21,,16,,10,18,18,,,,2,3,19,,29,,,27,19,,16,16,8,28,2,5,27,8,,,,,23,5,,5,14,16,,,,1,,15,13,13,31,13,14,19,34,,9,17,,6,26,22,9,8,,25,21,14,,7,,17,32,,15,,25,10,,9,19,23,30,17,,32,17,15,18,8,,,15,,,29,,18,,,21,4,16,30,10,,,,16,36,28,29,24,11,30,32,,9,,17,,3,25,8,4,,21,,7,,,21,,6,,21,,2,,1,18,,11,19,,11,4,25,21,34,,4 41 | 40,2,,28,2,15,12,,29,,18,,16,18,16,7,20,30,,16,,2,,19,23,14,15,33,,,2,14,8,19,12,12,,12,,,20,7,14,,5,,,18,,19,,,4,14,21,22,23,20,9,9,,,2,24,28,,,,30,,25,15,24,29,1,27,,4,,,,10,,,6,,10,15,9,,19,18,18,17,32,21,19,,23,10,11,,28,13,19,,14,,17,,25,34,,2,3,4,22,20,,,29,30,,22,15,8,17,2,,10,25,25,11,4,11,24,,,11,,15,,,26,,33,28,,23,, 42 | 41,8,29,3,12,4,9,,25,8,12,3,19,,15,20,14,,,,,,,25,22,31,3,34,,,,16,,26,,19,30,16,,23,17,2,,23,,,13,33,10,14,27,10,30,,,,11,,10,,,16,6,23,14,,30,,29,5,12,,26,18,2,,1,,10,,28,34,3,25,17,9,,31,,4,17,27,14,27,6,,,24,,,,4,25,,26,21,,15,,6,6,23,1,,,,,12,,16,,,,10,3,,,4,17,20,35,20,,,18,,31,19,,,,4,,9,,17,26,,8,12,25 43 | 42,29,16,20,19,,,22,31,9,34,,9,31,12,3,25,7,1,,15,26,16,,,1,23,35,,20,18,,,11,,27,28,3,12,,14,9,,2,26,,20,1,11,22,15,24,13,26,25,15,9,,,,,22,25,,8,18,8,,22,9,35,16,29,,,7,25,31,,,,25,9,24,12,12,,9,,5,,28,,23,7,,24,6,15,12,29,28,21,19,13,,,,10,,5,,21,9,5,28,16,,9,13,26,3,6,2,,24,12,24,14,,30,21,,6,30,10,28,16,15,13,2,18,10,,4,,9,4,31,1,21 44 | 43,24,,,17,11,,15,,6,8,,25,10,21,,6,24,15,6,3,16,,10,,30,16,,20,,24,,3,4,,29,4,14,8,15,2,,,,,23,,4,14,28,10,11,,13,1,14,,,23,12,18,4,3,6,34,29,,32,1,11,,11,7,23,,28,8,28,7,18,17,,,7,3,,,14,,,,8,,28,9,,,18,30,9,7,11,,,20,24,11,,22,,10,22,25,31,21,5,37,,,28,4,24,25,11,,,,22,,16,12,19,,28,10,,,,,3,13,,1,19,1,30,20,20,14,, 45 | 44,,19,4,,,13,20,14,,,9,23,19,28,22,21,31,21,11,,9,21,,11,26,34,28,,9,,,7,,10,10,,29,4,,,12,3,1,,14,19,,,6,12,5,18,17,,27,25,9,,19,3,,,27,10,,31,24,15,2,28,,,5,19,,24,7,30,,19,4,4,18,9,25,23,4,,,22,22,10,20,30,,,,8,25,31,25,26,,22,11,,,14,16,,9,22,,,7,8,,,2,28,,14,17,16,27,5,13,,22,,,12,3,,31,14,,20,14,,,9,,27,3,8,2,15,6, 46 | 45,17,,34,13,28,,26,17,15,,,5,,20,,11,21,,,25,8,12,,10,,18,10,16,17,,,22,2,25,15,,2,,,10,22,5,20,3,5,36,7,23,,20,,33,4,23,6,,1,29,7,12,12,,15,1,7,,21,16,28,15,34,14,6,,,,,,13,,21,,15,11,,,29,16,,10,16,6,,31,,6,,18,18,12,,,,,10,,11,,14,15,35,,5,2,,7,3,27,24,12,18,13,,,6,18,,11,,26,,2,,8,16,25,,37,12,,26,8,28,13,23,19,,,26, 47 | 46,23,1,12,,,,,32,2,17,,,24,2,13,,,,5,,12,20,11,21,5,11,,12,,,,,,,4,10,15,,,5,16,9,6,7,10,12,31,15,,28,,22,2,14,16,,,7,10,20,,,,19,9,28,1,19,1,,,,,15,,17,19,17,,29,11,11,27,31,,11,12,,,7,3,4,,1,,20,16,,,24,3,15,6,17,,9,26,27,18,,,8,18,26,20,28,,19,25,,4,7,12,17,,24,,,9,,,9,,,5,7,13,9,,25,,,,6,22,29,32,35,11,5 48 | 47,,,23,27,,,,1,20,,7,,25,,,,36,30,,28,24,,,12,20,2,30,3,,14,,18,27,,,31,,20,,,,24,,28,,4,20,35,,,6,24,16,11,3,,,1,8,,7,4,,6,11,18,30,,23,34,31,3,11,,,12,,,,2,26,10,2,37,,21,,,13,4,,24,21,,26,26,11,29,27,30,7,23,,,,,13,,,,3,,1,,14,36,,,,11,,,,22,31,8,,7,6,5,,10,22,16,,,18,34,,21,3,31,2,,35,16,23,16,24,15 49 | 48,28,,25,25,1,8,,,17,,19,13,30,,27,,,,,6,,17,,36,32,,15,,,,21,,,2,,,30,,19,,25,,24,33,17,,12,,,14,,14,3,,2,,6,20,18,28,10,,,27,24,,,,16,20,29,,25,7,4,,,34,14,20,18,,23,13,,12,8,,,26,,21,,,,,2,,,14,20,,25,,,20,23,,13,18,21,27,,,22,,22,5,,10,,12,,,,,18,,21,20,27,,,6,4,32,,32,1,11,29,34,,,,21,16,20,21,23 50 | 49,19,7,8,,27,,7,,,25,,21,7,27,8,9,,7,,,,,2,5,36,33,27,,13,,,,16,,,19,1,,21,25,,,9,29,,34,,37,16,6,,28,,,,12,4,24,33,11,,18,1,5,,6,13,4,,10,14,32,8,24,22,18,2,,,,22,16,35,,,,16,,18,5,12,,7,,12,,29,24,1,2,,10,24,,8,,4,,10,17,1,12,,33,,5,,,27,1,,24,15,,,9,25,19,,18,,,21,,9,,28,35,11,9,,29,15,10,,11,33,11,,9 51 | 50,25,,2,7,,,19,21,,27,17,33,16,,29,,12,,12,18,,,21,34,7,,,22,12,23,,,1,8,17,,21,23,,19,28,,22,24,26,,,1,5,24,,1,11,16,,,19,31,29,14,25,28,2,11,16,,8,,13,,22,35,,27,12,3,,33,,26,23,31,32,,11,,,11,,,11,26,,33,,,,,,,2,19,20,28,,25,20,18,3,11,13,,,12,,,,10,11,,,2,,25,10,25,,23,,24,,,,17,17,23,30,,9,,16,,22,24,31,14,17,,, 52 | -------------------------------------------------------------------------------- /test_runs.txt: -------------------------------------------------------------------------------- 1 | 2 | 3 | $ python aggregate.py test/data20_40.csv in ir 5 4 | 5 | Indegree algorithm, score: 0.0323684210526 6 | Iterative best removal with k = 5 score: 0.0634586466165 7 | Removed rankers (in order): ranker21, ranker29, ranker1, ranker38, ranker11 8 | Final score: 0.0634586466165 9 | Final ranker: 10 | 4 19 3 17 6 12 11 15 1 10 5 7 14 13 18 8 20 2 16 9 11 | 12 | $ python aggregate.py test/data20_40.csv in ir 5 ibf 13 | 14 | Indegree algorithm, score: 0.0323684210526 15 | Iterative best removal with k = 5 score: 0.0634586466165 16 | Removed rankers (in order): ranker21, ranker29, ranker1, ranker38, ranker11 17 | Iterative best flip, score: 0.0803007518797 18 | Final score: 0.0803007518797 19 | Final ranker: 20 | 4 19 3 17 6 12 11 15 1 10 5 7 14 13 18 8 20 2 16 9 21 | 22 | $ python aggregate.py test/data20_40.csv in ibf 23 | 24 | Indegree algorithm, score: 0.0323684210526 25 | Iterative best flip, score: 0.118947368421 26 | Final score: 0.118947368421 27 | Final ranker: 28 | 19 4 17 3 6 7 12 5 15 1 18 11 10 14 20 8 13 2 16 9 29 | 30 | $ python aggregate.py test/data20_40.csv pg 0.85 ibf 31 | 32 | Pagerank algorithm, alpha = 0.85 , score: 0.0297368421053 33 | Iterative best flip, score: 0.141578947368 34 | Final score: 0.141578947368 35 | Final ranker: 36 | 19 17 4 3 6 12 7 5 18 1 15 11 10 14 8 20 13 2 16 9 37 | 38 | $ python aggregate.py test/data20_40.csv pg 0.85 ibf ibf 39 | 40 | Pagerank algorithm, alpha = 0.85 , score: 0.0297368421053 41 | Iterative best flip, score: 0.141578947368 42 | Iterative best flip, score: 0.141578947368 43 | Final score: 0.141578947368 44 | Final ranker: 45 | 19 17 4 3 6 12 7 5 18 1 15 11 10 14 8 20 13 2 16 9 46 | 47 | $ python aggregate.py test/data20_40.csv pg 0.85 ir 2 ibf 48 | 49 | Pagerank algorithm, alpha = 0.85 , score: 0.0297368421053 50 | Iterative best removal with k = 2 score: 0.0493074792244 51 | Removed rankers (in order): ranker21, ranker29 52 | Iterative best flip, score: 0.0808864265928 53 | Final score: 0.0808864265928 54 | Final ranker: 55 | 19 4 17 3 12 6 15 11 1 7 5 14 10 18 13 8 20 2 16 9 56 | --------------------------------------------------------------------------------