├── .gitignore ├── README.md ├── TCSYNTAX.md ├── configs ├── default.ini └── hackerrank.ini ├── executer.py ├── tcgen.py ├── testcode.py └── utils.py /.gitignore: -------------------------------------------------------------------------------- 1 | # Byte-compiled / optimized / DLL files 2 | __pycache__/ 3 | *.py[cod] 4 | *$py.class 5 | 6 | # intermediate folder 7 | intermediate -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # CP-test 2 | 3 | A tool to generate test-cases for competitive programming. With it, you can also execute your code on generated test-cases and compare it with other code automatically. 4 | 5 | ### What it can do? 6 | * Generate random test-cases 7 | * Run code (supports various languages) automatically on these test-cases 8 | * Run two code simultaneously and compare the result on these test-cases 9 | * HackerRank problem-setting style test-cases zip 10 | 11 | ### Why? 12 | * Sometimes solely for generating test-cases 13 | * To check if code written generates proper output and does not give a run-time error 14 | * To compare your code with brute-force code/any other code to check for corner cases 15 | 16 | ## Usage 17 | 18 | To generate test-cases (E.g.: 5 test-cases) 19 | 20 | ``` 21 | python3 testcode.py -T testcase_syntax.tcs -N 5 22 | ``` 23 | 24 | To generate test-cases and run your code on generated test-cases 25 | ``` 26 | python3 testcode.py -T testcase_syntax.tcs -I1 my_code.cpp 27 | ``` 28 | 29 | To generate test-cases and run two codes and compare output 30 | ``` 31 | python3 testcode.py -T testcase_syntax.tcs -I1 my_code.cpp -I2 my_bruteforce.py 32 | ``` 33 | 34 | To generate HackerRank style test-cases zip 35 | ``` 36 | python3 testcode.py -T testcase_syntax.tcs -I1 my_code.cpp -C configs/hackerrank.ini 37 | ``` 38 |
39 | The default value of `-N` is 10. To generate more/less test-cases pass it as parameter.
40 | The default config used is `configs/default.ini`. Use `-C` flag to pass custom config.
41 | 42 | You need to specify syntax of test-cases in a file and pass it with `-T` argument.
43 | This test-case syntax is very simple and can be easily written. Refer to [TCSYNTAX.md](TCSYNTAX.md) 44 | 45 | ### Supported languages 46 | 47 | Currently code files supported are: 48 | 49 | - C11 50 | - C++14 51 | - Python3 52 | -------------------------------------------------------------------------------- /TCSYNTAX.md: -------------------------------------------------------------------------------- 1 | # Test-Case Syntax 2 | Here a guide to write syntax for test-cases. Refer examples for better understanding. 3 | 4 | ## General format 5 | Every line consist of minimum 2 parts 6 | 7 | 1. Type of data to be generated 8 | 2. Variable name (should contain alphabets only) 9 | 10 | The other parts can be 11 | 12 | 3. Minimum value of data elements 13 | 4. Maximum value of data elements 14 | 15 | For string: 16 | 17 | 3. Content of string (lowercase, uppercase, digits, etc.) 18 | 19 | For loop: 20 | 21 | 3. Number of iteration 22 | 23 | ## Rules 24 | * A semicolon at end of line indicates a new line in generated cases. If it is not placed output is generated on single line. 25 | 26 | 27 | Note: In below examples, `between a and b` also includes a and b 28 | 29 | *** 30 | #### Generate single Integer 31 | 32 | ``` 33 | int variable-name min-value max-value 34 | ``` 35 | Example: 36 | ``` 37 | int mx 0 1000; 38 | int n 0 mx; 39 | ``` 40 | Generates,
41 | a single integer between 0 and 1000 -> mx (0th line)
42 | a single integer between 0 and mx -> n (1st line)
43 | 44 | --- 45 | #### Generate row array of distinct integers 46 | ``` 47 | drarray_size variable-name min-value max-value 48 | ``` 49 | Example: 50 | ``` 51 | drarray_25 ar 0 100; 52 | int sz 0 20; 53 | drarray_sz arr 1 sz; 54 | ``` 55 | Generates,
56 | a row array of size 25 of distinct integers between 0 and 100 -> ar (0th line)
57 | a single integer between 0 and 20 -> sz (1st line)
58 | a row array of size sz of all integers between 1 and sz in random order -> arr (2nd line)
59 | 60 | --- 61 | #### Generate row array of integers 62 | ``` 63 | rarray_size variable-name min-value max-value 64 | ``` 65 | Example: 66 | ``` 67 | rarray_10 ar 0 100; 68 | int sz 0 20; 69 | rarray_sz arr 0 1000; 70 | ``` 71 | Generates,
72 | a row array of size 10 of integers between 0 and 100 -> ar (0th line)
73 | a single integer between 0 and 20 -> sz (1st line)
74 | a row array of size sz of integers between 0 and 1000 -> arr (2nd line)
75 | 76 | --- 77 | #### Generate fixed length string 78 | ``` 79 | flstring_size variable-name content 80 | ``` 81 | Content (single or any combination): 82 | 83 | - u - uppercase 84 | - l - lowercase 85 | - d - digits 86 | - ? - custom character string (See example) 87 | 88 | Example: 89 | ``` 90 | flstring_10 str ul; 91 | flstring_100 strr ? $#&; 92 | ``` 93 | Generates,
94 | a string of length 10 containing lowercase and uppercase characters -> str (0th line)
95 | a string of length 100 containing $#& characters -> strr (1st line)
96 | 97 | --- 98 | #### Generate variable length string 99 | ``` 100 | rlstring_minsize_maxsize variable-name content 101 | ``` 102 | Content (single or any combination): 103 | 104 | - u - uppercase 105 | - l - lowercase 106 | - d - digits 107 | - ? - custom character string (See example) 108 | 109 | Example: 110 | ``` 111 | int n 1 15 20; 112 | rlstring_10_20 str d; 113 | rlstring_10_n strr ? *+.; 114 | ``` 115 | Generates,
116 | a single integer between 15 and 20 -> n (0th line)
117 | a string containing digits whose length is between 10 and 20 -> str (1st line)
118 | a string containing \*+. characters whose length is between 10 and n -> strr (2nd line)
119 | 120 | --- 121 | #### Looping 122 | ``` 123 | loop variable-name no-of-iterations 124 | ``` 125 | 126 | Lines to be looped are to be indented by 4 spaces.
127 | Nested loops are also possible. 128 | 129 | Example: 130 | ``` 131 | int t 1 50; 132 | loop lp t 133 | rlstring_10_20 str d; 134 | int n 1 100; 135 | int k 1 10; 136 | ``` 137 | Generates, 138 | a single integer between 1 and 50 -> t (0th line)
139 | generate t times string containing digits whose length is between 10 and 20 -> str (2nd line)
140 | generate t times single integer between 1 and 100 -> n (3rd line)
141 | a single integer between 1 and 10 -> k (after loop)
142 | -------------------------------------------------------------------------------- /configs/default.ini: -------------------------------------------------------------------------------- 1 | [code1] 2 | # Name of temporary file generated after converting stdout to file-io 3 | fileio = code1_fileio 4 | # Name of output file for code1 5 | output = code1_output 6 | # Name of output binary executable 7 | binary = code1 8 | 9 | [code2] 10 | # Name of temporary file generated after converting stdout to file-io 11 | fileio = code2_fileio 12 | # Name of output file for code2 13 | output = code2_output 14 | # Name of output binary executable 15 | binary = code2 16 | 17 | [testcases] 18 | # Name of file in which testcases are generated 19 | output = tc_input 20 | 21 | [output] 22 | # Intermediate folder name 23 | intermediate = intermediate 24 | # Type of test groups (1 - group by tests; 2 - group by category) 25 | group_type = 1 26 | # Index the output file (i.e. add test-number to output files name) 27 | # for group by category, indexing is always done 28 | # this is for files only, folders are always indexed if group by tests 29 | index_output = no 30 | # Index number length (eg - 1 : 1, 2 : 01, 3 : 001, etc.) 31 | index_length = 0 32 | # Whether to zip output generated 33 | zip_output = no 34 | 35 | [result] 36 | # Name of file in which diffs are generated 37 | output = diff_result 38 | # Whether to generate final report or not 39 | make_report = no 40 | # Name of file in which final report is generated 41 | report = final_report 42 | 43 | [cleanup] 44 | # Whether to keep or delete file-io code (if any) 45 | clean_fileio = yes 46 | # Whether to keep or delete geneared binaries (if any) 47 | clean_binary = yes 48 | # Whether to keep or delete code2 and diff (if any) 49 | clean_compare = no 50 | # Whether to clean everything except report (testcases, input, output, diffs) 51 | clean_all = no 52 | -------------------------------------------------------------------------------- /configs/hackerrank.ini: -------------------------------------------------------------------------------- 1 | [code1] 2 | fileio = code1_fileio 3 | output = output 4 | binary = code1 5 | 6 | [code2] 7 | fileio = code2_fileio 8 | output = code2_output 9 | binary = code2 10 | 11 | [testcases] 12 | output = input 13 | 14 | [output] 15 | intermediate = hackerrank 16 | group_type = 2 17 | index_output = yes 18 | index_length = 2 19 | zip_output = yes 20 | 21 | [result] 22 | output = diff_result 23 | make_report = no 24 | report = final_report 25 | 26 | [cleanup] 27 | clean_fileio = yes 28 | clean_binary = yes 29 | clean_compare = yes 30 | clean_all = yes 31 | -------------------------------------------------------------------------------- /executer.py: -------------------------------------------------------------------------------- 1 | import subprocess 2 | import sys 3 | import time 4 | import re 5 | 6 | 7 | def c_cpp_code_to_fileio(inp, out_name, isplus): 8 | if not isplus: 9 | out_code = '#include \n' 10 | else: 11 | out_code = '#include \n' 12 | 13 | file_io_lines = "int main(int argc, char const *argv[]) {" 14 | file_io_lines += '\n\tfreopen(argv[1], "r", stdin);' 15 | file_io_lines += '\n\tfreopen(argv[2], "w", stdout);' 16 | main_func_pattern = r'int[\s]*main[\s]*\([^\)]*\)[\s]*{' 17 | 18 | with open(inp, 'r') as in_f: 19 | out_code += in_f.read() 20 | 21 | out_code = re.sub(main_func_pattern, file_io_lines, out_code) 22 | 23 | file_ext = ".c" if not isplus else ".cpp" 24 | with open(out_name + file_ext, 'w') as op_f: 25 | op_f.write(out_code) 26 | 27 | return out_name + file_ext 28 | 29 | 30 | def py_code_to_fileio(inp, out_name): 31 | out_code = 'import sys\n' 32 | out_code += "fin = open(sys.argv[1], 'r')\n" 33 | out_code += "fout = open(sys.argv[2], 'w+')\n" 34 | 35 | with open(inp, 'r') as in_f: 36 | out_code += in_f.read() 37 | 38 | inp_pattern = r'input[\s]*\([\s]*\)([\s]*\.[\s]*[lr]?strip[\s]*\([\s]*\))*' 39 | out_pattern = r'print[^\S\r\n]*\((.*)\)' 40 | 41 | out_code = re.sub(inp_pattern, 'fin.readline().strip()', out_code) 42 | out_code = re.sub(out_pattern, 'print(str(\\1), file=fout)', out_code) 43 | 44 | with open(out_name + ".py", 'w') as op_f: 45 | op_f.write(out_code) 46 | 47 | return out_name + ".py" 48 | 49 | 50 | def compile_c_code(inp, out_bin): 51 | try: 52 | start_time = time.time() 53 | subprocess.check_call([r"/usr/bin/gcc", "--std=c11", "-Wall", 54 | "-o", out_bin, inp + ".c"]) 55 | end_time = time.time() 56 | return end_time - start_time 57 | except subprocess.CalledProcessError as _: 58 | print("Compilation Error in file : " + inp) 59 | sys.exit(1) 60 | 61 | 62 | def compile_cpp_code(inp, out_bin): 63 | try: 64 | start_time = time.time() 65 | subprocess.check_call([r"/usr/bin/g++", "--std=c++14", "-Wall", 66 | "-o", out_bin, inp + ".cpp"]) 67 | end_time = time.time() 68 | return end_time - start_time 69 | except subprocess.CalledProcessError as _: 70 | print("Compilation Error in file : " + inp) 71 | sys.exit(1) 72 | 73 | 74 | def run_c_cpp_bin(binary, in_tc, out_res): 75 | try: 76 | start_time = time.time() 77 | subprocess.check_call(["./" + binary, in_tc, out_res]) 78 | end_time = time.time() 79 | return end_time - start_time 80 | except subprocess.CalledProcessError as _: 81 | print("Runtime Error in binary_file : " + binary) 82 | sys.exit(1) 83 | 84 | 85 | def run_py_code(inp, in_tc, out_res): 86 | try: 87 | inp = inp + ".py" 88 | start_time = time.time() 89 | subprocess.check_call([r"/usr/bin/python3", inp, in_tc, out_res]) 90 | end_time = time.time() 91 | return end_time - start_time 92 | except subprocess.CalledProcessError as _: 93 | print("Runtime Error in python_file : " + inp) 94 | sys.exit(1) 95 | -------------------------------------------------------------------------------- /tcgen.py: -------------------------------------------------------------------------------- 1 | from random import randint, choice, sample 2 | import string 3 | import time 4 | import re 5 | 6 | values = {} 7 | 8 | 9 | def get_value(s_val): 10 | numptn = re.compile(r'^-?([0-9]+)$') 11 | varptn = re.compile(r'^([a-zA-Z]+)$') 12 | arrptn = re.compile(r'^([a-zA-Z]+)([0-9]+)$') 13 | 14 | if numptn.match(s_val): 15 | return int(s_val) 16 | elif varptn.match(s_val): 17 | return values[s_val] 18 | elif arrptn.match(s_val): 19 | ptnsch = re.search(arrptn, s_val) 20 | return values[ptnsch.group(1)][int(ptnsch.group(2))] 21 | 22 | 23 | def generate_int(line): 24 | minv = get_value(line[3]) 25 | maxv = get_value(line[4]) 26 | randn = randint(minv, maxv) 27 | values[line[2]] = randn 28 | return str(randn) 29 | 30 | 31 | def generate_drarray(line): 32 | output = "" 33 | ssize = line[1].split('_')[1] 34 | size = get_value(ssize) 35 | minv = get_value(line[3]) 36 | maxv = get_value(line[4]) 37 | rns = sample(range(minv, maxv+1), size) 38 | values[line[2]] = rns 39 | 40 | for val in rns: 41 | output += str(val) + " " 42 | 43 | return output 44 | 45 | 46 | def generate_rarray(line): 47 | output = "" 48 | ssize = line[1].split('_')[1] 49 | size = get_value(ssize) 50 | minv = get_value(line[3]) 51 | maxv = get_value(line[4]) 52 | rns = [randint(minv, maxv) for _ in range(size)] 53 | values[line[2]] = rns 54 | 55 | for val in rns: 56 | output += str(val) + " " 57 | 58 | return output 59 | 60 | 61 | def generate_rlstring(line): 62 | output = "" 63 | minlen = line[1].split('_')[1] 64 | maxlen = line[1].split('_')[2] 65 | minlen = get_value(minlen) 66 | maxlen = get_value(maxlen) 67 | rnlen = randint(minlen, maxlen) 68 | 69 | letters = "" 70 | if line[3] == "?": 71 | letters += line[4].strip() 72 | else: 73 | if "u" in line[3]: 74 | letters += string.ascii_uppercase 75 | if "l" in line[3]: 76 | letters += string.ascii_lowercase 77 | if "d" in line[3]: 78 | letters += string.digits 79 | 80 | for _ in range(rnlen): 81 | output += choice(letters) 82 | 83 | return output 84 | 85 | 86 | def generate_flstring(line): 87 | output = "" 88 | flen = line[1].split('_')[1] 89 | flen = get_value(flen) 90 | letters = "" 91 | if line[3] == "?": 92 | letters += line[4].strip() 93 | else: 94 | if "u" in line[3]: 95 | letters += string.ascii_uppercase 96 | if "l" in line[3]: 97 | letters += string.ascii_lowercase 98 | if "d" in line[3]: 99 | letters += string.digits 100 | 101 | for _ in range(flen): 102 | output += choice(letters) 103 | 104 | return output 105 | 106 | 107 | def generate_loop(i, synlist): 108 | output = "" 109 | line = synlist[i] 110 | 111 | ittrs = line[3] 112 | ittrs = get_value(ittrs) 113 | 114 | nll = 0 115 | for ln in range(i+1, len(synlist)): 116 | nspaces = synlist[ln][0] - line[0] 117 | if nspaces > 0 and nspaces % 4 == 0: 118 | nll += 1 119 | else: 120 | break 121 | 122 | for _ in range(ittrs-1): 123 | output += generate_tc(synlist[i+1:i+1+nll]) 124 | 125 | return output 126 | 127 | 128 | def generate_tc(synlist): 129 | output = "" 130 | 131 | i = 0 132 | while True: 133 | if i >= len(synlist): 134 | break 135 | 136 | line = synlist[i] 137 | 138 | if line[1] == 'int': 139 | output += generate_int(line) 140 | 141 | elif "drarray" in line[1]: 142 | output += generate_drarray(line) 143 | 144 | elif "rarray" in line[1]: 145 | output += generate_rarray(line) 146 | 147 | elif "rlstring" in line[1]: 148 | output += generate_rlstring(line) 149 | 150 | elif "flstring" in line[1]: 151 | output += generate_flstring(line) 152 | 153 | elif "loop" in line[1]: 154 | output += generate_loop(i, synlist) 155 | 156 | if output and output[-1] != '\n': 157 | if line[-1] == ';': 158 | output += "\n" 159 | elif output[-1] != " ": 160 | output += " " 161 | 162 | i += 1 163 | 164 | return output 165 | 166 | 167 | def get_tokens(line): 168 | synline = line.strip().split() 169 | if not synline: 170 | return None 171 | indentspace = len(line) - len(line.lstrip()) 172 | tokens = [indentspace] 173 | tokens.extend(synline) 174 | 175 | if tokens[-1][-1] == ';': 176 | tokens[-1] = tokens[-1][:-1] 177 | tokens.append(';') 178 | 179 | return tokens 180 | 181 | 182 | def get_tcs(tc_syntax, tc_output): 183 | with open(tc_syntax, 'r') as syntax_file: 184 | synlist = [] 185 | for line in syntax_file.readlines(): 186 | tkns = get_tokens(line) 187 | if tkns: 188 | synlist.append(tkns) 189 | start_time = time.time() 190 | tc_out = generate_tc(synlist) 191 | end_time = time.time() 192 | 193 | with open(tc_output, 'w') as out_file: 194 | out_file.write(tc_out) 195 | 196 | return end_time - start_time 197 | -------------------------------------------------------------------------------- /testcode.py: -------------------------------------------------------------------------------- 1 | import argparse 2 | import configparser 3 | 4 | import tcgen 5 | import executer 6 | import utils 7 | 8 | parser = argparse.ArgumentParser() 9 | config = configparser.RawConfigParser(allow_no_value=False) 10 | 11 | 12 | parser.add_argument('-T', metavar='--testcases', type=str, required=True, 13 | help='file containg testcase syntax') 14 | parser.add_argument('-I1', metavar='--input1', type=str, required=False, 15 | help='file containg code1') 16 | parser.add_argument('-I2', metavar='--input2', type=str, required=False, 17 | help='file containg code2') 18 | parser.add_argument('-C', metavar='--config', type=str, 19 | default='configs/default.ini', help='config file') 20 | parser.add_argument('-N', metavar='--no_test', type=int, default=10, 21 | help='no of times to run test') 22 | 23 | args = parser.parse_args() 24 | config.read(args.C) 25 | 26 | tc_syntax = args.T 27 | tc_nos = args.N 28 | 29 | interm = config['output']['intermediate'] + '/' 30 | 31 | code1_file = args.I2 if args.I1 is None and args.I2 is not None else args.I1 32 | code1_fio = interm + config['code1']['fileio'] 33 | code1_bin = interm + config['code1']['binary'] 34 | 35 | code2_file = None if args.I1 is None and args.I2 is not None else args.I2 36 | code2_fio = interm + config['code2']['fileio'] 37 | code2_bin = interm + config['code2']['binary'] 38 | 39 | tc_out = interm + config['testcases']['output'] 40 | code1_out = interm + config['code1']['output'] 41 | code2_out = interm + config['code2']['output'] 42 | result = interm + config['result']['output'] 43 | 44 | report = interm + config['result']['report'] + ".txt" 45 | make_report = config['result'].getboolean('make_report') 46 | 47 | grp_type = config['output'].getint('group_type') 48 | idx_out = config['output'].getboolean('index_output') or (grp_type != 1) 49 | idx_len = config['output'].getint('index_length') 50 | 51 | gen_zip = config['output'].getboolean('zip_output') 52 | 53 | clean_fileio = config['cleanup'].getboolean('clean_fileio') 54 | clean_binary = config['cleanup'].getboolean('clean_binary') 55 | clean_compare = config['cleanup'].getboolean('clean_compare') 56 | clean_all = config['cleanup'].getboolean('clean_all') 57 | 58 | 59 | if __name__ == "__main__": 60 | 61 | utils.create_folder(interm) 62 | 63 | print("-" * 10 + " Compiling " + "-" * 10) 64 | 65 | if code1_file is not None: 66 | if ".cpp" in code1_file: 67 | executer.c_cpp_code_to_fileio(code1_file, code1_fio, isplus=True) 68 | c1_time = executer.compile_cpp_code(code1_fio, code1_bin) 69 | print("Code1 CPP compiled in %.5f sec" % c1_time) 70 | elif ".c" in code1_file: 71 | executer.c_cpp_code_to_fileio(code1_file, code1_fio, isplus=False) 72 | c1_time = executer.compile_c_code(code1_fio, code1_bin) 73 | print("Code1 C compiled in %.5f sec" % c1_time) 74 | elif ".py" in code1_file: 75 | executer.py_code_to_fileio(code1_file, code1_fio) 76 | print("Code1 PY converted") 77 | 78 | if code2_file is not None: 79 | if ".cpp" in code2_file: 80 | executer.c_cpp_code_to_fileio(code2_file, code2_fio, isplus=True) 81 | c2_time = executer.compile_cpp_code(code2_fio, code2_bin) 82 | print("Code2 CPP compiled in %.5f sec" % c2_time) 83 | elif ".c" in code2_file: 84 | executer.c_cpp_code_to_fileio(code2_file, code2_fio, isplus=False) 85 | c2_time = executer.compile_c_code(code2_fio, code2_bin) 86 | print("Code1 C compiled in %.5f sec" % c2_time) 87 | elif ".py" in code2_file: 88 | executer.py_code_to_fileio(code2_file, code2_fio) 89 | print("Code2 PY converted") 90 | 91 | print() 92 | 93 | try: 94 | stats = [] 95 | created_folders = set() 96 | 97 | ttc_out = tc_out + ".tmp" 98 | tc1_out = code1_out + ".tmp" 99 | tc2_out = code2_out + ".tmp" 100 | tresult = result + ".tmp" 101 | 102 | for i in range(tc_nos): 103 | 104 | print("-" * 10 + " Test - " + str(i) + " " + "-" * 10) 105 | 106 | tc_time = tcgen.get_tcs(tc_syntax, ttc_out) 107 | print("Testcases generated in %.5f sec" % tc_time) 108 | 109 | grp_args = {'grptype': grp_type, 110 | 'idxfile': idx_out, 111 | 'idxlen': idx_len} 112 | 113 | foldr = utils.copy_to_grp(i, ttc_out, **grp_args) 114 | created_folders.add(foldr) 115 | 116 | if code1_file is not None: 117 | if ".cpp" in code1_file or ".c" in code1_file: 118 | c1tm = executer.run_c_cpp_bin(code1_bin, ttc_out, tc1_out) 119 | elif ".py" in code1_file: 120 | c1tm = executer.run_py_code(code1_fio, ttc_out, tc1_out) 121 | print("Code1 executed in %.5f sec" % c1tm) 122 | 123 | foldr = utils.copy_to_grp(i, tc1_out, **grp_args) 124 | created_folders.add(foldr) 125 | 126 | stats.append({'code1_time': c1tm}) 127 | 128 | if code2_file is not None: 129 | if ".cpp" in code2_file or ".c" in code2_file: 130 | c2tm = executer.run_c_cpp_bin(code2_bin, ttc_out, tc2_out) 131 | elif ".py" in code2_file: 132 | c2tm = executer.run_py_code(code2_fio, ttc_out, tc2_out) 133 | print("Code2 executed in %.5f sec" % c2tm) 134 | 135 | foldr = utils.copy_to_grp(i, tc2_out, **grp_args) 136 | created_folders.add(foldr) 137 | 138 | diffs = utils.compare_outputs(tc1_out, tc2_out, tresult) 139 | if diffs == 0: 140 | print("Success : both outputs are same") 141 | elif diffs == -1: 142 | print("Failure : invalid output generated") 143 | else: 144 | print("Failure : output different at %d positions" % diffs) 145 | 146 | if diffs != -1: 147 | foldr = utils.copy_to_grp(i, tresult, **grp_args) 148 | created_folders.add(foldr) 149 | 150 | stats[-1].update({'code2_time': c2tm, 'diff': diffs}) 151 | 152 | print() 153 | 154 | except KeyboardInterrupt as _: 155 | i = i-1 156 | print("Error : KeyboardInterrupt") 157 | 158 | finally: 159 | if stats: 160 | print() 161 | print("Tests done : " + str(i+1) + "/" + str(tc_nos)) 162 | if make_report: 163 | utils.write_stats(stats, report, code2_file is None) 164 | print("Report written to " + report) 165 | print() 166 | 167 | if code1_file is not None: 168 | if clean_fileio: 169 | if ".py" in code1_file: 170 | utils.delete_file(code1_fio + ".py") 171 | elif ".cpp" in code1_file: 172 | utils.delete_file(code1_fio + ".cpp") 173 | elif ".c" in code1_file: 174 | utils.delete_file(code1_fio + ".c") 175 | 176 | if clean_binary: 177 | if ".cpp" in code1_file or ".c" in code1_file: 178 | utils.delete_file(code1_bin) 179 | 180 | utils.delete_file(tc1_out) 181 | 182 | if code2_file is not None: 183 | if clean_fileio: 184 | if ".py" in code2_file: 185 | utils.delete_file(code2_fio + ".py") 186 | elif ".cpp" in code2_file: 187 | utils.delete_file(code2_fio + ".cpp") 188 | elif ".c" in code2_file: 189 | utils.delete_file(code2_fio + ".c") 190 | 191 | if clean_binary: 192 | if ".cpp" in code2_file or ".c" in code2_file: 193 | utils.delete_file(code2_bin) 194 | 195 | utils.delete_file(tc2_out) 196 | 197 | utils.delete_file(ttc_out) 198 | utils.delete_file(tresult) 199 | 200 | if clean_compare: 201 | if grp_type == 2: 202 | print(created_folders) 203 | utils.delete_folder(code2_out) 204 | utils.delete_folder(result) 205 | elif grp_type == 1: 206 | fend = (str(i).zfill(idx_len) if idx_out else "") + ".txt" 207 | c2sidx = code2_out.rfind('/') + 1 208 | ressidx = code2_out.rfind('/') + 1 209 | c2fname = code2_out[c2sidx:] 210 | resfname = result[ressidx:] 211 | for fld in created_folders: 212 | utils.delete_file(interm + fld + "/" + c2fname + fend) 213 | utils.delete_file(interm + fld + "/" + resfname + fend) 214 | 215 | if gen_zip: 216 | utils.delete_file(interm + interm[:-1] + ".zip") 217 | utils.make_zip(interm[:-1]) 218 | 219 | if clean_all: 220 | print("**WARNING**") 221 | print("Cleaning everything except report.") 222 | print("To avoid this set 'clean_all' flag to 'no' in your config") 223 | utils.delete_folders(interm, created_folders) 224 | print() 225 | -------------------------------------------------------------------------------- /utils.py: -------------------------------------------------------------------------------- 1 | import os 2 | import shutil 3 | import tempfile 4 | 5 | 6 | def compare_outputs(code1_op, code2_op, result_file): 7 | with open(code1_op) as code_file: 8 | code1_op_lines = code_file.readlines() 9 | 10 | with open(code2_op) as code_file: 11 | code2_op_lines = code_file.readlines() 12 | 13 | if len(code1_op_lines) != len(code2_op_lines): 14 | print("Error - both output files dont have same no of lines") 15 | return -1 16 | 17 | res = open(result_file, 'w') 18 | diff = 0 19 | 20 | i = 0 21 | for c1l, c2l in zip(code1_op_lines, code2_op_lines): 22 | c1l = c1l.strip() 23 | c2l = c2l.strip() 24 | 25 | i += 1 26 | 27 | if c1l != c2l: 28 | res.write("-" * 10 + " Line - " + str(i) + " " + "-" * 10 + "\n") 29 | res.write("code1 >" + "\n") 30 | res.write(c1l + "\n") 31 | res.write("code2 >" + "\n") 32 | res.write(c2l + "\n") 33 | res.write("\n") 34 | diff += 1 35 | 36 | res.close() 37 | return diff 38 | 39 | 40 | def write_stats(stats, report_file, single_file): 41 | res = open(report_file, 'w') 42 | 43 | stat_size = len(stats) 44 | 45 | max_code1_time = max(stats, 46 | key=lambda x: x['code1_time'])['code1_time'] 47 | min_code1_time = min(stats, 48 | key=lambda x: x['code1_time'])['code1_time'] 49 | avg_code1_time = sum(s['code1_time'] for s in stats)/stat_size 50 | 51 | res.write("-" * 10 + " Code1 File " + "-" * 10 + "\n") 52 | res.write("Minimum time : " + str(min_code1_time) + " sec" + "\n") 53 | res.write("Maximum time : " + str(max_code1_time) + " sec" + "\n") 54 | res.write("Average time : " + str(avg_code1_time) + " sec" + "\n") 55 | res.write("\n") 56 | 57 | if not single_file: 58 | max_code2_time = max(stats, 59 | key=lambda x: x['code2_time'])['code2_time'] 60 | min_code2_time = min(stats, 61 | key=lambda x: x['code2_time'])['code2_time'] 62 | avg_code2_time = sum(s['code2_time'] for s in stats)/stat_size 63 | 64 | res.write("-" * 10 + " Code2 File " + "-" * 10 + "\n") 65 | res.write("Minimum time : " + str(min_code2_time) + " sec" + "\n") 66 | res.write("Maximum time : " + str(max_code2_time) + " sec" + "\n") 67 | res.write("Average time : " + str(avg_code2_time) + " sec" + "\n") 68 | res.write("\n") 69 | 70 | is_wrong = False 71 | 72 | res.write("-" * 10 + " Wrong Testcases " + "-" * 10 + "\n") 73 | for i in range(stat_size): 74 | if stats[i]['diff'] == -1: 75 | res.write("Testcase " + str(i) + " produced invalid output" 76 | + "\n") 77 | is_wrong = True 78 | elif stats[i]['diff'] != 0: 79 | res.write("Testcase " + str(i) + " different at " 80 | + str(stats[i]['diff']) + " positions" + "\n") 81 | is_wrong = True 82 | 83 | if not is_wrong: 84 | res.write("No wrong testcase" + "\n") 85 | 86 | res.close() 87 | 88 | 89 | def copy_to_grp(i, filename, grptype, idxfile, idxlen): 90 | sidx = filename.rfind('/') + 1 91 | didx = filename.rfind('.') 92 | 93 | if grptype == 1: 94 | fldrnam = str(i).zfill(idxlen) 95 | elif grptype == 2: 96 | idxfile = True 97 | fldrnam = filename[sidx:didx] 98 | else: 99 | fldrnam = str(i) 100 | 101 | fileext = (str(i).zfill(idxlen) if idxfile else "") + ".txt" 102 | 103 | if not os.path.exists(filename[:sidx] + fldrnam): 104 | os.makedirs(filename[:sidx] + fldrnam) 105 | new_filename = filename[:sidx] + fldrnam + '/' 106 | new_filename += filename[sidx:didx] + fileext 107 | shutil.copyfile(filename, new_filename) 108 | 109 | return fldrnam 110 | 111 | 112 | def create_folder(folder): 113 | if not os.path.exists(folder): 114 | os.makedirs(folder) 115 | 116 | 117 | def delete_file(filename): 118 | if os.path.exists(filename): 119 | os.remove(filename) 120 | 121 | 122 | def delete_folder(foldername): 123 | if os.path.exists(foldername): 124 | shutil.rmtree(foldername) 125 | 126 | 127 | def delete_folders(basepath, folder_list): 128 | for folder in folder_list: 129 | delete_folder(basepath + "/" + folder) 130 | 131 | 132 | def make_zip(folder): 133 | tmpzip = tempfile.gettempdir() + "/" + folder 134 | shutil.make_archive(tmpzip, 'zip', folder) 135 | shutil.move(tmpzip + ".zip", folder + "/" + folder + ".zip") 136 | --------------------------------------------------------------------------------