├── README.md └── create_env.py /README.md: -------------------------------------------------------------------------------- 1 | # Conda Meta Environment Parser 2 | 3 | This tiny parser takes as input a meta-environment file, transforms it into an environment file and invokes conda on it. 4 | The input, usually called **environment.yml.meta**, is a standard conda environment file which can be enriched with conditions for certain lines/blocks, e.g. to only include them on specific platforms or if certain flags are given. 5 | 6 | ## Example 7 | 8 | The following meta-file 9 | 10 | ``` 11 | name: demo_env 12 | dependencies: 13 | - numpy=1.16.4 14 | - tensorflow=1.14.0 15 | - tensorflow-gpu=1.14.0 [use_gpu] 16 | - winreg-helpers [platform == win32] 17 | ``` 18 | 19 | can be used to set up a conda environment with 20 | 21 | `python create_env.py` (A) 22 | 23 | or 24 | 25 | `python create_env.py --flag use_gpu` (B). 26 | 27 | In (A), the conda package *tensorflow-gpu* is not installed. In both cases, *winreg-helpers* is only installed on windows platforms. 28 | 29 | ## Quickstart 30 | 31 | To use CME Parser in your own project, create an **environment.yml.meta** file in your project folder, e.g. like this: 32 | 33 | ``` 34 | name: demo_env2 35 | channels: 36 | - conda-forge 37 | - defaults 38 | dependencies: 39 | - numpy=1.16.4 40 | - matplotlib=3.1.0 [allow_plot and plot_tool == matplotlib] 41 | - ggplot=0.11.5 [allow_plot and plot_tool == ggplot] 42 | - pip=19.1.1 43 | - some-linux-package=1.2.3 [platform startswith linux] 44 | - pip: 45 | - selenium=3.141.0 46 | ``` 47 | 48 | If **create_env.py** gets called without an `--input` argument, it will try to find **environment.yml.meta** either in the same directory or its parent. 49 | Thus, either directly add **create_env.py** to your project folder next to the meta-env file or add this repo as git submodule: 50 | 51 | ``` 52 | $ cd your_project_folder 53 | $ git rm environment.yml # if exists 54 | $ echo "environment.yml" >> .gitignore 55 | $ git add environment.yml.meta 56 | $ git submodule add https://github.com/silvanmelchior/cme_parser 57 | $ git commit -m "cme parser added" 58 | $ git push 59 | ``` 60 | 61 | If you now execute (in the case of using a submodule) 62 | 63 | `python cme_parser/create_env.py --flag allow_plot --variable plot_tool matplotlib` 64 | 65 | in your project folder, CME Parser will find the meta-env file and create **environment.yml**. 66 | In general, if not specified differently with a `--output` argument, the parser will write the output into the same location as the input, removing *.meta* from the filename. 67 | If *.meta* is not present, it will append *_out* to the filename instead. 68 | In all cases, the parser will in the end invoke conda to create the environment specified by the output file. 69 | 70 | ## Requirements 71 | 72 | CME Parser is written in pure Python and does only need standard library packages. 73 | It can be executed e.g. in the conda default environment (called *base*) and was tested with Python 2.7 and 3.6 / 3.7. 74 | 75 | ## Syntax 76 | 77 | If a line contains `[condition]`, the parser only outputs it if the condition is met (removing the condition itself). 78 | If a line contains `[[condition` and a later line `]]`, the parser outputs everything in between only if the condition is met (removing the two lines marking the block borders). 79 | Nested blocks are allowed. 80 | 81 | A condition can either be an arbitrary name, which is then interpreted as a flag, or a triplet of an arbitrary name, some operator and a string. 82 | In the latter, the name is interpreted as variable and the operator can be one of `==`, `!=`, `startswith`, `endswith`, `contains`. 83 | The string does not need any special terminators (e.g. no `""`). 84 | 85 | Conditions can be combined by adding `and` and `or` in between them. 86 | Furthermore, `not` can be added in front of a condition. The precedence order is `or` (highest), `and`, `not` (lowest), so e.g. `a and not b` is evaluated as `a and (not b)`. 87 | Brackets to enforce another evaluation order are currently not supported. 88 | 89 | Flags are evaluated as *false* per default, except given to the parser with `-f flag_name`. 90 | Variables have to be given to the parser with `-v var_name var_value`, otherwise an error is risen. 91 | An exception are variables of the form `var_name%value`, which are interpreted as variable `var_name` with default value `value`. 92 | Furthermore, the special variable `platform` is reserved and populated with the result of `sys.platform`. 93 | 94 | ## Command Line Options 95 | 96 | | Short | Long | Description | 97 | | --- | --- | --- | 98 | | -h | --help | show usage help message | 99 | | -i | --input | specify meta environment input file | 100 | | -o | --output | specify environment output file | 101 | | -f | --flag | set custom flag for parsing | 102 | | -v | --variable | define custom variable for parsing | 103 | | -q | --quiet | quietly overwrite output if already exists | 104 | | -p | --parse-only | do not invoke conda afterwards | 105 | | -c | --no-comment | do not add auto-generated comment to output | 106 | -------------------------------------------------------------------------------- /create_env.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | """ 3 | MIT License 4 | 5 | Copyright (c) 2019 Silvan Melchior 6 | 7 | Permission is hereby granted, free of charge, to any person obtaining a copy 8 | of this software and associated documentation files (the "Software"), to deal 9 | in the Software without restriction, including without limitation the rights 10 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 11 | copies of the Software, and to permit persons to whom the Software is 12 | furnished to do so, subject to the following conditions: 13 | 14 | The above copyright notice and this permission notice shall be included in all 15 | copies or substantial portions of the Software. 16 | 17 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 18 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 19 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 20 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 21 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 22 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 23 | SOFTWARE. 24 | """ 25 | 26 | import os 27 | import sys 28 | import argparse 29 | 30 | 31 | class MetaFileParser: 32 | 33 | def __init__(self): 34 | self._flags = [] 35 | self._vars = {} 36 | self.file_buffer = None 37 | self.line_cnt = None 38 | 39 | def read_argparse(self, flags_list, var_list): 40 | self._flags = flags_list 41 | for attr, val in var_list: 42 | self._vars[attr] = val 43 | 44 | def parse_file(self, file_in): 45 | self.file_buffer = [] 46 | self.line_cnt = 0 47 | try: 48 | if not self._parse_file(file_in, True, 0): 49 | raise ValueError('Unexpected closing bracket') 50 | except ValueError as e: 51 | raise ValueError('Line %d: %s' % (self.line_cnt, e)) 52 | 53 | def _parse_file(self, file_in, do_print, depth): 54 | while True: 55 | try: 56 | line = next(file_in) 57 | self.line_cnt += 1 58 | dbl_bracket_start = line.find('[[') 59 | dbl_bracket_end = line.find(']]') 60 | if dbl_bracket_start != -1: 61 | condition = line[dbl_bracket_start + 2:] 62 | if do_print: 63 | do_print_ = self._eval_condition(condition) 64 | else: 65 | do_print_ = False 66 | self._parse_file(file_in, do_print_, depth+1) 67 | elif dbl_bracket_end != -1: 68 | return False 69 | else: 70 | if do_print: 71 | self._parse_line(line) 72 | except StopIteration: 73 | if depth == 0: 74 | return True 75 | else: 76 | raise ValueError('Unexpected end of file, expected ]]') 77 | 78 | def _parse_line(self, line): 79 | bracket_start = line.find('[') 80 | bracket_end = line.find(']') 81 | if bracket_start != -1 and bracket_end != -1: 82 | condition = line[bracket_start+1:bracket_end] 83 | line = line[:bracket_start] + line[bracket_end+1:] 84 | if not self._eval_condition(condition): 85 | return 86 | 87 | self.file_buffer.append(line) 88 | 89 | def _eval_condition(self, condition): 90 | cond_ = condition.replace('==', ' == ').replace('!=', ' != ') 91 | fields = cond_.split() 92 | return self._eval_fields(fields) 93 | 94 | def _eval_fields(self, fields): 95 | if 'or' in fields: 96 | idx = fields.index('or') 97 | left = self._eval_fields(fields[:idx]) 98 | right = self._eval_fields(fields[idx + 1:]) 99 | return left or right 100 | if 'and' in fields: 101 | idx = fields.index('and') 102 | left = self._eval_fields(fields[:idx]) 103 | right = self._eval_fields(fields[idx + 1:]) 104 | return left and right 105 | if len(fields) > 0 and fields[0] == 'not': 106 | return not self._eval_fields(fields[1:]) 107 | 108 | if len(fields) == 1: 109 | return self._eval_flag(fields[0]) 110 | 111 | if len(fields) == 3: 112 | if fields[1] in ['==', '!=', 'startswith', 'endswith', 'contains']: 113 | left = self._eval_expression(fields[0]) 114 | if fields[1] == '==': 115 | return left == fields[2] 116 | elif fields[1] == '!=': 117 | return left != fields[2] 118 | elif fields[1] == 'startswith': 119 | return left.startswith(fields[2]) 120 | elif fields[1] == 'endswith': 121 | return left.endswith(fields[2]) 122 | elif fields[1] == 'contains': 123 | return fields[2] in left 124 | 125 | raise ValueError('Syntax Error') 126 | 127 | def _eval_flag(self, name): 128 | if '%' in name: 129 | raise ValueError('Flags can\'t have default values') 130 | else: 131 | return name in self._flags 132 | 133 | def _eval_expression(self, name): 134 | if name == 'platform': 135 | return sys.platform 136 | else: 137 | return self._eval_var(name) 138 | 139 | def _eval_var(self, name): 140 | name_cleaned = name.split('%')[0] 141 | if name_cleaned not in self._vars: 142 | if '%' in name: 143 | return name.split('%')[1] 144 | else: 145 | raise ValueError('Custom variable \'%s\' neither set with command line options ' 146 | 'nor given default value' % name) 147 | else: 148 | return self._vars[name_cleaned] 149 | 150 | def write_result(self, file_out): 151 | for line in self.file_buffer: 152 | file_out.write(line) 153 | 154 | 155 | def main(): 156 | # Read command line arguments 157 | argparser = argparse.ArgumentParser() 158 | argparser.add_argument('-i', '--input', 159 | help='specify meta environment input file') 160 | argparser.add_argument('-o', '--output', 161 | help='specify environment output file') 162 | argparser.add_argument('-f', '--flag', type=str, action='append', default=[], 163 | help='set custom flag for parsing') 164 | argparser.add_argument('-v', '--variable', type=str, nargs=2, action='append', default=[], 165 | help='define custom variable for parsing') 166 | argparser.add_argument('-q', '--quiet', action='store_true', 167 | help='quietly overwrite output if already exists') 168 | argparser.add_argument('-p', '--parse-only', action='store_true', 169 | help='do not invoke conda afterwards') 170 | argparser.add_argument('-c', '--no-comment', action='store_true', 171 | help='do not add auto-generated comment to output') 172 | args = argparser.parse_args() 173 | 174 | # Create parser instance 175 | metafileparser = MetaFileParser() 176 | metafileparser.read_argparse(args.flag, args.variable) 177 | 178 | # Get input file 179 | dir_path = os.path.dirname(os.path.realpath(__file__)) 180 | if args.input is None: 181 | meta_file_name = 'environment.yml.meta' 182 | meta_file = os.path.join(dir_path, meta_file_name) 183 | if not os.path.exists(meta_file): 184 | meta_file = os.path.join(dir_path, '..', meta_file_name) 185 | else: 186 | meta_file_name = os.path.basename(args.input) 187 | meta_file = args.input 188 | 189 | if not os.path.exists(meta_file): 190 | sys.stderr.write('Error: Could not find input file') 191 | sys.exit(1) 192 | 193 | # Get output file 194 | if args.output is None: 195 | if meta_file.endswith('.meta'): 196 | env_file = meta_file[:-5] 197 | else: 198 | env_file = meta_file + '_out' 199 | else: 200 | env_file = args.output 201 | 202 | if os.path.exists(env_file) and not args.quiet: 203 | input_query = 'Output file %s already exists, overwrite ([y]/n)?' % env_file 204 | while True: 205 | try: 206 | input_ = raw_input 207 | except NameError: 208 | input_ = input 209 | answer = input_(input_query) 210 | if answer in ['', 'Y', 'y', 'yes', 'Yes', 'YES']: 211 | break 212 | elif answer in ['N', 'n', 'no', 'No', 'NO']: 213 | return 214 | input_query = 'Invalid answer, please answer yes or no ([y]/n):' 215 | 216 | # Read input 217 | with open(meta_file, 'r') as file_in: 218 | try: 219 | metafileparser.parse_file(file_in) 220 | except ValueError as e: 221 | sys.stderr.write('Error: %s\n' % e) 222 | sys.exit(1) 223 | 224 | # Write output 225 | with open(env_file, 'w') as file_out: 226 | if not args.no_comment: 227 | file_out.write('# This file was auto-generated from %s\n' % meta_file_name) 228 | metafileparser.write_result(file_out) 229 | 230 | sys.stdout.write('Created %s successfully\n' % env_file) 231 | 232 | # Invoke conda 233 | if not args.parse_only: 234 | os.system('conda env create -f %s' % env_file) 235 | 236 | 237 | if __name__ == '__main__': 238 | main() 239 | --------------------------------------------------------------------------------