├── README.md └── pyp3 /README.md: -------------------------------------------------------------------------------- 1 | #The Python3 Pyed Piper 2 | 3 | **Note:** All respect to original author Toby Rosen of Sony Imageworks. This page is the first section from https://code.google.com/archive/p/pyp/ 4 | 5 | ls | pyp "p.replace('maybe','yes') | pp.sort() | pp[1:3] |p , p , p.strip('abc') | whitespace | p[3], 'no' | p.upper() " 6 | 7 | ##Piping Python Through Pipes 8 | 9 | Pyp is a linux command line text manipulation tool similar to awk or sed, but which uses standard python string and list methods as well as custom functions evolved to generate fast results in an intense production environment. Pyed Pyper was developed at Sony Pictures Imageworks to facilitate the construction of complex image manipulation "one-liner" commands during visual effects work on Alice in Wonderland, Green Lantern, and the The Amazing Spiderman. 10 | 11 | Because pyp employs it's own internal piping syntax ("|") similar to unix pipes, complex operations can be proceduralized by feeding the output of one python command to the input of the next. This greatly simplifies the generation and troubleshooting of multistep operations without the use of temporary variables or nested parentheses. The variable "p" represents each line as a string, while "pp" is entire input as python list: \> ls | pyp "p[0] | pp.sort() | p + ' first letter, sorted!'" #gives sorted list of first letters of every line 12 | 13 | In practice, the ability to easily construct complicated command sequences can largely replace "for each" loops on the command line, thus significantly speeding up work-flow using standard unix command recycling. 14 | 15 | pyp output has been optimized for typical production scenarios. For example, if text is broken up into an array using the "split()" method, the output will be automatically numbered by field making selecting a particular field trivial. Numerous other conveniences have been included, such as an accessible history of all inter-pipe sub-results, an ability to perform mathematical operations, and a complement of variables based on common metacharcter split/join operations. 16 | 17 | For power users, commands can be easily saved and recalled from disk as macros, providing an alternative to quick and dirty scripting. For the truly advanced user, additional methods can be added to the pyp class via a config file, allowing tight integration with larger facilities data structures or custom toolsets. 18 | -------------------------------------------------------------------------------- /pyp3: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | #version 2.11.30 3 | #author tobyrosen@gmail.com 4 | 5 | #Copyright (c) 2011,2012, Sony Pictures Imageworks 6 | #Distributed under the New BSD License. 7 | #All rights reserved. 8 | #Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: 9 | #Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. 10 | #Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation 11 | #and/or other materials provided with the distribution. 12 | #THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, 13 | #THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS 14 | #BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF 15 | #SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER 16 | #IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED 17 | #OF THE POSSIBILITY OF SUCH DAMAGE. 18 | 19 | 20 | import optparse 21 | import sys 22 | import os 23 | import time 24 | import json 25 | import glob 26 | import tempfile 27 | import datetime 28 | import getpass 29 | import re 30 | import math 31 | 32 | 33 | #try to import user customized classes if they exist. default is null class. 34 | try: 35 | from PypCustom import PypCustom 36 | except ImportError: 37 | class PypCustom(): 38 | pass 39 | 40 | try: 41 | from PypCustom import PowerPipeListCustom 42 | except ImportError : 43 | class PowerPipeListCustom(): 44 | pass 45 | 46 | try: 47 | from PypCustom import PypStrCustom 48 | except ImportError : 49 | class PypStrCustom(): 50 | pass 51 | 52 | try : 53 | from PypCustom import PypListCustom 54 | except ImportError: 55 | class PypListCustom(): 56 | pass 57 | 58 | try: 59 | from PypCustom import PypFunctionCustom 60 | except ImportError: 61 | class PypFunctionCustom(): 62 | pass 63 | 64 | 65 | class Colors(object): 66 | '''defines basic color scheme''' 67 | OFF = chr(27) + '[0m' 68 | RED = chr(27) + '[31m' 69 | GREEN = chr(27) + '[32m' 70 | YELLOW = chr(27) + '[33m' 71 | MAGENTA = chr(27) + '[35m' 72 | CYAN = chr(27) + '[36m' 73 | WHITE = chr(27) + '[37m' 74 | BLUE = chr(27) + '[34m' 75 | BOLD = chr(27) + '[1m' 76 | COLORS = [OFF, RED, GREEN, YELLOW, MAGENTA, CYAN, WHITE, BLUE, BOLD] 77 | 78 | class NoColors(object): 79 | '''defines basic null color scheme''' 80 | OFF = '' 81 | RED = '' 82 | GREEN ='' 83 | YELLOW = '' 84 | MAGENTA = '' 85 | CYAN = '' 86 | WHITE ='' 87 | BLUE = '' 88 | BOLD = '' 89 | COLORS = [OFF, RED, GREEN, YELLOW, MAGENTA, CYAN, WHITE, BLUE, BOLD] 90 | 91 | 92 | 93 | 94 | class HistoryObject(object): 95 | ''' 96 | wrapped object that allows history to be stored in container. Used for keeping history 97 | when rearrangeing pp lists 98 | ''' 99 | def __init__(self, wrapped_obj): 100 | self.__wrapped = wrapped_obj 101 | self.current = 0 102 | 103 | def wrapped_object(self): 104 | return self.__wrapped 105 | 106 | def set_wrapped_object(self, new_value): 107 | self.__wrapped = new_value 108 | 109 | 110 | def __len__(self): 111 | try: 112 | length = len(self.__wrapped_obj) 113 | except : #bouleons etc 114 | length = 1 115 | 116 | return length 117 | 118 | def __getitem__(self, index): 119 | return self.__wrapped[index] 120 | 121 | def __getattr__(self, name): 122 | return getattr(self.__wrapped, name) 123 | 124 | def __lt__(self, other): 125 | return self.__wrapped < other.__wrapped 126 | 127 | def __str__(self): 128 | return str(self.__wrapped) 129 | 130 | def __contains__(self, item): 131 | if item in self.__wrapped: 132 | return True 133 | else: 134 | return False 135 | 136 | class PowerPipeList(list,PowerPipeListCustom): 137 | ''' 138 | defines pp object, allows manipulation of entire input using python list methods 139 | ''' 140 | def __init__(self, *args): 141 | super(PowerPipeList, self).__init__(*args) 142 | try: 143 | PowerPipeListCustom.__init__(self) 144 | except AttributeError: 145 | pass 146 | self.pyp = Pyp() 147 | 148 | def stdDev(self,inputs,mean): 149 | sumsq=0.0 150 | for i in range(len(inputs)): 151 | sumsq += (float(inputs[i])-mean) **2 152 | return math.sqrt(sumsq/len(inputs)) 153 | 154 | 155 | def stats(self): 156 | 157 | float_pp = [float(x.wrapped_object()) for x in self if x.wrapped_object()] 158 | 159 | max_pp = max(float_pp) 160 | min_pp = min(float_pp) 161 | 162 | n_pp = len(float_pp) 163 | sum_pp = sum(float_pp) 164 | mean_pp = sum_pp / n_pp 165 | 166 | 167 | stddev_pp= self.stdDev(float_pp, mean_pp) 168 | 169 | stat_pp =[] 170 | 171 | for p in self: 172 | if p.wrapped_object(): 173 | p.type = dict 174 | stat_dict = {} 175 | stat_dict['original'] = float(p.wrapped_object()) 176 | stat_dict['max'] = max_pp 177 | stat_dict['min'] = min_pp 178 | stat_dict['n'] = n_pp 179 | stat_dict['sum'] = sum_pp 180 | stat_dict['mean'] = mean_pp 181 | stat_dict['stddev'] = stddev_pp 182 | 183 | p.set_wrapped_object(stat_dict) 184 | stat_pp.append(p) 185 | 186 | return stat_pp 187 | 188 | 189 | 190 | 191 | 192 | def divide(self, n_split): 193 | ''' 194 | splits list into subarrays with n_split members 195 | @param n_split: number of members produced by split 196 | @type n_split: int 197 | @return : new array split up by n_split 198 | @rtype : list 199 | ''' 200 | sub_out = [] 201 | out = [] 202 | n = 0 203 | pyp = Pyp() 204 | inputs = self.pyp.flatten_list(self) 205 | 206 | while inputs: 207 | input = inputs.pop(0) 208 | n = n + 1 209 | sub_out.append(input) 210 | if not n % n_split or not inputs: 211 | out.append([sub_out]) 212 | sub_out = [] 213 | 214 | return out 215 | 216 | def delimit(self, delimiter): 217 | ''' 218 | splits up array based on delimited instead of newlines 219 | @param delimiter: delimiter used for split 220 | @type delimiter: str 221 | @return: new string split by delimiter and joined by ' ' 222 | @rtype: list 223 | ''' 224 | return ' '.join(self.pyp.flatten_list(self)).split(delimiter) 225 | 226 | def oneline(self,delimiter = ' '): 227 | ''' 228 | combines list to one line with optional delimeter 229 | @param delimiter: delimiter used for joining to one line 230 | @type delimiter: str 231 | @return: one line output joined by delimiter 232 | @rtype: list 233 | ''' 234 | 235 | flat_list = self.flatten_list(self) 236 | return delimiter.join(flat_list) 237 | 238 | def uniq(self): 239 | ''' 240 | returns only unique elements from list 241 | @return: unique items 242 | @rtype: list 243 | ''' 244 | strings= self.pyp.flatten_list(self) 245 | 246 | return list(set(strings)) 247 | 248 | def flatten_list(self, iterables): 249 | ''' 250 | returns a list of strings from nested lists 251 | @param iterables: nested lists containing strs or PypStrs 252 | @type iterables: list 253 | @return: unnested list of strings 254 | @rtype: list 255 | ''' 256 | return self.pyp.flatten_list(iterables) 257 | 258 | def unlist(self): 259 | ''' 260 | splits a list into one element per line 261 | @param self: nested list 262 | @type self: list 263 | @return: unnested list 264 | @rtype: list 265 | ''' 266 | return self.pyp.flatten_list(self) 267 | 268 | 269 | 270 | def contains(self,target,input): 271 | if '__contains__' in dir(input): 272 | if target in input: 273 | return True 274 | 275 | else: 276 | if target == input: 277 | return True 278 | 279 | return False 280 | 281 | def after(self, target, after_n=1): 282 | ''' 283 | consolidates after_n lines after matching target text to 1 line 284 | @param target: target string to find 285 | @type target: str 286 | @param after_n: number of lines to consolidate 287 | @type after_n: int 288 | @return: list of after_n members 289 | @rtype: list 290 | ''' 291 | out = [] 292 | n = 0 293 | inputs = self.pyp.flatten_list(self) 294 | for input in inputs: 295 | n = n + 1 296 | if self.contains(target,input): 297 | out.append([ [input] + inputs[n:n + after_n] ]) 298 | 299 | return out 300 | 301 | def before(self, target, before_n=1): 302 | ''' 303 | consolidates before_n lines before matching target text to 1 line 304 | @param target: target string to find 305 | @type target: str 306 | @param before_n: number of lines to consolidate 307 | @type before_n: int 308 | @return: list of before_n members 309 | @rtype: list 310 | ''' 311 | out = [] 312 | n = 0 313 | inputs = self.pyp.flatten_list(self) 314 | for input in inputs: 315 | n = n + 1 316 | if self.contains(target,input): 317 | out.append([ [input] + inputs[n - before_n - 1:n - 1] ]) 318 | return out 319 | 320 | 321 | 322 | 323 | def matrix(self, target, matrix_n=1): 324 | ''' 325 | consolidates matrix_n lines surrounding matching target text to 1 line 326 | @param target: target string to find 327 | @type target: str 328 | @param matrix_n: number of lines to consolidate 329 | @type matrix_n: int 330 | @return: list of matrix_n members 331 | @rtype: list 332 | ''' 333 | out = [] 334 | n = 0 335 | inputs = self.pyp.flatten_list(self) 336 | for input in inputs: 337 | n = n + 1 338 | if self.contains(target,input): 339 | out.append([ inputs[n - matrix_n - 1:n - 1] + [input] + inputs[n:n + matrix_n] ]) 340 | return out 341 | 342 | 343 | 344 | class PypStr(str,PypStrCustom): 345 | ''' 346 | defines p string object, allows manipulation of input line by line using python 347 | string methods 348 | 349 | ''' 350 | def __init__(self, *args): 351 | super(PypStr, self).__init__() 352 | try: 353 | PypStrCustom.__init__(self) 354 | except AttributeError: 355 | pass 356 | 357 | 358 | try: 359 | self.dir = os.path.split(self.rstrip('/'))[0] 360 | self.file = os.path.split(self)[1] 361 | self.ext = self.split('.')[-1] 362 | p.max = 0 363 | except: 364 | pass 365 | 366 | def trim(self,delim='/'): 367 | ''' 368 | returns everything but the last directory/file 369 | @param self: directory path 370 | @type self: str 371 | @return: directory path missing without last directory/file 372 | @rtype: PypStr 373 | ''' 374 | return PypStr(delim.join(self.split(delim)[0:-1])) 375 | 376 | 377 | def kill(self, *args): 378 | ''' 379 | replaces to_kill with '' in string 380 | @param args: strings to remove 381 | @type args : strs 382 | @return: string without to_kill 383 | @rtype: PypStr 384 | ''' 385 | for arg in args: 386 | self = self.replace(arg, '') 387 | 388 | return PypStr(self) 389 | 390 | def letters(self): 391 | ''' 392 | returns only letters 393 | @return: list of strings with only letters 394 | @rtype: PypList 395 | ''' 396 | 397 | new_string='' 398 | for letter in list(self): 399 | if letter.isalpha(): 400 | new_string = new_string + letter 401 | else: 402 | new_string = new_string + ' ' 403 | return [PypStr(x) for x in new_string.split() if x] 404 | 405 | def punctuation(self): 406 | ''' 407 | returns only punctuation 408 | @return: list of strings with only punctuation 409 | @rtype: PypList 410 | ''' 411 | 412 | new_string='' 413 | for letter in list(self): 414 | if letter in """!"#$%&\'()*+,-./:;<=>?@[\\]^_`{|}~""": 415 | new_string = new_string + letter 416 | else: 417 | new_string = new_string + ' ' 418 | return [PypStr(x) for x in new_string.split() if x] 419 | 420 | def digits(self): 421 | ''' 422 | returns only digits 423 | @return: list of string with only digits 424 | @rtype: PypList 425 | ''' 426 | new_string='' 427 | for letter in list(self): 428 | if letter.isdigit(): 429 | new_string = new_string + letter 430 | else: 431 | new_string = new_string + ' ' 432 | return [PypStr(x) for x in new_string.split() if x] 433 | 434 | 435 | 436 | def clean(self,delim = '_'): 437 | ''' 438 | returns a metacharater sanitized version of input. ignores underscores and slashes and dots. 439 | @return: string with delim (default '_') replacing bad metacharacters 440 | @rtype: PypStr 441 | @param delim: delimeter to rejoin cleaned string with. default is "_" 442 | @type delime: str 443 | ''' 444 | 445 | for char in self: 446 | if not char.isalnum() and char not in ['/','.',delim]: 447 | self = self.replace(char, ' ') 448 | return PypStr(delim.join([x for x in self.split() if x.strip()])) 449 | 450 | def re(self,to_match,group=0): 451 | ''' 452 | returns characters that match a regex using to_match 453 | @return: portion of string that matches regex 454 | @rtype: PypStr 455 | @param to_match: regex used for matching 456 | @type to_match: str 457 | ''' 458 | 459 | match = re.search(to_match,self) 460 | if match: 461 | return PypStr(match.group(group)) 462 | else: 463 | return '' 464 | 465 | def rereplace(self, to_match, to_replace): 466 | ''' 467 | returns this string, with all occurrences of the regex to_match replaced with to_replace 468 | @return: this string, with regex-substitutions made 469 | @rtype: PypStr 470 | @param to_match: regex used for matching 471 | @type to_match: str 472 | @param to_replace: string to replace with - see re.sub for full syntax 473 | @type to_replace: str 474 | ''' 475 | return PypStr(re.sub(to_match, to_replace, self)) 476 | 477 | 478 | class PypList(list,PypListCustom): 479 | ''' 480 | defines p list object, allows manipulation of input line by line using python 481 | list methods 482 | ''' 483 | 484 | def __init__(self, *args): 485 | super(PypList, self).__init__(*args) 486 | try: 487 | PypListCustom.__init__(self) 488 | except AttributeError: 489 | pass 490 | class Pyp(object): 491 | ''' 492 | pyp engine. manipulates input stream using python methods 493 | @ivar history: master record of all manipulations 494 | @type history: dict 495 | @ivar pwd: current directory 496 | @type pwd: str 497 | @ivar p: current input line being manipulated 498 | @type p: str or list 499 | @ivar n: current input line number 500 | @type n: int 501 | ''' 502 | 503 | def __init__(self): 504 | self.history = {} #dictionary of all data organized input line by input line 505 | 506 | try: #occasionally, python loses pwd info 507 | self.pwd = os.getcwd() 508 | except: 509 | self.pwd ='' 510 | 511 | def get_custom_execute(self): 512 | '''returns customized paths to macro files if they are setup''' 513 | custom_ob = PypCustom() 514 | custom_attrs = dir(custom_ob) 515 | 516 | if 'custom_execute' in custom_attrs and custom_ob.custom_execute: 517 | final_execute = custom_ob.custom_execute 518 | else: 519 | final_execute = self.default_final_execute 520 | 521 | return final_execute 522 | 523 | def default_final_execute(self,cmds,null): 524 | ''' 525 | defines execute command 526 | @param cmds: list 527 | @param null: str 528 | ''' 529 | for cmd in cmds: 530 | os.system(cmd) 531 | 532 | 533 | def get_custom_macro_paths(self): 534 | '''returns customized paths to macro files if they are setup''' 535 | home = os.path.expanduser('~') 536 | custom_ob = PypCustom() 537 | custom_attrs = dir(custom_ob) 538 | 539 | if 'user_macro_path' in custom_attrs: 540 | user_macro_path = custom_ob.user_macro_path 541 | else: 542 | user_macro_path = home + '/pyp_user_macros.json' 543 | 544 | 545 | if 'group_macro_path' in custom_attrs: 546 | group_macro_path = custom_ob.group_macro_path 547 | else: 548 | group_macro_path = home + '/pyp_group_macros.json' 549 | 550 | return user_macro_path,group_macro_path 551 | 552 | def cmds_split(self, cmds, macros): 553 | ''' 554 | splits total commmand array based on pipes taking into account quotes, 555 | parantheses and escapes. returns array of commands that will be processed procedurally. 556 | Substitutes macros without executable commands. 557 | 558 | @param cmds: user supplied command set 559 | @type cmds: list 560 | @param macros: user defined marcros 561 | @type macros: dict 562 | @return: list of commands to be evaluated 563 | @rtype: list 564 | ''' 565 | cmds = cmds.strip('|') 566 | cmd_array = [] 567 | cmd = '' 568 | open_single = False 569 | open_double = False 570 | open_parenth = 0 571 | escape = False 572 | letters = list(cmds) 573 | while letters: 574 | letter = letters.pop(0) 575 | if cmd and cmd[-1] == '\\': escape = True 576 | 577 | #COUNTS QUOTES 578 | if letter == "'": 579 | if open_single and not escape: 580 | open_single = not open_single 581 | else: 582 | open_single = True 583 | if letter == '"': 584 | if open_double and not escape: 585 | open_double = not open_double 586 | else: 587 | open_double = True 588 | 589 | #COUNTS REAL PARANTHESES 590 | if not open_single and not open_double: 591 | if letter == '(' : 592 | open_parenth = open_parenth + 1 593 | if letter == ')': 594 | open_parenth = open_parenth - 1 595 | 596 | #MONEY MOVE--substitutes command for macro or starts building new command after adding command to cmd_array 597 | if cmd.strip() in macros and letter in ['|', '[', '%', ',', '+', ' ']: 598 | cmd = cmd.strip() 599 | letters = list('|'.join(macros[cmd]['command']) + letter + ''.join(letters)) 600 | cmd = '' 601 | elif letter == '|' and not open_single and not open_double and not open_parenth:# 602 | cmd_array.append(cmd) 603 | cmd = '' 604 | else: 605 | cmd = cmd + letter 606 | escape = False 607 | 608 | #for last command, either recursively run cmd_split or add last command to array 609 | if cmd.strip() in macros and not options.macro_save_name: #allows macro be split and also to be correctly overwritten 610 | return self.cmds_split('|'.join(cmd_array + macros[cmd]['command']), macros) #this is by definition the last cmd. 611 | else: 612 | cmd_array.append(cmd) #gets last cmd 613 | 614 | 615 | return [x for x in cmd_array if x.strip()] #gets rid of extra spaces and nulls 616 | 617 | def load_macros(self,macro_path): 618 | ''' 619 | loads macro file; returns macros dict 620 | @param macro_path: file path to macro file 621 | @type macro_path: str 622 | @return: dictionary of user defined macros 623 | @rtype: dict 624 | ''' 625 | #macro_path = self.macro_path 626 | if os.path.exists(macro_path): 627 | macro_ob = open(macro_path) 628 | macros = json.load(macro_ob) 629 | macro_ob.close() 630 | else: 631 | macros = {} 632 | 633 | for macro in macros: #unicode sucks 634 | macros[macro]['command'] = [str(x) for x in macros[macro]['command'] ] 635 | 636 | return macros 637 | 638 | def write_macros(self, macros,macro_path, cmds): 639 | ''' 640 | writes macro file 641 | @param macros: dictionary of user defined macros 642 | @type macros: dict 643 | @param macro_path: file path to macro file 644 | @type macro_path: str 645 | @param cmds: commands to be saved as a macro 646 | @type cmds: list 647 | ''' 648 | 649 | if options.macro_save_name: 650 | macro = options.macro_save_name 651 | macro_name = macro.split('#')[0].strip() 652 | macros[macro_name] = {} 653 | macros[macro_name]['command'] = cmds 654 | macros[macro_name]['user'] = getpass.getuser() 655 | macros[macro_name]['date'] =str(datetime.datetime.now()).split('.')[0] 656 | 657 | if '#' in macro: #deals with comments 658 | macros[macro_name]['comments'] = '#' + macro.split('#')[1].strip() 659 | else: 660 | macros[macro_name]['comments'] = '' 661 | macro_ob = open(macro_path, 'w') 662 | json.dump(macros, macro_ob) 663 | macro_ob.close() 664 | self.load_macros(macro_path) 665 | if macro_name in macros: 666 | print(Colors.YELLOW + macro_name , "successfully saved!" + Colors.OFF) 667 | sys.exit() 668 | else: 669 | print(Colors.RED + macro_name, 'was not saved...unknown error!' + Colors.OFF) 670 | sys.exit(1) 671 | 672 | def delete_macros(self, macros,macro_path): 673 | ''' 674 | deletes macro from file 675 | @param macros: dictionary of user defined macros 676 | @type macros: dict 677 | @param macro_path: file path to macro file 678 | @type macro_path: str 679 | ''' 680 | if options.macro_delete_name: 681 | if options.macro_delete_name in macros: 682 | del macros[options.macro_delete_name] 683 | json_ob = open(macro_path, 'w') 684 | json.dump(macros, json_ob) 685 | json_ob.close() 686 | print(Colors.MAGENTA + options.macro_delete_name + " macro has been successfully obliterated" + Colors.OFF) 687 | sys.exit() 688 | else: 689 | print(Colors.RED + options.macro_delete_name + " does not exist" + Colors.OFF) 690 | sys.exit(1) 691 | 692 | def list_macros(self, macros): 693 | ''' 694 | prints out formated macros, takes dictionary macros as input 695 | @param macros: dictionary of user defined macros 696 | @type macros: dict 697 | ''' 698 | if options.macro_list or options.macro_find_name: 699 | macros_sorted = [x for x in macros] 700 | macros_sorted.sort() 701 | for macro_name in macros_sorted: 702 | if options.macro_list or options.macro_find_name in macro_name or options.macro_find_name in macros[macro_name]['user']: 703 | print(Colors.MAGENTA + macro_name + '\n\t ' + Colors.YELLOW+macros[macro_name]['user'] \ 704 | + '\t' + macros[macro_name]['date']\ 705 | +'\n\t\t' + Colors.OFF + '"'\ 706 | + '|'.join(macros[macro_name]['command']) + '"' + Colors.GREEN + '\n\t\t'\ 707 | + macros[macro_name].get('comments', '') + Colors.OFF + '\n') 708 | sys.exit() 709 | 710 | def load_file(self): 711 | ''' 712 | loads file for pyp processing 713 | @return: file data 714 | @rtype: list 715 | ''' 716 | if options.text_file: 717 | if not os.path.exists(options.text_file): 718 | print(Colors.RED + options.text_file + " does not exist" + Colors.OFF) 719 | sys.exit() 720 | else: 721 | f = [x.rstrip() for x in open(options.text_file) ] 722 | return f 723 | else: 724 | return [] 725 | 726 | 727 | def shell(self, command): 728 | ''' 729 | executes a shell commands, returns output in array sh 730 | @param command: shell command to be evaluated 731 | @type command: str 732 | @return: output of shell command 733 | @rtype: list 734 | ''' 735 | sh = [x.strip() for x in os.popen(command).readlines()] 736 | return sh 737 | 738 | def shelld(self, command, *args): 739 | ''' 740 | executes a shell commands, returns output in dictionary based on args 741 | @param command: shell command to be evaluated 742 | @type command: str 743 | @param args: optional delimiter. default is ":". 744 | @type args: list 745 | @return: hashed output of shell command based on delimiter 746 | @rtype: dict 747 | 748 | ''' 749 | if not args: 750 | ofs = ':' 751 | else: 752 | ofs = args[0] 753 | shd = {} 754 | for line in [x.strip() for x in os.popen(command).readlines()]: 755 | try: 756 | key = line.split(ofs)[0] 757 | value = ofs.join(line.split(ofs)[1:]) 758 | shd[key] = value 759 | except IndexError: 760 | pass 761 | 762 | return shd 763 | 764 | 765 | def rekeep(self,to_match): 766 | ''' 767 | keeps lines based on regex string matches 768 | @param to_match: regex 769 | @type to_match: str 770 | @return: True if any of the strings match regex else False 771 | @rtype: bool 772 | ''' 773 | 774 | match = [] 775 | flat_p = self.flatten_list(self.p) 776 | for item in flat_p: 777 | if re.search(to_match,item): 778 | match.append(item) 779 | if match: 780 | return True 781 | else: 782 | return False 783 | 784 | def relose(self,to_match): 785 | ''' 786 | loses lines based on regex string matches 787 | @param to_match: regex 788 | @type to_match: str 789 | @return: False if any of the strings match regex else True 790 | @rtype: bool 791 | ''' 792 | 793 | return not self.rekeep(to_match) 794 | 795 | 796 | def keep(self,*args): 797 | ''' 798 | keeps lines based on string matches 799 | @param args: strings to search for 800 | @type args: list 801 | @return: True if any of the strings are found else False 802 | @rtype: bool 803 | ''' 804 | 805 | kept = [] 806 | for arg in args: 807 | flat_p = self.flatten_list(self.p) 808 | for item in flat_p: 809 | if arg in item: 810 | kept.append(arg) 811 | 812 | if kept: 813 | return True 814 | else: 815 | return False 816 | 817 | def lose(self,*args): 818 | ''' 819 | removes lines based on string matches 820 | @param args: strings to search for 821 | @type args: list 822 | @return: True if any of the strings are not found else False 823 | @rtype: bool 824 | ''' 825 | return not self.keep(*args) 826 | 827 | def array_tracer(self,input,analysis_mode,power_pipe=''): 828 | ''' 829 | generates colored, numbered output for lists and dictionaries and other types 830 | @param input: one line of input from evaluted pyp command 831 | @type input: any 832 | @param power_pipe: Output from powerpipe (pp) evaluation 833 | @type power_pipe: bool 834 | @return: colored output based on input contents 835 | @rtype: str 836 | ''' 837 | if not input and input not in [0,0.0]: #TRANSLATE FALSES TO EMPTY STRINGS OR ARRAYS. SUPPLIES DUMMY INPUT TO KEEP LINES IF NEEDED. 838 | if options.keep_false or power_pipe: 839 | input = ' ' 840 | else: 841 | return '' 842 | 843 | #BASIC VARIABLES 844 | nf = 0 845 | output = '' 846 | 847 | if power_pipe:#labels line number of powerpipes 848 | n_index = Colors.MAGENTA + '[%s]' % (self.n) + Colors.GREEN 849 | final_color = Colors.OFF 850 | else: #for line by line 851 | n_index = '' 852 | final_color='' 853 | 854 | 855 | #DEALS WITH DIFFERENT TYPES OF INPUTS 856 | 857 | input_type = type(input) 858 | 859 | if input_type in [ list, PypList, PowerPipeList] :#deals with lists 860 | for field in input: 861 | if not nf == len(input): 862 | if type(field) in [str, PypStr]: 863 | COLOR = Colors.GREEN 864 | else: 865 | COLOR = Colors.MAGENTA 866 | output = str(output) + Colors.BOLD + Colors.BLUE + "[%s]" % nf + Colors.OFF + COLOR + str(field) + Colors.GREEN 867 | nf = nf + 1 868 | final_output = n_index + Colors.GREEN + Colors.BOLD + '[' + Colors.OFF + output + Colors.GREEN + Colors.BOLD + ']' + Colors.OFF 869 | 870 | elif input_type in [str, PypStr] : #clean output for non-pp operations. this should have no color 871 | if analysis_mode: #put colors into output only if it's in analysis mode. 872 | n_index = Colors.GREEN 873 | final_color = Colors.OFF 874 | 875 | final_output = n_index + str(input) + final_color 876 | 877 | elif input_type in [int, float] : 878 | final_output = n_index + Colors.YELLOW + str(input) + Colors.OFF #had to comppromise here to keep string output clean. 879 | 880 | elif input_type is dict: #deals with dictionaries 881 | for field in sorted(input,key=lambda x : x): 882 | output = output + Colors.OFF + Colors.BOLD + Colors.BLUE + str(field) + Colors.GREEN + ": " + Colors.OFF + Colors.GREEN + str(input[field]) + Colors.BOLD + Colors.GREEN + ',\n ' 883 | final_output = n_index + Colors.GREEN + Colors.BOLD + '{' + output.strip().strip(' ,') + Colors.GREEN + Colors.BOLD + '}' + Colors.OFF 884 | 885 | else: #catches every else 886 | final_output = n_index + Colors.MAGENTA + str(input) + Colors.OFF 887 | 888 | return final_output 889 | 890 | def cmd_split(self, cmds): 891 | ''' 892 | takes a command (as previously split up by pipes and input as array cmds), 893 | and returns individual terms (cmd_array) that will be evaluated individually. 894 | Also returns a string_format string that will be used to stitch together 895 | the output with the proper spacing based on the presence of "+" and "," 896 | @param cmds: individual commands separated by pipes 897 | @type cmds: list 898 | @return: individual commands with corresponding string format 899 | @rtype: list 900 | ''' 901 | 902 | string_format = '%s' 903 | cmd_array = [] 904 | cmd = '' 905 | open_quote = False 906 | open_parenth = 0 907 | open_bracket = 0 908 | 909 | for letter in cmds: 910 | 911 | if letter in [ "'" , '"']: 912 | if cmd and cmd[-1] == '\\': 913 | open_quote = True 914 | else: 915 | open_quote = not open_quote 916 | 917 | if not open_quote: #this all ignores text in () or [] from being split by by , or + 918 | if letter == '(' : 919 | open_parenth = open_parenth + 1 920 | elif letter == ')': 921 | open_parenth = open_parenth - 1 922 | elif letter == '[' : 923 | open_bracket = open_bracket + 1 924 | elif letter == ']': 925 | open_bracket = open_bracket - 1 926 | 927 | if not open_parenth and not open_bracket and letter in [',', '+']: #these are actual formatting characters 928 | cmd_array.append(cmd) 929 | cmd = '' 930 | string_format = string_format + letter.replace('+', '%s').replace(',', ' %s') 931 | continue 932 | 933 | cmd = cmd + letter 934 | 935 | 936 | cmd_array.append(cmd) 937 | 938 | output = [(cmd_array, string_format)] 939 | return output 940 | 941 | def all_meta_split(self, input_str): 942 | ''' 943 | splits a string on any metacharacter 944 | @param input_str: input string 945 | @type input_str: str 946 | @return: list with no metacharacters 947 | @rtype: list 948 | ''' 949 | 950 | for char in input_str: 951 | if not char.isalnum(): 952 | input_str = input_str.replace(char, ' ') 953 | return [x for x in input_str.split() if x.strip()] 954 | 955 | def string_splitter(self): 956 | ''' 957 | splits self.p based on common metacharacters. returns a 958 | dictionary of this information. 959 | @return: input split up by common metacharacters 960 | @rtype: dict> 961 | ''' 962 | 963 | whitespace =self.p.split(None) 964 | slash =self.p.split('/') 965 | underscore =self.p.split('_') 966 | colon =self.p.split(':') 967 | dot =self.p.split('.') 968 | minus =self.p.split('-') 969 | all= self.all_meta_split(self.p) 970 | comma = self.p.split(',') 971 | tab = self.p.split('\t') 972 | backslash = self.p.split('\\') 973 | 974 | 975 | split_variables_raw = { 976 | 977 | 'whitespace' :whitespace, 978 | 'slash' :slash, 979 | 'underscore' :underscore, 980 | 'colon' :colon, 981 | 'dot' :dot, 982 | 'minus' :minus, 983 | 'all' : all, 984 | 'comma' : comma, 985 | 'tab': tab, 986 | 'backslash': backslash, 987 | 988 | 'w' :whitespace, 989 | 's' :slash, 990 | 'u' :underscore, 991 | 'c' :colon, 992 | 'd' :dot, 993 | 'm' :minus, 994 | 'a' : all, 995 | 'mm' : comma, 996 | 't': tab, 997 | 'b': backslash, 998 | } 999 | #gets rid of empty fields 1000 | split_variables = dict((x, PypList([PypStr(y) for y in split_variables_raw[x]])) for x in split_variables_raw) 1001 | return split_variables 1002 | 1003 | def join_and_format(self, join_type): 1004 | ''' 1005 | joins self.p arrays with a specified metacharacter 1006 | @param join_type: metacharacter to join array 1007 | @type join_type: str 1008 | @return: string joined by metacharacter 1009 | @rtype: str 1010 | ''' 1011 | 1012 | temp_joins = [] 1013 | derived_string_format = self.history[self.n]['string_format'][-1] 1014 | len_derived_str_format = len(derived_string_format.strip('%').split('%')) 1015 | 1016 | if len(self.p) == len_derived_str_format: 1017 | string_format = derived_string_format #normal output 1018 | for sub_p in self.p: 1019 | if type(sub_p) in [list, PypList]: 1020 | temp_joins.append(join_type.join(sub_p)) 1021 | else: #deals with printing lists and strings 1022 | temp_joins.append(sub_p) 1023 | 1024 | return PypStr(string_format % tuple(temp_joins)) 1025 | 1026 | else: #deals with piping pure arrays to p 1027 | return PypStr(join_type.join(PypStr(x)for x in self.p)) 1028 | 1029 | def array_joiner(self): 1030 | ''' 1031 | generates a dict of self.p arrays joined with various common metacharacters 1032 | @return: input joined by common metacharacters 1033 | @rtype: dict 1034 | ''' 1035 | whitespace = self.join_and_format(' ') 1036 | slash = self.join_and_format(os.sep) 1037 | underscore = self.join_and_format('_') 1038 | colon = self.join_and_format(':') 1039 | dot = self.join_and_format('.') 1040 | minus = self.join_and_format('-') 1041 | all = self.join_and_format(' ') 1042 | comma = self.join_and_format(',') 1043 | tab = self.join_and_format('\t') 1044 | backslash = self.join_and_format('\\') 1045 | 1046 | join_variables = { 1047 | 'w' : whitespace, 1048 | 's' : slash, 1049 | 'u' : underscore, 1050 | 'c' : colon, 1051 | 'd' : dot, 1052 | 'm' : minus, 1053 | 'a' : all, 1054 | 'mm' : comma, 1055 | 't': tab, 1056 | 'b':backslash, 1057 | 1058 | 'whitespace' : whitespace, 1059 | 'slash' : slash, 1060 | 'underscore' : underscore, 1061 | 'colon' : colon, 1062 | 'dot' : dot, 1063 | 'minus' : minus, 1064 | 'all' : all, 1065 | 'comma' : comma, 1066 | 'tab': tab, 1067 | 'backslash' : backslash, 1068 | 1069 | 1070 | } 1071 | 1072 | return join_variables 1073 | 1074 | 1075 | 1076 | def generic_variables(self): 1077 | ''' 1078 | generates a dict of self.p arrays joined with various common metacharacters 1079 | @return: input joined by common metacharacters 1080 | @rtype: dict 1081 | ''' 1082 | 1083 | string_input = str(self.p) 1084 | whitespace = string_input 1085 | slash =string_input 1086 | underscore = string_input 1087 | colon = string_input 1088 | dot = string_input 1089 | minus =string_input 1090 | all = string_input 1091 | comma = string_input 1092 | tab = string_input 1093 | backslash = string_input 1094 | 1095 | generic_variables = { 1096 | 'w' : whitespace, 1097 | 's' : slash, 1098 | 'u' : underscore, 1099 | 'c' : colon, 1100 | 'd' : dot, 1101 | 'm' : minus, 1102 | 'a' : all, 1103 | 'mm' : comma, 1104 | 't': tab, 1105 | 'b':backslash, 1106 | 1107 | 'whitespace' : whitespace, 1108 | 'slash' : slash, 1109 | 'underscore' : underscore, 1110 | 'colon' : colon, 1111 | 'dot' : dot, 1112 | 'minus' : minus, 1113 | 'all' : all, 1114 | 'comma' : comma, 1115 | 'tab': tab, 1116 | 'backslash' : backslash, 1117 | 1118 | 1119 | } 1120 | 1121 | return generic_variables 1122 | 1123 | def translate_preset_variables(self, translate_preset_variables,file_input, second_stream_input): 1124 | ''' 1125 | translates variables to protected namespace dictionary for feeding into eval command. 1126 | @param file_input: data from file 1127 | @type file_input: list 1128 | @param second_stream_input: input from second stream 1129 | @type second_stream_input: list 1130 | @return: values of preset variable for direct use by users 1131 | @rtype: dict 1132 | ''' 1133 | 1134 | 1135 | #generic variables 1136 | presets = { 1137 | 'n' : self.kept_n, 1138 | 'on' : self.n, 1139 | 'fpp' : file_input, 1140 | 'spp' : second_stream_input, 1141 | 'nk': 1000 + self.kept_n, 1142 | 'shell': self.shell, 1143 | 'shelld' : self.shelld, 1144 | 'keep': self.keep, 1145 | 'lose': self.lose, 1146 | 'k': self.keep, 1147 | 'l':self.lose, 1148 | 'rekeep':self.rekeep, 1149 | 'relose':self.relose, 1150 | 'rek':self.rekeep, 1151 | 'rel':self.relose, 1152 | 'quote': '"', 1153 | 'apost':"'", 1154 | 'qu':'"', 1155 | 'dollar': '$', 1156 | 'pwd': self.pwd, 1157 | 'date': datetime.datetime.now(), 1158 | 'math':math, 1159 | 'env': os.environ.get, 1160 | 'glob' : glob.glob, 1161 | 'letters': 'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ', 1162 | 'digits': '0123456789', 1163 | 'punctuation': """!"#$%&\'()*+,-./:;<=>?@[\\]^_`{|}~""", 1164 | 'str':(PypStr) 1165 | } 1166 | 1167 | #removes nested entries from history 1168 | history = [] 1169 | for hist in self.history[self.n]['history']: 1170 | if type(hist) in (list,PypList): 1171 | hist = self.unlist_p(hist) 1172 | history.append(hist) 1173 | presets['history'] = presets['h'] = history 1174 | 1175 | # file 1176 | if options.text_file: 1177 | try: 1178 | fp = file_input[self.kept_n] 1179 | except IndexError: 1180 | fp = '' 1181 | presets['fp'] = fp 1182 | 1183 | # second stream 1184 | try: 1185 | sp = second_stream_input[self.kept_n] 1186 | except IndexError: 1187 | sp = '' 1188 | 1189 | presets['sp'] = sp 1190 | 1191 | #original input 1192 | if self.history[self.n]['output']: 1193 | presets['o'] = self.history[self.n]['history'][0] 1194 | 1195 | else: 1196 | presets['o'] = '' 1197 | presets['original'] = presets['o'] 1198 | 1199 | # p cleanup 1200 | p = self.p 1201 | if type(p) in [str]: 1202 | presets['p'] = PypStr(p) 1203 | elif type(p) in [list]: 1204 | presets['p'] = PypList(p) 1205 | else: 1206 | presets['p'] = p 1207 | 1208 | #custom functions 1209 | presets.update(PypFunctionCustom.__dict__) #adds user defined functions 1210 | return presets 1211 | 1212 | def blank_history(self): 1213 | ''' 1214 | initializes history dict for a particular n, 1215 | where n is the line of the input being processed 1216 | ''' 1217 | history= {} #creates dict 1218 | history['error'] = '' # error data 1219 | history['history'] = [] # 1220 | #history['history'].append(self.p) # records current p 1221 | history['string_format'] = [] #used for formating output 1222 | history['original_splits'] = {}#dict of original splits 1223 | history['output'] = True 1224 | return history 1225 | 1226 | 1227 | def safe_eval(self, cmd, variables): 1228 | ''' 1229 | evaluates a str command (cmd) safely. takes a dictionary of protected 1230 | namespace variables as input.returns output of python call, which can 1231 | be any type of python data type (typically str or array though). 1232 | @param cmd: command to be evaluated using python 1233 | @type cmd: str 1234 | @param variables: preset variables used for evaluation 1235 | @type variables: dictionary 1236 | @return: output from python evaluation 1237 | @rtype: list 1238 | ''' 1239 | if not self.history[self.n]['error'] and self.history[self.n]['output'] is not False: #if no errors, go forward 1240 | total_output = [] 1241 | for cm_tuple in self.cmd_split(cmd):#cm_tuple consists of commands and string format.cmd_split splits each command into terms. 1242 | string_format = cm_tuple[1] #how the results will eventually be formatted. 1243 | for cm in cm_tuple[0]:#cm is the expression seperated by a + or a , 1244 | #evaluate cm and add to dictionary catching any error. 1245 | try: 1246 | output = eval(cm, variables) #500 lines of code wrap this line!!! 1247 | except KeyboardInterrupt: 1248 | print(Colors.RED + "killed by user" + Colors.OFF) 1249 | sys.exit() 1250 | except IndexError: #ignores index errors like all other shell commands 1251 | self.history[self.n]['string_format'].append('%s') 1252 | return [''] # this is total_output. 1253 | except Exception as err: 1254 | self.history[self.n]['error'] = Colors.RED + 'error: ' + str(err) + Colors.OFF, Colors.RED + cmd + Colors.OFF 1255 | self.history[self.n]['string_format'].append('%s')#adds default string format if error 1256 | return [''] 1257 | break 1258 | #totals output for each cm 1259 | try: 1260 | if output is True : #allows truth tests 1261 | output = self.p 1262 | except: 1263 | pass 1264 | total_output.append(output) 1265 | self.history[self.n]['string_format'].append(string_format) 1266 | 1267 | 1268 | return total_output 1269 | 1270 | 1271 | def get_user_input(self, total_output,second_stream_input,file_input, power_pipe): 1272 | ''' 1273 | figures out what to show user in terms of powerpipe output. does NOT update history dictionary. 1274 | 1275 | @param total_output: line output from eval 1276 | @type total_output: list 1277 | @param second_stream_input: entire input from second string 1278 | @type second_stream_input: list 1279 | @param file_input: entire input from file 1280 | @type file_input: list 1281 | @param power_pipe: kind of power pipe 1282 | @type power_pipe: string 1283 | @return: output for display 1284 | @rtype: list 1285 | ''' 1286 | 1287 | try: #who knows what could happen with user input 1288 | n = self.n 1289 | if power_pipe == 'pp' and total_output or not power_pipe: #standard input 1290 | user_output = total_output 1291 | elif power_pipe == 'spp' and second_stream_input: 1292 | user_output = [second_stream_input[n]] 1293 | elif power_pipe == 'fpp' and file_input: 1294 | user_output = [file_input[n]] 1295 | elif power_pipe: # power pipe variable is referenced, but does not exist. 1296 | print(Colors.RED + "YOU'RE LIST VARIABLE DOES NOT EXIST: " + Colors.GREEN + power_pipe + Colors.OFF) 1297 | sys.exit() 1298 | except: #default output is null per line 1299 | user_output =[''] 1300 | return user_output 1301 | 1302 | def update_history(self, total_output,second_stream_input,file_input, power_pipe): 1303 | ''' 1304 | updates history dictionary with output from python evaluation 1305 | @param total_output: line output from eval 1306 | @type total_output: list 1307 | @param second_stream_input: entire input from second string 1308 | @type second_stream_input: list 1309 | @param file_input: entire input from file 1310 | @type file_input: list 1311 | @param power_pipe: kind of power pipe 1312 | @type power_pipe: string 1313 | ''' 1314 | #marks null output as False EXCEPT when output is zero, powerpipe. Null output is not printed out. 1315 | if (total_output in [False]\ 1316 | or [x for x in self.flatten_list(total_output) if x is False ] == total_output )\ 1317 | and not power_pipe: 1318 | 1319 | #make irrelevent data false 1320 | self.history[self.n]['history'].append(False) 1321 | self.history[self.n]['output']=False 1322 | else: # good output 1323 | string_format = self.history[self.n]['string_format'][-1] 1324 | output_array = [] 1325 | history_array = [] 1326 | contains_list = False 1327 | #actual output is p or pp unless spp is invoked. 1328 | user_input = self.get_user_input(total_output, second_stream_input, file_input, power_pipe) 1329 | 1330 | #update history array 1331 | 1332 | contains_list = False 1333 | for out in total_output: # forms an array called_output array of strings or array_traced strings 1334 | history_array.append(out) # for feeding back to pipe 1335 | if type(out) in [list, PypList]: 1336 | contains_list = True 1337 | 1338 | contains_number = True if [x for x in total_output if type(x) in [int,float]] == total_output else False 1339 | contains_dict = True if [x for x in total_output if type(x) in [dict]] == total_output else False 1340 | #toggle analysis mode if not all input is strings 1341 | 1342 | if [x for x in total_output if type(x) in [str,PypStr]] == total_output: 1343 | analysis_mode = False 1344 | else: 1345 | analysis_mode = True 1346 | 1347 | #update actual output 1348 | for out in user_input: 1349 | traced_output = self.array_tracer(out,analysis_mode, power_pipe) 1350 | output_array.append(traced_output) # for output 1351 | 1352 | self.history[self.n]['output'] = string_format % (tuple(output_array)) 1353 | 1354 | if contains_list or contains_number or contains_dict:# or format_mismatch: #this section prevents buildup of recursive lists. 1355 | self.history[self.n]['history'].append(total_output) # just adds list to total output if list 1356 | else: 1357 | self.history[self.n]['history'].append(string_format % (tuple(history_array))) # adds properly formatted string if string. 1358 | 1359 | def flatten_list(self, iterables): 1360 | ''' 1361 | returns a list of strings from nested lists 1362 | @param iterables: nested list to flatten 1363 | @type iterables: list 1364 | ''' 1365 | out = [] 1366 | if '__iter__' in dir(iterables) or type(iterables) in [list, PowerPipeList,tuple,PypList]: 1367 | #expand history objects 1368 | if [x for x in iterables if type(x) in [HistoryObject]]: 1369 | 1370 | expanded_iterables = [] 1371 | for iterable in iterables: 1372 | if type(iterable) in [HistoryObject]: 1373 | iterable = iterable.wrapped_object() 1374 | expanded_iterables.append(iterable) 1375 | iterables = expanded_iterables 1376 | #if [x for x in iterables if type(x) in [str, PypStr,HistoryObject,type,int,float]]: 1377 | 1378 | if [x for x in iterables if type(x) not in [list,PowerPipeList,tuple,PypList]]: #str,int,etc 1379 | out = out + iterables #add the lists and be done 1380 | 1381 | else: 1382 | for x in iterables: 1383 | out = out + self.flatten_list(x) 1384 | else: #catches non iterables 1385 | out = [iterables] #non-iterables 1386 | return out 1387 | 1388 | 1389 | 1390 | def stamp_history(self,power_pipe_inputs, new_inputs): 1391 | ''' 1392 | makes new object stamped with all history info 1393 | @param power_pipe_inputs: PowerPipeList 1394 | @param new_inputs: Booleon 1395 | ''' 1396 | history_inputs= [] 1397 | input_index = 0 1398 | 1399 | for input in power_pipe_inputs: 1400 | 1401 | input_type = type(input) 1402 | input_copy = input 1403 | input = HistoryObject(input) 1404 | if input_index in self.history and self.history[input_index]['output'] is False: #ignore false 1405 | input_index = input_index + 1 1406 | continue 1407 | elif not new_inputs and input_index in self.history: #stamps inputs already in history 1408 | input.history = self.history[input_index] #get from history dict 1409 | else: 1410 | input.history = self.blank_history() #new input. make fresh entry 1411 | input.history['history'] = [input_copy] 1412 | input.history['string_format'] = ['%s'] 1413 | input.type = input_type 1414 | history_inputs.append(input) 1415 | input_index = input_index + 1 1416 | return PowerPipeList(history_inputs) 1417 | 1418 | 1419 | def power_pipe_sum(self, ints): 1420 | ''' 1421 | overloaded sum operator. ignores non-digit strings. converts everything else to ints or floats 1422 | @param ints: varies 1423 | ''' 1424 | 1425 | new_ints = [] 1426 | for line in ints: 1427 | if 'history' in dir(line): 1428 | line = line.wrapped_object() 1429 | if type(line) in [int,float]: 1430 | new_ints.append(line) 1431 | elif type(line) in [str, PypStr]: 1432 | if line.isdigit(): 1433 | new_ints.append(int(line)) 1434 | elif line.replace('.','').isdigit(): 1435 | new_ints.append(float(line)) 1436 | return sum(new_ints) 1437 | 1438 | 1439 | 1440 | def power_pipe_eval(self, cmd, inputs, second_stream_input, file_input, power_pipe_type): 1441 | ''' 1442 | evaluates pp statement. returns sanitized result. 1443 | @param cmd: power pipe command 1444 | @type cmd: str 1445 | @param inputs: inputs from std-in or previous python eval 1446 | @type inputs: list 1447 | @param power_pipe_type: kind of powerpipe 1448 | @type power_pipe_type: str 1449 | @return: 'p' and output of python evaluation 1450 | @rtype: list 1451 | ''' 1452 | variables = {} 1453 | padded_output = [] 1454 | 1455 | variables['str'] = PypStr #useful for list comps 1456 | 1457 | inputs = self.flatten_list(inputs) 1458 | 1459 | 1460 | variables['pp'] = self.stamp_history(inputs, False) 1461 | variables['spp'] = self.stamp_history(second_stream_input, True) 1462 | variables['fpp'] = self.stamp_history(file_input, True) 1463 | variables['sum'] = self.power_pipe_sum 1464 | variables['math'] = math 1465 | 1466 | 1467 | 1468 | try: 1469 | output = eval(cmd, variables) #1000 lines of code wrap this line!!! 1470 | except KeyboardInterrupt: 1471 | print(Colors.RED + "killed by user" + Colors.OFF) 1472 | sys.exit() 1473 | except Exception as err: 1474 | print(Colors.RED + 'error: ' + str(err) + Colors.OFF, Colors.RED + cmd + Colors.OFF) 1475 | sys.exit() 1476 | 1477 | if output is None: #allows use of inplace methods like sort 1478 | output = variables[power_pipe_type] 1479 | 1480 | if type(output) in [int, float]: #keeps output in array 1481 | output = [output] 1482 | 1483 | if type(output) in [HistoryObject]: #makes sure output is in list of lists 1484 | output = [output] 1485 | 1486 | if type(output) in [str, PypStr,tuple,type]: #makes sure output is in list of lists 1487 | output = [[output]] 1488 | 1489 | if [x for x in output if type(x) in [tuple]]:#changes tuples to lists 1490 | output = [PypList(x) for x in output if type(x) in [tuple]] 1491 | 1492 | 1493 | 1494 | history_output = self.restore_history(output) 1495 | 1496 | if len(history_output) == 1: #turn off powerpipe if output is single item 1497 | power_pipe_type = '' 1498 | 1499 | return history_output, power_pipe_type 1500 | 1501 | 1502 | def restore_history(self, output): 1503 | ''' 1504 | restores history dictionary to reflect changes. recasts all objects to their original form. 1505 | ''' 1506 | 1507 | self.history = {} 1508 | # try to preserve history...if this fails, we go with zero history 1509 | output_index = 0 1510 | history_output = [] 1511 | for out in output: 1512 | if 'history' in dir(out): 1513 | history = out.history 1514 | out = self.recursive_recast(out) #recast back 1515 | else: #non stamped and deeply stamped both need new entries 1516 | out = self.recursive_recast(out) #recast right away 1517 | history = self.blank_history() #new input. make fresh entry 1518 | history['history'] = [self.unlist_p(out)] 1519 | history['string_format'] = ['%s'] 1520 | 1521 | history_output.append(out) 1522 | self.history[output_index] = history 1523 | output_index = output_index + 1 1524 | 1525 | return history_output 1526 | 1527 | 1528 | def recursive_recast(self, out): 1529 | ''' 1530 | takes nested list. recasts all objects to their original type if they are stamped 1531 | @param out: varies 1532 | ''' 1533 | try: 1534 | if 'type' in dir(out): #its stamped 1535 | return out.type(out) 1536 | elif type(out) in [list,PowerPipeList]: #try to return list 1537 | return [self.recursive_recast(x) for x in out] 1538 | 1539 | elif type(out) in [tuple]: 1540 | return ' '.join(out) #user is using commas to join items with spaces 1541 | else: #for everything else. 1542 | return out 1543 | except: 1544 | return out #who knows what people will try. 1545 | 1546 | 1547 | 1548 | 1549 | def detect_power_pipe(self, command, power_pipe_type): 1550 | ''' 1551 | detects presense of powerpipe 1552 | @param command: command to be evaluated 1553 | @type command: str 1554 | @param power_pipe_type: kind of powerpipe (future use) 1555 | @type power_pipe_type: str 1556 | @return: True if powerpipe else False 1557 | @rtype: bool 1558 | ''' 1559 | open_quote = False 1560 | cmd_raw = list(command) 1561 | cmd = [] 1562 | 1563 | for letter in cmd_raw: 1564 | if letter not in ['"', "'"] and not letter.isalnum(): 1565 | letter = ' ' 1566 | cmd.append(letter) 1567 | 1568 | 1569 | cmds = ''.join(cmd).split() 1570 | for cmd in cmds: 1571 | cmd = list(cmd) 1572 | test_cmd = '' 1573 | while cmd: 1574 | letter = cmd.pop(0) 1575 | test_cmd = test_cmd + letter 1576 | if not open_quote: 1577 | if power_pipe_type == test_cmd and not cmd: 1578 | return True 1579 | 1580 | if letter in [ "'" , '"']: 1581 | if cmd and cmd[0] == '\\': 1582 | open_quote = True 1583 | else: 1584 | open_quote = not open_quote 1585 | return False 1586 | 1587 | def format_input(self, cmd, input_set, second_stream_input, file_input): 1588 | ''' 1589 | checks for powerpipe presence, evaluates powerpipe pp and returns 1590 | formatted output if detected 1591 | @param cmd: user command 1592 | @type cmd: str 1593 | @param input_set: input from std-in or previous python evaluation 1594 | @type input_set: list 1595 | @return: command, input set, presence of powerpipe 1596 | @rtype: list 1597 | ''' 1598 | #POWER PIPES 1599 | 1600 | power_pipe = '' #power pipe is off by default 1601 | if self.detect_power_pipe(cmd, 'pp') : 1602 | input_set,power_pipe = self.power_pipe_eval( cmd, input_set, second_stream_input, file_input,'pp') 1603 | cmd = 'p' 1604 | elif self.detect_power_pipe(cmd, 'spp') : 1605 | second_stream_input, power_pipe = self.power_pipe_eval( cmd, input_set, second_stream_input, file_input,'spp') 1606 | cmd = 'p' 1607 | elif self.detect_power_pipe(cmd, 'fpp'): 1608 | file_input, power_pipe = self.power_pipe_eval( cmd, input_set, second_stream_input, file_input,'fpp') 1609 | cmd = 'p' 1610 | return cmd, input_set,second_stream_input, file_input, power_pipe 1611 | 1612 | def unlist_p(self, p): 1613 | ''' 1614 | changes one item arrays to strings for input cleanup 1615 | @param p: input from std-in or previous pyp evaluation 1616 | @type p: list 1617 | @return: will return a string if input is a list and has one member 1618 | @rtype: list,str 1619 | ''' 1620 | if type(p) in [list, PypList] and len(p) == 1: 1621 | p = p[0] 1622 | 1623 | return p 1624 | 1625 | 1626 | def process(self, inputs, file_input, cmds, second_stream_input, process_as_list): 1627 | ''' 1628 | takes primary data from input stream (can be string, array or dictionary), applies user commands to it, 1629 | captures this output. Also, generates several variables such as line counter and 1630 | various string split and join variables. Outputs string, array, or dictionary in 1631 | a format deliniated by the string formating stamp 1632 | @param inputs: inputs from std-in or previous pyp eval 1633 | @type inputs: str,list 1634 | @param file_input: inputs from file 1635 | @type file_input: list 1636 | @param cmds: python commands to be evaluated 1637 | @type cmds: list 1638 | @param second_stream_input: second stream input 1639 | @type second_stream_input: list 1640 | ''' 1641 | 1642 | while cmds: #cmds are commands that will be executed on the input stream 1643 | if process_as_list: #non powerpipe runs are handled remotely 1644 | self.kept_n = 0 # counter of kept lines. needs to be avail for eval, so starts as 0 1645 | self.n = -1 # overall line counter. will change to 0 asap. 1646 | 1647 | cmd = cmds.pop(0) 1648 | cmd, input_set,second_stream_input, file_input, power_pipe = self.format_input(cmd, inputs,second_stream_input, file_input) 1649 | 1650 | original_input_set = input_set[:] 1651 | 1652 | while input_set: 1653 | self.process_line(cmd,input_set, original_input_set,second_stream_input,file_input ,power_pipe) 1654 | 1655 | if process_as_list and self.history[self.n]['output'] != False: #increments line n 1656 | self.kept_n = self.kept_n + 1 #only update if output is kept 1657 | 1658 | 1659 | new_input = [self.history[x]['history'][-1] for x in self.history ] # takes last output as new input 1660 | 1661 | self.process(new_input, file_input, cmds, second_stream_input,process_as_list) #process new input 1662 | 1663 | 1664 | def process_line(self,cmd, input_set,original_input_set, second_stream_input,file_input ,power_pipe): 1665 | ''' 1666 | processes individual cmds per line. updates history and counters 1667 | @param inputs: inputs from std-in or previous pyp eval 1668 | @type inputs: str,list 1669 | @param file_input: inputs from file 1670 | @type file_input: list 1671 | @param cmds: python commands to be evaluated 1672 | @type cmds: list 1673 | @param second_stream_input: second stream input 1674 | @type second_stream_input: list 1675 | ''' 1676 | 1677 | self.p = self.unlist_p(input_set.pop(0)) # p is main line variable being manipulated 1678 | self.n = self.n + 1 # raises counters 1679 | variables = {} 1680 | 1681 | if self.p is False: #skip false output but n is updated 1682 | return 1683 | 1684 | if not self.n in self.history: # initializes self.history dict for line. always for quick out. 1685 | self.history[self.n] = self.blank_history() 1686 | self.history[self.n]['history'].append(self.p) 1687 | 1688 | if type(self.p) in [ str, PypStr]: # p is string 1689 | variables = self.string_splitter() 1690 | if not self.history[self.n]['original_splits']: #makes variables dealing with original input 1691 | self.history[self.n]['original_splits'] = dict(('o' + x, variables[x]) for x in variables) 1692 | if not self.history[self.n]['output']: #kills o variables if there is no output 1693 | self.history[self.n]['original_splits'] = dict(('o' + x, '') for x in variables) 1694 | elif type(self.p) in [list, PypList] and not power_pipe: # p is list of lists, constructs various joins 1695 | try: #for going from list to powerpipe 1696 | variables = self.array_joiner() 1697 | except: 1698 | pass 1699 | else: 1700 | variables = self.generic_variables() 1701 | 1702 | 1703 | variables.update(self.translate_preset_variables(original_input_set,file_input, second_stream_input)) #add incrementals 1704 | variables.update(self.history[self.n]['original_splits']) # updates with original splits 1705 | 1706 | total_output = self.safe_eval(cmd, variables) 1707 | self.update_history(total_output,second_stream_input,file_input ,power_pipe) 1708 | 1709 | 1710 | 1711 | 1712 | 1713 | def output(self, total_cmds): 1714 | ''' 1715 | generates final output. 1716 | @param total_cmds: all commands executed 1717 | @type total_cmds: list 1718 | ''' 1719 | execute_cmds = [] 1720 | for self.history_index in self.history: 1721 | error = self.history[self.history_index]['error'] 1722 | 1723 | if not error : #no error 1724 | 1725 | cmd = self.history[self.history_index]['output'] #color formated output 1726 | #if 1: ########################TESTING 1727 | if cmd: #kept commands 1728 | if options.execute or options.execute_aux: #executes command 1729 | execute_cmds.append(cmd) 1730 | else: 1731 | print(cmd) # normal output 1732 | elif options.keep_false: #prints blank lines for options outputs that evaluate as False ({}, [], False, (), '', etc) 1733 | print() 1734 | 1735 | 1736 | else: #error 1737 | print(Colors.RED + self.history[self.history_index]['error'][0] + Colors.RED + ' : ' + self.history[self.history_index]['error'][1] + Colors.OFF) 1738 | 1739 | 1740 | if execute_cmds: 1741 | if not options.execute_aux: 1742 | threads = 10 1743 | else: 1744 | threads = options.execute_aux 1745 | 1746 | self.final_execute(execute_cmds, threads) 1747 | 1748 | 1749 | def initilize_input(self): 1750 | ''' 1751 | decides what type of input to use (all arrays. can be from rerun file, yaml, or st-in. 1752 | also does some basic processing, for using different delimeters or looking for jts numbers 1753 | @return: starting input for pyp processing 1754 | @rtype: list 1755 | ''' 1756 | 1757 | if options.manual: 1758 | print(Docs.manual) 1759 | sys.exit() 1760 | 1761 | if options.unmodified_config: 1762 | print(Docs.unmodified_config) 1763 | sys.exit() 1764 | 1765 | rerun_path = '%s/pyp_rerun_%d.txt' %(tempfile.gettempdir(),os.getppid()) 1766 | 1767 | 1768 | if options.rerun: #deals with rerunning script with -r flag 1769 | if not os.path.exists(rerun_path): 1770 | gpid = int(os.popen("ps -p %d -oppid=" % os.getppid()).read().strip()) 1771 | rerun_gpid_path = '%s/pyp_rerun_%d.txt' %(tempfile.gettempdir() ,gpid) 1772 | if os.path.exists(rerun_gpid_path): 1773 | rerun_path = rerun_gpid_path 1774 | else: 1775 | print(Colors.RED + rerun_path + " does not exist" + Colors.OFF) 1776 | sys.exit() 1777 | pipe_input = open(rerun_path) 1778 | 1779 | elif options.blank_inputs: 1780 | pipe_input = [] 1781 | end_n = int(options.blank_inputs) 1782 | for n in range(0,end_n): 1783 | pipe_input.append('') 1784 | 1785 | 1786 | elif options.no_input: 1787 | pipe_input = [''] 1788 | 1789 | else: 1790 | pipe_input = sys.stdin 1791 | 1792 | if not pipe_input: 1793 | pipe_input = [''] #for using control d to activate comands with no input 1794 | 1795 | return pipe_input #is iterable file object or list 1796 | 1797 | 1798 | def input_filter(self,line): 1799 | if options.ignore_blanks: 1800 | if line.strip(): 1801 | return line.strip() 1802 | else: 1803 | return False 1804 | else: 1805 | return self.strip_last_newline(line) 1806 | 1807 | def strip_last_newline(self, line): 1808 | return line[::-1].replace('\n','',1)[::-1] #strips trailing newlines 1809 | 1810 | 1811 | def write_rerun(self, pipe_input): 1812 | if not options.rerun: 1813 | rerun_path = '%s/pyp_rerun_%d.txt' %(tempfile.gettempdir(),os.getppid()) 1814 | rerun_file = open(rerun_path, 'w') 1815 | rerun_file.write('\n'.join([str(x) for x in pipe_input])) 1816 | rerun_file.close() 1817 | 1818 | 1819 | 1820 | 1821 | def process_master_switch(self,inputs, file_input, cmds, second_stream_input): 1822 | ''' 1823 | switches between line by line and list operation based on presence of pp, spp, fpp 1824 | @param inputs: 1825 | @param file_input: 1826 | @param cmds: 1827 | @param second_stream_input: 1828 | ''' 1829 | 1830 | process_as_list = False 1831 | inputs_to_write = [] 1832 | for cmd in cmds: 1833 | for pp_type in ['pp','spp','fpp']: 1834 | if self.detect_power_pipe(cmd, pp_type): 1835 | process_as_list = True 1836 | 1837 | if options.execute or options.execute_aux: 1838 | process_as_list = True 1839 | 1840 | 1841 | if process_as_list: #normal operation allowing list operations 1842 | inputs_to_write = [] #safe copy 1843 | clean_inputs = [] 1844 | for input in inputs: 1845 | clean_input = self.input_filter(input) 1846 | if clean_input is not False: 1847 | clean_inputs.append(PypStr(clean_input)) 1848 | inputs_to_write.append(self.strip_last_newline(input)) 1849 | 1850 | self.process(clean_inputs, file_input, cmds, second_stream_input, process_as_list) 1851 | self.output(cmds) 1852 | 1853 | 1854 | else: #line by line mode eraseing memory after every print 1855 | #self.n = -1 # overall line counter. will change to 0 asap. 1856 | self.kept_n = 0 # counter of kept lines. needs to be avail for eval, so starts as 0 1857 | cmds_copy = cmds[:] 1858 | 1859 | for line in inputs : 1860 | cmds = cmds_copy[:] 1861 | line = self.input_filter(line) 1862 | 1863 | if line is not False: #catches lines filtered out by --ignore_blanks 1864 | line = PypStr(line) 1865 | inputs_to_write.append(self.strip_last_newline(line)) 1866 | self.process([line], file_input, cmds, second_stream_input,process_as_list) #recursive processing to generate history dict 1867 | if self.history and self.history[0]['output'] != False: #first part allows blank "" 1868 | self.kept_n = self.kept_n + 1 1869 | 1870 | self.output(cmds) #output text or execute commands from history dict 1871 | self.history = {} #new history every run 1872 | 1873 | 1874 | 1875 | self.write_rerun(inputs_to_write) 1876 | 1877 | 1878 | def main(self): 1879 | '''generates input stream based on file, std-in options, rerun, starts process loop, generates output''' 1880 | second_stream_input = [PypStr(x) for x in args[1:]] #2nd stream input 1881 | file_input = [PypStr(x) for x in self.load_file() ]# file input 1882 | 1883 | #load custom executer if possible. 1884 | self.final_execute=self.get_custom_execute() 1885 | 1886 | #load user and group macros. 1887 | user_macro_path,group_macro_path=self.get_custom_macro_paths() 1888 | user_macros = self.load_macros(user_macro_path) 1889 | group_macros = self.load_macros(group_macro_path) 1890 | group_macros.update(user_macros) #merges group and user macros 1891 | macros = group_macros 1892 | 1893 | #macros for action ie saving and deleting 1894 | action_macros = group_macros if options.macro_group else user_macros 1895 | action_macros_path = group_macro_path if options.macro_group else user_macro_path 1896 | 1897 | self.list_macros(macros) 1898 | self.delete_macros(action_macros, action_macros_path) 1899 | 1900 | 1901 | if not args: # default command is just to print. 1902 | cmds = ['p'] 1903 | else: 1904 | cmds = self.cmds_split(args[0], macros) 1905 | 1906 | self.write_macros(action_macros, action_macros_path, cmds) #needs cmds before we write macros 1907 | 1908 | inputs = self.initilize_input() #figure out our input stream 1909 | 1910 | self.process_master_switch(inputs, file_input, cmds, second_stream_input) 1911 | 1912 | 1913 | 1914 | 1915 | 1916 | 1917 | 1918 | class Docs(): 1919 | manual = ''' 1920 | =================================================================================== 1921 | PYED PIPER MANUAL 1922 | 1923 | pyp is a command line utility for parsing text output and generating complex 1924 | unix commands using standard python methods. pyp is powered by python, so any 1925 | standard python string or list operation is available. 1926 | 1927 | The variable "p" represents EACH line of the input as a python string, so for 1928 | example, you can replace all "FOO" with "GOO" using "p.replace('FOO','GOO')". 1929 | Likewise, the variable "pp" represents the ENTIRE input as a python array, so 1930 | to sort the input alphabetically line-by-line, use "pp.sort()" 1931 | 1932 | Standard python relies on whitespace formating such as indentions. Since this 1933 | is not convenient with command line operations, pyp employs an internal piping 1934 | structure ("|") similar to unix pipes. This allows passing of the output of 1935 | one command to the input of the next command without nested "(())" structures. 1936 | It also allows easy spliting and joining of text using single, commonsense 1937 | variables (see below). An added bonus is that any subresult between pipes 1938 | is available, making it easy to refer to the original input if needed. 1939 | 1940 | Filtering output is straightforward using python Logic operations. Any output 1941 | that is "True" is kept while anything "False" is eliminated. So "p.isdigit()" 1942 | will keep all lines that are completely numbers. 1943 | 1944 | The output of pyp has been optimized for typical command line scenarios. For 1945 | example, if text is broken up into an array using the "split()" method, the 1946 | output will be conveniently numbered by field because a field selection is 1947 | anticipated. If the variable "pp" is employed, the output will be numbered 1948 | line-by-line to facilitate picking any particular line or range of lines. In 1949 | both cases, standard python methods (list[start:end]) can be used to select 1950 | fields or lines of interest. Also, the standard python string and list objects 1951 | have been overloaded with commonly used methods and attributes. For example, 1952 | "pp.uniq()" returns all unique members in an array, and p.kill('foo') will 1953 | eliminate all "foo" in the input. 1954 | 1955 | pyp commands can be easily saved to disk and recalled using user-defined macros, 1956 | so a complicated parsing operation requiring 20 or more steps can be recalled 1957 | easily, providing an alternative to quick and dirty scripts. For more advanced 1958 | users, these macros can be saved to a central location, allowing other users to 1959 | execute them. Also, an additional text file (PypCustom.py) can be set up that 1960 | allows additional methods to be added to the pyp str and list methods, allowing 1961 | tight integration with larger facilities data structures or custom tool sets. 1962 | 1963 | ----------------------------------------------------------------------------------- 1964 | PIPING IN THE PIPER 1965 | ----------------------------------------------------------------------------------- 1966 | You can pipe data WITHIN a pyp statement using standard unix style pipes ("|"), 1967 | where "p" now represents the evaluation of the python statement before the "|". 1968 | You can also refer back to the ORIGINAL, unadulterated input using the variable 1969 | "o" or "original" at any time...and the variable "h" or "history" allows you 1970 | to refer back to ANY subresult generated between pipes ("|"). 1971 | 1972 | All pyp statements should be enclosed in double quotes, with single quotes being 1973 | used to enclose any strings.''' + Colors.YELLOW + ''' 1974 | 1975 | echo 'FOO IS AN ' | pyp "p.replace('FOO','THIS') | p + 'EXAMPLE'" 1976 | ==> THIS IS AN EXAMPLE''' + Colors.GREEN + ''' 1977 | 1978 | ----------------------------------------------------------------------------------- 1979 | THE TYPE OF COLOR IS THE TYPE 1980 | ----------------------------------------------------------------------------------- 1981 | pyp uses a simple color and numerical indexing scheme to help you identify what 1982 | kind of objects you are working with. Don't worry about the specifics right now, 1983 | just keep in mind that different types can be readily identified: 1984 | 1985 | strings: hello world 1986 | 1987 | integers or floats:''' + Colors.YELLOW + ''' 1984''' + Colors.GREEN + ''' 1988 | 1989 | split-up line: ''' + Colors.BOLD + ''' [''' + Colors.BLUE + '[0]' + Colors.OFF + Colors.GREEN \ 1990 | + 'hello' + Colors.BOLD + Colors.BLUE + '[1]' + Colors.OFF + Colors.GREEN + '''world''' +\ 1991 | Colors.BOLD + '] ' + Colors.OFF + Colors.GREEN + ''' 1992 | 1993 | entire input list: ''' +\ 1994 | Colors.MAGENTA + '[0]' + Colors.GREEN + 'first line\n' + Colors.OFF +\ 1995 | Colors.MAGENTA + ' [1]' + Colors.GREEN + 'second line' + Colors.OFF + Colors.GREEN + ''' 1996 | 1997 | dictionaries:''' + Colors.BOLD + ''' {''' + Colors.BLUE + 'hello world' + Colors.BOLD +\ 1998 | Colors.GREEN + ': ' + Colors.OFF + Colors.GREEN + '1984' + Colors.BOLD + '}' + Colors.OFF + Colors.GREEN + ''' 1999 | 2000 | other objects:''' + Colors.MAGENTA + ' RANDOM_OBJECT' + Colors.GREEN + ''' 2001 | 2002 | The examples below will use a yellow/blue color scheme to seperate them 2003 | from the main text however. Also, all colors can be removed using the 2004 | --turn_off_color flag. 2005 | 2006 | ----------------------------------------------------------------------------------- 2007 | STRING OPERATIONS 2008 | ----------------------------------------------------------------------------------- 2009 | Here is a simple example for splitting the output of "ls" (unix file list) on '.':''' + Colors.YELLOW + ''' 2010 | 2011 | ls random_frame.jpg | pyp "p.split('.')" 2012 | ==> [''' + Colors.BLUE + '[0]' + Colors.YELLOW + 'random_frame' + Colors.BLUE + '[1]' + Colors.YELLOW + 'jpg] ''' + Colors.GREEN + ''' 2013 | 2014 | The variable "p" represents each line piped in from "ls". Notice the output has 2015 | index numbers, so it's trivial to pick a particular field or range of fields, 2016 | i.e. pyp "p.split('.')[0]" is the FIRST field. There are some pyp generated 2017 | variables that make this simpler, for example the variable "d" or "dot" is the 2018 | same as p.split('.'):''' + Colors.YELLOW + ''' 2019 | 2020 | ls random_frame.jpg | pyp "dot" 2021 | ==> [''' + Colors.BLUE + '[0]' + Colors.YELLOW + 'random_frame' + Colors.BLUE + '[1]' + Colors.YELLOW + '''jpg] 2022 | 2023 | ls random_frame.jpg | pyp "dot[0]" 2024 | ==> random_frame''' + Colors.GREEN + ''' 2025 | 2026 | To Join lists back together, just pipe them to the same or another built-in 2027 | variable(in this case "u", or "underscore"):''' + Colors.YELLOW + ''' 2028 | 2029 | ls random_frame.jpg | pyp "dot" 2030 | ==> [''' + Colors.BLUE + '[0]' + Colors.YELLOW + 'random_frame' + Colors.BLUE + '[1]' + Colors.YELLOW + '''jpg] 2031 | 2032 | ls random_frame.jpg | pyp "dot|underscore" 2033 | ==> random_frame_jpg ''' + Colors.GREEN + ''' 2034 | 2035 | To add text, just enclose it in quotes, and use "+" or "," just like python: ''' + Colors.YELLOW + ''' 2036 | 2037 | ls random_frame.jpg | pyp "'mkdir seq.tp_' , d[0]+ '_v1/misc_vd8'" 2038 | ==> mkdir seq.tp_random_frame_v1/misc_vd8'" ''' + Colors.GREEN + ''' 2039 | 2040 | A fundamental difference between pyp and standard python is that pyp allows you 2041 | to print out strings and lists on the same line using the standard "+" and "," 2042 | notation that is used for string construction. This allows you to have a string 2043 | and then print out the results of a particular split so it's easy to pick out 2044 | your field of interest: ''' + Colors.YELLOW + ''' 2045 | 2046 | ls random_frame.jpg | pyp "'mkdir', dot" 2047 | ==> mkdir [''' + Colors.BLUE + '[0]' + Colors.YELLOW + 'random_frame' + Colors.BLUE + '[1]' + Colors.YELLOW + '''jpg] '''+ Colors.GREEN + ''' 2048 | 2049 | In the same way, two lists can be displayed on the same line using "+" or ",". 2050 | If you are trying to actually combine two lists, enclose them in parentheses:''' + Colors.YELLOW + ''' 2051 | 2052 | ls random_frame.jpg | pyp "(underscore + dot)" 2053 | ==> [''' + Colors.BLUE + '[0]' + Colors.YELLOW + 'random' + Colors.BLUE + '[1]' + Colors.YELLOW +'frame.jpg'\ 2054 | + Colors.BLUE + '[2]' + Colors.YELLOW + 'random_frame'+ Colors.BLUE + '[3]' + Colors.YELLOW + '''jpg] ''' + Colors.GREEN + ''' 2055 | 2056 | This behaviour with '+' and ',' holds true in fact for ANY object, making 2057 | it easy to build statements without having to worry about whether they 2058 | are strings or not. 2059 | 2060 | ----------------------------------------------------------------------------------- 2061 | ENTIRE INPUT LIST OPERATIONS 2062 | ----------------------------------------------------------------------------------- 2063 | To perform operations that operate on the ENTIRE array of std-in, Use the variable 2064 | "pp", which you can manipulate using any standard python list methods. For example, 2065 | to sort the input, use:''' + Colors.YELLOW + ''' 2066 | 2067 | pp.sort()''' + Colors.GREEN + ''' 2068 | 2069 | When in array context, each line will be numbered with it's index in the array, 2070 | so it's easy to, for example select the 6th line of input by using "pp[5]". 2071 | You can pipe this back to p to continue modifying this input on a 2072 | line-by-line basis: ''' + Colors.YELLOW + ''' 2073 | 2074 | pp.sort() | p ''' + Colors.GREEN + ''' 2075 | 2076 | You can add arbitrary entries to your std-in stream at this point using 2077 | list addition. For example, to add an entry to the start and end:''' + Colors.YELLOW + ''' 2078 | 2079 | ['first entry'] + pp + ['last entry'] ''' + Colors.GREEN + ''' 2080 | 2081 | The new pp will reflect these changes for all future operations. 2082 | 2083 | There are several methods that have been added to python's normal list methods 2084 | to facilitate common operations. For example, keeping unique members or 2085 | consolidating all input to a single line can be accomplished with: '''+ Colors.YELLOW + ''' 2086 | 2087 | pp.uniq() 2088 | pp.oneline()'''+ Colors.GREEN + ''' 2089 | 2090 | Also, there are a few useful python math functions that work on lists of 2091 | integers or floats like sum, min, and max. For example, to add up 2092 | all of the integers in the last column of input: '''+ Colors.YELLOW + ''' 2093 | 2094 | whitespace[-1] | int(p) | sum(pp) '''+ Colors.GREEN + ''' 2095 | 2096 | 2097 | ----------------------------------------------------------------------------------- 2098 | MATH OPERATIONS 2099 | ----------------------------------------------------------------------------------- 2100 | To perform simple math, use the integer or float functions (int() or float()) 2101 | AND put the math in "()" + ''' + Colors.YELLOW + ''' 2102 | 2103 | echo 665 | pyp "(int(p) + 1)" 2104 | ==> 666 ''' + Colors.GREEN + ''' 2105 | ----------------------------------------------------------------------------------- 2106 | LOGIC FILTERS 2107 | ----------------------------------------------------------------------------------- 2108 | To filter output based on a python function that returns a Booleon (True or False), 2109 | just pipe the input to this function, and all lines that return True will keep 2110 | their current value, while all lines that return False will be eliminated. ''' + Colors.YELLOW + ''' 2111 | 2112 | echo 666 | pyp "p.isdigit()" 2113 | ==> 666''' + Colors.GREEN + ''' 2114 | 2115 | Keep in mind, that if the Boolean is True, the entire value of p is returned. 2116 | This comes in handy when you want to test on one field, but use something else. 2117 | For example, a[2].isdigit() will return p, not a[2] if a[2] is a digit. 2118 | 2119 | Standard python logic operators such as "and","or","not", and 'in' work as well. 2120 | 2121 | For example to filter output based on the presence of "GOO" in the line, use this:''' + Colors.YELLOW + ''' 2122 | 2123 | echo GOO | pyp "'G' in p" 2124 | ==> GOO'''+ Colors.GREEN + ''' 2125 | 2126 | The pyp functions "keep(STR)" and "lose(STR)", and their respective shortcuts, 2127 | "k(STR)" and "i(STR)", are very useful for simple OR style string 2128 | filtering. See below. 2129 | 2130 | Also note, all lines that test False ('', {}, [], False, 0) are eliminated from 2131 | the output completely. You can instead print out a blank line if something tests 2132 | false using --keep_false. This is useful if you need placeholders to keep lists 2133 | in sync, for example. 2134 | ----------------------------------------------------------------------------------- 2135 | SECOND STREAM, TEXT FILE, AND BLANK INPUT 2136 | ----------------------------------------------------------------------------------- 2137 | Normally, pyp receives its input by piping into it like a standard unix shell 2138 | command, but sometimes it's necessary to combine two streams of inputs, such as 2139 | consolidating the output of two shell commands line by line. pyp provides 2140 | for this with the second stream input. Essentially anything after the pyp 2141 | command that is not associated with an option flag is brought into pyp as 2142 | the second stream, and can be accessed seperately from the primary stream 2143 | by using the variable 'sp' 2144 | 2145 | To input a second stream of data, just tack on strings or execute (use backticks) 2146 | a command to the end of the pyp command, and then access this array using the 2147 | variable 'sp' ''' + Colors.YELLOW + ''' 2148 | 2149 | echo random_frame.jpg | pyp "p, sp" `echo "random_string"` 2150 | ===> random_frame.jpg random_string''' + Colors.GREEN + ''' 2151 | 2152 | In a similar way, text input can be read in from a text file using the 2153 | --text_file flag. You can access the entire file as a list using the variable 2154 | 'fpp', while the variable 'fp' reads in one line at a time. This text file 2155 | capability is very useful for lining up std-in data piped into pyp with 2156 | data in a text file like this:''' + Colors.YELLOW + ''' 2157 | 2158 | echo normal_input | pyp -text_file example.txt "p, fp" ''' + Colors.GREEN + ''' 2159 | 2160 | This setup is geared mostly towards combining data from std-in with that in 2161 | a text file. If the text file is your only data, you should cat it, and pipe 2162 | this into pyp. 2163 | 2164 | If you need to generate output from pyp with no input, use --blank_inputs. 2165 | This is useful for generating text based on line numbers using the 'n' 2166 | variable. 2167 | 2168 | ----------------------------------------------------------------------------------- 2169 | TEXT FILE AND SECOND STREAM LIST OPERATIONS 2170 | ----------------------------------------------------------------------------------- 2171 | List operations can be performed on file inputs and second stream 2172 | inputs using the variables spp and fpp, respectively. For example to sort 2173 | a file input, use: ''' + Colors.YELLOW + ''' 2174 | 2175 | fpp.sort() ''' + Colors.GREEN + ''' 2176 | 2177 | Once this operation takes place, the sorted fpp will be used for all future 2178 | operations, such as referring to the file input line-by-line using fp. 2179 | 2180 | You can add these inputs to the std-in stream using simple list 2181 | additions like this: ''' + Colors.YELLOW + ''' 2182 | 2183 | pp + fpp ''' + Colors.GREEN + ''' 2184 | 2185 | If pp is 10 lines, and fpp is 10 line, this will result in a new pp stream 2186 | of 20 lines. fpp will remain untouched, only pp will change with this 2187 | operation. 2188 | 2189 | Of course, you can trim these to your needs using standard 2190 | python list selection techniques: ''' + Colors.YELLOW + ''' 2191 | 2192 | pp[0:5] + fpp[0:5] ''' + Colors.GREEN + ''' 2193 | 2194 | This will result in a new composite input stream of 10 lines. 2195 | 2196 | Keep in mind that the length of fpp and spp is trimmed to reflect 2197 | that of std-in. If you need to see more of your file or second 2198 | stream input, you can extend your std-in stream simply:''' + Colors.YELLOW + ''' 2199 | 2200 | pp + ['']*10 ''' + Colors.GREEN + ''' 2201 | 2202 | will add 10 blank lines to std-in, and thus reveal another 10 2203 | lines of fpp if available. 2204 | 2205 | 2206 | ----------------------------------------------------------------------------------- 2207 | MACRO USAGE 2208 | ----------------------------------------------------------------------------------- 2209 | Macros are a way to permently store useful commands for future recall. They are 2210 | stored in your home directory by default. Facilites are provided to store public 2211 | macros as well, which is useful for sharing complex commands within your work group. 2212 | Paths to these text files can be reset to anything you choose my modifying the 2213 | PypCustom.py config file. Macros can become quite complex, and provide 2214 | a useful intermediate between shell commands and scripts, especially for solving 2215 | one-off problems. Macro listing, saving, deleting, and searching capabilities are 2216 | accessible using --macrolist, --macro_save, --macro_delete, --macro_find flags. 2217 | Run pyp --help for more details. 2218 | 2219 | you can pyp to and from macros just like any normal pyp command. ''' + Colors.YELLOW + ''' 2220 | pyp "a[0]| my_favorite_macros | 'ls', p" ''' + Colors.GREEN + ''' 2221 | 2222 | Note, if the macro returns a list, you can access individual elements using 2223 | [n] syntax:''' + Colors.YELLOW + ''' 2224 | pyp "my_list_macro[2]" ''' + Colors.GREEN + ''' 2225 | 2226 | Also, if the macro uses %s, you can append a %(string,..) to then end to string 2227 | substitute: ''' + Colors.YELLOW + ''' 2228 | pyp "my_string_substitution_macro%('test','case')" ''' + Colors.GREEN + ''' 2229 | 2230 | By default, macros are saved in your home directory. This can be modifed to any 2231 | directory by modifying the user_macro_path attribute in your PypCustom.py. If 2232 | you work in a group, you can also save macros for use by others in a specific 2233 | location by modifying group_macro_path. See the section below on custom 2234 | methods about how to set up this file. 2235 | ----------------------------------------------------------------------------------- 2236 | CUSTOM METHODS 2237 | ----------------------------------------------------------------------------------- 2238 | pyed pyper relies on overloading the standard python string and list objects 2239 | with its own custom methods. If you'd like to try writing your own methods 2240 | either to simplify a common task or integrate custom functions using a 2241 | proprietary API, it's straightforward to do. You'll have to setup a config 2242 | file first: 2243 | 2244 | pyp --unmodified_config > PypCustom.py 2245 | sudo chmod 666 PypCustom.py 2246 | 2247 | There are example functions for string, list, powerpipe, and generic methods. 2248 | to get you started. When pyp runs, it looks for this text file and automatically 2249 | loads any found functions, overloading them into the appropriate objects. You 2250 | can then use your custom methods just like any other pyp function. 2251 | ----------------------------------------------------------------------------------- 2252 | TIPS AND TRICKS 2253 | ----------------------------------------------------------------------------------- 2254 | If you have to cut and paste data (from an email for example), execute pyp, paste 2255 | in your data, then hit CTRL-D. This will put the data into the disk buffer. Then, 2256 | just rerun pyp with --rerun, and you'll be able to access this data for further 2257 | pyp manipulations! 2258 | 2259 | If you have split up a line into a list, and want to process this list line by 2260 | line, simply pipe the list to pp and then back to p: pyp "w | pp |p" 2261 | 2262 | Using --rerun is also great way to buffer data into pyp from long-running scripts 2263 | 2264 | pyp is an easy way to generate commands before executing them...iteratively keep 2265 | adding commands until you are confident, then use the --execute flag or pipe them 2266 | to sh. You can use ";" to set up dependencies between these commands...which is 2267 | an easy way to work out command sequences that would typically be executed in a 2268 | "foreach" loop. 2269 | 2270 | Break out complex intermediate steps into macros. Macros can be run at any point in a 2271 | pyp command. 2272 | 2273 | If you find yourself shelling out constantly to particular commands, it might 2274 | be worth adding python methods to the PypCustom.py config, especially if you 2275 | are at a large facility. 2276 | 2277 | Many command line tools (like stat) use a KEY:VALUE format. The shelld function 2278 | will turn this into a python dictionary, so you can access specific data using 2279 | their respective keys by using something like this: shelld(COMMAND)[KEY] 2280 | 2281 | =================================================================================== 2282 | HERE ARE THE BUILT IN VARIABLES: 2283 | 2284 | STD-IN (PRIMARY INPUT) 2285 | ------------- 2286 | p line-by-line std-in variable. p represents whatever was 2287 | evaluated to before the previous pipe (|). 2288 | 2289 | pp python list of ALL std-in input. In-place methods like 2290 | sort() will work as well as list methods like sorted(LIST) 2291 | 2292 | SECOND STREAM 2293 | -------------- 2294 | sp line-by-line input second stream input, like p, but from all 2295 | non-flag arguments AFTER pyp command: pyp "p, sp" SP1 SP2 SP3 ... 2296 | 2297 | spp python list of ALL second stream list input. Modifications of 2298 | this list will be picked up with future references to sp 2299 | 2300 | FILE INPUT 2301 | -------------- 2302 | fp line-by-line file input using --text_file TEXT_FILE. fp on 2303 | the first line of output is the first line of the text file 2304 | 2305 | fpp python list of ALL text file input. Modifications of 2306 | this list will be picked up with future references to fp 2307 | 2308 | 2309 | COMMON VARIABLES 2310 | ---------------- 2311 | original original line by line input to pyp 2312 | o same as original 2313 | 2314 | quote a literal " (double quotes can't be used in a pyp expression) 2315 | paran a literal ' 2316 | dollar a literal $ 2317 | 2318 | n line counter (1st line is 0, 2nd line is 1,...use the form "(n+3)" 2319 | to modify this value. n changes to reflect filtering and list ops. 2320 | nk n + 1000 2321 | 2322 | date date and time. Returns the current datetime.datetime.now() object. 2323 | pwd present working directory 2324 | 2325 | history history array of all previous results: 2326 | so pyp "a|u|s|i|h[-3]" shows eval of s 2327 | h same as history 2328 | 2329 | digits all numbers [0-9] 2330 | letters all upper and lowercase letters (useful when combined with variable n). 2331 | letters[n] will print out "a" on the first line, "b" on the second... 2332 | punctuation all punctuation [!"#$%&'()*+,-./:;<=>?@[\]^_`{|}~] 2333 | 2334 | 2335 | =================================================================================== 2336 | THE FOLLOWING ARE SPLIT OR JOINED BASED ON p BEING A STRING OR AN ARRAY: 2337 | 2338 | s OR slash p split/joined on "/" 2339 | d OR dot p split/joined on "." 2340 | w OR whitespace p split/joined on whitespace (on spaces,tabs,etc) 2341 | u OR underscore p split/joined on '_' 2342 | c OR colon p split/joined on ':' 2343 | mm OR comma p split/joined on ',' 2344 | m OR minus p split/joined on '-' 2345 | a OR all p split on [' '-_=$...] (on "All" metacharacters) 2346 | 2347 | Also, the ORIGINAL INPUT (history[0]) lines are split on delimiters as above, but 2348 | stored in os, od, ow, ou, oc, omm, om and oa as well as oslash, odot, owhitepace, 2349 | ocomma, ominus, and oall''' + Colors.GREEN + ''' 2350 | 2351 | =================================================================================== 2352 | HERE ARE THE BUILT IN FUNCTIONS AND ATTRIBUTES: 2353 | 2354 | Function Notes 2355 | -------------------------------------------------------------------------------- 2356 | STRING (all python STRING methods like p.replace(STRING1,STRING2) work) 2357 | -------------------------------------------------------------------------------- 2358 | p.digits() returns a list of contiguous numbers present in p 2359 | p.letters() returns a list of contiguous letters present in p 2360 | p.punctuation() returns a list of contiguous punctuation present in p 2361 | 2362 | p.trim(delimiter) removes last field from string based on delimiter 2363 | with the default being "/" 2364 | p.kill(STR1,STR2...) removes specified strings 2365 | p.clean(delimeter) removes all metacharacters except for slashes, dots and 2366 | the joining delimeter (default is "_") 2367 | p.re(REGEX) returns portion of string that matches REGEX regular 2368 | expression. 2369 | p.rereplace(REG,STR) find REG regular expression and replace with STR. 2370 | this is more reliable than p.replace(p.re(REGEX),STR) 2371 | 2372 | p.dir directory of path 2373 | p.file file name of path 2374 | p.ext file extension (jpg, tif, hip, etc) of path 2375 | 2376 | These fuctions will work with all pyp strings eg: p, o, dot[0], p.trim(), etc. 2377 | Strings returned by native python functions (like split()) won't have these 2378 | available, but you can still access them using str(STRING). Basically, 2379 | manually recasting anything using as a str(STRING) will endow them with 2380 | the custom pyp methods and attributes. 2381 | 2382 | -------------------------------------------------------------------------------- 2383 | LIST (all LIST methods like pp.sort(), pp[-1], and pp.reverse() work) 2384 | -------------------------------------------------------------------------------- 2385 | pp.delimit(DELIM) split input on delimiter instead of newlines 2386 | pp.divide(N) consolidates N consecutive lines to 1 line. 2387 | pp.before(STRING, N) searches for STRING, colsolidates N lines BEFORE it to 2388 | the same line. Default N is 1. 2389 | pp.after(STRING, N) searches for STRING, colsolidates N lines AFTER it to 2390 | same line. Default N is 1. 2391 | pp.matrix(STRING, N) returns pp.before(STRING, N) and pp.after(STRING, N). 2392 | Default N is 1. 2393 | pp.oneline(DELIM) combines all list elements to one line with delimiter. 2394 | Default delimeter is space. 2395 | pp.uniq() returns only unique elements 2396 | pp.unlist() breaks up ALL lists up into seperate single lines 2397 | 2398 | pp + [STRING] normal python list addition extends list 2399 | pp + spp + fpp normal python list addition combines several inputs. 2400 | new input will be pp; spp and fpp are unaffected. 2401 | sum(pp), max(pp),... normal python list math works if pp is properly cast 2402 | i.e. all members of pp should be integers or floats. 2403 | 2404 | These functions will also work on file and second stream lists: fpp and spp 2405 | 2406 | 2407 | -------------------------------------------------------------------------------- 2408 | NATIVE PYP FUNCTIONS 2409 | -------------------------------------------------------------------------------- 2410 | keep(STR1,STR2,...) keep all lines that have at least one STRING in them 2411 | k(STR1,STR2,...) shortcut for keep(STR1,STR2,...) 2412 | lose(STR1,STR2,...) lose all lines that have at least one STRING in them 2413 | l(STR1,STR2,...) shortcut for lose(STR1,STR2,...) 2414 | 2415 | rekeep(REGEX) keep all lines that match REGEX regular expression 2416 | rek(REGEX) shortcut for rekeep(REGEX) 2417 | relose(REGEX) lose all lines that match REGEX regular expression 2418 | rel(REGEX) shortcut for relose(REGEX) 2419 | 2420 | shell(SCRIPT) returns output of SCRIPT in a list. 2421 | shelld(SCRIPT,DELIM) returns output of SCRIPT in dictionary key/value seperated 2422 | on ':' (default) or supplied delimeter 2423 | env(ENVIROMENT_VAR) returns value of evironment variable using os.environ.get() 2424 | glob(PATH) returns globed files/directories at PATH. Make sure to use 2425 | '*' wildcard 2426 | str(STR) turns any object into an PypStr object, allowing use 2427 | of custom pyp methods as well as normal string methods. 2428 | 2429 | SIMPLE EXAMPLES: 2430 | =================================================================================== 2431 | pyp "'foo ' + p" ==> "foo" + current line 2432 | pyp "p.replace('x','y') | p + o" ==> current line w/replacement + original line 2433 | pyp "p.split(':')[0]" ==> first field of string split on ':' 2434 | pyp "slash[1:3]" ==> array of fields 1 and 2 of string split on '/' 2435 | pyp "s[1:3]|s" ==> string of above joined with '/' 2436 | ''' + Colors.OFF 2437 | 2438 | 2439 | 2440 | unmodified_config = ''' 2441 | #!/usr/bin/env python 2442 | # This must be saved in same directory as pyp (or be in the python path) 2443 | # make sure to name this PypCustom.py and change permission to 666 2444 | 2445 | import sys 2446 | import os 2447 | 2448 | 2449 | class Colors(object): 2450 | OFF = chr(27) + '[0m' 2451 | RED = chr(27) + '[31m' 2452 | GREEN = chr(27) + '[32m' 2453 | YELLOW = chr(27) + '[33m' 2454 | MAGENTA = chr(27) + '[35m' 2455 | CYAN = chr(27) + '[36m' 2456 | WHITE = chr(27) + '[37m' 2457 | BLUE = chr(27) + '[34m' 2458 | BOLD = chr(27) + '[1m' 2459 | COLORS = [OFF, RED, GREEN, YELLOW, MAGENTA, CYAN, WHITE, BLUE, BOLD] 2460 | 2461 | 2462 | class NoColors(object): 2463 | OFF = '' 2464 | RED = '' 2465 | GREEN ='' 2466 | YELLOW = '' 2467 | MAGENTA = '' 2468 | CYAN = '' 2469 | WHITE ='' 2470 | BLUE = '' 2471 | BOLD = '' 2472 | COLORS = [OFF, RED, GREEN, YELLOW, MAGENTA, CYAN, WHITE, BLUE, BOLD] 2473 | 2474 | 2475 | class PypCustom(object): 2476 | 'modify below paths to set macro paths' 2477 | def __init__(self): 2478 | self.user_macro_path = os.path.expanduser('~')+ '/pyp_user_macros.json' 2479 | self.group_macro_path = os.path.expanduser('~')+ '/pyp_user_macros.json' 2480 | self.custom_execute = False 2481 | 2482 | 2483 | class PowerPipeListCustom(): 2484 | 'this is used for pp functions (list fuctions like sort) that operate on all inputs at once.' 2485 | def __init__(self, *args): 2486 | pass 2487 | 2488 | def test(self): 2489 | print 'test' #pp.test() will print "test" 2490 | 2491 | 2492 | class PypStrCustom(): 2493 | 'this is used for string functions using p and other pyp string variables' 2494 | def __init__(self, *args): 2495 | self.test_attr = 'test attr' 2496 | 2497 | def test(self): 2498 | print 'test' #p.test() will print "test" is p is a str 2499 | 2500 | 2501 | class PypListCustom(): 2502 | def __init__(self, *args): 2503 | pass 2504 | 2505 | def test(self): 2506 | print 'test' #p.test() will print "test" is p is a list broken up from a str 2507 | 2508 | 2509 | class PypFunctionCustom(object): 2510 | 'this is used for custom functions and variables (non-instance)' 2511 | test_var = 'works' 2512 | 2513 | def __init__(self, *args): 2514 | pass 2515 | 2516 | def test(self): 2517 | print 'working func ' + self 2518 | ''' 2519 | 2520 | 2521 | usage = """ 2522 | pyp is a python-centric command line text manipulation tool. It allows you to format, replace, augment 2523 | and otherwise mangle text using standard python syntax with a few golden-oldie tricks from unix commands 2524 | of the past. You can pipe data into pyp or cut and paste text, and then hit ctrl-D to get your input into pyp. 2525 | 2526 | After it's in, you can use the standard repertoire of python commands to modify the text. The key variables 2527 | are "p", which represents EACH LINE of the input as a PYTHON STRING. and "pp", which represents ALL of the 2528 | inputs as a PYTHON ARRAY. 2529 | 2530 | You can pipe data WITHIN a pyp statement using standard unix style pipes ("|"), where "p" now represents the 2531 | evaluation of the python statement before the "|". You can also refer back to the ORIGINAL, unadulterated 2532 | input using the variable "o" or "original" at any time...and the variable "h" or "history" allows you 2533 | to refer back to ANY subresult generated between pipes ("|"). 2534 | 2535 | All pyp statements should be enclosed in double quotes, with single quotes being used to enclose any strings. 2536 | 2537 | echo 'FOO IS AN ' | pyp "p.replace('FOO','THIS') | p + 'EXAMPLE'" 2538 | ==> THIS IS AN EXAMPLE 2539 | 2540 | Splitting texton metacharacters is often critical for picking out particular fields of interest, 2541 | so common SPLITS and JOINS have been assigned variables. For example, "underscore" or "u" will split a string 2542 | to an array based on undercores ("_"), while "underscore" or "u" will ALSO join an array with underscores ("_") 2543 | back to a string. 2544 | 2545 | Here are a few key split/join variables; run with --manual for all variable and see examples below in the string section. 2546 | 2547 | s OR slash splits AND joins on "/" 2548 | u OR underscore splits AND joins on "_" 2549 | w OR whitespace splits on whitespace (spaces,tabs,etc) AND joins with spaces 2550 | a OR all splits on ALL metacharacters [!@#$%^&*()...] AND joins with spaces 2551 | 2552 | EXAMPLES: 2553 | ------------------------------------------------------------------------------ 2554 | List Operations # all python list methods work 2555 | ------------------------------------------------------------------------------ 2556 | print all lines ==> pyp "pp" 2557 | sort all input lines ==> pyp "pp.sort()" 2558 | eliminate duplicates ==> pyp "pp.uniq()" 2559 | combine all lines to one line ==> pyp "pp.oneline()" 2560 | print line after FOO ==> pyp "pp.after('FOO')" 2561 | list comprehenision ==> pyp "[x for x in pp]" 2562 | return to string context after sort ==> pyp "pp.sort() | p" 2563 | 2564 | ------------------------------------------------------------------------------- 2565 | String Operations # all python str methods work 2566 | ------------------------------------------------------------------------------- 2567 | print line ==> pyp "p" 2568 | combine line with FOO ==> pyp "p +'FOO'" 2569 | above, but combine with original input ==> pyp "p +'FOO'| p + o" 2570 | 2571 | replace FOO with GOO ==> pyp "p.replace('FOO','GOO')" 2572 | remove all GOO and FOO ==> pyp "p.kill('GOO','FOO')" 2573 | 2574 | string substitution ==> pyp "'%s FOO %s %s GOO'%(p,p,5)" 2575 | 2576 | split up line by FOO ==> pyp "p.split('FOO')" 2577 | split up line by '/' ==> pyp "slash" 2578 | select 1st field split up by '/' ==> pyp "slash[0]" 2579 | select fields 3 through 5 split up by '/' ==> pyp "s[2:6]" 2580 | above joined together with '/' ==> pyp "s[2:6] | s" 2581 | 2582 | ------------------------------------------------------------------------------- 2583 | Logic Filters # all python Booleon methods work 2584 | ------------------------------------------------------------------------------- 2585 | keep all lines with GOO and FOO ==> pyp "'GOO' in p and 'FOO' in p" 2586 | keep all lines with GOO or FOO ==> pyp "keep('GOO','FOO')" 2587 | keep all lines that are numbers ==> pyp "p.isdigit()" 2588 | 2589 | lose all lines with GOO and FOO ==> pyp "'GOO' not in p and 'FOO' not in p" 2590 | lose all lines with GOO or FOO ==> pyp "lose('GOO','FOO')" 2591 | lose all lines that are numbers ==> pyp "not p.isdigit()" 2592 | 2593 | ------------------------------------------------------------------------------- 2594 | TO SEE EXTENDED HELP, use --manual 2595 | 2596 | """ 2597 | 2598 | if __name__ == '__main__': 2599 | parser = optparse.OptionParser(Docs.usage) 2600 | 2601 | parser.add_option("-m", "--manual", action='store_true', help="prints out extended help") 2602 | parser.add_option("-l", "--macro_list", action='store_true', help="lists all available macros") 2603 | parser.add_option("-s", "--macro_save", dest='macro_save_name', type='string', help='saves current command as macro. use "#" for adding comments EXAMPLE: pyp -s "great_macro # prints first letter" "p[1]"') 2604 | parser.add_option("-f", "--macro_find", dest='macro_find_name', type='string', help='searches for macros with keyword or user name') 2605 | parser.add_option("-d", "--macro_delete", dest='macro_delete_name', type='string', help='deletes specified public macro') 2606 | parser.add_option("-g", "--macro_group", action='store_true', help="specify group macros for save and delete; default is user") 2607 | parser.add_option("-t", "--text_file", type='string', help="specify text file to load. for advanced users, you should typically cat a file into pyp") 2608 | parser.add_option("-x", "--execute", action='store_true', help="execute all commands.") 2609 | parser.add_option("-a", "--execute_aux", type='int', help="execute commands with auxillary data with custom executer") 2610 | parser.add_option("-c", "--turn_off_color", action='store_true', help="prints raw, uncolored output") 2611 | parser.add_option("-u", "--unmodified_config", action='store_true', help="prints out generic PypCustom.py config file") 2612 | parser.add_option("-b", "--blank_inputs", action='store', type='string', help="generate this number of blank input lines; useful for generating numbered lists with variable 'n'") 2613 | parser.add_option("-n", "--no_input", action='store_true', help="use with command that generates output with no input; same as --blank_inputs 1") 2614 | parser.add_option("-i", "--ignore_blanks", action='store_true', help="remove blank lines from output") 2615 | parser.add_option("-k", "--keep_false", action='store_true', help="print blank lines for lines that test as False. default is to filter out False lines from the output") 2616 | parser.add_option("-r", "--rerun", action="store_true", help="rerun based on automatically cached data from the last run. use this after executing \"pyp\", pasting input into the shell, and hitting CTRL-D TWICE") 2617 | 2618 | (options, args) = parser.parse_args() 2619 | 2620 | if options.turn_off_color or options.execute or options.execute_aux: # overall color switch asap. 2621 | Colors = NoColors 2622 | 2623 | #try: 2624 | pyp = Pyp().main() 2625 | #except Exception, err: 2626 | # print Colors.RED + str(err) + Colors.OFF 2627 | --------------------------------------------------------------------------------