├── .gitignore ├── MANIFEST ├── PKG-INFO ├── README.md ├── setup.py └── src └── YamlLibrary ├── __init__.py ├── fetchyaml.py └── version.py /.gitignore: -------------------------------------------------------------------------------- 1 | # Byte-compiled / optimized / DLL files 2 | __pycache__/ 3 | *.py[cod] 4 | *$py.class 5 | 6 | # C extensions 7 | *.so 8 | 9 | # Distribution / packaging 10 | .Python 11 | env/ 12 | build/ 13 | develop-eggs/ 14 | dist/ 15 | downloads/ 16 | eggs/ 17 | .eggs/ 18 | lib/ 19 | lib64/ 20 | parts/ 21 | sdist/ 22 | var/ 23 | *.egg-info/ 24 | .installed.cfg 25 | *.egg 26 | .idea/ 27 | 28 | # PyInstaller 29 | # Usually these files are written by a python script from a template 30 | # before PyInstaller builds the exe, so as to inject date/other infos into it. 31 | *.manifest 32 | *.spec 33 | 34 | # Installer logs 35 | pip-log.txt 36 | pip-delete-this-directory.txt 37 | 38 | # Unit test / coverage reports 39 | htmlcov/ 40 | .tox/ 41 | .coverage 42 | .coverage.* 43 | .cache 44 | nosetests.xml 45 | coverage.xml 46 | *,cover 47 | .hypothesis/ 48 | 49 | # Translations 50 | *.mo 51 | *.pot 52 | 53 | # Django stuff: 54 | *.log 55 | local_settings.py 56 | 57 | # Flask instance folder 58 | instance/ 59 | 60 | # Scrapy stuff: 61 | .scrapy 62 | 63 | # Sphinx documentation 64 | docs/_build/ 65 | 66 | # PyBuilder 67 | target/ 68 | 69 | # IPython Notebook 70 | .ipynb_checkpoints 71 | 72 | # pyenv 73 | .python-version 74 | 75 | # celery beat schedule file 76 | celerybeat-schedule 77 | 78 | # dotenv 79 | .env 80 | 81 | # Spyder project settings 82 | .spyderproject 83 | 84 | -------------------------------------------------------------------------------- /MANIFEST: -------------------------------------------------------------------------------- 1 | # file GENERATED by distutils, do NOT edit 2 | setup.py 3 | src/YamlLibrary/__init__.py 4 | src/YamlLibrary/fetchyaml.py 5 | src/YamlLibrary/version.py 6 | -------------------------------------------------------------------------------- /PKG-INFO: -------------------------------------------------------------------------------- 1 | Metadata-Version: 1.0 2 | Name: robotframework-yamllibrary 3 | Version: 0.1.2 4 | Summary: Yaml utility library for Robot Framework 5 | Home-page: https://github.com/divfor/robotframework-yamllibrary 6 | Author: Fred Huang 7 | Author-email: divfor@gmail.com 8 | License: Apache V2.0 9 | Description: N/A 10 | Platform: Linux, Windows and Mac 11 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | 功能简述： 2 | 3 | 1. 读取json或者yaml字典或序列化字符串，根据路径(以小数点连接各级key)返回叶子节点的值或者一个子树(最小子树是一个值不带key)。路径下遇到数组(list)时用元素位置(从0开始的数组下标)作为数组元素的key。不知道数组下标或者字典键的时候，可以插入分支路径和匹配值去自动寻找下标或键。 4 | 5 | 2. 对上面返回的子树，给出一个比较树(最小为单个值)，按相等/正则/数学不等式方式比较子树上叶子节点的值，子树的部分或全部结构的键和值。遇到数组会进行遍历比较（也就是支持乱序查找）。 6 | 7 | 如果你需要使用树型的json或者yaml字符串而不是字典，比如读取或者定义大量变量，一次性比较很多变量等，YamlLibrary是一个好的选择。 Get Tree接受一个字符串和一个路径作为参数，将字符串转成yaml树（注意json是yaml的子集，所以也可以用）然后返回给定路径下的子树（注意节点值是最小的子树，所以可以用来取节点值）。Node should Match则在Get Tree的基础上增加一个动作：将取出来的子树跟第三个参数--一个描述比较期望值的子树，一次性完成全部比较。 8 | 9 | 安装： 10 | pip install robotframework-yamllibrary 11 | 12 | 使用库：Robotframework: Library YamlLibrary 13 | 假如有两个yaml串，一个是数据，另一个做比较器用于检查部分路径上的值： 14 | ```python 15 | yaml_string = { 16 | Employer: { 17 | Name: Microsoft, 18 | Office: { China: [ Beijing, Shanghai ], UK: [ London ] }, 19 | Staff : [ { ID: 001, name: Fred, age: 30 }, { ID: 102, name: Jenny, age: 21 } ], 20 | }, 21 | School: { middle: xxx_school, high_school: zzz_school, colleage_school: sss_school }, 22 | } 23 | # define a matcher string to compare some parts of tree structure 24 | cmp_from_top_path = { Employer: { Staff : [ { name: Jenny, ID: y > 100, age: 16 < y < 50 } ] } } 25 | cmp_from_sub_path = { China: [ Beijing, Shanghai ], UK: [ London ] } 26 | cmp_some_list_items = [ { name: Jenny, ID: y > 100, age: 16 < y < 50 } ] 27 | 28 | ``` 29 | 30 | 关键字使用举例: 31 | ```robotframework 32 | ${company}= | Get Tree | ${yaml_string} | Employer.Name 33 | ${staff_list}= | Get Tree | ${yaml_string} | Employer.Staff 34 | ${education_schools} = | Get Tree | ${yaml_string} | School 35 | ${fred_age}= | Get Tree | ${yaml_string} | Employer.Staff.0.age 36 | ${fred_age}= | Get Tree | ${yaml_string} | Employer.Staff/name=Fred/age 37 | ${fred_age}= | Get Tree | ${yaml_string} | Employer.Staff/name~^Fred$/age 38 | Nodes Should Match | ${yml_string} | Employer.Staff/name~^Fred$/age | 16 < y < 60 39 | Nodes Should Match | ${yml_string} | . | ${cmp_from_top_path} 40 | Nodes Should Match | ${yml_string} | Employer.Office | ${cmp_from_sub_path} 41 | Nodes Should Match | ${yml_string} | Employer.Staff | ${cmp_some_list_items} 42 | ``` 43 | 44 | 说明： 45 | 读取数据库或者web接口测试得到json或者yaml文档（参数1），然后可以用上面两个关键字读取路径（参数2）下的子文档/值，以及对值进行比较（参数3）。值比较的方式有相等=，正则表达式匹配～，和数学表达式（不等式，y作变量）。全部比较成功才会返回True，否则False。 46 | 47 | 路径以小数点分割，碰到数组元素则用数字下标（从0开始），单个小数点代表全树。 48 | 对于有多个同级的元素形成的列表，需要确定数组下标，这里采取下沉到其子元素定位的办法：用/或者%分割的一段子路径（相对路径，小数点分割），一个=或～符号，一个比较值，这三个部分匹配得到所在数组的下标应该是唯一的（由使用者写好这三部分确保这个唯一性）。注意路径比较的时候是不可以用数学表达式。这种使用方式对于不知道字典键也是可以使用的。 49 | 50 | Debug：此库调用了robotframework的logger，Set Log Level为Debug可以得到带debug的log，展开就可以看到出错的地方 51 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | # Copyright (c) 2010 Franz Allan Valencia See 4 | # 5 | # Licensed under the Apache License, Version 2.0 (the "License"); 6 | # you may not use this file except in compliance with the License. 7 | # You may obtain a copy of the License at 8 | # 9 | # http://www.apache.org/licenses/LICENSE-2.0 10 | # 11 | # Unless required by applicable law or agreed to in writing, software 12 | # distributed under the License is distributed on an "AS IS" BASIS, 13 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 14 | # See the License for the specific language governing permissions and 15 | # limitations under the License. 16 | 17 | 18 | """Setup script for Robot's YamlLibrary distributions""" 19 | 20 | import sys, os 21 | from distutils.core import setup 22 | 23 | sys.path.insert(0, os.path.join('src', 'YamlLibrary')) 24 | 25 | from version import VERSION 26 | 27 | def main(): 28 | setup(name = 'robotframework-yamllibrary', 29 | version = VERSION, 30 | description = 'Yaml utility library for Robot Framework', 31 | author = 'Fred Huang', 32 | author_email = 'divfor@gmail.com', 33 | url = 'https://github.com/divfor/robotframework-yamllibrary', 34 | package_dir = { '' : 'src'}, 35 | packages = ['YamlLibrary'], 36 | install_requires = ['pyyaml>=3.0'], 37 | ) 38 | 39 | 40 | if __name__ == "__main__": 41 | main() 42 | -------------------------------------------------------------------------------- /src/YamlLibrary/__init__.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | from fetchyaml import FetchYaml 4 | 5 | 6 | class YamlLibrary(FetchYaml): 7 | """ 8 | TO BE DEFINED 9 | """ 10 | 11 | ROBOT_LIBRARY_SCOPE = 'GLOBAL' 12 | -------------------------------------------------------------------------------- /src/YamlLibrary/fetchyaml.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | import yaml 3 | import re 4 | from robot.api import logger 5 | 6 | 7 | class FetchYaml(object): 8 | """ 9 | FetchYaml is a library to fetch sub-document from yaml doc tree, compare it with a matcher doc tree 10 | it requires string buffer input that can be loaded by yaml.load() function 11 | """ 12 | def __init__(self): 13 | #self._mathexpr = re.compile("^[y>=< 0-9\*\/\-\+\.\(\)]+$") 14 | self._mathexpr = re.compile("^[=><= 0-9\*\/\-\+\.\(\)]* y [=>< 0-9\*\/\-\+\.\(\)]*$") 15 | 16 | def get_tree(self, yml_src, path='.'): 17 | """travel given 'path' in src to return a sub-tree that may 18 | be a single python dictionary, a list of dictionaries, or 19 | a single node (the value from a key/value pair) 20 | """ 21 | dct = self._smart_load(yml_src) 22 | return self._get_tree_by_smart_path(dct, path) 23 | 24 | def compare_tree(self, src, dst): 25 | """Recursively compare 'src' (json-like dict/list tree) with 26 | matcher 'dst' (same tree structure and keys, except value part are value/expr/regex). 27 | 28 | src: a python dictionary holding yaml/json/bson data 29 | 30 | dst: a subset of above python dictionary, with values in string/number/math-expr/regex 31 | 32 | return: True or False 33 | """ 34 | if dst is None or (isinstance(dst, (list, tuple, dict, basestring)) and len(dst) == 0): 35 | return self._cmp_null(src, dst) 36 | elif isinstance(dst, (list, tuple)): 37 | return self._cmp_list(src, dst) 38 | elif isinstance(dst, dict): 39 | return self._cmp_dict(src, dst) 40 | elif isinstance(dst, basestring) and self._mathexpr.search(dst) and len(dst.split('y')) > 1: 41 | logger.debug("compare_tree: eval '%s' with %s instead of y" % (str(dst), str(src))) 42 | return self._eval_math_expr(src, dst) 43 | elif isinstance(dst, (basestring, bool, int, long, float)): 44 | return self._cmp_base_types(src, dst) 45 | logger.debug("compare_tree: src not in [dict, list, basestring, int, long, float, bool]:\n%s" % str(src)) 46 | return False 47 | 48 | def nodes_should_match(self, yml_doc_data, yml_path, yml_doc_matcher): 49 | """compare sub-tree from given yaml path against a yaml tree of matcher. 50 | 51 | A matcher could be 'direct_value' for equal matching, '~regex' for regression 52 | expression matching, or a 'y'-variable math regression for evaluating (python eval) 53 | 54 | for example: {'age': 30, 'name': '~^Smith .+', 'salary': 'y > 20000'} 55 | 56 | Return False if any matcher fails. 57 | 58 | Return True if all matchers succeed. 59 | """ 60 | dct = self._smart_load(yml_doc_data) 61 | src = self._get_tree_by_smart_path(dct, yml_path) 62 | dst = self._smart_load(yml_doc_matcher) 63 | if not self.compare_tree(src, dst): 64 | raise AssertionError("nodes under '%s' do not satisfied matcher:\nactual:\n'%s'\nmatcher:\n'%s'" % 65 | (yml_path, str(src), str(dst))) 66 | logger.info("The matcher has been verified passed.") 67 | 68 | @staticmethod 69 | def _smart_load(src): 70 | if src is None or isinstance(src, (int, long, float, bool)): 71 | return src 72 | if isinstance(src, basestring): 73 | try: 74 | return yaml.load(src) 75 | except: 76 | return yaml.load(src.replace('\/', '/')) 77 | if isinstance(src, (dict, list)): 78 | return yaml.load(yaml.dump(src)) 79 | raise ValueError("_smart_load: Unknown format to yaml: %s (type is %s)" % (str(src), str(type(src)))) 80 | 81 | @staticmethod 82 | def _tokenize(s): 83 | if s is None: 84 | raise StopIteration 85 | tokens = (re.sub(r'\\(\\|\.)', r'\1', m.group(0)) 86 | for m in re.finditer(r'((\\.|[^.\\])*)', s)) 87 | # an empty string superfluous token is added after all non-empty token 88 | for token in tokens: 89 | if len(token) != 0: 90 | next(tokens) 91 | yield token 92 | 93 | def _get_tree_by_direct_path(self, dct, key): 94 | key = iter(key) 95 | try: 96 | head = next(key) 97 | except StopIteration: 98 | return dct 99 | if isinstance(dct, (list, tuple)): 100 | try: 101 | idx = int(head) 102 | except ValueError: 103 | raise ValueError("_direct_path: list index not a integer: %r" % head) 104 | try: 105 | value = dct[idx] 106 | except IndexError: 107 | raise IndexError("_direct_path: list index out of range: %d to %d" % (idx, len(dct))) 108 | elif isinstance(dct, dict): 109 | try: 110 | value = dct[head] 111 | except KeyError: 112 | raise KeyError("_direct_path: dict misses key %r" % (head, )) 113 | except: 114 | raise TypeError("_direct_path: can't query leaf key %r with value %r" % (head, dct)) 115 | else: 116 | value = dct 117 | 118 | for ty in (str, unicode, int, long, float, bool): 119 | if isinstance(value, ty): 120 | return ty(value) 121 | 122 | return self._get_tree_by_direct_path(value, key) 123 | 124 | def _get_tree_by_smart_path(self, dct, key): 125 | if key == '.' or key == '/': 126 | return dct 127 | s = key 128 | sp = re.split('[/%]', s, 2) # path_left, middle_locator, path_right 129 | while len(sp) == 3: 130 | left, locator, right = sp 131 | logger.debug("_smart_path: path is %s" % s) 132 | logger.debug("_smart_path: split path to left/middle/right: '%s', '%s', '%s'" % (left, locator, right)) 133 | blocks = self._get_tree_by_direct_path(dct, self._tokenize(left)) if left else dct 134 | if not blocks: 135 | logger.debug("_smart_path: Can not get the TBD part under path '%s'" % left) 136 | return None 137 | if not isinstance(blocks, (list, dict)): 138 | raise TypeError("_smart_path: Node is not a list or dictionary: %s" % left) 139 | if len(blocks) == 0: 140 | logger.debug("_smart_path: Node is an empty list: %s" % left) 141 | return None 142 | psv = re.split('(~|=)', locator) 143 | if not isinstance(psv, list) or len(psv) < 3: 144 | raise ValueError("_smart_path: expect 'sub-path=value or sub-path~regex' but received '%s'" % locator) 145 | path, sign, expr = psv 146 | index = -1 147 | search_pairs = blocks.iteritems() if isinstance(blocks, dict) else enumerate(blocks) 148 | for i, block in search_pairs: 149 | v = self._get_tree_by_direct_path(block, self._tokenize(path)) 150 | logger.debug("_smart_path: stepped into branch search path '%s' and got its leaf value: %s" % (path, str(v))) 151 | if not v: continue 152 | if not isinstance(v, (basestring, bool, int, long, float)): 153 | raise ValueError("_smart_path: expect basic type to index block but received '%s'" % left) 154 | index = 1 155 | if sign == '~' and re.compile(expr).search(v): break 156 | if sign == '=': 157 | if isinstance(v, basestring) and unicode(v) == expr: break 158 | if isinstance(v, int) and v == int(expr): break 159 | if isinstance(v, long) and v == long(expr): break 160 | if isinstance(v, float) and v == float(expr): break 161 | if isinstance(v, bool) and v == bool(expr): break 162 | index = -1 163 | if index < 0: 164 | return None 165 | s = '.'.join((left, str(i))) if left else str(i) 166 | s = '.'.join((s, right)) if right else s 167 | logger.debug("_smart_path: '%s' indexed as '%s', new path will be '%s'" % (locator, i, s)) 168 | sp = re.split('[/%]', s, 2) 169 | if len(sp) < 3: 170 | return self._get_tree_by_direct_path(dct, self._tokenize(s)) 171 | return None 172 | 173 | @staticmethod 174 | def _cmp_string(src, dst): 175 | if not isinstance(src, basestring): 176 | logger.debug("_cmp_string: receives non-string: %s" % str(src)) 177 | return False 178 | if len(dst) == 0: 179 | return len(src) == 0 180 | if dst[0] != '~': 181 | if len(src) == len(dst) and unicode(src) == unicode(dst): 182 | return True 183 | if dst[0] == '~': 184 | if len(dst) < 2: 185 | logger.debug("_cmp_string: regexp is empty!") 186 | return False 187 | if re.compile(dst[1:]).search(src): 188 | logger.debug("_cmp_string: src '%s' matches regexp '%s'" % (str(src), str(dst))) 189 | return True 190 | logger.debug("_cmp_string: string not match regexp: %s ~ %s" % (src, dst)) 191 | return False 192 | 193 | @staticmethod 194 | def _cmp_number(src, dst): 195 | for t in (int, long, float): 196 | if isinstance(dst, t): 197 | try: 198 | if t(src) == t(dst): 199 | return True 200 | except: 201 | logger.debug("_cmp_number: expect type %s but get %s" % (str(type(dst)), str(type(src)))) 202 | logger.debug("_cmp_number: Can not convert: %s" % str(src)) 203 | logger.debug("_cmp_number: number '%s' not equal to '%s'" % (str(src), str(dst))) 204 | return False 205 | 206 | @staticmethod 207 | def _cmp_bool(src, dst): 208 | try: 209 | if bool(src) == bool(dst): 210 | return True 211 | except: 212 | logger.debug("_cmp_bool: expect bool but get %s" % str(type(src))) 213 | logger.debug("_cmp_bool: Can not convert to bool: %s" % str(src)) 214 | return False 215 | 216 | @staticmethod 217 | def _cmp_null(src, dst=None): 218 | if src is None: 219 | return True 220 | if isinstance(src, basestring) and src in ('', 'null', 'undefined'): 221 | return True 222 | if isinstance(src, (list, tuple, dict, basestring)) and len(src) == 0: 223 | return True 224 | logger.debug("_cmp_null: src=[%s] does not matches None" % str(src)) 225 | return False 226 | 227 | @staticmethod 228 | def _eval_math_expr(src, dst): 229 | if not isinstance(src, (int, long, float)): 230 | logger.debug("_eval_math_expr: receives src not in int/long/float/bool: %s" % str(src)) 231 | return False 232 | ev = eval(str(src).join(dst.split('y'))) 233 | if not isinstance(ev, bool): 234 | logger.debug("_eval_math_expr: eval return non-bool value") 235 | return False 236 | if ev == False: 237 | logger.debug("_eval_math_expr: '%s' not satisfied math expr '%s'" % (str(src), dst)) 238 | return ev 239 | 240 | def _cmp_dict(self, src, dst): 241 | if not isinstance(src, dict): 242 | logger.debug("_cmp_dict: src tree '%s' is not a dict" % str(src)) 243 | return False 244 | try: 245 | for key in dst.keys(): 246 | if not self.compare_tree(src[key], dst[key]): 247 | return False 248 | except KeyError: 249 | logger.debug("_cmp_dict: matcher key '%s' not found in '%s'" % (str(key), str(src))) 250 | return False 251 | return True 252 | 253 | def _cmp_list(self, src, dst): 254 | if not isinstance(src, (tuple, list)): 255 | logger.debug("_cmp_list: src '%s' is not a list or tuple" % str(src)) 256 | return False 257 | try: 258 | for v in dst: 259 | if v is None or isinstance(v, (basestring, int, long, float, bool)): 260 | if all([not self._cmp_base_types(s, v) for s in src]): 261 | logger.debug("_cmp_list: '%s' from '%s' not match any value in list '%s'" 262 | % (str(v), str(dst), str(src))) 263 | return False 264 | elif all([not self.compare_tree(s, v) for s in src]): 265 | logger.debug("_cmp_list: item '%s' from '%s' not found in src list '%s'" 266 | % (str(v), str(dst), str(src))) 267 | return False 268 | except IndexError: 269 | logger.debug("_cmp_list: matcher index '%s' out of range in list '%s'" % (str(key), str(src))) 270 | return False 271 | logger.debug("_cmp_list: src '%s' matches '%s'" % (str(src), str(dst))) 272 | return True 273 | 274 | def _cmp_base_types(self, src, dst): 275 | if dst is None: 276 | return self._cmp_null(src, dst) 277 | if isinstance(dst, bool): 278 | return self._cmp_bool(src, dst) 279 | elif isinstance(dst, (int, long, float)): 280 | return self._cmp_number(src, dst) 281 | elif isinstance(dst, basestring): 282 | return self._cmp_string(src, dst) 283 | logger.debug("Unknown type of dst: %s" % str(dst)) 284 | 285 | def _strip_bson_id(self, dct): 286 | if isinstance(dct, (list, tuple)): 287 | for child in dct: 288 | self._strip_bson_id(child) 289 | if isinstance(dct, dict): 290 | for key in dct.keys(): 291 | if key == '_id': 292 | del dct['_id'] 293 | if isinstance(dct[key], dict): 294 | self._strip_bson_id(dct[key]) 295 | 296 | @staticmethod 297 | def _strip_number_long(s): 298 | rex = re.compile('^NumberLong\(([0-9]+)\)$') 299 | result = rex.search(s) 300 | return result.group(1) if result else s 301 | -------------------------------------------------------------------------------- /src/YamlLibrary/version.py: -------------------------------------------------------------------------------- 1 | # encoding=utf-8 2 | 3 | VERSION = '0.2.8' 4 | --------------------------------------------------------------------------------