├── .gitignore ├── README.md ├── atest ├── infinite.robomachine ├── one_state_no_actions.robomachine ├── pass_and_fail_machine.robomachine ├── selenium_demo_fsm.robomachine ├── selenium_demo_machine.robomachine └── testingmachine.robomachine ├── copyright.txt ├── example └── webapp │ ├── .gitignore │ ├── KeywordLibrary.py │ ├── LICENSE.md │ ├── model.robomachine │ └── test.sh ├── pyproject.toml ├── release.sh ├── requirements.txt ├── setup.cfg ├── setup.py ├── src └── robomachine │ ├── __init__.py │ ├── allpairsstrategy.py │ ├── generator.py │ ├── model.py │ ├── parsing.py │ ├── rules.py │ ├── runner.py │ └── strategies.py └── test ├── __init__.py ├── generation_test.py ├── model_test.py ├── robomachina_test.py ├── rules_test.py ├── strategies_test.py └── variable_test.py /.gitignore: -------------------------------------------------------------------------------- 1 | *.pyc 2 | atest/*.robot 3 | log.html 4 | output.xml 5 | report.html 6 | env/ 7 | .idea 8 | build 9 | dist 10 | *.egg-info 11 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # RoboMachine 2 | 3 | A test data generator for [Robot Framework](http://www.robotframework.org). 4 | 5 | ## What is this tool for? 6 | 7 | You know those ugly bugs that users report that somehow were missed by all your ATDD-, TDD- and exploratory testing? Those bugs that lurk in the corners of complexity of the system that you are building. Those bugs that will make users of your application hate you.. 8 | 9 | This tool is here to help you seek and destroy those bugs before users will find them. 10 | 11 | It gives you the ability to generate a huge number of tests that can go through a very vast number of similar (or not so similar) test scenarios. 12 | 13 | ## What is it? 14 | 15 | If you know Regular Expressions: 16 | A regular expression is a pattern that represents all the possible strings that match that pattern. RoboMachine works in a similar way. A machine represents a set of tests in a compact way. This machine can be used to generate all (or part of) the tests that it represents. 17 | 18 | If you know Model-Based Testing or automata theory: 19 | With RoboMachine you define an extended finite state machine (in a Robot Framework -like syntax) that represents a set of tests. RoboMachine also contains algorithms that can be used to generate real executable Robot Framework tests from a RoboMachine model. 20 | 21 | If you know Combinatorial Testing or like data driven testing: 22 | With RoboMachine you can define sets of data and rules about that data, that can be used to generate data driven tests (and also keyword driven tests). This allows you to keep your data in compact sets and let the system generate the real test data vectors from there. 23 | 24 | ## What does it look like? 25 | 26 | Here is an example machine (using combinatorial techniques) that can generate tests for [SeleniumLibrary demo](http://code.google.com/p/robotframework-seleniumlibrary/wiki/Demo). 27 | 28 | ```robotframework 29 | *** Settings *** 30 | Suite Setup Open Browser ${LOGIN_PAGE_URL} googlechrome 31 | Suite Teardown Close Browser 32 | Test Setup Go to ${LOGIN_PAGE_URL} 33 | Library SeleniumLibrary 34 | 35 | *** Variables *** 36 | ${USERNAME_FIELD} username_field 37 | ${PASSWORD_FIELD} password_field 38 | ${LOGIN_BUTTON} LOGIN 39 | ${VALID_USERNAME} demo 40 | ${VALID_PASSWORD} mode 41 | ${LOGIN_PAGE_URL} http://localhost:7272/html/ 42 | 43 | *** Machine *** 44 | ${USERNAME} any of ${VALID_USERNAME} ${VALID_PASSWORD} invalid123 ${EMPTY} 45 | ${PASSWORD} any of ${VALID_PASSWORD} ${VALID_USERNAME} password123 ${EMPTY} 46 | 47 | # Add an equivalence rule to reduce the number of generated tests 48 | ${USERNAME} == ${VALID_PASSWORD} <==> ${PASSWORD} == ${VALID_USERNAME} 49 | 50 | Login Page 51 | Title Should Be Login Page 52 | [Actions] 53 | Submit Credentials ==> Welcome Page when ${USERNAME} == ${VALID_USERNAME} and ${PASSWORD} == ${VALID_PASSWORD} 54 | Submit Credentials ==> Error Page otherwise 55 | 56 | Welcome Page 57 | Title Should Be Welcome Page 58 | 59 | Error Page 60 | Title Should Be Error Page 61 | 62 | *** Keywords *** 63 | Submit Credentials 64 | Input Text ${USERNAME_FIELD} ${USERNAME} 65 | Input Password ${PASSWORD_FIELD} ${PASSWORD} 66 | Click Button ${LOGIN_BUTTON} 67 | ``` 68 | 69 | Here is another example machine (using model-based testing with finite state machine): 70 | 71 | ```robotframework 72 | *** Settings *** 73 | Suite Setup Open Browser ${LOGIN_PAGE_URL} googlechrome 74 | Suite Teardown Close Browser 75 | Test Setup Go to ${LOGIN_PAGE_URL} 76 | Library SeleniumLibrary 77 | 78 | *** Variables *** 79 | ${USERNAME_FIELD} username_field 80 | ${PASSWORD_FIELD} password_field 81 | ${LOGIN_BUTTON} LOGIN 82 | ${VALID_USERNAME} demo 83 | ${VALID_PASSWORD} mode 84 | ${LOGIN_PAGE_URL} http://localhost:7272/html/ 85 | 86 | *** Machine *** 87 | Login Page 88 | Title Should Be Login Page 89 | [Actions] 90 | Submit Valid Credentials ==> Welcome Page 91 | Submit Invalid Credentials ==> Error Page 92 | 93 | Welcome Page 94 | Title Should Be Welcome Page 95 | [Actions] 96 | Go to ${LOGIN_PAGE_URL} ==> Login Page 97 | 98 | Error Page 99 | Title Should Be Error Page 100 | [Actions] 101 | Go to ${LOGIN_PAGE_URL} ==> Login Page 102 | 103 | *** Keywords *** 104 | Submit Valid Credentials 105 | Submit Credentials ${VALID_USERNAME} ${VALID_PASSWORD} 106 | 107 | Submit Invalid Credentials 108 | Submit Credentials invalid_username invalid_password 109 | 110 | Submit Credentials 111 | [Arguments] ${username} ${password} 112 | Input Text ${USERNAME_FIELD} ${USERNAME} 113 | Input Password ${PASSWORD_FIELD} ${PASSWORD} 114 | Click Button ${LOGIN_BUTTON} 115 | ``` 116 | 117 | NOTE! This machine can generate infinite number of test sequences thus you need to constraint the generation. 118 | For example: 119 | 120 | robomachine --tests-max 10 --actions-max 20 --to-state 'Welcome Page' --generation-algorithm random [MACHINE_FILE_NAME] 121 | 122 | will generate 10 random tests with at most 20 actions each and all tests ending to state 'Welcome Page'. 123 | 124 | ## Installation 125 | 126 | From [Python Package Index](http://pypi.python.org/pypi) 127 | 128 | pip install RoboMachine 129 | 130 | 131 | From source: 132 | 133 | git clone git://github.com/mkorpela/RoboMachine.git 134 | cd RoboMachine 135 | python setup.py install 136 | 137 | After this you should have a commandline tool called `robomachine` available. 138 | See `robomachine --help` for commandline tool usage. 139 | 140 | ## Syntax 141 | 142 | * Only supported Robot Framework format is space separated format. 143 | * You should use `.robomachine` suffix for your Machine files. 144 | * You can have a Robot Framework *Settings* table at the beginning of the machine file. 145 | * You can have a Robot Framework *Variables* table after the optional Settings table. 146 | * *Machine* table is a must. 147 | * You can have a Robot Framework *Keywords* table after the Machine table. 148 | 149 | ## Machine table syntax 150 | 151 | Used machine variables must be introduced at the beginning of the machine table. Valid machine variable is a valid Robot Framework variable that contains only uppercase characters A-Z, numbers (not at the start) 0-9 and 152 | underscores (`_`). For example: `${VALID_123}` is a valid machine variable. 153 | 154 | An example of a valid machine variable definition line:: 155 | 156 | ${VALID_VARIABLE_1} any of Value 1 Value 2 Value 3 157 | 158 | Rules about the machine variables should be next. Rule building blocks: 159 | * Variable conditions: 160 | * `${VARIABLE} == value` 161 | * `${VARIABLE} != value` 162 | * `${VARIABLE} < value` 163 | * `${VARIABLE} <= value` 164 | * `${VARIABLE} > value` 165 | * `${VARIABLE} >= value` 166 | * `${VARIABLE} in (a, b)` # short for ${VARIABLE} == a or ${VARIABLE} == b 167 | * `${VARIABLE} ~ REGEX` # python regexp: re.search(REGEX, ${VARIABLE}) != None 168 | * `${VARIABLE} !~ REGEX` # negated regexp match 169 | * And rule: [some condition] and [some other condition] 170 | * Or rule: [some condition] or [some other condition] 171 | * Not rule: not ([some condition]) 172 | * Implication rule: [some condition] ==> [some condition] 173 | * Equivalence rule: [some condition] <==> [some condition] 174 | * [some condition]: Variable condition OR (rule) 175 | 176 | Rules can be used to remove variable combinations that should not be used in 177 | test generation. 178 | 179 | An example of a valid rule line: 180 | 181 | ${VARIABLE_1} == Value 1 ==> (${FOO} == bar or ${FOO} == zoo) 182 | 183 | Which means: When `${VARIABLE_1}` value is "Value 1" then `${FOO}` should either be "bar" or "zoo". 184 | 185 | State blocks should be next. First state block is the starting state. 186 | 187 | State block starts with a line containing the states name. Valid state name contains only upper and lowercase characters a-zA-Z, numbers (not at the start) 0-9 and spaces (not at the start or end and only one 188 | between words). 189 | 190 | This is followed by the Robot Framework steps that should be executed when in that 191 | state. 192 | 193 | This can be followed by an actions block definition. 194 | 195 | An actions block starts with `[Actions]` tag and is followed by one or more action lines. 196 | 197 | An action line has four parts: 198 | * A Robot Framework step that is executed when the action happens (action label) (you can also leave this out - use a tau transition) 199 | * ` ==> ` right arrow with two spaces before and after 200 | * Name of the state that the machine ends up when the action is taken (end state) 201 | * Optional rule (when the action is available) this either starts with 202 | ` when ` and the rule or ` otherwise ` - meaning this action should be taken when 203 | all of the other actions with same action label are not available 204 | 205 | An example of a valid state definition:: 206 | 207 | ```robotframework 208 | State Name 209 | Robot Framework Step with ${arguments} 210 | Log something WARN 211 | [Actions] 212 | Log other thing ==> Other State when ${FOO} == bar 213 | Log other thing ==> Another State when (${FOO} == zoo and ${BAR} == goo) 214 | Log other thing ==> End State otherwise 215 | Log nothing ==> End State 216 | ==> Some state # This here is a tau transition that does not write a step to the test 217 | ``` 218 | -------------------------------------------------------------------------------- /atest/infinite.robomachine: -------------------------------------------------------------------------------- 1 | *** Machine *** 2 | ${VAR1} any of A B C D E F G H I J K L M N O P 3 | ${VAR2} any of A B C D E F G H I J K L M N O P 4 | ${VAR3} any of A B C D E F G H I J K L M N O P 5 | ${VAR4} any of A B C D E F G H I J K L M N O P 6 | ${VAR5} any of A B C D E F G H I J K L M N O P 7 | ${VAR6} any of A B C D E F G H I J K L M N O P 8 | ${VAR7} any of A B C D E F G H I J K L M N O P 9 | ${VAR8} any of A B C D E F G H I J K L M N O P 10 | 11 | State 1 12 | [Actions] 13 | action 1 ==> State 1 14 | action 2 ==> State 2 15 | action 3 ==> State 3 16 | action 4 ==> State 4 17 | action 5 ==> State 5 18 | action 6 ==> State 6 19 | action 7 ==> State 7 20 | action 8 ==> State 8 21 | action 9 ==> State 9 22 | 23 | State 2 24 | [Actions] 25 | action 1 ==> State 1 26 | action 2 ==> State 2 27 | action 3 ==> State 3 28 | action 4 ==> State 4 29 | action 5 ==> State 5 30 | action 6 ==> State 6 31 | action 7 ==> State 7 32 | action 8 ==> State 8 33 | action 9 ==> State 9 34 | 35 | State 3 36 | [Actions] 37 | action 1 ==> State 1 38 | action 2 ==> State 2 39 | action 3 ==> State 3 40 | action 4 ==> State 4 41 | action 5 ==> State 5 42 | action 6 ==> State 6 43 | action 7 ==> State 7 44 | action 8 ==> State 8 45 | action 9 ==> State 9 46 | 47 | State 4 48 | [Actions] 49 | action 1 ==> State 1 50 | action 2 ==> State 2 51 | action 3 ==> State 3 52 | action 4 ==> State 4 53 | action 5 ==> State 5 54 | action 6 ==> State 6 55 | action 7 ==> State 7 56 | action 8 ==> State 8 57 | action 9 ==> State 9 58 | 59 | State 5 60 | [Actions] 61 | action 1 ==> State 1 62 | action 2 ==> State 2 63 | action 3 ==> State 3 64 | action 4 ==> State 4 65 | action 5 ==> State 5 66 | action 6 ==> State 6 67 | action 7 ==> State 7 68 | action 8 ==> State 8 69 | action 9 ==> State 9 70 | 71 | State 6 72 | [Actions] 73 | action 1 ==> State 1 74 | action 2 ==> State 2 75 | action 3 ==> State 3 76 | action 4 ==> State 4 77 | action 5 ==> State 5 78 | action 6 ==> State 6 79 | action 7 ==> State 7 80 | action 8 ==> State 8 81 | action 9 ==> State 9 82 | 83 | State 7 84 | [Actions] 85 | action 1 ==> State 1 86 | action 2 ==> State 2 87 | action 3 ==> State 3 88 | action 4 ==> State 4 89 | action 5 ==> State 5 90 | action 6 ==> State 6 91 | action 7 ==> State 7 92 | action 8 ==> State 8 93 | action 9 ==> State 9 94 | 95 | State 8 96 | [Actions] 97 | action 1 ==> State 1 98 | action 2 ==> State 2 99 | action 3 ==> State 3 100 | action 4 ==> State 4 101 | action 5 ==> State 5 102 | action 6 ==> State 6 103 | action 7 ==> State 7 104 | action 8 ==> State 8 105 | action 9 ==> State 9 106 | 107 | State 9 108 | [Actions] 109 | action 1 ==> State 1 110 | action 2 ==> State 2 111 | action 3 ==> State 3 112 | action 4 ==> State 4 113 | action 5 ==> State 5 114 | action 6 ==> State 6 115 | action 7 ==> State 7 116 | action 8 ==> State 8 117 | action 9 ==> State 9 118 | -------------------------------------------------------------------------------- /atest/one_state_no_actions.robomachine: -------------------------------------------------------------------------------- 1 | *** Machine *** 2 | ${LOG_MESSAGE} any of Hello World! Hello Kitty! 3 | 4 | Data driving 5 | Log ${LOG_MESSAGE} 6 | -------------------------------------------------------------------------------- /atest/pass_and_fail_machine.robomachine: -------------------------------------------------------------------------------- 1 | *** Machine *** 2 | Pass Or Fail 3 | [Actions] 4 | No Operation ==> Passing # Should definetly pass 5 | Fail ==> Failing # this should be failing 6 | 7 | Passing 8 | Log this will be reached 9 | 10 | Failing 11 | Log this will not be reached 12 | -------------------------------------------------------------------------------- /atest/selenium_demo_fsm.robomachine: -------------------------------------------------------------------------------- 1 | *** Settings *** 2 | Suite Setup Open Browser ${LOGIN_PAGE_URL} googlechrome 3 | Suite Teardown Close Browser 4 | Test Setup Go to ${LOGIN_PAGE_URL} 5 | Library SeleniumLibrary 6 | 7 | *** Variables *** 8 | ${USERNAME_FIELD} username_field 9 | ${PASSWORD_FIELD} password_field 10 | ${LOGIN_BUTTON} LOGIN 11 | ${VALID_USERNAME} demo 12 | ${VALID_PASSWORD} mode 13 | ${LOGIN_PAGE_URL} http://localhost:7272/html/ 14 | 15 | *** Machine *** 16 | Login Page 17 | Title Should Be Login Page 18 | [Actions] 19 | Submit Valid Credentials ==> Welcome Page 20 | Submit Invalid Credentials ==> Error Page 21 | 22 | Welcome Page 23 | Title Should Be Welcome Page 24 | [Actions] 25 | Go to ${LOGIN_PAGE_URL} ==> Login Page 26 | 27 | Error Page 28 | Title Should Be Error Page 29 | [Actions] 30 | Go to ${LOGIN_PAGE_URL} ==> Login Page 31 | 32 | *** Keywords *** 33 | Submit Valid Credentials 34 | Submit Credentials ${VALID_USERNAME} ${VALID_PASSWORD} 35 | 36 | Submit Invalid Credentials 37 | Submit Credentials invalid_username invalid_password 38 | 39 | Submit Credentials 40 | [Arguments] ${username} ${password} 41 | Input Text ${USERNAME_FIELD} ${USERNAME} 42 | Input Password ${PASSWORD_FIELD} ${PASSWORD} 43 | Click Button ${LOGIN_BUTTON} 44 | -------------------------------------------------------------------------------- /atest/selenium_demo_machine.robomachine: -------------------------------------------------------------------------------- 1 | *** Settings *** 2 | Suite Setup Open Browser ${LOGIN_PAGE_URL} googlechrome 3 | Suite Teardown Close Browser 4 | Test Setup Go to ${LOGIN_PAGE_URL} 5 | Library SeleniumLibrary 6 | 7 | *** Variables *** 8 | ${USERNAME_FIELD} username_field 9 | ${PASSWORD_FIELD} password_field 10 | ${LOGIN_BUTTON} LOGIN 11 | ${VALID_USERNAME} demo 12 | ${VALID_PASSWORD} mode 13 | ${LOGIN_PAGE_URL} http://localhost:7272/html/ 14 | 15 | *** Machine *** 16 | ${USERNAME} any of ${VALID_USERNAME} ${VALID_PASSWORD} invalid123 ${EMPTY} 17 | ${PASSWORD} any of ${VALID_PASSWORD} ${VALID_USERNAME} password123 ${EMPTY} 18 | 19 | # Add an equivalence rule to reduce the number of generated tests 20 | ${USERNAME} == ${VALID_PASSWORD} <==> ${PASSWORD} == ${VALID_USERNAME} 21 | 22 | Login Page 23 | Title Should Be Login Page 24 | [Actions] 25 | Submit Credentials ==> Welcome Page when ${USERNAME} == ${VALID_USERNAME} and ${PASSWORD} == ${VALID_PASSWORD} 26 | Submit Credentials ==> Error Page otherwise 27 | 28 | Welcome Page 29 | Title Should Be Welcome Page 30 | 31 | Error Page 32 | Title Should Be Error Page 33 | 34 | *** Keywords *** 35 | Submit Credentials 36 | Input Text ${USERNAME_FIELD} ${USERNAME} 37 | Input Password ${PASSWORD_FIELD} ${PASSWORD} 38 | Click Button ${LOGIN_BUTTON} 39 | -------------------------------------------------------------------------------- /atest/testingmachine.robomachine: -------------------------------------------------------------------------------- 1 | *** Machine *** 2 | 3 | ${FOO} any of foo bar zoo 4 | ${BAR} any of some one out there 5 | 6 | (${FOO} != bar and ${FOO} != zoo) ==> ${BAR} == there 7 | ${BAR} == some ==> (${FOO} == bar or ${FOO} == zoo) 8 | 9 | Starting State 10 | Log hello ${FOO} 11 | [Actions] 12 | No Operation ==> End State 13 | Log nothing ==> Starting State 14 | 15 | End State 16 | Log ending ${FOO} 17 | -------------------------------------------------------------------------------- /copyright.txt: -------------------------------------------------------------------------------- 1 | Copyright 2011-2022 Mikko Korpela 2 | 3 | Licensed under the Apache License, Version 2.0 (the "License"); 4 | you may not use this file except in compliance with the License. 5 | You may obtain a copy of the License at 6 | 7 | http://www.apache.org/licenses/LICENSE-2.0 8 | 9 | Unless required by applicable law or agreed to in writing, software 10 | distributed under the License is distributed on an "AS IS" BASIS, 11 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | See the License for the specific language governing permissions and 13 | limitations under the License. 14 | -------------------------------------------------------------------------------- /example/webapp/.gitignore: -------------------------------------------------------------------------------- 1 | *.robot 2 | *.dot 3 | *.dot.* 4 | -------------------------------------------------------------------------------- /example/webapp/KeywordLibrary.py: -------------------------------------------------------------------------------- 1 | """ 2 | This is the keyword library for the webapp sample application, 3 | a mock for a proper System Under Test. 4 | """ 5 | from robot.api import logger 6 | from random import random 7 | 8 | from robot.libraries.BuiltIn import BuiltIn 9 | 10 | 11 | # --------------------------------------------- 12 | # Error-inducing decorators 13 | # 14 | # Indended usage: 15 | # Decorate a function in the keyword library 16 | # with one of the supplied decorators and try 17 | # to find the failure(s) with your tests. 18 | # The idea is to learn what types of tests 19 | # that are required to find certain types 20 | # of failures. 21 | # --------------------------------------------- 22 | class SimulatedFailureException(Exception): 23 | """ 24 | Raise this exception in the supplied 25 | decorators when a function is supposed 26 | to fail 27 | """ 28 | 29 | 30 | def fail_every_nth_time(count, hard=True): 31 | """ 32 | This decorator will skip the decorated function every 33 | n-th time it is called. A hard failure (hard=True) will 34 | raise an exception. A soft failure (hard=False) will stop 35 | the decorated function from doing what it was supposed to. 36 | 37 | *Usage example:* 38 | 39 | @fail_every_nth_time(12) 40 | def my_func(*args): 41 | ... 42 | 43 | This example will cause my_func to return an unexpected 44 | value every 12:th time it is called 45 | """ 46 | def decorator(func): 47 | def inner(*args, **kwargs): 48 | inner.count += 1 49 | if inner.count % count == 0: 50 | if hard is True: 51 | raise SimulatedFailureException( 52 | '\n Fail in method `{:s}` (call count)'.format( 53 | func.__name__)) 54 | else: 55 | logger.warn('Skipping method `{:s}` (call count)'.format( 56 | func.__name__)) 57 | else: 58 | func(*args, **kwargs) 59 | return inner 60 | inner.count = 0 61 | inner.__name__ = func.__name__ 62 | return inner 63 | return decorator 64 | 65 | 66 | def fail_with_probability(prob, hard=True): 67 | """ 68 | This decorator will skip the decorated function based on the supplied 69 | probability of failure [0..1]. A hard failure (hard=True) will 70 | raise an exception. A soft failure (hard=False) will stop 71 | the decorated function from doing what it was supposed to. 72 | 73 | *Usage example:* 74 | 75 | @fail_with_probability(0.25) 76 | def my_func(*args): 77 | ... 78 | 79 | This example will cause my_func to return an unexpected 80 | value on average 25% of the times it is called 81 | """ 82 | def decorator(func): 83 | def inner(*args, **kwargs): 84 | if random() <= prob: 85 | if hard is True: 86 | raise SimulatedFailureException( 87 | '\n Fail in method `{:s}` (probability)'.format( 88 | func.__name__)) 89 | else: 90 | logger.warn('Skipping method `{:s}` (probability)'.format( 91 | func.__name__)) 92 | else: 93 | func(*args, **kwargs) 94 | return inner 95 | inner.__name__ = func.__name__ 96 | return inner 97 | return decorator 98 | 99 | 100 | def fail_in_config(bad_configs, hard=True): 101 | """ 102 | This decorator will skip the decorated function if the supplied 103 | dictionary is a subset of the available RobotFramework variables. 104 | A hard failure (hard=True) will raise an exception. A soft failure 105 | (hard=False) will stop the decorated function from doing what it 106 | was supposed to. 107 | 108 | *Usage example:* 109 | 110 | @fail_in_config({'${USER}': 'monkey'}) 111 | @fail_in_config({'${CITY}': 'Boston'}) 112 | def my_func(*args): 113 | ... 114 | 115 | This example will cause my_func to return an unexpected 116 | value if ${USER} is 'monkey' or if ${CITY} is 'Boston' 117 | """ 118 | def decorator(func): 119 | def inner(*args, **kwargs): 120 | variables = BuiltIn().get_variables() 121 | if dict(variables, **bad_configs) == dict(variables): 122 | if hard is True: 123 | raise SimulatedFailureException( 124 | '\n Fail in method `{:s}` '.format(func.__name__) + 125 | '(configuration: {:s})'.format(str(bad_configs))) 126 | else: 127 | logger.warn( 128 | 'Skipping method `{:s}` '.format(func.__name__) + 129 | '(configuration: {:s})'.format(str(bad_configs))) 130 | else: 131 | func(*args, **kwargs) 132 | return inner 133 | inner.__name__ = func.__name__ 134 | return inner 135 | return decorator 136 | 137 | 138 | # ---------------------------- 139 | # The actual keyword library 140 | # ---------------------------- 141 | class KeywordLibrary(object): 142 | """ 143 | This is the implementation of the keywords used in the robomachine test. 144 | 145 | This class mimics the observable behavior of a real application and 146 | is intended to be used for educational purposes. Insert errors, re-run 147 | your tests and try to find them! 148 | """ 149 | def __init__(self): 150 | self._browser = None 151 | self._state = None 152 | self._name = None 153 | self._password = None 154 | self._page_title = '' 155 | self._change_state('Login Page') 156 | 157 | # 158 | # ASSERTS 159 | # 160 | def assert_state_is(self, state): 161 | """Asserts the page title""" 162 | if state != self._state: 163 | raise Exception( 164 | 'Wrong state! The state is ' + 165 | '`{:s}`, expected `{:s}`.'.format(self._state, state)) 166 | else: 167 | logger.info('State: `%s`' % self._state) 168 | 169 | def assert_page_title_is(self, title): 170 | """Asserts the page title""" 171 | if title != self._page_title: 172 | raise Exception( 173 | 'Wrong page! The title at page `{:s}` '.format(self._state) + 174 | 'was `{:s}`, expected `{:s}`.'.format(self._page_title, title)) 175 | else: 176 | logger.info('Page title: `%s`' % self._page_title) 177 | 178 | # 179 | # USER ACTIONS 180 | # 181 | # In a live situation, we would call functions 182 | # that invokes functionality in the System Under 183 | # Test instead of our simple SUT mock 184 | # (the self._change_state() method) 185 | def click_login_button(self): 186 | """Execute the login""" 187 | if self._name == 'My Name' and self._password == 'mypassword': 188 | self._change_state('Welcome Page') 189 | else: 190 | self._change_state('Error Page') 191 | 192 | def click_log_out_button(self): 193 | """Log out and go to the login page""" 194 | self._change_state('Login Page') 195 | 196 | def enter_password(self, password): 197 | """Enter the password""" 198 | self._password = password 199 | logger.info('Password = `%s`' % password) 200 | 201 | def enter_username(self, name): 202 | """Enter the username""" 203 | self._name = name 204 | logger.info('User name = `%s`' % name) 205 | 206 | def start_browser(self, browser): 207 | """Emulate a browser start""" 208 | self._browser = browser 209 | 210 | # 211 | # APPLICATION FLOW ACTIONS 212 | # 213 | def exit_page(self): 214 | """Exit current page and change state""" 215 | if self._state == 'Error Page': 216 | self._change_state('Login Page') 217 | 218 | elif self._state == 'Profile Page': 219 | self._change_state('Welcome Page') 220 | 221 | def go_to(self, state): 222 | """Change state""" 223 | self._change_state(state) 224 | 225 | # 226 | # PRIVATE MEMBERS 227 | # 228 | def _change_state(self, new_state): 229 | """Set page titles according to current page""" 230 | if new_state == 'Login Page': 231 | self._name = '' 232 | self._password = '' 233 | self._page_title = 'Please log in!' 234 | 235 | elif new_state == 'Welcome Page': 236 | self._page_title = 'This is the welcome page!' 237 | 238 | elif new_state == 'Profile Page': 239 | self._page_title = 'Hello, %s!' % self._name 240 | 241 | elif new_state == 'Error Page': 242 | self._page_title = 'Oops, something went wrong!' 243 | 244 | self._state = new_state 245 | logger.info('Changed state to `%s`' % new_state) 246 | -------------------------------------------------------------------------------- /example/webapp/LICENSE.md: -------------------------------------------------------------------------------- 1 | The files in this folder are all copyright (c) 2017 David Kaplan, handitover.se 2 | 3 | Licensed under the Apache License, Version 2.0 (the "License"); 4 | you may not use this file except in compliance with the License. 5 | You may obtain a copy of the License at 6 | 7 | http://www.apache.org/licenses/LICENSE-2.0 8 | 9 | Unless required by applicable law or agreed to in writing, software 10 | distributed under the License is distributed on an "AS IS" BASIS, 11 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | See the License for the specific language governing permissions and 13 | limitations under the License. -------------------------------------------------------------------------------- /example/webapp/model.robomachine: -------------------------------------------------------------------------------- 1 | *** Settings *** 2 | Library KeywordLibrary.py 3 | Test Setup go to Init 4 | 5 | 6 | *** Variables *** 7 | ${VALID_USERNAME} My Name 8 | ${VALID_PASSWORD} mypassword 9 | 10 | 11 | *** Machine *** 12 | ${USERNAME} any of ${VALID_USERNAME} ${VALID_PASSWORD} invalid123 ${EMPTY} 13 | ${PASSWORD} any of ${VALID_PASSWORD} ${VALID_USERNAME} password123 ${EMPTY} 14 | ${BROWSER} any of Safari Chrome Firefox Edge 15 | 16 | 17 | # Add an equivalence rule to reduce the number of generated tests. 18 | # N.B: Such rules can't be used with allpairs-random strategy! 19 | # ${USERNAME} == ${VALID_PASSWORD} <==> ${PASSWORD} == ${VALID_USERNAME} 20 | 21 | 22 | Init 23 | start browser ${BROWSER} 24 | assert state is Init 25 | [Actions] 26 | go to Login Page ==> Login Page 27 | 28 | Login Page 29 | assert state is Login Page 30 | assert page title is Please log in! 31 | [Actions] 32 | submit credentials ==> Welcome Page when ${USERNAME} == ${VALID_USERNAME} and ${PASSWORD} == ${VALID_PASSWORD} 33 | submit credentials ==> Error Page otherwise 34 | 35 | Welcome Page 36 | assert state is Welcome Page 37 | assert page title is This is the welcome page! 38 | [Actions] 39 | go to Profile Page ==> Profile Page 40 | click log out button ==> Login Page 41 | 42 | Profile Page 43 | assert state is Profile Page 44 | assert page title is Hello, ${USERNAME}! 45 | [Actions] 46 | exit page ==> Welcome Page 47 | 48 | Error Page 49 | assert state is Error Page 50 | assert page title is Oops, something went wrong! 51 | [Actions] 52 | exit page ==> Login Page 53 | 54 | 55 | *** Keywords *** 56 | submit credentials 57 | enter username ${USERNAME} 58 | enter password ${PASSWORD} 59 | click login button 60 | 61 | -------------------------------------------------------------------------------- /example/webapp/test.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | BASENAME=model 3 | 4 | export PYTHONPATH=../.. 5 | INFILE=$BASENAME.robomachine 6 | OUTFILE=$BASENAME.robot 7 | 8 | python -m robomachine.runner --tests-max 20 --actions-max 8 \ 9 | --output $OUTFILE --generate-dot-graph svg \ 10 | --generation-algorithm allpairs-random \ 11 | $INFILE -------------------------------------------------------------------------------- /pyproject.toml: -------------------------------------------------------------------------------- 1 | [build-system] 2 | requires = ["setuptools"] 3 | build-backend = "setuptools.build_meta" -------------------------------------------------------------------------------- /release.sh: -------------------------------------------------------------------------------- 1 | cd .. 2 | git clone git@github.com:mkorpela/RoboMachine.git robomachinerelease 3 | cd robomachinerelease 4 | python setup.py sdist 5 | twine upload -r RoboMachine dist/*.* 6 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | argparse~=1.4.0 2 | pyparsing~=2.4.7 3 | allpairspy~=2.5.0 4 | pip~=22.2.2 5 | wheel~=0.37.1 6 | setuptools~=65.3.0 7 | robotframework~=6.0.1 -------------------------------------------------------------------------------- /setup.cfg: -------------------------------------------------------------------------------- 1 | [metadata] 2 | name = RoboMachine 3 | author = Mikko Korpela 4 | author_email = mikko.korpela@gmail.com 5 | url = https://github.com/mkorpela/RoboMachine 6 | download_url = https://pypi.org/project/RoboMachine/ 7 | project_urls = 8 | Source = https://github.com/mkorpela/RoboMachine 9 | version = attr: robomachine.__version__ 10 | description = Test data generator for Robot Framework 11 | long_description = file: README.md 12 | long_description_content_type = text/markdown 13 | license = Apache License, Version 2.0 14 | classifiers = 15 | Intended Audience :: Developers 16 | Natural Language :: English 17 | Programming Language :: Python :: 3 18 | Topic :: Software Development :: Testing 19 | License :: OSI Approved :: Apache Software License 20 | Framework :: Robot Framework 21 | 22 | [options] 23 | python_requires = >=3.6 24 | package_dir = 25 | = src 26 | packages=find: 27 | include_package_data = True 28 | install_requires = 29 | robotframework>=3.2 30 | pyparsing==2.4.7 31 | argparse 32 | allpairspy 33 | 34 | [options.packages.find] 35 | where=src 36 | 37 | [options.entry_points] 38 | console_scripts = 39 | robomachine = robomachine.runner:main -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | from distutils.core import setup 4 | 5 | setup() 6 | -------------------------------------------------------------------------------- /src/robomachine/__init__.py: -------------------------------------------------------------------------------- 1 | # Copyright 2011-2012 Mikko Korpela 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | try: 16 | from StringIO import StringIO 17 | except: 18 | from io import StringIO 19 | 20 | __version__ = "0.10.0" 21 | from .parsing import parse 22 | 23 | from .generator import Generator, DepthFirstSearchStrategy 24 | 25 | def generate(machine, max_tests=1000, max_actions=None, to_state=None, output=None, 26 | strategy=DepthFirstSearchStrategy): 27 | generator = Generator() 28 | return generator.generate(machine, max_tests, max_actions, to_state, output, strategy) 29 | 30 | def transform(text): 31 | output = StringIO() 32 | generate(parse(text), output=output) 33 | return output.getvalue() -------------------------------------------------------------------------------- /src/robomachine/allpairsstrategy.py: -------------------------------------------------------------------------------- 1 | # Copyright 2012 Mikko Korpela 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | from allpairspy import AllPairs 16 | from .strategies import RandomStrategy 17 | 18 | 19 | class AllPairsRandomStrategy(RandomStrategy): 20 | 21 | def __init__(self, machine, max_actions, to_state=None): 22 | if machine.rules: 23 | raise AssertionError('ERROR! AllPairs does not work correctly with rules') 24 | RandomStrategy.__init__(self, machine, max_actions, to_state) 25 | 26 | def tests(self): 27 | for values in self._generate_all_pairs_variable_values(): 28 | test = self._generate_test(values) 29 | if not test and self._to_state and self._to_state != self._machine.start_state.name: 30 | continue 31 | yield test, [v.current_value for v in self._machine.variables] 32 | 33 | def _generate_all_pairs_variable_values(self): 34 | if len(list(self._machine.variables)) < 2: 35 | for var in self._machine.variables: 36 | return [v for v in var.values] 37 | return [[]] 38 | return AllPairs([v.values for v in self._machine.variables]) 39 | -------------------------------------------------------------------------------- /src/robomachine/generator.py: -------------------------------------------------------------------------------- 1 | # Copyright 2011-2012 Mikko Korpela 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | 16 | from __future__ import print_function 17 | 18 | from .parsing import parse 19 | from .strategies import DepthFirstSearchStrategy 20 | 21 | try: 22 | from StringIO import StringIO 23 | except ImportError: 24 | from io import StringIO 25 | 26 | 27 | class Generator(object): 28 | def __init__(self): 29 | self.visited_states = set() 30 | self.visited_actions = set() 31 | 32 | def _write_test(self, name, machine, output, test, values): 33 | output.write('\n{:s}\n'.format(name)) 34 | if values: 35 | machine.write_variable_setting_step(values, output) 36 | machine.start_state.write_to(output) 37 | self.visited_states.add(machine.start_state) 38 | for action in test: 39 | self.visited_actions.add(action) 40 | self.visited_states.add(action.next_state) 41 | action.write_to(output) 42 | 43 | def _write_tests(self, machine, max_tests, max_actions, to_state, output, strategy): 44 | i = 1 45 | skipped = 0 46 | generated_tests = set() 47 | 48 | strategy_class = strategy(machine, max_actions, to_state) 49 | for test, values in strategy_class.tests(): 50 | if i + skipped > max_tests: 51 | print('--tests-max generation try limit {:d} reached with {:d} tests generated'.format(max_tests, i - 1)) 52 | break 53 | if (tuple(test), tuple(values)) in generated_tests: 54 | skipped += 1 55 | continue 56 | else: 57 | generated_tests.add((tuple(test), tuple(values))) 58 | self._write_test('Test {:d}'.format(i), machine, output, test, values) 59 | i += 1 60 | 61 | 62 | def generate(self, machine, max_tests=1000, max_actions=None, to_state=None, output=None, 63 | strategy=DepthFirstSearchStrategy): 64 | max_actions = -1 if max_actions is None else max_actions 65 | machine.write_settings_table(output) 66 | machine.write_variables_table(output) 67 | output.write('*** Test Cases ***') 68 | self._write_tests(machine, max_tests, max_actions, to_state, output, strategy) 69 | machine.write_keywords_table(output) 70 | 71 | 72 | def transform(self, text): 73 | output = StringIO() 74 | self.generate(parse(text), output=output) 75 | return output.getvalue() 76 | -------------------------------------------------------------------------------- /src/robomachine/model.py: -------------------------------------------------------------------------------- 1 | # Copyright 2011-2012 Mikko Korpela 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | import re 16 | 17 | 18 | class RoboMachine(object): 19 | 20 | def __init__(self, states, variables, rules, settings_table=None, 21 | variables_table=None, keywords_table=None): 22 | self.states = states or [] 23 | self.variables = variables or [] 24 | self.rules = rules or [] 25 | self._settings_table = settings_table or [] 26 | self._variables_table = variables_table or [] 27 | self._keywords_table = keywords_table or [] 28 | for state in self.states: 29 | state.set_machine(self) 30 | for variable in self.variables: 31 | variable.set_machine(self) 32 | 33 | @property 34 | def start_state(self): 35 | return self.states[0] 36 | 37 | @property 38 | def variable_value_mapping(self): 39 | return dict((v.name, v.current_value) for v in self.variables) 40 | 41 | def find_state_by_name(self, name): 42 | for state in self.states: 43 | if state.name == name: 44 | return state 45 | return None 46 | 47 | def find_variable_by_name(self, name): 48 | for variable in self.variables: 49 | if variable.name == name: 50 | return variable 51 | return None 52 | 53 | def write_settings_table(self, output): 54 | for content in self._settings_table: 55 | output.write(content) 56 | 57 | def write_variables_table(self, output): 58 | for content in self._variables_table: 59 | output.write(content) 60 | 61 | def write_keywords_table(self, output): 62 | for content in self._keywords_table: 63 | output.write('\n'+content) 64 | if not self._keywords_table: 65 | output.write('\n*** Keywords ***\n') 66 | if self.variables: 67 | self.write_variable_setter(output) 68 | for state in self.states: 69 | if state.steps: 70 | output.write(state.name+'\n') 71 | state.write_steps_to(output) 72 | 73 | def write_variable_setter(self, output): 74 | output.write('Set Machine Variables\n') 75 | output.write(' [Arguments] {:s}\n'.format(' '.join(variable.name for variable in self.variables))) 76 | for variable in self.variables: 77 | output.write(' Set Test Variable \\{:s}\n'.format(variable.name)) 78 | 79 | @staticmethod 80 | def write_variable_setting_step(values, output): 81 | output.write(' Set Machine Variables {:s}\n'.format(' '.join(values))) 82 | 83 | def rules_are_ok(self, values): 84 | value_mapping = dict((v.name, value) for v, value in zip(self.variables, values)) 85 | for rule in self.rules: 86 | if not rule.is_valid(value_mapping=value_mapping): 87 | return False 88 | return True 89 | 90 | def apply_variable_values(self, values): 91 | for variable, value in zip(self.variables, values): 92 | variable.set_current_value(value) 93 | 94 | 95 | class State(object): 96 | 97 | def __init__(self, name, steps, actions): 98 | self.name = name 99 | self.steps = steps or [] 100 | self._actions = actions or [] 101 | 102 | @property 103 | def actions(self): 104 | result = [] 105 | names = set() 106 | for action in self._actions: 107 | if action.is_available() and action.name not in names: 108 | result.append(action) 109 | names.add(action.name) 110 | return result 111 | 112 | def set_machine(self, machine): 113 | for action in self._actions: 114 | action.set_machine(machine) 115 | 116 | def write_steps_to(self, output): 117 | for step in self.steps: 118 | output.write(step + '\n') 119 | 120 | def write_to(self, output): 121 | if self.steps: 122 | output.write(' {:s}\n'.format(self.name)) 123 | 124 | 125 | class Action(object): 126 | 127 | def __init__(self, name, next_state, condition=None): 128 | self.name = name 129 | self._next_state_name = next_state 130 | self.condition = condition 131 | self._machine = None 132 | 133 | def set_machine(self, machine): 134 | self._machine = machine 135 | if not self.next_state: 136 | raise AssertionError('Invalid end state "{:s}" in '.format(self._next_state_name) + 137 | 'action "{:s}"!'.format(self.name)) 138 | 139 | @property 140 | def next_state(self): 141 | return self._machine.find_state_by_name(self._next_state_name) 142 | 143 | def is_available(self): 144 | if not self.condition: 145 | return True 146 | if self.condition == 'otherwise': 147 | return True 148 | return self.condition.is_valid(value_mapping=self._machine.variable_value_mapping) 149 | 150 | def write_to(self, output): 151 | if self.name: 152 | output.write(' {:s}\n'.format(self.name)) 153 | self.next_state.write_to(output) 154 | 155 | 156 | class Variable(object): 157 | REGEX = r'\$\{[_A-Z][_A-Z0-9]*\}' 158 | PATTERN = re.compile(REGEX) 159 | _NO_VALUE = object() 160 | 161 | def __init__(self, name, values): 162 | self.name = name 163 | self.values = values 164 | self._current_value = Variable._NO_VALUE 165 | self._machine = None 166 | 167 | def set_machine(self, machine): 168 | self._machine = machine 169 | 170 | def set_current_value(self, value): 171 | self._current_value = value 172 | 173 | @property 174 | def current_value(self): 175 | if self._current_value is Variable._NO_VALUE: 176 | raise AssertionError('No current value set') 177 | return self._resolve_value(self._current_value) 178 | 179 | def _resolve_value(self, value): 180 | return self.PATTERN.sub(self._resolve_variable, value) 181 | 182 | def _resolve_variable(self, var_match): 183 | var = self._machine.find_variable_by_name(var_match.group(0)) 184 | if not var: 185 | return var_match.group(0) 186 | return var.current_value 187 | -------------------------------------------------------------------------------- /src/robomachine/parsing.py: -------------------------------------------------------------------------------- 1 | # Copyright 2011-2012 Mikko Korpela 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | from __future__ import print_function 16 | 17 | from pyparsing import (CharsNotIn, Forward, Literal, LineEnd, OneOrMore, Optional, 18 | Regex, StringEnd, White, Word, ZeroOrMore, 19 | delimitedList, printables, 20 | ParseBaseException) 21 | from .model import RoboMachine, State, Action, Variable 22 | from .rules import (AndRule, Condition, EquivalenceRule, OrRule, 23 | NotRule, ImplicationRule, UnequalCondition, 24 | GreaterThanCondition, GreaterThanOrEqualCondition, 25 | LessThanCondition, LessThanOrEqualCondition, 26 | RegexCondition, RegexNegatedCondition) 27 | 28 | 29 | end_of_line = Regex(r' *\n') ^ LineEnd() 30 | 31 | settings_table = Literal('*** Settings ***') + Regex(r'[^\*]+(?=\*)') 32 | settings_table.setParseAction(lambda t: '\n'.join(t)) 33 | variables_table = Literal('*** Variables ***') + Regex(r'[^\*]+(?=\*)') 34 | variables_table.setParseAction(lambda t: '\n'.join(t)) 35 | keywords_table = Literal('*** Keywords ***') + CharsNotIn('') + StringEnd() 36 | keywords_table.setParseAction(lambda t: '\n'.join(t)) 37 | 38 | state_name = Regex(r'\w+( \w+)*') 39 | state_name.leaveWhitespace() 40 | state_name = state_name.setResultsName('state_name') 41 | 42 | robo_step = Regex(r'([\w\$\{\}][ \w\$\{\}]*[\w\}]|\w)') 43 | robo_step.leaveWhitespace() 44 | robo_step = robo_step.setResultsName('robo_step') 45 | 46 | variable = Regex(Variable.REGEX) 47 | 48 | variable_value = Regex(r'[\w\$\{\}!?\-\=\_\.\/]+( [\w\$\{\}!?\-\=\_\.\/]+)*') 49 | 50 | splitter = Literal(' ') + OneOrMore(' ') 51 | splitter.setParseAction(lambda t: ' ') 52 | 53 | variable_values = (variable_value + ZeroOrMore(splitter + variable_value)).setResultsName('variable_values') 54 | variable_values.setParseAction(lambda t: [[t[2 * i] for i in range(int((len(t) + 1) / 2))]]) 55 | 56 | variable_definition = variable.setResultsName( 57 | 'variable_name') + splitter + 'any of' + splitter + variable_values + end_of_line 58 | variable_definition.leaveWhitespace() 59 | variable_definition.setParseAction(lambda t: [Variable(t.variable_name, list(t.variable_values))]) 60 | 61 | rule = Forward() 62 | 63 | condition_rule = variable + ' == ' + variable_value 64 | condition_rule.setParseAction(lambda t: [Condition(t[0], t[2])]) 65 | condition_rule.leaveWhitespace() 66 | 67 | unequal_condition_rule = variable + ' != ' + variable_value 68 | unequal_condition_rule.setParseAction(lambda t: [UnequalCondition(t[0], t[2])]) 69 | unequal_condition_rule.leaveWhitespace() 70 | 71 | cond_gt_rule = variable + ' > ' + variable_value 72 | cond_gt_rule.setParseAction(lambda t: [GreaterThanCondition(t[0], t[2])]) 73 | cond_gt_rule.leaveWhitespace() 74 | 75 | cond_ge_rule = variable + ' >= ' + variable_value 76 | cond_ge_rule.setParseAction(lambda t: [GreaterThanOrEqualCondition(t[0], t[2])]) 77 | cond_ge_rule.leaveWhitespace() 78 | 79 | cond_lt_rule = variable + ' < ' + variable_value 80 | cond_lt_rule.setParseAction(lambda t: [LessThanCondition(t[0], t[2])]) 81 | cond_lt_rule.leaveWhitespace() 82 | 83 | cond_le_rule = variable + ' <= ' + variable_value 84 | cond_le_rule.setParseAction(lambda t: [LessThanOrEqualCondition(t[0], t[2])]) 85 | cond_le_rule.leaveWhitespace() 86 | 87 | cond_in_rule = (variable + ' in ' + Literal('(').suppress() + 88 | variable_value + ZeroOrMore((Literal(',') + Optional(Literal(' '))).suppress() + variable_value) + 89 | Literal(')').suppress()) 90 | cond_in_rule.setParseAction(lambda t: [OrRule([Condition(t[0], t[i]) for i in range(2, len(t))])]) 91 | cond_in_rule.leaveWhitespace() 92 | 93 | cond_regex_rule = variable + ' ~ ' + delimitedList(Word(printables), delim=' ', combine=True) 94 | cond_regex_rule.setParseAction(lambda t: [RegexCondition(t[0], t[2])]) 95 | cond_regex_rule.leaveWhitespace() 96 | 97 | cond_regex_neg_rule = variable + ' !~ ' + delimitedList(Word(printables), delim=' ', combine=True) 98 | cond_regex_neg_rule.setParseAction(lambda t: [RegexNegatedCondition(t[0], t[2])]) 99 | cond_regex_neg_rule.leaveWhitespace() 100 | 101 | closed_rule = condition_rule ^ unequal_condition_rule ^ cond_gt_rule ^ cond_ge_rule ^ cond_lt_rule ^ \ 102 | cond_le_rule ^ cond_in_rule ^ cond_regex_rule ^ cond_regex_neg_rule ^ ('(' + rule + ')') 103 | closed_rule.setParseAction(lambda t: [t[1]] if len(t) == 3 else t) 104 | 105 | not_rule = Literal('not ') + closed_rule 106 | not_rule.leaveWhitespace() 107 | not_rule.setParseAction(lambda t: [NotRule(t[1])]) 108 | 109 | equivalence_rule = closed_rule + splitter + '<==>' + splitter + closed_rule 110 | equivalence_rule.leaveWhitespace() 111 | equivalence_rule.setParseAction(lambda t: [EquivalenceRule(t[0], t[4])]) 112 | 113 | implication_rule = closed_rule + splitter + '==>' + splitter + closed_rule 114 | implication_rule.leaveWhitespace() 115 | implication_rule.setParseAction(lambda t: [ImplicationRule(t[0], t[4])]) 116 | 117 | and_rule = closed_rule + ZeroOrMore(splitter + 'and' + splitter + closed_rule) 118 | and_rule.setParseAction(lambda t: [AndRule([t[i] for i in range(len(t)) if i % 4 == 0])]) 119 | and_rule.leaveWhitespace() 120 | 121 | or_rule = closed_rule + ZeroOrMore(splitter + 'or' + splitter + closed_rule) 122 | or_rule.setParseAction(lambda t: [OrRule([t[i] for i in range(len(t)) if i % 4 == 0])]) 123 | or_rule.leaveWhitespace() 124 | 125 | rule << (not_rule ^ equivalence_rule ^ implication_rule ^ and_rule ^ or_rule ^ closed_rule) 126 | 127 | step = Regex(r' [^\n\[][^\n]*(?=\n)') + LineEnd() 128 | step.leaveWhitespace() 129 | step.setParseAction(lambda t: [t[0]]) 130 | 131 | action_header = White(min=2) + '[Actions]' 132 | 133 | condition = splitter + Literal('when') + splitter + rule 134 | condition = condition ^ Regex(r' +otherwise') 135 | 136 | 137 | def parse_condition(cond): 138 | if len(cond) > 1: 139 | return [cond[3]] 140 | return ['otherwise'] 141 | 142 | 143 | condition.leaveWhitespace() 144 | condition.setParseAction(parse_condition) 145 | condition = Optional(condition).setResultsName('condition') 146 | action = White(min=4) + Optional(robo_step + White(min=2)) + \ 147 | '==>' + White(min=2) + state_name + condition + end_of_line 148 | action.leaveWhitespace() 149 | action.setParseAction(lambda t: [Action(t.robo_step.rstrip(), t.state_name, t.condition)]) 150 | 151 | actions = action_header + end_of_line + OneOrMore(action).setResultsName('actions') 152 | actions = Optional(actions) 153 | actions.leaveWhitespace() 154 | actions.setResultsName('actions') 155 | 156 | comment = Regex(r'(^\s*\#[^\n]*\n)|(\s\s+\#[^\n]*(?=\n))|(\n\s*\#[^\n]*)') 157 | comment.leaveWhitespace() 158 | 159 | steps = ZeroOrMore(step).setResultsName('steps') 160 | 161 | state = state_name + end_of_line + steps + actions 162 | state.leaveWhitespace() 163 | state.setParseAction(lambda p: State(p.state_name, list(p.steps), list(p.actions))) 164 | 165 | machine_header = Literal('*** Machine ***') + end_of_line 166 | states = state + ZeroOrMore(OneOrMore(LineEnd()) + state) 167 | states.setParseAction(lambda t: [[t[2 * i] for i in range(int((len(t) + 1) / 2))]]) 168 | states = states.setResultsName('states') 169 | variables = ZeroOrMore(variable_definition).setResultsName('variables') 170 | rules = ZeroOrMore(rule + end_of_line).setResultsName('rules') 171 | rules.setParseAction(lambda t: [t[i] for i in range(len(t)) if i % 2 == 0]) 172 | machine = Optional(settings_table).setResultsName('settings_table') + \ 173 | Optional(variables_table).setResultsName('variables_table') + \ 174 | machine_header + ZeroOrMore(end_of_line) + variables + \ 175 | ZeroOrMore(end_of_line) + rules + \ 176 | ZeroOrMore(end_of_line) + states + \ 177 | Optional(keywords_table).setResultsName('keywords_table') 178 | 179 | 180 | def create_robomachine(p): 181 | # For some reason, p.rules contains only the _first_ rule. Work around it 182 | # by finding rule elements based on their type. 183 | def is_rule(obj): 184 | return isinstance(obj, (EquivalenceRule, ImplicationRule, AndRule, 185 | OrRule, NotRule)) 186 | 187 | rules = [v for v in p if is_rule(v)] 188 | return RoboMachine(list(p.states), 189 | list(p.variables), 190 | rules, # p.rules contains only first rule(!) 191 | settings_table=p.settings_table, 192 | variables_table=p.variables_table, 193 | keywords_table=p.keywords_table) 194 | 195 | 196 | machine.setParseAction(create_robomachine) 197 | machine.ignore(comment) 198 | machine.setWhitespaceChars(' ') 199 | 200 | 201 | class RoboMachineParsingException(Exception): 202 | pass 203 | 204 | 205 | def resolve_whitespace(text): 206 | output_texts = [] 207 | for index, line in enumerate(text.splitlines()): 208 | if '\t' in line: 209 | print('WARNING! tab detected on line [{:d}]: {:r}'.format(index, line)) 210 | output_texts.append(line.rstrip()) 211 | return '\n'.join(output_texts).strip() + '\n' 212 | 213 | 214 | def parse(text): 215 | try: 216 | return machine.parseString(resolve_whitespace(text), parseAll=True)[0] 217 | except ParseBaseException as pe: 218 | print('Exception at line {:d}'.format(pe.lineno)) 219 | print(pe.msg) 220 | print('line: "{:s}"'.format(pe.line)) 221 | raise RoboMachineParsingException(pe.msg) 222 | except AssertionError as ae: 223 | print(ae) 224 | raise RoboMachineParsingException(ae) 225 | -------------------------------------------------------------------------------- /src/robomachine/rules.py: -------------------------------------------------------------------------------- 1 | # Copyright 2011-2012 Mikko Korpela 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | import re 16 | 17 | 18 | class EquivalenceRule(object): 19 | 20 | def __init__(self, condition1, condition2): 21 | self._condition1 = condition1 22 | self._condition2 = condition2 23 | 24 | def __str__(self): 25 | return '{:s} <==> {:s}'.format(str(self._condition1), str(self._condition2)) 26 | 27 | def is_valid(self, value_mapping): 28 | return self._condition1.is_valid(value_mapping) == self._condition2.is_valid(value_mapping) 29 | 30 | 31 | class ImplicationRule(object): 32 | 33 | def __init__(self, condition1, condition2): 34 | self._condition1 = condition1 35 | self._condition2 = condition2 36 | 37 | def __str__(self): 38 | return '{:s} ==> {:s}'.format(str(self._condition1), str(self._condition2)) 39 | 40 | def is_valid(self, value_mapping): 41 | return not self._condition1.is_valid(value_mapping) or self._condition2.is_valid(value_mapping) 42 | 43 | 44 | class AndRule(object): 45 | 46 | def __init__(self, conditions): 47 | self._conditions = conditions 48 | 49 | def __str__(self): 50 | return ' and '.join(str(c) for c in self._conditions) 51 | 52 | def is_valid(self, value_mapping): 53 | return not(any(not(c.is_valid(value_mapping)) for c in self._conditions)) 54 | 55 | 56 | class OrRule(object): 57 | 58 | def __init__(self, conditions): 59 | self._conditions = conditions 60 | 61 | def __str__(self): 62 | return ' or '.join(str(c) for c in self._conditions) 63 | 64 | def is_valid(self, value_mapping): 65 | return any(c.is_valid(value_mapping) for c in self._conditions) 66 | 67 | 68 | class NotRule(object): 69 | 70 | def __init__(self, condition): 71 | self._condition = condition 72 | 73 | def __str__(self): 74 | return 'not ({:s})'.format(str(self._condition)) 75 | 76 | def is_valid(self, value_mapping): 77 | return not self._condition.is_valid(value_mapping) 78 | 79 | 80 | class Condition(object): 81 | 82 | def __init__(self, variable_name, value): 83 | self._name = variable_name.strip() 84 | self._value = value.strip() 85 | 86 | def __str__(self): 87 | return '{:s} == {:s}'.format(self._name, self._value) 88 | 89 | def is_valid(self, value_mapping): 90 | return value_mapping[self._name].strip() == self._value.strip() 91 | 92 | 93 | def UnequalCondition(variable_name, value): 94 | return NotRule(Condition(variable_name, value)) 95 | 96 | 97 | class GreaterThanCondition(object): 98 | 99 | def __init__(self, variable_name, value): 100 | self._name = variable_name.strip() 101 | self._value = value.strip() 102 | 103 | def __str__(self): 104 | return '{:s} > {:s}'.format(self._name, self._value) 105 | 106 | def is_valid(self, value_mapping): 107 | return value_mapping[self._name].strip() > self._value.strip() 108 | 109 | 110 | class GreaterThanOrEqualCondition(object): 111 | 112 | def __init__(self, variable_name, value): 113 | self._name = variable_name.strip() 114 | self._value = value.strip() 115 | 116 | def __str__(self): 117 | return '{:s} >= {:s}'.format(self._name, self._value) 118 | 119 | def is_valid(self, value_mapping): 120 | return value_mapping[self._name].strip() >= self._value.strip() 121 | 122 | 123 | class LessThanCondition(object): 124 | 125 | def __init__(self, variable_name, value): 126 | self._name = variable_name.strip() 127 | self._value = value.strip() 128 | 129 | def __str__(self): 130 | return '{:s} < {:s}'.format(self._name, self._value) 131 | 132 | def is_valid(self, value_mapping): 133 | return value_mapping[self._name].strip() < self._value.strip() 134 | 135 | 136 | class LessThanOrEqualCondition(object): 137 | 138 | def __init__(self, variable_name, value): 139 | self._name = variable_name.strip() 140 | self._value = value.strip() 141 | 142 | def __str__(self): 143 | return '{:s} <= {:s}'.format(self._name, self._value) 144 | 145 | def is_valid(self, value_mapping): 146 | return value_mapping[self._name].strip() <= self._value.strip() 147 | 148 | 149 | class RegexCondition(object): 150 | 151 | def __init__(self, variable_name, value): 152 | self._name = variable_name.strip() 153 | self._value = value.strip() 154 | 155 | def __str__(self): 156 | return '{:s} ~ {:s}'.format(self._name, self._value) 157 | 158 | def is_valid(self, value_mapping): 159 | return re.search(self._value.strip(), value_mapping[self._name].strip()) is not None 160 | 161 | 162 | class RegexNegatedCondition(object): 163 | 164 | def __init__(self, variable_name, value): 165 | self._name = variable_name.strip() 166 | self._value = value.strip() 167 | 168 | def __str__(self): 169 | return '{:s} !~ {:s}'.format(self._name, self._value) 170 | 171 | def is_valid(self, value_mapping): 172 | return re.search(self._value.strip(), value_mapping[self._name].strip()) is None 173 | -------------------------------------------------------------------------------- /src/robomachine/runner.py: -------------------------------------------------------------------------------- 1 | # Copyright 2011-2019 Mikko Korpela 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | from __future__ import print_function 16 | 17 | import os 18 | import re 19 | import subprocess 20 | import sys 21 | from .parsing import RoboMachineParsingException, parse 22 | 23 | from . import __version__ 24 | import argparse 25 | 26 | from .generator import Generator 27 | from .strategies import DepthFirstSearchStrategy, RandomStrategy 28 | 29 | if sys.version_info.major == 3: 30 | unicode = str 31 | 32 | parser = argparse.ArgumentParser(description='RoboMachine {:s} - '.format(__version__) + 33 | 'a test data generator for Robot Framework', 34 | formatter_class=argparse.RawTextHelpFormatter) 35 | parser.add_argument('input', type=str, help='input file') 36 | parser.add_argument('--output', '-o', type=str, default=None, 37 | help='output file (default is input file with txt suffix)') 38 | parser.add_argument('--tests-max', '-t', 39 | type=int, default=1000, 40 | help='maximum number of tests to generate (default 1000)') 41 | parser.add_argument('--to-state', '-T', 42 | type=str, default=None, 43 | help='The state that all generated tests should end in.\n' + 44 | 'If none given, all states are considered valid test end states') 45 | parser.add_argument('--actions-max', '-a', 46 | type=int, default=100, 47 | help='maximum number of actions to generate (default 100)') 48 | parser.add_argument('--generation-algorithm', '-g', 49 | type=str, default='dfs', choices=['dfs', 'random', 'allpairs-random'], 50 | help='''\ 51 | Use test generation algorithm: 52 | allpairs-random = generate tests randomly, use allpairs algorithm for parameter value selection 53 | dfs = depth first search (default) 54 | random = generate tests randomly''') 55 | parser.add_argument('--do-not-execute', action='store_true', default=False, 56 | help='Do not execute generated tests with pybot command') 57 | parser.add_argument('--generate-dot-graph', '-D', 58 | type=str, default='none', choices=['none', 'png', 'svg'], 59 | help='''\ 60 | Generates a directional graph visualizing your test model. Select file format: 61 | none - Do not generate a file (default) 62 | png - bitmap 63 | svg - vector''') 64 | 65 | 66 | def main(): 67 | args = parser.parse_args() 68 | generator = Generator() 69 | strategy_class = _select_strategy(args.generation_algorithm) 70 | all_actions = set() 71 | 72 | if args.input.endswith('.txt') and not args.output: 73 | sys.exit('txt input not allowed when no output') 74 | try: 75 | with open(args.input, 'r') as inp: 76 | machine = parse(inp.read()) 77 | except IOError as e: 78 | sys.exit(unicode(e)) 79 | except RoboMachineParsingException as e: 80 | sys.exit(1) 81 | 82 | # File names: 83 | output_base_name = os.path.splitext(args.output or args.input)[0] 84 | output_test_file = output_base_name + '.robot' 85 | output_dot_file = output_base_name + '.dot' 86 | 87 | # Find unique actions: 88 | for state in machine.states: 89 | for action in state._actions: 90 | action._parent_state = state 91 | all_actions.add(action) 92 | 93 | # DOT Graph: 94 | if args.generate_dot_graph != 'none': 95 | # Generate graph in dot format: 96 | dot_graph = 'digraph TestModel {\n' 97 | # 98 | # Nodes: 99 | for state in machine.states: 100 | dot_graph += ' {:s} [label=\"{:s}\"];\n'.format(state.name.replace(' ', '_'), state.name) 101 | # 102 | # Transitions: 103 | for action in all_actions: 104 | state = action._parent_state 105 | action_name = action.name if action.name != '' else '[tau]' 106 | dot_state_name = state.name.replace(' ', '_') 107 | dot_next_state_name = action.next_state.name.replace(' ', '_') 108 | dot_action_name = re.sub(r'\s\s+', ' ', action_name) 109 | 110 | dot_graph += ' {:s} -> {:s} [label="{:s}"];\n'.format( 111 | dot_state_name, dot_next_state_name, dot_action_name) 112 | dot_graph += '}\n' 113 | # 114 | # Write to STDOUT: 115 | print('-' * 78) 116 | print('Dot graph') 117 | print('---------') 118 | print(dot_graph) 119 | print('-' * 78) 120 | # 121 | # Write to file: 122 | with open(output_dot_file, 'w') as out: 123 | out.write(dot_graph) 124 | try: 125 | retcode = subprocess.call(['dot', '-O', '-T' + args.generate_dot_graph, output_dot_file]) 126 | except OSError: 127 | retcode = -1 128 | if retcode == 0: 129 | print('Generated dot files: {:s}, {:s}.{:s}'.format( 130 | output_dot_file, output_dot_file, args.generate_dot_graph)) 131 | else: 132 | print('ERROR: Something went wrong during the dot file generation!\n' + 133 | ' Maybe you haven\'t yet installed the dot tool?') 134 | 135 | # Generate tests: 136 | with open(output_test_file, 'w') as out: 137 | generator.generate(machine, 138 | max_tests=args.tests_max, 139 | max_actions=args.actions_max, 140 | to_state=args.to_state, 141 | output=out, 142 | strategy=strategy_class) 143 | print('Generated test file: {:s}'.format(output_test_file)) 144 | 145 | # Coverage information: 146 | covered_states = generator.visited_states 147 | covered_actions = generator.visited_actions 148 | uncovered_states = set(machine.states).difference(generator.visited_states) 149 | uncovered_actions = all_actions.difference(generator.visited_actions) 150 | # 151 | # Write to STDOUT: 152 | print('-' * 78) 153 | # 154 | # Covered states: 155 | print('Covered states ({:d}/{:d}):'.format(len(covered_states), len(machine.states))) 156 | if covered_states: 157 | for state in covered_states: 158 | print(' {:s}'.format(state.name)) 159 | else: 160 | print(' -none-') 161 | # 162 | # Covered actions: 163 | print('\nCovered actions ({:d}/{:d}):'.format(len(covered_actions), len(all_actions))) 164 | if covered_actions: 165 | for action in covered_actions: 166 | action_name = action.name if action.name != '' else '[tau]' 167 | print(' {:s} ({:s} -> {:s})'.format(action_name, action._parent_state.name, action.next_state.name)) 168 | else: 169 | print(' -none-') 170 | # 171 | # Uncovered states: 172 | if uncovered_states: 173 | print('\nUncovered states ({:d}/{:d}):'.format(len(uncovered_states), len(machine.states))) 174 | for state in uncovered_states: 175 | print(' {:s}'.format(state.name)) 176 | # 177 | # Uncovered actions: 178 | if uncovered_actions: 179 | print('\nUncovered actions ({:d}/{:d}):'.format(len(uncovered_actions), len(all_actions))) 180 | for action in uncovered_actions: 181 | action_name = action.name if action.name != '' else '[tau]' 182 | print(' {:s} ({:s} -> {:s})'.format(action_name, action._parent_state.name, action.next_state.name)) 183 | print('-' * 78) 184 | 185 | # Run tests: 186 | if not args.do_not_execute: 187 | print('\nRunning generated tests with robot:') 188 | retcode = subprocess.call(['robot', output_test_file]) 189 | sys.exit(retcode) 190 | 191 | 192 | def _select_strategy(strategy): 193 | if strategy == 'random': 194 | return RandomStrategy 195 | if strategy == 'dfs': 196 | return DepthFirstSearchStrategy 197 | if strategy == 'allpairs-random': 198 | try: 199 | from src.robomachine.allpairsstrategy import AllPairsRandomStrategy 200 | return AllPairsRandomStrategy 201 | except ImportError: 202 | print('ERROR! allpairs-random strategy needs the AllPairs module') 203 | print('please install it from Python Package Index') 204 | print('pip install allpairspy') 205 | raise 206 | 207 | 208 | if __name__ == '__main__': 209 | main() 210 | -------------------------------------------------------------------------------- /src/robomachine/strategies.py: -------------------------------------------------------------------------------- 1 | # Copyright 2011-2012 Mikko Korpela 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | import random 15 | 16 | class _Strategy(object): 17 | 18 | def __init__(self, machine, max_actions, to_state=None): 19 | self._machine = machine 20 | self._max_actions = max_actions 21 | self._to_state = to_state 22 | assert not to_state or self._machine.find_state_by_name(to_state) 23 | 24 | def _matching_to_state(self, test): 25 | return not self._to_state or self._to_state == test[-1].next_state.name 26 | 27 | 28 | class DepthFirstSearchStrategy(_Strategy): 29 | 30 | def tests(self): 31 | for values in self._variable_value_sets(self._machine.variables): 32 | self._machine.apply_variable_values(values) 33 | for test in self._generate_all_from(self._machine.start_state, self._max_actions): 34 | yield test, [v.current_value for v in self._machine.variables] 35 | 36 | def _variable_value_sets(self, variables): 37 | if not variables: 38 | return ([],) 39 | return (vs for vs in self._var_set(variables) if self._machine.rules_are_ok(vs)) 40 | 41 | def _var_set(self, vars): 42 | if not vars: 43 | return [[]] 44 | return ([val]+sub_set for val in vars[0].values for sub_set in self._var_set(vars[1:])) 45 | 46 | def _generate_all_from(self, state, max_actions): 47 | if not state.actions or max_actions == 0: 48 | if self._to_state and self._to_state != state.name: 49 | return 50 | yield [] 51 | else: 52 | at_least_one_generated = False 53 | for action in state.actions: 54 | for test in self._generate_all_from(action.next_state, max_actions-1): 55 | at_least_one_generated = True 56 | yield [action]+test 57 | if not at_least_one_generated and self._to_state == state.name: 58 | yield [] 59 | 60 | 61 | class RandomStrategy(_Strategy): 62 | 63 | def tests(self): 64 | while True: 65 | test = self._generate_test(self._generate_variable_values()) 66 | if not test and self._to_state and self._to_state != self._machine.start_state.name: 67 | continue 68 | yield test, [v.current_value for v in self._machine.variables] 69 | 70 | def _generate_test(self, values): 71 | test = [] 72 | self._machine.apply_variable_values(values) 73 | current_state = self._machine.start_state 74 | while self._max_actions > len(test) and current_state.actions: 75 | action = random.choice(current_state.actions) 76 | current_state = action.next_state 77 | test.append(action) 78 | while test and not self._matching_to_state(test): 79 | test.pop() 80 | return test 81 | 82 | def _generate_variable_values(self): 83 | while True: 84 | candidate = [random.choice(v.values) for v in self._machine.variables] 85 | if self._machine.rules_are_ok(candidate): 86 | return candidate 87 | 88 | -------------------------------------------------------------------------------- /test/__init__.py: -------------------------------------------------------------------------------- 1 | # Copyright 2011-2012 Mikko Korpela 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | -------------------------------------------------------------------------------- /test/generation_test.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) 2011-2012 Mikko Korpela 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | # ------------------------------------------------------------------------ 15 | # Copyright 2017 David Kaplan 16 | # 17 | # Changes: 18 | # - Python 3 support 19 | 20 | try: 21 | from StringIO import StringIO 22 | except: 23 | from io import StringIO 24 | 25 | import unittest 26 | from src.robomachine.model import RoboMachine, State, Action 27 | from src import robomachine 28 | 29 | 30 | class GenerationTestCase(unittest.TestCase): 31 | 32 | def test_generation_with_no_tests(self): 33 | mock_machine = RoboMachine(states=[], variables=[], rules=[]) 34 | output = StringIO() 35 | robomachine.generate(machine=mock_machine, output=output, strategy=_MockStrategyWithNoTests) 36 | self.assertTrue(output.getvalue().startswith('*** Test Cases ***\n*** Keywords ***')) 37 | 38 | def test_generation_with_a_test_limit(self): 39 | mock_machine = RoboMachine(states=[State('bar', [], [])], variables=[], rules=[]) 40 | output = StringIO() 41 | robomachine.generate(machine=mock_machine, max_tests=10, output=output, strategy=_MockStrategyWithHundredTests) 42 | self.assertEqual(10, sum(1 for line in output.getvalue().splitlines() if line.startswith('Test '))) 43 | 44 | def test_generation_with_same_generated_tests(self): 45 | mock_machine = RoboMachine(states=[State('bar', [], [])], variables=[], rules=[]) 46 | output = StringIO() 47 | robomachine.generate(machine=mock_machine, max_tests=1000, output=output, strategy=_MockStrategyWithSameTests) 48 | self.assertEqual(1, sum(1 for line in output.getvalue().splitlines() if line.startswith('Test '))) 49 | 50 | 51 | class _MockStrategy(object): 52 | 53 | def __init__(self, machine, *others): 54 | self._machine = machine 55 | 56 | def _action(self): 57 | a = Action('foo', 'bar') 58 | a.set_machine(self._machine) 59 | return a 60 | 61 | def _visited_actions(self): 62 | pass 63 | 64 | def _visited_states(self): 65 | pass 66 | 67 | 68 | class _MockStrategyWithNoTests(_MockStrategy): 69 | 70 | def tests(self): 71 | return [] 72 | 73 | 74 | class _MockStrategyWithHundredTests(_MockStrategy): 75 | 76 | def tests(self): 77 | return [([self._action()], []) for i in range(100)] 78 | 79 | 80 | class _MockStrategyWithSameTests(_MockStrategy): 81 | 82 | def tests(self): 83 | test = ([self._action()], []) 84 | while True: 85 | yield test 86 | 87 | 88 | if __name__ == '__main__': 89 | unittest.main() 90 | -------------------------------------------------------------------------------- /test/model_test.py: -------------------------------------------------------------------------------- 1 | # Copyright 2011-2012 Mikko Korpela 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | import unittest 16 | from src.robomachine.model import RoboMachine, Variable 17 | 18 | 19 | class MachinaModelTestCase(unittest.TestCase): 20 | 21 | def test_empty_machina_model(self): 22 | empty_machina = RoboMachine(None, None, None) 23 | self.assertEqual([], empty_machina.states) 24 | self.assertEqual([], empty_machina.variables) 25 | self.assertEqual(None, empty_machina.find_state_by_name('some name')) 26 | 27 | def test_variable_model(self): 28 | var_model = Variable('name', ['1', '2']) 29 | self.assertEqual('name', var_model.name) 30 | self.assertEqual(['1', '2'], var_model.values) 31 | 32 | def test_variable_current_value(self): 33 | var_model = Variable('foo', ['bar', 'zoo']) 34 | try: 35 | var_model.current_value 36 | self.fail('should throw assertion error as no current value has been set') 37 | except AssertionError: 38 | pass 39 | var_model.set_current_value('bar') 40 | self.assertEqual('bar', var_model.current_value) 41 | 42 | if __name__ == '__main__': 43 | unittest.main() 44 | -------------------------------------------------------------------------------- /test/robomachina_test.py: -------------------------------------------------------------------------------- 1 | # Copyright 2011-2012 Mikko Korpela 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | try: 16 | from StringIO import StringIO 17 | except: 18 | from io import StringIO 19 | 20 | import unittest 21 | from src.robomachine import parsing 22 | from src import robomachine 23 | from src.robomachine.parsing import comment, RoboMachineParsingException 24 | from src.robomachine.strategies import RandomStrategy 25 | 26 | 27 | _MACHINA = """\ 28 | *** Settings *** 29 | Default Tags foo 30 | 31 | *** Variables *** 32 | ${BAR} zoo 33 | 34 | *** Machine *** 35 | Start State 36 | Log In Start State 37 | [Actions] 38 | Some keyword ==> End State 39 | 40 | End State 41 | Log In End State 42 | 43 | *** Keywords *** 44 | Some keyword 45 | No Operation 46 | """ 47 | 48 | _TESTS = """\ 49 | *** Settings *** 50 | Default Tags foo 51 | 52 | *** Variables *** 53 | ${BAR} zoo 54 | 55 | *** Test Cases *** 56 | Test 1 57 | Start State 58 | Some keyword 59 | End State 60 | 61 | *** Keywords *** 62 | 63 | Some keyword 64 | No Operation 65 | Start State 66 | Log In Start State 67 | End State 68 | Log In End State 69 | """ 70 | 71 | class TestRoboMachina(unittest.TestCase): 72 | 73 | def test_transform_to_robot_test_cases(self): 74 | tests = robomachine.transform(_MACHINA) 75 | self.assertEqual(_TESTS, tests) 76 | 77 | def test_parse_machina_state_names(self): 78 | m = robomachine.parse(_MACHINA) 79 | self.assertEqual(['Start State', 'End State'], [s.name for s in m.states]) 80 | 81 | def test_parse_machina_state_actions(self): 82 | m = robomachine.parse(_MACHINA) 83 | self.assertEqual(['Some keyword'], [a.name for a in m.states[0].actions]) 84 | self.assertEqual([], [a.name for a in m.states[1].actions]) 85 | 86 | def test_parse_machina_state_steps(self): 87 | m = robomachine.parse(_MACHINA) 88 | self.assertEqual([' Log In Start State'], m.states[0].steps) 89 | self.assertEqual([' Log In End State'], m.states[1].steps) 90 | 91 | 92 | _MACHINA2 = """\ 93 | *** Machine *** 94 | A 95 | Foo bar 96 | Bar foo 97 | [Actions] 98 | first ==> B 99 | second ==> C 100 | ==> D 101 | 102 | B 103 | [Actions] 104 | something else ==> A # This is also a comment 105 | other thing ==> C 106 | 107 | C 108 | No Operation #This is a comment 109 | 110 | D 111 | Log tau action can only get here 112 | """ 113 | 114 | _TESTS2_GENERATE_ALL_DFS_MAX_ACTIONS_2 = """\ 115 | *** Test Cases *** 116 | Test 1 117 | A 118 | first 119 | something else 120 | A 121 | 122 | Test 2 123 | A 124 | first 125 | other thing 126 | C 127 | 128 | Test 3 129 | A 130 | second 131 | C 132 | 133 | Test 4 134 | A 135 | D 136 | 137 | *** Keywords *** 138 | A 139 | Foo bar 140 | Bar foo 141 | C 142 | No Operation #This is a comment 143 | D 144 | Log tau action can only get here 145 | """ 146 | 147 | _INVALID_STATE_MACHINE = """\ 148 | *** Machine *** 149 | State 1 150 | [Actions] 151 | ==> Invalid 152 | """ 153 | 154 | class TestParsing(unittest.TestCase): 155 | 156 | def test_header_parsing(self): 157 | header = parsing.machine_header.parseString('*** Machine ***\n') 158 | self.assertEqual('*** Machine ***', header[0]) 159 | 160 | def test_state_parsing(self): 161 | state = """\ 162 | A 163 | Foo bar 164 | Bar foo 165 | [Actions] 166 | first ==> B 167 | second ==> C 168 | """ 169 | state = parsing.state.parseString(state)[0] 170 | self.assertEqual('A', state.name) 171 | self.assertEqual([' Foo bar', ' Bar foo'], state.steps) 172 | self.assertEqual(['first', 'second'], [a.name for a in state.actions]) 173 | self.assertEqual(['B', 'C'], [a._next_state_name for a in state.actions]) 174 | 175 | def test_steps_parsing(self): 176 | steps = """\ 177 | Foo bar 178 | Bar foo 179 | """ 180 | steps = parsing.steps.parseString(steps) 181 | self.assertEqual([' Foo bar', ' Bar foo'], list(steps.steps)) 182 | 183 | def test_step_parsing(self): 184 | self._should_parse_step(' Foo bar\n') 185 | self._should_parse_step(' Log ${value}\n') 186 | self._should_parse_step(' ${value}= Set Variable something\n') 187 | self._should_parse_step(' Log [something]\n') 188 | 189 | def test_action_parsing(self): 190 | action = parsing.action.parseString(' action name ==> state')[0] 191 | self.assertEqual('action name', action.name) 192 | self.assertEqual('state', action._next_state_name) 193 | 194 | def test_tau_action_parsing(self): 195 | action = parsing.action.parseString(' ==> state')[0] 196 | self.assertEqual('', action.name) 197 | self.assertEqual('state', action._next_state_name) 198 | 199 | def _should_parse_step(self, step): 200 | self.assertEqual(step[:-1], parsing.step.parseString(step)[0]) 201 | 202 | def test_parsing(self): 203 | m = robomachine.parse(_MACHINA2) 204 | self.assertEqual(m.states[0].name, 'A') 205 | self.assertEqual(m.states[0].steps, [' Foo bar', ' Bar foo']) 206 | self.assertEqual([a.name for a in m.states[0].actions], ['first', 'second', '']) 207 | self.assertEqual([a.next_state.name for a in m.states[0].actions], ['B', 'C', 'D']) 208 | self.assertEqual(m.states[1].name, 'B') 209 | self.assertEqual(m.states[1].steps, []) 210 | self.assertEqual([a.name for a in m.states[1].actions], ['something else', 'other thing']) 211 | self.assertEqual([a.next_state.name for a in m.states[1].actions], ['A', 'C']) 212 | self.assertEqual(m.states[2].name, 'C') 213 | self.assertEqual(m.states[2].steps, [' No Operation #This is a comment']) 214 | self.assertEqual(m.states[2].actions, []) 215 | 216 | def test_invalid_state_parsing(self): 217 | try: 218 | m = robomachine.parse(_INVALID_STATE_MACHINE) 219 | self.fail('Should raise exception') 220 | except RoboMachineParsingException: 221 | pass 222 | 223 | 224 | _VAR_PROBLEM_MACHINE = """\ 225 | *** Machine *** 226 | ${FOO} any of bar 227 | ${BAR} any of ${FOO} 228 | 229 | State 230 | No Operation 231 | [Actions] 232 | ==> End when ${BAR} == bar 233 | 234 | End 235 | No Operation 236 | """ 237 | 238 | _TEST_VAR_PROBLEM_MACHINE = """\ 239 | *** Test Cases *** 240 | Test 1 241 | Set Machine Variables bar bar 242 | State 243 | End 244 | 245 | *** Keywords *** 246 | Set Machine Variables 247 | [Arguments] ${FOO} ${BAR} 248 | Set Test Variable \${FOO} 249 | Set Test Variable \${BAR} 250 | State 251 | No Operation 252 | End 253 | No Operation 254 | """ 255 | 256 | class TestTestGeneration(unittest.TestCase): 257 | 258 | def test_generate_all_dfs_max_actions_2(self): 259 | m = robomachine.parse(_MACHINA2) 260 | out = StringIO() 261 | robomachine.generate(m, max_actions=2, output=out) 262 | self.assertEqual(_TESTS2_GENERATE_ALL_DFS_MAX_ACTIONS_2, out.getvalue()) 263 | 264 | def test_generate_all_with_variable_resolving_problem(self): 265 | m = robomachine.parse(_VAR_PROBLEM_MACHINE) 266 | out = StringIO() 267 | robomachine.generate(m, max_actions=1, output=out) 268 | self.assertEqual(_TEST_VAR_PROBLEM_MACHINE, out.getvalue()) 269 | 270 | def test_generate_random_with_variable_resolving_problem(self): 271 | m = robomachine.parse(_VAR_PROBLEM_MACHINE) 272 | out = StringIO() 273 | robomachine.generate(m, max_actions=1, output=out, strategy=RandomStrategy) 274 | self.assertEqual(_TEST_VAR_PROBLEM_MACHINE, out.getvalue()) 275 | 276 | 277 | class TestComment(unittest.TestCase): 278 | 279 | def test_whole_line_comment(self): 280 | self._verify_comment('# comment\n') 281 | 282 | def test_comment_line_with_whitespace(self): 283 | self._verify_comment(' # comment\n') 284 | 285 | def test_content_and_comment(self): 286 | self._verify_comment(' Some content # commenting it\n', 287 | ' # commenting it') 288 | 289 | def _verify_comment(self, line, expected=None): 290 | expected = expected or line 291 | self.assertEqual(expected, comment.searchString(line)[0][0]) 292 | 293 | if __name__ == '__main__': 294 | unittest.main() 295 | -------------------------------------------------------------------------------- /test/rules_test.py: -------------------------------------------------------------------------------- 1 | # Copyright 2011-2012 Mikko Korpela 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | import unittest 16 | from src.robomachine.rules import (Condition, AndRule, EquivalenceRule, OrRule, 17 | NotRule, ImplicationRule, UnequalCondition, 18 | GreaterThanCondition, GreaterThanOrEqualCondition, 19 | LessThanCondition, LessThanOrEqualCondition, 20 | RegexCondition, RegexNegatedCondition) 21 | 22 | 23 | class RulesTestCases(unittest.TestCase): 24 | 25 | _TRUE = lambda:0 26 | _TRUE.is_valid = lambda value_mapping: True 27 | _FALSE = lambda:0 28 | _FALSE.is_valid = lambda value_mapping: False 29 | 30 | def test_condition(self): 31 | condition = Condition('${VARIABLE}', 'value') 32 | self.assertTrue(condition.is_valid(value_mapping={'${VARIABLE}':'value'})) 33 | self.assertFalse(condition.is_valid(value_mapping={'${VARIABLE}':'wrong value'})) 34 | 35 | def test_unequal_condition(self): 36 | condition = UnequalCondition('${VARIABLE}', 'value') 37 | self.assertFalse(condition.is_valid(value_mapping={'${VARIABLE}':'value'})) 38 | self.assertTrue(condition.is_valid(value_mapping={'${VARIABLE}':'wrong value'})) 39 | 40 | def test_greater_than_condition(self): 41 | condition = GreaterThanCondition('${VARIABLE}', '1') 42 | self.assertFalse(condition.is_valid(value_mapping={'${VARIABLE}':'0'})) 43 | self.assertFalse(condition.is_valid(value_mapping={'${VARIABLE}':'1'})) 44 | self.assertTrue(condition.is_valid(value_mapping={'${VARIABLE}':'2'})) 45 | 46 | def test_greater_than_or_equal_condition(self): 47 | condition = GreaterThanOrEqualCondition('${VARIABLE}', '1') 48 | self.assertFalse(condition.is_valid(value_mapping={'${VARIABLE}':'0'})) 49 | self.assertTrue(condition.is_valid(value_mapping={'${VARIABLE}':'1'})) 50 | self.assertTrue(condition.is_valid(value_mapping={'${VARIABLE}':'2'})) 51 | 52 | def test_less_than_condition(self): 53 | condition = LessThanCondition('${VARIABLE}', '1') 54 | self.assertTrue(condition.is_valid(value_mapping={'${VARIABLE}':'0'})) 55 | self.assertFalse(condition.is_valid(value_mapping={'${VARIABLE}':'1'})) 56 | self.assertFalse(condition.is_valid(value_mapping={'${VARIABLE}':'2'})) 57 | 58 | def test_less_than_or_equal_condition(self): 59 | condition = LessThanOrEqualCondition('${VARIABLE}', '1') 60 | self.assertTrue(condition.is_valid(value_mapping={'${VARIABLE}':'0'})) 61 | self.assertTrue(condition.is_valid(value_mapping={'${VARIABLE}':'1'})) 62 | self.assertFalse(condition.is_valid(value_mapping={'${VARIABLE}':'2'})) 63 | 64 | def test_regex_condition(self): 65 | condition = RegexCondition('${VARIABLE}', '.*a') 66 | self.assertTrue(condition.is_valid(value_mapping={'${VARIABLE}':'abc'})) 67 | self.assertTrue(condition.is_valid(value_mapping={'${VARIABLE}':'bar'})) 68 | self.assertFalse(condition.is_valid(value_mapping={'${VARIABLE}':'foo'})) 69 | 70 | def test_regex_negated_condition(self): 71 | condition = RegexNegatedCondition('${VARIABLE}', '.*a$') 72 | self.assertTrue(condition.is_valid(value_mapping={'${VARIABLE}':'foob'})) 73 | self.assertTrue(condition.is_valid(value_mapping={'${VARIABLE}':'barb'})) 74 | self.assertFalse(condition.is_valid(value_mapping={'${VARIABLE}':'fooa'})) 75 | 76 | def test_and_rule(self): 77 | self.assertTrue(AndRule([self._TRUE for _ in range(10)]).is_valid({})) 78 | self.assertFalse(AndRule([self._FALSE]+[self._TRUE for _ in range(10)]).is_valid({})) 79 | 80 | def test_equivalence_rule(self): 81 | self.assertTrue(EquivalenceRule(self._TRUE, self._TRUE).is_valid({})) 82 | self.assertFalse(EquivalenceRule(self._FALSE, self._TRUE).is_valid({})) 83 | self.assertFalse(EquivalenceRule(self._TRUE, self._FALSE).is_valid({})) 84 | self.assertTrue(EquivalenceRule(self._FALSE, self._FALSE).is_valid({})) 85 | 86 | def test_or_rule(self): 87 | self.assertFalse(OrRule([self._FALSE for _ in range(10)]).is_valid({})) 88 | self.assertTrue(OrRule([self._TRUE]+[self._FALSE for _ in range(10)]).is_valid({})) 89 | 90 | def test_not_rule(self): 91 | self.assertFalse(NotRule(self._TRUE).is_valid({})) 92 | self.assertTrue(NotRule(self._FALSE).is_valid({})) 93 | 94 | def test_implication_rule(self): 95 | self.assertTrue(ImplicationRule(self._TRUE, self._TRUE).is_valid({})) 96 | self.assertTrue(ImplicationRule(self._FALSE, self._TRUE).is_valid({})) 97 | self.assertFalse(ImplicationRule(self._TRUE, self._FALSE).is_valid({})) 98 | self.assertTrue(ImplicationRule(self._FALSE, self._FALSE).is_valid({})) 99 | 100 | if __name__ == '__main__': 101 | unittest.main() 102 | -------------------------------------------------------------------------------- /test/strategies_test.py: -------------------------------------------------------------------------------- 1 | # Copyright 2011-2012 Mikko Korpela 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | import unittest 16 | from src.robomachine.model import RoboMachine, State, Action, Variable 17 | from src.robomachine.rules import * 18 | from src.robomachine.strategies import DepthFirstSearchStrategy, RandomStrategy 19 | from src.robomachine.allpairsstrategy import AllPairsRandomStrategy 20 | 21 | class StrategyTestCase(object): 22 | 23 | def test_can_generate_test_from_simple_machine(self): 24 | action12 = Action('to state2', 'state2') 25 | action21 = Action('to state1', 'state1') 26 | states = [State('state1', [], [action12]), 27 | State('state2', [], [action21])] 28 | at_least_one_test_generated = False 29 | for test in self.strategy_class(RoboMachine(states, [], []), 2).tests(): 30 | self.assertEqual(([action12, action21], []), test) 31 | at_least_one_test_generated = True 32 | break 33 | self.assertTrue(at_least_one_test_generated) 34 | 35 | def test_can_generate_test_from_variable_machine(self): 36 | variables = [Variable('var', ['a', 'b', 'c'])] 37 | at_least_one_test_generated = False 38 | for test in self.strategy_class(RoboMachine([State('state', [], [])], variables, []), 0).tests(): 39 | self.assertEqual([], test[0]) 40 | self.assertEqual(1, len(test[1])) 41 | self.assertTrue(test[1][0] in ['a', 'b', 'c']) 42 | at_least_one_test_generated = True 43 | break 44 | self.assertTrue(at_least_one_test_generated) 45 | 46 | def test_obeys_rules(self): 47 | variables = [Variable('${VAR}', ['a', 'b', 'c', 'd', 'e', 'f', 'g'])] 48 | rules = [Condition('${VAR}', 'e')] 49 | at_least_one_test_generated = False 50 | for test in self.strategy_class(RoboMachine([State('state', [], [])], variables, rules), 0).tests(): 51 | self.assertEqual([], test[0]) 52 | self.assertEqual(1, len(test[1])) 53 | self.assertEqual('e', test[1][0]) 54 | at_least_one_test_generated = True 55 | break 56 | self.assertTrue(at_least_one_test_generated) 57 | 58 | def test_obeys_to_state(self): 59 | a12 = Action('s1->s2', 's2') 60 | a21 = Action('s2->s1', 's1') 61 | a23 = Action('s2->s3', 's3') 62 | a31 = Action('s3->s1', 's1') 63 | states = [State('s1', [], [a12]), 64 | State('s2', [], [a21, a23]), 65 | State('s3', [], [a31])] 66 | tests_to_generate = 5 67 | for test in self.strategy_class(RoboMachine(states, [], []), 17, 's3').tests(): 68 | self.assertEqual(states[-1].name, test[0][-1].next_state.name) 69 | tests_to_generate -= 1 70 | if tests_to_generate == 0: 71 | break 72 | self.assertEqual(0, tests_to_generate) 73 | 74 | def test_obeys_to_state_circular(self): 75 | a12 = Action('a12', 's2') 76 | a23 = Action('a23', 's3') 77 | a31 = Action('a31', 's1') 78 | states = [State('s1', [], [a12]), 79 | State('s2', [], [a23]), 80 | State('s3', [], [a31])] 81 | at_least_one_test_generated = False 82 | for test in self.strategy_class(RoboMachine(states, [], []), 4, 's3').tests(): 83 | self.assertEqual(states[-1].name, test[0][-1].next_state.name) 84 | at_least_one_test_generated = True 85 | break 86 | self.assertTrue(at_least_one_test_generated) 87 | 88 | def test_to_state_is_start_state(self): 89 | at_least_one_test_generated = False 90 | for test in self.strategy_class(RoboMachine([State('s1', [], [])], [], []), 4, 's1').tests(): 91 | self.assertEqual(([], []), test) 92 | at_least_one_test_generated = True 93 | break 94 | self.assertTrue(at_least_one_test_generated) 95 | 96 | 97 | class DepthFirstSearchStrategyTestCase(StrategyTestCase, unittest.TestCase): 98 | strategy_class = DepthFirstSearchStrategy 99 | 100 | class RandomStrategyTestCase(StrategyTestCase, unittest.TestCase): 101 | strategy_class = RandomStrategy 102 | 103 | class AllPairsRandomStrategyTestCase(StrategyTestCase, unittest.TestCase): 104 | strategy_class = AllPairsRandomStrategy 105 | 106 | def test_generates_all_pairs(self): 107 | variables = [Variable('${A}', list('123')), 108 | Variable('${B}', list('456')), 109 | Variable('${C}', list('789'))] 110 | tests = list(AllPairsRandomStrategy(RoboMachine([State('s', [], [])], variables, []), 1, 's').tests()) 111 | self.assertEqual(len(tests), 9) 112 | 113 | def test_obeys_to_state(self): 114 | # NOT APPLICABLE 115 | pass 116 | 117 | def test_obeys_rules(self): 118 | # NOT APPLICABLE 119 | pass 120 | 121 | def test_all_pairs_raises_error_if_rules(self): 122 | variables = [Variable('${A}', list('123')), 123 | Variable('${B}', list('456'))] 124 | rules = [OrRule([UnequalCondition('${A}', '1'), UnequalCondition('${B}', '4')])] 125 | self.assertRaises(AssertionError, lambda: AllPairsRandomStrategy(RoboMachine([State('s', [], [])], variables, rules), 1, 's').tests()) 126 | 127 | if __name__ == '__main__': 128 | unittest.main() 129 | -------------------------------------------------------------------------------- /test/variable_test.py: -------------------------------------------------------------------------------- 1 | # Copyright 2011-2012 Mikko Korpela 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | try: 16 | from StringIO import StringIO 17 | except: 18 | from io import StringIO 19 | 20 | import unittest 21 | from src.robomachine import parsing 22 | from src import robomachine 23 | import pyparsing 24 | 25 | class VariableParsingTestCases(unittest.TestCase): 26 | 27 | def test_variable_parsing(self): 28 | self._parse_var('${VARIABLE}') 29 | self._parse_var('${VAR2}') 30 | self._parse_var('${_VAR_WITH_UNDERSCORE}') 31 | self._invalid_var('') 32 | self._invalid_var('${foo}') 33 | self._invalid_var('${1}') 34 | self._invalid_var('${2FOO}') 35 | self._invalid_var('${FOO BAR}') 36 | 37 | def _parse_var(self, var_name): 38 | self.assertEqual(var_name, parsing.variable.parseString(var_name)[0]) 39 | 40 | def _invalid_var(self, var_name): 41 | try: 42 | parsing.variable.parseString(var_name) 43 | self.fail('Should not parse invalid variable name "{:s}"'.format(var_name)) 44 | except pyparsing.ParseException as e: 45 | pass 46 | 47 | def test_variable_definition_parsing(self): 48 | v = parsing.variable_definition.parseString('${ABC123} any of one two 123\n')[0] 49 | self.assertEqual('${ABC123}', v.name) 50 | self.assertEqual(['one', 'two', '123'], v.values) 51 | 52 | def test_variable_definition_parsing_when_more_than_two_spaces(self): 53 | v = parsing.variable_definition.parseString('${ABC123} any of one two 123\n')[0] 54 | self.assertEqual('${ABC123}', v.name) 55 | self.assertEqual(['one', 'two', '123'], v.values) 56 | 57 | 58 | class ConditionActionParsingTestCases(unittest.TestCase): 59 | 60 | def test_conditional_action_parsing(self): 61 | a = parsing.action.parseString(' My Action ==> End Step when ${FOO} == bar\n')[0] 62 | self.assertEqual('My Action', a.name) 63 | self.assertEqual('End Step', a._next_state_name) 64 | self.assertEqual('${FOO} == bar', str(a.condition)) 65 | 66 | def test_multiconditional_action_parsing(self): 67 | a = parsing.action.parseString(' My Action ==> End Step when ${BAR} == a and ${FOO} == cee\n')[0] 68 | self.assertEqual('My Action', a.name) 69 | self.assertEqual('End Step', a._next_state_name) 70 | self.assertEqual('${BAR} == a and ${FOO} == cee', str(a.condition)) 71 | 72 | def test_otherwise_conditional_action_parsing(self): 73 | a = parsing.action.parseString(' My Action ==> End Step otherwise\n')[0] 74 | self.assertEqual('My Action', a.name) 75 | self.assertEqual('End Step', a._next_state_name) 76 | self.assertEqual('otherwise', a.condition) 77 | 78 | 79 | class RuleParsingTestCases(unittest.TestCase): 80 | 81 | def test_equivalence_rule_parsing(self): 82 | rule = parsing.rule.parseString('${USERNAME} == ${VALID_PASSWORD} <==> ${PASSWORD} == ${VALID_USERNAME}\n')[0] 83 | self.assertEqual('${USERNAME} == ${VALID_PASSWORD} <==> ${PASSWORD} == ${VALID_USERNAME}', str(rule)) 84 | value_mapping = {'${USERNAME}':'${VALID_PASSWORD}', 85 | '${PASSWORD}':'${VALID_USERNAME}'} 86 | self.assertTrue(rule.is_valid(value_mapping=value_mapping)) 87 | value_mapping['${USERNAME}']='something else' 88 | self.assertFalse(rule.is_valid(value_mapping=value_mapping)) 89 | value_mapping['${USERNAME}']='${VALID_PASSWORD}' 90 | value_mapping['${PASSWORD}']='something' 91 | self.assertFalse(rule.is_valid(value_mapping=value_mapping)) 92 | value_mapping['${USERNAME}']='not valid' 93 | value_mapping['${PASSWORD}']='nothis validus' 94 | self.assertTrue(rule.is_valid(value_mapping=value_mapping)) 95 | 96 | def test_implication_rule_parsing(self): 97 | rule = parsing.rule.parseString('${VARIABLE} != value ==> ${OTHER} == other')[0] 98 | self.assertEqual('not (${VARIABLE} == value) ==> ${OTHER} == other', str(rule)) 99 | 100 | def test_condition_parsing(self): 101 | rule = parsing.rule.parseString('${VARIABLE} == value')[0] 102 | self.assertEqual('${VARIABLE} == value', str(rule)) 103 | 104 | def test_and_rule_parsing(self): 105 | rule = parsing.rule.parseString('${VARIABLE} == value and ${VAR2} == baluu')[0] 106 | self.assertEqual('${VARIABLE} == value and ${VAR2} == baluu', str(rule)) 107 | 108 | def test_or_rule_parsing(self): 109 | rule = parsing.rule.parseString('${VARIABLE} == value or ${VAR2} == baluu')[0] 110 | self.assertEqual('${VARIABLE} == value or ${VAR2} == baluu', str(rule)) 111 | 112 | def test_not_rule_parsing(self): 113 | rule = parsing.rule.parseString('not (${VAR2} == baluu)')[0] 114 | self.assertEqual('not (${VAR2} == baluu)', str(rule)) 115 | 116 | def test_complex(self): 117 | rule = parsing.rule.parseString('${FOO} == bar or (not (${BAR} == foo and ${ZOO} == fii))')[0] 118 | self.assertEqual('${FOO} == bar or not (${BAR} == foo and ${ZOO} == fii)', str(rule)) 119 | 120 | 121 | _LOGIN_MACHINE = """\ 122 | *** Machine *** 123 | ${USERNAME} any of demo mode invalid ${EMPTY} 124 | ${PASSWORD} any of mode demo invalid ${EMPTY} 125 | 126 | ${USERNAME} == mode <==> ${PASSWORD} == demo 127 | 128 | Login Page 129 | Title Should Be Login Page 130 | [Actions] 131 | Submit Credentials ==> Welcome Page when ${USERNAME} == demo and ${PASSWORD} == mode 132 | Submit Credentials ==> Error Page otherwise 133 | 134 | Welcome Page 135 | Title Should Be Welcome Page 136 | 137 | Error Page 138 | Title Should Be Error Page 139 | """ 140 | 141 | _LOGIN_TESTS_GENERATE_ALL_DFS = """\ 142 | *** Test Cases *** 143 | Test 1 144 | Set Machine Variables demo mode 145 | Login Page 146 | Submit Credentials 147 | Welcome Page 148 | 149 | Test 2 150 | Set Machine Variables demo invalid 151 | Login Page 152 | Submit Credentials 153 | Error Page 154 | 155 | Test 3 156 | Set Machine Variables demo ${EMPTY} 157 | Login Page 158 | Submit Credentials 159 | Error Page 160 | 161 | Test 4 162 | Set Machine Variables mode demo 163 | Login Page 164 | Submit Credentials 165 | Error Page 166 | 167 | Test 5 168 | Set Machine Variables invalid mode 169 | Login Page 170 | Submit Credentials 171 | Error Page 172 | 173 | Test 6 174 | Set Machine Variables invalid invalid 175 | Login Page 176 | Submit Credentials 177 | Error Page 178 | 179 | Test 7 180 | Set Machine Variables invalid ${EMPTY} 181 | Login Page 182 | Submit Credentials 183 | Error Page 184 | 185 | Test 8 186 | Set Machine Variables ${EMPTY} mode 187 | Login Page 188 | Submit Credentials 189 | Error Page 190 | 191 | Test 9 192 | Set Machine Variables ${EMPTY} invalid 193 | Login Page 194 | Submit Credentials 195 | Error Page 196 | 197 | Test 10 198 | Set Machine Variables ${EMPTY} ${EMPTY} 199 | Login Page 200 | Submit Credentials 201 | Error Page 202 | 203 | *** Keywords *** 204 | Set Machine Variables 205 | [Arguments] ${USERNAME} ${PASSWORD} 206 | Set Test Variable \${USERNAME} 207 | Set Test Variable \${PASSWORD} 208 | Login Page 209 | Title Should Be Login Page 210 | Welcome Page 211 | Title Should Be Welcome Page 212 | Error Page 213 | Title Should Be Error Page 214 | """ 215 | 216 | 217 | class VariableMachineParsingTestCases(unittest.TestCase): 218 | 219 | def test_machine_parsing(self): 220 | m = parsing.parse(_LOGIN_MACHINE) 221 | self.assertEqual('${USERNAME}', m.variables[0].name) 222 | self.assertEqual('${PASSWORD}', m.variables[1].name) 223 | self.assertEqual(2, len(m.variables)) 224 | m.apply_variable_values(['demo', 'mode']) 225 | self.assertEqual('${USERNAME} == demo and ${PASSWORD} == mode', str(m.states[0].actions[0].condition)) 226 | self.assertEqual('Welcome Page', m.states[0].actions[0].next_state.name) 227 | m.apply_variable_values(['invalid', 'invalid']) 228 | self.assertEqual('otherwise', m.states[0].actions[0].condition) 229 | self.assertEqual('Error Page', m.states[0].actions[0].next_state.name) 230 | 231 | 232 | class TestGenerationTestCases(unittest.TestCase): 233 | 234 | def test_generate_all_dfs(self): 235 | m = parsing.parse(_LOGIN_MACHINE) 236 | out = StringIO() 237 | robomachine.generate(m, output=out) 238 | self.assertEqual(_LOGIN_TESTS_GENERATE_ALL_DFS, out.getvalue()) 239 | 240 | if __name__ == '__main__': 241 | unittest.main() 242 | --------------------------------------------------------------------------------