├── .gitignore ├── LICENSE ├── MANIFEST.in ├── README.md ├── _config.yml ├── build.sh ├── data ├── container.json ├── request.json └── stitch.json ├── docs ├── algorithms.md ├── evolutionary_genes.png └── evolutionary_genes.puml ├── figure_1.png ├── requirements.txt ├── run_me.py ├── setup.py ├── stitcher ├── __init__.py ├── bidding.py ├── evolutionary.py ├── iterative_repair.py ├── stitch.py ├── validators.py └── vis.py └── tests ├── __init__.py ├── stitcher_bidding_test.py ├── stitcher_evolutionary_test.py ├── stitcher_iterative_repair_test.py ├── stitcher_stich_test.py ├── stitcher_test.py ├── stitcher_validators_test.py └── stitcher_vis_test.py /.gitignore: -------------------------------------------------------------------------------- 1 | **.pyc* 2 | *.idea/ 3 | .coverage 4 | dist/ 5 | graph_stitcher.egg-info/ -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Copyright (C) 2015 by Thijs Metsch 2 | 3 | Permission is hereby granted, free of charge, to any person obtaining 4 | a copy of this software and associated documentation files (the 5 | "Software"), to deal in the Software without restriction, including 6 | without limitation the rights to use, copy, modify, merge, publish, 7 | distribute, sublicense, and/or sell copies of the Software, and to 8 | permit persons to whom the Software is furnished to do so, subject to 9 | the following conditions: 10 | 11 | The above copyright notice and this permission notice shall be 12 | included in all copies or substantial portions of the Software. 13 | 14 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, 15 | EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF 16 | MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND 17 | NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE 18 | LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION 19 | OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION 20 | WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. -------------------------------------------------------------------------------- /MANIFEST.in: -------------------------------------------------------------------------------- 1 | include setup.py 2 | include LICENSE 3 | include README.md 4 | recursive-include data *.json -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Graph stitching 2 | 3 | This tool is a little framework to determine possible merges between two graphs 4 | based on a set of required additional relationships (aka stitches / edges). 5 | Based on a set of possible resulting candidate graphs we validate which of the 6 | candidates graphs is the "best". 7 | 8 | An example use case is a packaging problem: e.g. set of entities need to be 9 | placed in an container which already contains a set of different entities. 10 | 11 | Let's assume the entities to be placed can be described using a *request* graph 12 | (as in entities having a relationship) and the existing *container* can be 13 | described as a graph as well (entities having relationships and are already 14 | ranked). Nodes & edges in the existing container should be ranked to indicate 15 | how loaded/busy they are. 16 | 17 | Adding a new set of entities (e.g. a lamp described by the lamp bulb & fitting) 18 | to the existing container (e.g. a house) automatically requires certain 19 | relationships to be added (e.g. the power cable which is plugged into the 20 | socket). But not all possible relationship setups might be good as e.g. adding 21 | another consumer (the lamp) to a certain power socket might impact other 22 | consumers (the microwave) of the same socket as the fuse might give up. How 23 | loaded the power socket is can be expressed through the previously mentioned 24 | rank. 25 | 26 | Hence **stitching** two graphs together is done by adding relationships (edges) 27 | between certain types of nodes from the *container* and the *request*. As 28 | multiple stitches are possible we need to know determine and validate (adhering 29 | conditions like compositions and attribute requirements) the possible 30 | candidates and determine the best. Defining the best one can be done using a 31 | very simple principle. A good stitch is defined by: 32 | 33 | * all new relationships are satisfied 34 | * the resulting graph is stable and none of the existing nodes (entities) are 35 | impacted by the requested once. 36 | 37 | Through the pluggable architecture of this tool we can now implement 38 | different algorithms on how to determine if a resulting graph is good 39 | & stable: 40 | 41 | * through a simple rule saying for a node of type A, the maximum number of 42 | incoming dependencies is Y 43 | * through a simple rule saying for a node of type B, that it cannot take more 44 | incoming relationships when it's rank is over an threshold K. 45 | * (many more .e.g implement your own) 46 | 47 | This tool obviously can also be used to enhance the placement of workload after 48 | a temporal scheduling decision has been made (based on round-robin, preemptive, 49 | priority, fair share, fifo, ... algorithms) within clusters. 50 | 51 | ## graph stitcher's internals 52 | 53 | As mentioned above graph stitcher is pluggable, to test different algorithms of 54 | **graph stitching** & validation of the same. Hence it has a pluggable 55 | interface for the *stitch()* and *validate()* routine (see *BaseStitcher* 56 | class). 57 | 58 | To stitch two graphs the tool needs to know which relationships are needed: 59 | 60 | Type of source node | Type of target node 61 | ----------------------------------------- 62 | type_a | type_x 63 | ... | ... 64 | 65 | These are stored in the file (stitch.json). Sample graphs & tests can be found 66 | in the **tests** directory as well. 67 | 68 | There is a *conditional_filter()* function implemented which can do some basic 69 | filtering. Implementing your own is possible by passing conditions and the 70 | *conditional_filter()* routine you want to use as parameters to the *stich()* 71 | call. 72 | 73 | The basic filter support the following operations: 74 | 75 | * based on required (exception: not equal operation) target attributes - 76 | example below: node a requires it's stitched target to have an attribute 77 | 'foo' with value 'y' 78 | * this can also be done with: not equal, larger than, less than or by a 79 | regular expression. 80 | * the notion that two nodes require the same or different target - example 81 | below: node 1 & 2 need to have the same stitched target node and node 3 & 4 82 | need to have different stitched target nodes. 83 | * the notion that stitched target nodes (not) share a common attribute - 84 | example below: node x & y need to be stitched to target nodes which share 85 | the same attribute value for the attribute with the name 'group'. 86 | 87 | The following dictionary can be passed in as a composition condition: 88 | 89 | { 90 | 'attributes': [('eq', ('a', ('foo', 'y'))), 91 | ('neq', ('a', ('foo', 5))), 92 | ('lt', ('a', ('foo', 4))), 93 | ('lg', ('a', ('foo', 7))), 94 | ('regex', ('a', ('foo', '^a')))], 95 | 'compositions': [('same', ('1', '2')), 96 | ('diff', ('3', '4')), 97 | ('share', ('group', ['x', 'y'])), 98 | ('nshare', ('group', ['a', 'b']))] 99 | } 100 | 101 | This graph stitcher is mostly developed to test & play around. Also to check if 102 | [evolutionary algorithms](https://en.wikipedia.org/wiki/Evolutionary_algorithm) 103 | can be developed to determine the best resulting graph. More details on the 104 | algorithms in place can be found in the [/docs](/docs/algorithms.md) directory. 105 | 106 | ## Running it 107 | 108 | Just do a: 109 | 110 | $ ./run_me.py 111 | 112 | You will hopefully see something similar to the following diagram. The *k*, 113 | *l*, *m* nodes form the request. All other nodes represent the container (node 114 | colors indicate the rank, node forms the different types). The stitches are 115 | dotted lines. Each graph is a candidate solution, the results of the 116 | validation are shown as titles of the graphs. 117 | 118 | ![output](./figure_1.png?raw=true "Output") 119 | 120 | To test the evolutionary algorithm run: 121 | 122 | $ ./run_me.py -a evolutionary 123 | 124 | Please note that it might not always find a set of good solutions, as the 125 | container and the request are pretty small. Also note that currently the 126 | fitness function expresses a fitness for the given conditions; and does not 127 | include a fitness value for the validation phase. 128 | 129 | To test the bidding algorithm in which the container nodes try to find the 130 | optimal solution run: 131 | 132 | $ ./run_me.py -a bidding 133 | -------------------------------------------------------------------------------- /_config.yml: -------------------------------------------------------------------------------- 1 | theme: jekyll-theme-cayman -------------------------------------------------------------------------------- /build.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | 3 | python3 -m pycodestyle -r **/*.py 4 | 5 | python3 -m pylint -r n **/*.py 6 | 7 | python3 -m nose --with-coverage --cover-package=stitcher 8 | -------------------------------------------------------------------------------- /data/container.json: -------------------------------------------------------------------------------- 1 | { 2 | "directed": true, 3 | "graph": { 4 | "multigraph": true 5 | }, 6 | "nodes": [ 7 | { 8 | "type": "a", 9 | "id": "A", 10 | "rank": 0 11 | }, 12 | { 13 | "type": "b", 14 | "id": "C", 15 | "rank": 9 16 | }, 17 | { 18 | "type": "a", 19 | "id": "B", 20 | "rank": 5 21 | }, 22 | { 23 | "type": "c", 24 | "id": "E", 25 | "rank": 4 26 | }, 27 | { 28 | "type": "b", 29 | "id": "D", 30 | "rank": 6 31 | }, 32 | { 33 | "type": "c", 34 | "id": "F", 35 | "rank": 3 36 | }, 37 | { 38 | "type": "x", 39 | "id": "1", 40 | "rank": 0 41 | }, 42 | { 43 | "type": "y", 44 | "id": "2", 45 | "rank": 0 46 | } 47 | ], 48 | "links": [ 49 | { 50 | "source": "A", 51 | "target": "C" 52 | }, 53 | { 54 | "source": "B", 55 | "target": "C" 56 | }, 57 | { 58 | "source": "E", 59 | "target": "D" 60 | }, 61 | { 62 | "source": "D", 63 | "target": "C" 64 | }, 65 | { 66 | "source": "F", 67 | "target": "D" 68 | }, 69 | { 70 | "source": "1", 71 | "target": "A" 72 | }, 73 | { 74 | "source": "1", 75 | "target": "2" 76 | }, 77 | { 78 | "source": "2", 79 | "target": "C" 80 | } 81 | ], 82 | "multigraph": false 83 | } -------------------------------------------------------------------------------- /data/request.json: -------------------------------------------------------------------------------- 1 | { 2 | "directed": true, 3 | "graph": { 4 | "multigraph": true 5 | }, 6 | "nodes": [ 7 | { 8 | "type": "x", 9 | "id": "k", 10 | "rank": 0 11 | }, 12 | { 13 | "type": "z", 14 | "id": "m", 15 | "rank": 0 16 | }, 17 | { 18 | "type": "y", 19 | "id": "l", 20 | "rank": 0 21 | } 22 | ], 23 | "links": [ 24 | { 25 | "source": "k", 26 | "target": "l" 27 | }, 28 | { 29 | "source": "l", 30 | "target": "m" 31 | } 32 | ], 33 | "multigraph": false 34 | } -------------------------------------------------------------------------------- /data/stitch.json: -------------------------------------------------------------------------------- 1 | { 2 | "x": "a", 3 | "y": "b", 4 | "z": "c" 5 | } -------------------------------------------------------------------------------- /docs/algorithms.md: -------------------------------------------------------------------------------- 1 | 2 | # Algorithms 3 | 4 | The following section describe the different approaches for stitching the 5 | request graph to the container graph. Note that container graphs could be 6 | sub-graphs of larger container graph for performance reasons. 7 | 8 | ## Full solution 9 | 10 | This is the simplest way to stitch the request to the container graph. It 11 | starts by determining all possible combinations on how the request nodes can 12 | be stitched/mapped to the container. Iterating over that stitch/edge list it 13 | first eliminates all those candidates for which the type mapping is violated. 14 | Following that all stitches which do not meet the conditions are filtered out. 15 | A set of possible candidates is returned which can be validated further. 16 | 17 | Obviously the Big O is not great, as time complexity is correlated with the 18 | number of possible combinations. The number of combinations is dependant on 19 | the number of nodes in the request and container. Some optimization has been 20 | done by using dictionary, and assuring most methods are lookups on 21 | dictionary instead of lists. 22 | 23 | ## Evolutionary 24 | 25 | The [evolutionary](https://en.wikipedia.org/wiki/Evolutionary_algorithm) 26 | algorithm for finding possible solutions uses a more 27 | [darwinism](https://en.wikipedia.org/wiki/Darwinism) based approach. Random 28 | candidates are created which are put into an initial population. Out of that 29 | initial population the best candidates survive, while the others die off. A 30 | candidate is defined by it's genes. Whereby a gene set of a candidate 31 | represent it's individual stitches. 32 | 33 | Within the algorithm, first the best possible parents for new candidates are 34 | determined (default: best 20% of the population. This, like all others 35 | parameters is adjustable.). For diversity a certain percentage (default 10%) 36 | of the population is copied into the new population. New children are now 37 | created from this parent set using a crossover function. Following this a 38 | certain percentage (default 10%) of the children mutates. Overall the 39 | algorithm tries to keep the population size stable - this can be tuned to 40 | allow for growth or decay. The algorithm is finished when either a preset 41 | maximum number of iterations is reached, or the fitness goal is reached 42 | (default: 0.0). This allows for creation of multiple candidates (fitness 43 | goal != 0.0) and ensures the algorithm does not run forever when it dips into 44 | sub-optimal solution space and cannot reach the fitness goal. 45 | 46 | Each candidate contains a genes dictionary which represent the stitches - in 47 | the following example the dictionary would contain *{'a': 'x', 'b': 'y'}*': 48 | 49 | ![evolutionary_genes](evolutionary_genes.png "Gen set of a candidate.") 50 | 51 | The fitness of the candidate is determined by how good the stitches are. The 52 | sum of the fitness values for each stitch determine the candidate's fitness 53 | (and ultimately the utility of the agents in a Multi-Agent System (MAS)). 54 | Stitches which do not adhere the conditions or type conditions get higher 55 | values for the fitness then those, satisfying all criteria. 56 | 57 | The crossover function allows for combing the genes of a candidate with those 58 | of another and result in a new child. When doing the crossover the stitches 59 | with the best individual fitness value are preferred. This assures that the 60 | algorithm finds good solutions fast. 61 | 62 | It is suggested to write more advanced fitness functions which eventually will 63 | obsolete the validation run as needed in the full solution approach. Making 64 | the validation step obsolete can be done by e.g. including the include the 65 | number of nodes & edges in the fitness value. 66 | 67 | Mutation is done by randomly changing certain stitches to different targets. 68 | 69 | The time complexity of this algorithms is obviously much better than working 70 | in the full solution space. By implementing specialized fitness functions it 71 | can be assured that the solutions found are 'optimal'. Note however, that the 72 | algorithm might not always find a solution. Based on size and complexity it 73 | is necessary to tune the parameters introduced earlier to the use case at hand. 74 | 75 | ## Bidding 76 | 77 | The nodes in the container, just like in a Multi-Agent System [1](1), pursue a 78 | certain self-interest, as well as an interest to be able to stitch the request 79 | collectively. Using a [english](https://en.wikipedia.org/wiki/English_auction) 80 | auction (This could be flipped to a 81 | [dutch](https://en.wikipedia.org/wiki/Dutch_auction) one as well) concept the 82 | nodes in the container bid on the node in the request, and hence eventually 83 | express a interest in for a stitch. By bidding credits the nodes in 84 | the container can hide their actually capabilities, capacities and just 85 | express as interest in the form of a value. The more 86 | [intelligence](https://en.wikipedia.org/wiki/Intelligent_agent) is implemented 87 | in the node, the better it can calculate it's bids. 88 | 89 | The algorithm starts by traversing the container graph and each node 90 | calculates it's bids. During traversal the current assignments and bids from 91 | other nodes are 'gossiped' along. The amount of credits bid, depends on if the 92 | node in the request graph matches the type requirement and how well the stitch 93 | fits. The nodes in the container need to base their bids on the bids of their 94 | surrounding environment (especially in cases in which the _same_, _diff_, 95 | _share_, _nshare_ conditions are used). Hence they not only need to know about 96 | the current assignments, but also the neighbours bids. 97 | 98 | For the simple _lg_ and _lt_ conditions, the larger the delta between 99 | what the node in the request graphs asks for (through the conditions) and what 100 | the node in the container offers, the higher the bid is. Whenever a current 101 | bid for a request node's stitch to the container is higher than the current 102 | assignment, the same is updated. In the case of conditions that express that 103 | nodes in the request need to be stitched to the same or different container 104 | node, the credits are calculated based on the container node's knowledge about 105 | other bids, and what is currently assigned. Should a pair of request node be 106 | stitched - with the _diff_ conditions in place - the current container node 107 | will increase it's bid by 50%, if one is already assigned somewhere else. Does 108 | the current container node know about a bid from another node, it will 109 | increase it's bid by 25%. 110 | 111 | For example a container with nodes A, B, C, D needs to be stitched to a 112 | request of nodes X, Y. For X, Y there is a _share_ filter defined - which 113 | either A & B, or C & D can fulfill. Let's assume A bids 5 credits for 114 | X, and B bids 2 credits for Y, and C bids 4 credits for X and D bids 6 115 | credits for Y. Hence the group C & D would be the better fit. When evaluating 116 | D, D needs to know X is currently assigned to A - but also needs to know the 117 | bid of C so is can increase it's bid on Y. When C is revisited it can increase 118 | it's bid given D has Y assigned. As soon as the nodes A & B are revisited they 119 | will eventually drop their bids, as they now know C & D can serve the request 120 | X, Y better. They hence sacrifice their bis for the needs of the greater good. 121 | So the fact sth is assigned to a neighbour matters more then the bid of the 122 | neighbour (increase by 50%) - but still, the knowledge of the neighbour's bid 123 | is crucial (increase by 25%) - e.g. if bid from C would be 0, D should not 124 | bit for Y. 125 | 126 | The ability to increase the bids by 25% or 50% is important to differentiate 127 | between the fact that sth is assigned somewhere else, or if the environment 128 | the node knows about includes a bid that can help it's own bid. 129 | 130 | Note this algorithm does not guarantee stability. Also for better results in 131 | specific use cases it is encourage to change how the credits are calculated. 132 | For example based on the utility of the container node. Especially for the 133 | _share_ attribute condition there might be cases that the algorithm does not 134 | find the optimal solution, as the increasing percentages (50%, 25%) are not 135 | optimal. The time complexity depends on the number of nodes in the container 136 | graph, their structure and how often bids & assignment change. This is 137 | because currently the container graph is traversed synchronously. In future a 138 | async approach would be beneficial, which will also allow for parallel 139 | calculation of bids. 140 | 141 | [1]: https://www.cs.ox.ac.uk/people/michael.wooldridge/pubs/imas/IMAS2e.html 142 | "An Introduction to MultiAgent Systems." 143 | -------------------------------------------------------------------------------- /docs/evolutionary_genes.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tmetsch/graph_stitcher/45b058813f39a5f4c13c01b159197f14e4a45745/docs/evolutionary_genes.png -------------------------------------------------------------------------------- /docs/evolutionary_genes.puml: -------------------------------------------------------------------------------- 1 | @startditaa 2 | /-------------------------\ 3 | | +---------+ | 4 | | request : a ... b | | 5 | | +---------+ | 6 | | |...| | 7 | | v v | 8 | | +-+-------+ | 9 | | container : x ... y | | 10 | | +---------+ | 11 | \-------------------------/ 12 | fitness of candidate 13 | @endditaa -------------------------------------------------------------------------------- /figure_1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tmetsch/graph_stitcher/45b058813f39a5f4c13c01b159197f14e4a45745/figure_1.png -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | # Install using: pip install -r requirements.txt 2 | matplotlib>=1.4.2 3 | networkx>=2.4 4 | pygraphviz>=1.3rc2 5 | pydot>=1.0.32 6 | -------------------------------------------------------------------------------- /run_me.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | """ 4 | Sample executor showing the result of the stitching & validation methods. 5 | """ 6 | 7 | import argparse 8 | import json 9 | import sys 10 | import logging 11 | 12 | from networkx.readwrite import json_graph 13 | 14 | from stitcher import bidding 15 | from stitcher import evolutionary 16 | from stitcher import iterative_repair 17 | from stitcher import stitch 18 | from stitcher import validators 19 | from stitcher import vis 20 | 21 | FORMAT = "%(asctime)s - %(filename)s - %(lineno)s - " \ 22 | "%(levelname)s - %(message)s" 23 | logging.basicConfig(format=FORMAT, level=logging.DEBUG) 24 | 25 | 26 | def main(algo): 27 | """ 28 | main routine. 29 | """ 30 | container_tmp = json.load(open('data/container.json')) 31 | container = json_graph.node_link_graph(container_tmp, directed=True) 32 | request_tmp = json.load(open('data/request.json')) 33 | request = json_graph.node_link_graph(request_tmp, directed=True) 34 | rels = json.load(open('data/stitch.json')) 35 | conditions = {} 36 | 37 | if algo == 'evolutionary': 38 | # to show the true power of this :-) 39 | conditions = {'compositions': [('diff', ('k', 'l'))]} 40 | stitcher = evolutionary.EvolutionarySticher(rels) 41 | elif algo == 'bidding': 42 | # to show the true power of this :-) 43 | conditions = {'attributes': [('lt', ('l', ('rank', 9)))]} 44 | stitcher = bidding.BiddingStitcher(rels) 45 | elif algo == 'repair': 46 | # to show the true power of this :-) 47 | conditions = {'attributes': [('eq', ('k', ('rank', 5)))]} 48 | stitcher = iterative_repair.IterativeRepairStitcher(rels) 49 | else: 50 | stitcher = stitch.GlobalStitcher(rels) 51 | graphs = stitcher.stitch(container, request, conditions=conditions) 52 | 53 | if graphs: 54 | results = validators.validate_incoming_edges(graphs, {'b': 5}) 55 | # XXX: disable this if you do not want to see the results. 56 | vis.show(graphs, request, results) 57 | 58 | 59 | if __name__ == '__main__': 60 | OPT = ['global', 'evolutionary', 'bidding', 'repair'] 61 | PARSER = argparse.ArgumentParser() 62 | PARSER.add_argument('-a', choices=OPT, default='global', 63 | help='Select the stitching algorithm.') 64 | ALGO = PARSER.parse_args(sys.argv[1:]).a 65 | main(ALGO) 66 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | """ 2 | Setup. 3 | """ 4 | 5 | from setuptools import setup 6 | 7 | setup(name='graph_stitcher', 8 | version='0.0.20', 9 | author='Thijs Metsch', 10 | url='https://github.com/tmetsch/graph_stitcher', 11 | description=('This tool is a little framework to determine possible' 12 | 'merges between two graphs based on a set of required' 13 | 'additional relationships (aka as stitches / edges).'), 14 | license='MIT', 15 | keywords='graph stitching algorithms framework', 16 | packages=['stitcher'], 17 | classifiers=[ 18 | 'Development Status :: 3 - Alpha', 19 | 'License :: OSI Approved :: MIT License', 20 | 'Topic :: Scientific/Engineering', 21 | 'Topic :: Utilities' 22 | ]) 23 | -------------------------------------------------------------------------------- /stitcher/__init__.py: -------------------------------------------------------------------------------- 1 | """ 2 | Module implementing the graph stitcher. 3 | """ 4 | 5 | TYPE_ATTR = 'type' 6 | 7 | 8 | class Stitcher: 9 | """ 10 | A stitcher. 11 | """ 12 | 13 | def __init__(self, rels): 14 | """ 15 | Initiate the stitcher. 16 | 17 | ;:param rels: A dictionary defining what type of nodes in the request 18 | must be stitched to what type of nodes in the container. 19 | """ 20 | self.rels = rels 21 | 22 | def stitch(self, container, request, conditions=None): 23 | """ 24 | Stitch a request graph into an existing graph container. Returns a set 25 | of possible options. 26 | 27 | :param container: A graph describing the existing container with 28 | ranks. 29 | :param request: A graph describing the request. 30 | :param conditions: Dictionary with conditions - e.g. node a & b need 31 | to be related to node c. 32 | :return: List of resulting graphs(s). 33 | """ 34 | raise NotImplementedError('Not implemented yet.') 35 | -------------------------------------------------------------------------------- /stitcher/bidding.py: -------------------------------------------------------------------------------- 1 | """ 2 | Implements stitching, and filtering functions based on a bidding concept. 3 | """ 4 | 5 | import copy 6 | import logging 7 | import re 8 | 9 | import networkx as nx 10 | 11 | import stitcher 12 | 13 | TMP = {} 14 | 15 | FACTOR_1 = 1.25 16 | FACTOR_2 = 1.5 17 | 18 | 19 | def _other_bids(all_bids): 20 | """ 21 | XXX: see if this can be simplified. Maybe by using diff struct in bids. 22 | """ 23 | res = [] 24 | for item in all_bids.values(): 25 | for sub_item in item: 26 | res.append(sub_item) 27 | return res 28 | 29 | 30 | def _attribute_condy(condy, my_bids, param, node_attr): 31 | """ 32 | Basic rules to deal with attribute conditions. 33 | """ 34 | node = param[0] 35 | attrn = param[1][0] 36 | attrv = param[1][1] 37 | if node not in my_bids: 38 | # I'm not bidding on this node - so no need to do sth. 39 | return 40 | 41 | if condy == 'lg': 42 | # huge delta -> high bid. 43 | my_bids[node] = node_attr[attrn] - attrv + my_bids[node] 44 | elif condy == 'lt': 45 | # huge delta -> high bid. 46 | my_bids[node] = attrv - node_attr[attrn] + my_bids[node] 47 | elif condy == 'eq': 48 | # attr not present or not equal -> drop bid. 49 | if attrn not in node_attr or attrv != node_attr[attrn]: 50 | my_bids.pop(node) 51 | elif condy == 'neq': 52 | # attr present and equal -> drop bid. 53 | if attrn in node_attr and attrv == node_attr[attrn]: 54 | my_bids.pop(node) 55 | elif condy == 'regex': 56 | # attr present and reges does not match -> drop bid. 57 | if attrn in node_attr and not re.search(attrv, node_attr[attrn]): 58 | my_bids.pop(node) 59 | 60 | 61 | def _same_condy(my_bids, param): 62 | flag = True 63 | # if I do a bid on all elements in param... 64 | for item in param: 65 | if item not in list(my_bids.keys()): 66 | flag = False 67 | break 68 | # ... I'll increase the bids. 69 | # TODO: only if my bid is better though... 70 | if flag: 71 | for item in param: 72 | my_bids[item] = my_bids[item] * FACTOR_2 73 | else: 74 | for item in param: 75 | # if not all can be stitched to me - I need to remove all bids. 76 | if item in my_bids: 77 | my_bids.pop(item) 78 | 79 | 80 | def _diff_condy(my_bids, param, assigned, all_bids): 81 | # if other entity has the others - I can increase my bid for the greater 82 | # good. 83 | for item in param: 84 | if set(param) - set(assigned.keys()) == {item}: 85 | # if others assigned increase by factor 2 86 | my_bids[item] = my_bids[item] * FACTOR_2 87 | elif set(param) - set(_other_bids(all_bids)) == {item}: 88 | # if others are bid on increase by factor 1 89 | my_bids[item] = my_bids[item] * FACTOR_1 90 | elif len(set(param) - set(_other_bids(all_bids))) == 0 \ 91 | and item in my_bids: 92 | # bids out for all - increase by factor 1 93 | my_bids[item] = my_bids[item] * FACTOR_1 94 | 95 | # keep highest bid - remove rest. 96 | keeper = max(my_bids, 97 | key=lambda k: my_bids[k] if k in param else float('-inf')) 98 | for item in param: 99 | if item in my_bids and item != keeper: 100 | my_bids.pop(item) 101 | 102 | 103 | def _nshare_condy(my_bids, param, assigned, node, container): 104 | attrn = param[0] 105 | attrv_assigned = None 106 | nodes = param[1] 107 | for item in nodes: 108 | if item in assigned and attrv_assigned is None: 109 | tmp = [n for n in container.nodes if n.name == assigned[item][0]] 110 | attrv_assigned = container.nodes[tmp[0]][attrn] 111 | elif attrv_assigned is not None and item in my_bids \ 112 | and container.nodes[node][attrn] != attrv_assigned: 113 | my_bids[item] = my_bids[item] * FACTOR_2 114 | 115 | 116 | CACHE = {} 117 | 118 | 119 | def _sub_stitch(container, request, mapping, condy): 120 | if (container, request) not in CACHE: 121 | opt_graph = nx.DiGraph() 122 | tmp = {} 123 | for node, attr in container.nodes(data=True): 124 | tmp[node] = Entity(str(node), mapping, request, opt_graph, 125 | conditions=condy) 126 | opt_graph.add_node(tmp[node], **attr) 127 | for src, trg, attr in container.edges(data=True): 128 | opt_graph.add_edge(tmp[src], tmp[trg], **attr) 129 | 130 | start_node = list(tmp.values())[0] 131 | 132 | # kick off 133 | assign, _ = start_node.trigger({'bids': [], 'assigned': {}}, 134 | 'sub-init') 135 | CACHE[container, request] = assign 136 | 137 | return CACHE[(container, request)] 138 | 139 | 140 | class Entity: 141 | """ 142 | A node in a graph that can bid on nodes of the request graph. 143 | """ 144 | 145 | def __init__(self, name, mapping, request, container, conditions=None): 146 | self.name = name 147 | self.container = container 148 | self.request = request 149 | self.mapping = mapping 150 | self.bids = {} 151 | self.conditions = conditions or {} 152 | 153 | def _share_condy(self, condy, param, my_bids, assigned): 154 | """ 155 | If I know that mine and surrounding bids are lower than what another 156 | combo can offer - drop bids & assignments! 157 | """ 158 | attrn = param[0] 159 | nodes = param[1] 160 | if attrn not in self.container.nodes[self]: 161 | for item in nodes: 162 | if item in my_bids: 163 | my_bids.pop(item) 164 | return [] 165 | my_attrv = self.container.nodes[self][attrn] 166 | bids = self.bids 167 | 168 | cache = {} 169 | # step 1) find possible groups that I know of. 170 | for item in bids: 171 | tmp = [n for n in self.container.nodes() if n.name == item][0] 172 | if attrn in self.container.nodes[tmp]: 173 | attrv = self.container.nodes[tmp][attrn] 174 | if attrv not in cache: 175 | cache[attrv] = [tmp] 176 | else: 177 | cache[attrv].append(tmp) 178 | 179 | options = {} 180 | # step 2) figure out what the groups are worth. 181 | for item in cache: 182 | sub_container = nx.subgraph(self.container, cache[item]) 183 | sub_request = nx.subgraph(self.request, nodes) 184 | sub_condy = copy.deepcopy(self.conditions) 185 | sub_condy['compositions'].remove(condy) 186 | assign = _sub_stitch(sub_container, sub_request, self.mapping, 187 | sub_condy) 188 | # set() so 2,1 == 1,2 in py 3. 189 | if set(assign.keys()) == set(nodes): 190 | # is complete 191 | options[item] = sum([val[1] for val in assign.values()]) 192 | for tmp in nodes: 193 | if tmp in my_bids and self in sub_container: 194 | my_bids[tmp] = my_bids[tmp] * FACTOR_2 195 | 196 | # step 3) drop my bids & assignments. 197 | res = [] 198 | if my_attrv in options and len(options) >= 2 \ 199 | and sorted(options, key=options.get, 200 | reverse=True)[0] != my_attrv: 201 | for item in nodes: 202 | if item in my_bids: 203 | my_bids.pop(item) 204 | res.append(item) 205 | for item in nodes: 206 | if item in assigned and assigned[item][0] == self.name: 207 | assigned.pop(item) 208 | return res 209 | 210 | def _apply_conditions(self, my_bids, assigned): 211 | """ 212 | Alter bids based on conditions. 213 | """ 214 | if 'attributes' in self.conditions: 215 | for item in self.conditions['attributes']: 216 | condy = item[0] 217 | param = item[1] 218 | node_attr = self.container.nodes[self] 219 | _attribute_condy(condy, my_bids, param, node_attr) 220 | if 'compositions' in self.conditions: 221 | for item in self.conditions['compositions']: 222 | condy = item[0] 223 | param = item[1] 224 | if condy == 'same': 225 | _same_condy(my_bids, param) 226 | elif condy == 'diff': 227 | _diff_condy(my_bids, param, assigned, self.bids) 228 | elif condy == 'share': 229 | self._share_condy(item, param, my_bids, assigned) 230 | elif condy == 'nshare': 231 | _nshare_condy(my_bids, param, assigned, 232 | self, 233 | self.container) 234 | return my_bids 235 | 236 | def _calc_credits(self, assigned): 237 | tmp = {} 238 | for node, attr in self.request.nodes(data=True): 239 | if self.mapping[attr[stitcher.TYPE_ATTR]] == \ 240 | self.container.nodes[self][stitcher.TYPE_ATTR]: 241 | tmp[node] = 1.0 242 | if tmp: 243 | self.bids[self.name] = self._apply_conditions(tmp, assigned) 244 | logging.info(self.name + ': current bids ' + repr(self.bids)) 245 | 246 | def trigger(self, msg, src): 247 | """ 248 | Triggers a bidding round on this entity. 249 | 250 | :param msg: dict containing the message from other entity. 251 | :param src: str indicating the src of where the message comes from. 252 | """ 253 | logging.info('%s -> %s msg: %s', str(src), str(self.name), str(msg)) 254 | 255 | # let's add those I didn't know of 256 | mod = msg['bids'] == self.bids 257 | for item in msg['bids']: 258 | # each entity controls own bids! 259 | self.bids[item] = msg['bids'][item] 260 | 261 | assigned = msg['assigned'] 262 | self._calc_credits(assigned) 263 | # assign and optimize were needed. 264 | if self.name in self.bids: 265 | for bid in self.bids[self.name]: 266 | rq_n = bid # request node 267 | crd = self.bids[self.name][bid] # bid 'credits'. 268 | if rq_n not in assigned: 269 | logging.info('Assigning: ' + rq_n + ' -> ' + self.name) 270 | assigned[rq_n] = (self.name, crd) 271 | else: 272 | if assigned[rq_n][1] < crd: 273 | # my bid is better. 274 | logging.info('Updated assignment: ' + rq_n + ' -> ' + 275 | self.name) 276 | assigned[rq_n] = (self.name, crd) 277 | 278 | # communicate 279 | # XXX: enable async. 280 | for neighbour in nx.all_neighbors(self.container, self): 281 | if neighbour.name != src and not mod: 282 | # only update neighbour when there is an update to be send. 283 | neighbour.trigger(src=self.name, 284 | msg={'bids': self.bids, 285 | 'assigned': assigned}) 286 | elif msg is not None and msg['bids'] != self.bids: 287 | # only call my caller back when he didn't knew me. 288 | neighbour.trigger(src=self.name, 289 | msg={'bids': self.bids, 290 | 'assigned': assigned}) 291 | return assigned, self.bids 292 | 293 | def __repr__(self): 294 | return self.name 295 | 296 | 297 | class BiddingStitcher(stitcher.Stitcher): 298 | """ 299 | Stitcher based on a bidding concept where each Agent of MAS bids for nodes 300 | from the request graph. 301 | """ 302 | 303 | def stitch(self, container, request, conditions=None, start=None): 304 | condy = conditions or {} 305 | opt_graph = nx.DiGraph() 306 | tmp = {} 307 | for node, attr in container.nodes(data=True): 308 | tmp[node] = Entity(str(node), self.rels, request, opt_graph, 309 | conditions=condy) 310 | opt_graph.add_node(tmp[node], **attr) 311 | for src, trg, attr in container.edges(data=True): 312 | opt_graph.add_edge(tmp[src], tmp[trg], **attr) 313 | 314 | start_node = list(tmp.values())[0] 315 | if start is not None: 316 | start_node = tmp[start] 317 | 318 | # kick off 319 | assign, _ = start_node.trigger({'bids': [], 'assigned': {}}, 'init') 320 | tmp_graph = nx.union(container, request) 321 | for item in assign: 322 | src = item 323 | trg = assign[item][0] 324 | tmp_graph.add_edge(src, trg) 325 | 326 | return [tmp_graph] 327 | -------------------------------------------------------------------------------- /stitcher/evolutionary.py: -------------------------------------------------------------------------------- 1 | """ 2 | Implements stitching, and filtering functions based on evolutionary algorithm. 3 | """ 4 | 5 | import logging 6 | import random 7 | import re 8 | import networkx as nx 9 | 10 | import stitcher 11 | 12 | LOG = logging.getLogger() 13 | 14 | 15 | def _eq_attr(node, attr, gens, container): 16 | """ 17 | Calcs fitness based on the fact that node need target node to have an attr 18 | with a certain value. 19 | """ 20 | trg_nd = container.nodes[gens[node]] 21 | if attr[0] not in trg_nd: 22 | return 10.1 23 | if attr[1] != trg_nd[attr[0]]: 24 | return 10.2 25 | return 0.0 26 | 27 | 28 | def _neq_attr(node, attr, gens, container): 29 | """ 30 | Calcs fitness based on the fact that node's target shall not have an attr 31 | with a certain value. 32 | """ 33 | trg_nd = container.nodes[gens[node]] 34 | if attr[0] in trg_nd and attr[1] == trg_nd[attr[0]]: 35 | return 10.1 36 | return 0.0 37 | 38 | 39 | def _lg_attr(node, attr, gens, container): 40 | """ 41 | Calcs fitness based on the fact that node's target node shall have an attr 42 | with a value larger than the given one. 43 | """ 44 | trg_nd = container.nodes[gens[node]] 45 | if attr[0] not in trg_nd: 46 | return 10.1 47 | if attr[1] >= trg_nd[attr[0]]: 48 | return 10.2 49 | return 0.0 50 | 51 | 52 | def _lt_attr(node, attr, gens, container): 53 | """ 54 | Calcs fitness based on the fact that node's target node shall have an attr 55 | with a value smaller than the given one. 56 | """ 57 | trg_nd = container.nodes[gens[node]] 58 | if attr[0] not in trg_nd: 59 | return 10.1 60 | if attr[1] < trg_nd[attr[0]]: 61 | return 10.2 62 | return 0.0 63 | 64 | 65 | def _regex_attr(node, attr, gens, container): 66 | """ 67 | Calcs fitness based on the fact that node's target node shall have an attr 68 | with a value smaller than the given one. 69 | """ 70 | trg_nd = container.nodes[gens[node]] 71 | if attr[0] not in trg_nd: 72 | return 10.1 73 | if not re.search(attr[1], trg_nd[attr[0]]): 74 | return 10.2 75 | return 0.0 76 | 77 | 78 | def _same_target(node1, node2, gens): 79 | """ 80 | Calcs fitness based on the fact that two nodes should share same target. 81 | """ 82 | shared_tg = None 83 | for src in gens: 84 | if shared_tg is None and src in [node1, node2]: 85 | shared_tg = gens[src] 86 | elif shared_tg is not None and gens[src] != shared_tg: 87 | return 10.0 88 | return 0.0 89 | 90 | 91 | def _diff_target(node1, node2, gens): 92 | """ 93 | Calc fitness based on the fact that two nodes should not share same target. 94 | """ 95 | shared_tg = None 96 | for src in gens: 97 | if shared_tg is None and src in [node1, node2]: 98 | shared_tg = gens[src] 99 | elif shared_tg is not None and gens[src] == shared_tg: 100 | return 10.0 101 | return 0.0 102 | 103 | 104 | def _share_attr(attrn, node_list, gens, container): 105 | """ 106 | Calcs fitness based on the fact that two nodes from the request should be 107 | stitched to two nodes in the container which share the same attribute 108 | value. 109 | """ 110 | attrv = None 111 | for node in node_list: 112 | trg = gens[node] 113 | if attrn not in container.nodes[trg]: 114 | return 10.1 115 | if attrv is None: 116 | attrv = container.nodes[trg][attrn] 117 | elif attrv != container.nodes[trg][attrn]: 118 | return 10.2 119 | return 0.0 120 | 121 | 122 | def _nshare_attr(attrn, node_list, gens, container): 123 | """ 124 | Calcs fitness based on the fact that two nodes from the request should be 125 | stitched to two nodes in the container which do not share the same 126 | attribute value. 127 | """ 128 | attrv = None 129 | for node in node_list: 130 | trg = gens[node] 131 | if attrn not in container.nodes[trg]: 132 | return 10.1 133 | if attrv is None: 134 | attrv = container.nodes[trg][attrn] 135 | elif attrv == container.nodes[trg][attrn]: 136 | return 10.2 137 | return 0.0 138 | 139 | 140 | def _my_filter(conditions, gens, container): 141 | """ 142 | Apply filters. 143 | """ 144 | res = 0.0 145 | if conditions is not None: 146 | if 'attributes' in conditions: 147 | for condition in conditions['attributes']: 148 | para1 = condition[1][0] 149 | para2 = condition[1][1] 150 | if condition[0] == 'eq': 151 | res += _eq_attr(para1, para2, gens, container) 152 | elif condition[0] == 'neq': 153 | res += _neq_attr(para1, para2, gens, container) 154 | elif condition[0] == 'lg': 155 | res += _lg_attr(para1, para2, gens, container) 156 | elif condition[0] == 'lt': 157 | res += _lt_attr(para1, para2, gens, container) 158 | elif condition[0] == 'regex': 159 | res += _regex_attr(para1, para2, gens, container) 160 | if 'compositions' in conditions: 161 | for condition in conditions['compositions']: 162 | para1 = condition[1][0] 163 | para2 = condition[1][1] 164 | if condition[0] == 'same': 165 | res += _same_target(para1, para2, gens) 166 | elif condition[0] == 'diff': 167 | res += _diff_target(para1, para2, gens) 168 | elif condition[0] == 'share': 169 | res += _share_attr(para1, para2, gens, container) 170 | elif condition[0] == 'nshare': 171 | res += _nshare_attr(para1, para2, gens, container) 172 | return res 173 | 174 | 175 | class Candidate: 176 | """ 177 | A candidate of a population for an evolutionary algorithm 178 | """ 179 | 180 | def __init__(self, gen): 181 | """ 182 | Initialize the candidate. 183 | 184 | :param gen: The genetics of the candidate 185 | """ 186 | self.gen = gen 187 | 188 | def fitness(self): 189 | """ 190 | Calculate the fitness of this candidate based on it's genes. 191 | :return: 192 | """ 193 | raise NotImplementedError('Not done yet.') 194 | 195 | def mutate(self): 196 | """ 197 | Mutate the candidate. 198 | """ 199 | raise NotImplementedError('Not done yet.') 200 | 201 | def crossover(self, partner): 202 | """ 203 | Cross this candidate with another one. 204 | """ 205 | raise NotImplementedError('Not done yet.') 206 | 207 | def __eq__(self, other): 208 | return self.gen == other.gen 209 | 210 | 211 | class GraphCandidate(Candidate): 212 | """ 213 | Candidate within a population. The DNA of this candidate is defined by a 214 | dictionary of source to target stitches. 215 | """ 216 | 217 | def __init__(self, gen, stitch, conditions, mutation_list, request, 218 | container): 219 | super(GraphCandidate, self).__init__(gen) 220 | self.stitch = stitch 221 | self.conditions = conditions 222 | self.mutation_list = mutation_list 223 | self.request = request 224 | self.container = container 225 | 226 | def fitness(self): 227 | fit = 0.0 228 | 229 | # 1. stitch 230 | for src in self.gen: 231 | trg = self.gen[src] 232 | if self.container.nodes[trg][stitcher.TYPE_ATTR] != \ 233 | self.stitch[self.request.nodes[src][stitcher.TYPE_ATTR]]: 234 | fit += 100 235 | 236 | # 2. conditions 237 | fit += _my_filter(self.conditions, self.gen, self.container) 238 | 239 | return fit 240 | 241 | def mutate(self): 242 | # let's mutate to an option outside of the shortlisted candidate list. 243 | src = random.choice(list(self.gen.keys())) 244 | 245 | done = False 246 | cutoff = len(self.gen) 247 | i = 0 248 | while not done and i <= cutoff: 249 | # break off as there might be no other match available. 250 | nd_trg = self.mutation_list[random.randint( 251 | 0, len(self.mutation_list) - 1)][0] 252 | if self.container.nodes[nd_trg][stitcher.TYPE_ATTR] == \ 253 | self.stitch[self.request.nodes[src][stitcher.TYPE_ATTR]]: 254 | done = True 255 | self.gen[src] = nd_trg 256 | i += 1 257 | 258 | def crossover(self, partner): 259 | tmp = {} 260 | 261 | for src in self.gen: 262 | # pick one from partner 263 | done = False 264 | nd_trg = '' 265 | cutoff = len(self.gen) 266 | i = 0 267 | while not done and i <= cutoff: 268 | nd_trg = random.choice(list(partner.gen.values())) 269 | if self.container.nodes[nd_trg][stitcher.TYPE_ATTR] == \ 270 | self.stitch[self.request.nodes[src][ 271 | stitcher.TYPE_ATTR]]: 272 | done = True 273 | i += 1 274 | if done: 275 | tmp[src] = nd_trg 276 | else: 277 | tmp[src] = self.gen[src] 278 | 279 | return self.__class__(tmp, self.stitch, self.conditions, 280 | self.mutation_list, self.request, self.container) 281 | 282 | def __repr__(self): 283 | return 'f: ' + str(self.fitness()) + ' - ' + repr(self.gen) 284 | 285 | 286 | class BasicEvolution: 287 | """ 288 | Implements basic evolutionary behaviour. 289 | """ 290 | 291 | def __init__(self, percent_cutoff=0.2, percent_diversity=0.1, 292 | percent_mutate=0.1, growth=1.0): 293 | """ 294 | Initialize. 295 | 296 | :param percent_cutoff: Which top percentage of the population survives. 297 | :param percent_diversity: Which percentage to randomly add from the 298 | unfit population portion - keeps diversity up. 299 | :param percent_mutate: Percentage of the new population which should 300 | mutate randomly. 301 | :param growth: growth percentage of the new population to indicate 302 | growth/shrinking. 303 | """ 304 | self.cutoff = percent_cutoff 305 | self.diversity = percent_diversity 306 | self.mutate = percent_mutate 307 | self.growth = growth 308 | 309 | def _darwin(self, population): 310 | """ 311 | Evolve the population - the strongest survive and some randomly picked 312 | will survive. A certain percentage will mutate. 313 | 314 | :param population: Population list sorted by fitness. 315 | :return: List of candidates 316 | """ 317 | # best possible parents 318 | new_population = population[0:int(len(population) * self.cutoff)] 319 | div_pool = population[int(len(population) * self.cutoff):-1] 320 | 321 | # diversity - select some random from 'bad' pool 322 | for _ in range(int(self.diversity * len(div_pool))): 323 | i = random.randint(0, len(div_pool) - 1) 324 | new_population.append(div_pool[i]) 325 | 326 | # create children 327 | len_pop = len(population) 328 | len_new_pop = len(new_population) 329 | for _ in range(int(len_pop * self.growth) - len_new_pop): 330 | candidate1 = population[random.randint(0, len_new_pop - 1)] 331 | candidate2 = population[random.randint(0, len_new_pop - 1)] 332 | child = candidate1.crossover(candidate2) 333 | # if child not in new_population: # TODO: check if helpful. 334 | new_population.append(child) 335 | 336 | # mutate some 337 | for _ in range(int(self.mutate * (len_pop - len_new_pop))): 338 | i = random.randint(len_new_pop, len_pop - 1) 339 | new_population[i].mutate() 340 | 341 | LOG.debug('New population length: %s', str(len(new_population))) 342 | return new_population 343 | 344 | def run(self, population, max_runs, fitness_goal=0.0, stabilizer=False): 345 | """ 346 | Play the game of life. 347 | 348 | :param population: The initial population. 349 | :param max_runs: Maximum number of iterations. 350 | :param fitness_goal: Breakoff value for fitness - default 0.0 351 | :param stabilizer: If True iteration will break off after population 352 | stabilizes - comes with a slight performance penalty. Note: only 353 | useful when the population has a stable length (growth=1.0) :-). 354 | :return: Number of iterations used and the final population. 355 | """ 356 | # evolve till max iter, 0.0 (goal) or all death 357 | iteration = 0 358 | fitness_sum = 0.0 359 | population.sort(key=lambda candidate: candidate.fitness()) 360 | while iteration <= max_runs: 361 | LOG.debug('Iteration: %s', str(iteration)) 362 | 363 | population = self._darwin(population) 364 | population.sort(key=lambda candidate: candidate.fitness()) 365 | 366 | if population[0].fitness() == fitness_goal: 367 | LOG.info('Found solution in iteration: %s', str(iteration)) 368 | break 369 | 370 | if stabilizer: 371 | new_fitness_sum = sum(candidate.fitness() for 372 | candidate in population) 373 | if new_fitness_sum == fitness_sum: 374 | # in the last solution all died... 375 | LOG.info('Population stabilized after iteration: %s', 376 | str(iteration - 1)) 377 | break 378 | fitness_sum = new_fitness_sum 379 | 380 | iteration += 1 381 | 382 | # show results 383 | if iteration >= max_runs: 384 | LOG.warning('Maximum number of iterations reached') 385 | LOG.info('Final population: %s', repr(population)) 386 | return iteration, population 387 | 388 | 389 | class EvolutionarySticher(stitcher.Stitcher): 390 | """ 391 | Stitcher which uses an evolutionary algorithm. 392 | """ 393 | 394 | def __init__(self, rels, max_iter=10, fit_goal=-1.0, cutoff=0.9, 395 | mutate=0.0, candidates=10): 396 | """ 397 | Initializes this stitcher. 398 | 399 | :param filename: Filename of the mapping file. 400 | :param max_iter: Number of maximum iterations (default 10). 401 | :param fit_goal: Fitness goal for the evolutionary algorithm (default 402 | -1.0 : results in multiple solutions). 403 | :param cutoff: Percentage of populate that survives (default 90%). 404 | :param mutate: Percentage of population that mutates (default None). 405 | :param candidates: Number of candidates to randomly generate 406 | (default 10). 407 | """ 408 | super(EvolutionarySticher, self).__init__(rels) 409 | self.max_iter = max_iter 410 | self.fit_goal = fit_goal 411 | self.cutoff = cutoff 412 | self.mutate = mutate 413 | self.candidates = candidates 414 | 415 | def stitch(self, container, request, conditions=None): 416 | evo = BasicEvolution(percent_cutoff=self.cutoff, 417 | percent_mutate=self.mutate) 418 | conditions = {} or conditions 419 | 420 | # initial population 421 | population = [] 422 | for _ in range(self.candidates): 423 | tmp = {} 424 | for item in request.nodes(): 425 | trg_cand = random.choice(list(container.nodes().keys())) 426 | tmp[item] = trg_cand 427 | population.append(GraphCandidate(tmp, self.rels, conditions, [], 428 | request, container)) 429 | 430 | _, population = evo.run(population, 431 | self.max_iter, 432 | fitness_goal=self.fit_goal) 433 | 434 | if population[0].fitness() != 0.0: 435 | logging.warning('Please rerun - did not find a viable solution') 436 | return [] 437 | 438 | graphs = [] 439 | for candidate in population: 440 | if candidate.fitness() == 0.0: 441 | tmp_graph = nx.union(container, request) 442 | for item in candidate.gen: 443 | tmp_graph.add_edge(item, candidate.gen[item]) 444 | graphs.append(tmp_graph) 445 | return graphs 446 | -------------------------------------------------------------------------------- /stitcher/iterative_repair.py: -------------------------------------------------------------------------------- 1 | """ 2 | Module using an iterative repair approach. 3 | """ 4 | 5 | import logging 6 | import random 7 | import re 8 | 9 | import networkx as nx 10 | 11 | import stitcher 12 | 13 | 14 | def convert_conditions(conditions): 15 | """ 16 | Convert the conditions to a mapping of nodename->list of conditions. 17 | """ 18 | if conditions is None: 19 | return {} 20 | res = {} 21 | if 'attributes' in conditions: 22 | for condition in conditions['attributes']: 23 | cond = condition[0] 24 | node = condition[1][0] 25 | attr = condition[1][1] 26 | if node in res: 27 | res[node].append((cond, attr)) 28 | else: 29 | res[node] = [(cond, attr)] 30 | if 'compositions' in conditions: 31 | for condition in conditions['compositions']: 32 | cond = condition[0] 33 | if cond in ['same', 'diff']: 34 | for item in condition[1]: 35 | node = item 36 | if node in res: 37 | res[node].append((cond, [item for item in condition[1] 38 | if item != node][0])) 39 | else: 40 | res[node] = [(cond, [item for item in condition[1] 41 | if item != node][0])] 42 | else: 43 | for item in condition[1][1]: 44 | node = item 45 | attr = condition[1][0] 46 | if node in res: 47 | res[node].append((cond, (attr, [item for item in 48 | condition[1] 49 | if item != node]))) 50 | else: 51 | res[node] = [(cond, (attr, [item for item in 52 | condition[1] 53 | if item != node]))] 54 | return res 55 | 56 | 57 | class IterativeRepairStitcher(stitcher.Stitcher): 58 | """ 59 | Stitcher using a iterative repair approach to solve the constraints. 60 | """ 61 | 62 | def __init__(self, rels, max_steps=30): 63 | super(IterativeRepairStitcher, self).__init__(rels) 64 | self.steps = max_steps 65 | 66 | def stitch(self, container, request, conditions=None): 67 | conditions = convert_conditions(conditions) 68 | mapping = {} 69 | 70 | # initial (random mapping) 71 | for node, attr in request.nodes(data=True): 72 | mapping[node] = self._pick_random(container, 73 | attr[stitcher.TYPE_ATTR]) 74 | logging.info('Initial random stitching: %s.', mapping) 75 | 76 | # start the solving process. 77 | res = self._solve(container, request, conditions, mapping) 78 | if res >= 0: 79 | logging.info('Found solution in %s iterations: %s.', res, mapping) 80 | tmp_graph = nx.union(container, request) 81 | for item in mapping: 82 | tmp_graph.add_edge(item, mapping[item]) 83 | return [tmp_graph] 84 | logging.error('Could not find a solution in %s steps.', self.steps) 85 | return [] 86 | 87 | def _solve(self, container, request, conditions, mapping): 88 | """ 89 | The actual iterative repair algorithm. 90 | """ 91 | for i in range(self.steps): 92 | logging.debug('Iteration: %s.', i) 93 | conflicts = self.find_conflicts(container, request, conditions, 94 | mapping) 95 | if not conflicts: 96 | return i 97 | logging.debug('Found %s conflict(s) in current stitch %s - %s', 98 | len(conflicts), mapping, conflicts) 99 | conflict = self.next_conflict(conflicts) 100 | logging.debug('Will try to fix conflict: %s.', conflict) 101 | self.fix_conflict(conflict, container, request, mapping) 102 | return -1 103 | 104 | def find_conflicts(self, container, request, conditions, mapping): 105 | """ 106 | Find the conflicts in a given mapping between request & container. 107 | 108 | Return None if no conflicts can be found. 109 | 110 | Overwrite this routine if you need additional conflict resolution 111 | mechanism - a sub optimal mapping (e.g. based on the rank) could also 112 | be seen as a conflict. 113 | """ 114 | res = [] 115 | for node in request.nodes(): 116 | if node in conditions: 117 | for condy in conditions[node]: 118 | tmp = _check_condition(node, condy, container, mapping) 119 | if tmp: 120 | res.append(tmp) 121 | return res 122 | 123 | def next_conflict(self, conflicts): 124 | """ 125 | Given a set of conflicts pick the conflict to solve next. 126 | 127 | Overwrite this routine if you prefer another ordering than the 128 | default. For example based on a rank of a sub optimal mapping. 129 | """ 130 | return random.choice(conflicts) 131 | 132 | def fix_conflict(self, conflict, container, request, mapping): 133 | """ 134 | Fix a given conflict. 135 | 136 | Overwrite this routine with your own optimized fixer if needed. 137 | """ 138 | # TODO: write solving routines for conditions that are available. 139 | tmp = self._pick_random(container, 140 | request.nodes[conflict[0]][stitcher.TYPE_ATTR]) 141 | mapping[conflict[0]] = tmp 142 | 143 | def _pick_random(self, container, req_node_type): 144 | """ 145 | Randomly pick a node in container for a node in req. 146 | """ 147 | i = 30 148 | while i >= 0: 149 | cand_name, attrs = random.choice(list(container.nodes(data=True))) 150 | if attrs[stitcher.TYPE_ATTR] == self.rels[req_node_type]: 151 | return cand_name 152 | i -= 1 153 | logging.warning('Checking if there is a node that matches...') 154 | for node, attr in container.nodes(data=True): 155 | if attr[stitcher.TYPE_ATTR] == self.rels[req_node_type]: 156 | return node 157 | raise Exception('No node in the container has the required type ' 158 | '%s.' % self.rels[req_node_type]) 159 | 160 | 161 | def _check_condition(node, condy, container, mapping): 162 | res = _check_attributes(node, condy, container, mapping) 163 | if not res: 164 | res = _check_compositions(node, condy, container, mapping) 165 | return res 166 | 167 | 168 | def _check_attributes(node, condy, container, mapping): 169 | cond = condy[0] 170 | attrs = condy[1] 171 | # attributes 172 | if cond == 'eq': 173 | if attrs[0] not in container.nodes[mapping[node]] or \ 174 | container.nodes[mapping[node]][attrs[0]] != attrs[1]: 175 | return node, condy 176 | elif cond == 'neq': 177 | if attrs[0] in container.nodes[mapping[node]] and \ 178 | container.nodes[mapping[node]][attrs[0]] == attrs[1]: 179 | return node, condy 180 | elif cond == 'lt': 181 | if attrs[0] not in container.nodes[mapping[node]] or \ 182 | container.nodes[mapping[node]][attrs[0]] > attrs[1]: 183 | return node, condy 184 | elif cond == 'gt': 185 | if attrs[0] not in container.nodes[mapping[node]] or \ 186 | container.nodes[mapping[node]][attrs[0]] < attrs[1]: 187 | return node, condy 188 | elif cond == 'regex': 189 | if attrs[0] not in container.nodes[mapping[node]] or \ 190 | not re.search(attrs[1], 191 | container.nodes[mapping[node]][attrs[0]]): 192 | return node, condy 193 | return None 194 | 195 | 196 | def _check_compositions(node, condy, container, mapping): 197 | cond = condy[0] 198 | attrs = condy[1] 199 | # compositions 200 | if cond == 'same': 201 | if mapping[node] != mapping[attrs]: 202 | return node, condy 203 | elif cond == 'diff': 204 | if mapping[node] == mapping[attrs]: 205 | return node, condy 206 | elif cond == 'share': 207 | attr_name = attrs[0] 208 | for ngbh in attrs[1]: 209 | if attr_name not in container.nodes[mapping[ngbh]] or \ 210 | container.nodes[mapping[ngbh]][attr_name] != \ 211 | container.nodes[mapping[node]][attr_name]: 212 | return node, condy 213 | elif cond == 'nshare': 214 | attr_name = attrs[0] 215 | for ngbh in attrs[1]: 216 | if attr_name not in container.nodes[mapping[ngbh]] or \ 217 | container.nodes[mapping[ngbh]][attr_name] == \ 218 | container.nodes[mapping[node]][attr_name]: 219 | return node, condy 220 | return None 221 | -------------------------------------------------------------------------------- /stitcher/stitch.py: -------------------------------------------------------------------------------- 1 | """ 2 | Implements stitching, validation and filtering functions. 3 | """ 4 | 5 | import copy 6 | import itertools 7 | import re 8 | 9 | import networkx as nx 10 | 11 | import stitcher 12 | 13 | 14 | def _find_nodes(graph, tzpe): 15 | """ 16 | Find nodes of a certain type in a graph. 17 | """ 18 | res = [] 19 | for node, values in graph.nodes(data=True): 20 | if values[stitcher.TYPE_ATTR] == tzpe: 21 | res.append(node) 22 | return res 23 | 24 | 25 | def my_filter(container, edge_list, conditions): 26 | """ 27 | Default filter which scans the candidate edges and eliminates those 28 | candidates which do not adhere the conditions. Following conditions are 29 | available: 30 | 31 | attributes - the target node e.g. needs to have a certain attribute 32 | composition - two nodes need to e.g. have same/different/shared attributes 33 | for the target nodes. 34 | 35 | :param edge_list: the candidate edge list as created by stitch(). 36 | :param conditions: dictionary containing the conditions. 37 | :return: The filtered dict with possible edges. 38 | """ 39 | if conditions is None: 40 | return edge_list 41 | if 'attributes' in conditions: 42 | for condition in conditions['attributes']: 43 | cond = condition[0] 44 | para1 = condition[1][0] 45 | para2 = condition[1][1] 46 | if cond == 'eq': 47 | _eq_attr_filter(container, para1, para2, edge_list) 48 | elif cond == 'neq': 49 | _neq_attr_filter(container, para1, para2, edge_list) 50 | elif cond == 'lg': 51 | _lg_attr_filter(container, para1, para2, edge_list) 52 | elif cond == 'lt': 53 | _lt_attr_filter(container, para1, para2, edge_list) 54 | elif cond == 'regex': 55 | _regex_attr_filter(container, para1, para2, edge_list) 56 | if 'compositions' in conditions: 57 | for condition in conditions['compositions']: 58 | cond = condition[0] 59 | para1 = condition[1][0] 60 | para2 = condition[1][1] 61 | if cond == 'same': 62 | _same_filter(para1, para2, edge_list) 63 | elif cond == 'diff': 64 | _diff_filter(para1, para2, edge_list) 65 | elif cond == 'share': 66 | _share_attr(container, para1, para2, edge_list) 67 | elif cond == 'nshare': 68 | _nshare_attr(container, para1, para2, edge_list) 69 | return edge_list 70 | 71 | 72 | def _eq_attr_filter(container, node, condition, candidate_list): 73 | """ 74 | Filter on attributes needed on target node. 75 | """ 76 | for candidate in list(candidate_list.keys()): 77 | attrn = condition[0] 78 | attrv = condition[1] 79 | for src, trg in candidate_list[candidate]: 80 | if src == node and attrn not in container.nodes[trg]: 81 | candidate_list.pop(candidate) 82 | break 83 | if src == node and attrn in container.nodes[trg] \ 84 | and attrv != container.nodes[trg][attrn]: 85 | candidate_list.pop(candidate) 86 | break 87 | 88 | 89 | def _neq_attr_filter(container, node, condition, candidate_list): 90 | """ 91 | Filter on attributes unequal to requested value. 92 | """ 93 | for candidate in list(candidate_list.keys()): 94 | attrn = condition[0] 95 | attrv = condition[1] 96 | for src, trg in candidate_list[candidate]: 97 | if src == node and attrn in container.nodes[trg] \ 98 | and attrv == container.nodes[trg][attrn]: 99 | candidate_list.pop(candidate) 100 | break 101 | 102 | 103 | def _lg_attr_filter(container, node, condition, candidate_list): 104 | """ 105 | Filter on attributes larger than requested value. 106 | """ 107 | for candidate in list(candidate_list.keys()): 108 | attrn = condition[0] 109 | attrv = condition[1] 110 | for src, trg in candidate_list[candidate]: 111 | if src == node and attrn not in container.nodes[trg]: 112 | candidate_list.pop(candidate) 113 | break 114 | if src == node and attrn in container.nodes[trg] \ 115 | and container.nodes[trg][attrn] <= attrv: 116 | candidate_list.pop(candidate) 117 | break 118 | 119 | 120 | def _lt_attr_filter(container, node, condition, candidate_list): 121 | """ 122 | Filter on attributes less than requested value. 123 | """ 124 | for candidate in list(candidate_list.keys()): 125 | attrn = condition[0] 126 | attrv = condition[1] 127 | for src, trg in candidate_list[candidate]: 128 | if src == node and attrn not in container.nodes[trg]: 129 | candidate_list.pop(candidate) 130 | break 131 | if src == node and attrn in container.nodes[trg] \ 132 | and attrv < container.nodes[trg][attrn]: 133 | candidate_list.pop(candidate) 134 | break 135 | 136 | 137 | def _regex_attr_filter(container, node, condition, candidate_list): 138 | """ 139 | Filter on attributes which match an regex. 140 | """ 141 | for candidate in list(candidate_list.keys()): 142 | attrn = condition[0] 143 | regex = condition[1] 144 | for src, trg in candidate_list[candidate]: 145 | if src == node and attrn not in container.nodes[trg]: 146 | candidate_list.pop(candidate) 147 | break 148 | if src == node and attrn in container.nodes[trg] \ 149 | and not re.search(regex, container.nodes[trg][attrn]): 150 | candidate_list.pop(candidate) 151 | break 152 | 153 | 154 | def _same_filter(node1, node2, candidate_list): 155 | """ 156 | Filter out candidates which do not adhere the same target composition 157 | request. 158 | """ 159 | for candidate in list(candidate_list.keys()): 160 | n1_trg = '' 161 | n2_trg = '' 162 | for src, trg in candidate_list[candidate]: 163 | if n1_trg == '' and src == node1: 164 | n1_trg = trg 165 | elif n1_trg != '' and src == node2: 166 | if trg != n1_trg: 167 | candidate_list.pop(candidate) 168 | break 169 | if n2_trg == '' and src == node2: 170 | n2_trg = trg 171 | elif n2_trg != '' and src == node1: 172 | if trg != n2_trg: 173 | candidate_list.pop(candidate) 174 | break 175 | 176 | 177 | def _diff_filter(node1, node2, candidate_list): 178 | """ 179 | Filter out candidates which do not adhere the different target composition 180 | request. 181 | """ 182 | for candidate in list(candidate_list.keys()): 183 | n1_trg = '' 184 | n2_trg = '' 185 | for src, trg in candidate_list[candidate]: 186 | if n1_trg == '' and src == node1: 187 | n1_trg = trg 188 | elif n1_trg != '' and src == node2: 189 | if trg == n1_trg: 190 | candidate_list.pop(candidate) 191 | break 192 | if n2_trg == '' and src == node2: 193 | n2_trg = trg 194 | elif n2_trg != '' and src == node1: 195 | if trg == n2_trg: 196 | candidate_list.pop(candidate) 197 | break 198 | 199 | 200 | def _share_attr(container, attrn, nlist, candidate_list): 201 | """ 202 | Filter out candidates which do not adhere the request that all target nodes 203 | stitched to in the nlist share the same attribute value for a given 204 | attribute name. 205 | """ 206 | for candidate in list(candidate_list.keys()): 207 | attrv = '' 208 | for src, trg in candidate_list[candidate]: 209 | if attrn not in container.nodes[trg]: 210 | candidate_list.pop(candidate) 211 | break 212 | elif src in nlist and attrv == '': 213 | attrv = container.nodes[trg][attrn] 214 | elif src in nlist and container.nodes[trg][attrn] != attrv: 215 | candidate_list.pop(candidate) 216 | break 217 | 218 | 219 | def _nshare_attr(container, attrn, nlist, candidate_list): 220 | """ 221 | Filter out candidates which do not adhere the request that all target nodes 222 | stitched to in the nlist do not share the same attribute value for a given 223 | attribute name. 224 | """ 225 | for candidate in list(candidate_list.keys()): 226 | attrv = '' 227 | for src, trg in candidate_list[candidate]: 228 | if attrn not in container.nodes[trg]: 229 | candidate_list.pop(candidate) 230 | break 231 | elif src in nlist and attrv == '': 232 | attrv = container.nodes[trg][attrn] 233 | elif src in nlist and container.nodes[trg][attrn] == attrv: 234 | candidate_list.pop(candidate) 235 | break 236 | 237 | 238 | class GlobalStitcher(stitcher.Stitcher): 239 | """ 240 | Base stitcher with the functions which need to be implemented. 241 | """ 242 | 243 | def stitch(self, container, request, conditions=None, 244 | candidate_filter=my_filter): 245 | """ 246 | Stitch a request graph into an existing graph container. Returns a set 247 | of possible options. 248 | 249 | :param container: A graph describing the existing container with 250 | ranks. 251 | :param request: A graph describing the request. 252 | :param conditions: Dictionary with conditions - e.g. node a & b need 253 | to be related to node c. 254 | :param candidate_filter: Function which allows for filtering useless 255 | options upfront. 256 | :return: The resulting graphs(s). 257 | """ 258 | res = [] 259 | # TODO: optimize this using concurrency & parallelism 260 | 261 | # 1. find possible mappings 262 | tmp = {} 263 | for node, attr in request.nodes(data=True): 264 | if attr[stitcher.TYPE_ATTR] in self.rels: 265 | candidates = _find_nodes(container, 266 | self.rels[attr[stitcher.TYPE_ATTR]]) 267 | for candidate in candidates: 268 | if node not in tmp: 269 | tmp[node] = [candidate] 270 | else: 271 | tmp[node].append(candidate) 272 | 273 | # 2. find candidates 274 | # dictionary so we have hashed keys (--> speed) 275 | candidate_edges = {} 276 | keys = list(tmp.keys()) 277 | per = [tmp[key] for key in keys] 278 | 279 | for edge_list in itertools.product(*per): 280 | j = 0 281 | edges = [] 282 | for item in edge_list: 283 | edges.append((keys[j], item)) 284 | j += 1 285 | if edges: 286 | candidate_edges[str(edges)] = edges 287 | 288 | # 3. (optional step): filter 289 | candidate_edges = candidate_filter(container, candidate_edges, 290 | conditions) 291 | 292 | # 4. create candidate containers 293 | tmp_graph = nx.union(container, request) 294 | for item in list(candidate_edges.values()): 295 | candidate_graph = copy.deepcopy(tmp_graph) # faster graph copy 296 | # candidate_graph = tmp_graph.copy() 297 | for src, trg in item: 298 | candidate_graph.add_edge(src, trg) 299 | res.append(candidate_graph) 300 | return res 301 | -------------------------------------------------------------------------------- /stitcher/validators.py: -------------------------------------------------------------------------------- 1 | """ 2 | contains validation routines. 3 | """ 4 | 5 | import stitcher 6 | 7 | 8 | def validate_incoming_edges(graphs, param=None): 9 | """ 10 | In case a node of a certain type has more then a threshold of incoming 11 | edges determine a possible stitches as a bad stitch. 12 | """ 13 | param = param or {} 14 | res = {} 15 | i = 0 16 | for candidate in graphs: 17 | res[i] = 'ok' 18 | for node, values in candidate.nodes(data=True): 19 | if values[stitcher.TYPE_ATTR] not in list(param.keys()): 20 | continue 21 | tmp = param[values[stitcher.TYPE_ATTR]] 22 | if len(candidate.in_edges(node)) >= tmp: 23 | res[i] = 'node ' + str(node) + ' has to many edges: ' + \ 24 | str(len(candidate.in_edges(node))) 25 | i += 1 26 | return res 27 | 28 | 29 | def validate_incoming_rank(graphs, param=None): 30 | """ 31 | In case a rank of a node and # of incoming edges increases determine 32 | possible stitches as a bad stitch. 33 | """ 34 | param = param or {} 35 | res = {} 36 | i = 0 37 | for candidate in graphs: 38 | res[i] = 'ok' 39 | for node, values in candidate.nodes(data=True): 40 | if values[stitcher.TYPE_ATTR] not in list(param.keys()): 41 | continue 42 | tmp = param[values[stitcher.TYPE_ATTR]] 43 | if len(candidate.in_edges(node)) > tmp[0] \ 44 | and values['rank'] >= tmp[1]: 45 | res[i] = 'node ' + str(node) + ' rank is >= ' + \ 46 | str(tmp[1]) + ' and # incoming edges is > ' \ 47 | + str(tmp[0]) 48 | i += 1 49 | return res 50 | -------------------------------------------------------------------------------- /stitcher/vis.py: -------------------------------------------------------------------------------- 1 | 2 | """ 3 | Visualize possible stitches with the outcome of the validator. 4 | """ 5 | 6 | import math 7 | import random 8 | import matplotlib.pyplot as plt 9 | import networkx as nx 10 | import numpy as np 11 | 12 | from mpl_toolkits.mplot3d import Axes3D 13 | 14 | import stitcher 15 | 16 | SPACE = 25 17 | TYPE_FORMAT = {'a': '^', 'b': 's', 'c': 'v'} 18 | 19 | 20 | def show(graphs, request, titles, prog='neato', size=None, 21 | type_format=None, filename=None): 22 | """ 23 | Display the results using matplotlib. 24 | """ 25 | if not size: 26 | size = _get_size(len(graphs)) 27 | fig, axarr = plt.subplots(size[0], size[1], figsize=(18, 10)) 28 | fig.set_facecolor('white') 29 | x_val = 0 30 | y_val = 0 31 | index = 0 32 | 33 | if size[0] == 1: 34 | axarr = np.array(axarr).reshape((1, size[1])) 35 | 36 | for candidate in graphs: 37 | # axarr[x_val, y_val].axis('off') 38 | axarr[x_val, y_val].xaxis.set_major_formatter(plt.NullFormatter()) 39 | axarr[x_val, y_val].yaxis.set_major_formatter(plt.NullFormatter()) 40 | axarr[x_val, y_val].xaxis.set_ticks([]) 41 | axarr[x_val, y_val].yaxis.set_ticks([]) 42 | axarr[x_val, y_val].set_title(titles[index]) 43 | # axarr[x_val, y_val].set_axis_bgcolor("white") 44 | if not type_format: 45 | type_format = TYPE_FORMAT 46 | _plot_subplot(candidate, request.nodes(), prog, type_format, 47 | axarr[x_val, y_val]) 48 | y_val += 1 49 | if y_val > size[1] - 1: 50 | y_val = 0 51 | x_val += 1 52 | index += 1 53 | fig.tight_layout() 54 | if filename is not None: 55 | plt.savefig(filename) 56 | else: 57 | plt.show() 58 | plt.close() 59 | 60 | 61 | def _plot_subplot(graph, new_nodes, prog, type_format, axes): 62 | """ 63 | Plot a single candidate graph. 64 | """ 65 | pos = nx.nx_agraph.graphviz_layout(graph, prog=prog) 66 | 67 | # draw the nodes 68 | for node, values in graph.nodes(data=True): 69 | shape = 'o' 70 | if values[stitcher.TYPE_ATTR] in type_format: 71 | shape = type_format[values[stitcher.TYPE_ATTR]] 72 | color = 'g' 73 | alpha = 0.8 74 | if node in new_nodes: 75 | color = 'b' 76 | alpha = 0.2 77 | elif 'rank' in values and values['rank'] > 7: 78 | color = 'r' 79 | elif 'rank' in values and values['rank'] < 7 and values['rank'] > 3: 80 | color = 'y' 81 | nx.draw_networkx_nodes(graph, pos, nodelist=[node], node_color=color, 82 | node_shape=shape, alpha=alpha, ax=axes) 83 | 84 | # draw the edges 85 | dotted_line = [] 86 | normal_line = [] 87 | for src, trg in graph.edges(): 88 | if src in new_nodes and trg not in new_nodes: 89 | dotted_line.append((src, trg)) 90 | else: 91 | normal_line.append((src, trg)) 92 | nx.draw_networkx_edges(graph, pos, edgelist=dotted_line, style='dotted', 93 | ax=axes) 94 | nx.draw_networkx_edges(graph, pos, edgelist=normal_line, ax=axes) 95 | 96 | # draw labels 97 | nx.draw_networkx_labels(graph, pos, ax=axes) 98 | 99 | 100 | def show_3d(graphs, request, titles, prog='neato', filename=None): 101 | """ 102 | Show the candidates in 3d - the request elevated above the container. 103 | """ 104 | fig = plt.figure(figsize=(18, 10)) 105 | fig.set_facecolor('white') 106 | i = 0 107 | 108 | size = _get_size(len(graphs)) 109 | 110 | for graph in graphs: 111 | axes = fig.add_subplot(size[0], size[1], i+1, 112 | projection=Axes3D.name) 113 | axes.set_title(titles[i]) 114 | axes._axis3don = False 115 | 116 | _plot_3d_subplot(graph, request, prog, axes) 117 | 118 | i += 1 119 | fig.tight_layout() 120 | if filename is not None: 121 | plt.savefig(filename) 122 | else: 123 | plt.show() 124 | plt.close() 125 | 126 | 127 | def _plot_3d_subplot(graph, request, prog, axes): 128 | """ 129 | Plot a single candidate graph in 3d. 130 | """ 131 | cache = {} 132 | 133 | tmp = graph.copy() 134 | for node in request.nodes(): 135 | tmp.remove_node(node) 136 | 137 | pos = nx.nx_agraph.graphviz_layout(tmp, prog=prog) 138 | 139 | # the container 140 | for item in tmp.nodes(): 141 | axes.plot([pos[item][0]], [pos[item][1]], [0], linestyle="None", 142 | marker="o", color='gray') 143 | axes.text(pos[item][0], pos[item][1], 0, item) 144 | 145 | for src, trg in tmp.edges(): 146 | axes.plot([pos[src][0], pos[trg][0]], 147 | [pos[src][1], pos[trg][1]], 148 | [0, 0], color='gray') 149 | 150 | # the new nodes 151 | for item in graph.nodes(): 152 | if item in request.nodes(): 153 | for nghb in graph.neighbors(item): 154 | if nghb in tmp.nodes(): 155 | x_val = pos[nghb][0] 156 | y_val = pos[nghb][1] 157 | if (x_val, y_val) in list(cache.values()): 158 | x_val = pos[nghb][0] + random.randint(10, SPACE) 159 | y_val = pos[nghb][0] + random.randint(10, SPACE) 160 | cache[item] = (x_val, y_val) 161 | 162 | # edge 163 | axes.plot([x_val, pos[nghb][0]], 164 | [y_val, pos[nghb][1]], 165 | [SPACE, 0], color='blue') 166 | 167 | axes.plot([x_val], [y_val], [SPACE], linestyle="None", marker="o", 168 | color='blue') 169 | axes.text(x_val, y_val, SPACE, item) 170 | 171 | for src, trg in request.edges(): 172 | if trg in cache and src in cache: 173 | axes.plot([cache[src][0], cache[trg][0]], 174 | [cache[src][1], cache[trg][1]], 175 | [SPACE, SPACE], color='blue') 176 | 177 | 178 | def _get_size(n_items): 179 | """ 180 | Calculate the size of the subplot layouts based on number of items. 181 | """ 182 | n_cols = math.ceil(math.sqrt(n_items)) 183 | n_rows = math.floor(math.sqrt(n_items)) 184 | if n_cols * n_rows < n_items: 185 | n_cols += 1 186 | return int(n_rows), int(n_cols) 187 | -------------------------------------------------------------------------------- /tests/__init__.py: -------------------------------------------------------------------------------- 1 | """ 2 | Unittests for the graph stitcher 3 | """ 4 | -------------------------------------------------------------------------------- /tests/stitcher_bidding_test.py: -------------------------------------------------------------------------------- 1 | """ 2 | Unittest for the bidding module. 3 | """ 4 | 5 | import logging 6 | import unittest 7 | 8 | import networkx as nx 9 | 10 | from stitcher import bidding 11 | 12 | FORMAT = "%(asctime)s - %(filename)s - %(lineno)s - " \ 13 | "%(levelname)s - %(message)s" 14 | logging.basicConfig(format=FORMAT, level=logging.DEBUG) 15 | 16 | 17 | class EntityTest(unittest.TestCase): 18 | """ 19 | Testcase for the Entity class. 20 | """ 21 | 22 | def setUp(self): 23 | self.map = {'type_a': 'type_x', 24 | 'type_b': 'type_y'} 25 | 26 | self.request = nx.DiGraph() 27 | self.request.add_node('a', **{'type': 'type_a'}) 28 | self.request.add_node('b', **{'type': 'type_b'}) 29 | self.request.add_node('c', **{'type': 'type_b'}) 30 | self.request.add_edge('a', 'b') 31 | self.request.add_edge('b', 'c') 32 | 33 | self.container = nx.DiGraph() 34 | self.x_attr = {'type': 'type_x', 35 | 'foo': 3, 36 | 'group_1': 'group_a', 37 | 'group_2': 'group_b'} 38 | self.y_attr = {'type': 'type_y', 39 | 'foo': 5, 40 | 'group_1': 'group_a', 41 | 'group_2': 'group_c'} 42 | 43 | def test_trigger_for_success(self): 44 | """ 45 | Test trigger for success. 46 | """ 47 | cut = bidding.Entity('x', self.map, self.request, self.container) 48 | self.container.add_node(cut, **self.x_attr) 49 | cut.trigger({'assigned': {}, 'bids': []}, 'init') 50 | 51 | def test_trigger_for_sanity(self): 52 | """ 53 | Test trigger for failure. 54 | """ 55 | # test calculation of credits to bid. 56 | # no conditions 57 | cut = bidding.Entity('x', self.map, self.request, self.container) 58 | self.container.add_node(cut, **self.x_attr) 59 | res, bids = cut.trigger({'assigned': {}, 'bids': {}}, 'init') 60 | self.assertEqual(res['a'][1], 1.0) # 1 because type matches. 61 | self.assertNotIn('b', bids) # a should not bid on b. 62 | 63 | # lg 64 | self.container = nx.DiGraph() 65 | condy = {'attributes': [('lg', ('a', ('foo', 1)))]} 66 | cut = bidding.Entity('x', self.map, self.request, self.container, 67 | condy) 68 | self.container.add_node(cut, **self.x_attr) 69 | res, bids = cut.trigger({'assigned': {}, 'bids': []}, 'init') 70 | self.assertEqual(res['a'][1], 3.0) # rank 3 - rank 1 + 1 for type = 2 71 | 72 | # lt 73 | self.container = nx.DiGraph() 74 | condy = {'attributes': [('lt', ('a', ('foo', 9)))]} 75 | cut = bidding.Entity('x', self.map, self.request, self.container, 76 | condy) 77 | self.container.add_node(cut, **self.x_attr) 78 | res, bids = cut.trigger({'assigned': {}, 'bids': []}, 'init') 79 | self.assertEqual(res['a'][1], 7.0) # 9-3+1=7 80 | 81 | # eq 82 | self.container = nx.DiGraph() 83 | condy = {'attributes': [('eq', ('a', ('foo', 2)))]} 84 | cut = bidding.Entity('x', self.map, self.request, self.container, 85 | condy) 86 | self.container.add_node(cut, **self.x_attr) 87 | res, _ = cut.trigger({'assigned': {}, 'bids': []}, 'init') 88 | self.assertNotIn('a', bids) # popped because not equal. 89 | 90 | # neq 91 | self.container = nx.DiGraph() 92 | condy = {'attributes': [('neq', ('a', ('group_1', 'group_a')))]} 93 | cut = bidding.Entity('x', self.map, self.request, self.container, 94 | condy) 95 | self.container.add_node(cut, **self.x_attr) 96 | res, _ = cut.trigger({'assigned': {}, 'bids': []}, 'init') 97 | self.assertNotIn('a', bids) # popped because equal. 98 | 99 | # regex 100 | self.container = nx.DiGraph() 101 | condy = {'attributes': [('regex', ('a', ('group_1', '^z')))]} 102 | cut = bidding.Entity('x', self.map, self.request, self.container, 103 | condy) 104 | self.container.add_node(cut, **self.x_attr) 105 | res, _ = cut.trigger({'assigned': {}, 'bids': []}, 'init') 106 | self.assertNotIn('a', bids) # popped because z not in group name. 107 | 108 | # same 109 | # 1. nothing assigned -> increase bit for both (50%) 110 | self.container = nx.DiGraph() 111 | condy = {'compositions': [('same', ['b', 'c'])]} 112 | cut = bidding.Entity('y', self.map, self.request, self.container, 113 | condy) 114 | self.container.add_node(cut, **self.y_attr) 115 | res, bids = cut.trigger({'assigned': {}, 'bids': []}, 'init') 116 | self.assertEqual(res['b'][1], 1.5) # 1 + 0.5 increase 117 | self.assertEqual(res['c'][1], 1.5) # 1 + 0.5 increase 118 | 119 | # 1. nothing assigned -> increase bit for both (50%) 120 | self.container = nx.DiGraph() 121 | condy = {'compositions': [('same', ['a', 'c'])]} 122 | cut = bidding.Entity('y', self.map, self.request, self.container, 123 | condy) 124 | self.container.add_node(cut, **self.y_attr) 125 | res, bids = cut.trigger({'assigned': {}, 'bids': []}, 'init') 126 | self.assertTrue('a' not in bids['y']) # not bid on a 127 | self.assertTrue('c' not in bids['y']) # not bid on c 128 | 129 | # 2. a,b, assigned to sth else -> nothing 130 | # so e.g. only better attribute matches can increase the need for it 131 | # to bid higher - and eventually stuff getting assigned here. 132 | 133 | # diff 134 | # 1. nothing assigned -> drop one bid 135 | self.container = nx.DiGraph() 136 | condy = {'compositions': [('diff', ['b', 'c'])]} 137 | cut = bidding.Entity('y', self.map, self.request, self.container, 138 | condy) 139 | self.container.add_node(cut, **self.y_attr) 140 | res, bids = cut.trigger({'assigned': {}, 'bids': []}, 'init') 141 | if 'c' in res: 142 | self.assertNotIn('b', bids['y']) # drop other bid. 143 | elif 'b' in res: 144 | self.assertNotIn('c', bids['y']) # drop other bid. 145 | 146 | # 2. a assigned, b no bid -> increase bid for b (50%) 147 | self.container = nx.DiGraph() 148 | condy = {'compositions': [('diff', ['a', 'c'])]} 149 | cut_x = bidding.Entity('x', self.map, self.request, self.container, 150 | condy) 151 | cut_y = bidding.Entity('y', self.map, self.request, self.container, 152 | condy) 153 | self.container.add_node(cut_x, **self.x_attr) 154 | self.container.add_node(cut_y, **self.y_attr) 155 | self.container.add_edge(cut_x, cut_y) 156 | res, bids = cut_x.trigger({'assigned': {}, 157 | 'bids': []}, 'init') 158 | self.assertEqual(res['a'][0], 'x') 159 | self.assertEqual(res['c'][0], 'y') 160 | self.assertEqual(res['c'][1], 1.5) # 1 (type) + 0.5 increase 161 | 162 | # 3. a,b, assigned to sth else -> if neighbour bids too - 163 | # increase bid a bit (25%) 164 | self.container = nx.DiGraph() 165 | condy = {'compositions': [('diff', ['a', 'c'])]} 166 | cut_x = bidding.Entity('x', self.map, self.request, self.container, 167 | condy) 168 | cut_y = bidding.Entity('y', self.map, self.request, self.container, 169 | condy) 170 | self.container.add_node(cut_x, **self.x_attr) 171 | self.container.add_node(cut_y, **self.y_attr) 172 | self.container.add_edge(cut_x, cut_y) 173 | res, bids = cut_x.trigger({'assigned': {'a': ('k', 10.0), 174 | 'c': ('l', 10.0)}, 175 | 'bids': []}, 'init') 176 | self.assertEqual(bids['y']['c'], 1.25) # y bid on c + 0.25 177 | self.assertEqual(bids['x']['a'], 1.25) # x bid on a + 0.25 178 | 179 | # share 180 | # 1. a assigned to x, I share attr with x -> increase bid (50%) 181 | self.container = nx.DiGraph() 182 | condy = {'compositions': [('share', ('group_1', ['a', 'b']))]} 183 | cut_x = bidding.Entity('x', self.map, self.request, self.container, 184 | condy) 185 | cut_y = bidding.Entity('y', self.map, self.request, self.container, 186 | condy) 187 | self.container.add_node(cut_x, **self.x_attr) 188 | self.container.add_node(cut_y, **self.y_attr) 189 | self.container.add_edge(cut_x, cut_y) 190 | res, bids = cut_x.trigger({'assigned': {}, 'bids': []}, 'init') 191 | self.assertEqual(res['b'][1], 1.5) # +50% b/c group complete. 192 | self.assertEqual(res['b'][0], 'y') # b should be stitched to y 193 | self.assertEqual(res['a'][1], 1.5) # +50% b/c group complete. 194 | self.assertEqual(res['a'][0], 'x') # a should be stitched to x 195 | 196 | # nshare 197 | # 1. a assigned, b no bid -> increase bid for b (50%) 198 | self.container = nx.DiGraph() 199 | condy = {'compositions': [('nshare', ('group_2', ['a', 'b']))]} 200 | cut_x = bidding.Entity('x', self.map, self.request, self.container, 201 | condy) 202 | cut_y = bidding.Entity('y', self.map, self.request, self.container, 203 | condy) 204 | self.container.add_node(cut_x, **self.x_attr) 205 | self.container.add_node(cut_y, **self.y_attr) 206 | self.container.add_edge(cut_x, cut_y) 207 | self.container.add_node(cut, **self.x_attr) 208 | res, bids = cut_x.trigger({'assigned': {}, 'bids': []}, 'init') 209 | self.assertEqual(res['b'][1], 1.5) # 1 + 0.5 increase 210 | self.assertEqual(res['b'][0], 'y') # b should be stitched to y 211 | self.assertEqual(res['a'][1], 1.0) # a sits fine on x 212 | self.assertEqual(res['a'][0], 'x') # a should be stitched to x 213 | 214 | 215 | class BiddingStitcherTest(unittest.TestCase): 216 | """ 217 | Testcase for the BiddingStitcher class. 218 | """ 219 | 220 | def setUp(self): 221 | rels = {'type_x': 'type_a', 222 | 'type_y': 'type_b', 223 | 'type_z': 'type_c'} 224 | self.cut = bidding.BiddingStitcher(rels) 225 | 226 | self.container = nx.DiGraph() 227 | self.container.add_node('A', **{'type': 'type_a'}) 228 | self.container.add_node('B', **{'type': 'type_b'}) 229 | self.container.add_node('C', **{'type': 'type_c'}) 230 | self.container.add_edge('A', 'B') 231 | self.container.add_edge('B', 'C') 232 | 233 | self.request = nx.DiGraph() 234 | self.request.add_node('X', **{'type': 'type_x'}) 235 | self.request.add_node('Y', **{'type': 'type_y'}) 236 | self.request.add_node('Z', **{'type': 'type_y'}) 237 | self.request.add_edge('X', 'Y') 238 | self.request.add_edge('Y', 'Z') 239 | 240 | def test_stitch_for_success(self): 241 | """ 242 | Test stitching for success. 243 | """ 244 | self.cut.stitch(self.container, self.request) 245 | 246 | def test_stitch_for_sanity(self): 247 | """ 248 | Test stitching for sanity 249 | """ 250 | # shouldn't matter were we start 251 | for node in self.container.nodes(): 252 | res = self.cut.stitch(self.container, self.request, start=node) 253 | self.assertIn(('X', 'A'), res[0].edges()) 254 | self.assertIn(('Y', 'B'), res[0].edges()) 255 | self.assertIn(('Z', 'B'), res[0].edges()) 256 | 257 | # cycles shouldn't matter either 258 | self.container.add_edge('C', 'A') # CYCLE! 259 | res = self.cut.stitch(self.container, self.request) 260 | self.assertIn(('X', 'A'), res[0].edges()) 261 | self.assertIn(('Y', 'B'), res[0].edges()) 262 | self.assertIn(('Z', 'B'), res[0].edges()) 263 | 264 | # complex test with two groups. 265 | container = nx.DiGraph() 266 | container.add_node('A', **{'type': 'type_a', 'group': 'a', 'rank': 2}) 267 | container.add_node('B', **{'type': 'type_b', 'group': 'a', 'rank': 3}) 268 | container.add_node('I', **{'type': 'type_b', 'rank': 1}) 269 | # XXX: 'group': 'c' is incomplete should only works because I uplevel 270 | # complete groups. 271 | container.add_node('J', **{'type': 'type_b', 'group': 'c', 'rank': 2}) 272 | container.add_node('X', **{'type': 'type_b', 'group': 'b', 'rank': 2}) 273 | container.add_node('Y', **{'type': 'type_a', 'group': 'b', 'rank': 5}) 274 | container.add_edge('A', 'B') 275 | container.add_edge('B', 'I') 276 | container.add_edge('I', 'J') 277 | container.add_edge('J', 'X') 278 | container.add_edge('X', 'Y') 279 | 280 | request = nx.DiGraph() 281 | request.add_node('1', **{'type': 'type_x'}) 282 | request.add_node('2', **{'type': 'type_y'}) 283 | request.add_edge('1', '2') 284 | 285 | condy = { 286 | 'compositions': [('share', ('group', ['1', '2']))], 287 | 'attributes': [('lg', ('1', ('rank', 2))), 288 | ('lg', ('2', ('rank', 2)))] 289 | } 290 | for node in container.nodes(): 291 | res = self.cut.stitch(container, request, conditions=condy, 292 | start=node) 293 | self.assertIn(('1', 'Y'), res[0].edges()) 294 | self.assertIn(('2', 'X'), res[0].edges()) 295 | -------------------------------------------------------------------------------- /tests/stitcher_evolutionary_test.py: -------------------------------------------------------------------------------- 1 | """ 2 | Unittest for the evolutionary module. 3 | """ 4 | 5 | import itertools 6 | import json 7 | import logging 8 | import unittest 9 | import networkx as nx 10 | 11 | from networkx.readwrite import json_graph 12 | 13 | from stitcher import evolutionary 14 | 15 | FORMAT = "%(asctime)s - %(filename)s - %(lineno)s - " \ 16 | "%(levelname)s - %(message)s" 17 | logging.basicConfig(format=FORMAT, level=logging.DEBUG) 18 | 19 | 20 | class TestCandidate(unittest.TestCase): 21 | """ 22 | Test abstract class. 23 | """ 24 | 25 | def setUp(self): 26 | self.cut = evolutionary.Candidate('a') 27 | 28 | def test_eq_for_success(self): 29 | """ 30 | Test eq for success. 31 | """ 32 | one = evolutionary.Candidate({'a': '1', 'b': '2'}) 33 | another = evolutionary.Candidate({'a': '1', 'b': '2'}) 34 | self.assertTrue(one == another) 35 | 36 | def test_fitness_for_failure(self): 37 | """ 38 | Test not impl error. 39 | """ 40 | self.assertRaises(NotImplementedError, self.cut.fitness) 41 | 42 | def test_mutate_for_failure(self): 43 | """ 44 | Test not impl error. 45 | """ 46 | self.assertRaises(NotImplementedError, self.cut.mutate) 47 | 48 | def test_crossover_for_failure(self): 49 | """ 50 | Test not impl error. 51 | """ 52 | self.assertRaises(NotImplementedError, self.cut.crossover, None) 53 | 54 | def test_eq_for_sanity(self): 55 | """ 56 | Objects are the same when there genomics are identical. 57 | """ 58 | one = evolutionary.Candidate({'a': '1', 'b': '2'}) 59 | another = evolutionary.Candidate({'a': '1', 'b': '2'}) 60 | yet_another = evolutionary.Candidate({'a': '1', 'b': '3'}) 61 | 62 | self.assertTrue(one == another) 63 | self.assertTrue(one != yet_another) 64 | self.assertIn(another, [one]) 65 | self.assertNotIn(yet_another, [one, another]) 66 | 67 | 68 | class TestGraphCandidate(unittest.TestCase): 69 | """ 70 | Tests the graph candidate. 71 | """ 72 | 73 | def setUp(self): 74 | self.container = nx.DiGraph() 75 | self.container.add_node('1', **{'type': 'a', 'group': 'foo', 'foo': 3, 76 | 'retest': 'aaa'}) 77 | self.container.add_node('2', **{'type': 'a', 'group': 'foo'}) 78 | self.container.add_node('3', **{'type': 'a', 'foo': 5, 79 | 'retest': 'bbb'}) 80 | self.container.add_node('4', **{'type': 'b', 'group': 'foo'}) 81 | self.container.add_node('5', **{'type': 'b', 'group': 'bar'}) 82 | self.container.add_node('6', **{'type': 'b', 'group': 'bar'}) 83 | 84 | self.request = nx.DiGraph() 85 | self.request.add_node('a', **{'type': 'x'}) 86 | self.request.add_node('b', **{'type': 'x'}) 87 | self.request.add_node('c', **{'type': 'y'}) 88 | self.request.add_node('d', **{'type': 'y'}) 89 | 90 | self.stitch = {'x': 'a', 'y': 'b'} # stitch x -> a and y -> b 91 | 92 | def test_fitness_for_success(self): 93 | """ 94 | Test fitness function for success. 95 | """ 96 | cut = evolutionary.GraphCandidate({'a': '1', 'c': '4'}, self.stitch, 97 | {}, [], self.request, self.container) 98 | cut.fitness() 99 | repr(cut) 100 | 101 | def test_mutate_for_success(self): 102 | """ 103 | Test mutate function for success. 104 | """ 105 | cut = evolutionary.GraphCandidate({'a': '1', 'c': '4'}, self.stitch, 106 | {}, ['2'], self.request, 107 | self.container) 108 | cut.mutate() 109 | 110 | def test_crossover_for_success(self): 111 | """ 112 | Test crossover function for success. 113 | """ 114 | cut = evolutionary.GraphCandidate({'a': '1', 'c': '4'}, self.stitch, 115 | {}, [], self.request, self.container) 116 | partner = evolutionary.GraphCandidate({'a': '2', 'c': '4'}, 117 | self.stitch, {}, [], 118 | self.request, self.container) 119 | cut.crossover(partner) 120 | 121 | # Test for failures - should not happen. 122 | 123 | def test_fitness_for_sanity(self): 124 | """ 125 | Test fitness function for sanity. 126 | """ 127 | # a should not be stitched to 3! 128 | cut = evolutionary.GraphCandidate({'a': '4'}, self.stitch, {}, [], 129 | self.request, self.container) 130 | self.assertEqual(cut.fitness(), 100.0) 131 | 132 | # a needs to be stitched to target node with attr foo = 3 133 | condy = {'attributes': [('eq', ('a', ('foo', 3)))]} 134 | cut = evolutionary.GraphCandidate({'a': '1'}, self.stitch, condy, [], 135 | self.request, self.container) 136 | self.assertEqual(cut.fitness(), 0.0) 137 | 138 | cut = evolutionary.GraphCandidate({'a': '2'}, self.stitch, condy, [], 139 | self.request, self.container) 140 | self.assertEqual(cut.fitness(), 10.1) 141 | 142 | condy = {'attributes': [('eq', ('a', ('foo', 9)))]} 143 | cut = evolutionary.GraphCandidate({'a': '1'}, self.stitch, condy, [], 144 | self.request, self.container) 145 | self.assertEqual(cut.fitness(), 10.2) 146 | 147 | # a needs to be stitched to target node with attr foo != 3 148 | condy = {'attributes': [('neq', ('a', ('foo', 3)))]} 149 | cut = evolutionary.GraphCandidate({'a': '3'}, self.stitch, condy, [], 150 | self.request, self.container) 151 | self.assertEqual(cut.fitness(), 0.0) 152 | 153 | cut = evolutionary.GraphCandidate({'a': '2'}, self.stitch, condy, [], 154 | self.request, self.container) 155 | self.assertEqual(cut.fitness(), 0.0) 156 | 157 | condy = {'attributes': [('neq', ('a', ('foo', 3)))]} 158 | cut = evolutionary.GraphCandidate({'a': '1'}, self.stitch, condy, [], 159 | self.request, self.container) 160 | self.assertEqual(cut.fitness(), 10.1) 161 | 162 | # a needs to be stitched to target node with attr foo > 4 163 | condy = {'attributes': [('lg', ('a', ('foo', 4)))]} 164 | cut = evolutionary.GraphCandidate({'a': '3'}, self.stitch, condy, [], 165 | self.request, self.container) 166 | self.assertEqual(cut.fitness(), 0.0) 167 | 168 | cut = evolutionary.GraphCandidate({'a': '2'}, self.stitch, condy, [], 169 | self.request, self.container) 170 | self.assertEqual(cut.fitness(), 10.1) 171 | 172 | condy = {'attributes': [('lg', ('a', ('foo', 4)))]} 173 | cut = evolutionary.GraphCandidate({'a': '1'}, self.stitch, condy, [], 174 | self.request, self.container) 175 | self.assertEqual(cut.fitness(), 10.2) 176 | 177 | # a needs to be stitched to target node with attr foo < 4 178 | condy = {'attributes': [('lt', ('a', ('foo', 4)))]} 179 | cut = evolutionary.GraphCandidate({'a': '1'}, self.stitch, condy, [], 180 | self.request, self.container) 181 | self.assertEqual(cut.fitness(), 0.0) 182 | 183 | cut = evolutionary.GraphCandidate({'a': '2'}, self.stitch, condy, [], 184 | self.request, self.container) 185 | self.assertEqual(cut.fitness(), 10.1) 186 | 187 | condy = {'attributes': [('lt', ('a', ('foo', 4)))]} 188 | cut = evolutionary.GraphCandidate({'a': '3'}, self.stitch, condy, [], 189 | self.request, self.container) 190 | self.assertEqual(cut.fitness(), 10.2) 191 | 192 | # node a requires target node to have an attribute retest which starts 193 | # with an 'c' 194 | condy = {'attributes': [('regex', ('a', ('retest', '^b')))]} 195 | cut = evolutionary.GraphCandidate({'a': '2'}, self.stitch, condy, [], 196 | self.request, self.container) 197 | self.assertEqual(cut.fitness(), 10.1) 198 | cut = evolutionary.GraphCandidate({'a': '1'}, self.stitch, condy, [], 199 | self.request, self.container) 200 | self.assertEqual(cut.fitness(), 10.2) 201 | cut = evolutionary.GraphCandidate({'a': '3'}, self.stitch, condy, [], 202 | self.request, self.container) 203 | self.assertEqual(cut.fitness(), 0.0) 204 | 205 | # a and b are stitched to 1 206 | condy = {'compositions': [('share', ('group', ['a', 'b']))]} 207 | cut = evolutionary.GraphCandidate({'a': '1', 'b': '1'}, self.stitch, 208 | condy, [], self.request, 209 | self.container) 210 | self.assertEqual(cut.fitness(), 0.0) 211 | 212 | # node c has no group attr. 213 | cut = evolutionary.GraphCandidate({'a': '1', 'b': '3'}, self.stitch, 214 | condy, [], self.request, 215 | self.container) 216 | self.assertEqual(cut.fitness(), 10.1) 217 | 218 | # c and d stitched to nodes with different group value. 219 | condy = {'compositions': [('share', ('group', ['c', 'd']))]} 220 | cut = evolutionary.GraphCandidate({'c': '4', 'd': '5'}, self.stitch, 221 | condy, [], self.request, 222 | self.container) 223 | self.assertEqual(cut.fitness(), 10.2) 224 | 225 | # a and b are stitched to 1 226 | condy = {'compositions': [('nshare', ('group', ['a', 'b']))]} 227 | cut = evolutionary.GraphCandidate({'a': '1', 'b': '1'}, self.stitch, 228 | condy, [], self.request, 229 | self.container) 230 | self.assertEqual(cut.fitness(), 10.2) 231 | 232 | # node c has no group attr. 233 | cut = evolutionary.GraphCandidate({'a': '1', 'b': '3'}, self.stitch, 234 | condy, [], self.request, 235 | self.container) 236 | self.assertEqual(cut.fitness(), 10.1) 237 | 238 | # c and d stitched to nodes with different group value. 239 | condy = {'compositions': [('nshare', ('group', ['c', 'd']))]} 240 | cut = evolutionary.GraphCandidate({'c': '4', 'd': '5'}, self.stitch, 241 | condy, [], self.request, 242 | self.container) 243 | self.assertEqual(cut.fitness(), 0.0) 244 | 245 | # a and b stitched to same target. 246 | condy = {'compositions': [('same', ['a', 'b'])]} 247 | cut = evolutionary.GraphCandidate({'a': '1', 'b': '1'}, self.stitch, 248 | condy, [], self.request, 249 | self.container) 250 | self.assertEqual(cut.fitness(), 0.0) 251 | 252 | # a and b not stitched to same target. 253 | condy = {'compositions': [('same', ['b', 'a'])]} 254 | cut = evolutionary.GraphCandidate({'a': '1', 'b': '2'}, self.stitch, 255 | condy, [], self.request, 256 | self.container) 257 | self.assertEqual(cut.fitness(), 10.0) 258 | 259 | # a and b not stitched to same target. 260 | condy = {'compositions': [('diff', ['a', 'b'])]} 261 | cut = evolutionary.GraphCandidate({'a': '1', 'b': '2'}, self.stitch, 262 | condy, [], self.request, 263 | self.container) 264 | self.assertEqual(cut.fitness(), 0.0) 265 | 266 | # a and n stitched to same target. 267 | condy = {'compositions': [('diff', ['b', 'a'])]} 268 | cut = evolutionary.GraphCandidate({'a': '1', 'b': '1'}, self.stitch, 269 | condy, [], self.request, 270 | self.container) 271 | self.assertEqual(cut.fitness(), 10.0) 272 | 273 | def test_special(self): 274 | """ 275 | Test share condition. 276 | """ 277 | condy = {'compositions': [('share', ('group', ['a', 'b'])), 278 | ('share', ('group', ['c', 'd']))]} 279 | cut = evolutionary.GraphCandidate({'a': '1', 'b': '1', 280 | 'c': '5', 'd': '6'}, 281 | self.stitch, condy, [], self.request, 282 | self.container) 283 | self.assertEqual(cut.fitness(), 0.0) 284 | 285 | def test_mutate_for_sanity(self): 286 | """ 287 | Test mutate function for sanity. 288 | """ 289 | cut = evolutionary.GraphCandidate({'a': '1'}, self.stitch, 290 | {}, ['2'], self.request, 291 | self.container) 292 | cut.mutate() 293 | # gens should have flipped - hard to test otherwise as random is 294 | # involved. 295 | self.assertDictEqual({'a': '2'}, cut.gen) 296 | 297 | def test_crossover_for_sanity(self): 298 | """ 299 | Test crossover function for sanity. 300 | """ 301 | cut = evolutionary.GraphCandidate({'a': '1'}, self.stitch, 302 | {}, [], self.request, self.container) 303 | partner = evolutionary.GraphCandidate({'a': '2'}, 304 | self.stitch, {}, [], 305 | self.request, self.container) 306 | child = cut.crossover(partner) 307 | # child's gens should have been taken from the partner. - hard to test 308 | # otherwise as random is involved. 309 | self.assertDictEqual(child.gen, partner.gen) 310 | 311 | # partner has a only non valid mappings 312 | self.container.add_node('y', **{'type': 'boo'}) 313 | partner = evolutionary.GraphCandidate({'a': 'y'}, 314 | self.stitch, {}, [], 315 | self.request, self.container) 316 | child = cut.crossover(partner) 317 | self.assertTrue(child == cut) 318 | 319 | 320 | class TestBasicEvolution(unittest.TestCase): 321 | """ 322 | Tests the filter functions and validates that the right candidates are 323 | eliminated. 324 | """ 325 | 326 | def setUp(self): 327 | self.cut = evolutionary.BasicEvolution() 328 | 329 | def test_run_for_success(self): 330 | """ 331 | Test basic evolutionary algorithm usage. 332 | """ 333 | population = _get_population('b') 334 | self.cut.run(population, 1) 335 | 336 | def test_run_for_failure(self): 337 | """ 338 | Test basic evolutionary algorithm usage for failure. 339 | """ 340 | population = _get_population('x') 341 | iterations, _ = self.cut.run(population, 100, stabilizer=True) 342 | self.assertEqual(iterations, 1) # should die in second run 343 | 344 | iterations, _ = self.cut.run(population, 0) 345 | self.assertEqual(iterations, 1) 346 | 347 | def test_run_for_sanity(self): 348 | """ 349 | Test basic evolutionary algorithm usage. 350 | """ 351 | population = _get_population('b') 352 | iteration, _ = self.cut.run(population, 1) 353 | self.assertEqual(iteration, 0) # done as we flip to b immediately. 354 | 355 | 356 | class EvolutionaryStitcherTest(unittest.TestCase): 357 | """ 358 | Testcase for the evolutionary algorithm based stitcher. 359 | """ 360 | 361 | def setUp(self): 362 | container_tmp = json.load(open('data/container.json')) 363 | self.container = json_graph.node_link_graph(container_tmp, 364 | directed=True) 365 | request_tmp = json.load(open('data/request.json')) 366 | self.request = json_graph.node_link_graph(request_tmp, 367 | directed=True) 368 | rels = json.load(open('data/stitch.json')) 369 | self.cut = evolutionary.EvolutionarySticher(rels) 370 | 371 | def test_stitch_for_success(self): 372 | """ 373 | test stitch for success. 374 | """ 375 | self.cut.stitch(self.container, self.request) 376 | 377 | def test_stitch_for_sanity(self): 378 | """ 379 | Test stitch for sanity. 380 | """ 381 | for _ in range(0, 25): 382 | # changes are high that within one run the algo finds no solution. 383 | self.cut.stitch(self.container, self.request) 384 | 385 | 386 | def _get_population(value): 387 | population = [] 388 | for item in itertools.permutations('abcde'): 389 | population.append(ExampleCandidate(''.join(item), value)) 390 | return population 391 | 392 | 393 | class ExampleCandidate(evolutionary.Candidate): 394 | """ 395 | Simple candidate. 396 | """ 397 | 398 | def __init__(self, gen, test_value): 399 | super(ExampleCandidate, self).__init__(gen) 400 | self.test_value = test_value 401 | 402 | def fitness(self): 403 | """ 404 | Simple but stupid fitness function. 405 | """ 406 | fit = 0.0 407 | for item in self.gen: 408 | if item != self.test_value: 409 | fit += 1 410 | return fit 411 | 412 | def mutate(self): 413 | """ 414 | Not mutating for stable env. 415 | """ 416 | 417 | def crossover(self, partner): 418 | """ 419 | To test (get stable environment) let's return a 'bbbbb' 420 | """ 421 | return self.__class__('bbbbb', self.test_value) 422 | 423 | def __repr__(self): 424 | return self.gen + ':' + str(self.fitness()) 425 | -------------------------------------------------------------------------------- /tests/stitcher_iterative_repair_test.py: -------------------------------------------------------------------------------- 1 | """ 2 | Unittests for iterative_repair module. 3 | """ 4 | 5 | import json 6 | import unittest 7 | 8 | import networkx as nx 9 | 10 | from networkx.readwrite import json_graph 11 | 12 | from stitcher import iterative_repair 13 | 14 | 15 | def _sample_data(): 16 | cont = nx.DiGraph() 17 | cont.add_node('1', **{'type': 'a', 'group': 'foo', 'rank': 1.0}) 18 | cont.add_node('2', **{'type': 'b', 'group': 'foo', 'rank': 1.0}) 19 | cont.add_node('3', **{'type': 'b', 'group': 'bar', 'rank': 2.0}) 20 | cont.add_node('4', **{'type': 'a', 'group': 'bar', 'rank': 2.0}) 21 | cont.add_edge('1', '2') 22 | cont.add_edge('2', '3') 23 | cont.add_edge('4', '3') 24 | 25 | req = nx.DiGraph() 26 | req.add_node('a', **{'type': 'x'}) 27 | req.add_node('b', **{'type': 'y'}) 28 | req.add_edge('a', 'b') 29 | return cont, req 30 | 31 | 32 | class IterativeRepairStitcherTest(unittest.TestCase): 33 | """ 34 | Test for class IterativeRepairStitcher. 35 | """ 36 | 37 | def setUp(self) -> None: 38 | container_tmp = json.load(open('data/container.json')) 39 | self.container = json_graph.node_link_graph(container_tmp, 40 | directed=True) 41 | request_tmp = json.load(open('data/request.json')) 42 | self.request = json_graph.node_link_graph(request_tmp, 43 | directed=True) 44 | rels = json.load(open('data/stitch.json')) 45 | self.cut = iterative_repair.IterativeRepairStitcher(rels) 46 | 47 | # Test for success. 48 | 49 | def test_stitch_for_success(self): 50 | """ 51 | Test fo success. 52 | """ 53 | self.cut.stitch(self.container, self.request) 54 | 55 | def test_find_conflicts_for_success(self): 56 | """ 57 | Test for success. 58 | """ 59 | cont, req = _sample_data() 60 | condy = {'attributes': [('eq', ('a', ('foo', 'bar')))]} 61 | self.cut.find_conflicts(cont, req, condy, {'a': '1'}) 62 | 63 | def test_next_conflict_for_success(self): 64 | """ 65 | Test for success. 66 | """ 67 | self.cut.next_conflict([('foo', 'bar'), ('bar', 'foo')]) 68 | 69 | def test_fix_for_success(self): 70 | """ 71 | Test for success. 72 | """ 73 | self.cut.fix_conflict(('k', ('eq', ('rank', 5))), 74 | self.container, 75 | self.request, 76 | {'k': 'A'}) 77 | 78 | # Test for failure. 79 | 80 | def test_stitch_for_failure(self): 81 | """ 82 | Test for failure. 83 | """ 84 | cont = nx.DiGraph() 85 | cont.add_node('foo', **{'type': 'a'}) 86 | 87 | req = nx.DiGraph() 88 | req.add_node('bar', **{'type': 'y'}) # no matching type in container. 89 | 90 | self.assertRaises(Exception, self.cut.stitch, cont, req) 91 | 92 | # test with unsolvable case. 93 | cont, req = _sample_data() 94 | res = self.cut.stitch(cont, req, { 95 | 'attributes': 96 | [('eq', ('a', ('buuha', 'asdf')))] 97 | }) 98 | self.assertTrue(len(res) == 0) 99 | 100 | # Test for sanity. 101 | 102 | def test_stitch_for_sanity(self): 103 | """ 104 | Test for sanity. 105 | """ 106 | condy = { 107 | 'attributes': [('eq', ('k', ('rank', 5)))] 108 | } 109 | res = self.cut.stitch(self.container, self.request, conditions=condy) 110 | 111 | # TODO: test with multigraph request! 112 | self.assertIsInstance(res, list) 113 | self.assertIsInstance(res[0], nx.DiGraph) 114 | 115 | def test_find_conflicts_for_sanity(self): 116 | """ 117 | Test for sanity. 118 | """ 119 | cont, req = _sample_data() 120 | 121 | # a doesn't have foo attr. 122 | condy = {'a': [('eq', ('foo', 'bar'))]} 123 | res = self.cut.find_conflicts(cont, req, condy, {'a': '1'}) 124 | self.assertEqual(condy['a'][0], res[0][1]) 125 | # a is in group foo 126 | condy = {'a': [('neq', ('group', 'foo'))]} 127 | res = self.cut.find_conflicts(cont, req, condy, {'a': '1'}) 128 | self.assertEqual(condy['a'][0], res[0][1]) 129 | # a's rank is 1.0 130 | condy = {'a': [('lt', ('rank', 0.5))]} 131 | res = self.cut.find_conflicts(cont, req, condy, {'a': '1'}) 132 | self.assertEqual(condy['a'][0], res[0][1]) 133 | # a's rank is 1.0 134 | condy = {'a': [('gt', ('rank', 2.0))]} 135 | res = self.cut.find_conflicts(cont, req, condy, {'a': '1'}) 136 | self.assertEqual(condy['a'][0], res[0][1]) 137 | # a's group name is a word 138 | condy = {'a': [('regex', ('group', '\\d'))]} 139 | res = self.cut.find_conflicts(cont, req, condy, {'a': '1'}) 140 | self.assertEqual(condy['a'][0], res[0][1]) 141 | # a & b not on same node... 142 | condy = {'a': [('same', 'b')], 'b': [('same', 'a')]} 143 | res = self.cut.find_conflicts(cont, req, condy, {'a': '1', 'b': '2'}) 144 | self.assertEqual(condy['a'][0], res[0][1]) 145 | # a & b not on same node... 146 | condy = {'a': [('diff', 'b')], 'b': [('diff', 'a')]} 147 | res = self.cut.find_conflicts(cont, req, condy, {'a': '1', 'b': '1'}) 148 | self.assertEqual(condy['a'][0], res[0][1]) 149 | # a & b not in same group 150 | condy = {'a': [('share', ('group', ['b']))], 151 | 'b': [('share', ('group', ['a']))]} 152 | res = self.cut.find_conflicts(cont, req, condy, {'a': '1', 'b': '3'}) 153 | self.assertEqual(condy['a'][0], res[0][1]) 154 | # a & b in same group 155 | condy = {'a': [('nshare', ('group', ['b']))], 156 | 'b': [('nshare', ('group', ['a']))]} 157 | res = self.cut.find_conflicts(cont, req, condy, {'a': '1', 'b': '2'}) 158 | self.assertEqual(condy['a'][0], res[0][1]) 159 | 160 | def test_next_conflict_for_sanity(self): 161 | """ 162 | Test for sanity. 163 | """ 164 | res = self.cut.next_conflict(['foo', 'bar']) 165 | self.assertIsNotNone(res) 166 | 167 | def test_fix_for_sanity(self): 168 | """ 169 | Test for sanity. 170 | """ 171 | cont, req = _sample_data() 172 | mapping = {'a': '1'} 173 | self.cut.fix_conflict(('a', ('eq', ('foo', 'bar'))), cont, req, 174 | mapping) 175 | self.assertIn('a', mapping) 176 | 177 | 178 | class TestConvertConditions(unittest.TestCase): 179 | """ 180 | Test the condition converter. 181 | """ 182 | 183 | def setUp(self) -> None: 184 | self.cond = { 185 | 'attributes': [('eq', ('a', ('foo', 'y'))), 186 | ('neq', ('a', ('foo', 5))), 187 | ('lt', ('a', ('foo', 4))), 188 | ('lg', ('a', ('foo', 7))), 189 | ('regex', ('a', ('foo', '^a')))], 190 | 'compositions': [('same', ('1', '2')), 191 | ('diff', ('3', '4')), 192 | ('diff', ('3', '1')), 193 | ('share', ('group', ['x', 'y'])), 194 | ('nshare', ('group', ['a', 'b']))] 195 | } 196 | 197 | # Test for success. 198 | 199 | def test_convert_for_success(self): 200 | """ 201 | Test for success. 202 | """ 203 | iterative_repair.convert_conditions(self.cond) 204 | 205 | # Test for failure 206 | 207 | # N/A 208 | 209 | # Test for sanity. 210 | 211 | def test_convert_for_sanity(self): 212 | """ 213 | Test for sanity. 214 | """ 215 | res = iterative_repair.convert_conditions(self.cond) 216 | self.assertIn('a', res) 217 | self.assertIn('b', res) 218 | self.assertIn('x', res) 219 | self.assertIn('y', res) 220 | self.assertIn('1', res) 221 | self.assertIn('2', res) 222 | self.assertIn('3', res) 223 | self.assertIn('4', res) 224 | self.assertTrue(len(res['a']) == 6) # eq, neq, lt, lg, regex, nshare 225 | self.assertTrue(len(res['b']) == 1) # nshare 226 | self.assertTrue(len(res['x']) == 1) # share 227 | self.assertTrue(len(res['y']) == 1) # share 228 | self.assertTrue(len(res['1']) == 2) # same, diff 229 | self.assertTrue(len(res['2']) == 1) # same 230 | self.assertTrue(len(res['3']) == 2) # 2x diff 231 | self.assertTrue(len(res['4']) == 1) # diff 232 | -------------------------------------------------------------------------------- /tests/stitcher_stich_test.py: -------------------------------------------------------------------------------- 1 | """ 2 | Unittest for the basic stitcher. 3 | """ 4 | 5 | import json 6 | import unittest 7 | import sys 8 | 9 | import networkx as nx 10 | 11 | from networkx.readwrite import json_graph 12 | 13 | from stitcher import stitch 14 | 15 | 16 | class TestFilteringConditions(unittest.TestCase): 17 | """ 18 | Tests the filter functions and validates that the right candidates are 19 | eliminated. 20 | """ 21 | 22 | def setUp(self): 23 | self.container = nx.DiGraph() 24 | self.container.add_node('1', **{'type': 'a', 'foo': 'x', 'bar': 5, 25 | 'retest': 'abcde'}) 26 | self.container.add_node('2', **{'type': 'a', 'foo': 'y', 'bar': 7}) 27 | self.container.add_node('3', **{'type': 'b', 'foo': 'x'}) 28 | self.container.add_edge('1', '2') 29 | self.container.add_edge('2', '3') 30 | 31 | self.request = nx.DiGraph() 32 | self.request.add_node('a', **{'type': 'x'}) 33 | self.request.add_node('b', **{'type': 'y'}) 34 | self.request.add_edge('a', 'b') 35 | 36 | self.cut = stitch.GlobalStitcher({'x': 'a', 'y': 'b', 'z': 'c'}) 37 | 38 | def assertItemsEqual(self, first, second): 39 | """ 40 | Python2->3 fix... 41 | """ 42 | if sys.version_info[0] >= 3: 43 | self.assertCountEqual(first, second) 44 | else: 45 | super(TestFilteringConditions, self).assertItemsEqual(first, 46 | second, 47 | 'override.') 48 | 49 | def test_filter_for_sanity(self): 50 | """ 51 | Test filter for sanity. 52 | """ 53 | # none given as condition should results input = output 54 | inp = [1, 2, 3] 55 | out = stitch.my_filter(self.container, [1, 2, 3], None) 56 | self.assertEqual(inp, out) 57 | 58 | # node a requires target node to have attribute foo set to y 59 | condy = {'attributes': [('eq', ('a', ('foo', 'y')))]} 60 | res1 = self.cut.stitch(self.container, self.request, conditions=condy) 61 | # only 1 option left - a->2, b->3! 62 | self.assertEqual(len(res1), 1) 63 | self.assertItemsEqual([('1', '2'), ('a', 'b'), ('2', '3'), ('b', '3'), 64 | ('a', '2')], res1[0].edges()) 65 | 66 | # node a requires target node to have attribute foo set to 5 67 | condy = {'attributes': [('eq', ('a', ('bar', 5)))]} 68 | res1 = self.cut.stitch(self.container, self.request, conditions=condy) 69 | # only 1 option left - a->1, b->3! 70 | self.assertEqual(len(res1), 1) 71 | self.assertItemsEqual([('1', '2'), ('a', 'b'), ('2', '3'), ('b', '3'), 72 | ('a', '1')], res1[0].edges()) 73 | 74 | # node a requires target node to have attribute foo not set to y 75 | condy = {'attributes': [('neq', ('a', ('foo', 'y')))]} 76 | res1 = self.cut.stitch(self.container, self.request, conditions=condy) 77 | # only 1 option left - a->1, b->3! 78 | self.assertEqual(len(res1), 1) 79 | self.assertItemsEqual([('1', '2'), ('a', 'b'), ('2', '3'), ('b', '3'), 80 | ('a', '1')], res1[0].edges()) 81 | 82 | # node a requires target node to have attribute foo not set to 5 83 | condy = {'attributes': [('neq', ('a', ('bar', 5)))]} 84 | res1 = self.cut.stitch(self.container, self.request, conditions=condy) 85 | # only 1 option left - a->2, b->3! 86 | self.assertEqual(len(res1), 1) 87 | self.assertItemsEqual([('1', '2'), ('a', 'b'), ('2', '3'), ('b', '3'), 88 | ('a', '2')], res1[0].edges()) 89 | 90 | # node a requires target node to have an attribute bar with value > 5 91 | condy = {'attributes': [('lg', ('a', ('bar', 5)))]} 92 | res1 = self.cut.stitch(self.container, self.request, conditions=condy) 93 | # only 1 option left - a->2, b->3! 94 | self.assertEqual(len(res1), 1) 95 | self.assertItemsEqual([('1', '2'), ('a', 'b'), ('2', '3'), ('b', '3'), 96 | ('a', '2')], res1[0].edges()) 97 | 98 | # node a requires target node to have an attribute xyz with value > 5 99 | condy = {'attributes': [('lg', ('a', ('xyz', 5)))]} 100 | res1 = self.cut.stitch(self.container, self.request, conditions=condy) 101 | # no stitch possible 102 | self.assertEqual(len(res1), 0) 103 | 104 | # node a requires target node to have an attribute bar with value < 6 105 | condy = {'attributes': [('lt', ('a', ('bar', 6)))]} 106 | res1 = self.cut.stitch(self.container, self.request, conditions=condy) 107 | # only 1 option left - a->1, b->3! 108 | self.assertEqual(len(res1), 1) 109 | self.assertItemsEqual([('1', '2'), ('a', 'b'), ('2', '3'), ('b', '3'), 110 | ('a', '1')], res1[0].edges()) 111 | 112 | # node a requires target node to have an attribute xyz with value < 5 113 | condy = {'attributes': [('lt', ('a', ('xyz', 5)))]} 114 | res1 = self.cut.stitch(self.container, self.request, conditions=condy) 115 | # no stitch possible 116 | self.assertEqual(len(res1), 0) 117 | 118 | # node a requires target node to have an attribute retest which starts 119 | # with an 'a' 120 | condy = {'attributes': [('regex', ('a', ('retest', '^a')))]} 121 | res1 = self.cut.stitch(self.container, self.request, conditions=condy) 122 | # only 1 option left - a->1, b->3! 123 | self.assertEqual(len(res1), 1) 124 | self.assertItemsEqual([('1', '2'), ('a', 'b'), ('2', '3'), ('b', '3'), 125 | ('a', '1')], res1[0].edges()) 126 | 127 | # node a requires target node to have an attribute retest which starts 128 | # with an 'c' 129 | condy = {'attributes': [('regex', ('a', ('retest', '^c')))]} 130 | res1 = self.cut.stitch(self.container, self.request, conditions=condy) 131 | # no options left. 132 | self.assertEqual(len(res1), 0) 133 | 134 | self.container.add_node('4', **{'type': 'b', 'foo': 'x'}) 135 | self.container.add_edge('3', '4') 136 | self.request.add_node('c', **{'type': 'y'}) 137 | self.request.add_edge('b', 'c') 138 | 139 | # node c & b to be stitched to same target! 140 | condy = {'compositions': [('same', ('b', 'c'))], 141 | 'attributes': [('eq', ('a', ('foo', 'x')))]} 142 | res1 = self.cut.stitch(self.container, self.request, conditions=condy) 143 | condy = {'compositions': [('same', ('c', 'b'))], 144 | 'attributes': [('eq', ('a', ('foo', 'x')))]} 145 | res2 = self.cut.stitch(self.container, self.request, conditions=condy) 146 | # only 2 options left: b&c->3 or b&c->4 147 | self.assertEqual(len(res1), 2) 148 | self.assertEqual(len(res2), 2) 149 | # should be identical. 150 | self.assertItemsEqual(res1[0].edges(), res2[0].edges()) 151 | self.assertItemsEqual(res1[1].edges(), res2[1].edges()) 152 | self.assertItemsEqual([('a', '1'), ('a', 'b'), ('b', 'c'), ('1', '2'), 153 | ('3', '4'), ('2', '3'), ('b', '3'), ('c', '3')], 154 | res1[0].edges()) 155 | self.assertItemsEqual([('a', '1'), ('a', 'b'), ('b', 'c'), ('1', '2'), 156 | ('3', '4'), ('2', '3'), ('b', '4'), ('c', '4')], 157 | res1[1].edges()) 158 | 159 | # node a & b to be stitched to different targets! 160 | condy = {'compositions': [('diff', ('b', 'c'))], 161 | 'attributes': [('eq', ('a', ('foo', 'x')))]} 162 | res1 = self.cut.stitch(self.container, self.request, conditions=condy) 163 | condy = {'compositions': [('diff', ('c', 'b'))], 164 | 'attributes': [('eq', ('a', ('foo', 'x')))]} 165 | res2 = self.cut.stitch(self.container, self.request, conditions=condy) 166 | # only 2 options left: b->3 & c->4 or b->4 & c->3 167 | self.assertEqual(len(res1), 2) 168 | self.assertEqual(len(res2), 2) 169 | # should be identical. 170 | self.assertItemsEqual(res1[0].edges(), res2[0].edges()) 171 | self.assertItemsEqual(res1[1].edges(), res2[1].edges()) 172 | either_called = False 173 | if ('b', '4') in res1[0].edges(): 174 | self.assertItemsEqual([('a', '1'), ('a', 'b'), ('b', 'c'), 175 | ('1', '2'), ('3', '4'), ('2', '3'), 176 | ('b', '4'), ('c', '3')], 177 | res1[0].edges()) 178 | self.assertItemsEqual([('a', '1'), ('a', 'b'), ('b', 'c'), 179 | ('1', '2'), ('3', '4'), ('2', '3'), 180 | ('b', '3'), ('c', '4')], 181 | res1[1].edges()) 182 | either_called = True 183 | elif ('b', '4') in res1[1].edges(): 184 | self.assertItemsEqual([('a', '1'), ('a', 'b'), ('b', 'c'), 185 | ('1', '2'), ('3', '4'), ('2', '3'), 186 | ('b', '4'), ('c', '3')], 187 | res1[1].edges()) 188 | self.assertItemsEqual([('a', '1'), ('a', 'b'), ('b', 'c'), 189 | ('1', '2'), ('3', '4'), ('2', '3'), 190 | ('b', '3'), ('c', '4')], 191 | res1[0].edges()) 192 | either_called = True 193 | self.assertTrue(either_called) 194 | 195 | def test_attr_sharing_filter_for_sanity(self): 196 | """ 197 | Test filter for sanity with a more complex setup. 198 | """ 199 | container = nx.DiGraph() 200 | container.add_node('a', **{'type': 'a', 'group': '1', 'geo': 'eu'}) 201 | container.add_node('b', **{'type': 'b', 'group': '1', 'geo': 'us'}) 202 | container.add_node('c', **{'type': 'a', 'group': '2'}) 203 | container.add_node('d', **{'type': 'b', 'group': '2'}) 204 | container.add_node('e', **{'type': 'b'}) 205 | container.add_edge('a', 'b') 206 | container.add_edge('b', 'c') 207 | container.add_edge('c', 'd') 208 | 209 | request = nx.DiGraph() 210 | request.add_node('1', **{'type': 'x'}) 211 | request.add_node('2', **{'type': 'y'}) 212 | request.add_node('3', **{'type': 'y'}) 213 | request.add_edge('1', '2') 214 | request.add_edge('1', '3') 215 | 216 | condy = {'compositions': [('share', ('group', ['1', '2'])), 217 | ('same', ('2', '3'))], 218 | 'attributes': [('eq', ('1', ('geo', 'eu')))]} 219 | res1 = self.cut.stitch(container, request, conditions=condy) 220 | 221 | # only one option possible 222 | self.assertEqual(len(res1), 1) 223 | # verify stitches 224 | self.assertIn(('1', 'a'), res1[0].edges()) 225 | self.assertIn(('2', 'b'), res1[0].edges()) 226 | self.assertIn(('3', 'b'), res1[0].edges()) 227 | 228 | condy = {'compositions': [('nshare', ('group', ['2', '3']))], 229 | 'attributes': [('eq', ('1', ('geo', 'eu'))), 230 | ('eq', ('2', ('geo', 'us')))]} 231 | res1 = self.cut.stitch(container, request, conditions=condy) 232 | 233 | # only one option possible 234 | self.assertEqual(len(res1), 1) 235 | # verify stitches 236 | self.assertIn(('1', 'a'), res1[0].edges()) 237 | self.assertIn(('2', 'b'), res1[0].edges()) 238 | self.assertIn(('3', 'd'), res1[0].edges()) 239 | 240 | 241 | class TestGlobalStitcher(unittest.TestCase): 242 | """ 243 | Test the global stitcher class. 244 | """ 245 | 246 | def setUp(self): 247 | container_tmp = json.load(open('data/container.json')) 248 | self.container = json_graph.node_link_graph(container_tmp, 249 | directed=True) 250 | request_tmp = json.load(open('data/request.json')) 251 | self.request = json_graph.node_link_graph(request_tmp, 252 | directed=True) 253 | rels = json.load(open('data/stitch.json')) 254 | self.cut = stitch.GlobalStitcher(rels) 255 | 256 | def test_stitch_for_success(self): 257 | """ 258 | Test stitch for success. 259 | """ 260 | self.cut.stitch(self.container, self.request) 261 | 262 | def test_stitch_for_sanity(self): 263 | """ 264 | Test stitch for sanity. 265 | """ 266 | # basic stitch test 267 | res1 = self.cut.stitch(self.container, self.request) 268 | self.assertTrue(len(res1) > 0) 269 | self.assertEqual(res1[0].number_of_edges(), 270 | self.container.number_of_edges() + 5) 271 | self.assertEqual(res1[0].number_of_nodes(), 272 | self.container.number_of_nodes() + 3) 273 | 274 | # let's add a node to the request which does not require to be 275 | # stitched to the container. Hence added edges = 3! 276 | self.request.add_node('n', **{'type': 'foo', 'rank': 7}) 277 | self.request.add_edge('k', 'n') 278 | self.request.add_edge('n', 'l') 279 | 280 | self.assertTrue(len(res1) > 0) 281 | self.assertEqual(res1[0].number_of_edges(), 282 | self.container.number_of_edges() + 5) 283 | self.assertEqual(res1[0].number_of_nodes(), 284 | self.container.number_of_nodes() + 3) 285 | -------------------------------------------------------------------------------- /tests/stitcher_test.py: -------------------------------------------------------------------------------- 1 | """ 2 | unittest for the module level stuff. 3 | """ 4 | 5 | import unittest 6 | 7 | import stitcher 8 | 9 | 10 | class StitcherTest(unittest.TestCase): 11 | """ 12 | Testcase for the top level module stuff. 13 | """ 14 | 15 | def setUp(self): 16 | self.cut = stitcher.Stitcher({'a': 'b'}) 17 | 18 | def test_stitch_for_failure(self): 19 | """ 20 | Test stitch for failure. 21 | """ 22 | self.assertRaises(NotImplementedError, self.cut.stitch, None, None) 23 | -------------------------------------------------------------------------------- /tests/stitcher_validators_test.py: -------------------------------------------------------------------------------- 1 | """ 2 | Unittest for some basic validators. 3 | """ 4 | 5 | import json 6 | import unittest 7 | 8 | from networkx.readwrite import json_graph 9 | 10 | from stitcher import stitch 11 | from stitcher import validators 12 | 13 | 14 | class TestIncomingEdgeStitcher(unittest.TestCase): 15 | """ 16 | Test the simple stitcher class based on # of incoming edges. 17 | """ 18 | 19 | def setUp(self): 20 | container_tmp = json.load(open('data/container.json')) 21 | self.container = json_graph.node_link_graph(container_tmp, 22 | directed=True) 23 | request_tmp = json.load(open('data/request.json')) 24 | self.request = json_graph.node_link_graph(request_tmp, 25 | directed=True) 26 | rels = json.load(open('data/stitch.json')) 27 | self.stitcher = stitch.GlobalStitcher(rels) 28 | 29 | def test_validate_for_success(self): 30 | """ 31 | Test validate for success. 32 | """ 33 | res1 = self.stitcher.stitch(self.container, self.request) 34 | validators.validate_incoming_edges(res1) 35 | 36 | def test_validate_for_sanity(self): 37 | """ 38 | Test validate for sanity. 39 | """ 40 | res1 = self.stitcher.stitch(self.container, self.request) 41 | res2 = validators.validate_incoming_edges(res1, {'b': 5}) 42 | 43 | self.assertTrue(len(res2) == 8) 44 | # 4 are ok & 4 are stupid 45 | count = 0 46 | for item in res2: 47 | if res2[item] != 'ok': 48 | count += 1 49 | self.assertEqual(count, 4) 50 | 51 | 52 | class TestNodeRankSticher(unittest.TestCase): 53 | """ 54 | Test the simple stitcher class based on ranks. 55 | """ 56 | 57 | def setUp(self): 58 | container_tmp = json.load(open('data/container.json')) 59 | self.container = json_graph.node_link_graph(container_tmp, 60 | directed=True) 61 | request_tmp = json.load(open('data/request.json')) 62 | self.request = json_graph.node_link_graph(request_tmp, 63 | directed=True) 64 | rels = json.load(open('data/stitch.json')) 65 | self.stitcher = stitch.GlobalStitcher(rels) 66 | 67 | def test_validate_for_success(self): 68 | """ 69 | Test validate for success. 70 | """ 71 | res1 = self.stitcher.stitch(self.container, self.request) 72 | validators.validate_incoming_rank(res1) 73 | 74 | def test_validate_for_sanity(self): 75 | """ 76 | Test validate for sanity. 77 | """ 78 | res1 = self.stitcher.stitch(self.container, self.request) 79 | res2 = validators.validate_incoming_rank(res1, {'a': (0, 3)}) 80 | 81 | self.assertTrue(len(res2) == 8) 82 | self.assertEqual(res2[4], 83 | 'node B rank is >= 3 and # incoming edges is > 0') 84 | -------------------------------------------------------------------------------- /tests/stitcher_vis_test.py: -------------------------------------------------------------------------------- 1 | """ 2 | Unittest for the vis module. 3 | """ 4 | 5 | import os 6 | import unittest 7 | 8 | import networkx as nx 9 | 10 | from stitcher import vis 11 | 12 | 13 | class TestVisualizations(unittest.TestCase): 14 | """ 15 | Tests the vis module. 16 | """ 17 | 18 | def setUp(self): 19 | self.container = nx.DiGraph() 20 | self.container.add_node('a', **{'type': 'a', 'rank': 1}) 21 | self.container.add_node('b', **{'type': 'a', 'rank': 5}) 22 | self.container.add_node('c', **{'type': 'a', 'rank': 10}) 23 | self.container.add_edge('a', 'b') 24 | self.container.add_edge('b', 'c') 25 | self.request = nx.DiGraph() 26 | self.request.add_node('1', **{'type': 'b'}) 27 | self.request.add_node('2', **{'type': 'b'}) 28 | self.request.add_edge('1', '2') 29 | self.container = nx.union(self.container, self.request) 30 | self.container.add_edge('1', 'a') 31 | self.container.add_edge('2', 'a') 32 | 33 | def test_show_for_success(self): 34 | """ 35 | Test the show routine. 36 | """ 37 | vis.show([self.container], self.request, ['Test'], 38 | filename='tmp.png') 39 | os.remove('tmp.png') 40 | 41 | def test_show3d_for_success(self): 42 | """ 43 | Test the show routine. 44 | """ 45 | vis.show_3d([self.container], self.request, ['Test'], 46 | filename='tmp.png') 47 | os.remove('tmp.png') 48 | 49 | def test_show3d_for_failure(self): 50 | """ 51 | Test the show routine for failure. 52 | """ 53 | # request nodes are not in container. 54 | self.container.remove_edge('1', 'a') 55 | self.container.remove_node('1') 56 | self.container.remove_edge('2', 'a') 57 | self.container.remove_node('2') 58 | self.assertRaises(nx.NetworkXError, vis.show_3d, [self.container], 59 | self.request, ['Test']) 60 | --------------------------------------------------------------------------------