├── helper_scripts ├── requirements.txt ├── run_benchmarks.sh ├── stop_consumer.sh ├── start_consumer.sh ├── setup_ssh_tunnel.sh ├── reset_log_store_docker.sh ├── set_consumer_behaviour.sh ├── analyze.py └── benchmark.py ├── config └── openvpn_server │ ├── pricewars-user │ ├── pricewars-ui │ ├── pricewars-marketplace │ ├── pricewars-consumer │ ├── pricewars-merchant │ ├── pricewars-producer │ ├── pricewars-ui.ovpn │ ├── pricewars-consumer.ovpn │ ├── pricewars-merchant.ovpn │ ├── pricewars-producer.ovpn │ └── pricewars-marketplace.ovpn ├── .gitignore ├── docs ├── modeling │ ├── sequence_diagram_flow.png │ ├── pricewars-architecture.odg │ ├── pricewars-architecture.png │ └── sequence_diagram.txt ├── api │ └── README.md └── pricewars_logo.svg ├── .gitattributes ├── .gitmodules ├── LICENSE.md ├── docker-compose.yml └── README.md /helper_scripts/requirements.txt: -------------------------------------------------------------------------------- 1 | matplotlib 2 | kafka-python -------------------------------------------------------------------------------- /config/openvpn_server/pricewars-user: -------------------------------------------------------------------------------- 1 | # config for pricewars-user 2 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | .DS_Store 2 | docker-mounts/ 3 | 4 | # python directories 5 | __pycache__ 6 | .idea 7 | -------------------------------------------------------------------------------- /docs/modeling/sequence_diagram_flow.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hpi-epic/pricewars/HEAD/docs/modeling/sequence_diagram_flow.png -------------------------------------------------------------------------------- /docs/modeling/pricewars-architecture.odg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hpi-epic/pricewars/HEAD/docs/modeling/pricewars-architecture.odg -------------------------------------------------------------------------------- /docs/modeling/pricewars-architecture.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hpi-epic/pricewars/HEAD/docs/modeling/pricewars-architecture.png -------------------------------------------------------------------------------- /.gitattributes: -------------------------------------------------------------------------------- 1 | # This setting forces Git to normalize line endings to LF on checkin and 2 | # prevents conversion to CRLF when the file is checked out. 3 | * text=auto eol=lf 4 | -------------------------------------------------------------------------------- /config/openvpn_server/pricewars-ui: -------------------------------------------------------------------------------- 1 | # config for client ui 2 | ifconfig-push 10.8.0.23 255.255.255.192 3 | #push "route 10.1.135.0 255.255.255.0 10.1.134.62" 4 | # push "dhcp-option WINS addr" 5 | # push "dhcp-option DNS addr" 6 | -------------------------------------------------------------------------------- /config/openvpn_server/pricewars-marketplace: -------------------------------------------------------------------------------- 1 | # config for client ui 2 | ifconfig-push 10.8.0.25 255.255.255.192 3 | #push "route 10.1.135.0 255.255.255.0 10.1.134.62" 4 | # push "dhcp-option WINS addr" 5 | # push "dhcp-option DNS addr" 6 | -------------------------------------------------------------------------------- /config/openvpn_server/pricewars-consumer: -------------------------------------------------------------------------------- 1 | # config for client consumer 2 | ifconfig-push 10.8.0.20 255.255.255.192 3 | #push "route 10.1.135.0 255.255.255.0 10.1.134.62" 4 | # push "dhcp-option WINS addr" 5 | # push "dhcp-option DNS addr" 6 | -------------------------------------------------------------------------------- /config/openvpn_server/pricewars-merchant: -------------------------------------------------------------------------------- 1 | # config for client merchant 2 | ifconfig-push 10.8.0.21 255.255.255.192 3 | #push "route 10.1.135.0 255.255.255.0 10.1.134.62" 4 | # push "dhcp-option WINS addr" 5 | # push "dhcp-option DNS addr" 6 | -------------------------------------------------------------------------------- /config/openvpn_server/pricewars-producer: -------------------------------------------------------------------------------- 1 | # config for client producer 2 | ifconfig-push 10.8.0.22 255.255.255.192 3 | #push "route 10.1.135.0 255.255.255.0 10.1.134.62" 4 | # push "dhcp-option WINS addr" 5 | # push "dhcp-option DNS addr" 6 | -------------------------------------------------------------------------------- /helper_scripts/run_benchmarks.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | declare map=( 4 | [abc,0]=1 5 | [abc,1]=2 6 | [abc,2]=3 7 | [abc,3]=4 8 | [def,0]="http://..." 9 | [def,1]=33 10 | [def,2]=2 11 | ) 12 | key="def" 13 | i=1 14 | echo "${map[$key,$i]}" # => 33 15 | i=0 16 | echo "${map[$key,$i]}" # => 33 17 | 18 | -------------------------------------------------------------------------------- /helper_scripts/stop_consumer.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | 4 | printf 'Stopping consumer: ' 5 | RET=`curl -s -o /dev/null -w "%{http_code}" -X DELETE -H "Content-Type: application/json" http://consumer:3000/setting` 6 | 7 | if [ "$RET" = "200" ] 8 | then 9 | echo 'successful.' 10 | else 11 | echo 'unsuccessful.' 12 | fi 13 | 14 | -------------------------------------------------------------------------------- /docs/api/README.md: -------------------------------------------------------------------------------- 1 | # API Documentation 2 | 3 | For more details regarding the API specification, the reader is kindly referred to the separate branch [gh-pages](https://github.com/hpi-epic/masterproject-pricewars/tree/gh-pages) within this repository. 4 | 5 | Due to the current github bug, that submodules with dependencies to private repositories cannot be resolved, the github page build process fails enforcing us to separate the API specification from the submodules for rendering those via [github.io](https://hpi-epic.github.io/masterproject-pricewars/api/#/). 6 | -------------------------------------------------------------------------------- /helper_scripts/start_consumer.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | printf 'Starting consumer: ' 4 | SETTINGS=`curl -s -X GET -H "Content-Type: application/json" http://consumer:3000/setting` 5 | SUCC=`echo $SETTINGS | grep -c 'consumer_per_minute'` 6 | 7 | if [ "$SUCC" = "1" ] 8 | then 9 | RET=`curl -s -o /dev/null -w "%{http_code}" -X POST -H "Content-Type: application/json" -d "$SETTINGS" http://consumer:3000/setting` 10 | if [ "$RET" = "200" ] 11 | then 12 | echo 'successful.' 13 | fi 14 | else 15 | echo 'unsuccessful (error: could not get settings before starting.) Exiting.' 16 | fi 17 | 18 | -------------------------------------------------------------------------------- /.gitmodules: -------------------------------------------------------------------------------- 1 | [submodule "merchant"] 2 | path = merchant 3 | url = https://github.com/hpi-epic/pricewars-merchant.git 4 | [submodule "consumer"] 5 | path = consumer 6 | url = https://github.com/hpi-epic/pricewars-consumer.git 7 | [submodule "producer"] 8 | path = producer 9 | url = https://github.com/hpi-epic/pricewars-producer.git 10 | [submodule "management-ui"] 11 | path = management-ui 12 | url = git@github.com:hpi-epic/pricewars-mgmt-ui.git 13 | [submodule "kafka-reverse-proxy"] 14 | path = kafka-reverse-proxy 15 | url = https://github.com/hpi-epic/pricewars-kafka-reverse-proxy.git 16 | [submodule "marketplace"] 17 | path = marketplace 18 | url = https://github.com/hpi-epic/pricewars-marketplace.git 19 | [submodule "analytics"] 20 | path = analytics 21 | url = https://github.com/hpi-epic/pricewars-analytics.git 22 | -------------------------------------------------------------------------------- /helper_scripts/setup_ssh_tunnel.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # 3 | ### Setup Port Forwarding 4 | iptables -t nat -A POSTROUTING -j MASQUERADE 5 | sysctl -w net.ipv4.ip_forward=1 6 | echo 1 > /proc/sys/net/ipv4/ip_forward 7 | # 8 | ### Adding routings 9 | iptables -t nat -A PREROUTING -p tcp --dport 7041 -j DNAT --to-destination 10.8.0.20:22 #consumer 10 | iptables -t nat -A PREROUTING -p tcp --dport 7046 -j DNAT --to-destination 10.8.0.25:22 #marketplace 11 | iptables -t nat -A PREROUTING -p tcp --dport 7047 -j DNAT --to-destination 10.8.0.21:22 #merchant 12 | iptables -t nat -A PREROUTING -p tcp --dport 7042 -j DNAT --to-destination 10.8.0.23:22 #ui 13 | iptables -t nat -A PREROUTING -p tcp --dport 7045 -j DNAT --to-destination 10.8.0.22:22 #producer 14 | iptables -t nat -A PREROUTING -p tcp --dport 7049 -j DNAT --to-destination 10.8.0.26:22 #logger 15 | # 16 | iptables-save 17 | # 18 | ### Delete nat rules if necessary 19 | #sudo iptables -t nat -F 20 | #sudo iptables -L -t nat --line-numbers 21 | #sudo iptables -t nat -D PREROUTING 4 22 | -------------------------------------------------------------------------------- /LICENSE.md: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright © 2016-present Marvin Bornstein, Johanna Latt, Jan Lindemann, Nikolai J. Podlesny, Sebastian Serth, Jan Selke 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /docs/modeling/sequence_diagram.txt: -------------------------------------------------------------------------------- 1 | title Setup 2 | 3 | Marketplace->+Producer: GET/decryption_key 4 | Producer-->-Marketplace: 5 | loop Producer add product 6 | MGMT-UI->+Producer: PUT/product/{id} 7 | Producer-->-MGMT-UI: 8 | end 9 | loop Merchant start 10 | MGMT-UI->+Merchant: POST/settings/execution {state: "init"} 11 | Merchant->+Marketplace: POST/merchants 12 | Marketplace-->-Merchant: 13 | Merchant-->-MGMT-UI: 14 | MGMT-UI->+Merchant: POST/settings/execution {state: "start"} 15 | Merchant-->-MGMT-UI: 16 | end 17 | loop Consumer start 18 | MGMT-UI->+Consumer: POST/setting 19 | Consumer-->-MGMT-UI: 20 | end 21 | 22 | note over Merchant,Marketplace,Consumer,MGMT-UI,Producer: Meta-Setup completed 23 | Merchant->+Producer:POST/buyers 24 | Producer-->-Merchant:products 25 | Merchant->+Marketplace:POST/offers 26 | Marketplace-->-Merchant: 27 | 28 | note over Merchant,Marketplace,Consumer,MGMT-UI,Producer: internal Setup completed & initial products on marketplace 29 | loop Consumer 30 | Consumer->+Marketplace:GET/offers 31 | Marketplace-->-Consumer:offers 32 | opt Buying Decision 33 | Consumer->+Marketplace:POST/offers/{id}/buy 34 | Marketplace->>+Merchant:POST/sold 35 | Marketplace-->-Consumer: 36 | Merchant->+Producer:GET/products/buy 37 | Producer-->-Merchant: 38 | Merchant->>-Marketplace:PATCH/offers/{id}/restock 39 | end 40 | end 41 | loop Merchant adjustprice 42 | Merchant->+Marketplace:GET/offers 43 | Marketplace-->-Merchant: 44 | loop for each product 45 | Merchant->+Marketplace:PUT/offers/{id} 46 | Marketplace-->-Merchant: 47 | end 48 | end 49 | 50 | -------------------------------------------------------------------------------- /helper_scripts/reset_log_store_docker.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | # Paths to Kafka and Flink (docker setup). 4 | # Please adapt according to your configuration. 5 | FLINK_PATH=/opt/flink 6 | KAFKA_PATH=/opt/kafka 7 | 8 | printf 'Emptying Kafka topics.' 9 | for topic in 'marketSituation' 'cumulativeRevenueBasedMarketshare' 'updateOffer' 'cumulativeRevenueBasedMarketshareHourly' 'cumulativeRevenueBasedMarketshareDaily' 'cumulativeTurnoverBasedMarketshare' 'cumulativeTurnoverBasedMarketshareHourly' 'cumulativeTurnoverBasedMarketshareDaily' 'cumulativeAmountBasedMarketshare' 'cumulativeAmountBasedMarketshareHourly' 'cumulativeAmountBasedMarketshareDaily' 'revenue' 'buyOffer' 'producer' 'revenuePerMinute' 'profitPerMinute' 'revenuePerHour' 'profitPerHour' 10 | do 11 | docker exec masterprojectpricewars_kafka_1 ${KAFKA_PATH}/bin/kafka-topics.sh --delete --zookeeper zookeeper:2181 --topic $topic > /dev/null 12 | printf '.' 13 | done 14 | 15 | printf '\nRemoving old CSV files and restarting reverse-kafka-proxy to empty its cache.' 16 | docker exec masterprojectpricewars_kafka-reverse-proxy_1 find /loggerapp/data/ -name '*.csv' -exec rm -f {} \; 17 | docker restart masterprojectpricewars_kafka-reverse-proxy_1 18 | 19 | printf '\nCanceling flink jobs.' 20 | docker exec masterprojectpricewars_flink-jobmanager_1 ${FLINK_PATH}/bin/flink list -r | awk '{split($0, a, " : "); print a[2]}' | while read line; do 21 | [ -z "$line" ] && continue 22 | docker exec masterprojectpricewars_flink-jobmanager_1 ${FLINK_PATH}/bin/flink cancel $line > /dev/null 23 | printf '.' 24 | done 25 | 26 | printf '\nRestarting flink jobs.' 27 | for file in `docker exec masterprojectpricewars_flink-jobmanager_1 ls /analytics/ | grep jar` 28 | do 29 | if [ "${file}" != "${file%.jar}" ];then 30 | docker exec masterprojectpricewars_flink-jobmanager_1 ${FLINK_PATH}/bin/flink run -d /analytics/$file > /dev/null 31 | fi 32 | printf '.' 33 | done 34 | 35 | echo '' 36 | -------------------------------------------------------------------------------- /helper_scripts/set_consumer_behaviour.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | echo 'Please be aware that this script currently does not ensure the probabilities add up to 100%.' 4 | echo 'Default to: 50% logit behaviour, 30% cheapest behaviour, and 20% random behaviour.' 5 | 6 | if [ -z "$PERCENTAGE_LOGIT" ]; then 7 | PERCENTAGE_LOGIT=50 8 | fi 9 | 10 | if [ -z "$PERCENTAGE_CHEAPEST" ]; then 11 | PERCENTAGE_CHEAPEST=30 12 | fi 13 | 14 | if [ -z "$PERCENTAGE_SEC_CHEAPEST" ]; then 15 | PERCENTAGE_SEC_CHEAPEST=10 16 | fi 17 | 18 | if [ -z "$PERCENTAGE_RANDOM" ]; then 19 | PERCENTAGE_RANDOM=10 20 | fi 21 | 22 | SETTINGS='{"consumer_per_minute":100,"amount_of_consumers":1,"probability_of_buy":100,"min_buying_amount":1,"max_buying_amount":1,"min_wait":0.1,"max_wait":2,"behaviors":[{"name":"first","description":"I am buying the first possible item","settings":{},"settings_description":"Behavior settings not necessary","amount":0},{"name":"random","description":"I am buying random items","settings":{},"settings_description":"Behavior settings not necessary","amount":'$PERCENTAGE_RANDOM'},{"name":"cheap","description":"I am buying the cheapest item","settings":{},"settings_description":"Behavior settings not necessary","amount":'$PERCENTAGE_CHEAPEST'},{"name":"expensive","description":"I am buying the most expensive item","settings":{},"settings_description":"Behavior settings not necessary","amount":0},{"name":"cheap_and_prime","description":"I am buying the cheapest item which supports prime shipping","settings":{},"settings_description":"Behavior settings not necessary","amount":0},{"name":"cheapest_best_quality","description":"I am buying the cheapest best quality available.","settings":{},"settings_description":"Behavior settings not necessary","amount":0},{"name":"cheapest_best_quality_with_prime","description":"I am buying the cheapest best quality available which supports prime.","settings":{},"settings_description":"Behavior settings not necessary","amount":0},{"name":"second_cheap","description":"I am buying the second cheapest item","settings":{},"settings_description":"Behavior settings not necessary","amount":'$PERCENTAGE_SEC_CHEAPEST'},{"name":"third_cheap","description":"I am buying the third cheapest item","settings":{},"settings_description":"Behavior settings not necessary","amount":0},{"name":"sigmoid_distribution_price","description":"I am with sigmoid distribution on the price regarding the producer prices","settings":{},"settings_description":"Behavior settings not necessary","amount":0},{"name":"logit_coefficients","description":"I am with logit coefficients","settings":{"coefficients":{"intercept":-6.6177961,"price_rank":0.2083944,"amount_of_all_competitors":0.253481,"average_price_on_market":-0.0079326,"quality_rank":-0.1835972}},"settings_description":"Key Value map for Feature and their coeffient","amount":'$PERCENTAGE_LOGIT'}],"timeout_if_no_offers_available":2,"timeout_if_too_many_requests":30,"max_buying_price":80,"debug":false,"producer_url":"http://producer:3050","product_popularity":{"1":100},"marketplace_url":"http://marketplace:8080"}' 23 | 24 | printf "Setting consumer behaviour to $PERCENTAGE_CHEAPEST%% cheapest, $PERCENTAGE_SEC_CHEAPEST%% second cheapest, $PERCENTAGE_LOGIT%% logit, and $PERCENTAGE_RANDOM%% random: " 25 | RET=`curl -s -o /dev/null -w "%{http_code}" -X PUT -H "Content-Type: application/json" -d "$SETTINGS" http://consumer:3000/setting` 26 | 27 | if [ "$RET" = "200" ] 28 | then 29 | echo 'successful.' 30 | else 31 | echo 'unsuccessful.' 32 | fi 33 | -------------------------------------------------------------------------------- /config/openvpn_server/pricewars-ui.ovpn: -------------------------------------------------------------------------------- 1 | ############################################## 2 | # Sample client-side OpenVPN 2.0 config file # 3 | # for connecting to multi-client server. # 4 | # # 5 | # This configuration can be used by multiple # 6 | # clients, however each client should have # 7 | # its own cert and key files. # 8 | # # 9 | # On Windows, you might want to rename this # 10 | # file so it has a .ovpn extension # 11 | ############################################## 12 | 13 | # Specify that we are a client and that we 14 | # will be pulling certain config file directives 15 | # from the server. 16 | client 17 | 18 | # Use the same setting as you are using on 19 | # the server. 20 | # On most systems, the VPN will not function 21 | # unless you partially or fully disable 22 | # the firewall for the TUN/TAP interface. 23 | ;dev tap 24 | dev tun 25 | 26 | # Windows needs the TAP-Win32 adapter name 27 | # from the Network Connections panel 28 | # if you have more than one. On XP SP2, 29 | # you may need to disable the firewall 30 | # for the TAP adapter. 31 | ;dev-node MyTap 32 | 33 | # Are we connecting to a TCP or 34 | # UDP server? Use the same setting as 35 | # on the server. 36 | ;proto tcp 37 | proto udp 38 | 39 | # The hostname/IP and port of the server. 40 | # You can have multiple remote entries 41 | # to load balance between the servers. 42 | remote 185.82.202.248 1194 43 | ;remote my-server-2 1194 44 | 45 | # Choose a random host from the remote 46 | # list for load-balancing. Otherwise 47 | # try hosts in the order specified. 48 | ;remote-random 49 | 50 | # Keep trying indefinitely to resolve the 51 | # host name of the OpenVPN server. Very useful 52 | # on machines which are not permanently connected 53 | # to the internet such as laptops. 54 | resolv-retry infinite 55 | 56 | # Most clients don't need to bind to 57 | # a specific local port number. 58 | nobind 59 | 60 | # Downgrade privileges after initialization (non-Windows only) 61 | ;user nobody 62 | ;group nogroup 63 | 64 | # Try to preserve some state across restarts. 65 | persist-key 66 | persist-tun 67 | 68 | # If you are connecting through an 69 | # HTTP proxy to reach the actual OpenVPN 70 | # server, put the proxy server/IP and 71 | # port number here. See the man page 72 | # if your proxy server requires 73 | # authentication. 74 | ;http-proxy-retry # retry on connection failures 75 | ;http-proxy [proxy server] [proxy port #] 76 | 77 | # Wireless networks often produce a lot 78 | # of duplicate packets. Set this flag 79 | # to silence duplicate packet warnings. 80 | ;mute-replay-warnings 81 | 82 | # SSL/TLS parms. 83 | # See the server config file for more 84 | # description. It's best to use 85 | # a separate .crt/.key file pair 86 | # for each client. A single ca 87 | # file can be used for all clients. 88 | 89 | # Verify server certificate by checking 90 | # that the certicate has the nsCertType 91 | # field set to "server". This is an 92 | # important precaution to protect against 93 | # a potential attack discussed here: 94 | # http://openvpn.net/howto.html#mitm 95 | # 96 | # To use this feature, you will need to generate 97 | # your server certificates with the nsCertType 98 | # field set to "server". The build-key-server 99 | # script in the easy-rsa folder will do this. 100 | ns-cert-type server 101 | 102 | # If a tls-auth key is used on the server 103 | # then every client must also have the key. 104 | ;tls-auth ta.key 1 105 | 106 | # Select a cryptographic cipher. 107 | # If the cipher option is used on the server 108 | # then you must also specify it here. 109 | ;cipher x 110 | 111 | # Enable compression on the VPN link. 112 | # Don't enable this unless it is also 113 | # enabled in the server config file. 114 | comp-lzo 115 | 116 | # Set log file verbosity. 117 | verb 3 118 | 119 | # Silence repeating messages 120 | ;mute 20 121 | -------------------------------------------------------------------------------- /config/openvpn_server/pricewars-consumer.ovpn: -------------------------------------------------------------------------------- 1 | ############################################## 2 | # Sample client-side OpenVPN 2.0 config file # 3 | # for connecting to multi-client server. # 4 | # # 5 | # This configuration can be used by multiple # 6 | # clients, however each client should have # 7 | # its own cert and key files. # 8 | # # 9 | # On Windows, you might want to rename this # 10 | # file so it has a .ovpn extension # 11 | ############################################## 12 | 13 | # Specify that we are a client and that we 14 | # will be pulling certain config file directives 15 | # from the server. 16 | client 17 | 18 | # Use the same setting as you are using on 19 | # the server. 20 | # On most systems, the VPN will not function 21 | # unless you partially or fully disable 22 | # the firewall for the TUN/TAP interface. 23 | ;dev tap 24 | dev tun 25 | 26 | # Windows needs the TAP-Win32 adapter name 27 | # from the Network Connections panel 28 | # if you have more than one. On XP SP2, 29 | # you may need to disable the firewall 30 | # for the TAP adapter. 31 | ;dev-node MyTap 32 | 33 | # Are we connecting to a TCP or 34 | # UDP server? Use the same setting as 35 | # on the server. 36 | ;proto tcp 37 | proto udp 38 | 39 | # The hostname/IP and port of the server. 40 | # You can have multiple remote entries 41 | # to load balance between the servers. 42 | remote 185.82.202.248 1194 43 | ;remote my-server-2 1194 44 | 45 | # Choose a random host from the remote 46 | # list for load-balancing. Otherwise 47 | # try hosts in the order specified. 48 | ;remote-random 49 | 50 | # Keep trying indefinitely to resolve the 51 | # host name of the OpenVPN server. Very useful 52 | # on machines which are not permanently connected 53 | # to the internet such as laptops. 54 | resolv-retry infinite 55 | 56 | # Most clients don't need to bind to 57 | # a specific local port number. 58 | nobind 59 | 60 | # Downgrade privileges after initialization (non-Windows only) 61 | ;user nobody 62 | ;group nogroup 63 | 64 | # Try to preserve some state across restarts. 65 | persist-key 66 | persist-tun 67 | 68 | # If you are connecting through an 69 | # HTTP proxy to reach the actual OpenVPN 70 | # server, put the proxy server/IP and 71 | # port number here. See the man page 72 | # if your proxy server requires 73 | # authentication. 74 | ;http-proxy-retry # retry on connection failures 75 | ;http-proxy [proxy server] [proxy port #] 76 | 77 | # Wireless networks often produce a lot 78 | # of duplicate packets. Set this flag 79 | # to silence duplicate packet warnings. 80 | ;mute-replay-warnings 81 | 82 | # SSL/TLS parms. 83 | # See the server config file for more 84 | # description. It's best to use 85 | # a separate .crt/.key file pair 86 | # for each client. A single ca 87 | # file can be used for all clients. 88 | 89 | # Verify server certificate by checking 90 | # that the certicate has the nsCertType 91 | # field set to "server". This is an 92 | # important precaution to protect against 93 | # a potential attack discussed here: 94 | # http://openvpn.net/howto.html#mitm 95 | # 96 | # To use this feature, you will need to generate 97 | # your server certificates with the nsCertType 98 | # field set to "server". The build-key-server 99 | # script in the easy-rsa folder will do this. 100 | ns-cert-type server 101 | 102 | # If a tls-auth key is used on the server 103 | # then every client must also have the key. 104 | ;tls-auth ta.key 1 105 | 106 | # Select a cryptographic cipher. 107 | # If the cipher option is used on the server 108 | # then you must also specify it here. 109 | ;cipher x 110 | 111 | # Enable compression on the VPN link. 112 | # Don't enable this unless it is also 113 | # enabled in the server config file. 114 | comp-lzo 115 | 116 | # Set log file verbosity. 117 | verb 3 118 | 119 | # Silence repeating messages 120 | ;mute 20 121 | -------------------------------------------------------------------------------- /config/openvpn_server/pricewars-merchant.ovpn: -------------------------------------------------------------------------------- 1 | ############################################## 2 | # Sample client-side OpenVPN 2.0 config file # 3 | # for connecting to multi-client server. # 4 | # # 5 | # This configuration can be used by multiple # 6 | # clients, however each client should have # 7 | # its own cert and key files. # 8 | # # 9 | # On Windows, you might want to rename this # 10 | # file so it has a .ovpn extension # 11 | ############################################## 12 | 13 | # Specify that we are a client and that we 14 | # will be pulling certain config file directives 15 | # from the server. 16 | client 17 | 18 | # Use the same setting as you are using on 19 | # the server. 20 | # On most systems, the VPN will not function 21 | # unless you partially or fully disable 22 | # the firewall for the TUN/TAP interface. 23 | ;dev tap 24 | dev tun 25 | 26 | # Windows needs the TAP-Win32 adapter name 27 | # from the Network Connections panel 28 | # if you have more than one. On XP SP2, 29 | # you may need to disable the firewall 30 | # for the TAP adapter. 31 | ;dev-node MyTap 32 | 33 | # Are we connecting to a TCP or 34 | # UDP server? Use the same setting as 35 | # on the server. 36 | ;proto tcp 37 | proto udp 38 | 39 | # The hostname/IP and port of the server. 40 | # You can have multiple remote entries 41 | # to load balance between the servers. 42 | remote 185.82.202.248 1194 43 | ;remote my-server-2 1194 44 | 45 | # Choose a random host from the remote 46 | # list for load-balancing. Otherwise 47 | # try hosts in the order specified. 48 | ;remote-random 49 | 50 | # Keep trying indefinitely to resolve the 51 | # host name of the OpenVPN server. Very useful 52 | # on machines which are not permanently connected 53 | # to the internet such as laptops. 54 | resolv-retry infinite 55 | 56 | # Most clients don't need to bind to 57 | # a specific local port number. 58 | nobind 59 | 60 | # Downgrade privileges after initialization (non-Windows only) 61 | ;user nobody 62 | ;group nogroup 63 | 64 | # Try to preserve some state across restarts. 65 | persist-key 66 | persist-tun 67 | 68 | # If you are connecting through an 69 | # HTTP proxy to reach the actual OpenVPN 70 | # server, put the proxy server/IP and 71 | # port number here. See the man page 72 | # if your proxy server requires 73 | # authentication. 74 | ;http-proxy-retry # retry on connection failures 75 | ;http-proxy [proxy server] [proxy port #] 76 | 77 | # Wireless networks often produce a lot 78 | # of duplicate packets. Set this flag 79 | # to silence duplicate packet warnings. 80 | ;mute-replay-warnings 81 | 82 | # SSL/TLS parms. 83 | # See the server config file for more 84 | # description. It's best to use 85 | # a separate .crt/.key file pair 86 | # for each client. A single ca 87 | # file can be used for all clients. 88 | 89 | # Verify server certificate by checking 90 | # that the certicate has the nsCertType 91 | # field set to "server". This is an 92 | # important precaution to protect against 93 | # a potential attack discussed here: 94 | # http://openvpn.net/howto.html#mitm 95 | # 96 | # To use this feature, you will need to generate 97 | # your server certificates with the nsCertType 98 | # field set to "server". The build-key-server 99 | # script in the easy-rsa folder will do this. 100 | ns-cert-type server 101 | 102 | # If a tls-auth key is used on the server 103 | # then every client must also have the key. 104 | ;tls-auth ta.key 1 105 | 106 | # Select a cryptographic cipher. 107 | # If the cipher option is used on the server 108 | # then you must also specify it here. 109 | ;cipher x 110 | 111 | # Enable compression on the VPN link. 112 | # Don't enable this unless it is also 113 | # enabled in the server config file. 114 | comp-lzo 115 | 116 | # Set log file verbosity. 117 | verb 3 118 | 119 | # Silence repeating messages 120 | ;mute 20 121 | -------------------------------------------------------------------------------- /config/openvpn_server/pricewars-producer.ovpn: -------------------------------------------------------------------------------- 1 | ############################################## 2 | # Sample client-side OpenVPN 2.0 config file # 3 | # for connecting to multi-client server. # 4 | # # 5 | # This configuration can be used by multiple # 6 | # clients, however each client should have # 7 | # its own cert and key files. # 8 | # # 9 | # On Windows, you might want to rename this # 10 | # file so it has a .ovpn extension # 11 | ############################################## 12 | 13 | # Specify that we are a client and that we 14 | # will be pulling certain config file directives 15 | # from the server. 16 | client 17 | 18 | # Use the same setting as you are using on 19 | # the server. 20 | # On most systems, the VPN will not function 21 | # unless you partially or fully disable 22 | # the firewall for the TUN/TAP interface. 23 | ;dev tap 24 | dev tun 25 | 26 | # Windows needs the TAP-Win32 adapter name 27 | # from the Network Connections panel 28 | # if you have more than one. On XP SP2, 29 | # you may need to disable the firewall 30 | # for the TAP adapter. 31 | ;dev-node MyTap 32 | 33 | # Are we connecting to a TCP or 34 | # UDP server? Use the same setting as 35 | # on the server. 36 | ;proto tcp 37 | proto udp 38 | 39 | # The hostname/IP and port of the server. 40 | # You can have multiple remote entries 41 | # to load balance between the servers. 42 | remote 185.82.202.248 1194 43 | ;remote my-server-2 1194 44 | 45 | # Choose a random host from the remote 46 | # list for load-balancing. Otherwise 47 | # try hosts in the order specified. 48 | ;remote-random 49 | 50 | # Keep trying indefinitely to resolve the 51 | # host name of the OpenVPN server. Very useful 52 | # on machines which are not permanently connected 53 | # to the internet such as laptops. 54 | resolv-retry infinite 55 | 56 | # Most clients don't need to bind to 57 | # a specific local port number. 58 | nobind 59 | 60 | # Downgrade privileges after initialization (non-Windows only) 61 | ;user nobody 62 | ;group nogroup 63 | 64 | # Try to preserve some state across restarts. 65 | persist-key 66 | persist-tun 67 | 68 | # If you are connecting through an 69 | # HTTP proxy to reach the actual OpenVPN 70 | # server, put the proxy server/IP and 71 | # port number here. See the man page 72 | # if your proxy server requires 73 | # authentication. 74 | ;http-proxy-retry # retry on connection failures 75 | ;http-proxy [proxy server] [proxy port #] 76 | 77 | # Wireless networks often produce a lot 78 | # of duplicate packets. Set this flag 79 | # to silence duplicate packet warnings. 80 | ;mute-replay-warnings 81 | 82 | # SSL/TLS parms. 83 | # See the server config file for more 84 | # description. It's best to use 85 | # a separate .crt/.key file pair 86 | # for each client. A single ca 87 | # file can be used for all clients. 88 | 89 | # Verify server certificate by checking 90 | # that the certicate has the nsCertType 91 | # field set to "server". This is an 92 | # important precaution to protect against 93 | # a potential attack discussed here: 94 | # http://openvpn.net/howto.html#mitm 95 | # 96 | # To use this feature, you will need to generate 97 | # your server certificates with the nsCertType 98 | # field set to "server". The build-key-server 99 | # script in the easy-rsa folder will do this. 100 | ns-cert-type server 101 | 102 | # If a tls-auth key is used on the server 103 | # then every client must also have the key. 104 | ;tls-auth ta.key 1 105 | 106 | # Select a cryptographic cipher. 107 | # If the cipher option is used on the server 108 | # then you must also specify it here. 109 | ;cipher x 110 | 111 | # Enable compression on the VPN link. 112 | # Don't enable this unless it is also 113 | # enabled in the server config file. 114 | comp-lzo 115 | 116 | # Set log file verbosity. 117 | verb 3 118 | 119 | # Silence repeating messages 120 | ;mute 20 121 | -------------------------------------------------------------------------------- /config/openvpn_server/pricewars-marketplace.ovpn: -------------------------------------------------------------------------------- 1 | ############################################## 2 | # Sample client-side OpenVPN 2.0 config file # 3 | # for connecting to multi-client server. # 4 | # # 5 | # This configuration can be used by multiple # 6 | # clients, however each client should have # 7 | # its own cert and key files. # 8 | # # 9 | # On Windows, you might want to rename this # 10 | # file so it has a .ovpn extension # 11 | ############################################## 12 | 13 | # Specify that we are a client and that we 14 | # will be pulling certain config file directives 15 | # from the server. 16 | client 17 | 18 | # Use the same setting as you are using on 19 | # the server. 20 | # On most systems, the VPN will not function 21 | # unless you partially or fully disable 22 | # the firewall for the TUN/TAP interface. 23 | ;dev tap 24 | dev tun 25 | 26 | # Windows needs the TAP-Win32 adapter name 27 | # from the Network Connections panel 28 | # if you have more than one. On XP SP2, 29 | # you may need to disable the firewall 30 | # for the TAP adapter. 31 | ;dev-node MyTap 32 | 33 | # Are we connecting to a TCP or 34 | # UDP server? Use the same setting as 35 | # on the server. 36 | ;proto tcp 37 | proto udp 38 | 39 | # The hostname/IP and port of the server. 40 | # You can have multiple remote entries 41 | # to load balance between the servers. 42 | remote 185.82.202.248 1194 43 | ;remote my-server-2 1194 44 | 45 | # Choose a random host from the remote 46 | # list for load-balancing. Otherwise 47 | # try hosts in the order specified. 48 | ;remote-random 49 | 50 | # Keep trying indefinitely to resolve the 51 | # host name of the OpenVPN server. Very useful 52 | # on machines which are not permanently connected 53 | # to the internet such as laptops. 54 | resolv-retry infinite 55 | 56 | # Most clients don't need to bind to 57 | # a specific local port number. 58 | nobind 59 | 60 | # Downgrade privileges after initialization (non-Windows only) 61 | ;user nobody 62 | ;group nogroup 63 | 64 | # Try to preserve some state across restarts. 65 | persist-key 66 | persist-tun 67 | 68 | # If you are connecting through an 69 | # HTTP proxy to reach the actual OpenVPN 70 | # server, put the proxy server/IP and 71 | # port number here. See the man page 72 | # if your proxy server requires 73 | # authentication. 74 | ;http-proxy-retry # retry on connection failures 75 | ;http-proxy [proxy server] [proxy port #] 76 | 77 | # Wireless networks often produce a lot 78 | # of duplicate packets. Set this flag 79 | # to silence duplicate packet warnings. 80 | ;mute-replay-warnings 81 | 82 | # SSL/TLS parms. 83 | # See the server config file for more 84 | # description. It's best to use 85 | # a separate .crt/.key file pair 86 | # for each client. A single ca 87 | # file can be used for all clients. 88 | 89 | # Verify server certificate by checking 90 | # that the certicate has the nsCertType 91 | # field set to "server". This is an 92 | # important precaution to protect against 93 | # a potential attack discussed here: 94 | # http://openvpn.net/howto.html#mitm 95 | # 96 | # To use this feature, you will need to generate 97 | # your server certificates with the nsCertType 98 | # field set to "server". The build-key-server 99 | # script in the easy-rsa folder will do this. 100 | ns-cert-type server 101 | 102 | # If a tls-auth key is used on the server 103 | # then every client must also have the key. 104 | ;tls-auth ta.key 1 105 | 106 | # Select a cryptographic cipher. 107 | # If the cipher option is used on the server 108 | # then you must also specify it here. 109 | ;cipher x 110 | 111 | # Enable compression on the VPN link. 112 | # Don't enable this unless it is also 113 | # enabled in the server config file. 114 | comp-lzo 115 | 116 | # Set log file verbosity. 117 | verb 3 118 | 119 | # Silence repeating messages 120 | ;mute 20 121 | -------------------------------------------------------------------------------- /docker-compose.yml: -------------------------------------------------------------------------------- 1 | version: '3.1' 2 | services: 3 | postgres: 4 | image: postgres:9.6.5 5 | environment: 6 | - POSTGRES_USER=pricewars 7 | - POSTGRES_PASSWORD=1337 8 | - POSTGRES_DB=marketplace 9 | 10 | redis: 11 | image: redis:latest 12 | ports: 13 | - "6379:6379" 14 | 15 | zookeeper: 16 | image: zookeeper:3.4 17 | ports: 18 | - "2181:2181" 19 | 20 | kafka: 21 | image: wurstmeister/kafka:1.1.0 22 | ports: 23 | - "9092:9092" 24 | - "9093:9093" 25 | environment: 26 | KAFKA_ADVERTISED_LISTENERS: INSIDE://kafka:9092,OUTSIDE://localhost:9093 27 | KAFKA_LISTENERS: INSIDE://:9092,OUTSIDE://:9093 28 | KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT 29 | KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE 30 | KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181 31 | KAFKA_LOG_DIRS: /kafka/kafka-logs 32 | KAFKA_LOG_RETENTION_HOURS: -1 33 | LOG4J_LOGGER_KAFKA: WARN 34 | LOG4J_LOGGER_ORG_APACHE_ZOOKEEPER: WARN 35 | depends_on: 36 | - zookeeper 37 | 38 | kafka-reverse-proxy: 39 | build: ./kafka-reverse-proxy 40 | ports: 41 | - "8001:8001" 42 | command: ["python3", "-u", "LoggerApp.py", "--kafka_url", "kafka:9092"] 43 | depends_on: 44 | - kafka 45 | links: 46 | - kafka 47 | 48 | flink-taskmanager: 49 | build: 50 | context: ./analytics 51 | dockerfile: Dockerfile.flink-taskmanager 52 | expose: 53 | - "6121" 54 | - "6122" 55 | environment: 56 | - JOB_MANAGER_RPC_ADDRESS=flink-jobmanager 57 | - TASK_MANAGER_NUMBER_OF_TASK_SLOTS=8 58 | # disable terminal output of this container 59 | logging: 60 | driver: "none" 61 | command: taskmanager 62 | depends_on: 63 | - flink-jobmanager 64 | links: 65 | - flink-jobmanager 66 | 67 | flink-jobmanager: 68 | build: 69 | context: ./analytics 70 | dockerfile: Dockerfile.flink-jobmanager 71 | ports: 72 | - "8081:8081" 73 | expose: 74 | - "6123" 75 | environment: 76 | - KAFKA_URL=kafka:9092 77 | - JOB_MANAGER_RPC_ADDRESS=flink-jobmanager 78 | depends_on: 79 | - kafka 80 | links: 81 | - kafka 82 | 83 | management-ui: 84 | build: ./management-ui 85 | ports: 86 | - "80:80" 87 | depends_on: 88 | - kafka-reverse-proxy 89 | - marketplace 90 | - producer 91 | 92 | marketplace: 93 | build: ./marketplace 94 | ports: 95 | - "8080:8080" 96 | environment: 97 | - POSTGRES_USER=pricewars 98 | - POSTGRES_PASSWORD=1337 99 | - POSTGRES_HOST=postgres 100 | - POSTGRES_PORT=5432 101 | - POSTGRES_DB=marketplace 102 | - KAFKA_URL=kafka:9092 103 | - REDIS_HOST=redis 104 | - PRICEWARS_PRODUCER_URL=http://producer:3050 105 | depends_on: 106 | - postgres 107 | - redis 108 | - kafka 109 | - producer 110 | links: 111 | - postgres 112 | - redis 113 | - kafka 114 | - producer 115 | 116 | producer: 117 | build: ./producer 118 | ports: 119 | - "3050:3050" 120 | environment: 121 | - KAFKA_URL=kafka:9092 122 | depends_on: 123 | - kafka 124 | 125 | consumer: 126 | build: ./consumer 127 | ports: 128 | - "3000:3000" 129 | environment: 130 | - RAILS_ENV=development 131 | - PRICEWARS_MARKETPLACE_URL=http://marketplace:8080 132 | command: "bash -c 'rm -rf /consumer/tmp/pids/server.pid; bundle exec rails s -b 0.0.0.0'" 133 | depends_on: 134 | - marketplace 135 | links: 136 | - marketplace 137 | 138 | merchant: 139 | build: ./merchant 140 | restart: on-failure:5 141 | ports: 142 | - "5003:5003" 143 | command: python3 -u merchant.py --port 5003 --strategy two_bound --marketplace http://marketplace:8080 --producer http://producer:3050 144 | volumes: 145 | - ./merchant:/merchant 146 | depends_on: 147 | - producer 148 | - marketplace 149 | links: 150 | - producer 151 | - marketplace 152 | 153 | networks: 154 | default: 155 | driver: bridge 156 | ipam: 157 | driver: default 158 | config: 159 | - subnet: 192.168.47.0/24 160 | -------------------------------------------------------------------------------- /helper_scripts/analyze.py: -------------------------------------------------------------------------------- 1 | """ 2 | Analyzes the dump of kafka data, that was created by benchmark.py. 3 | Results (e.g. a merchant's profit and revenue) are saved to a CSV file. 4 | """ 5 | 6 | import argparse 7 | import csv 8 | import datetime 9 | import os 10 | import json 11 | from collections import defaultdict 12 | 13 | import matplotlib 14 | matplotlib.use('Agg') # required for headless plotting 15 | 16 | import matplotlib.pyplot as plt 17 | 18 | 19 | def load_merchant_id_mapping(directory): 20 | with open(os.path.join(directory, 'merchant_id_mapping.json')) as file: 21 | return json.load(file) 22 | 23 | 24 | def analyze_kafka_dump(directory): 25 | merchant_id_mapping = load_merchant_id_mapping(directory) 26 | 27 | revenue = defaultdict(float) 28 | with open(os.path.join(directory, 'kafka', 'buyOffer')) as file: 29 | for event in json.load(file): 30 | revenue[event['merchant_id']] += event['amount'] * event['price'] 31 | 32 | holding_cost = defaultdict(float) 33 | with open(os.path.join(directory, 'kafka', 'holding_cost')) as file: 34 | for event in json.load(file): 35 | holding_cost[event['merchant_id']] += event['cost'] 36 | 37 | order_cost = defaultdict(float) 38 | with open(os.path.join(directory, 'kafka', 'producer')) as file: 39 | for event in json.load(file): 40 | order_cost[event['merchant_id']] += event['billing_amount'] 41 | 42 | profit = {merchant_id: revenue[merchant_id] - holding_cost[merchant_id] - order_cost[merchant_id] 43 | for merchant_id in merchant_id_mapping} 44 | 45 | with open(os.path.join(directory, 'results.csv'), 'w') as file: 46 | writer = csv.writer(file) 47 | writer.writerow(['name', 'revenue', 'holding_cost', 'order_cost', 'profit']) 48 | for merchant_id in sorted(merchant_id_mapping, key=merchant_id_mapping.get): 49 | writer.writerow([merchant_id_mapping[merchant_id], revenue[merchant_id], holding_cost[merchant_id], 50 | order_cost[merchant_id], profit[merchant_id]]) 51 | 52 | create_chart(directory, merchant_id_mapping, 53 | topic='inventory_level', value_name='level', label='Inventory Level', 54 | filename='inventory_levels.png', drawstyle='steps-post') 55 | create_chart(directory, merchant_id_mapping, 56 | topic='profitPerMinute', value_name='profit', label='Profit per Minute', filename='profit_per_minute.png') 57 | create_chart(directory, merchant_id_mapping, 58 | topic='revenuePerMinute', value_name='revenue', label='Revenue per Minute', filename='revenue_per_minute.png') 59 | create_chart(directory, merchant_id_mapping, 60 | topic='profit', value_name='profit', label='Cumulative Profit', filename='cumulative_profit.png') 61 | create_chart(directory, merchant_id_mapping, 62 | topic='cumulativeRevenue', value_name='revenue', label='Cumulative Revenue', filename='cumulative_revenue.png') 63 | 64 | 65 | def parse_timestamps(events): 66 | for event in events: 67 | # TODO: use better conversion; strptime discards timezone 68 | try: 69 | event['timestamp'] = datetime.datetime.strptime(event['timestamp'], '%Y-%m-%dT%H:%M:%S.%fZ') 70 | except ValueError: 71 | # Dates in topic 'inventory_level' have no milliseconds 72 | event['timestamp'] = datetime.datetime.strptime(event['timestamp'], '%Y-%m-%dT%H:%M:%SZ') 73 | 74 | 75 | def create_chart(directory, merchant_id_mapping, topic, value_name, label, filename, **options): 76 | try: 77 | input_file = os.path.join(directory, 'kafka', topic) 78 | events = json.load(open(input_file)) 79 | except FileNotFoundError: 80 | print('Could not find file', input_file) 81 | print('Skip generating graph', filename) 82 | return 83 | 84 | parse_timestamps(events) 85 | fig, ax = plt.subplots() 86 | for merchant_id in merchant_id_mapping: 87 | dates = [event['timestamp'] for event in events if event['merchant_id'] == merchant_id] 88 | values = [event[value_name] for event in events if event['merchant_id'] == merchant_id] 89 | # Cannot plot if no events belong to that merchant 90 | if len(dates) > 0: 91 | ax.plot(dates, values, label=merchant_id_mapping[merchant_id], **options) 92 | plt.xlabel('Time') 93 | plt.ylabel(label) 94 | fig.legend() 95 | ax.xaxis.set_major_formatter(matplotlib.dates.DateFormatter('%Y-%m-%d %H:%M:%S')) 96 | fig.autofmt_xdate() # rotate and align dates 97 | plt.tight_layout() # fit dates into picture 98 | fig.savefig(os.path.join(directory, filename)) 99 | 100 | 101 | def main(): 102 | parser = argparse.ArgumentParser(description='Analyzes data generated by benchmark.py') 103 | parser.add_argument('--directory', '-d', type=str, required=True) 104 | args = parser.parse_args() 105 | analyze_kafka_dump(args.directory) 106 | 107 | 108 | if __name__ == '__main__': 109 | main() 110 | -------------------------------------------------------------------------------- /docs/pricewars_logo.svg: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | -------------------------------------------------------------------------------- /helper_scripts/benchmark.py: -------------------------------------------------------------------------------- 1 | import argparse 2 | import os 3 | from os.path import dirname 4 | import subprocess 5 | import time 6 | import json 7 | import datetime 8 | import random 9 | import shlex 10 | 11 | import requests 12 | from kafka import KafkaConsumer 13 | 14 | from analyze import analyze_kafka_dump 15 | 16 | 17 | class PopenWrapper: 18 | """ 19 | This class is a context manager that wraps subprocess.Popen. 20 | Popen waits until the created process is finished when exiting the context. 21 | This wrapper additionally sends a terminate signal to the program before waiting for it to finish. 22 | """ 23 | 24 | def __init__(self, *args, **kwargs): 25 | self.args = args 26 | self.kwargs = kwargs 27 | 28 | def __enter__(self): 29 | self.process = subprocess.Popen(*self.args, **self.kwargs) 30 | return self.process 31 | 32 | def __exit__(self, *args): 33 | self.process.terminate() 34 | self.process.__exit__(*args) 35 | 36 | 37 | def dump_topic(topic, output_dir, kafka_host): 38 | consumer = KafkaConsumer(topic, 39 | bootstrap_servers=kafka_host, 40 | value_deserializer=lambda m: json.loads(m.decode('utf-8')), 41 | consumer_timeout_ms=2000, 42 | auto_offset_reset='earliest') 43 | 44 | events = [message.value for message in consumer] 45 | with open(os.path.join(output_dir, topic), 'w') as file: 46 | json.dump(events, file) 47 | 48 | 49 | def dump_kafka(output_dir, kafka_host): 50 | kafka_dir = os.path.join(output_dir, 'kafka') 51 | os.mkdir(kafka_dir) 52 | topics = KafkaConsumer(bootstrap_servers=kafka_host).topics() 53 | for topic in topics: 54 | try: 55 | dump_topic(topic, kafka_dir, kafka_host) 56 | except json.decoder.JSONDecodeError: 57 | print('Failed to dump kafka topic', topic) 58 | 59 | 60 | def save_merchant_id_mapping(output_dir, marketplace_url): 61 | merchants_info = requests.get(marketplace_url + '/merchants').json() 62 | merchant_mapping = {} 63 | for merchant_info in merchants_info: 64 | merchant_mapping[merchant_info['merchant_id']] = merchant_info['merchant_name'] 65 | with open(os.path.join(output_dir, 'merchant_id_mapping.json'), 'w') as file: 66 | json.dump(merchant_mapping, file) 67 | 68 | 69 | def clear_containers(pricewars_dir): 70 | subprocess.run(['docker-compose', 'rm', '--stop', '--force'], cwd=pricewars_dir) 71 | 72 | 73 | def set_consumer_ratios(resp, **kwargs): 74 | behaviors_to_use = {} 75 | for k, v in kwargs.items(): 76 | behavior = [b for b in resp['behaviors'] if b['name'] == k] 77 | 78 | if behavior: 79 | behaviors_to_use[k] = v 80 | else: 81 | print(f"Unable to set consumer behaviour '{k}': not implemented by consumer.") 82 | 83 | b_sum = sum(behaviors_to_use.values()) 84 | factor = 100 / b_sum 85 | for k, v in behaviors_to_use.items(): 86 | behaviors_to_use[k] = v*factor 87 | 88 | for b in resp['behaviors']: 89 | if b['name'] in behaviors_to_use: 90 | b['amount'] = int(behaviors_to_use[b['name']]) 91 | else: 92 | b['amount'] = 0 93 | 94 | 95 | def wait_for_marketplace(marketplace_url, timeout=300): 96 | """ 97 | Send requests to the marketplace until there is a response 98 | """ 99 | start = time.time() 100 | while time.time() - start < timeout: 101 | try: 102 | requests.get(marketplace_url) 103 | return 104 | except requests.exceptions.ConnectionError: 105 | pass 106 | raise RuntimeError('Cannot reach marketplace') 107 | 108 | def parse_arguments(): 109 | parser = argparse.ArgumentParser( 110 | description='Runs a simulation on the Pricewars platform', 111 | epilog='Usage example: python3 %(prog)s --duration 5 --output ~/results ' 112 | '--merchants "python3 merchant/merchant.py --port 5000"') 113 | parser.add_argument('--duration', '-d', metavar='MINUTES', type=float, required=True, help='Run that many minutes') 114 | parser.add_argument('--output', '-o', metavar='DIRECTORY', type=str, required=True) 115 | parser.add_argument('--merchants', '-m', metavar='MERCHANT', type=str, nargs='+', required=True, 116 | help='commands to start merchants') 117 | parser.add_argument('--marketplace_url', type=str, default='http://localhost:8080') 118 | parser.add_argument('--consumer_url', type=str, default='http://localhost:3000') 119 | parser.add_argument('--kafka_host', type=str, default='localhost:9093') 120 | parser.add_argument('--holding_cost', type=float, default=0.0) 121 | parser.add_argument('--suppress_debug_output', action="store_true", 122 | help='Show only error messages of the merchants and suppresses all other output') 123 | return parser.parse_args() 124 | 125 | def main(): 126 | pricewars_dir = dirname(dirname(os.path.abspath(__file__))) 127 | args = parse_arguments() 128 | duration_in_minutes = args.duration 129 | 130 | if not os.path.isdir(args.output): 131 | raise RuntimeError('Invalid output directory: ' + args.output) 132 | 133 | output_dir = os.path.join(args.output, datetime.datetime.now().strftime("%Y-%m-%dT%H-%M-%S%z")) 134 | os.mkdir(output_dir) 135 | clear_containers(pricewars_dir) 136 | 137 | # Start all services from the docker-compose file except the merchants. 138 | core_services = ['producer', 'marketplace', 'management-ui', 'flink-taskmanager', 'flink-jobmanager', 139 | 'kafka-reverse-proxy', 'kafka', 'zookeeper', 'redis', 'postgres', 'consumer'] 140 | 141 | stdout_target = subprocess.DEVNULL if args.suppress_debug_output else None 142 | 143 | # Build missing containers and wait until it finished 144 | subprocess.run(['docker-compose', 'up', '--no-start'], stdout=stdout_target) 145 | 146 | with PopenWrapper(['docker-compose', 'up'] + core_services, cwd=pricewars_dir, stdout=stdout_target): 147 | # configure marketplace 148 | wait_for_marketplace(args.marketplace_url) 149 | requests.put(args.marketplace_url + '/holding_cost_rate', json={'rate': args.holding_cost}) 150 | 151 | print('Starting merchants') 152 | merchants = [] 153 | for command in random.sample(args.merchants, len(args.merchants)): 154 | time.sleep(random.random() * 2) 155 | merchants.append(subprocess.Popen(shlex.split(command), stdout=stdout_target)) 156 | 157 | print('Starting consumer') 158 | consumer_settings = requests.get(args.consumer_url + '/setting').json() 159 | 160 | # for more randomized consumer behaviours use something like: 161 | # `prefer_cheap = random.randint(4, 7)` and 162 | # `cheapest_best_quality = random.randint(2, 4)` 163 | #set_consumer_ratios(consumer_settings, prefer_cheap = 1, cheapest_best_quality = 4) 164 | 165 | response = requests.post(args.consumer_url + '/setting', json=consumer_settings) 166 | response.raise_for_status() 167 | 168 | # Run for the given amount of time 169 | print('Run for', duration_in_minutes, 'minutes') 170 | time.sleep(duration_in_minutes * 60) 171 | 172 | print('Stopping consumer') 173 | requests.delete(args.consumer_url + '/setting') 174 | 175 | print('Stopping merchants') 176 | for merchant in merchants: 177 | merchant.terminate() 178 | merchant.wait() 179 | 180 | print('Saving Kafka data') 181 | dump_kafka(output_dir, args.kafka_host) 182 | save_merchant_id_mapping(output_dir, args.marketplace_url) 183 | 184 | analyze_kafka_dump(output_dir) 185 | 186 | 187 | if __name__ == '__main__': 188 | main() 189 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | Price Wars 2 | 3 | # A Simulation Platform for Dynamic Pricing Competition 4 | 5 | 6 | Currently, retailers lack the possibility to test, develop, and evaluate their algorithms appropriately before releasing them into the real world. At the same time, it is challenging for researchers to investigate how pricing strategies interact with each other under heavy competition. 7 | 8 | We built an open platform to simulate dynamic pricing competition allowing both practitioners and researchers to study the effects of automated repricing mechanisms competing with each other using market scenarios that mimic real-world marketplaces. 9 | 10 | We built the platform in a way that one can participate and deploy own merchants with only a few lines of Python code. It allows merchants to deploy the full width of pricing strategies, from simple rule-based strategies to more sophisticated data-driven strategies using machine learning. For practitioners, the platform further provides a possibility to evaluate their pricing strategies appropriately before releasing them in production. 11 | 12 | For more information about the platform and publications, see the our chair's [project site](https://hpi.de/en/plattner/projects/price-wars-an-interactive-simulation-platform.html) on Dynamic Pricing under Competition. 13 | 14 | On the master branch, this repository contains the docker-setup that allows running the simulation locally. On the [gh-pages](https://github.com/hpi-epic/pricewars/tree/gh-pages) branch one can find the [API specification](https://hpi-epic.github.io/pricewars/) of all components. 15 | 16 | ## Application Overview 17 | 18 | **Repositories** 19 | * Management UI: [https://github.com/hpi-epic/pricewars-mgmt-ui](https://github.com/hpi-epic/pricewars-mgmt-ui) 20 | * Consumer: [https://github.com/hpi-epic/pricewars-consumer](https://github.com/hpi-epic/pricewars-consumer) 21 | * Producer: [https://github.com/hpi-epic/pricewars-producer](https://github.com/hpi-epic/pricewars-producer) 22 | * Marketplace: [https://github.com/hpi-epic/pricewars-marketplace](https://github.com/hpi-epic/pricewars-marketplace) 23 | * Merchant: [https://github.com/hpi-epic/pricewars-merchant](https://github.com/hpi-epic/pricewars-merchant) 24 | * Kafka RESTful API: [https://github.com/hpi-epic/pricewars-kafka-rest](https://github.com/hpi-epic/pricewars-kafka-rest) 25 | * Analytics: [https://github.com/hpi-epic/pricewars-analytics](https://github.com/hpi-epic/pricewars-analytics) 26 | 27 | ## Architecture 28 | 29 | ![FMC Architecture Diagram](/docs/modeling/pricewars-architecture.png?raw=true) 30 | 31 | ## Sequence Diagram 32 | 33 | ![Sequence Diagram](/docs/modeling/sequence_diagram_flow.png?raw=true) 34 | 35 | ## Deployment 36 | 37 | ### Requirements 38 | 39 | * [Docker](https://www.docker.com/) 40 | * If you are on Linux read [this](https://docs.docker.com/install/linux/linux-postinstall/) for running docker as non-root user. 41 | * [Docker Compose](https://docs.docker.com/compose/install/) 42 | 43 | ### Setup 44 | Clone the repository and its subrepositories: 45 | 46 | ``` 47 | git clone --recursive git@github.com:hpi-epic/pricewars.git 48 | ``` 49 | 50 | For the next step bring a fast internet line and some time. 51 | This can take up to 30 minutes at the first-time setup. 52 | Build docker images and containers with the following command. 53 | 54 | ``` 55 | cd pricewars 56 | docker-compose build 57 | ``` 58 | 59 | ### Run Price Wars 60 | 61 | The Price Wars platform can be started with: 62 | 63 | ``` 64 | docker-compose up 65 | ``` 66 | 67 | This will start all services and one example merchant. 68 | You can shut down the platform with `CTRL + C` or `docker-compose stop`. 69 | 70 | Warning: There might be routing problems if the docker network overlaps with your local network. 71 | If this is the case, change the ip address in `docker-compose.yml` under the `networks` entry. 72 | 73 | After starting the Pricewars platform with `docker-compose up`, it can be controlled with the [Management UI](http://localhost) 74 | 75 | 1. \[Optional] Configure available products in the [Config/Producer section](http://localhost/index.html#/config/producer) 76 | 2. Start the [Consumer](http://localhost/index.html#/config/consumer) 77 | 3. Merchants are trading products now. The [Dashboard](http://localhost/index.html#/dashboard/overview) shows graphs about sales, profits and more. 78 | 79 | In the [Merchant repository](https://github.com/hpi-epic/pricewars-merchant) you can learn how to build your own merchant and run it on the platform. 80 | 81 | #### Cleaning up containers and existing state 82 | Run the following commands to run the platform in a clean state. 83 | 84 | ``` 85 | docker-compose down 86 | docker-compose up 87 | ``` 88 | 89 | #### Updating the Docker setup 90 | First, stop your running containers. 91 | 92 | ``` 93 | docker-compose stop 94 | ``` 95 | 96 | Update repositories. 97 | ``` 98 | git pull 99 | git submodule update 100 | ``` 101 | 102 | Rebuild all images that have changed: 103 | ``` 104 | docker-compose build 105 | ``` 106 | 107 | #### Help - My Docker Setup is not working as expected! 108 | 109 | ##### Some containers quit unexpectedly: 110 | You can see the status of the containers with `docker-compose ps`. 111 | In case a container is not running, you can see its last logs with `docker-compose logs `. 112 | - __Postgres__: Bring the platform in a clean state with `docker-compose down` and run it again. 113 | - __Zookeeper / Kafka__: If you just stopped some older containers: Wait! There is a timeout for Zookeeper to notice that Kafka has been stopped (timestamp-based, so it works even if Zookeeper is not running). Bring the platform in a clean state with `docker-compose rm --stop` and run it again. 114 | - __Others__: Try to read the logs or read on. 115 | 116 | ##### The command `docker-compose up` is hanging: 117 | - Reset the containers and the network: `docker system prune` (and restart the Docker service). 118 | - Terminate Docker and ensure, that all docker processes are stopped (especially the network service). 119 | - Restart your computer and wait (even though it might be slow) for about five to ten minutes. 120 | - Reset Docker to factory defaults (should be your last attempt, as this requires re-building of all images): 121 | - macOS: Click on "Preferences" > "Reset" > "Reset to factory defaults" 122 | 123 | ### Native 124 | For details regarding the deployment of the component, we kindly refer to the deployment section of the microservice specific README.md file. The links can be found above. 125 | 126 | ## Benchmark Tool 127 | 128 | You can run a benchmark on the Price Wars platform with the benchmark tool [benchmark.py](helper_scripts/benchmark.py). 129 | This tool allows to run the platform in a given configuration for a specific time period. 130 | Afterwards, results of this run are written to the output directory. 131 | 132 | Firstly, install necessary Python libraries: 133 | ``` 134 | python3 -m pip install -r helper_scripts/requirements.txt 135 | ``` 136 | 137 | Example command: 138 | ``` 139 | python3 helper_scripts/benchmark.py --duration 30 --output --merchants 140 | ``` 141 | 142 | This starts the whole platform and two merchants to compete against each other for 30 minutes. 143 | As merchant start command you can use for example: `"python3 merchant/merchant.py --strategy cheapest --port 5000"`. 144 | The quotes must be included. 145 | Run `python3 helper_scripts/benchmark.py --help` to see all arguments. 146 | 147 | ## Developement 148 | There are different ways to develop the containerized marketplace services. These approaches have different trade-offs between setup time and time to run the platform with the new code. 149 | 150 | #### Rebuilding the docker image 151 | This method does not require any extra setup. The workflow is to modify code, then rebuild the docker images. Use the following commands: 152 | ``` 153 | docker-compose down 154 | docker rmi 155 | ``` 156 | On the next start of the platform, the new image will be built. 157 | 158 | Rebuilding an image takes a few seconds up to over a minute depending on the service. 159 | 160 | #### Mounting a volume 161 | If rebuilding the image to too tedious, an alternative approach is mounting the executable (or source code) onto the docker container. This way, the program can be built locally and is used by the container on the next start. 162 | 163 | The directory can be mounted in the `docker-compose.yml` file with the `volumes` option. 164 | 165 | #### Run service natively 166 | It is possible to run a service natively. This removes the docker abstraction and is the fastest way to restart the service with a new version. 167 | 168 | However, the service must be reconfigured to connect to the docker services and vice versa. 169 | 170 | Visit the corresponding subrepository on how to run a service natively and how to configure it. 171 | --------------------------------------------------------------------------------