├── .github └── pull_request_template.md ├── .gitignore ├── README ├── README.md ├── py_tpcc.egg-info ├── PKG-INFO ├── SOURCES.txt ├── dependency_links.txt ├── entry_points.txt ├── not-zip-safe └── top_level.txt ├── pytpcc ├── CONFIG_EXAMPLE ├── MONGODB_EXAMPLE ├── README_v1.1 ├── __init__.py ├── constants.py ├── coordinator.py ├── drivers │ ├── __init__.py │ ├── abstractdriver.py │ ├── cassandradriver.py │ ├── couchdbdriver.py │ ├── csvdriver.py │ ├── hbasedriver.py │ ├── membasedriver.py │ ├── mongodbdriver.py │ ├── redisdriver.py │ ├── scalarisdriver.py │ ├── sqlitedriver.py │ └── tokyocabinetdriver.py ├── message.py ├── runtime │ ├── __init__.py │ ├── executor.py │ └── loader.py ├── tpcc.py ├── tpcc.sql ├── util │ ├── __init__.py │ ├── nurand.py │ ├── rand.py │ ├── results.py │ └── scaleparameters.py ├── verify.js └── worker.py ├── setup.cfg ├── setup.py └── vldb2019 ├── paper.pdf ├── poster.pdf ├── posterHiRes.pdf └── results ├── dbSizes.csv ├── dbSizes.json ├── results.csv └── results.json /.github/pull_request_template.md: -------------------------------------------------------------------------------- 1 | Thanks for submitting a PR to the py-tpcc repo. Please include the following fields (if relevant) prior to submitting your PR. 2 | 3 | **Jira Ticket:** < Ticket Number > 4 | 5 | **Whats Changed:** 6 | High level explanation of changes 7 | 8 | **Patch testing results:** 9 | If applicable, link a patch test showing code changes running successfully 10 | 11 | **Related PRs:** 12 | If applicable, link related PRs 13 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | *.swp 2 | *.pyc 3 | *.config 4 | *.csv 5 | *.out 6 | *.log 7 | *.log.* 8 | temp* 9 | -------------------------------------------------------------------------------- /README: -------------------------------------------------------------------------------- 1 | + ----------------------------------------------- + 2 | + Python TPC-C + 3 | + ----------------------------------------------- + 4 | 5 | 6 | The basic idea is that you will need to create a new driver file that 7 | implements the functions defined in "abstractdriver.py". One function will 8 | load in the tuples into your database for a given table. Then there are five 9 | separate functions that execute the given transaction based on a set of input 10 | parameters. All the work for generating the tuples and the input parameters 11 | for the transactions has been done for you. 12 | 13 | Here's what you need to do to get started: 14 | 15 | (1) Download the source code from Github: 16 | 17 | https://github.com/apavlo/py-tpcc/tree/master/pytpcc 18 | 19 | (2) Create a new file in the 'drivers' directory for your system that follows 20 | the proper naming convention. For example, if your system is 'MongoDB', then 21 | your new file will be called 'mongodbdriver.py' and that file will contain a 22 | new class called 'MongodbDriver' (note the capitalization). 23 | 24 | (3) Inside your class you will need to implement the required functions of 25 | defined in AbstractDriver. There is documentation on what these need to do 26 | also available on Github: 27 | 28 | https://github.com/apavlo/py-tpcc/wiki 29 | 30 | (3) Try running your system. I would start by defining the configuration file 31 | that gets returned with by the 'makeDefaultConfig' function in your driver and 32 | then implement the data loading part first, since that will guide how you 33 | actually execute the transactions. Using 'MongoDB' as an example again, you 34 | can print out the driver's configuration dict to a file: 35 | 36 | $ python ./tpcc.py --print-config mongodb > mongodb.config 37 | 38 | Make any changes you need to 'mongodb.config' (e.g., passwords, hostnames). 39 | Then test the loader: 40 | 41 | $ python ./tpcc.py --no-execute --config=mongodb.config mongodb 42 | 43 | You can use the CSV driver if you want to see what the data or transaction 44 | input parameters will look like. The following command will dump out just the 45 | input to the driver's functions to files in /tmp/tpcc-* 46 | 47 | $ python ./tpcc.py csv 48 | 49 | You can also look at my SqliteDriver implementation to get an idea of what 50 | your transaction implementation functions need to do: 51 | 52 | https://github.com/apavlo/py-tpcc/blob/master/pytpcc/drivers/sqlitedriver.py 53 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | ## TPC-C in Python for MongoDB 2 | Approved in July of 1992, TPC Benchmark C is an on-line transaction processing (OLTP) benchmark. TPC-C is more complex than previous OLTP benchmarks such as TPC-A because of its multiple transaction types, more complex database and overall execution structure. TPC-C involves a mix of five concurrent transactions of different types and complexity either executed on-line or queued for deferred execution. The database is comprised of nine types of tables with a wide range of record and population sizes. TPC-C is measured in transactions per minute (tpmC). While the benchmark portrays the activity of a wholesale supplier, TPC-C is not limited to the activity of any particular business segment, but, rather represents any industry that must manage, sell, or distribute a product or service. 3 | 4 | To learn more about TPC-C, please see the [TPC-C](https://www.tpc.org/tpcc/) documentation. 5 | 6 | This repo is an experimental variant of Python TPC-C implementation based on the original [here](http://github.com/apavlo/py-tpcc). 7 | 8 | The structure of the repo is: 9 | 10 | 1. **pytpcc** - the code for pytpcc with driver (DB) specific code in **drivers** subdirectory. 11 | 2. **vldb2019** - 2019 VLDB paper, poster and results generated from this code 12 | * [VLDB Paper](vldb2019/paper.pdf) 13 | * [VLDB Poster](vldb2019/poster.pdf) 14 | * [Result directory](vldb2019/results) 15 | 16 | All the tests were run using [MongoDB Atlas](https://www.mongodb.com/cloud/atlas?jmp=VLDB2019). 17 | Use code `VLDB2019` to get $150 credit to get started with MongoDB Atlas. 18 | 19 | -------------------------------------------------------------------------------- /py_tpcc.egg-info/PKG-INFO: -------------------------------------------------------------------------------- 1 | Metadata-Version: 1.0 2 | Name: py-tpcc 3 | Version: 0.0dev 4 | Summary: Python implementation of the TPC-C benchmark 5 | Home-page: http://www.cs.brown.edu/~pavlo/ 6 | Author: Andy Pavlo 7 | Author-email: pavlo@cs.brown.edu 8 | License: BSD 9 | Description: UNKNOWN 10 | Platform: UNKNOWN 11 | -------------------------------------------------------------------------------- /py_tpcc.egg-info/SOURCES.txt: -------------------------------------------------------------------------------- 1 | setup.cfg 2 | setup.py 3 | py_tpcc.egg-info/PKG-INFO 4 | py_tpcc.egg-info/SOURCES.txt 5 | py_tpcc.egg-info/dependency_links.txt 6 | py_tpcc.egg-info/entry_points.txt 7 | py_tpcc.egg-info/not-zip-safe 8 | py_tpcc.egg-info/top_level.txt 9 | pytpcc/__init__.py -------------------------------------------------------------------------------- /py_tpcc.egg-info/dependency_links.txt: -------------------------------------------------------------------------------- 1 | 2 | -------------------------------------------------------------------------------- /py_tpcc.egg-info/entry_points.txt: -------------------------------------------------------------------------------- 1 | 2 | # -*- Entry points: -*- 3 | -------------------------------------------------------------------------------- /py_tpcc.egg-info/not-zip-safe: -------------------------------------------------------------------------------- 1 | 2 | -------------------------------------------------------------------------------- /py_tpcc.egg-info/top_level.txt: -------------------------------------------------------------------------------- 1 | pytpcc 2 | -------------------------------------------------------------------------------- /pytpcc/CONFIG_EXAMPLE: -------------------------------------------------------------------------------- 1 | # HypertableDriver Configuration File 2 | # Created 2011-05-02 01:43:37.859545 3 | [hypertable] 4 | 5 | # hostname 6 | host = localhost 7 | 8 | # namespace name 9 | namespace = tpcc 10 | 11 | # port 12 | port = 38080 13 | 14 | #clientnodes splited by spaces 15 | clients =u1 u2 192.168.3.21 16 | #directories of the code on the client node 17 | path =./code/tpcc/py-tpcc/mtpcc 18 | -------------------------------------------------------------------------------- /pytpcc/MONGODB_EXAMPLE: -------------------------------------------------------------------------------- 1 | # MongodbDriver Configuration File 2 | # generate example with default values via `tpcc.py --print-config mongodb` 3 | [mongodb] 4 | # The mongodb connection string or full URI or mongodb+srv string 5 | uri = mongodb://localhost:27017 6 | causal_consistency = True 7 | findandmodify = True 8 | name = tpcc 9 | secondary_reads = True 10 | denormalize = True 11 | retry_writes = True 12 | # user = username 13 | # passwd = passwd 14 | -------------------------------------------------------------------------------- /pytpcc/README_v1.1: -------------------------------------------------------------------------------- 1 | 1. 3 newly added files message.py worker.py coordinator.py. Copy them into the old pytpcc directory. 2 | 2. Coordinator is the main part. Use it like the old tpcc.py. e.g., python coordinator.py --config hypertable.config --clientprocs 5 hypertable 3 | 3. Old argument --clients is replaced with --clientprocs, which specifies how many worker processes you want to run on each client node. 4 | 4. All clientnodes(name or ip) must be specified in the configure file. 5 | 5. The directory of the code on the client side should be specified in the configure file,too. The default address is user's home address, which is default using ssh. 6 | It should remain the same for each client node, which is not a problem for now. 7 | 6. Execnet python module should be installed on each client. Here is how to install it. http://codespeak.net/execnet/install.html 8 | 9 | -------------------------------------------------------------------------------- /pytpcc/__init__.py: -------------------------------------------------------------------------------- 1 | # 2 | -------------------------------------------------------------------------------- /pytpcc/constants.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | # ----------------------------------------------------------------------- 3 | # Copyright (C) 2011 4 | # Andy Pavlo 5 | # http://www.cs.brown.edu/~pavlo/ 6 | # 7 | # Original Java Version: 8 | # Copyright (C) 2008 9 | # Evan Jones 10 | # Massachusetts Institute of Technology 11 | # 12 | # Permission is hereby granted, free of charge, to any person obtaining 13 | # a copy of this software and associated documentation files (the 14 | # "Software"), to deal in the Software without restriction, including 15 | # without limitation the rights to use, copy, modify, merge, publish, 16 | # distribute, sublicense, and/or sell copies of the Software, and to 17 | # permit persons to whom the Software is furnished to do so, subject to 18 | # the following conditions: 19 | # 20 | # The above copyright notice and this permission notice shall be 21 | # included in all copies or substantial portions of the Software. 22 | # 23 | # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, 24 | # EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF 25 | # MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT 26 | # IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR 27 | # OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, 28 | # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR 29 | # OTHER DEALINGS IN THE SOFTWARE. 30 | # ----------------------------------------------------------------------- 31 | 32 | MONEY_DECIMALS = 2 33 | 34 | # Item constants 35 | NUM_ITEMS = 100000 36 | MIN_IM = 1 37 | MAX_IM = 10000 38 | MIN_PRICE = 1.00 39 | MAX_PRICE = 100.00 40 | MIN_I_NAME = 14 41 | MAX_I_NAME = 24 42 | MIN_I_DATA = 26 43 | MAX_I_DATA = 50 44 | 45 | # Warehouse constants 46 | MIN_TAX = 0 47 | MAX_TAX = 0.2000 48 | TAX_DECIMALS = 4 49 | INITIAL_W_YTD = 300000.00 50 | MIN_NAME = 6 51 | MAX_NAME = 10 52 | MIN_STREET = 10 53 | MAX_STREET = 20 54 | MIN_CITY = 10 55 | MAX_CITY = 20 56 | STATE = 2 57 | ZIP_LENGTH = 9 58 | ZIP_SUFFIX = "11111" 59 | 60 | # Stock constants 61 | MIN_QUANTITY = 10 62 | MAX_QUANTITY = 100 63 | DIST = 24 64 | STOCK_PER_WAREHOUSE = 100000 65 | 66 | # District constants 67 | DISTRICTS_PER_WAREHOUSE = 10 68 | INITIAL_D_YTD = 30000.00 # different from Warehouse 69 | INITIAL_NEXT_O_ID = 3001 70 | 71 | # Customer constants 72 | CUSTOMERS_PER_DISTRICT = 3000 73 | INITIAL_CREDIT_LIM = 50000.00 74 | MIN_DISCOUNT = 0.0000 75 | MAX_DISCOUNT = 0.5000 76 | DISCOUNT_DECIMALS = 4 77 | INITIAL_BALANCE = -10.00 78 | INITIAL_YTD_PAYMENT = 10.00 79 | INITIAL_PAYMENT_CNT = 1 80 | INITIAL_DELIVERY_CNT = 0 81 | MIN_FIRST = 6 82 | MAX_FIRST = 10 83 | MIDDLE = "OE" 84 | PHONE = 16 85 | MIN_C_DATA = 300 86 | MAX_C_DATA = 500 87 | GOOD_CREDIT = "GC" 88 | BAD_CREDIT = "BC" 89 | 90 | # Order constants 91 | MIN_CARRIER_ID = 1 92 | MAX_CARRIER_ID = 10 93 | # HACK: This is not strictly correct, but it works 94 | NULL_CARRIER_ID = 0 95 | # o_id < than this value, carrier != null, >= -> carrier == null 96 | NULL_CARRIER_LOWER_BOUND = 2101 97 | MIN_OL_CNT = 5 98 | MAX_OL_CNT = 15 99 | INITIAL_ALL_LOCAL = 1 100 | INITIAL_ORDERS_PER_DISTRICT = 3000 101 | 102 | # Used to generate new order transactions 103 | MAX_OL_QUANTITY = 10 104 | 105 | # Order line constants 106 | INITIAL_QUANTITY = 5 107 | MIN_AMOUNT = 0.01 108 | 109 | # History constants 110 | MIN_DATA = 12 111 | MAX_DATA = 24 112 | INITIAL_AMOUNT = 10.00 113 | 114 | # New order constants 115 | INITIAL_NEW_ORDERS_PER_DISTRICT = 900 116 | 117 | # TPC-C 2.4.3.4 (page 31) says this must be displayed when new order rolls back. 118 | INVALID_ITEM_MESSAGE = "Item number is not valid" 119 | 120 | # Used to generate stock level transactions 121 | MIN_STOCK_LEVEL_THRESHOLD = 10 122 | MAX_STOCK_LEVEL_THRESHOLD = 20 123 | 124 | # Used to generate payment transactions 125 | MIN_PAYMENT = 1.0 126 | MAX_PAYMENT = 5000.0 127 | 128 | # Indicates "brand" items and stock in i_data and s_data. 129 | ORIGINAL_STRING = "ORIGINAL" 130 | 131 | # Table Names 132 | TABLENAME_ITEM = "ITEM" 133 | TABLENAME_WAREHOUSE = "WAREHOUSE" 134 | TABLENAME_DISTRICT = "DISTRICT" 135 | TABLENAME_CUSTOMER = "CUSTOMER" 136 | TABLENAME_STOCK = "STOCK" 137 | TABLENAME_ORDERS = "ORDERS" 138 | TABLENAME_NEW_ORDER = "NEW_ORDER" 139 | TABLENAME_ORDER_LINE = "ORDER_LINE" 140 | TABLENAME_HISTORY = "HISTORY" 141 | 142 | ALL_TABLES = [ 143 | TABLENAME_ITEM, 144 | TABLENAME_WAREHOUSE, 145 | TABLENAME_DISTRICT, 146 | TABLENAME_CUSTOMER, 147 | TABLENAME_STOCK, 148 | TABLENAME_ORDERS, 149 | TABLENAME_NEW_ORDER, 150 | TABLENAME_ORDER_LINE, 151 | TABLENAME_HISTORY, 152 | ] 153 | 154 | # Transaction Types 155 | def enum(*sequential, **named): 156 | enums = dict(map(lambda x: (x, x), sequential)) 157 | # dict(zip(sequential, range(len(sequential))), **named) 158 | return type('Enum', (), enums) 159 | TransactionTypes = enum( 160 | "DELIVERY", 161 | "NEW_ORDER", 162 | "ORDER_STATUS", 163 | "PAYMENT", 164 | "STOCK_LEVEL", 165 | ) 166 | -------------------------------------------------------------------------------- /pytpcc/coordinator.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # -*- coding: utf-8 -*- 3 | # ----------------------------------------------------------------------- 4 | # Copyright (C) 2011 5 | # Andy Pavlo & Yang Lu 6 | # http:##www.cs.brown.edu/~pavlo/ 7 | # 8 | # Permission is hereby granted, free of charge, to any person obtaining 9 | # a copy of this software and associated documentation files (the 10 | # "Software"), to deal in the Software without restriction, including 11 | # without limitation the rights to use, copy, modify, merge, publish, 12 | # distribute, sublicense, and/or sell copies of the Software, and to 13 | # permit persons to whom the Software is furnished to do so, subject to 14 | # the following conditions: 15 | # 16 | # The above copyright notice and this permission notice shall be 17 | # included in all copies or substantial portions of the Software. 18 | # 19 | # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, 20 | # EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF 21 | # MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT 22 | # IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR 23 | # OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, 24 | # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR 25 | # OTHER DEALINGS IN THE SOFTWARE. 26 | # ----------------------------------------------------------------------- 27 | 28 | import sys 29 | import os 30 | import string 31 | import datetime 32 | import logging 33 | import re 34 | import argparse 35 | import glob 36 | import time 37 | import pickle 38 | import execnet 39 | import worker 40 | import message 41 | from ConfigParser import SafeConfigParser 42 | from pprint import pprint, pformat 43 | 44 | from util import * 45 | from runtime import * 46 | import drivers 47 | 48 | logging.basicConfig(level = logging.INFO, 49 | format="%(asctime)s [%(funcName)s:%(lineno)03d] %(levelname)-5s: %(message)s", 50 | datefmt="%m-%d-%Y %H:%M:%S", 51 | filename='results.log') 52 | 53 | ## ============================================== 54 | ## createDriverClass 55 | ## ============================================== 56 | def createDriverClass(name): 57 | full_name = "%sDriver" % name.title() 58 | mod = __import__('drivers.%s' % full_name.lower(), globals(), locals(), [full_name]) 59 | klass = getattr(mod, full_name) 60 | return klass 61 | ## DEF 62 | 63 | ## ============================================== 64 | ## getDrivers 65 | ## ============================================== 66 | def getDrivers(): 67 | drivers = [] 68 | for f in map(lambda x: os.path.basename(x).replace("driver.py", ""), glob.glob("./drivers/*driver.py")): 69 | if f != "abstract": drivers.append(f) 70 | return drivers 71 | ## DEF 72 | 73 | ## ============================================== 74 | ## startLoading 75 | ## ============================================== 76 | def startLoading(scalParameters,args,config,channels): 77 | #Split the warehouses into chunks 78 | procs = len(channels) 79 | w_ids = map(lambda x:[], range(procs)) 80 | for w_id in range(scaleParameters.starting_warehouse, scaleParameters.ending_warehouse+1): 81 | idx = w_id % procs 82 | w_ids[idx].append(w_id) 83 | print w_ids 84 | 85 | load_start=time.time() 86 | for i in range(len(channels)): 87 | m=message.Message(header=message.CMD_LOAD,data=[scalParameters,args,config,w_ids[i]]) 88 | channels[i].send(pickle.dumps(m,-1)) 89 | for ch in channels: 90 | ch.receive() 91 | pass 92 | return time.time()-load_start 93 | 94 | 95 | ## ============================================== 96 | ## startExecution 97 | ## ============================================== 98 | def startExecution(scaleParameters, args, config,channels): 99 | procs = len(channels) 100 | total_results = results.Results() 101 | 102 | for ch in channels: 103 | m=message.Message(header=message.CMD_EXECUTE,data=[scaleParameters,args,config]) 104 | ch.send(pickle.dumps(m,-1)) 105 | for ch in channels: 106 | r=pickle.loads(ch.receive()).data 107 | total_results.append(r) 108 | return total_results 109 | ## DEF 110 | 111 | 112 | ## ============================================== 113 | ## main 114 | ## ============================================== 115 | if __name__ == '__main__': 116 | aparser = argparse.ArgumentParser(description='Python implementation of the TPC-C Benchmark') 117 | aparser.add_argument('system', choices=getDrivers(), 118 | help='Target system driver') 119 | aparser.add_argument('--config', type=file, 120 | help='Path to driver configuration file') 121 | aparser.add_argument('--reset', action='store_true', 122 | help='Instruct the driver to reset the contents of the database') 123 | aparser.add_argument('--scalefactor', default=1, type=float, metavar='SF', 124 | help='Benchmark scale factor') 125 | aparser.add_argument('--warehouses', default=4, type=int, metavar='W', 126 | help='Number of Warehouses') 127 | aparser.add_argument('--duration', default=60, type=int, metavar='D', 128 | help='How long to run the benchmark in seconds') 129 | aparser.add_argument('--ddl', default=os.path.realpath(os.path.join(os.path.dirname(__file__), "tpcc.sql")), 130 | help='Path to the TPC-C DDL SQL file') 131 | ## number of processes per node 132 | aparser.add_argument('--clientprocs', default=1, type=int, metavar='N', 133 | help='Number of processes on each client node.') 134 | 135 | aparser.add_argument('--stop-on-error', action='store_true', 136 | help='Stop the transaction execution when the driver throws an exception.') 137 | aparser.add_argument('--no-load', action='store_true', 138 | help='Disable loading the data') 139 | aparser.add_argument('--no-execute', action='store_true', 140 | help='Disable executing the workload') 141 | aparser.add_argument('--print-config', action='store_true', 142 | help='Print out the default configuration file for the system and exit') 143 | aparser.add_argument('--debug', action='store_true', 144 | help='Enable debug log messages') 145 | args = vars(aparser.parse_args()) 146 | 147 | if args['debug']: logging.getLogger().setLevel(logging.DEBUG) 148 | 149 | ## Create a handle to the target client driver 150 | driverClass = createDriverClass(args['system']) 151 | assert driverClass != None, "Failed to find '%s' class" % args['system'] 152 | driver = driverClass(args['ddl']) 153 | assert driver != None, "Failed to create '%s' driver" % args['system'] 154 | if args['print_config']: 155 | config = driver.makeDefaultConfig() 156 | print driver.formatConfig(config) 157 | print 158 | sys.exit(0) 159 | 160 | ## Load Configuration file 161 | if args['config']: 162 | logging.debug("Loading configuration file '%s'" % args['config']) 163 | cparser = ConfigParser() 164 | cparser.read(os.path.realpath(args['config'].name)) 165 | config = dict(cparser.items(args['system'])) 166 | else: 167 | logging.debug("Using default configuration for %s" % args['system']) 168 | defaultConfig = driver.makeDefaultConfig() 169 | config = dict(map(lambda x: (x, defaultConfig[x][1]), defaultConfig.keys())) 170 | config['reset'] = args['reset'] 171 | config['load'] = False 172 | config['execute'] = False 173 | if config['reset']: logging.info("Reseting database") 174 | driver.loadConfig(config) 175 | logging.info("Initializing TPC-C benchmark using %s" % driver) 176 | 177 | 178 | ##Get a list of clientnodes from configuration file. 179 | clients=[] 180 | channels=[] 181 | assert config['clients']!='' 182 | clients=re.split(r"\s+",str(config['clients'])) 183 | #print clients, len(clients),args['clientprocs'] 184 | ##Create ssh channels to client nodes 185 | for node in clients: 186 | cmd = 'ssh='+ node 187 | cmd += r"//chdir=" 188 | cmd += config['path'] 189 | #print cmd 190 | for i in range(args['clientprocs']): 191 | gw=execnet.makegateway(cmd) 192 | ch=gw.remote_exec(worker) 193 | channels.append(ch) 194 | 195 | ## Create ScaleParameters 196 | scaleParameters = scaleparameters.makeWithScaleFactor(args['warehouses'], args['scalefactor']) 197 | nurand = rand.setNURand(nurand.makeForLoad()) 198 | if args['debug']: logging.debug("Scale Parameters:\n%s" % scaleParameters) 199 | 200 | ## DATA LOADER!!! 201 | load_time = None 202 | if not args['no_load']: 203 | load_time = startLoading(scaleParameters, args, config,channels) 204 | #print load_time 205 | ## IF 206 | 207 | ## WORKLOAD DRIVER!!! 208 | if not args['no_execute']: 209 | results = startExecution(scaleParameters, args, config,channels) 210 | assert results 211 | logging.info(results.show(load_time, driver, len(channels))) 212 | print results.show(load_time, driver, len(channels)) 213 | ## IF 214 | 215 | ## MAIN 216 | -------------------------------------------------------------------------------- /pytpcc/drivers/__init__.py: -------------------------------------------------------------------------------- 1 | # 2 | -------------------------------------------------------------------------------- /pytpcc/drivers/abstractdriver.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | # ----------------------------------------------------------------------- 3 | # Copyright (C) 2011 4 | # Andy Pavlo 5 | # http://www.cs.brown.edu/~pavlo/ 6 | # 7 | # Permission is hereby granted, free of charge, to any person obtaining 8 | # a copy of this software and associated documentation files (the 9 | # "Software"), to deal in the Software without restriction, including 10 | # without limitation the rights to use, copy, modify, merge, publish, 11 | # distribute, sublicense, and/or sell copies of the Software, and to 12 | # permit persons to whom the Software is furnished to do so, subject to 13 | # the following conditions: 14 | # 15 | # The above copyright notice and this permission notice shall be 16 | # included in all copies or substantial portions of the Software. 17 | # 18 | # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, 19 | # EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF 20 | # MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT 21 | # IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR 22 | # OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, 23 | # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR 24 | # OTHER DEALINGS IN THE SOFTWARE. 25 | # ----------------------------------------------------------------------- 26 | 27 | from datetime import datetime 28 | 29 | import constants 30 | 31 | ## ============================================== 32 | ## AbstractDriver 33 | ## ============================================== 34 | class AbstractDriver(object): 35 | def __init__(self, name, ddl): 36 | self.name = name 37 | self.driver_name = "%sDriver" % self.name.title() 38 | self.ddl = ddl 39 | 40 | def __str__(self): 41 | return self.driver_name 42 | 43 | def makeDefaultConfig(self): 44 | """This function needs to be implemented by all sub-classes. 45 | It should return the items that need to be in your implementation's configuration file. 46 | Each item in the list is a triplet containing: ( , , ) 47 | """ 48 | raise NotImplementedError("%s does not implement makeDefaultConfig" % (self.driver_name)) 49 | 50 | def loadConfig(self, config): 51 | """Initialize the driver using the given configuration dict""" 52 | raise NotImplementedError("%s does not implement loadConfig" % (self.driver_name)) 53 | 54 | def formatConfig(self, config): 55 | """Return a formatted version of the config dict that can be used with the --config command line argument""" 56 | ret = "# %s Configuration File\n" % (self.driver_name) 57 | ret += "# Created %s\n" % (datetime.now()) 58 | ret += "[%s]" % self.name 59 | 60 | for name in config.keys(): 61 | desc, default = config[name] 62 | if not default: 63 | default = "" 64 | ret += "\n\n# %s\n%-20s = %s" % (desc, name, default) 65 | return ret 66 | 67 | def loadStart(self): 68 | """Optional callback to indicate to the driver that the data loading phase is about to begin.""" 69 | return None 70 | 71 | def loadFinish(self): 72 | """Optional callback to indicate to the driver that the data loading phase is finished.""" 73 | return None 74 | 75 | def loadFinishItem(self): 76 | """Optional callback to indicate to the driver that the ITEM data has been passed to the driver.""" 77 | return None 78 | 79 | def loadFinishWarehouse(self, w_id): 80 | """Optional callback to indicate to the driver that the data for the given warehouse is finished.""" 81 | return None 82 | 83 | def loadFinishDistrict(self, w_id, d_id): 84 | """Optional callback to indicate to the driver that the data for the given district is finished.""" 85 | return None 86 | 87 | def loadTuples(self, tableName, tuples): 88 | """Load a list of tuples into the target table""" 89 | raise NotImplementedError("%s does not implement loadTuples" % (self.driver_name)) 90 | 91 | def executeStart(self): 92 | """Optional callback before the execution phase starts""" 93 | return None 94 | 95 | def executeFinish(self): 96 | """Callback after the execution phase finishes""" 97 | return None 98 | 99 | def executeTransaction(self, txn, params): 100 | """Execute a transaction based on the given name""" 101 | 102 | if constants.TransactionTypes.DELIVERY == txn: 103 | result = self.doDelivery(params) 104 | elif constants.TransactionTypes.NEW_ORDER == txn: 105 | result = self.doNewOrder(params) 106 | elif constants.TransactionTypes.ORDER_STATUS == txn: 107 | result = self.doOrderStatus(params) 108 | elif constants.TransactionTypes.PAYMENT == txn: 109 | result = self.doPayment(params) 110 | elif constants.TransactionTypes.STOCK_LEVEL == txn: 111 | result = self.doStockLevel(params) 112 | else: 113 | assert False, "Unexpected TransactionType: " + txn 114 | return result 115 | 116 | def doDelivery(self, params): 117 | """Execute DELIVERY Transaction 118 | Parameters Dict: 119 | w_id 120 | o_carrier_id 121 | ol_delivery_d 122 | """ 123 | raise NotImplementedError("%s does not implement doDelivery" % (self.driver_name)) 124 | 125 | def doNewOrder(self, params): 126 | """Execute NEW_ORDER Transaction 127 | Parameters Dict: 128 | w_id 129 | d_id 130 | c_id 131 | o_entry_d 132 | i_ids 133 | i_w_ids 134 | i_qtys 135 | """ 136 | raise NotImplementedError("%s does not implement doNewOrder" % (self.driver_name)) 137 | 138 | def doOrderStatus(self, params): 139 | """Execute ORDER_STATUS Transaction 140 | Parameters Dict: 141 | w_id 142 | d_id 143 | c_id 144 | c_last 145 | """ 146 | raise NotImplementedError("%s does not implement doOrderStatus" % (self.driver_name)) 147 | 148 | def doPayment(self, params): 149 | """Execute PAYMENT Transaction 150 | Parameters Dict: 151 | w_id 152 | d_id 153 | h_amount 154 | c_w_id 155 | c_d_id 156 | c_id 157 | c_last 158 | h_date 159 | """ 160 | raise NotImplementedError("%s does not implement doPayment" % (self.driver_name)) 161 | 162 | def doStockLevel(self, params): 163 | """Execute STOCK_LEVEL Transaction 164 | Parameters Dict: 165 | w_id 166 | d_id 167 | threshold 168 | """ 169 | raise NotImplementedError("%s does not implement doStockLevel" % (self.driver_name)) 170 | ## CLASS 171 | -------------------------------------------------------------------------------- /pytpcc/drivers/cassandradriver.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | # copyright (C) 2011 3 | # Jingxin Feng, Xiaowei Wang 4 | # jxfeng@cs.brown.edu 5 | # xiaowei@cs.brown.edu 6 | # 7 | # Permission is hereby granted, free of charge, to any person obtaining 8 | # a copy of this software and associated documentation files (the 9 | # "Software"), to deal in the Software without restriction, including 10 | # without limitation the rights to use, copy, modify, merge, publish, 11 | # distribute, sublicense, and/or sell copies of the Software, and to 12 | # permit persons to whom the Software is furnished to do so, subject to 13 | # the following conditions: 14 | # 15 | # The above copyright notice and this permission notice shall be 16 | # included in all copies or substantial portions of the Software. 17 | # 18 | # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, 19 | # EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF 20 | # MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT 21 | # IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR 22 | # OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, 23 | # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR 24 | # OTHER DEALINGS IN THE SOFTWARE. 25 | # ----------------------------------------------------------------------- 26 | 27 | 28 | import pycassa 29 | from pycassa.index import * 30 | from pycassa.system_manager import * 31 | 32 | import os 33 | import logging 34 | import commands 35 | import uuid 36 | from pprint import pprint,pformat 37 | import constants 38 | 39 | from abstractdriver import * 40 | ## ============================================== 41 | ## AbstractDriver 42 | ## ============================================== 43 | class CassandraDriver(AbstractDriver): 44 | 45 | DEFAULT_CONFIG = { 46 | "hostname": ("The host address to the Cassandra database","localhost"), 47 | "port": ("Port number",9160), 48 | "name": ("Name","tpcc"), 49 | "keyspace":("Keyspace", "Keyspace1"), 50 | "replicationfactor": ("ReplicationFactor", 1) 51 | } 52 | 53 | 54 | 55 | def __init__(self, ddl): 56 | super(CassandraDriver,self).__init__("cassandra",ddl) 57 | self.conn = None 58 | self.name = "cassandra" 59 | self.database = None 60 | 61 | self.new_ordercf= None 62 | self.orderscf= None 63 | self.order_linecf= None 64 | self.customercf = None 65 | self.warehousecf = None 66 | self.districtcf = None 67 | self.historycf = None 68 | self.stockcf = None 69 | self.itemcf = None 70 | def makeDefaultConfig(self): 71 | return CassandraDriver.DEFAULT_CONFIG 72 | 73 | def loadConfig(self,config): 74 | for key in CassandraDriver.DEFAULT_CONFIG.keys(): 75 | assert key in config, "Missing parameter '%s' in %s configuration" % (key, self.name) 76 | 77 | 78 | connection = str(config["hostname"]+':'+str(config["port"])) 79 | 80 | 81 | keyspace = str(config["keyspace"]) 82 | self.sys = SystemManager(connection) 83 | keyspaces = self.sys.list_keyspaces() 84 | fl = 0 85 | for i in range(len(keyspaces)): 86 | if str(keyspaces[i]) == keyspace: 87 | fl = 1 88 | break 89 | if fl == 0: 90 | self.sys.create_keyspace(keyspace, SIMPLE_STRATEGY,{'replication_factor' : str(config["replicationfactor"])}) 91 | self.sys.create_column_family(keyspace, 'NEW_ORDER', comparator_type = UTF8_TYPE) 92 | self.sys.create_column_family(keyspace, 'ORDERS', comparator_type = UTF8_TYPE) 93 | self.sys.create_column_family(keyspace, 'ORDER_LINE', comparator_type = UTF8_TYPE) 94 | self.sys.create_column_family(keyspace, 'CUSTOMER', comparator_type = UTF8_TYPE) 95 | self.sys.create_column_family(keyspace, 'WAREHOUSE', comparator_type = UTF8_TYPE) 96 | self.sys.create_column_family(keyspace, 'DISTRICT', comparator_type = UTF8_TYPE) 97 | self.sys.create_column_family(keyspace, 'HISTORY', comparator_type = UTF8_TYPE) 98 | self.sys.create_column_family(keyspace, 'STOCK', comparator_type = UTF8_TYPE) 99 | self.sys.create_column_family(keyspace, 'ITEM', comparator_type = UTF8_TYPE) 100 | 101 | 102 | self.sys.alter_column(keyspace,'WAREHOUSE','W_ID',UTF8_TYPE) 103 | self.sys.alter_column(keyspace,'WAREHOUSE','W_NAME',UTF8_TYPE) 104 | self.sys.alter_column(keyspace,'WAREHOUSE','W_STREET_1',UTF8_TYPE) 105 | self.sys.alter_column(keyspace,'WAREHOUSE','W_STREET_2',UTF8_TYPE) 106 | self.sys.alter_column(keyspace,'WAREHOUSE','W_CITY',UTF8_TYPE) 107 | self.sys.alter_column(keyspace,'WAREHOUSE','W_STATE',UTF8_TYPE) 108 | self.sys.alter_column(keyspace,'WAREHOUSE','W_ZIP',UTF8_TYPE) 109 | self.sys.alter_column(keyspace,'WAREHOUSE','W_TAX',UTF8_TYPE) 110 | self.sys.alter_column(keyspace,'WAREHOUSE','W_YTD',UTF8_TYPE) 111 | 112 | 113 | self.sys.alter_column(keyspace,'DISTRICT','D_ID',UTF8_TYPE) 114 | self.sys.alter_column(keyspace,'DISTRICT','D_W_ID',UTF8_TYPE) 115 | self.sys.alter_column(keyspace,'DISTRICT','D_NAME',UTF8_TYPE) 116 | self.sys.alter_column(keyspace,'DISTRICT','D_STREET_1',UTF8_TYPE) 117 | self.sys.alter_column(keyspace,'DISTRICT','D_STREET_2',UTF8_TYPE) 118 | self.sys.alter_column(keyspace,'DISTRICT','D_CITY',UTF8_TYPE) 119 | self.sys.alter_column(keyspace,'DISTRICT','D_STATE',UTF8_TYPE) 120 | self.sys.alter_column(keyspace,'DISTRICT','D_ZIP',UTF8_TYPE) 121 | self.sys.alter_column(keyspace,'DISTRICT','D_TAX',UTF8_TYPE) 122 | self.sys.alter_column(keyspace,'DISTRICT','D_YTD',UTF8_TYPE) 123 | self.sys.alter_column(keyspace,'DISTRICT','D_NEXT_O_ID',UTF8_TYPE) 124 | 125 | 126 | self.sys.alter_column(keyspace,'CUSTOMER','C_ID',UTF8_TYPE) 127 | self.sys.alter_column(keyspace,'CUSTOMER','C_D_ID',UTF8_TYPE) 128 | self.sys.alter_column(keyspace,'CUSTOMER','C_W_ID',UTF8_TYPE) 129 | self.sys.alter_column(keyspace,'CUSTOMER','C_FIRST',UTF8_TYPE) 130 | self.sys.alter_column(keyspace,'CUSTOMER','C_MIDDLE',UTF8_TYPE) 131 | self.sys.alter_column(keyspace,'CUSTOMER','C_LAST',UTF8_TYPE) 132 | self.sys.alter_column(keyspace,'CUSTOMER','C_STREET_1',UTF8_TYPE) 133 | self.sys.alter_column(keyspace,'CUSTOMER','C_STREET_2',UTF8_TYPE) 134 | self.sys.alter_column(keyspace,'CUSTOMER','C_CITY',UTF8_TYPE) 135 | self.sys.alter_column(keyspace,'CUSTOMER','C_STATE',UTF8_TYPE) 136 | self.sys.alter_column(keyspace,'CUSTOMER','C_ZIP',UTF8_TYPE) 137 | self.sys.alter_column(keyspace,'CUSTOMER','C_PHONE',UTF8_TYPE) 138 | self.sys.alter_column(keyspace,'CUSTOMER','C_SINCE',UTF8_TYPE) 139 | self.sys.alter_column(keyspace,'CUSTOMER','C_CREDIT',UTF8_TYPE) 140 | self.sys.alter_column(keyspace,'CUSTOMER','C_CREDIT_LIM',UTF8_TYPE) 141 | self.sys.alter_column(keyspace,'CUSTOMER','C_DISCOUNT',UTF8_TYPE) 142 | self.sys.alter_column(keyspace,'CUSTOMER','C_BALANCE',UTF8_TYPE) 143 | self.sys.alter_column(keyspace,'CUSTOMER','C_YTD_PAYMENT',UTF8_TYPE) 144 | self.sys.alter_column(keyspace,'CUSTOMER','C_PAYMENT_CNT',UTF8_TYPE) 145 | self.sys.alter_column(keyspace,'CUSTOMER','C_DELIVERY_CNT',UTF8_TYPE) 146 | self.sys.alter_column(keyspace,'CUSTOMER','C_DATA',UTF8_TYPE) 147 | 148 | 149 | 150 | 151 | 152 | 153 | 154 | 155 | self.sys.alter_column(keyspace,'HISTORY','H_C_ID',UTF8_TYPE) 156 | self.sys.alter_column(keyspace,'HISTORY','H_C_D_ID',UTF8_TYPE) 157 | self.sys.alter_column(keyspace,'HISTORY','H_C_W_ID',UTF8_TYPE) 158 | self.sys.alter_column(keyspace,'HISTORY','H_D_ID',UTF8_TYPE) 159 | self.sys.alter_column(keyspace,'HISTORY','H_W_ID',UTF8_TYPE) 160 | self.sys.alter_column(keyspace,'HISTORY','H_DATE',UTF8_TYPE) 161 | self.sys.alter_column(keyspace,'HISTORY','H_AMOUNT',UTF8_TYPE) 162 | self.sys.alter_column(keyspace,'HISTORY','H_DATA',UTF8_TYPE) 163 | 164 | 165 | 166 | self.sys.alter_column(keyspace,'NEW_ORDER','NO_O_ID',UTF8_TYPE) 167 | self.sys.alter_column(keyspace,'NEW_ORDER','NO_D_ID',UTF8_TYPE) 168 | self.sys.alter_column(keyspace,'NEW_ORDER','NO_W_ID',UTF8_TYPE) 169 | 170 | self.sys.alter_column(keyspace,'ORDERS','O_ID',UTF8_TYPE) 171 | self.sys.alter_column(keyspace,'ORDERS','O_D_ID',UTF8_TYPE) 172 | self.sys.alter_column(keyspace,'ORDERS','O_W_ID',UTF8_TYPE) 173 | self.sys.alter_column(keyspace,'ORDERS','O_C_ID',UTF8_TYPE) 174 | self.sys.alter_column(keyspace,'ORDERS','O_ENTRY_D',UTF8_TYPE) 175 | self.sys.alter_column(keyspace,'ORDERS','O_CARRIER_ID',UTF8_TYPE) 176 | self.sys.alter_column(keyspace,'ORDERS','O_OL_CNT',UTF8_TYPE) 177 | self.sys.alter_column(keyspace,'ORDERS','O_ALL_LOCAL',UTF8_TYPE) 178 | 179 | 180 | 181 | self.sys.alter_column(keyspace,'ORDER_LINE','OL_O_ID',UTF8_TYPE) 182 | self.sys.alter_column(keyspace,'ORDER_LINE','OL_D_ID',UTF8_TYPE) 183 | self.sys.alter_column(keyspace,'ORDER_LINE','OL_W_ID',UTF8_TYPE) 184 | self.sys.alter_column(keyspace,'ORDER_LINE','OL_NUMBER',UTF8_TYPE) 185 | self.sys.alter_column(keyspace,'ORDER_LINE','OL_I_ID',UTF8_TYPE) 186 | self.sys.alter_column(keyspace,'ORDER_LINE','OL_SUPPLY_W_ID',UTF8_TYPE) 187 | self.sys.alter_column(keyspace,'ORDER_LINE','OL_DELIVERY_D',UTF8_TYPE) 188 | self.sys.alter_column(keyspace,'ORDER_LINE','OL_QUANTITY',UTF8_TYPE) 189 | self.sys.alter_column(keyspace,'ORDER_LINE','OL_AMOUNT',UTF8_TYPE) 190 | self.sys.alter_column(keyspace,'ORDER_LINE','OL_DIST_INFO',UTF8_TYPE) 191 | 192 | 193 | self.sys.alter_column(keyspace,'ITEM','I_ID',UTF8_TYPE) 194 | self.sys.alter_column(keyspace,'ITEM','I_IM_ID',UTF8_TYPE) 195 | self.sys.alter_column(keyspace,'ITEM','I_NAME',UTF8_TYPE) 196 | self.sys.alter_column(keyspace,'ITEM','I_PRICE',UTF8_TYPE) 197 | self.sys.alter_column(keyspace,'ITEM','I_DATA',UTF8_TYPE) 198 | 199 | 200 | self.sys.alter_column(keyspace,'STOCK','S_I_ID',UTF8_TYPE) 201 | self.sys.alter_column(keyspace,'STOCK','S_W_ID',UTF8_TYPE) 202 | self.sys.alter_column(keyspace,'STOCK','S_QUANTITY',UTF8_TYPE) 203 | self.sys.alter_column(keyspace,'STOCK','S_DIST_01',UTF8_TYPE) 204 | self.sys.alter_column(keyspace,'STOCK','S_DIST_02',UTF8_TYPE) 205 | self.sys.alter_column(keyspace,'STOCK','S_DIST_03',UTF8_TYPE) 206 | self.sys.alter_column(keyspace,'STOCK','S_DIST_04',UTF8_TYPE) 207 | self.sys.alter_column(keyspace,'STOCK','S_DIST_05',UTF8_TYPE) 208 | self.sys.alter_column(keyspace,'STOCK','S_DIST_06',UTF8_TYPE) 209 | self.sys.alter_column(keyspace,'STOCK','S_DIST_07',UTF8_TYPE) 210 | self.sys.alter_column(keyspace,'STOCK','S_DIST_08',UTF8_TYPE) 211 | self.sys.alter_column(keyspace,'STOCK','S_DIST_09',UTF8_TYPE) 212 | self.sys.alter_column(keyspace,'STOCK','S_DIST_10',UTF8_TYPE) 213 | self.sys.alter_column(keyspace,'STOCK','S_YTD',UTF8_TYPE) 214 | self.sys.alter_column(keyspace,'STOCK','S_ORDER_CNT',UTF8_TYPE) 215 | self.sys.alter_column(keyspace,'STOCK','S_REMOTE_CNT',UTF8_TYPE) 216 | self.sys.alter_column(keyspace,'STOCK','S_DATA',UTF8_TYPE) 217 | 218 | 219 | self.sys.create_index(keyspace,'CUSTOMER','C_ID', UTF8_TYPE) 220 | self.sys.create_index(keyspace,'CUSTOMER','C_D_ID', UTF8_TYPE) 221 | self.sys.create_index(keyspace,'CUSTOMER','C_W_ID', UTF8_TYPE) 222 | self.sys.create_index(keyspace,'CUSTOMER','C_LAST', UTF8_TYPE) 223 | self.sys.create_index(keyspace,'NEW_ORDER','NO_O_ID', UTF8_TYPE) 224 | self.sys.create_index(keyspace,'NEW_ORDER','NO_D_ID', UTF8_TYPE) 225 | self.sys.create_index(keyspace,'NEW_ORDER','NO_W_ID', UTF8_TYPE) 226 | self.sys.create_index(keyspace,'ORDERS','O_ID', UTF8_TYPE) 227 | self.sys.create_index(keyspace,'ORDERS','O_D_ID', UTF8_TYPE) 228 | self.sys.create_index(keyspace,'ORDERS','O_W_ID', UTF8_TYPE) 229 | self.sys.create_index(keyspace,'ORDERS','O_C_ID', UTF8_TYPE) 230 | self.sys.create_index(keyspace,'ORDER_LINE','OL_O_ID', UTF8_TYPE) 231 | self.sys.create_index(keyspace,'ORDER_LINE','OL_D_ID', UTF8_TYPE) 232 | self.sys.create_index(keyspace,'ORDER_LINE','OL_W_ID', UTF8_TYPE) 233 | self.sys.create_index(keyspace,'STOCK','S_W_ID', UTF8_TYPE) 234 | self.sys.create_index(keyspace,'STOCK','S_QUANTITY', UTF8_TYPE) 235 | 236 | 237 | self.conn = pycassa.connect(str(config["keyspace"]),[connection]) 238 | self.new_ordercf=pycassa.ColumnFamily(self.conn,'NEW_ORDER') 239 | self.orderscf=pycassa.ColumnFamily(self.conn, 'ORDERS') 240 | self.order_linecf=pycassa.ColumnFamily(self.conn, 'ORDER_LINE') 241 | self.customercf=pycassa.ColumnFamily(self.conn, 'CUSTOMER') 242 | self.warehousecf = pycassa.ColumnFamily(self.conn,'WAREHOUSE') 243 | self.districtcf = pycassa.ColumnFamily(self.conn, 'DISTRICT') 244 | self.historycf = pycassa.ColumnFamily(self.conn,'HISTORY') 245 | self.stockcf = pycassa.ColumnFamily(self.conn,'STOCK') 246 | self.itemcf = pycassa.ColumnFamily(self.conn, 'ITEM') 247 | 248 | def loadTuples(self, tableName, tuples): 249 | if len(tuples) == 0: 250 | return 251 | logging.debug("loading") 252 | col_fam = pycassa.ColumnFamily(self.conn, tableName) 253 | if tableName == 'ITEM': 254 | for row in tuples: 255 | row_key = str(row[0]).zfill(5) 256 | i_id = str(row[0]) 257 | i_im_id = str(row[1]) 258 | i_name = str(row[2]) 259 | i_price = str(row[3]) 260 | i_data = str(row[4]) 261 | col_fam.insert(row_key, {'I_ID':i_id, 'I_IM_ID':i_im_id, 'I_NAME':i_name, 'I_PRICE':i_price, 'I_DATA':i_data}) 262 | if tableName == 'WAREHOUSE': 263 | if len(tuples[0])!=9: return 264 | for row in tuples: 265 | row_key = str(row[0]).zfill(5) #w_ID 266 | w_id = str(row[0]) 267 | w_name =str(row[1]) 268 | w_street_1 = str(row[2]) 269 | w_street_2 = str(row[3]) 270 | w_city = str(row[4]) 271 | w_state = str(row[5]) 272 | w_zip = str(row[6]) 273 | w_tax = str(row[7]) 274 | w_ytd = str(row[8]) 275 | col_fam.insert(row_key,{'W_ID':w_id, 'W_NAME':w_name, 'W_STREET_1': w_street_1, 'W_STREET_2': w_street_2, 'W_CITY':w_city, 'W_STATE':w_state, 'W_ZIP':w_zip, 'W_TAX':w_tax, 'W_YTD':w_ytd}) 276 | if tableName == 'CUSTOMER': 277 | for row in tuples: 278 | row_key = str(row[0]).zfill(5)+ str(row[1]).zfill(5)+ str(row[2]).zfill(5) 279 | c_id = str(row[0]) 280 | c_d_id =str(row[1]) 281 | c_w_id =str(row[2]) 282 | c_first =str(row[3]) 283 | c_middle = str(row[4]) 284 | c_last = str(row[5]) 285 | c_street_1 = str(row[6]) 286 | c_street_2 = str(row[7]) 287 | c_city = str(row[8]) 288 | c_state = str(row[9]) 289 | c_zip = str(row[10]) 290 | c_phone = str(row[11]) 291 | c_since = str(row[12]) 292 | c_credit = str(row[13]) 293 | c_credit_lim = str(row[14]) 294 | c_discount = str(row[15]) 295 | c_balance = str(row[16]) 296 | c_ytd_payment = str(row[17]) 297 | c_payment_cnt = str(row[18]) 298 | c_delivery_cnt = str(row[19]) 299 | c_data = str(row[20]) 300 | col_fam.insert(row_key, {'C_ID':c_id, 'C_D_ID':c_d_id, 'C_W_ID':c_w_id, 'C_FIRST':c_first, 'C_MIDDLE':c_middle, 'C_LAST':c_last, 'C_STREET_1':c_street_1,'C_STREET_2':c_street_2, 'C_CITY':c_city, 'C_STATE':c_state, 'C_ZIP':c_zip, 'C_PHONE':c_phone, 'C_SINCE':c_since, 'C_CREDIT':c_credit, 'C_CREDIT_LIM':c_credit_lim, 'C_DISCOUNT':c_discount, 'C_BALANCE':c_balance, 'C_YTD_PAYMENT':c_ytd_payment, 'C_PAYMENT_CNT':c_payment_cnt, 'C_DELIVERY_CNT':c_delivery_cnt, 'C_DATA':c_data}) 301 | 302 | if tableName == 'ORDERS': 303 | for row in tuples: 304 | row_key = str(row[0]).zfill(5)+str(row[1]).zfill(5)+ str(row[2]).zfill(5) 305 | o_id = str(row[0]) 306 | o_d_id = str(row[1]) 307 | o_w_id = str(row[2]) 308 | o_c_id = str(row[3]) 309 | o_entry_d = str(row[4]) 310 | o_carrier_id = str(row[5]) 311 | o_ol_cnt = str(row[6]) 312 | o_all_local = str(row[7]) 313 | col_fam.insert(row_key,{'O_ID':o_id, 'O_D_ID':o_d_id, 'O_W_ID':o_w_id, 'O_C_ID':o_c_id, 'O_ENTRY_D':o_entry_d, 'O_CARRIER_ID':o_carrier_id, 'O_OL_CNT':o_ol_cnt, 'O_ALL_LOCAL':o_all_local}) 314 | 315 | 316 | if tableName == 'STOCK': 317 | for row in tuples: 318 | row_key = str(row[0]).zfill(5)+str(row[1]).zfill(5) 319 | s_i_id = str(row[0]) 320 | s_w_id = str(row[1]) 321 | s_quantity = str(row[2]) 322 | s_dist_01 = str(row[3]) 323 | s_dist_02 = str(row[4]) 324 | s_dist_03 = str(row[5]) 325 | s_dist_04 = str(row[6]) 326 | s_dist_05 = str(row[7]) 327 | s_dist_06 = str(row[8]) 328 | s_dist_07 = str(row[9]) 329 | s_dist_08 = str(row[10]) 330 | s_dist_09 = str(row[11]) 331 | s_dist_10 = str(row[12]) 332 | s_ytd = str(row[13]) 333 | s_order_cnt = str(row[14]) 334 | s_remote_cnt = str(row[15]) 335 | s_data = str(row[16]) 336 | col_fam.insert(row_key,{'S_I_ID':s_i_id, 'S_W_ID':s_w_id, 'S_QUANTITY':s_quantity, 'S_DIST_01':s_dist_01,'S_DIST_02':s_dist_02,'S_DIST_03':s_dist_03,'S_DIST_04':s_dist_04,'S_DIST_05':s_dist_05,'S_DIST_06':s_dist_06,'S_DIST_07':s_dist_07,'S_DIST_08':s_dist_08,'S_DIST_09':s_dist_09,'S_DIST_10':s_dist_10, 'S_YTD': s_ytd, 'S_ORDER_CNT':s_order_cnt, 'S_REMOTE_CNT':s_remote_cnt, 'S_DATA':s_data}) 337 | 338 | if tableName == 'DISTRICT': 339 | for row in tuples: 340 | row_key = str(row[0]).zfill(5)+str(row[1]).zfill(5) 341 | d_id = str(row[0]) 342 | d_w_id = str(row[1]) 343 | d_name = str(row[2]) 344 | d_street_1 = str(row[3]) 345 | d_street_2 = str(row[4]) 346 | d_city = str(row[5]) 347 | d_state = str(row[6]) 348 | d_zip = str(row[7]) 349 | d_tax =str(row[8]) 350 | d_ytd = str(row[9]) 351 | d_next_o_id = str(row[10]) 352 | col_fam.insert(row_key,{'D_ID':d_id, 'D_W_ID':d_w_id, 'D_NAME':d_name, 'D_STREET_1':d_street_1, 'D_STREET_2':d_street_2,'D_CITY':d_city, 'D_STATE':d_state, 'D_ZIP':d_zip, 'D_TAX':d_tax, 'D_YTD':d_ytd, 'D_NEXT_O_ID':d_next_o_id}) 353 | 354 | if tableName == 'NEW_ORDER': 355 | for row in tuples: 356 | row_key = str(row[0]).zfill(5)+str(row[1]).zfill(5)+str(row[2]).zfill(5) 357 | no_o_id = str(row[0]) 358 | no_d_id = str(row[1]) 359 | no_w_id = str(row[2]) 360 | col_fam.insert(row_key,{'NO_O_ID':no_o_id, 'NO_D_ID':no_d_id, 'NO_W_ID':no_w_id}) 361 | if tableName == 'ORDER_LINE': 362 | for row in tuples: 363 | row_key = str(row[0]).zfill(5)+str(row[1]).zfill(5)+str(row[2]).zfill(5)+str(row[3]).zfill(5) 364 | ol_o_id = str(row[0]) 365 | ol_d_id = str(row[1]) 366 | ol_w_id = str(row[2]) 367 | ol_number = str(row[3]) 368 | ol_i_id = str(row[4]) 369 | ol_supply_w_id = str(row[5]) 370 | ol_delivery_d = str(row[6]) 371 | ol_quantity = str(row[7]) 372 | ol_amount = str(row[8]) 373 | ol_dist_info = str(row[9]) 374 | col_fam.insert(row_key,{'OL_O_ID':ol_o_id, 'OL_D_ID':ol_d_id, 'OL_W_ID':ol_w_id, 'OL_NUMBER':ol_number, 'OL_I_ID':ol_i_id, 'OL_SUPPLY_W_ID':ol_supply_w_id, 'OL_DELIVERY_D': ol_delivery_d, 'OL_QUANTITY':ol_quantity,'OL_AMOUNT':ol_amount, 'OL_DIST_INFO':ol_dist_info}) 375 | 376 | if tableName == 'HISTORY': 377 | for i in range(len(tuples)): 378 | #row_key = str(i) 379 | row_key = str(uuid.uuid1()) 380 | h_c_id = str(tuples[i][0]) 381 | h_c_d_id = str(tuples[i][1]) 382 | h_c_w_id = str(tuples[i][2]) 383 | h_d_id = str(tuples[i][3]) 384 | h_w_id = str(tuples[i][4]) 385 | h_date = str(tuples[i][5]) 386 | h_amount = str(tuples[i][6]) 387 | h_data = str(tuples[i][7]) 388 | col_fam.insert(row_key, {'H_C_ID':h_c_id, 'H_C_D_ID':h_c_d_id, 'H_C_W_ID':h_c_w_id, 'H_D_ID':h_d_id, 'H_W_ID':h_w_id, 'H_DATE':h_date,'H_AMOUNT':h_amount, 'H_DATA':h_data}) 389 | # print tableName+'--' + str(len(tuples)) 390 | 391 | def loadFinish(self): 392 | logging.info("Commiting changes to database") 393 | 394 | 395 | 396 | 397 | ##----------------------------------- 398 | ## doDelivery 399 | ##---------------------------------- 400 | def doDelivery(self, params): 401 | logging.debug("do delivery") 402 | 403 | w_id = params["w_id"] 404 | o_carrier_id = params["o_carrier_id"] 405 | ol_delivery_d = params["ol_delivery_d"] 406 | 407 | 408 | 409 | result = [ ] 410 | for d_id in range(1, constants.DISTRICTS_PER_WAREHOUSE+1): 411 | did_expr = create_index_expression('NO_D_ID',str(d_id)) 412 | wid_expr = create_index_expression('NO_W_ID',str(w_id)) 413 | clause = create_index_clause([did_expr,wid_expr],count=1) 414 | newOrder=self.new_ordercf.get_indexed_slices(clause) 415 | flag=0 416 | for key, column in newOrder: 417 | #print column 418 | no_o_id=column['NO_O_ID'] 419 | flag=1 420 | if flag==0: 421 | continue 422 | if int(no_o_id)<=-1: 423 | continue 424 | if no_o_id==None: 425 | continue 426 | # print no_o_id 427 | # print d_id 428 | # print w_id 429 | orders_rowkey=no_o_id.zfill(5)+str(d_id).zfill(5)+str(w_id).zfill(5) 430 | #print orders_rowkey 431 | o=self.orderscf.get(orders_rowkey) 432 | 433 | 434 | c_id=str(o['O_C_ID']) 435 | 436 | oid_expr = create_index_expression('OL_O_ID',str(no_o_id)) 437 | did_expr = create_index_expression('OL_D_ID',str(d_id)) 438 | wid_expr = create_index_expression('OL_W_ID',str(w_id)) 439 | 440 | clause = create_index_clause([oid_expr,did_expr,wid_expr],count=100000) 441 | orderLine=self.order_linecf.get_indexed_slices(clause) 442 | 443 | ol_total=0 444 | for key, column in orderLine: 445 | ol_total+=float(column['OL_AMOUNT']) 446 | 447 | deleteKey=no_o_id.zfill(5)+str(d_id).zfill(5)+str(w_id).zfill(5) 448 | self.new_ordercf.remove(deleteKey) 449 | self.orderscf.insert(deleteKey, {'O_CARRIER_ID': str(o_carrier_id)}) 450 | self.order_linecf.insert(deleteKey,{'OL_DELIVERY_D':str(ol_delivery_d)}) 451 | 452 | c=self.customercf.get(str(c_id).zfill(5)+str(d_id).zfill(5)+str(w_id).zfill(5)) 453 | old_balance=float(c['C_BALANCE']) 454 | new_balance=str(old_balance+ol_total) 455 | self.customercf.insert(str(c_id).zfill(5)+str(d_id).zfill(5)+str(w_id).zfill(5),{'C_BALANCE': str(new_balance)}) 456 | 457 | result.append((str(d_id),str(no_o_id))) 458 | ##for 459 | 460 | return result 461 | ##----------------------------------- 462 | ## doNewOrder 463 | ##----------------------------------- 464 | 465 | def doNewOrder(self, params): 466 | logging.debug("do new order") 467 | w_id = params["w_id"] 468 | d_id = params["d_id"] 469 | c_id = params["c_id"] 470 | o_entry_d = params["o_entry_d"] 471 | i_ids = params["i_ids"] 472 | i_w_ids = params["i_w_ids"] 473 | i_qtys = params["i_qtys"] 474 | 475 | 476 | assert len(i_ids) > 0 477 | assert len(i_ids) ==len(i_w_ids) 478 | assert len(i_ids) ==len(i_qtys) 479 | 480 | all_local = True 481 | items = [ ] 482 | for i in range(len(i_ids)): 483 | all_local = all_local and i_w_ids[i] == w_id 484 | ol_i_id = i_ids[i] 485 | itm=self.itemcf.get(str(ol_i_id).zfill(5), columns=['I_PRICE','I_NAME','I_DATA']) 486 | items.append(itm) 487 | assert len(items)==len(i_ids) 488 | 489 | 490 | for itm in items: 491 | if len(itm)==0: 492 | return 493 | 494 | 495 | 496 | 497 | 498 | #getWarehouseTaxRate 499 | w_tax_c = self.warehousecf.get(str(w_id).zfill(5),columns=['W_TAX']) 500 | w_tax =float(w_tax_c['W_TAX']) 501 | #getDistrict 502 | row_key = str(d_id).zfill(5) +str(w_id).zfill(5) 503 | o=self.districtcf.get(row_key, columns=['D_TAX','D_NEXT_O_ID']) 504 | d_tax = float(o['D_TAX']) 505 | #incrementNextOrderId 506 | d_next_o_id = int(o['D_NEXT_O_ID']) 507 | 508 | #getCustomer 509 | row_key = str(c_id).zfill(5) +str(d_id).zfill(5)+str(w_id).zfill(5) 510 | customer_info = self.customercf.get(row_key,columns=['C_DISCOUNT','C_LAST','C_CREDIT']) 511 | c_discount=float(customer_info['C_DISCOUNT']) 512 | 513 | o_carrier_id = constants.NULL_CARRIER_ID 514 | ol_cnt = len(i_ids) 515 | 516 | #incrementNextOrderId 517 | row_key = str(d_id).zfill(5)+str(w_id).zfill(5) 518 | self.districtcf.insert(row_key,{'D_NEXT_O_ID':str(d_next_o_id+1)}) 519 | 520 | #createOrder 521 | 522 | order_rowkey=str(d_next_o_id).zfill(5)+str(d_id).zfill(5)+str(w_id).zfill(5) 523 | # print "d_next_o_id " +str(d_next_o_id) 524 | # print "d_id "+str(d_id) 525 | # print "order_rowkey " + order_rowkey 526 | self.orderscf.insert(order_rowkey,{'O_ID':str(d_next_o_id), 'O_D_ID':str(d_id), 'O_W_ID':str(w_id), 'O_C_ID':str(c_id), 'O_ENTRY_D':str(o_entry_d), 'O_CARRIER_ID':str(o_carrier_id), 'O_OL_CNT':str(ol_cnt), 'O_ALL_LOCAL':str(all_local)}) 527 | 528 | #createNewOrder 529 | neworder_rowkey=str(d_next_o_id).zfill(5)+str(d_id).zfill(5)+str(w_id).zfill(5) 530 | # print 'neworder_rowkey ' + neworder_rowkey 531 | self.new_ordercf.insert(neworder_rowkey, {'NO_O_ID':str(d_next_o_id), 'NO_D_ID':str(d_id), 'NO_W_ID':str(w_id)}) 532 | #getItemInfo 533 | total = 0 534 | item_data = [ ] 535 | for i in range(len(i_ids)): 536 | itemInfo = items[i] 537 | i_name = itemInfo['I_NAME'] 538 | i_data = itemInfo['I_DATA'] 539 | i_price =float(itemInfo['I_PRICE']) 540 | 541 | #"getStockInfo": "SELECT S_QUANTITY, S_DATA, S_YTD, S_ORDER_CNT, S_REMOTE_CNT, S_DIST_%02d FROM STOCK WHERE S_I_ID = ? AND S_W_ID = ?", # d_id, ol_i_id, ol_supply_w_id 542 | ol_i_id = i_ids[i] 543 | ol_number = i+1 544 | ol_supply_w_id = i_w_ids[i] 545 | ol_quantity = i_qtys[i] 546 | 547 | stockInfo = self.stockcf.get(str(ol_i_id).zfill(5)+str(ol_supply_w_id).zfill(5)) 548 | #"updateStock": "UPDATE STOCK SET S_QUANTITY = ?, S_YTD = ?, S_ORDER_CNT = ?, S_REMOTE_CNT = ? WHERE S_I_ID = ? AND S_W_ID = ?", # s_quantity, s_order_cnt, s_remote_cnt, ol_i_id, ol_supply_w_id 549 | if len(stockInfo)==0: 550 | logging.warn("No STOCK record for (ol_i_id=%d, ol_supply_w_id=%d)" % (ol_i_id, ol_supply_w_id)) 551 | continue 552 | s_quantity = int(stockInfo['S_QUANTITY']) 553 | s_ytd = int(stockInfo['S_YTD']) 554 | s_order_cnt = int(stockInfo['S_ORDER_CNT']) 555 | s_remote_cnt = int(stockInfo['S_REMOTE_CNT']) 556 | s_data = stockInfo['S_DATA'] 557 | if d_id < 10: 558 | s_dist_col='S_DIST_'+'0'+str(d_id) 559 | else: 560 | s_dist_col='S_DIST_'+str(d_id) 561 | s_dist_xx = stockInfo[s_dist_col] 562 | 563 | 564 | ## Update stock 565 | s_ytd += ol_quantity 566 | if s_quantity >= ol_quantity + 10: 567 | s_quantity = s_quantity - ol_quantity 568 | else: 569 | s_quantity = s_quantity + 91 - ol_quantity 570 | s_order_cnt += 1 571 | if ol_supply_w_id != w_id: s_remote_cnt += 1 572 | self.stockcf.insert(str(ol_i_id).zfill(5)+str(ol_supply_w_id).zfill(5),{'S_QUANTITY':str(s_quantity), 'S_YTD':str(s_ytd), 'S_ORDER_CNT':str(s_order_cnt) , 'S_REMOTE_CNT':str(s_remote_cnt)}) 573 | 574 | ##"createOrderLine": "INSERT INTO ORDER_LINE (OL_O_ID, OL_D_ID, OL_W_ID, OL_NUMBER, OL_I_ID, OL_SUPPLY_W_ID, OL_DELIVERY_D, OL_QUANTITY, OL_AMOUNT, OL_DIST_INFO) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)", # o_id, d_id, w_id, ol_number, ol_i_id, ol_supply_w_id, ol_quantity, ol_amount, ol_dist_info 575 | if i_data.find(constants.ORIGINAL_STRING) != -1 and s_data.find(constants.ORIGINAL_STRING)!= -1: 576 | brand_generic = 'B' 577 | else: 578 | brand_generic = 'G' 579 | ol_amount = ol_quantity * i_price 580 | total += ol_amount 581 | 582 | orderline_rowkey=str(d_next_o_id).zfill(5)+str(d_id).zfill(5)+str(w_id).zfill(5) 583 | self.order_linecf.insert(orderline_rowkey,{'OL_O_ID': str(d_next_o_id), 'OL_D_ID':str(d_id), 'OL_W_ID':str(w_id), 'OL_NUMBER':str(ol_number), 'OL_I_ID':str(ol_i_id), 'OL_SUPPLY_W_ID':str(ol_supply_w_id), 'OL_DELIVERY_D': str(o_entry_d), 'OL_QUANTITY':str(ol_quantity),'OL_AMOUNT':str(ol_amount), 'OL_DIST_INFO':str(s_dist_xx)}) 584 | item_data.append( (i_name, s_quantity, brand_generic,i_price, ol_amount) ) 585 | total *= (1 - c_discount) * (1 + w_tax + d_tax) 586 | misc = [ (w_tax, d_tax, d_next_o_id, total) ] 587 | return [ customer_info, misc, item_data ] 588 | ##---------------------------- 589 | ## doPayment 590 | ##---------------------------- 591 | 592 | def doPayment(self, params): 593 | logging.debug("do payment") 594 | w_id = params["w_id"] 595 | d_id = params["d_id"] 596 | h_amount = params["h_amount"] 597 | c_w_id = params["c_w_id"] 598 | c_d_id = params["c_d_id"] 599 | c_id = params["c_id"] 600 | c_last = params["c_last"] 601 | h_date = params["h_date"] 602 | 603 | 604 | if c_id != None: 605 | #getCustomerByCustomerId 606 | row_key = str(c_id).zfill(5) +str(d_id).zfill(5)+str(w_id).zfill(5) 607 | customer = self.customercf.get(row_key) 608 | assert len(customer)>0 609 | c_balance = float(str(customer['C_BALANCE']))- h_amount 610 | c_ytd_payment = float(str(customer['C_YTD_PAYMENT'])) + h_amount 611 | c_payment_cnt = int(str(customer['C_PAYMENT_CNT']))+1 612 | c_data = str(customer['C_DATA']) 613 | c_credit = str(customer['C_CREDIT']) 614 | else: 615 | #getCustomerByLastName 616 | c_1_expr = create_index_expression('C_W_ID',str(w_id)) 617 | c_2_expr = create_index_expression('C_D_ID',str(d_id)) 618 | c_3_expr = create_index_expression('C_LAST',str(c_last)) 619 | clause = create_index_clause([c_1_expr,c_2_expr,c_3_expr],count=1000) 620 | 621 | newcustomer = self.customercf.get_indexed_slices(clause) 622 | firstnames=[] 623 | 624 | namecnt=0 625 | for key, column in newcustomer: 626 | firstnames.append(column['C_FIRST']) 627 | namecnt+=1 628 | # print namecnt 629 | index = (namecnt-1)/2 630 | firstname=firstnames[index] 631 | c_4_expr = create_index_expression('C_LAST',str(c_last)) 632 | clause = create_index_clause([c_1_expr,c_2_expr,c_3_expr,c_4_expr],count=1) 633 | newcustomer = self.customercf.get_indexed_slices(clause) 634 | for key, column in newcustomer: 635 | c_id = column['C_ID'] 636 | c_balance = float(column['C_BALANCE'])- h_amount 637 | c_ytd_payment = float(column['C_YTD_PAYMENT']) + h_amount 638 | c_payment_cnt = int(column['C_PAYMENT_CNT'])+1 639 | c_data = column['C_DATA'] 640 | c_credit =column['C_CREDIT'] 641 | row_key = str(c_id).zfill(5) +str(d_id).zfill(5)+str(w_id).zfill(5) 642 | customer = self.customercf.get(row_key) 643 | warehouse = self.warehousecf.get(str(w_id).zfill(5)) 644 | district = self.districtcf.get(str(d_id).zfill(5)+str(w_id).zfill(5)) 645 | 646 | self.warehousecf.insert(str(w_id).zfill(5),{'W_YTD':str(float(warehouse['W_YTD'])+h_amount)}) 647 | 648 | self.districtcf.insert(str(d_id).zfill(5)+str(w_id).zfill(5),{'D_YTD': str(float(district['D_YTD'])+h_amount)}) 649 | 650 | if c_credit == constants.BAD_CREDIT: 651 | newData = " ".join(map(str, [c_id, c_d_id, c_w_id, d_id, w_id, h_amount])) 652 | c_data = (newData + "|" + c_data) 653 | if len(c_data) > constants.MAX_C_DATA: c_data = c_data[:constants.MAX_C_DATA] 654 | self.customercf.insert(str(c_id).zfill(5)+str(c_d_id).zfill(5)+str(c_w_id).zfill(5),{ 'C_BALANCE' : str(c_balance), 'C_YTD_PAYMENT':str(c_ytd_payment) , 'C_PAYMENT_CNT':str(c_payment_cnt), 'C_DATA' : str(c_data)}) 655 | else: 656 | c_data = "" 657 | self.customercf.insert(str(c_id).zfill(5)+str(c_d_id).zfill(5)+str(c_w_id).zfill(5),{ 'C_BALANCE' : str(c_balance), 'C_YTD_PAYMENT':str(c_ytd_payment) , 'C_PAYMENT_CNT':str(c_payment_cnt)}) 658 | h_data= "%s %s" % (warehouse['W_NAME'], district['D_NAME']) 659 | self.historycf.insert(str(uuid.uuid1()), {'H_C_ID':str(c_id), 'H_C_D_ID':str(c_d_id), 'H_C_W_ID':str(c_w_id), 'H_D_ID':str(d_id), 'H_W_ID':str(w_id), 'H_DATE':str(h_date),'H_AMOUNT':str(h_amount), 'H_DATA':str(h_data)}) 660 | return [warehouse, district, customer] 661 | ##----------------------------------- 662 | ## doOrderStatus 663 | ##----------------------------------- 664 | def doOrderStatus(self, params): 665 | logging.info("do orderStatus") 666 | w_id = params["w_id"] 667 | d_id = params["d_id"] 668 | c_id = params["c_id"] 669 | c_last = params["c_last"] 670 | 671 | assert w_id, pformat(params) 672 | assert d_id, pformat(params) 673 | 674 | 675 | if c_id == None: 676 | last_expr = create_index_expression('C_LAST',str(c_last)) 677 | did_expr = create_index_expression('C_D_ID',str(d_id)) 678 | wid_expr = create_index_expression('C_W_ID',str(w_id)) 679 | clause = create_index_clause([last_expr,did_expr,wid_expr],count=10000) 680 | all_customers=self.customercf.get_indexed_slices(clause) 681 | first_names=[ ] 682 | c_ids=[] 683 | namecnt=0 684 | for key, column in all_customers: 685 | first_names.append(column['C_FIRST']) 686 | c_ids.append(column['C_ID']) 687 | namecnt = namecnt+1 688 | namecnt=len(first_names) 689 | assert namecnt>0 690 | index=(namecnt-1)/2 691 | first_name=first_names[index] 692 | assert first_name!=None 693 | c_id=c_ids[index] 694 | assert c_id!=None 695 | 696 | key1=str(c_id).zfill(5)+str(d_id).zfill(5)+str(w_id).zfill(5) 697 | res1=self.customercf.get(key1) 698 | customer=[res1['C_ID'],res1['C_FIRST'],res1['C_MIDDLE'],res1['C_LAST'],res1['C_BALANCE']] 699 | 700 | cid_expr = create_index_expression('O_C_ID',str(c_id)) 701 | did_expr = create_index_expression('O_D_ID',str(d_id)) 702 | wid_expr = create_index_expression('O_W_ID',str(w_id)) 703 | clause = create_index_clause([cid_expr,did_expr,wid_expr],count=100000) 704 | all_orders=self.orderscf.get_indexed_slices(clause) 705 | 706 | last_order_oid=0 707 | order=[] 708 | for key, column in all_orders: 709 | if int(column['O_ID'])>last_order_oid: 710 | last_order_oid=int(column['O_ID']) 711 | if last_order_oid>0: 712 | o=self.orderscf.get(str(last_order_oid).zfill(5)+str(d_id).zfill(5)+str(w_id).zfill(5)) 713 | order=[o['O_ID'],o['O_CARRIER_ID'],o['O_ENTRY_D']] 714 | 715 | 716 | orderLines = [] 717 | if last_order_oid>0: 718 | oid_expr = create_index_expression('OL_O_ID',str(last_order_oid)) 719 | did_expr = create_index_expression('OL_D_ID',str(d_id)) 720 | wid_expr = create_index_expression('OL_W_ID',str(w_id)) 721 | clause = create_index_clause([oid_expr,did_expr,wid_expr]) 722 | orderLine=self.order_linecf.get_indexed_slices(clause) 723 | for key, column in orderLine: 724 | orderLines.append([column['OL_SUPPLY_W_ID'],column['OL_I_ID'],column['OL_QUANTITY'],column['OL_AMOUNT'],column['OL_DELIVERY_D']]) 725 | 726 | return [ customer, order, orderLines ] 727 | 728 | ##---------------------------- 729 | ## doStockLevel 730 | ##---------------------------- 731 | 732 | 733 | def doStockLevel(self, params): 734 | logging.info("do stocklevel") 735 | w_id = params["w_id"] 736 | d_id = params["d_id"] 737 | threshold = params["threshold"] 738 | 739 | 740 | #"getOId": "SELECT D_NEXT_O_ID FROM DISTRICT WHERE D_W_ID = ? AND D_ID = ?", 741 | d = self.districtcf.get(str(d_id).zfill(5)+str(w_id).zfill(5),columns=['D_NEXT_O_ID']) 742 | assert d 743 | #getStockCount 744 | o_id = d['D_NEXT_O_ID'] 745 | 746 | 747 | s_q_expr = create_index_expression('S_QUANTITY',str(threshold), LT) 748 | s_q_expr2 = create_index_expression('S_W_ID',str(w_id)) 749 | clause = create_index_clause([s_q_expr,s_q_expr2]) 750 | newstock = self.stockcf.get_indexed_slices(clause) 751 | 752 | 753 | ol_expr = create_index_expression('OL_W_ID',str(w_id)) 754 | ol_expr2 = create_index_expression('OL_D_ID',str(d_id)) 755 | ol_expr3 = create_index_expression('OL_O_ID',str(o_id),LT) 756 | ol_expr4 = create_index_expression('OL_O_ID', str(int(o_id)-20),GTE) 757 | clause2 = create_index_clause([ol_expr,ol_expr2]) 758 | neworderline = self.order_linecf.get_indexed_slices(clause2) 759 | 760 | count = 0 761 | for key, column in newstock: 762 | for key2, column2 in neworderline: 763 | tmp1 = column['S_I_ID'] 764 | s_i_id = int(tmp1) 765 | tmp2 = column2['OL_I_ID'] 766 | ol_i_id = int(tmp2) 767 | if s_i_id == ol_i_id: 768 | count= count+1 769 | 770 | return count 771 | 772 | -------------------------------------------------------------------------------- /pytpcc/drivers/csvdriver.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | # ----------------------------------------------------------------------- 3 | # Copyright (C) 2011 4 | # Andy Pavlo 5 | # http://www.cs.brown.edu/~pavlo/ 6 | # 7 | # Permission is hereby granted, free of charge, to any person obtaining 8 | # a copy of this software and associated documentation files (the 9 | # "Software"), to deal in the Software without restriction, including 10 | # without limitation the rights to use, copy, modify, merge, publish, 11 | # distribute, sublicense, and/or sell copies of the Software, and to 12 | # permit persons to whom the Software is furnished to do so, subject to 13 | # the following conditions: 14 | # 15 | # The above copyright notice and this permission notice shall be 16 | # included in all copies or substantial portions of the Software. 17 | # 18 | # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, 19 | # EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF 20 | # MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT 21 | # IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR 22 | # OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, 23 | # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR 24 | # OTHER DEALINGS IN THE SOFTWARE. 25 | # ----------------------------------------------------------------------- 26 | 27 | import os 28 | import csv 29 | from datetime import datetime 30 | from pprint import pprint,pformat 31 | 32 | from abstractdriver import * 33 | 34 | ## ============================================== 35 | ## CSVDriver 36 | ## ============================================== 37 | class CsvDriver(AbstractDriver): 38 | DEFAULT_CONFIG = { 39 | "table_directory": ("The path to the directory to store the table CSV files", "/tmp/tpcc-tables" ), 40 | "txn_directory": ("The path to the directory to store the txn CSV files", "/tmp/tpcc-txns" ), 41 | } 42 | 43 | def __init__(self, ddl): 44 | super(CsvDriver, self).__init__("csv", ddl) 45 | self.table_directory = None 46 | self.table_outputs = { } 47 | self.txn_directory = None 48 | self.txn_outputs = { } 49 | self.txn_params = { } 50 | ## DEF 51 | 52 | def makeDefaultConfig(self): 53 | return CsvDriver.DEFAULT_CONFIG 54 | ## DEF 55 | 56 | def loadConfig(self, config): 57 | for key in CsvDriver.DEFAULT_CONFIG.keys(): 58 | assert key in config, "Missing parameter '%s' in %s configuration" % (key, self.name) 59 | 60 | self.table_directory = config["table_directory"] 61 | assert self.table_directory 62 | if not os.path.exists(self.table_directory): os.makedirs(self.table_directory) 63 | 64 | self.txn_directory = config["txn_directory"] 65 | assert self.txn_directory 66 | if not os.path.exists(self.txn_directory): os.makedirs(self.txn_directory) 67 | ## DEF 68 | 69 | def loadTuples(self, tableName, tuples): 70 | if not tableName in self.table_outputs: 71 | path = os.path.join(self.table_directory, "%s.csv" % tableName) 72 | self.table_outputs[tableName] = csv.writer(open(path, 'wb'), quoting=csv.QUOTE_ALL) 73 | ## IF 74 | self.table_outputs[tableName].writerows(tuples) 75 | ## DEF 76 | 77 | def executeTransaction(self, txn, params): 78 | if not txn in self.txn_outputs: 79 | path = os.path.join(self.txn_directory, "%s.csv" % txn) 80 | self.txn_outputs[txn] = csv.writer(open(path, 'wb'), quoting=csv.QUOTE_ALL) 81 | self.txn_params[txn] = params.keys()[:] 82 | self.txn_outputs[txn].writerow(["Timestamp"] + self.txn_params[txn]) 83 | ## IF 84 | row = [datetime.now()] + [params[k] for k in self.txn_params[txn]] 85 | self.txn_outputs[txn].writerow(row) 86 | ## DEF 87 | ## CLASS 88 | 89 | 90 | -------------------------------------------------------------------------------- /pytpcc/drivers/sqlitedriver.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | # ----------------------------------------------------------------------- 3 | # Copyright (C) 2011 4 | # Andy Pavlo 5 | # http://www.cs.brown.edu/~pavlo/ 6 | # 7 | # Original Java Version: 8 | # Copyright (C) 2008 9 | # Evan Jones 10 | # Massachusetts Institute of Technology 11 | # 12 | # Permission is hereby granted, free of charge, to any person obtaining 13 | # a copy of this software and associated documentation files (the 14 | # "Software"), to deal in the Software without restriction, including 15 | # without limitation the rights to use, copy, modify, merge, publish, 16 | # distribute, sublicense, and/or sell copies of the Software, and to 17 | # permit persons to whom the Software is furnished to do so, subject to 18 | # the following conditions: 19 | # 20 | # The above copyright notice and this permission notice shall be 21 | # included in all copies or substantial portions of the Software. 22 | # 23 | # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, 24 | # EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF 25 | # MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT 26 | # IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR 27 | # OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, 28 | # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR 29 | # OTHER DEALINGS IN THE SOFTWARE. 30 | # ----------------------------------------------------------------------- 31 | 32 | from __future__ import with_statement 33 | 34 | import os 35 | import sqlite3 36 | import logging 37 | import commands 38 | from pprint import pprint,pformat 39 | 40 | import constants 41 | from abstractdriver import * 42 | 43 | TXN_QUERIES = { 44 | "DELIVERY": { 45 | "getNewOrder": "SELECT NO_O_ID FROM NEW_ORDER WHERE NO_D_ID = ? AND NO_W_ID = ? AND NO_O_ID > -1 LIMIT 1", # 46 | "deleteNewOrder": "DELETE FROM NEW_ORDER WHERE NO_D_ID = ? AND NO_W_ID = ? AND NO_O_ID = ?", # d_id, w_id, no_o_id 47 | "getCId": "SELECT O_C_ID FROM ORDERS WHERE O_ID = ? AND O_D_ID = ? AND O_W_ID = ?", # no_o_id, d_id, w_id 48 | "updateOrders": "UPDATE ORDERS SET O_CARRIER_ID = ? WHERE O_ID = ? AND O_D_ID = ? AND O_W_ID = ?", # o_carrier_id, no_o_id, d_id, w_id 49 | "updateOrderLine": "UPDATE ORDER_LINE SET OL_DELIVERY_D = ? WHERE OL_O_ID = ? AND OL_D_ID = ? AND OL_W_ID = ?", # o_entry_d, no_o_id, d_id, w_id 50 | "sumOLAmount": "SELECT SUM(OL_AMOUNT) FROM ORDER_LINE WHERE OL_O_ID = ? AND OL_D_ID = ? AND OL_W_ID = ?", # no_o_id, d_id, w_id 51 | "updateCustomer": "UPDATE CUSTOMER SET C_BALANCE = C_BALANCE + ? WHERE C_ID = ? AND C_D_ID = ? AND C_W_ID = ?", # ol_total, c_id, d_id, w_id 52 | }, 53 | "NEW_ORDER": { 54 | "getWarehouseTaxRate": "SELECT W_TAX FROM WAREHOUSE WHERE W_ID = ?", # w_id 55 | "getDistrict": "SELECT D_TAX, D_NEXT_O_ID FROM DISTRICT WHERE D_ID = ? AND D_W_ID = ?", # d_id, w_id 56 | "incrementNextOrderId": "UPDATE DISTRICT SET D_NEXT_O_ID = ? WHERE D_ID = ? AND D_W_ID = ?", # d_next_o_id, d_id, w_id 57 | "getCustomer": "SELECT C_DISCOUNT, C_LAST, C_CREDIT FROM CUSTOMER WHERE C_W_ID = ? AND C_D_ID = ? AND C_ID = ?", # w_id, d_id, c_id 58 | "createOrder": "INSERT INTO ORDERS (O_ID, O_D_ID, O_W_ID, O_C_ID, O_ENTRY_D, O_CARRIER_ID, O_OL_CNT, O_ALL_LOCAL) VALUES (?, ?, ?, ?, ?, ?, ?, ?)", # d_next_o_id, d_id, w_id, c_id, o_entry_d, o_carrier_id, o_ol_cnt, o_all_local 59 | "createNewOrder": "INSERT INTO NEW_ORDER (NO_O_ID, NO_D_ID, NO_W_ID) VALUES (?, ?, ?)", # o_id, d_id, w_id 60 | "getItemInfo": "SELECT I_PRICE, I_NAME, I_DATA FROM ITEM WHERE I_ID = ?", # ol_i_id 61 | "getStockInfo": "SELECT S_QUANTITY, S_DATA, S_YTD, S_ORDER_CNT, S_REMOTE_CNT, S_DIST_%02d FROM STOCK WHERE S_I_ID = ? AND S_W_ID = ?", # d_id, ol_i_id, ol_supply_w_id 62 | "updateStock": "UPDATE STOCK SET S_QUANTITY = ?, S_YTD = ?, S_ORDER_CNT = ?, S_REMOTE_CNT = ? WHERE S_I_ID = ? AND S_W_ID = ?", # s_quantity, s_order_cnt, s_remote_cnt, ol_i_id, ol_supply_w_id 63 | "createOrderLine": "INSERT INTO ORDER_LINE (OL_O_ID, OL_D_ID, OL_W_ID, OL_NUMBER, OL_I_ID, OL_SUPPLY_W_ID, OL_DELIVERY_D, OL_QUANTITY, OL_AMOUNT, OL_DIST_INFO) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)", # o_id, d_id, w_id, ol_number, ol_i_id, ol_supply_w_id, ol_quantity, ol_amount, ol_dist_info 64 | }, 65 | 66 | "ORDER_STATUS": { 67 | "getCustomerByCustomerId": "SELECT C_ID, C_FIRST, C_MIDDLE, C_LAST, C_BALANCE FROM CUSTOMER WHERE C_W_ID = ? AND C_D_ID = ? AND C_ID = ?", # w_id, d_id, c_id 68 | "getCustomersByLastName": "SELECT C_ID, C_FIRST, C_MIDDLE, C_LAST, C_BALANCE FROM CUSTOMER WHERE C_W_ID = ? AND C_D_ID = ? AND C_LAST = ? ORDER BY C_FIRST", # w_id, d_id, c_last 69 | "getLastOrder": "SELECT O_ID, O_CARRIER_ID, O_ENTRY_D FROM ORDERS WHERE O_W_ID = ? AND O_D_ID = ? AND O_C_ID = ? ORDER BY O_ID DESC LIMIT 1", # w_id, d_id, c_id 70 | "getOrderLines": "SELECT OL_SUPPLY_W_ID, OL_I_ID, OL_QUANTITY, OL_AMOUNT, OL_DELIVERY_D FROM ORDER_LINE WHERE OL_W_ID = ? AND OL_D_ID = ? AND OL_O_ID = ?", # w_id, d_id, o_id 71 | }, 72 | 73 | "PAYMENT": { 74 | "getWarehouse": "SELECT W_NAME, W_STREET_1, W_STREET_2, W_CITY, W_STATE, W_ZIP FROM WAREHOUSE WHERE W_ID = ?", # w_id 75 | "updateWarehouseBalance": "UPDATE WAREHOUSE SET W_YTD = W_YTD + ? WHERE W_ID = ?", # h_amount, w_id 76 | "getDistrict": "SELECT D_NAME, D_STREET_1, D_STREET_2, D_CITY, D_STATE, D_ZIP FROM DISTRICT WHERE D_W_ID = ? AND D_ID = ?", # w_id, d_id 77 | "updateDistrictBalance": "UPDATE DISTRICT SET D_YTD = D_YTD + ? WHERE D_W_ID = ? AND D_ID = ?", # h_amount, d_w_id, d_id 78 | "getCustomerByCustomerId": "SELECT C_ID, C_FIRST, C_MIDDLE, C_LAST, C_STREET_1, C_STREET_2, C_CITY, C_STATE, C_ZIP, C_PHONE, C_SINCE, C_CREDIT, C_CREDIT_LIM, C_DISCOUNT, C_BALANCE, C_YTD_PAYMENT, C_PAYMENT_CNT, C_DATA FROM CUSTOMER WHERE C_W_ID = ? AND C_D_ID = ? AND C_ID = ?", # w_id, d_id, c_id 79 | "getCustomersByLastName": "SELECT C_ID, C_FIRST, C_MIDDLE, C_LAST, C_STREET_1, C_STREET_2, C_CITY, C_STATE, C_ZIP, C_PHONE, C_SINCE, C_CREDIT, C_CREDIT_LIM, C_DISCOUNT, C_BALANCE, C_YTD_PAYMENT, C_PAYMENT_CNT, C_DATA FROM CUSTOMER WHERE C_W_ID = ? AND C_D_ID = ? AND C_LAST = ? ORDER BY C_FIRST", # w_id, d_id, c_last 80 | "updateBCCustomer": "UPDATE CUSTOMER SET C_BALANCE = ?, C_YTD_PAYMENT = ?, C_PAYMENT_CNT = ?, C_DATA = ? WHERE C_W_ID = ? AND C_D_ID = ? AND C_ID = ?", # c_balance, c_ytd_payment, c_payment_cnt, c_data, c_w_id, c_d_id, c_id 81 | "updateGCCustomer": "UPDATE CUSTOMER SET C_BALANCE = ?, C_YTD_PAYMENT = ?, C_PAYMENT_CNT = ? WHERE C_W_ID = ? AND C_D_ID = ? AND C_ID = ?", # c_balance, c_ytd_payment, c_payment_cnt, c_w_id, c_d_id, c_id 82 | "insertHistory": "INSERT INTO HISTORY VALUES (?, ?, ?, ?, ?, ?, ?, ?)", 83 | }, 84 | 85 | "STOCK_LEVEL": { 86 | "getOId": "SELECT D_NEXT_O_ID FROM DISTRICT WHERE D_W_ID = ? AND D_ID = ?", 87 | "getStockCount": """ 88 | SELECT COUNT(DISTINCT(OL_I_ID)) FROM ORDER_LINE, STOCK 89 | WHERE OL_W_ID = ? 90 | AND OL_D_ID = ? 91 | AND OL_O_ID < ? 92 | AND OL_O_ID >= ? 93 | AND S_W_ID = ? 94 | AND S_I_ID = OL_I_ID 95 | AND S_QUANTITY < ? 96 | """, 97 | }, 98 | } 99 | 100 | 101 | ## ============================================== 102 | ## SqliteDriver 103 | ## ============================================== 104 | class SqliteDriver(AbstractDriver): 105 | DEFAULT_CONFIG = { 106 | "database": ("The path to the SQLite database", "/tmp/tpcc.db" ), 107 | } 108 | 109 | def __init__(self, ddl): 110 | super(SqliteDriver, self).__init__("sqlite", ddl) 111 | self.database = None 112 | self.conn = None 113 | self.cursor = None 114 | 115 | ## ---------------------------------------------- 116 | ## makeDefaultConfig 117 | ## ---------------------------------------------- 118 | def makeDefaultConfig(self): 119 | return SqliteDriver.DEFAULT_CONFIG 120 | 121 | ## ---------------------------------------------- 122 | ## loadConfig 123 | ## ---------------------------------------------- 124 | def loadConfig(self, config): 125 | for key in SqliteDriver.DEFAULT_CONFIG.keys(): 126 | assert key in config, "Missing parameter '%s' in %s configuration" % (key, self.name) 127 | 128 | self.database = str(config["database"]) 129 | 130 | if config["reset"] and os.path.exists(self.database): 131 | logging.debug("Deleting database '%s'" % self.database) 132 | os.unlink(self.database) 133 | 134 | if os.path.exists(self.database) == False: 135 | logging.debug("Loading DDL file '%s'" % (self.ddl)) 136 | ## HACK 137 | cmd = "sqlite3 %s < %s" % (self.database, self.ddl) 138 | (result, output) = commands.getstatusoutput(cmd) 139 | assert result == 0, cmd + "\n" + output 140 | ## IF 141 | 142 | self.conn = sqlite3.connect(self.database) 143 | self.cursor = self.conn.cursor() 144 | 145 | ## ---------------------------------------------- 146 | ## loadTuples 147 | ## ---------------------------------------------- 148 | def loadTuples(self, tableName, tuples): 149 | if len(tuples) == 0: return 150 | 151 | p = ["?"]*len(tuples[0]) 152 | sql = "INSERT INTO %s VALUES (%s)" % (tableName, ",".join(p)) 153 | self.cursor.executemany(sql, tuples) 154 | 155 | logging.debug("Loaded %d tuples for tableName %s" % (len(tuples), tableName)) 156 | return 157 | 158 | ## ---------------------------------------------- 159 | ## loadFinish 160 | ## ---------------------------------------------- 161 | def loadFinish(self): 162 | logging.info("Commiting changes to database") 163 | self.conn.commit() 164 | 165 | ## ---------------------------------------------- 166 | ## doDelivery 167 | ## ---------------------------------------------- 168 | def doDelivery(self, params): 169 | q = TXN_QUERIES["DELIVERY"] 170 | 171 | w_id = params["w_id"] 172 | o_carrier_id = params["o_carrier_id"] 173 | ol_delivery_d = params["ol_delivery_d"] 174 | 175 | result = [ ] 176 | for d_id in range(1, constants.DISTRICTS_PER_WAREHOUSE+1): 177 | self.cursor.execute(q["getNewOrder"], [d_id, w_id]) 178 | newOrder = self.cursor.fetchone() 179 | if newOrder == None: 180 | ## No orders for this district: skip it. Note: This must be reported if > 1% 181 | continue 182 | assert len(newOrder) > 0 183 | no_o_id = newOrder[0] 184 | 185 | self.cursor.execute(q["getCId"], [no_o_id, d_id, w_id]) 186 | c_id = self.cursor.fetchone()[0] 187 | 188 | self.cursor.execute(q["sumOLAmount"], [no_o_id, d_id, w_id]) 189 | ol_total = self.cursor.fetchone()[0] 190 | 191 | self.cursor.execute(q["deleteNewOrder"], [d_id, w_id, no_o_id]) 192 | self.cursor.execute(q["updateOrders"], [o_carrier_id, no_o_id, d_id, w_id]) 193 | self.cursor.execute(q["updateOrderLine"], [ol_delivery_d, no_o_id, d_id, w_id]) 194 | 195 | # These must be logged in the "result file" according to TPC-C 2.7.2.2 (page 39) 196 | # We remove the queued time, completed time, w_id, and o_carrier_id: the client can figure 197 | # them out 198 | # If there are no order lines, SUM returns null. There should always be order lines. 199 | assert ol_total != None, "ol_total is NULL: there are no order lines. This should not happen" 200 | assert ol_total > 0.0 201 | 202 | self.cursor.execute(q["updateCustomer"], [ol_total, c_id, d_id, w_id]) 203 | 204 | result.append((d_id, no_o_id)) 205 | ## FOR 206 | 207 | self.conn.commit() 208 | return result 209 | 210 | ## ---------------------------------------------- 211 | ## doNewOrder 212 | ## ---------------------------------------------- 213 | def doNewOrder(self, params): 214 | q = TXN_QUERIES["NEW_ORDER"] 215 | 216 | w_id = params["w_id"] 217 | d_id = params["d_id"] 218 | c_id = params["c_id"] 219 | o_entry_d = params["o_entry_d"] 220 | i_ids = params["i_ids"] 221 | i_w_ids = params["i_w_ids"] 222 | i_qtys = params["i_qtys"] 223 | 224 | assert len(i_ids) > 0 225 | assert len(i_ids) == len(i_w_ids) 226 | assert len(i_ids) == len(i_qtys) 227 | 228 | all_local = True 229 | items = [ ] 230 | for i in range(len(i_ids)): 231 | ## Determine if this is an all local order or not 232 | all_local = all_local and i_w_ids[i] == w_id 233 | self.cursor.execute(q["getItemInfo"], [i_ids[i]]) 234 | items.append(self.cursor.fetchone()) 235 | assert len(items) == len(i_ids) 236 | 237 | ## TPCC defines 1% of neworder gives a wrong itemid, causing rollback. 238 | ## Note that this will happen with 1% of transactions on purpose. 239 | for item in items: 240 | if len(item) == 0: 241 | ## TODO Abort here! 242 | return 243 | ## FOR 244 | 245 | ## ---------------- 246 | ## Collect Information from WAREHOUSE, DISTRICT, and CUSTOMER 247 | ## ---------------- 248 | self.cursor.execute(q["getWarehouseTaxRate"], [w_id]) 249 | w_tax = self.cursor.fetchone()[0] 250 | 251 | self.cursor.execute(q["getDistrict"], [d_id, w_id]) 252 | district_info = self.cursor.fetchone() 253 | d_tax = district_info[0] 254 | d_next_o_id = district_info[1] 255 | 256 | self.cursor.execute(q["getCustomer"], [w_id, d_id, c_id]) 257 | customer_info = self.cursor.fetchone() 258 | c_discount = customer_info[0] 259 | 260 | ## ---------------- 261 | ## Insert Order Information 262 | ## ---------------- 263 | ol_cnt = len(i_ids) 264 | o_carrier_id = constants.NULL_CARRIER_ID 265 | 266 | self.cursor.execute(q["incrementNextOrderId"], [d_next_o_id + 1, d_id, w_id]) 267 | self.cursor.execute(q["createOrder"], [d_next_o_id, d_id, w_id, c_id, o_entry_d, o_carrier_id, ol_cnt, all_local]) 268 | self.cursor.execute(q["createNewOrder"], [d_next_o_id, d_id, w_id]) 269 | 270 | ## ---------------- 271 | ## Insert Order Item Information 272 | ## ---------------- 273 | item_data = [ ] 274 | total = 0 275 | for i in range(len(i_ids)): 276 | ol_number = i + 1 277 | ol_supply_w_id = i_w_ids[i] 278 | ol_i_id = i_ids[i] 279 | ol_quantity = i_qtys[i] 280 | 281 | itemInfo = items[i] 282 | i_name = itemInfo[1] 283 | i_data = itemInfo[2] 284 | i_price = itemInfo[0] 285 | 286 | self.cursor.execute(q["getStockInfo"] % (d_id), [ol_i_id, ol_supply_w_id]) 287 | stockInfo = self.cursor.fetchone() 288 | if len(stockInfo) == 0: 289 | logging.warn("No STOCK record for (ol_i_id=%d, ol_supply_w_id=%d)" % (ol_i_id, ol_supply_w_id)) 290 | continue 291 | s_quantity = stockInfo[0] 292 | s_ytd = stockInfo[2] 293 | s_order_cnt = stockInfo[3] 294 | s_remote_cnt = stockInfo[4] 295 | s_data = stockInfo[1] 296 | s_dist_xx = stockInfo[5] # Fetches data from the s_dist_[d_id] column 297 | 298 | ## Update stock 299 | s_ytd += ol_quantity 300 | if s_quantity >= ol_quantity + 10: 301 | s_quantity = s_quantity - ol_quantity 302 | else: 303 | s_quantity = s_quantity + 91 - ol_quantity 304 | s_order_cnt += 1 305 | 306 | if ol_supply_w_id != w_id: s_remote_cnt += 1 307 | 308 | self.cursor.execute(q["updateStock"], [s_quantity, s_ytd, s_order_cnt, s_remote_cnt, ol_i_id, ol_supply_w_id]) 309 | 310 | if i_data.find(constants.ORIGINAL_STRING) != -1 and s_data.find(constants.ORIGINAL_STRING) != -1: 311 | brand_generic = 'B' 312 | else: 313 | brand_generic = 'G' 314 | 315 | ## Transaction profile states to use "ol_quantity * i_price" 316 | ol_amount = ol_quantity * i_price 317 | total += ol_amount 318 | 319 | self.cursor.execute(q["createOrderLine"], [d_next_o_id, d_id, w_id, ol_number, ol_i_id, ol_supply_w_id, o_entry_d, ol_quantity, ol_amount, s_dist_xx]) 320 | 321 | ## Add the info to be returned 322 | item_data.append( (i_name, s_quantity, brand_generic, i_price, ol_amount) ) 323 | ## FOR 324 | 325 | ## Commit! 326 | self.conn.commit() 327 | 328 | ## Adjust the total for the discount 329 | #print "c_discount:", c_discount, type(c_discount) 330 | #print "w_tax:", w_tax, type(w_tax) 331 | #print "d_tax:", d_tax, type(d_tax) 332 | total *= (1 - c_discount) * (1 + w_tax + d_tax) 333 | 334 | ## Pack up values the client is missing (see TPC-C 2.4.3.5) 335 | misc = [ (w_tax, d_tax, d_next_o_id, total) ] 336 | 337 | return [ customer_info, misc, item_data ] 338 | 339 | ## ---------------------------------------------- 340 | ## doOrderStatus 341 | ## ---------------------------------------------- 342 | def doOrderStatus(self, params): 343 | q = TXN_QUERIES["ORDER_STATUS"] 344 | 345 | w_id = params["w_id"] 346 | d_id = params["d_id"] 347 | c_id = params["c_id"] 348 | c_last = params["c_last"] 349 | 350 | assert w_id, pformat(params) 351 | assert d_id, pformat(params) 352 | 353 | if c_id != None: 354 | self.cursor.execute(q["getCustomerByCustomerId"], [w_id, d_id, c_id]) 355 | customer = self.cursor.fetchone() 356 | else: 357 | # Get the midpoint customer's id 358 | self.cursor.execute(q["getCustomersByLastName"], [w_id, d_id, c_last]) 359 | all_customers = self.cursor.fetchall() 360 | assert len(all_customers) > 0 361 | namecnt = len(all_customers) 362 | index = (namecnt-1)/2 363 | customer = all_customers[index] 364 | c_id = customer[0] 365 | assert len(customer) > 0 366 | assert c_id != None 367 | 368 | self.cursor.execute(q["getLastOrder"], [w_id, d_id, c_id]) 369 | order = self.cursor.fetchone() 370 | if order: 371 | self.cursor.execute(q["getOrderLines"], [w_id, d_id, order[0]]) 372 | orderLines = self.cursor.fetchall() 373 | else: 374 | orderLines = [ ] 375 | 376 | self.conn.commit() 377 | return [ customer, order, orderLines ] 378 | 379 | ## ---------------------------------------------- 380 | ## doPayment 381 | ## ---------------------------------------------- 382 | def doPayment(self, params): 383 | q = TXN_QUERIES["PAYMENT"] 384 | 385 | w_id = params["w_id"] 386 | d_id = params["d_id"] 387 | h_amount = params["h_amount"] 388 | c_w_id = params["c_w_id"] 389 | c_d_id = params["c_d_id"] 390 | c_id = params["c_id"] 391 | c_last = params["c_last"] 392 | h_date = params["h_date"] 393 | 394 | if c_id != None: 395 | self.cursor.execute(q["getCustomerByCustomerId"], [w_id, d_id, c_id]) 396 | customer = self.cursor.fetchone() 397 | else: 398 | # Get the midpoint customer's id 399 | self.cursor.execute(q["getCustomersByLastName"], [w_id, d_id, c_last]) 400 | all_customers = self.cursor.fetchall() 401 | assert len(all_customers) > 0 402 | namecnt = len(all_customers) 403 | index = (namecnt-1)/2 404 | customer = all_customers[index] 405 | c_id = customer[0] 406 | assert len(customer) > 0 407 | c_balance = customer[14] - h_amount 408 | c_ytd_payment = customer[15] + h_amount 409 | c_payment_cnt = customer[16] + 1 410 | c_data = customer[17] 411 | 412 | self.cursor.execute(q["getWarehouse"], [w_id]) 413 | warehouse = self.cursor.fetchone() 414 | 415 | self.cursor.execute(q["getDistrict"], [w_id, d_id]) 416 | district = self.cursor.fetchone() 417 | 418 | self.cursor.execute(q["updateWarehouseBalance"], [h_amount, w_id]) 419 | self.cursor.execute(q["updateDistrictBalance"], [h_amount, w_id, d_id]) 420 | 421 | # Customer Credit Information 422 | if customer[11] == constants.BAD_CREDIT: 423 | newData = " ".join(map(str, [c_id, c_d_id, c_w_id, d_id, w_id, h_amount])) 424 | c_data = (newData + "|" + c_data) 425 | if len(c_data) > constants.MAX_C_DATA: c_data = c_data[:constants.MAX_C_DATA] 426 | self.cursor.execute(q["updateBCCustomer"], [c_balance, c_ytd_payment, c_payment_cnt, c_data, c_w_id, c_d_id, c_id]) 427 | else: 428 | c_data = "" 429 | self.cursor.execute(q["updateGCCustomer"], [c_balance, c_ytd_payment, c_payment_cnt, c_w_id, c_d_id, c_id]) 430 | 431 | # Concatenate w_name, four spaces, d_name 432 | h_data = "%s %s" % (warehouse[0], district[0]) 433 | # Create the history record 434 | self.cursor.execute(q["insertHistory"], [c_id, c_d_id, c_w_id, d_id, w_id, h_date, h_amount, h_data]) 435 | 436 | self.conn.commit() 437 | 438 | # TPC-C 2.5.3.3: Must display the following fields: 439 | # W_ID, D_ID, C_ID, C_D_ID, C_W_ID, W_STREET_1, W_STREET_2, W_CITY, W_STATE, W_ZIP, 440 | # D_STREET_1, D_STREET_2, D_CITY, D_STATE, D_ZIP, C_FIRST, C_MIDDLE, C_LAST, C_STREET_1, 441 | # C_STREET_2, C_CITY, C_STATE, C_ZIP, C_PHONE, C_SINCE, C_CREDIT, C_CREDIT_LIM, 442 | # C_DISCOUNT, C_BALANCE, the first 200 characters of C_DATA (only if C_CREDIT = "BC"), 443 | # H_AMOUNT, and H_DATE. 444 | 445 | # Hand back all the warehouse, district, and customer data 446 | return [ warehouse, district, customer ] 447 | 448 | ## ---------------------------------------------- 449 | ## doStockLevel 450 | ## ---------------------------------------------- 451 | def doStockLevel(self, params): 452 | q = TXN_QUERIES["STOCK_LEVEL"] 453 | 454 | w_id = params["w_id"] 455 | d_id = params["d_id"] 456 | threshold = params["threshold"] 457 | 458 | self.cursor.execute(q["getOId"], [w_id, d_id]) 459 | result = self.cursor.fetchone() 460 | assert result 461 | o_id = result[0] 462 | 463 | self.cursor.execute(q["getStockCount"], [w_id, d_id, o_id, (o_id - 20), w_id, threshold]) 464 | result = self.cursor.fetchone() 465 | 466 | self.conn.commit() 467 | 468 | return int(result[0]) 469 | 470 | ## CLASS -------------------------------------------------------------------------------- /pytpcc/drivers/tokyocabinetdriver.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | # ----------------------------------------------------------------------- 3 | # Copyright (C) 2011 4 | # Marcelo Martins 5 | # http://www.cs.brown.edu/~martins/ 6 | # 7 | # Permission is hereby granted, free of charge, to any person obtaining 8 | # a copy of this software and associated documentation files (the 9 | # "Software"), to deal in the Software without restriction, including 10 | # without limitation the rights to use, copy, modify, merge, publish, 11 | # distribute, sublicense, and/or sell copies of the Software, and to 12 | # permit persons to whom the Software is furnished to do so, subject to 13 | # the following conditions: 14 | # 15 | # The above copyright notice and this permission notice shall be 16 | # included in all copies or substantial portions of the Software. 17 | # 18 | # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, 19 | # EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF 20 | # MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT 21 | # IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR 22 | # OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, 23 | # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR 24 | # OTHER DEALINGS IN THE SOFTWARE. 25 | # ----------------------------------------------------------------------- 26 | 27 | from __future__ import with_statement 28 | from abstractdriver import * 29 | from pprint import pprint, pformat 30 | from pyrant import protocol 31 | 32 | import constants 33 | import logging 34 | import os 35 | import pyrant 36 | import sys 37 | 38 | TABLE_COLUMNS = { 39 | constants.TABLENAME_ITEM: [ 40 | "I_ID", # INTEGER 41 | "I_IM_ID", # INTEGER 42 | "I_NAME", # VARCHAR 43 | "I_PRICE", # FLOAT 44 | "I_DATA", # VARCHAR 45 | ], 46 | constants.TABLENAME_WAREHOUSE: [ 47 | "W_ID", # SMALLINT 48 | "W_NAME", # VARCHAR 49 | "W_STREET_1", # VARCHAR 50 | "W_STREEET_2", # VARCHAR 51 | "W_CITY", # VARCHAR 52 | "W_STATE", # VARCHAR 53 | "W_ZIP", # VARCHAR 54 | "W_TAX", # FLOAT 55 | "W_YTD", # FLOAT 56 | ], 57 | constants.TABLENAME_DISTRICT: [ 58 | "D_ID", # TINYINT 59 | "D_W_ID", # SMALLINT 60 | "D_NAME", # VARCHAR 61 | "D_STREET_1", # VARCHAR 62 | "D_STREET_2", # VARCHAR 63 | "D_CITY", # VARCHAR 64 | "D_STATE", # VARCHAR 65 | "D_ZIP", # VARCHAR 66 | "D_TAX", # FLOAT 67 | "D_YTD", # FLOAT 68 | "D_NEXT_O_ID", # INT 69 | ], 70 | constants.TABLENAME_CUSTOMER: [ 71 | "C_ID", # INTEGER 72 | "C_D_ID", # TINYINT 73 | "C_W_ID", # SMALLINT 74 | "C_FIRST", # VARCHAR 75 | "C_MIDDLE", # VARCHAR 76 | "C_LAST", # VARCHAR 77 | "C_STREET_1", # VARCHAR 78 | "C_STREET_2", # VARCHAR 79 | "C_CITY", # VARCHAR 80 | "C_STATE", # VARCHAR 81 | "C_ZIP", # VARCHAR 82 | "C_PHONE", # VARCHAR 83 | "C_SINCE", # TIMESTAMP 84 | "C_CREDIT", # VARCHAR 85 | "C_CREDIT_LIM", # FLOAT 86 | "C_DISCOUNT", # FLOAT 87 | "C_BALANCE", # FLOAT 88 | "C_YTD_PAYMENT", # FLOAT 89 | "C_PAYMENT_CNT", # INTEGER 90 | "C_DELIVERY_CNT", # INTEGER 91 | "C_DATA", # VARCHAR 92 | ], 93 | constants.TABLENAME_STOCK: [ 94 | "S_I_ID", # INTEGER 95 | "S_W_ID", # SMALLINT 96 | "S_QUANTITY", # INTEGER 97 | "S_DIST_01", # VARCHAR 98 | "S_DIST_02", # VARCHAR 99 | "S_DIST_03", # VARCHAR 100 | "S_DIST_04", # VARCHAR 101 | "S_DIST_05", # VARCHAR 102 | "S_DIST_06", # VARCHAR 103 | "S_DIST_07", # VARCHAR 104 | "S_DIST_08", # VARCHAR 105 | "S_DIST_09", # VARCHAR 106 | "S_DIST_10", # VARCHAR 107 | "S_YTD", # INTEGER 108 | "S_ORDER_CNT", # INTEGER 109 | "S_REMOTE_CNT", # INTEGER 110 | "S_DATA", # VARCHAR 111 | ], 112 | constants.TABLENAME_ORDERS: [ 113 | "O_ID", # INTEGER 114 | "O_C_ID", # INTEGER 115 | "O_D_ID", # TINYINT 116 | "O_W_ID", # SMALLINT 117 | "O_ENTRY_ID", # TIMESTAMP 118 | "O_CARRIER_ID", # INTEGER 119 | "O_OL_CNT", # INTEGER 120 | "O_ALL_LOCAL", # INTEGER 121 | ], 122 | constants.TABLENAME_NEW_ORDER: [ 123 | "NO_O_ID", # INTEGER 124 | "NO_D_ID", # TINYINT 125 | "NO_W_ID", # SMALLINT 126 | ], 127 | constants.TABLENAME_ORDER_LINE: [ 128 | "OL_O_ID", # INTEGER 129 | "OL_D_ID", # TINYINT 130 | "OL_W_ID", # SMALLINT 131 | "OL_NUMBER", # INTEGER 132 | "OL_I_ID", # INTEGER 133 | "OL_SUPPLY_W_ID", # SMALLINT 134 | "OL_DELIVERY_D", # TIMESTAMP 135 | "OL_QUANTITY", # INTEGER 136 | "OL_AMOUNT", # FLOAT 137 | "OL_DIST_INFO", # VARCHAR 138 | ], 139 | constants.TABLENAME_HISTORY: [ 140 | "H_C_ID", # INTEGER 141 | "H_C_D_ID", # TINYINT 142 | "H_C_W_ID", # SMALLINT 143 | "H_D_ID", # TINYINT 144 | "H_W_ID", # SMALLINT 145 | "H_DATA", # TIMESTAMP 146 | "H_AMOUNT", # FLOAT 147 | "H_DATA", # VARCHAR 148 | ], 149 | } 150 | TABLE_INDEXES = { 151 | constants.TABLENAME_ITEM: [ 152 | "I_ID", 153 | ], 154 | constants.TABLENAME_WAREHOUSE: [ 155 | "W_ID", 156 | ], 157 | constants.TABLENAME_DISTRICT: [ 158 | "D_ID", 159 | "D_W_ID", 160 | ], 161 | constants.TABLENAME_CUSTOMER: [ 162 | "C_ID", 163 | "C_D_ID", 164 | "C_W_ID", 165 | ], 166 | constants.TABLENAME_STOCK: [ 167 | "S_I_ID", 168 | "S_W_ID", 169 | ], 170 | constants.TABLENAME_ORDERS: [ 171 | "O_ID", 172 | "O_D_ID", 173 | "O_W_ID", 174 | "O_C_ID", 175 | ], 176 | constants.TABLENAME_NEW_ORDER: [ 177 | "NO_O_ID", 178 | "NO_D_ID", 179 | "NO_W_ID", 180 | ], 181 | constants.TABLENAME_ORDER_LINE: [ 182 | "OL_O_ID", 183 | "OL_D_ID", 184 | "OL_W_ID", 185 | ], 186 | } 187 | 188 | ## ============================================== 189 | ## TokyocabinetDriver 190 | ## ============================================== 191 | class TokyocabinetDriver(AbstractDriver): 192 | 193 | ## Tokyo Tyrant provides one connection per *table*, not per database. 194 | 195 | ## Config files have no hierarchy. Let's set up our own hierarchy as a 196 | ## stringfied dictionary and evaluate it later. 197 | 198 | ## Each table connection defines the host and port location of the Tokyo 199 | ## Tyrant server 200 | 201 | DEFAULT_CONFIG = { "servers": ("Tokyo Tyrant server configuration", '{ 0: { "' + 202 | constants.TABLENAME_ITEM + '":{ "host": "localhost", "port": 1978, },"' + 203 | constants.TABLENAME_WAREHOUSE + '" : { "host": "localhost", "port": 1979, },"' + 204 | constants.TABLENAME_DISTRICT + '" : { "host": "localhost", "port": 1980, },"' + 205 | constants.TABLENAME_CUSTOMER + '" : { "host": "localhost", "port": 1981, },"' + 206 | constants.TABLENAME_STOCK + '" : { "host": "localhost", "port": 1982, },"' + 207 | constants.TABLENAME_ORDERS + '" : { "host": "localhost", "port": 1983, },"' + 208 | constants.TABLENAME_NEW_ORDER + '" : { "host": "localhost", "port": 1984, },"' + 209 | constants.TABLENAME_ORDER_LINE+ '" : { "host": "localhost", "port": 1985, },"' + 210 | constants.TABLENAME_HISTORY + '" : { "host": "localhost", "port": 1986, }, }, }' ), } 211 | 212 | def __init__(self, ddl): 213 | super(TokyocabinetDriver, self).__init__("tokyocabinet", ddl) 214 | self.databases = dict() 215 | self.conn = dict() 216 | self.numServers = 0 217 | 218 | ##----------------------------------------------- 219 | ## self.tupleToString 220 | ##----------------------------------------------- 221 | def tupleToString(self, tuple, sep=":"): 222 | """Tokyo-Cabinet table-type databases only accept strings as keys. 223 | This function transforms a compound key (tuple) into a string. 224 | Tuples elements are separated by the sep char""" 225 | return sep.join(str(t) for t in tuple) 226 | 227 | ##----------------------------------------------- 228 | ## self.getServer 229 | ##----------------------------------------------- 230 | def getServer(self, warehouseID): 231 | """Tokyo Cabinet does not support data partitioning. For the TPC-C 232 | benchmark, we manually partition data based on the warehouse ID""" 233 | return (warehouseID % self.numServers) 234 | 235 | ## ---------------------------------------------- 236 | ## makeDefaultConfig 237 | ## ---------------------------------------------- 238 | def makeDefaultConfig(self): 239 | return TokyocabinetDriver.DEFAULT_CONFIG 240 | 241 | ## ---------------------------------------------- 242 | ## loadConfig 243 | ## ---------------------------------------------- 244 | def loadConfig(self, config): 245 | for key in TokyocabinetDriver.DEFAULT_CONFIG.keys(): 246 | assert key in config, "Missing parameter '%s' in %s configuration" % (key, self.name) 247 | 248 | if config["servers"]: 249 | # Whn reading from INI file, we need to convert the server 250 | # description-object from string to a real dictionary 251 | config["servers"] = eval(config["servers"]) 252 | for serverId, tables in config["servers"].iteritems(): 253 | self.databases[serverId] = tables 254 | 255 | # First connect to databases 256 | for serverId, tables in self.databases.iteritems(): 257 | self.conn[serverId] = dict() 258 | for tab, values in tables.iteritems(): 259 | self.conn[serverId][tab] = pyrant.Tyrant(values["host"], values["port"]) 260 | ## FOR 261 | 262 | # Remove previous data 263 | if config["reset"]: 264 | for serverId, tables in self.conn.iteritems(): 265 | for tab in tables.keys(): 266 | logging.debug("Deleting database '%s' at server '%s'" % (tab, serverId)) 267 | self.conn[serverId][tab].clear() 268 | ## FOR 269 | ## FOR 270 | ## IF 271 | 272 | self.numServers = len(self.databases.keys()) 273 | logging.info("Number of servers: %s" % self.numServers) 274 | 275 | ## ------------------------------------------- 276 | ## loadTuples 277 | ## ------------------------------------------- 278 | def loadTuples(self, tableName, tuples): 279 | """Load tuples into tables of database 280 | Each table is a connection to a Tyrant server. Each record is a key-value pair, 281 | where key = primary key, values = concatenation of columns (dictionary). If 282 | key is compound we transform it into a string, since TC does not support 283 | compound keys. Data partitioning occurs based on Warehouse ID.""" 284 | 285 | if len(tuples) == 0: return 286 | 287 | logging.debug("Loading %d tuples of tableName %s" % (len(tuples), tableName)) 288 | 289 | assert tableName in TABLE_COLUMNS, "Unexpected table %s" % tableName 290 | columns = TABLE_COLUMNS[tableName] 291 | num_columns = xrange(len(columns)) 292 | records = list() 293 | 294 | if tableName == constants.TABLENAME_WAREHOUSE: 295 | for t in tuples: 296 | w_key = t[0] # W_ID 297 | sID = self.getServer(w_key) 298 | cols = dict(map(lambda i: (columns[i], t[i]), num_columns)) 299 | records.append((str(w_key), cols)) 300 | ## FOR 301 | 302 | try: 303 | self.conn[sID][tableName].multi_set(records) 304 | except KeyError, err: 305 | sys.stderr.write("%s(%s): server ID does not exist or is offline\n" %(KeyError, err)) 306 | sys.exit(1) 307 | 308 | elif tableName == constants.TABLENAME_DISTRICT: 309 | for t in tuples: 310 | w_key = t[1] # W_ID 311 | sID = self.getServer(w_key) 312 | d_key = self.tupleToString(t[:2]) # D_ID, D_W_ID 313 | cols = dict(map(lambda i: (columns[i], t[i]), num_columns)) 314 | records.append((d_key, cols)) 315 | ## FOR 316 | 317 | try: 318 | self.conn[sID][tableName].multi_set(records) 319 | except KeyError, err: 320 | sys.stderr.write("%s(%s): server ID does not exist or is offline\n" %(KeyError, err)) 321 | sys.exit(1) 322 | 323 | ## Item table doesn't have a w_id for partition. Replicate it to all 324 | ## servers 325 | elif tableName == constants.TABLENAME_ITEM: 326 | for t in tuples: 327 | i_key = str(t[0]) 328 | cols = dict(map(lambda i: (columns[i], t[i]), num_columns)) 329 | records.append((i_key, cols)) 330 | ## FOR 331 | 332 | for i in xrange(self.numServers): 333 | try: 334 | self.conn[i][tableName].multi_set(records) 335 | except KeyError, err: 336 | sys.stderr.write("%s(%s): server ID doesn't exist or is offline\n" %(KeyError, err)) 337 | sys.exit(1) 338 | ## FOR 339 | ## FOR 340 | 341 | elif tableName == constants.TABLENAME_CUSTOMER: 342 | for t in tuples: 343 | w_key = t[2] # W_ID 344 | sID = self.getServer(w_key) 345 | c_key = self.tupleToString(t[:3]) # C_ID, C_D_ID, C_W_ID 346 | cols = dict(map(lambda i: (columns[i], t[i]), num_columns)) 347 | records.append((c_key, cols)) 348 | ## FOR 349 | 350 | try: 351 | self.conn[sID][tableName].multi_set(records) 352 | except KeyError, err: 353 | sys.stderr.write("%s(%s): server ID does not exist or is offline\n" %(KeyError, err)) 354 | sys.exit(1) 355 | 356 | elif tableName == constants.TABLENAME_HISTORY: 357 | for t in tuples: 358 | w_key = t[4] # W_ID 359 | # Not really a primary key, but we need to generate 360 | # something tobe our key 361 | h_key = self.tupleToString(t[:3]) # H_C_ID, H_C_D, H_C_W 362 | sID = self.getServer(w_key) 363 | cols = dict(map(lambda i: (columns[i], t[i]), num_columns)) 364 | records.append((h_key, cols)) 365 | ## FOR 366 | 367 | try: 368 | self.conn[sID][tableName].multi_set(records) 369 | except KeyError, err: 370 | sys.stderr.write("%s(%s): server ID does not exist or is offline\n" %(KeyError, err)) 371 | sys.exit(1) 372 | 373 | elif tableName == constants.TABLENAME_STOCK: 374 | for t in tuples: 375 | w_key = t[1] # W_ID 376 | sID = self.getServer(w_key) 377 | s_key = self.tupleToString(t[:2]) # S_ID, S_W_ID 378 | cols = dict(map(lambda i: (columns[i], t[i]), num_columns)) 379 | records.append((s_key, cols)) 380 | ## FOR 381 | 382 | try: 383 | self.conn[sID][tableName].multi_set(records) 384 | except KeyError, err: 385 | sys.stderr.write("%s(%s): server ID does not exist or is offline\n" %(KeyError, err)) 386 | sys.exit(1) 387 | 388 | elif tableName == constants.TABLENAME_ORDERS: 389 | for t in tuples: 390 | w_key = t[3] # W_ID 391 | sID = self.getServer(w_key) 392 | o_key = self.tupleToString(t[1:4]) # O_ID, O_D_ID, O_W_ID 393 | cols = dict(map(lambda i: (columns[i], t[i]), num_columns)) 394 | records.append((o_key, cols)) 395 | ## FOR 396 | 397 | try: 398 | self.conn[sID][tableName].multi_set(records) 399 | except KeyError, err: 400 | sys.stderr.write("%s(%s): server ID does not exist or is offline\n" %(KeyError, err)) 401 | sys.exit(1) 402 | 403 | elif tableName == constants.TABLENAME_NEW_ORDER: 404 | for t in tuples: 405 | w_key = t[2] # W_ID 406 | sID = self.getServer(w_key) 407 | no_key = self.tupleToString(t[:3]) # NO_O_ID, NO_D_ID, NO_W_ID 408 | cols = dict(map(lambda i: (columns[i], t[i]), num_columns)) 409 | records.append((no_key, cols)) 410 | ## FOR 411 | 412 | try: 413 | self.conn[sID][tableName].multi_set(records) 414 | except KeyError, err: 415 | sys.stderr.write("%s(%s): server ID does not exist or is offline\n" %(KeyError, err)) 416 | sys.exit(1) 417 | 418 | elif tableName == constants.TABLENAME_ORDER_LINE: 419 | for t in tuples: 420 | w_key = t[2] # W_ID 421 | sID = self.getServer(w_key) 422 | ol_key = self.tupleToString(t[:4]) # OL_O_ID, OL_D_ID, OL_W_ID, OL_NUMBER 423 | cols = dict(map(lambda i: (columns[i], t[i]), num_columns)) 424 | records.append((ol_key, cols)) 425 | ## FOR 426 | 427 | try: 428 | self.conn[sID][tableName].multi_set(records) 429 | except KeyError, err: 430 | sys.stderr.write("%s(%s): server ID does not exist or is offline\n" %(KeyError, err)) 431 | sys.exit(1) 432 | 433 | logging.debug("Loaded %s tuples for tableName %s" % (len(tuples), tableName)) 434 | return 435 | 436 | ## ------------------------------------------- 437 | ## loadFinish 438 | ## ------------------------------------------- 439 | def loadFinish(self): 440 | 441 | conn = dict() 442 | 443 | logging.info("Creating indexes...") 444 | # Add indexes to database after loading all data 445 | for serverId, tables in self.databases.iteritems(): 446 | conn[serverId] = dict() 447 | for tab, connValues in tables.iteritems(): 448 | conn[serverId][tab] = protocol.TyrantProtocol(connValues["host"], connValues["port"]) 449 | for index_name in TABLE_COLUMNS[tab]: 450 | conn[serverId][tab].add_index(index_name) 451 | ## FOR 452 | ## FOR 453 | 454 | logging.info("Optimizing indexes...") 455 | # Optimize indexes for faster access 456 | for serverId, tables in self.databases.iteritems(): 457 | for tab, connValues in tables.iteritems(): 458 | for index_name in TABLE_COLUMNS[tab]: 459 | conn[serverId][tab].optimize_index(index_name) 460 | ## FOR 461 | ## FOR 462 | 463 | logging.info("Syncing to disk...") 464 | # Finally, flush everything to disk 465 | for tab in TABLE_COLUMNS.keys(): 466 | for sID in self.conn.keys(): 467 | self.conn[sID][tab].sync() 468 | ## FOR 469 | 470 | logging.info("Finished loading tables") 471 | 472 | ## -------------------------------------------- 473 | ## doDelivery 474 | ## -------------------------------------------- 475 | def doDelivery(self, params): 476 | """Execute DELIVERY Transaction 477 | Parameters Dict: 478 | w_id 479 | o_carrier_id 480 | ol_delivery_id 481 | """ 482 | 483 | w_id = params["w_id"] 484 | o_carrier_id = params["o_carrier_id"] 485 | ol_delivery_d = params["ol_delivery_d"] 486 | 487 | sID = self.getServer(w_id) 488 | 489 | newOrderQuery = self.conn[sID][constants.TABLENAME_NEW_ORDER].query 490 | ordersQuery = self.conn[sID][constants.TABLENAME_ORDERS].query 491 | orderLineQuery = self.conn[sID][constants.TABLENAME_ORDER_LINE].query 492 | customerQuery = self.conn[sID][constants.TABLENAME_CUSTOMER].query 493 | 494 | results = [ ] 495 | for d_id in xrange(1, constants.DISTRICTS_PER_WAREHOUSE+1): 496 | 497 | # getNewOrder 498 | # SELECT NO_O_ID FROM NEW_ORDER WHERE NO_D_ID = ? AND NO_W_ID = ? AND NO_O_ID > -1 LIMIT 1 499 | 500 | newOrders = newOrderQuery.filter(NO_D_ID = d_id, NO_W_ID = w_id, NO_O_ID__gt = -1).columns("NO_O_ID") 501 | if len(newOrders) == 0: 502 | ## No orders for this district: skip it. Note: This must 503 | ## reported if > 1% 504 | continue 505 | assert(newOrders) > 0 506 | no_o_id = int(newOrders[0]["NO_O_ID"]) 507 | 508 | # sumOLAmount 509 | # SELECT SUM(OL_AMOUNT) FROM ORDER_LINE WHERE OL_O_ID = ? AND OL_D_ID = ? AND OL_W_ID = ? 510 | 511 | olines = orderLineQuery.filter(OL_O_ID = no_o_id, OL_D_ID = d_id, OL_W_ID = w_id).columns("OL_AMOUNT") 512 | 513 | # These must be logged in the "result file" according to TPC-C 514 | # 2.7.22 (page 39) 515 | # We remove the queued time, completed time, w_id, and 516 | # o_carrier_id: the client can figure them out 517 | # If there are no order lines, SUM returns null. There should 518 | # always be order lines. 519 | assert len(olines) > 0, "ol_total is NULL: there are no order lines. This should not happen" 520 | 521 | ol_total = sum(float(i["OL_AMOUNT"]) for i in olines) 522 | 523 | assert ol_total > 0.0 524 | 525 | # deleteNewOrder 526 | # DELETE FROM NEW_ORDER WHERE NO_D_ID = ? AND NO_W_ID = ? AND NO_O_ID = ? 527 | 528 | orders = newOrderQuery.filter(NO_D_ID = d_id, NO_W_ID = w_id, NO_O_ID = no_o_id) 529 | orders.delete(quick=True) 530 | 531 | # HACK: not transactionally safe 532 | # getCId 533 | # SELECT O_C_ID FROM ORDERS WHERE O_ID = ? AND O_D_ID = ? AND O_W_ID = ? 534 | 535 | orders = ordersQuery.filter(O_ID = no_o_id, O_D_ID = d_id, O_W_ID = w_id) 536 | assert len(orders) > 0 537 | c_id = int(orders.columns("O_C_ID")[0]["O_C_ID"]) 538 | 539 | # updateOrders 540 | # UPDATE ORDERS SET O_CARRIER_ID = ? WHERE O_ID = ? AND O_D_ID = ? AND O_W_ID = ? 541 | 542 | records = list() 543 | for record in orders: 544 | key, cols = record 545 | cols["O_CARRIER_ID"] = o_carrier_id 546 | records.append((key, cols)) 547 | ## FOR 548 | 549 | self.conn[sID][constants.TABLENAME_ORDERS].multi_set(records) 550 | 551 | # updateOrderLine 552 | # UPDATE ORDER_LINE SET OL_DELIVERY_D = ? WHERE OL_O_ID = ? AND OL_D_ID = ? AND OL_W_ID = ? 553 | 554 | orders = orderLineQuery.filter(OL_O_ID = no_o_id, OL_D_ID = d_id, OL_W_ID = w_id) 555 | records = list() 556 | for record in orders: 557 | key, cols = record 558 | cols["OL_DELIVERY_D"] = ol_delivery_d 559 | records.append((key, cols)) 560 | ## FOR 561 | 562 | self.conn[sID][constants.TABLENAME_ORDER_LINE].multi_set(records) 563 | 564 | # updateCustomer 565 | # UPDATE CUSTOMER SET C_BALANCE = C_BALANCE + ? WHERE C_ID = ? AND C_D_ID = ? AND C_W_ID = ? 566 | 567 | customers = customerQuery.filter(C_ID = c_id, C_D_ID = d_id, C_W_ID = w_id) 568 | records = list() 569 | for record in customers: 570 | key, cols = record 571 | cols["C_BALANCE"] = float(cols["C_BALANCE"]) + ol_total 572 | records.append((key, cols)) 573 | ## FOR 574 | 575 | self.conn[sID][constants.TABLENAME_CUSTOMER].multi_add(records) 576 | 577 | results.append((d_id, no_o_id)) 578 | ## FOR 579 | 580 | return results 581 | 582 | def doNewOrder(self, params): 583 | """Execute NEW_ORDER Transaction 584 | Parameters Dict: 585 | w_id 586 | d_id 587 | c_id 588 | o_entry_id 589 | i_ids 590 | i_w_ids 591 | i_qtys 592 | """ 593 | 594 | w_id = params["w_id"] 595 | d_id = params["d_id"] 596 | c_id = params["c_id"] 597 | o_entry_d = params["o_entry_d"] 598 | i_ids = params["i_ids"] 599 | i_w_ids = params["i_w_ids"] 600 | i_qtys = params["i_qtys"] 601 | 602 | assert len(i_ids) > 0 603 | assert len(i_ids) == len(i_w_ids) 604 | assert len(i_ids) == len(i_qtys) 605 | 606 | sID = self.getServer(w_id) 607 | 608 | warehouseQuery = self.conn[sID][constants.TABLENAME_WAREHOUSE].query 609 | districtQuery = self.conn[sID][constants.TABLENAME_DISTRICT].query 610 | customerQuery = self.conn[sID][constants.TABLENAME_CUSTOMER].query 611 | orderQuery = self.conn[sID][constants.TABLENAME_ORDERS].query 612 | newOrderQuery = self.conn[sID][constants.TABLENAME_NEW_ORDER].query 613 | itemQuery = self.conn[sID][constants.TABLENAME_ITEM].query 614 | 615 | all_local = True 616 | items = [ ] 617 | for i in xrange(len(i_ids)): 618 | ## Determine if this is an all local order or not 619 | all_local = all_local and i_w_ids[i] == w_id 620 | # getItemInfo 621 | retItems = itemQuery.filter(I_PRICE = i_ids[i]) 622 | if len(items) > 0: 623 | items.append(retItems.columns()) 624 | else: 625 | items.append([]) 626 | assert len(items) == len(i_ids) 627 | 628 | ## TPCC define 1% of neworder gives a wrong itemid, causing rollback. 629 | ## Note that this will happen with 1% of transactions on purpose. 630 | for item in items: 631 | if len(item) == 0: 632 | ## TODO Abort here! 633 | return 634 | ## FOR 635 | 636 | ## ----------------- 637 | ## Collect Information from WAREHOUSE, DISTRICT, and CUSTOMER 638 | ## ----------------- 639 | 640 | # getWarehouseTaxRate 641 | # SELECT W_TAX FROM WAREHOUSE WHERE W_ID = ? 642 | 643 | taxes = warehouseQuery.filter(W_ID = w_id) 644 | w_tax = float(taxes.columns("W_TAX")[0]["W_TAX"]) 645 | 646 | # getDistrict 647 | # SELECT D_TAX, D_NEXT_O_ID FROM DISTRICT WHERE D_ID = ? AND D_W_ID = ? 648 | 649 | districts = districtQuery.filter(D_ID = d_id, D_W_ID = w_id) 650 | districtInfo = districts.columns("D_TAX", "D_NEXT_O_ID")[0] 651 | d_tax = float(districtInfo["D_TAX"]) 652 | d_next_o_id = int(districtInfo["D_NEXT_O_ID"]) 653 | 654 | # incrementNextOrderId 655 | # HACK: This is not transactionally safe! 656 | # UPDATE DISTRICT SET D_NEXT_O_ID = ? WHERE D_ID = ? AND D_W_ID = ? 657 | records = list() 658 | for record in districts: 659 | key, cols = record 660 | cols["D_NEXT_O_ID"] = d_next_o_id + 1 661 | records.append((key, cols)) 662 | ## FOR 663 | 664 | self.conn[sID][constants.TABLENAME_DISTRICT].multi_set(records) 665 | 666 | # getCustomer 667 | # SELECT C_DISCOUNT, C_LAST, C_CREDIT FROM CUSTOMER WHERE C_W_ID = ? AND C_D_ID = ? AND C_ID = ? 668 | 669 | customers = customerQuery.filter(C_W_ID = w_id, C_D_ID = d_id, C_ID = c_id) 670 | customerInfo = customers.columns("C_DISCOUNT", "C_LAST", "C_CREDIT")[0] 671 | c_discount = float(customerInfo["C_DISCOUNT"]) 672 | 673 | ## ----------------- 674 | ## Insert Order Information 675 | ## ----------------- 676 | ol_cnt = len(i_ids) 677 | o_carrier_id = constants.NULL_CARRIER_ID 678 | 679 | # createOrder 680 | # INSERT INTO ORDERS (O_ID, O_D_ID, O_W_ID, O_C_ID, O_ENTRY_D, O_CARRIER_ID, O_OL_CNT, 681 | # O_ALL_LOCAL) VALUES (?, ?, ?, ?, ?, ?, ?, ?) 682 | 683 | key = self.tupleToString((d_next_o_id, d_id, w_id)) 684 | cols = {"O_ID": d_next_o_id, "O_D_ID": d_id, "O_W_ID": w_id, "O_C_ID": 685 | c_id, "O_ENTRY_D": o_entry_d, "O_CARRIER_ID": 686 | o_carrier_id, "O_OL_CNT": o_ol_cnt, "O_ALL_LOCAL": 687 | all_local} 688 | self.conn[sID][constants.TABLENAME_ORDERS].multi_set([(key, cols)]) 689 | 690 | # createNewOrder 691 | # INSERT INTO NEW_ORDER (NO_O_ID, NO_D_ID, NO_W_ID) VALUES (?, ?, ?) 692 | 693 | key = self.tupleToString((d_next_o_id, d_id, w_id)) 694 | cols = {"NO_O_ID": d_next_o_id, "NO_D_ID": d_id, "NO_W_ID": w_id} 695 | self.conn[sID][constants.TABLENAME_NEW_ORDER].multi_set([(key, cols)]) 696 | 697 | ## ------------------------------- 698 | ## Insert Order Item Information 699 | ## ------------------------------- 700 | 701 | item_data = [ ] 702 | total = 0 703 | for i in xrange(len(i_id)): 704 | ol_number = i+1 705 | ol_supply_w_id = i_w_ids[i] 706 | ol_i_id = i_ids[i] 707 | ol_quantity = i_qtys[i] 708 | 709 | # getItemInfo 710 | # SELECT I_PRICE, I_NAME, I_DATA FROM ITEM WHERE I_ID = ? 711 | 712 | key, itemInfo = items[i] 713 | i_price = float(itemInfo["I_PRICE"]) 714 | i_name = itemInfo["I_NAME"] 715 | i_data = itemInfo["I_DATA"] 716 | 717 | # getStockInfo 718 | # SELECT S_QUANTITY, S_DATA, S_YTD, S_ORDER_CNT, S_REMOTE_CNT, S_DIST_%02d FROM STOCK 719 | # WHERE S_I_ID = ? AND S_W_ID = ? 720 | 721 | stocks = stockQuery.filter(S_I_ID = ol_i_id, S_W_ID = ol_supply_w_id) 722 | if len(stocks) == 0: 723 | logging.warn("No STOCK record for (ol_i_id=%d, ol_supply_w_id=%d)" 724 | % (ol_i_id, ol_supply_w_id)) 725 | continue 726 | stockInfo = stock.columns("S_QUANTITY", "S_DATA", "S_YTD", "S_ORDER_CNT", 727 | "S_REMOTE_CNT", "S_DIST_%02d"%d_id)[0] 728 | 729 | s_quantity = int(stockInfo["S_QUANTITY"]) 730 | s_ytd = float(stockInfo["S_YTD"]) 731 | s_order_cnt = int(stockInfo["S_ORDER_CNT"]) 732 | s_remote_cnt = int(stockInfo["S_REMOTE_CNT"]) 733 | s_data = stockInfo["S_DATA"] 734 | s_dist_xx = stockInfo["S_DIST_%02d"%d_id] # Fetches data from the 735 | # s_dist_[d_id] column 736 | 737 | # updateStock 738 | # UPDATE STOCK SET S_QUANTITY = ?, S_YTD = ?, S_ORDER_CNT = ?, S_REMOTE_CNT = ? 739 | # WHERE S_I_ID = ? AND S_W_ID = ? 740 | 741 | s_ytd += ol_quantity 742 | if s_quantity >= ol_quantity + 10: 743 | s_quantity = s_quantity - ol_quantity 744 | else: 745 | s_quantity = s_quantity + 91 - ol_quantity 746 | s_order_cnt += 1 747 | 748 | if ol_supply_w_id != w_id: s_remote_cnt += 1 749 | 750 | sSID = self.getServer(ol_supply_w_id) 751 | stockQuery = self.conn[sID][constants.TABLENAME_STOCK].query 752 | stocks = stockQuery.filter(S_I_ID = ol_i_id, S_W_ID = ol_supply_w_id) 753 | records = list() 754 | for record in stocks: 755 | key, cols = record 756 | cols["S_QUANTITY"] = s_quantity 757 | cols["S_YTD"] = s_ytd 758 | cols["S_ORDER_CNT"] = s_order_cnt 759 | cols["S_REMOTE_CNT"] = s_remote_cnt 760 | records.append((key, cols)) 761 | ## FOR 762 | 763 | self.conn[sSID][constants.TABLENAME_STOCK].multi_set(records) 764 | 765 | if i_data.find(constants.ORIGINAL_STRING) != -1 and s_data.find(constants.ORIGINAL_STRING) != -1: 766 | brand_generic = 'B' 767 | else: 768 | brand_generic = 'G' 769 | 770 | ## Transaction profile states to use "ol_quantity * i_price" 771 | ol_amount = ol_quantity * i_price 772 | total += ol_amount 773 | 774 | # createOrderLine 775 | # INSERT INTO ORDER_LINE (OL_O_ID, OL_D_ID, OL_W_ID, OL_NUMBER, OL_I_ID, OL_SUPPLY_W_ID, 776 | # OL_DELIVERY_D, OL_QUANTITY, OL_AMOUNT, OL_DIST_INFO) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?) 777 | 778 | key = tupletoString((d_next_o_id, d_id, w_id, ol_number)) 779 | cols = {"OL_O_ID": d_next_o_id, "OL_D_ID": d_id, "OL_W_ID": w_id, 780 | "OL_NUMBER": ol_number, "OL_I_ID": ol_i_id, 781 | "OL_SUPPLY_W_ID": ol_supply_w_id, "OL_DELIVERY_D": 782 | ol_entry_d, "OL_QUANTITY": ol_quantity, "OL_AMOUNT": 783 | ol_amount, "OL_DIST_INFO": s_dist_xx} 784 | self.conn[sID][constants.TABLENAME_ORDER_LINE].multi_set([(key, cols)]) 785 | 786 | ## Add the info to be returned 787 | item_data.append((i_name, s_quantity, brand_generic, i_price, ol_amount)) 788 | ## FOR 789 | 790 | ## Adjust the total for the discount 791 | #print "c_discount:", c_discount, type(c_discount) 792 | #print "w_tax:", w_tax, type(w_tax) 793 | #print "d_tax:", d_tax, type(d_tax) 794 | total *= (1 - c_discount) * (1 + w_tax + d_tax) 795 | 796 | ## Pack up values the client is missing (see TPC-C 2.4.3.5) 797 | misc = [(w_tax, d_tax, d_next_o_id, total)] 798 | 799 | return [ customerInfo, misc, item_data ] 800 | 801 | def doOrderStatus(self, params): 802 | """Execute ORDER_STATUS Transaction 803 | Parameters Dict: 804 | w_id 805 | d_id 806 | c_id 807 | c_last 808 | """ 809 | w_id = params["w_id"] 810 | d_id = params["d_id"] 811 | c_id = params["c_id"] 812 | c_last = params["c_last"] 813 | 814 | assert w_id, pformat(params) 815 | assert d_id, pformat(params) 816 | 817 | sID = self.getServer(w_id) 818 | 819 | customerQuery = self.conn[sID][constants.TABLENAME_CUSTOMER].query 820 | orderQuery = self.conn[sID][constants.TABLENAME_ORDERS].query 821 | orderLineQuery= self.conn[sID][constants.TABLENAME_ORDER_LINE].query 822 | 823 | if c_id != None: 824 | # getCustomerByCustomerId 825 | # SELECT C_ID, C_FIRST, C_MIDDLE, C_LAST, C_BALANCE FROM CUSTOMER 826 | # WHERE C_W_ID = ? AND C_D_ID = ? AND C_ID = ? 827 | 828 | customers = customerQuery.filter(C_W_ID = w_id, C_D_ID = d_id, C_ID = c_id) 829 | customerInfo = customers.columns("C_ID", "C_FIRST", "C_MIDDLE", "C_LAST", "C_BALANCE")[0] 830 | else: 831 | # Get the midpoint customer's id 832 | # getCustomersByLastName 833 | # SELECT C_ID, C_FIRST, C_MIDDLE, C_LAST, C_BALANCE FROM CUSTOMER 834 | # WHERE C_W_ID = ? AND C_D_ID = ? AND C_LAST = ? ORDER BY C_FIRST 835 | 836 | customers = customerQuery.filter(C_W_ID = w_id, C_D_ID = d_id, C_LAST__contains = c_last).order_by("C_FIRST", numeric=False) 837 | all_customers = customers.columns("C_ID", "C_FIRST", "C_MIDDLE", "C_LAST", "C_BALANCE") 838 | namecnt = len(all_customers) 839 | assert namecnt > 0 840 | index = (namecnt-1)/2 841 | customerInfo = all_customers[index] 842 | c_id = int(customerInfo["C_ID"]) 843 | assert len(customerInfo) > 0 844 | 845 | # getLastOrder TODO: LIMIT 1 846 | # SELECT O_ID, O_CARRIER_ID, O_ENTRY_D FROM ORDERS 847 | # WHERE O_W_ID = ? AND O_D_ID = ? AND O_C_ID = ? ORDER BY O_ID DESC LIMIT 1 848 | 849 | orders = orderQuery.filter(O_W_ID = w_id, O_D_ID = d_id, O_C_ID = c_id).order_by("-O_ID", numeric=True) 850 | orderInfo = orders.columns("O_ID", "O_CARRIER_ID", "O_ENTRY_D")[0] 851 | 852 | # getOrderLines 853 | # SELECT OL_SUPPLY_W_ID, OL_I_ID, OL_QUANTITY, OL_AMOUNT, OL_DELIVERY_D FROM ORDER_LINE 854 | # WHERE OL_W_ID = ? AND OL_D_ID = ? AND OL_O_ID = ? 855 | if len(orders) > 0: 856 | o_id = int(orderInfo["O_ID"]) 857 | orders = orderLineQuery.filter(OL_W_ID = w_id, OL_D_ID = d_id, OL_O_ID = o_id) 858 | orderLines = orders.columns("OL_SUPPLY_W_ID", "OL_I_ID", "OL_QUANTITY", "OL_AMOUNT", "OL_DELIVERY_D") 859 | else: 860 | orderLines = [ ] 861 | 862 | return [customerInfo, orderInfo, orderLines] 863 | 864 | def doPayment(self, params): 865 | """Execute PAYMENT Transaction 866 | Parameters Dict: 867 | w_id 868 | d_id 869 | h_amount 870 | c_w_id 871 | c_d_id 872 | c_id 873 | c_last 874 | h_date 875 | """ 876 | 877 | w_id = params["w_id"] 878 | d_id = params["d_id"] 879 | h_amount = params["h_amount"] 880 | c_w_id = params["c_w_id"] 881 | c_d_id = params["c_d_id"] 882 | c_id = params["c_id"] 883 | c_last = params["c_last"] 884 | h_date = params["h_date"] 885 | 886 | sID = self.getServer(w_id) 887 | cSID = self.getServer(c_w_id) 888 | 889 | customerQuery = self.conn[cSID][constants.TABLENAME_CUSTOMER].query 890 | warehouseQuery = self.conn[sID][constants.TABLENAME_WAREHOUSE].query 891 | districtQuery = self.conn[sID][constants.TABLENAME_DISTRICT].query 892 | 893 | if c_id != None: 894 | # getCustomerByCustomerId 895 | # SELECT C_ID, C_FIRST, C_MIDDLE, C_LAST, C_STREET_1, C_STREET_2, C_CITY, C_STATE, 896 | # C_ZIP, C_PHONE, C_SINCE, C_CREDIT, C_CREDIT_LIM, C_DISCOUNT, C_BALANCE, 897 | # C_YTD_PAYMENT, C_PAYMENT_CNT, C_DATA FROM CUSTOMER 898 | # WHERE C_W_ID = ? AND C_D_ID = ? AND C_ID = ? 899 | 900 | customers = customerQuery.filter(C_W_ID = c_w_id, C_D_ID = d_id, C_ID = c_id) 901 | customerInfo = customers.columns("C_ID", "C_BALANCE", "C_YTD_PAYMENT", "C_PAYMENT_CNT", "C_DATA")[0] 902 | else: 903 | # Get the midpoint customer's id 904 | # getCustomersByLastName 905 | # SELECT C_ID, C_FIRST, C_MIDDLE, C_LAST, C_STREET_1, C_STREET_2, C_CITY, C_STATE, 906 | # C_ZIP, C_PHONE, C_SINCE, C_CREDIT, C_CREDIT_LIM, C_DISCOUNT, C_BALANCE, 907 | # C_YTD_PAYMENT, C_PAYMENT_CNT, C_DATA FROM CUSTOMER 908 | # WHERE C_W_ID = ? AND C_D_ID = ? AND C_LAST = ? ORDER BY C_FIRST 909 | 910 | customers = customerQuery.filter(C_W_ID = c_w_id, C_D_ID = d_id, C_LAST__contains = c_last).order_by("C_FIRST", numeric=False) 911 | all_customers = customers.columns("C_ID", "C_BALANCE", "C_YTD_PAYMENT", "C_PAYMENT_CNT", "C_DATA") 912 | namecnt = len(all_customers) 913 | assert namecnt > 0 914 | index = (namecnt-1)/2 915 | customerInfo = all_customers[index] 916 | c_id = int(customerInfo["C_ID"]) 917 | assert len(customerInfo) > 0 918 | 919 | c_balance = float(customerInfo["C_BALANCE"]) - h_amount 920 | c_ytd_payment = float(customerInfo["C_YTD_PAYMENT"]) + h_amount 921 | c_payment_cnt = int(customerInfo["C_PAYMENT_CNT"]) + 1 922 | c_data = customerInfo["C_DATA"] 923 | 924 | # getWarehouse 925 | # SELECT W_NAME, W_STREET_1, W_STREET_2, W_CITY, W_STATE, W_ZIP FROM WAREHOUSE WHERE W_ID = ? 926 | 927 | warehouses = warehouseQuery.filter(W_ID = w_id) 928 | warehouseInfo = warehouses.columns("W_NAME","W_STREET_1", "W_STREET_2", "W_CITY", "W_STATE", "W_ZIP")[0] 929 | 930 | # updateWarehouseBalance 931 | # UPDATE WAREHOUSE SET W_YTD = W_YTD + ? WHERE W_ID = ? 932 | 933 | warehouses = warehouseQuery.filter(W_ID = w_id) 934 | records = list() 935 | for record in warehouses: 936 | key, cols = record 937 | cols["W_YTD"] = float(cols["W_YTD"]) + h_amount 938 | records.append((key, cols)) 939 | ## FOR 940 | self.conn[sID][constants.TABLENAME_WAREHOUSE].multi_set(records) 941 | 942 | # getDistrict 943 | # SELECT D_NAME, D_STREET_1, D_STREET_2, D_CITY, D_STATE, D_ZIP FROM DISTRICT 944 | # WHERE D_W_ID = ? AND D_ID = ? 945 | 946 | districts = districtQuery.filter(D_W_ID = w_id, D_ID = d_id) 947 | districtInfo = districts.columns("D_NAME", "D_STREET_1", "D_STREET_2", "D_CITY", "D_STATE", "D_ZIP")[0] 948 | 949 | # updateDistrictBalance 950 | # UPDATE DISTRICT SET D_YTD = D_YTD + ? WHERE D_W_ID = ? AND D_ID = ? 951 | 952 | districts = districtQuery.filter(W_ID = w_id, D_ID = d_id) 953 | records = list() 954 | for record in districts: 955 | key, cols = record 956 | cols["D_YTD"] = float(cols["D_YTD"]) + h_amount 957 | records.append((key, cols)) 958 | ## FOR 959 | self.conn[sID][constants.TABLENAME_DISTRICT].multi_set(records) 960 | 961 | # Customer Credit Information 962 | customers = customerQuery.filter(C_W_ID = c_w_id, C_D_ID = c_d_id, C_ID = c_id) 963 | cInfo = customers.columns("C_CREDIT")[0] 964 | 965 | if cInfo["C_CREDIT"] == constants.BAD_CREDIT: 966 | newData = " ".join(map(str, [c_id, c_d_id, c_w_id, d_id, w_id, h_amount])) 967 | c_data = (newData + "|" + c_data) 968 | if len(c_data) > constants.MAX_C_DATA: c_data = c_data[:constants.MAX_C_DATA] 969 | 970 | # updateBCCustomer 971 | # UPDATE CUSTOMER SET C_BALANCE = ?, C_YTD_PAYMENT = ?, C_PAYMENT_CNT = ?, C_DATA = ? 972 | # WHERE C_W_ID = ? AND C_D_ID = ? AND C_ID = ? 973 | 974 | records = list() 975 | for record in customers: 976 | key, cols = record 977 | cols["C_BALANCE"] = c_balance 978 | cols["C_YTD_PAYMENT"] = c_ytd_payment 979 | cols["C_PAYMENT_CNT"] = c_payment_cnt 980 | cols["C_DATA"] = c_data 981 | records.append((key, cols)) 982 | ## FOR 983 | self.conn[cSID][constants.TABLENAME_CUSTOMER].multi_set(records) 984 | else: 985 | c_data = "" 986 | 987 | records = list() 988 | # updateGCCustomer 989 | # UPDATE CUSTOMER SET C_BALANCE = ?, C_YTD_PAYMENT = ?, C_PAYMENT_CNT = ? 990 | # WHERE C_W_ID = ? AND C_D_ID = ? AND C_ID = ? 991 | 992 | for record in customers: 993 | key, cols = record 994 | cols["C_BALANCE"] = c_balance 995 | cols["C_YTD_PAYMENT"] = c_ytd_payment 996 | cols["C_PAYMENT_CNT"] = c_payment_cnt 997 | records.append((key, cols)) 998 | ## FOR 999 | self.conn[cSID][constants.TABLENAME_CUSTOMER].multi_set(records) 1000 | 1001 | # Concatenate w_name, four space, d_name 1002 | h_data = "%s %s" % (warehouseInfo["W_NAME"], districtInfo["D_NAME"]) 1003 | 1004 | # Create the history record 1005 | # INSERT INTO HISTORY VALUES (?, ?, ?, ?, ?, ?, ?, ?) 1006 | 1007 | h_key = self.tupleToString((c_id, c_d_id, c_w_id)) 1008 | 1009 | cols = {"H_C_ID": c_id, "H_C_D_ID": c_d_id, "H_C_W_ID": c_w_id, "H_D_ID": 1010 | d_id, "H_W_ID": w_id, "H_DATE": h_date, "H_AMOUNT": 1011 | h_amount, "H_DATA": h_data} 1012 | self.conn[sID][constants.TABLENAME_HISTORY].multi_set([(key, cols)]) 1013 | 1014 | # TPC-C 2.5.3.3: Must display the following fields: 1015 | # W_ID, D_ID, C_ID, C_D_ID, C_W_ID, W_STREET_1, W_STREET_2, W_CITY, 1016 | # W_STATE, W_ZIP, D_STREET_1, D_STREET_2, D_CITY, D_STATE, D_ZIP, 1017 | # C_FIRST, C_MIDDLE, C_LAST, C_STREET_1, C_STREET_2, C_CITY, C_STATE, 1018 | # C_ZIP, C_PHONE, C_SINCE, C_CREDIT, C_CREDIT_LIM, C_DISCOUNT, 1019 | # C_BALANCE, the first 200 characters of C_DATA (only if C_CREDIT = 1020 | # "BC"), H_AMMOUNT, and H_DATE. 1021 | 1022 | # Hand back all the warehouse, district, and customer data 1023 | return [ warehouseInfo, districtInfo, customerInfo ] 1024 | 1025 | def doStockLevel(self, params): 1026 | """Execute STOCK_LEVEL Transaction 1027 | Parameters Dict: 1028 | w_id 1029 | d_id 1030 | threshold 1031 | """ 1032 | w_id = params["w_id"] 1033 | d_id = params["d_id"] 1034 | threshold = params["threshold"] 1035 | 1036 | sID = self.getServer(w_id) 1037 | 1038 | districtQuery = self.conn[sID][constants.TABLENAME_DISTRICT].query 1039 | orderLineQuery = self.conn[sID][constants.TABLENAME_ORDER_LINE].query 1040 | stockQuery = self.conn[sID][constants.TABLENAME_STOCK].query 1041 | 1042 | # getOId 1043 | # "SELECT D_NEXT_O_ID FROM DISTRICT WHERE D_W_ID = ? AND D_ID = ?" 1044 | 1045 | districts = districtQuery.filter(D_W_ID = w_id, D_ID = d_id) 1046 | o_id = int(districts.columns("D_NEXT_O_ID")[0]["D_NEXT_O_ID"]) 1047 | 1048 | # getStockCount 1049 | # SELECT COUNT(DISTINCT(OL_I_ID)) FROM ORDER_LINE, STOCK WHERE OL_W_ID = ? AND OL_D_ID = ? 1050 | # AND OL_O_ID < ? AND OL_O_ID >= ? AND S_W_ID = ? AND S_I_ID = OL_I_ID AND S_QUANTITY < ? 1051 | 1052 | orders = orderLineQuery.filter(OL_W_ID = w_id, OL_D_ID = d_id, OL_O_ID__between = [(o_id-20), (o_id-1)]) 1053 | ol_i_ids = set([i["OL_I_ID"] for i in orders.columns("OL_I_ID")]) 1054 | 1055 | stocks = stockQuery.filter(S_W_ID = w_id, S_I_ID__in = list(ol_i_ids), S_QUANTITY__lt = threshold).columns("S_W_ID") 1056 | 1057 | cnt = len(stocks) 1058 | 1059 | return cnt 1060 | 1061 | ## CLASS 1062 | -------------------------------------------------------------------------------- /pytpcc/message.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # -*- coding: utf-8 -*- 3 | # ----------------------------------------------------------------------- 4 | # Copyright (C) 2011 5 | # Andy Pavlo & Yang Lu 6 | # http:##www.cs.brown.edu/~pavlo/ 7 | # 8 | # Permission is hereby granted, free of charge, to any person obtaining 9 | # a copy of this software and associated documentation files (the 10 | # "Software"), to deal in the Software without restriction, including 11 | # without limitation the rights to use, copy, modify, merge, publish, 12 | # distribute, sublicense, and/or sell copies of the Software, and to 13 | # permit persons to whom the Software is furnished to do so, subject to 14 | # the following conditions: 15 | # 16 | # The above copyright notice and this permission notice shall be 17 | # included in all copies or substantial portions of the Software. 18 | # 19 | # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, 20 | # EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF 21 | # MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT 22 | # IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR 23 | # OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, 24 | # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR 25 | # OTHER DEALINGS IN THE SOFTWARE. 26 | # ----------------------------------------------------------------------- 27 | 28 | import sys 29 | import os 30 | import string 31 | import datetime 32 | import logging 33 | import re 34 | import argparse 35 | import glob 36 | import time 37 | from pprint import pprint,pformat 38 | 39 | from util import * 40 | from runtime import * 41 | import drivers 42 | 43 | EMPTY = 0 44 | CMD_LOAD = 1 45 | CMD_EXECUTE = 2 46 | CMD_STOP = 3 47 | LOAD_COMPLETED = 4 48 | EXECUTE_COMPLETED = 5 49 | 50 | class Message: 51 | def __init__(self,header=EMPTY,data=None): 52 | self.header=header 53 | self.data=data 54 | -------------------------------------------------------------------------------- /pytpcc/runtime/__init__.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | 3 | __all__ = ["executor", "loader"] 4 | -------------------------------------------------------------------------------- /pytpcc/runtime/executor.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | # ----------------------------------------------------------------------- 3 | # Copyright (C) 2011 4 | # Andy Pavlo 5 | # http://www.cs.brown.edu/~pavlo/ 6 | # 7 | # Original Java Version: 8 | # Copyright (C) 2008 9 | # Evan Jones 10 | # Massachusetts Institute of Technology 11 | # 12 | # Permission is hereby granted, free of charge, to any person obtaining 13 | # a copy of this software and associated documentation files (the 14 | # "Software"), to deal in the Software without restriction, including 15 | # without limitation the rights to use, copy, modify, merge, publish, 16 | # distribute, sublicense, and/or sell copies of the Software, and to 17 | # permit persons to whom the Software is furnished to do so, subject to 18 | # the following conditions: 19 | # 20 | # The above copyright notice and this permission notice shall be 21 | # included in all copies or substantial portions of the Software. 22 | # 23 | # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, 24 | # EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF 25 | # MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT 26 | # IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR 27 | # OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, 28 | # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR 29 | # OTHER DEALINGS IN THE SOFTWARE. 30 | # ----------------------------------------------------------------------- 31 | 32 | import sys 33 | import multiprocessing 34 | import time 35 | import random 36 | import traceback 37 | import logging 38 | from datetime import datetime 39 | from pprint import pprint,pformat 40 | 41 | import constants 42 | from util import * 43 | 44 | 45 | class Executor: 46 | 47 | def __init__(self, driver, scaleParameters, stop_on_error = False): 48 | self.driver = driver 49 | self.scaleParameters = scaleParameters 50 | self.stop_on_error = stop_on_error 51 | ## DEF 52 | 53 | def execute(self, duration): 54 | global_result = results.Results() 55 | assert global_result, "Failed to return a Results object" 56 | logging.debug("Executing benchmark for %d seconds" % duration) 57 | start = global_result.startBenchmark() 58 | debug = logging.getLogger().isEnabledFor(logging.DEBUG) 59 | # Batch Results 60 | batch_result = results.Results() 61 | start_batch = batch_result.startBenchmark() 62 | while (time.time() - start) <= duration: 63 | txn, params = self.doOne() 64 | global_txn_id = global_result.startTransaction(txn) 65 | batch_txn_id = batch_result.startTransaction(txn) 66 | if debug: logging.debug("Executing '%s' transaction" % txn) 67 | try: 68 | (val, retries) = self.driver.executeTransaction(txn, params) 69 | except KeyboardInterrupt: 70 | return -1 71 | except (Exception, AssertionError) as ex: 72 | logging.warn("Failed to execute Transaction '%s': %s" % (txn, ex)) 73 | traceback.print_exc(file=sys.stdout) 74 | print("Aborting some transaction with some error %s %s" % (txn, ex)) 75 | global_result.abortTransaction(global_txn_id) 76 | batch_result.abortTransaction(batch_txn_id) 77 | if self.stop_on_error: raise 78 | continue 79 | 80 | if val is None: 81 | global_result.abortTransaction(global_txn_id, retries) 82 | batch_result.abortTransaction(batch_txn_id, retries) 83 | continue 84 | 85 | batch_result.stopTransaction(batch_txn_id, retries) 86 | global_result.stopTransaction(global_txn_id, retries) 87 | 88 | if time.time() - start_batch > 900: # every 15 minutes 89 | batch_result.stopBenchmark() 90 | logging.info(batch_result.show()) 91 | batch_result = results.Results() 92 | start_batch = batch_result.startBenchmark() 93 | 94 | ## WHILE 95 | batch_result.stopBenchmark() 96 | global_result.stopBenchmark() 97 | return (global_result) 98 | ## DEF 99 | 100 | def doOne(self): 101 | """Selects and executes a transaction at random. The number of new order transactions executed per minute is the official "tpmC" metric. See TPC-C 5.4.2 (page 71).""" 102 | 103 | ## This is not strictly accurate: The requirement is for certain 104 | ## *minimum* percentages to be maintained. This is close to the right 105 | ## thing, but not precisely correct. See TPC-C 5.2.4 (page 68). 106 | x = rand.number(1, 100) 107 | params = None 108 | txn = None 109 | if x <= 4: ## 4% 110 | txn, params = (constants.TransactionTypes.STOCK_LEVEL, self.generateStockLevelParams()) 111 | elif x <= 4 + 4: ## 4% 112 | txn, params = (constants.TransactionTypes.DELIVERY, self.generateDeliveryParams()) 113 | elif x <= 4 + 4 + 4: ## 4% 114 | txn, params = (constants.TransactionTypes.ORDER_STATUS, self.generateOrderStatusParams()) 115 | elif x <= 43 + 4 + 4 + 4: ## 43% 116 | txn, params = (constants.TransactionTypes.PAYMENT, self.generatePaymentParams()) 117 | else: ## 45% 118 | assert x > 100 - 45, "Random number wasn't within specified range or percentages don't add up (%d)" % x 119 | txn, params = (constants.TransactionTypes.NEW_ORDER, self.generateNewOrderParams()) 120 | 121 | return (txn, params) 122 | ## DEF 123 | 124 | ## ---------------------------------------------- 125 | ## generateDeliveryParams 126 | ## ---------------------------------------------- 127 | def generateDeliveryParams(self): 128 | """Return parameters for DELIVERY""" 129 | w_id = self.makeWarehouseId() 130 | o_carrier_id = rand.number(constants.MIN_CARRIER_ID, constants.MAX_CARRIER_ID) 131 | ol_delivery_d = datetime.now() 132 | return makeParameterDict(locals(), "w_id", "o_carrier_id", "ol_delivery_d") 133 | ## DEF 134 | 135 | ## ---------------------------------------------- 136 | ## generateNewOrderParams 137 | ## ---------------------------------------------- 138 | def generateNewOrderParams(self): 139 | """Return parameters for NEW_ORDER""" 140 | w_id = self.makeWarehouseId() 141 | d_id = self.makeDistrictId() 142 | c_id = self.makeCustomerId() 143 | ol_cnt = rand.number(constants.MIN_OL_CNT, constants.MAX_OL_CNT) 144 | o_entry_d = datetime.now() 145 | 146 | ## 1% of transactions roll back 147 | rollback = rand.number(1, 100) == 1 148 | 149 | i_ids = [ ] 150 | i_w_ids = [ ] 151 | i_qtys = [ ] 152 | for i in range(0, ol_cnt): 153 | if rollback and i + 1 == ol_cnt: 154 | i_ids.append(self.scaleParameters.items + 1) 155 | else: 156 | i_id = self.makeItemId() 157 | while i_id in i_ids: 158 | i_id = self.makeItemId() 159 | i_ids.append(i_id) 160 | 161 | ## 1% of items are from a remote warehouse 162 | remote = (rand.number(1, 100) == 1) 163 | if self.scaleParameters.warehouses > 1 and remote: 164 | i_w_ids.append(rand.numberExcluding(self.scaleParameters.starting_warehouse, self.scaleParameters.ending_warehouse, w_id)) 165 | else: 166 | i_w_ids.append(w_id) 167 | 168 | i_qtys.append(rand.number(1, constants.MAX_OL_QUANTITY)) 169 | ## FOR 170 | 171 | return makeParameterDict(locals(), "w_id", "d_id", "c_id", "o_entry_d", "i_ids", "i_w_ids", "i_qtys") 172 | ## DEF 173 | 174 | ## ---------------------------------------------- 175 | ## generateOrderStatusParams 176 | ## ---------------------------------------------- 177 | def generateOrderStatusParams(self): 178 | """Return parameters for ORDER_STATUS""" 179 | w_id = self.makeWarehouseId() 180 | d_id = self.makeDistrictId() 181 | c_last = None 182 | c_id = None 183 | 184 | ## 60%: order status by last name 185 | if rand.number(1, 100) <= 60: 186 | c_last = rand.makeRandomLastName(self.scaleParameters.customersPerDistrict) 187 | 188 | ## 40%: order status by id 189 | else: 190 | c_id = self.makeCustomerId() 191 | 192 | return makeParameterDict(locals(), "w_id", "d_id", "c_id", "c_last") 193 | ## DEF 194 | 195 | ## ---------------------------------------------- 196 | ## generatePaymentParams 197 | ## ---------------------------------------------- 198 | def generatePaymentParams(self): 199 | """Return parameters for PAYMENT""" 200 | x = rand.number(1, 100) 201 | y = rand.number(1, 100) 202 | 203 | w_id = self.makeWarehouseId() 204 | d_id = self.makeDistrictId() 205 | c_w_id = None 206 | c_d_id = None 207 | c_id = None 208 | c_last = None 209 | h_amount = rand.fixedPoint(2, constants.MIN_PAYMENT, constants.MAX_PAYMENT) 210 | h_date = datetime.now() 211 | 212 | ## 85%: paying through own warehouse (or there is only 1 warehouse) 213 | if self.scaleParameters.warehouses == 1 or x <= 85: 214 | c_w_id = w_id 215 | c_d_id = d_id 216 | ## 15%: paying through another warehouse: 217 | else: 218 | ## select in range [1, num_warehouses] excluding w_id 219 | c_w_id = rand.numberExcluding(self.scaleParameters.starting_warehouse, self.scaleParameters.ending_warehouse, w_id) 220 | assert c_w_id != w_id, "Failed to generate W_ID that's not equal to C_W_ID" 221 | c_d_id = self.makeDistrictId() 222 | 223 | ## 60%: payment by last name 224 | if y <= 60: 225 | c_last = rand.makeRandomLastName(self.scaleParameters.customersPerDistrict) 226 | ## 40%: payment by id 227 | else: 228 | assert y > 60, "Bad random payment value generated %d" % y 229 | c_id = self.makeCustomerId() 230 | 231 | return makeParameterDict(locals(), "w_id", "d_id", "h_amount", "c_w_id", "c_d_id", "c_id", "c_last", "h_date") 232 | ## DEF 233 | 234 | ## ---------------------------------------------- 235 | ## generateStockLevelParams 236 | ## ---------------------------------------------- 237 | def generateStockLevelParams(self): 238 | """Returns parameters for STOCK_LEVEL""" 239 | w_id = self.makeWarehouseId() 240 | d_id = self.makeDistrictId() 241 | threshold = rand.number(constants.MIN_STOCK_LEVEL_THRESHOLD, constants.MAX_STOCK_LEVEL_THRESHOLD) 242 | return makeParameterDict(locals(), "w_id", "d_id", "threshold") 243 | ## DEF 244 | 245 | def makeWarehouseId(self): 246 | w_id = rand.number(self.scaleParameters.starting_warehouse, self.scaleParameters.ending_warehouse) 247 | assert(w_id >= self.scaleParameters.starting_warehouse), "Invalid W_ID: %d" % w_id 248 | assert(w_id <= self.scaleParameters.ending_warehouse), "Invalid W_ID: %d" % w_id 249 | return w_id 250 | ## DEF 251 | 252 | def makeDistrictId(self): 253 | return rand.number(1, self.scaleParameters.districtsPerWarehouse) 254 | ## DEF 255 | 256 | def makeCustomerId(self): 257 | return rand.NURand(1023, 1, self.scaleParameters.customersPerDistrict) 258 | ## DEF 259 | 260 | def makeItemId(self): 261 | return rand.NURand(8191, 1, self.scaleParameters.items) 262 | ## DEF 263 | ## CLASS 264 | 265 | def makeParameterDict(values, *args): 266 | return dict(map(lambda x: (x, values[x]), args)) 267 | ## DEF 268 | -------------------------------------------------------------------------------- /pytpcc/runtime/loader.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | # ----------------------------------------------------------------------- 3 | # Copyright (C) 2011 4 | # Andy Pavlo 5 | # http:##www.cs.brown.edu/~pavlo/ 6 | # 7 | # Original Java Version: 8 | # Copyright (C) 2008 9 | # Evan Jones 10 | # Massachusetts Institute of Technology 11 | # 12 | # Permission is hereby granted, free of charge, to any person obtaining 13 | # a copy of this software and associated documentation files (the 14 | # "Software"), to deal in the Software without restriction, including 15 | # without limitation the rights to use, copy, modify, merge, publish, 16 | # distribute, sublicense, and/or sell copies of the Software, and to 17 | # permit persons to whom the Software is furnished to do so, subject to 18 | # the following conditions: 19 | # 20 | # The above copyright notice and this permission notice shall be 21 | # included in all copies or substantial portions of the Software. 22 | # 23 | # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, 24 | # EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF 25 | # MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT 26 | # IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR 27 | # OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, 28 | # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR 29 | # OTHER DEALINGS IN THE SOFTWARE. 30 | # ----------------------------------------------------------------------- 31 | 32 | import os 33 | import sys 34 | 35 | import logging 36 | from datetime import datetime 37 | from random import shuffle 38 | from pprint import pprint,pformat 39 | 40 | import constants 41 | from util import * 42 | 43 | class Loader: 44 | 45 | def __init__(self, handle, scaleParameters, w_ids, needLoadItems): 46 | self.handle = handle 47 | self.scaleParameters = scaleParameters 48 | self.w_ids = w_ids 49 | self.needLoadItems = needLoadItems 50 | self.batch_size = 500 51 | 52 | ## ============================================== 53 | ## execute 54 | ## ============================================== 55 | def execute(self): 56 | 57 | ## Item Table 58 | if self.needLoadItems: 59 | logging.debug("Loading ITEM table") 60 | self.loadItems() 61 | self.handle.loadFinishItem() 62 | 63 | ## Then create the warehouse-specific tuples 64 | for w_id in self.w_ids: 65 | self.loadWarehouse(w_id) 66 | self.handle.loadFinishWarehouse(w_id) 67 | ## FOR 68 | 69 | return (None) 70 | 71 | ## ============================================== 72 | ## loadItems 73 | ## ============================================== 74 | def loadItems(self): 75 | ## Select 10% of the rows to be marked "original" 76 | originalRows = rand.selectUniqueIds(self.scaleParameters.items // 10, 1, self.scaleParameters.items) 77 | 78 | ## Load all of the items 79 | tuples = [ ] 80 | total_tuples = 0 81 | for i in range(1, self.scaleParameters.items+1): 82 | original = (i in originalRows) 83 | tuples.append(self.generateItem(i, original)) 84 | total_tuples += 1 85 | if len(tuples) == self.batch_size: 86 | logging.debug("LOAD - %s: %5d / %d" % (constants.TABLENAME_ITEM, total_tuples, self.scaleParameters.items)) 87 | self.handle.loadTuples(constants.TABLENAME_ITEM, tuples) 88 | tuples = [ ] 89 | ## FOR 90 | if len(tuples) > 0: 91 | logging.debug("LOAD - %s: %5d / %d" % (constants.TABLENAME_ITEM, total_tuples, self.scaleParameters.items)) 92 | self.handle.loadTuples(constants.TABLENAME_ITEM, tuples) 93 | ## DEF 94 | 95 | ## ============================================== 96 | ## loadWarehouse 97 | ## ============================================== 98 | def loadWarehouse(self, w_id): 99 | logging.debug("LOAD - %s: %d / %d" % (constants.TABLENAME_WAREHOUSE, w_id, len(self.w_ids))) 100 | 101 | ## WAREHOUSE 102 | w_tuples = [ self.generateWarehouse(w_id) ] 103 | self.handle.loadTuples(constants.TABLENAME_WAREHOUSE, w_tuples) 104 | 105 | ## DISTRICT 106 | d_tuples = [ ] 107 | for d_id in range(1, self.scaleParameters.districtsPerWarehouse+1): 108 | d_next_o_id = self.scaleParameters.customersPerDistrict + 1 109 | d_tuples = [ self.generateDistrict(w_id, d_id, d_next_o_id) ] 110 | 111 | c_tuples = [ ] 112 | h_tuples = [ ] 113 | 114 | ## Select 10% of the customers to have bad credit 115 | selectedRows = rand.selectUniqueIds(self.scaleParameters.customersPerDistrict // 10, 1, self.scaleParameters.customersPerDistrict) 116 | 117 | ## TPC-C 4.3.3.1. says that o_c_id should be a permutation of [1, 3000]. But since it 118 | ## is a c_id field, it seems to make sense to have it be a permutation of the 119 | ## customers. For the "real" thing this will be equivalent 120 | cIdPermutation = [ ] 121 | 122 | for c_id in range(1, self.scaleParameters.customersPerDistrict+1): 123 | badCredit = (c_id in selectedRows) 124 | c_tuples.append(self.generateCustomer(w_id, d_id, c_id, badCredit, True)) 125 | h_tuples.append(self.generateHistory(w_id, d_id, c_id)) 126 | cIdPermutation.append(c_id) 127 | ## FOR 128 | assert cIdPermutation[0] == 1 129 | assert cIdPermutation[self.scaleParameters.customersPerDistrict - 1] == self.scaleParameters.customersPerDistrict 130 | shuffle(cIdPermutation) 131 | 132 | o_tuples = [ ] 133 | ol_tuples = [ ] 134 | no_tuples = [ ] 135 | 136 | for o_id in range(1, self.scaleParameters.customersPerDistrict+1): 137 | o_ol_cnt = rand.number(constants.MIN_OL_CNT, constants.MAX_OL_CNT) 138 | 139 | ## The last newOrdersPerDistrict are new orders 140 | newOrder = ((self.scaleParameters.customersPerDistrict - self.scaleParameters.newOrdersPerDistrict) < o_id) 141 | o_tuples.append(self.generateOrder(w_id, d_id, o_id, cIdPermutation[o_id - 1], o_ol_cnt, newOrder)) 142 | 143 | ## Generate each OrderLine for the order 144 | for ol_number in range(0, o_ol_cnt): 145 | ol_tuples.append(self.generateOrderLine(w_id, d_id, o_id, ol_number, self.scaleParameters.items, newOrder)) 146 | ## FOR 147 | 148 | ## This is a new order: make one for it 149 | if newOrder: no_tuples.append([o_id, d_id, w_id]) 150 | ## FOR 151 | 152 | self.handle.loadTuples(constants.TABLENAME_DISTRICT, d_tuples) 153 | self.handle.loadTuples(constants.TABLENAME_CUSTOMER, c_tuples) 154 | self.handle.loadTuples(constants.TABLENAME_ORDERS, o_tuples) 155 | self.handle.loadTuples(constants.TABLENAME_ORDER_LINE, ol_tuples) 156 | self.handle.loadTuples(constants.TABLENAME_NEW_ORDER, no_tuples) 157 | self.handle.loadTuples(constants.TABLENAME_HISTORY, h_tuples) 158 | self.handle.loadFinishDistrict(w_id, d_id) 159 | ## FOR 160 | 161 | ## Select 10% of the stock to be marked "original" 162 | s_tuples = [ ] 163 | selectedRows = rand.selectUniqueIds(self.scaleParameters.items // 10, 1, self.scaleParameters.items) 164 | total_tuples = 0 165 | for i_id in range(1, self.scaleParameters.items+1): 166 | original = (i_id in selectedRows) 167 | s_tuples.append(self.generateStock(w_id, i_id, original)) 168 | if len(s_tuples) >= self.batch_size: 169 | logging.debug("LOAD - %s [W_ID=%d]: %5d / %d" % (constants.TABLENAME_STOCK, w_id, total_tuples, self.scaleParameters.items)) 170 | self.handle.loadTuples(constants.TABLENAME_STOCK, s_tuples) 171 | s_tuples = [ ] 172 | total_tuples += 1 173 | ## FOR 174 | if len(s_tuples) > 0: 175 | logging.debug("LOAD - %s [W_ID=%d]: %5d / %d" % (constants.TABLENAME_STOCK, w_id, total_tuples, self.scaleParameters.items)) 176 | self.handle.loadTuples(constants.TABLENAME_STOCK, s_tuples) 177 | ## DEF 178 | 179 | ## ============================================== 180 | ## generateItem 181 | ## ============================================== 182 | def generateItem(self, id, original): 183 | i_id = id 184 | i_im_id = rand.number(constants.MIN_IM, constants.MAX_IM) 185 | i_name = rand.astring(constants.MIN_I_NAME, constants.MAX_I_NAME) 186 | i_price = rand.fixedPoint(constants.MONEY_DECIMALS, constants.MIN_PRICE, constants.MAX_PRICE) 187 | i_data = rand.astring(constants.MIN_I_DATA, constants.MAX_I_DATA) 188 | if original: i_data = self.fillOriginal(i_data) 189 | 190 | return [i_id, i_im_id, i_name, i_price, i_data] 191 | ## DEF 192 | 193 | ## ============================================== 194 | ## generateWarehouse 195 | ## ============================================== 196 | def generateWarehouse(self, w_id): 197 | w_tax = self.generateTax() 198 | w_ytd = constants.INITIAL_W_YTD 199 | w_address = self.generateAddress() 200 | return [w_id] + w_address + [w_tax, w_ytd] 201 | ## DEF 202 | 203 | ## ============================================== 204 | ## generateDistrict 205 | ## ============================================== 206 | def generateDistrict(self, d_w_id, d_id, d_next_o_id): 207 | d_tax = self.generateTax() 208 | d_ytd = constants.INITIAL_D_YTD 209 | d_address = self.generateAddress() 210 | return [d_id, d_w_id] + d_address + [d_tax, d_ytd, d_next_o_id] 211 | ## DEF 212 | 213 | ## ============================================== 214 | ## generateCustomer 215 | ## ============================================== 216 | def generateCustomer(self, c_w_id, c_d_id, c_id, badCredit, doesReplicateName): 217 | c_first = rand.astring(constants.MIN_FIRST, constants.MAX_FIRST) 218 | c_middle = constants.MIDDLE 219 | 220 | assert 1 <= c_id and c_id <= constants.CUSTOMERS_PER_DISTRICT 221 | if c_id <= 1000: 222 | c_last = rand.makeLastName(c_id - 1) 223 | else: 224 | c_last = rand.makeRandomLastName(constants.CUSTOMERS_PER_DISTRICT) 225 | 226 | c_phone = rand.nstring(constants.PHONE, constants.PHONE) 227 | c_since = datetime.now() 228 | c_credit = constants.BAD_CREDIT if badCredit else constants.GOOD_CREDIT 229 | c_credit_lim = constants.INITIAL_CREDIT_LIM 230 | c_discount = rand.fixedPoint(constants.DISCOUNT_DECIMALS, constants.MIN_DISCOUNT, constants.MAX_DISCOUNT) 231 | c_balance = constants.INITIAL_BALANCE 232 | c_ytd_payment = constants.INITIAL_YTD_PAYMENT 233 | c_payment_cnt = constants.INITIAL_PAYMENT_CNT 234 | c_delivery_cnt = constants.INITIAL_DELIVERY_CNT 235 | c_data = rand.astring(constants.MIN_C_DATA, constants.MAX_C_DATA) 236 | 237 | c_street1 = rand.astring(constants.MIN_STREET, constants.MAX_STREET) 238 | c_street2 = rand.astring(constants.MIN_STREET, constants.MAX_STREET) 239 | c_city = rand.astring(constants.MIN_CITY, constants.MAX_CITY) 240 | c_state = rand.astring(constants.STATE, constants.STATE) 241 | c_zip = self.generateZip() 242 | 243 | return [ c_id, c_d_id, c_w_id, c_first, c_middle, c_last, \ 244 | c_street1, c_street2, c_city, c_state, c_zip, \ 245 | c_phone, c_since, c_credit, c_credit_lim, c_discount, c_balance, \ 246 | c_ytd_payment, c_payment_cnt, c_delivery_cnt, c_data ] 247 | ## DEF 248 | 249 | ## ============================================== 250 | ## generateOrder 251 | ## ============================================== 252 | def generateOrder(self, o_w_id, o_d_id, o_id, o_c_id, o_ol_cnt, newOrder): 253 | """Returns the generated o_ol_cnt value.""" 254 | o_entry_d = datetime.now() 255 | o_carrier_id = constants.NULL_CARRIER_ID if newOrder else rand.number(constants.MIN_CARRIER_ID, constants.MAX_CARRIER_ID) 256 | o_all_local = constants.INITIAL_ALL_LOCAL 257 | return [ o_id, o_c_id, o_d_id, o_w_id, o_entry_d, o_carrier_id, o_ol_cnt, o_all_local ] 258 | ## DEF 259 | 260 | ## ============================================== 261 | ## generateOrderLine 262 | ## ============================================== 263 | def generateOrderLine(self, ol_w_id, ol_d_id, ol_o_id, ol_number, max_items, newOrder): 264 | ol_i_id = rand.number(1, max_items) 265 | ol_supply_w_id = ol_w_id 266 | ol_delivery_d = datetime.now() 267 | ol_quantity = constants.INITIAL_QUANTITY 268 | 269 | ## 1% of items are from a remote warehouse 270 | remote = (rand.number(1, 100) == 1) 271 | if self.scaleParameters.warehouses > 1 and remote: 272 | ol_supply_w_id = rand.numberExcluding(self.scaleParameters.starting_warehouse, 273 | self.scaleParameters.ending_warehouse, 274 | ol_w_id) 275 | 276 | ol_amount = rand.fixedPoint(constants.MONEY_DECIMALS, constants.MIN_AMOUNT, constants.MAX_PRICE * constants.MAX_OL_QUANTITY) 277 | if newOrder: 278 | ol_delivery_d = None 279 | ol_dist_info = rand.astring(constants.DIST, constants.DIST) 280 | 281 | return [ ol_o_id, ol_d_id, ol_w_id, ol_number, ol_i_id, ol_supply_w_id, ol_delivery_d, ol_quantity, ol_amount, ol_dist_info ] 282 | ## DEF 283 | 284 | ## ============================================== 285 | ## generateStock 286 | ## ============================================== 287 | def generateStock(self, s_w_id, s_i_id, original): 288 | s_quantity = rand.number(constants.MIN_QUANTITY, constants.MAX_QUANTITY); 289 | s_ytd = 0; 290 | s_order_cnt = 0; 291 | s_remote_cnt = 0; 292 | 293 | s_data = rand.astring(constants.MIN_I_DATA, constants.MAX_I_DATA); 294 | if original: self.fillOriginal(s_data) 295 | 296 | s_dists = [ ] 297 | for i in range(0, constants.DISTRICTS_PER_WAREHOUSE): 298 | s_dists.append(rand.astring(constants.DIST, constants.DIST)) 299 | 300 | return [ s_i_id, s_w_id, s_quantity ] + \ 301 | s_dists + \ 302 | [ s_ytd, s_order_cnt, s_remote_cnt, s_data ] 303 | ## DEF 304 | 305 | ## ============================================== 306 | ## generateHistory 307 | ## ============================================== 308 | def generateHistory(self, h_c_w_id, h_c_d_id, h_c_id): 309 | h_w_id = h_c_w_id 310 | h_d_id = h_c_d_id 311 | h_date = datetime.now() 312 | h_amount = constants.INITIAL_AMOUNT 313 | h_data = rand.astring(constants.MIN_DATA, constants.MAX_DATA) 314 | return [ h_c_id, h_c_d_id, h_c_w_id, h_d_id, h_w_id, h_date, h_amount, h_data ] 315 | ## DEF 316 | 317 | ## ============================================== 318 | ## generateAddress 319 | ## ============================================== 320 | def generateAddress(self): 321 | """ 322 | Returns a name and a street address 323 | Used by both generateWarehouse and generateDistrict. 324 | """ 325 | name = rand.astring(constants.MIN_NAME, constants.MAX_NAME) 326 | return [ name ] + self.generateStreetAddress() 327 | ## DEF 328 | 329 | ## ============================================== 330 | ## generateStreetAddress 331 | ## ============================================== 332 | def generateStreetAddress(self): 333 | """ 334 | Returns a list for a street address 335 | Used for warehouses, districts and customers. 336 | """ 337 | street1 = rand.astring(constants.MIN_STREET, constants.MAX_STREET) 338 | street2 = rand.astring(constants.MIN_STREET, constants.MAX_STREET) 339 | city = rand.astring(constants.MIN_CITY, constants.MAX_CITY) 340 | state = rand.astring(constants.STATE, constants.STATE) 341 | zip = self.generateZip() 342 | 343 | return [ street1, street2, city, state, zip ] 344 | ## DEF 345 | 346 | ## ============================================== 347 | ## generateTax 348 | ## ============================================== 349 | def generateTax(self): 350 | return rand.fixedPoint(constants.TAX_DECIMALS, constants.MIN_TAX, constants.MAX_TAX) 351 | ## DEF 352 | 353 | ## ============================================== 354 | ## generateZip 355 | ## ============================================== 356 | def generateZip(self): 357 | length = constants.ZIP_LENGTH - len(constants.ZIP_SUFFIX) 358 | return rand.nstring(length, length) + constants.ZIP_SUFFIX 359 | ## DEF 360 | 361 | ## ============================================== 362 | ## fillOriginal 363 | ## ============================================== 364 | def fillOriginal(self, data): 365 | """ 366 | a string with ORIGINAL_STRING at a random position 367 | """ 368 | originalLength = len(constants.ORIGINAL_STRING) 369 | position = rand.number(0, len(data) - originalLength) 370 | out = data[:position] + constants.ORIGINAL_STRING + data[position + originalLength:] 371 | assert len(out) == len(data) 372 | return out 373 | ## DEF 374 | ## CLASS 375 | -------------------------------------------------------------------------------- /pytpcc/tpcc.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # -*- coding: utf-8 -*- 3 | # ----------------------------------------------------------------------- 4 | # Copyright (C) 2011 5 | # Andy Pavlo 6 | # http:##www.cs.brown.edu/~pavlo/ 7 | # 8 | # Permission is hereby granted, free of charge, to any person obtaining 9 | # a copy of this software and associated documentation files (the 10 | # "Software"), to deal in the Software without restriction, including 11 | # without limitation the rights to use, copy, modify, merge, publish, 12 | # distribute, sublicense, and/or sell copies of the Software, and to 13 | # permit persons to whom the Software is furnished to do so, subject to 14 | # the following conditions: 15 | # 16 | # The above copyright notice and this permission notice shall be 17 | # included in all copies or substantial portions of the Software. 18 | # 19 | # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, 20 | # EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF 21 | # MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT 22 | # IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR 23 | # OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, 24 | # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR 25 | # OTHER DEALINGS IN THE SOFTWARE. 26 | # ----------------------------------------------------------------------- 27 | 28 | import sys 29 | import os 30 | import string 31 | import datetime 32 | import logging 33 | import re 34 | import argparse 35 | import glob 36 | import time 37 | import multiprocessing 38 | from configparser import ConfigParser 39 | from pprint import pprint, pformat 40 | 41 | from util import results, scaleparameters 42 | from runtime import executor, loader 43 | 44 | logging.basicConfig(level=logging.INFO, 45 | format="%(asctime)s [%(funcName)s:%(lineno)03d] %(levelname)-5s: %(message)s", 46 | datefmt="%m-%d-%Y %H:%M:%S", 47 | # 48 | filename='results.log') 49 | 50 | console = logging.StreamHandler() 51 | console.setLevel(logging.INFO) 52 | console.setFormatter(logging.Formatter( 53 | '%(asctime)s [%(funcName)s:%(lineno)03d] %(levelname)-5s: %(message)s')) 54 | logging.getLogger('').addHandler(console) 55 | 56 | 57 | ## ============================================== 58 | ## createDriverClass 59 | ## ============================================== 60 | def createDriverClass(name): 61 | full_name = "%sDriver" % name.title() 62 | mod = __import__('drivers.%s' % full_name.lower(), globals(), locals(), [full_name]) 63 | klass = getattr(mod, full_name) 64 | return klass 65 | ## DEF 66 | 67 | ## ============================================== 68 | ## getDrivers 69 | ## ============================================== 70 | def getDrivers(): 71 | drivers = [] 72 | for f in [os.path.basename(drv).replace("driver.py", "") for drv in glob.glob("./drivers/*driver.py")]: 73 | if f != "abstract": 74 | drivers.append(f) 75 | return drivers 76 | ## DEF 77 | 78 | ## ============================================== 79 | ## startLoading 80 | ## ============================================== 81 | def startLoading(driverClass, scaleParameters, args, config): 82 | logging.debug("Creating client pool with %d processes", args['clients']) 83 | pool = multiprocessing.Pool(args['clients']) 84 | 85 | # Split the warehouses into chunks 86 | w_ids = [[] for _ in range(args['clients'])] 87 | for w_id in range(scaleParameters.starting_warehouse, scaleParameters.ending_warehouse+1): 88 | idx = w_id % args['clients'] 89 | w_ids[idx].append(w_id) 90 | ## FOR 91 | 92 | loader_results = [] 93 | try: 94 | del args['config'] 95 | except KeyError: 96 | print() 97 | for i in range(args['clients']): 98 | r = pool.apply_async(loaderFunc, (driverClass, scaleParameters, args, config, w_ids[i])) 99 | loader_results.append(r) 100 | ## FOR 101 | 102 | pool.close() 103 | logging.debug("Waiting for %d loaders to finish", args['clients']) 104 | pool.join() 105 | ## DEF 106 | 107 | ## ============================================== 108 | ## loaderFunc 109 | ## ============================================== 110 | def loaderFunc(driverClass, scaleParameters, args, config, w_ids): 111 | driver = driverClass(args['ddl']) 112 | assert driver != None, "Driver in loadFunc is none!" 113 | logging.debug("Starting client execution: %s [warehouses=%d]", driver, len(w_ids)) 114 | 115 | config['load'] = True 116 | config['execute'] = False 117 | config['reset'] = False 118 | config['warehouses'] = args['warehouses'] 119 | driver.loadConfig(config) 120 | 121 | try: 122 | loadItems = (1 in w_ids) 123 | l = loader.Loader(driver, scaleParameters, w_ids, loadItems) 124 | driver.loadStart() 125 | l.execute() 126 | driver.loadFinish() 127 | except KeyboardInterrupt: 128 | return -1 129 | except (Exception, AssertionError) as ex: 130 | logging.warn("Failed to load data: %s", ex) 131 | raise 132 | 133 | ## DEF 134 | 135 | ## ============================================== 136 | ## startExecution 137 | ## ============================================== 138 | def startExecution(driverClass, scaleParameters, args, config): 139 | logging.debug("Creating client pool with %d processes", args['clients']) 140 | pool = multiprocessing.Pool(args['clients']) 141 | debug = logging.getLogger().isEnabledFor(logging.DEBUG) 142 | try: 143 | del args['config'] 144 | except KeyError: 145 | print() 146 | worker_results = [] 147 | for _ in range(args['clients']): 148 | r = pool.apply_async(executorFunc, (driverClass, scaleParameters, args, config, debug,)) 149 | worker_results.append(r) 150 | ## FOR 151 | pool.close() 152 | pool.join() 153 | 154 | total_results = results.Results() 155 | for asyncr in worker_results: 156 | asyncr.wait() 157 | r = asyncr.get() 158 | assert r != None, "No results object returned by thread!" 159 | if r == -1: 160 | sys.exit(1) 161 | total_results.append(r) 162 | ## FOR 163 | 164 | return total_results 165 | ## DEF 166 | 167 | ## ============================================== 168 | ## executorFunc 169 | ## ============================================== 170 | def executorFunc(driverClass, scaleParameters, args, config, debug): 171 | driver = driverClass(args['ddl']) 172 | assert driver != None, "No driver in executorFunc" 173 | logging.debug("Starting client execution: %s", driver) 174 | 175 | config['execute'] = True 176 | config['reset'] = False 177 | driver.loadConfig(config) 178 | 179 | e = executor.Executor(driver, scaleParameters, stop_on_error=args['stop_on_error']) 180 | driver.executeStart() 181 | results = e.execute(args['duration']) 182 | driver.executeFinish() 183 | 184 | return results 185 | ## DEF 186 | 187 | ## ============================================== 188 | ## main 189 | ## ============================================== 190 | if __name__ == '__main__': 191 | aparser = argparse.ArgumentParser(description='Python implementation of the TPC-C Benchmark') 192 | aparser.add_argument('system', choices=getDrivers(), 193 | help='Target system driver') 194 | aparser.add_argument('--config', type=open, 195 | help='Path to driver configuration file') 196 | aparser.add_argument('--reset', action='store_true', 197 | help='Instruct the driver to reset the contents of the database') 198 | aparser.add_argument('--scalefactor', default=1, type=float, metavar='SF', 199 | help='Benchmark scale factor') 200 | aparser.add_argument('--warehouses', default=4, type=int, metavar='W', 201 | help='Number of Warehouses') 202 | aparser.add_argument('--duration', default=60, type=int, metavar='D', 203 | help='How long to run the benchmark in seconds') 204 | aparser.add_argument('--ddl', 205 | default=os.path.realpath(os.path.join(os.path.dirname(__file__), "tpcc.sql")), 206 | help='Path to the TPC-C DDL SQL file') 207 | aparser.add_argument('--clients', default=1, type=int, metavar='N', 208 | help='The number of blocking clients to fork') 209 | aparser.add_argument('--stop-on-error', action='store_true', 210 | help='Stop the transaction execution when the driver throws an exception.') 211 | aparser.add_argument('--no-load', action='store_true', 212 | help='Disable loading the data') 213 | aparser.add_argument('--no-execute', action='store_true', 214 | help='Disable executing the workload') 215 | aparser.add_argument('--print-config', action='store_true', 216 | help='Print out the default configuration file for the system and exit') 217 | aparser.add_argument('--debug', action='store_true', 218 | help='Enable debug log messages') 219 | args = vars(aparser.parse_args()) 220 | 221 | if args['debug']: 222 | logging.getLogger().setLevel(logging.DEBUG) 223 | 224 | ## Create a handle to the target client driver 225 | driverClass = createDriverClass(args['system']) 226 | assert driverClass != None, "Failed to find '%s' class" % args['system'] 227 | driver = driverClass(args['ddl']) 228 | assert driver != None, "Failed to create '%s' driver" % args['system'] 229 | if args['print_config']: 230 | config = driver.makeDefaultConfig() 231 | print(driver.formatConfig(config)) 232 | print() 233 | sys.exit(0) 234 | 235 | ## Load Configuration file 236 | if args['config']: 237 | logging.debug("Loading configuration file '%s'", args['config']) 238 | cparser = ConfigParser() 239 | cparser.read(os.path.realpath(args['config'].name)) 240 | config = dict(cparser.items(args['system'])) 241 | else: 242 | logging.debug("Using default configuration for %s", args['system']) 243 | defaultConfig = driver.makeDefaultConfig() 244 | config = dict([(param, defaultConfig[param][1]) for param in defaultConfig.keys()]) 245 | config['reset'] = args['reset'] 246 | config['load'] = False 247 | config['execute'] = False 248 | if config['reset']: 249 | logging.info("Reseting database") 250 | config['warehouses'] = args['warehouses'] 251 | driver.loadConfig(config) 252 | logging.info("Initializing TPC-C benchmark using %s", driver) 253 | 254 | ## Create ScaleParameters 255 | scaleParameters = scaleparameters.makeWithScaleFactor(args['warehouses'], args['scalefactor']) 256 | if args['debug']: 257 | logging.debug("Scale Parameters:\n%s", scaleParameters) 258 | 259 | ## DATA LOADER!!! 260 | load_time = None 261 | if not args['no_load']: 262 | logging.info("Loading TPC-C benchmark data using %s", (driver)) 263 | load_start = time.time() 264 | if args['clients'] == 1: 265 | l = loader.Loader( 266 | driver, 267 | scaleParameters, 268 | range(scaleParameters.starting_warehouse, scaleParameters.ending_warehouse+1), 269 | True 270 | ) 271 | driver.loadStart() 272 | l.execute() 273 | driver.loadFinish() 274 | else: 275 | startLoading(driverClass, scaleParameters, args, config) 276 | load_time = time.time() - load_start 277 | ## IF 278 | 279 | ## WORKLOAD DRIVER!!! 280 | if not args['no_execute']: 281 | if args['clients'] == 1: 282 | e = executor.Executor(driver, scaleParameters, stop_on_error=args['stop_on_error']) 283 | driver.executeStart() 284 | results = e.execute(args['duration']) 285 | driver.executeFinish() 286 | else: 287 | results = startExecution(driverClass, scaleParameters, args, config) 288 | assert results, "No results from execution for %d client!" % args['clients'] 289 | logging.info("Final Results") 290 | logging.info("Threads: %d", args['clients']) 291 | logging.info(results.show(load_time, driver, args['clients'])) 292 | ## IF 293 | 294 | ## MAIN 295 | -------------------------------------------------------------------------------- /pytpcc/tpcc.sql: -------------------------------------------------------------------------------- 1 | CREATE TABLE WAREHOUSE ( 2 | W_ID SMALLINT DEFAULT '0' NOT NULL, 3 | W_NAME VARCHAR(16) DEFAULT NULL, 4 | W_STREET_1 VARCHAR(32) DEFAULT NULL, 5 | W_STREET_2 VARCHAR(32) DEFAULT NULL, 6 | W_CITY VARCHAR(32) DEFAULT NULL, 7 | W_STATE VARCHAR(2) DEFAULT NULL, 8 | W_ZIP VARCHAR(9) DEFAULT NULL, 9 | W_TAX FLOAT DEFAULT NULL, 10 | W_YTD FLOAT DEFAULT NULL, 11 | CONSTRAINT W_PK_ARRAY PRIMARY KEY (W_ID) 12 | ); 13 | 14 | CREATE TABLE DISTRICT ( 15 | D_ID TINYINT DEFAULT '0' NOT NULL, 16 | D_W_ID SMALLINT DEFAULT '0' NOT NULL REFERENCES WAREHOUSE (W_ID), 17 | D_NAME VARCHAR(16) DEFAULT NULL, 18 | D_STREET_1 VARCHAR(32) DEFAULT NULL, 19 | D_STREET_2 VARCHAR(32) DEFAULT NULL, 20 | D_CITY VARCHAR(32) DEFAULT NULL, 21 | D_STATE VARCHAR(2) DEFAULT NULL, 22 | D_ZIP VARCHAR(9) DEFAULT NULL, 23 | D_TAX FLOAT DEFAULT NULL, 24 | D_YTD FLOAT DEFAULT NULL, 25 | D_NEXT_O_ID INT DEFAULT NULL, 26 | PRIMARY KEY (D_W_ID,D_ID) 27 | ); 28 | 29 | CREATE TABLE ITEM ( 30 | I_ID INTEGER DEFAULT '0' NOT NULL, 31 | I_IM_ID INTEGER DEFAULT NULL, 32 | I_NAME VARCHAR(32) DEFAULT NULL, 33 | I_PRICE FLOAT DEFAULT NULL, 34 | I_DATA VARCHAR(64) DEFAULT NULL, 35 | CONSTRAINT I_PK_ARRAY PRIMARY KEY (I_ID) 36 | ); 37 | 38 | CREATE TABLE CUSTOMER ( 39 | C_ID INTEGER DEFAULT '0' NOT NULL, 40 | C_D_ID TINYINT DEFAULT '0' NOT NULL, 41 | C_W_ID SMALLINT DEFAULT '0' NOT NULL, 42 | C_FIRST VARCHAR(32) DEFAULT NULL, 43 | C_MIDDLE VARCHAR(2) DEFAULT NULL, 44 | C_LAST VARCHAR(32) DEFAULT NULL, 45 | C_STREET_1 VARCHAR(32) DEFAULT NULL, 46 | C_STREET_2 VARCHAR(32) DEFAULT NULL, 47 | C_CITY VARCHAR(32) DEFAULT NULL, 48 | C_STATE VARCHAR(2) DEFAULT NULL, 49 | C_ZIP VARCHAR(9) DEFAULT NULL, 50 | C_PHONE VARCHAR(32) DEFAULT NULL, 51 | C_SINCE TIMESTAMP DEFAULT CURRENT_TIMESTAMP NOT NULL, 52 | C_CREDIT VARCHAR(2) DEFAULT NULL, 53 | C_CREDIT_LIM FLOAT DEFAULT NULL, 54 | C_DISCOUNT FLOAT DEFAULT NULL, 55 | C_BALANCE FLOAT DEFAULT NULL, 56 | C_YTD_PAYMENT FLOAT DEFAULT NULL, 57 | C_PAYMENT_CNT INTEGER DEFAULT NULL, 58 | C_DELIVERY_CNT INTEGER DEFAULT NULL, 59 | C_DATA VARCHAR(500), 60 | PRIMARY KEY (C_W_ID,C_D_ID,C_ID), 61 | UNIQUE (C_W_ID,C_D_ID,C_LAST,C_FIRST), 62 | CONSTRAINT C_FKEY_D FOREIGN KEY (C_D_ID, C_W_ID) REFERENCES DISTRICT (D_ID, D_W_ID) 63 | ); 64 | CREATE INDEX IDX_CUSTOMER ON CUSTOMER (C_W_ID,C_D_ID,C_LAST); 65 | 66 | CREATE TABLE HISTORY ( 67 | H_C_ID INTEGER DEFAULT NULL, 68 | H_C_D_ID TINYINT DEFAULT NULL, 69 | H_C_W_ID SMALLINT DEFAULT NULL, 70 | H_D_ID TINYINT DEFAULT NULL, 71 | H_W_ID SMALLINT DEFAULT '0' NOT NULL, 72 | H_DATE TIMESTAMP DEFAULT CURRENT_TIMESTAMP NOT NULL, 73 | H_AMOUNT FLOAT DEFAULT NULL, 74 | H_DATA VARCHAR(32) DEFAULT NULL, 75 | CONSTRAINT H_FKEY_C FOREIGN KEY (H_C_ID, H_C_D_ID, H_C_W_ID) REFERENCES CUSTOMER (C_ID, C_D_ID, C_W_ID), 76 | CONSTRAINT H_FKEY_D FOREIGN KEY (H_D_ID, H_W_ID) REFERENCES DISTRICT (D_ID, D_W_ID) 77 | ); 78 | 79 | CREATE TABLE STOCK ( 80 | S_I_ID INTEGER DEFAULT '0' NOT NULL REFERENCES ITEM (I_ID), 81 | S_W_ID SMALLINT DEFAULT '0 ' NOT NULL REFERENCES WAREHOUSE (W_ID), 82 | S_QUANTITY INTEGER DEFAULT '0' NOT NULL, 83 | S_DIST_01 VARCHAR(32) DEFAULT NULL, 84 | S_DIST_02 VARCHAR(32) DEFAULT NULL, 85 | S_DIST_03 VARCHAR(32) DEFAULT NULL, 86 | S_DIST_04 VARCHAR(32) DEFAULT NULL, 87 | S_DIST_05 VARCHAR(32) DEFAULT NULL, 88 | S_DIST_06 VARCHAR(32) DEFAULT NULL, 89 | S_DIST_07 VARCHAR(32) DEFAULT NULL, 90 | S_DIST_08 VARCHAR(32) DEFAULT NULL, 91 | S_DIST_09 VARCHAR(32) DEFAULT NULL, 92 | S_DIST_10 VARCHAR(32) DEFAULT NULL, 93 | S_YTD INTEGER DEFAULT NULL, 94 | S_ORDER_CNT INTEGER DEFAULT NULL, 95 | S_REMOTE_CNT INTEGER DEFAULT NULL, 96 | S_DATA VARCHAR(64) DEFAULT NULL, 97 | PRIMARY KEY (S_W_ID,S_I_ID) 98 | ); 99 | 100 | CREATE TABLE ORDERS ( 101 | O_ID INTEGER DEFAULT '0' NOT NULL, 102 | O_C_ID INTEGER DEFAULT NULL, 103 | O_D_ID TINYINT DEFAULT '0' NOT NULL, 104 | O_W_ID SMALLINT DEFAULT '0' NOT NULL, 105 | O_ENTRY_D TIMESTAMP DEFAULT CURRENT_TIMESTAMP NOT NULL, 106 | O_CARRIER_ID INTEGER DEFAULT NULL, 107 | O_OL_CNT INTEGER DEFAULT NULL, 108 | O_ALL_LOCAL INTEGER DEFAULT NULL, 109 | PRIMARY KEY (O_W_ID,O_D_ID,O_ID), 110 | UNIQUE (O_W_ID,O_D_ID,O_C_ID,O_ID), 111 | CONSTRAINT O_FKEY_C FOREIGN KEY (O_C_ID, O_D_ID, O_W_ID) REFERENCES CUSTOMER (C_ID, C_D_ID, C_W_ID) 112 | ); 113 | CREATE INDEX IDX_ORDERS ON ORDERS (O_W_ID,O_D_ID,O_C_ID); 114 | 115 | CREATE TABLE NEW_ORDER ( 116 | NO_O_ID INTEGER DEFAULT '0' NOT NULL, 117 | NO_D_ID TINYINT DEFAULT '0' NOT NULL, 118 | NO_W_ID SMALLINT DEFAULT '0' NOT NULL, 119 | CONSTRAINT NO_PK_TREE PRIMARY KEY (NO_D_ID,NO_W_ID,NO_O_ID), 120 | CONSTRAINT NO_FKEY_O FOREIGN KEY (NO_O_ID, NO_D_ID, NO_W_ID) REFERENCES ORDERS (O_ID, O_D_ID, O_W_ID) 121 | ); 122 | 123 | CREATE TABLE ORDER_LINE ( 124 | OL_O_ID INTEGER DEFAULT '0' NOT NULL, 125 | OL_D_ID TINYINT DEFAULT '0' NOT NULL, 126 | OL_W_ID SMALLINT DEFAULT '0' NOT NULL, 127 | OL_NUMBER INTEGER DEFAULT '0' NOT NULL, 128 | OL_I_ID INTEGER DEFAULT NULL, 129 | OL_SUPPLY_W_ID SMALLINT DEFAULT NULL, 130 | OL_DELIVERY_D TIMESTAMP DEFAULT NULL, 131 | OL_QUANTITY INTEGER DEFAULT NULL, 132 | OL_AMOUNT FLOAT DEFAULT NULL, 133 | OL_DIST_INFO VARCHAR(32) DEFAULT NULL, 134 | PRIMARY KEY (OL_W_ID,OL_D_ID,OL_O_ID,OL_NUMBER), 135 | CONSTRAINT OL_FKEY_O FOREIGN KEY (OL_O_ID, OL_D_ID, OL_W_ID) REFERENCES ORDERS (O_ID, O_D_ID, O_W_ID), 136 | CONSTRAINT OL_FKEY_S FOREIGN KEY (OL_I_ID, OL_SUPPLY_W_ID) REFERENCES STOCK (S_I_ID, S_W_ID) 137 | ); 138 | --CREATE INDEX IDX_ORDER_LINE_3COL ON ORDER_LINE (OL_W_ID,OL_D_ID,OL_O_ID); 139 | --CREATE INDEX IDX_ORDER_LINE_2COL ON ORDER_LINE (OL_W_ID,OL_D_ID); 140 | CREATE INDEX IDX_ORDER_LINE_TREE ON ORDER_LINE (OL_W_ID,OL_D_ID,OL_O_ID); 141 | -------------------------------------------------------------------------------- /pytpcc/util/__init__.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | 3 | __all__ = ["scaleparameters", "rand", "nurand", "results"] -------------------------------------------------------------------------------- /pytpcc/util/nurand.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | # ----------------------------------------------------------------------- 3 | # Copyright (C) 2011 4 | # Andy Pavlo 5 | # http://www.cs.brown.edu/~pavlo/ 6 | # 7 | # Original Java Version: 8 | # Copyright (C) 2008 9 | # Evan Jones 10 | # Massachusetts Institute of Technology 11 | # 12 | # Permission is hereby granted, free of charge, to any person obtaining 13 | # a copy of this software and associated documentation files (the 14 | # "Software"), to deal in the Software without restriction, including 15 | # without limitation the rights to use, copy, modify, merge, publish, 16 | # distribute, sublicense, and/or sell copies of the Software, and to 17 | # permit persons to whom the Software is furnished to do so, subject to 18 | # the following conditions: 19 | # 20 | # The above copyright notice and this permission notice shall be 21 | # included in all copies or substantial portions of the Software. 22 | # 23 | # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, 24 | # EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF 25 | # MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT 26 | # IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR 27 | # OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, 28 | # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR 29 | # OTHER DEALINGS IN THE SOFTWARE. 30 | # ----------------------------------------------------------------------- 31 | 32 | import random 33 | 34 | def makeForLoad(): 35 | """Create random NURand constants, appropriate for loading the database.""" 36 | cLast = random.randint(0, 255) 37 | cId = random.randint(0, 1023) 38 | orderLineItemId = random.randint(0, 8191) 39 | return NURandC(cLast, cId, orderLineItemId) 40 | return NURandC(cLast, cId, orderLineItemId) 41 | 42 | def validCRun(cRun, cLoad): 43 | """Returns true if the cRun value is valid for running. See TPC-C 2.1.6.1 (page 20)""" 44 | cDelta = abs(cRun - cLoad) 45 | return 65 <= cDelta and cDelta <= 119 and cDelta != 96 and cDelta != 112 46 | 47 | def makeForRun(loadC): 48 | """Create random NURand constants for running TPC-C. TPC-C 2.1.6.1. (page 20) specifies the valid range for these constants.""" 49 | cRun = random.randint(0, 255) 50 | while validCRun(cRun, loadC.cLast) == False: 51 | cRun = random.randint(0, 255) 52 | assert validCRun(cRun, loadC.cLast) 53 | 54 | cId = random.randint(0, 1023) 55 | orderLineItemId = random.randint(0, 8191) 56 | return NURandC(cRun, cId, orderLineItemId) 57 | 58 | class NURandC: 59 | def __init__(self, cLast, cId, orderLineItemId): 60 | self.cLast = cLast 61 | self.cId = cId 62 | self.orderLineItemId = orderLineItemId 63 | -------------------------------------------------------------------------------- /pytpcc/util/rand.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | # ----------------------------------------------------------------------- 3 | # Copyright (C) 2011 4 | # Andy Pavlo 5 | # http://www.cs.brown.edu/~pavlo/ 6 | # 7 | # Original Java Version: 8 | # Copyright (C) 2008 9 | # Evan Jones 10 | # Massachusetts Institute of Technology 11 | # 12 | # Permission is hereby granted, free of charge, to any person obtaining 13 | # a copy of this software and associated documentation files (the 14 | # "Software"), to deal in the Software without restriction, including 15 | # without limitation the rights to use, copy, modify, merge, publish, 16 | # distribute, sublicense, and/or sell copies of the Software, and to 17 | # permit persons to whom the Software is furnished to do so, subject to 18 | # the following conditions: 19 | # 20 | # The above copyright notice and this permission notice shall be 21 | # included in all copies or substantial portions of the Software. 22 | # 23 | # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, 24 | # EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF 25 | # MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT 26 | # IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR 27 | # OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, 28 | # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR 29 | # OTHER DEALINGS IN THE SOFTWARE. 30 | # ----------------------------------------------------------------------- 31 | 32 | import random 33 | from . import nurand 34 | 35 | SYLLABLES = [ "BAR", "OUGHT", "ABLE", "PRI", "PRES", "ESE", "ANTI", "CALLY", "ATION", "EING" ] 36 | 37 | nurandVar = None # NURand 38 | def setNURand(nu): 39 | global nurandVar 40 | nurandVar = nu 41 | ## DEF 42 | 43 | def NURand(a, x, y): 44 | """A non-uniform random number, as defined by TPC-C 2.1.6. (page 20).""" 45 | global nurandVar 46 | assert x <= y 47 | if nurandVar is None: 48 | setNURand(nurand.makeForLoad()) 49 | 50 | if a == 255: 51 | c = nurandVar.cLast 52 | elif a == 1023: 53 | c = nurandVar.cId 54 | elif a == 8191: 55 | c = nurandVar.orderLineItemId 56 | else: 57 | raise Exception("a = " + a + " is not a supported value") 58 | 59 | return (((number(0, a) | number(x, y)) + c) % (y - x + 1)) + x 60 | ## DEF 61 | 62 | def number(minimum, maximum): 63 | value = random.randint(minimum, maximum) 64 | assert minimum <= value and value <= maximum 65 | return value 66 | ## DEF 67 | 68 | def numberExcluding(minimum, maximum, excluding): 69 | """An in the range [minimum, maximum], excluding excluding.""" 70 | assert minimum < maximum 71 | assert minimum <= excluding and excluding <= maximum 72 | 73 | ## Generate 1 less number than the range 74 | num = number(minimum, maximum-1) 75 | 76 | ## Adjust the numbers to remove excluding 77 | if num >= excluding: num += 1 78 | assert minimum <= num and num <= maximum and num != excluding 79 | return num 80 | ## DEF 81 | 82 | def fixedPoint(decimal_places, minimum, maximum): 83 | assert decimal_places > 0 84 | assert minimum < maximum 85 | 86 | multiplier = 1 87 | for i in range(0, decimal_places): 88 | multiplier *= 10 89 | 90 | int_min = int(minimum * multiplier + 0.5) 91 | int_max = int(maximum * multiplier + 0.5) 92 | 93 | return float(number(int_min, int_max) / float(multiplier)) 94 | ## DEF 95 | 96 | def selectUniqueIds(numUnique, minimum, maximum): 97 | rows = set() 98 | for i in range(0, numUnique): 99 | index = None 100 | while index == None or index in rows: 101 | index = number(minimum, maximum) 102 | ## WHILE 103 | rows.add(index) 104 | ## FOR 105 | assert len(rows) == numUnique 106 | return rows 107 | ## DEF 108 | 109 | def astring(minimum_length, maximum_length): 110 | """A random alphabetic string with length in range [minimum_length, maximum_length].""" 111 | return randomString(minimum_length, maximum_length, 'a', 26) 112 | ## DEF 113 | 114 | def nstring(minimum_length, maximum_length): 115 | """A random numeric string with length in range [minimum_length, maximum_length].""" 116 | return randomString(minimum_length, maximum_length, '0', 10) 117 | ## DEF 118 | 119 | def randomString(minimum_length, maximum_length, base, numCharacters): 120 | length = number(minimum_length, maximum_length) 121 | baseByte = ord(base) 122 | string = "" 123 | for i in range(length): 124 | string += chr(baseByte + number(0, numCharacters-1)) 125 | return string 126 | ## DEF 127 | 128 | def makeLastName(number): 129 | """A last name as defined by TPC-C 4.3.2.3. Not actually random.""" 130 | global SYLLABLES 131 | assert 0 <= number and number <= 999 132 | indicies = [ number//100, (number//10)%10, number%10 ] 133 | return "".join(map(lambda x: SYLLABLES[x], indicies)) 134 | ## DEF 135 | 136 | def makeRandomLastName(maxCID): 137 | """A non-uniform random last name, as defined by TPC-C 4.3.2.3. The name will be limited to maxCID.""" 138 | min_cid = 999 139 | if (maxCID - 1) < min_cid: min_cid = maxCID - 1 140 | return makeLastName(NURand(255, 0, min_cid)) 141 | ## DEF 142 | -------------------------------------------------------------------------------- /pytpcc/util/results.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | # ----------------------------------------------------------------------- 3 | # Copyright (C) 2011 4 | # Andy Pavlo 5 | # http://www.cs.brown.edu/~pavlo/ 6 | # 7 | # Permission is hereby granted, free of charge, to any person obtaining 8 | # a copy of this software and associated documentation files (the 9 | # "Software"), to deal in the Software without restriction, including 10 | # without limitation the rights to use, copy, modify, merge, publish, 11 | # distribute, sublicense, and/or sell copies of the Software, and to 12 | # permit persons to whom the Software is furnished to do so, subject to 13 | # the following conditions: 14 | # 15 | # The above copyright notice and this permission notice shall be 16 | # included in all copies or substantial portions of the Software. 17 | # 18 | # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, 19 | # EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF 20 | # MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT 21 | # IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR 22 | # OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, 23 | # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR 24 | # OTHER DEALINGS IN THE SOFTWARE. 25 | # ----------------------------------------------------------------------- 26 | 27 | import logging 28 | import time 29 | from collections import Counter 30 | 31 | class Results: 32 | 33 | def __init__(self): 34 | self.start = None 35 | self.stop = None 36 | self.txn_id = 0 37 | 38 | self.txn_counters = {} 39 | self.txn_times = {} 40 | self.txn_mins = {} 41 | self.txn_maxs = {} 42 | self.latencies = {} 43 | self.txn_retries = {} 44 | self.txn_aborts = {} 45 | self.retries = {} 46 | self.running = {} 47 | 48 | def startBenchmark(self): 49 | """Mark the benchmark as having been started""" 50 | assert self.start is None, "start is not none on start" 51 | logging.debug("Starting benchmark statistics collection") 52 | self.start = time.time() 53 | return self.start 54 | 55 | def stopBenchmark(self): 56 | """Mark the benchmark as having been stopped""" 57 | assert self.start != None, "start is none on stop" 58 | assert self.stop is None, "stop isn't none on stop" 59 | logging.debug("Stopping benchmark statistics collection") 60 | self.stop = time.time() 61 | 62 | def startTransaction(self, txn): 63 | self.txn_id += 1 64 | id = self.txn_id 65 | self.running[id] = (txn, time.time()) 66 | return id 67 | 68 | def abortTransaction(self, id, retries=0): 69 | """Abort a transaction and discard its times""" 70 | assert id in self.running, "Didn't find self in running" 71 | txn_name, txn_start = self.running[id] 72 | del self.running[id] 73 | total_retries = self.txn_retries.get(txn_name, 0) 74 | self.txn_retries[txn_name] = total_retries + retries 75 | total_aborts = self.txn_aborts.get(txn_name, 0) 76 | self.txn_aborts[txn_name] = total_aborts + 1 77 | if txn_name not in self.retries: 78 | self.retries[txn_name] = [] 79 | self.retries[txn_name].append(retries) 80 | # txn_start is currently unused, which means we don't include aborted txn in timing metrics 81 | 82 | def stopTransaction(self, id, retries=0): 83 | """Record that the benchmark completed an invocation of the given transaction""" 84 | assert id in self.running, "Didn't find self in running" 85 | txn_name, txn_start = self.running[id] 86 | del self.running[id] 87 | 88 | duration = time.time() - txn_start 89 | total_time = self.txn_times.get(txn_name, 0) 90 | self.txn_times[txn_name] = total_time + duration 91 | if txn_name not in self.latencies: 92 | self.latencies[txn_name] = [] 93 | self.latencies[txn_name].append(duration) 94 | 95 | total_aborts = self.txn_aborts.get(txn_name, 0) 96 | self.txn_aborts[txn_name] = total_aborts 97 | total_retries = self.txn_retries.get(txn_name, 0) 98 | self.txn_retries[txn_name] = total_retries + retries 99 | if txn_name not in self.retries: 100 | self.retries[txn_name] = [] 101 | self.retries[txn_name].append(retries) 102 | 103 | total_cnt = self.txn_counters.get(txn_name, 0) 104 | self.txn_counters[txn_name] = total_cnt + 1 105 | 106 | min_time = self.txn_mins.get(txn_name, 10000000) 107 | if duration < min_time: 108 | self.txn_mins[txn_name] = duration 109 | 110 | max_time = self.txn_maxs.get(txn_name, 0) 111 | if duration > max_time: 112 | self.txn_maxs[txn_name] = duration 113 | 114 | def append(self, r): 115 | for txn_name in r.txn_counters.keys(): 116 | orig_cnt = self.txn_counters.get(txn_name, 0) 117 | orig_min = self.txn_mins.get(txn_name, 10000000) 118 | orig_max = self.txn_maxs.get(txn_name, 0) 119 | orig_time = self.txn_times.get(txn_name, 0) 120 | orig_retries = self.txn_retries.get(txn_name, 0) 121 | orig_aborts = self.txn_aborts.get(txn_name, 0) 122 | 123 | self.txn_counters[txn_name] = orig_cnt + r.txn_counters[txn_name] 124 | self.txn_mins[txn_name] = orig_min if orig_min < r.txn_mins[txn_name] else r.txn_mins[txn_name] 125 | self.txn_maxs[txn_name] = orig_max if orig_max > r.txn_maxs[txn_name] else r.txn_maxs[txn_name] 126 | self.txn_times[txn_name] = orig_time + r.txn_times[txn_name] 127 | self.txn_retries[txn_name] = orig_retries + r.txn_retries[txn_name] 128 | self.txn_aborts[txn_name] = orig_aborts + r.txn_aborts[txn_name] 129 | print("%s [cnt=%d, time=%d]" % (txn_name, self.txn_counters[txn_name], self.txn_times[txn_name])) 130 | # logging.debug("%s [cnt=%d, time=%d]" % (txn_name, self.txn_counters[txn_name], self.txn_times[txn_name])) 131 | if txn_name not in self.latencies: 132 | self.latencies[txn_name] = [] 133 | self.latencies[txn_name].extend(r.latencies[txn_name]) 134 | if txn_name not in self.retries: 135 | self.retries[txn_name] = [] 136 | self.retries[txn_name].extend(r.retries[txn_name]) 137 | 138 | ## HACK 139 | self.start = r.start 140 | self.stop = r.stop 141 | 142 | def __str__(self): 143 | return self.show() 144 | 145 | def show(self, load_time=None, driver=None, threads=1): 146 | if not self.start: 147 | return "Benchmark not started" 148 | if not self.stop: 149 | duration = time.time() - self.start 150 | else: 151 | duration = self.stop - self.start 152 | 153 | col_width = 16 154 | num_columns = 13 155 | total_width = (col_width*num_columns)-8 156 | f = "\n " + (("%-" + str(col_width) + "s")*num_columns) 157 | line = "-"*total_width 158 | 159 | ret = u"\n" + "="*total_width + "\n" 160 | if load_time: 161 | ret += "Data Loading Time: %d seconds\n\n" % (load_time) 162 | 163 | ret += "Execution Results after %d seconds\n%s" % (duration, line) 164 | ret += f % ("", "Complete", u"Time (µs)", u"Percentage", u"Retries", u"minLatMs", u"p50", u"p75", u"p90", u"p95", u"p99", u"maxLatMs", u"Aborts") 165 | 166 | total_time = 0 167 | total_cnt = 0 168 | total_retries = 0 169 | total_aborts = 0 170 | for txn in self.txn_counters: 171 | txn_time = self.txn_times[txn] 172 | txn_cnt = self.txn_counters[txn] 173 | total_time += txn_time 174 | total_cnt += txn_cnt 175 | 176 | result_doc = {} 177 | for txn in sorted(self.txn_counters): 178 | txn_time = self.txn_times[txn] 179 | txn_cnt = self.txn_counters[txn] 180 | min_latency = u"%5.2f" % (1000 * self.txn_mins[txn]) 181 | max_latency = u"%6.2f" % (1000 * self.txn_maxs[txn]) 182 | samples = len(self.latencies[txn]) 183 | lat = sorted(self.latencies[txn]) 184 | ip50 = u"%6.2f" % (1000* lat[int(samples/2)]) 185 | ip75 = u"%6.2f" % (1000*lat[int(samples/100.0*75)]) 186 | ip90 = u"%6.2f" % (1000*lat[int(samples/100.0*90)]) 187 | ip95 = u"%6.2f" % (1000*lat[int(samples/100.0*95)]) 188 | ip99 = u"%6.2f" % (1000*lat[int(samples/100.0*99)]) 189 | txn_aborts = self.txn_aborts[txn] 190 | total_aborts += txn_aborts 191 | txn_retries = self.txn_retries[txn] 192 | total_retries += txn_retries 193 | perc_cnt = u"%5.02f" % ((100.0*txn_cnt / total_cnt)) 194 | perc_time = u"%5.02f" % ((100.0*txn_time / total_time)) 195 | just_retries = [x for x in self.retries[txn] if x>0] 196 | freq_dist = {str(k):v for k,v in dict(Counter(self.retries[txn])).items()} 197 | ret += f % (txn, str(txn_cnt), u"%9.3f" % (txn_time), perc_cnt, 198 | str(len(just_retries))+","+str(sum(just_retries)), 199 | min_latency, ip50, ip75, ip90, ip95, ip99, max_latency, txn_aborts) 200 | result_doc[txn] = {'latency':{'min':1000*self.txn_mins[txn], 'max':1000*self.txn_maxs[txn], 'p50':1000* lat[int(samples/2)], 201 | 'p75':1000*lat[int(samples/100.0*75)],'p90':1000*lat[int(samples/100.0*90)], 202 | 'p95':1000*lat[int(samples/100.0*95)],'p99':1000*lat[int(samples/100.0*99)]}, 203 | 'total':txn_cnt} 204 | if just_retries: 205 | result_doc[txn]['retries']={'retries_ops':freq_dist, 'retries_txn_total':total_retries, 'retries_total_ops':len(just_retries)} 206 | 207 | print(self.txn_counters) 208 | txn_new_order = self.txn_counters.get('NEW_ORDER', 0) 209 | ret += "\n" + line # ("-"*total_width) 210 | #total_rate = "%.02f txn/s" % ((total_cnt / total_time)) 211 | lat = sorted(self.latencies.get('NEW_ORDER',[0])) 212 | samples = len(lat) 213 | ret += f % ("TOTAL", str(total_cnt), u"%12.3f" % total_time, "", "", "", "", "", "", "", "", "", "") 214 | if driver != None: 215 | # print(driver) 216 | result_doc['tpmc'] = txn_new_order*60/duration 217 | result_doc['denorm'] = driver.denormalize 218 | result_doc['duration'] = duration 219 | result_doc['warehouses'] = driver.warehouses 220 | result_doc['date'] = time.strftime("%Y-%m-%d %H:%M:%S") 221 | result_doc['threads'] = threads 222 | result_doc['txn'] = not driver.no_transactions 223 | result_doc['batch_writes'] = driver.batch_writes 224 | result_doc['find_and_modify'] = driver.find_and_modify 225 | result_doc['read_preference'] = driver.read_preference 226 | result_doc['write_concern'] = driver.write_concern.document['w'] 227 | result_doc['causal'] = driver.causal_consistency 228 | result_doc['all_in_one_txn'] = driver.all_in_one_txn 229 | result_doc['retry_writes'] = driver.retry_writes 230 | result_doc['read_concern'] = driver.read_concern 231 | result_doc['total_retries'] = total_retries 232 | result_doc['total'] = total_cnt 233 | result_doc['aborts'] = total_aborts 234 | ret += "\n%s TpmC for %s %s thr %s txn %d WH: %d %d total %d durSec, batch %s %d retries %s%% %s fnM %s p50 %s p75 %s p90 %s p95 %s p99 %s max %s WC %s causal %s 10in1 %s retry %s %d %d" % ( 235 | time.strftime("%Y-%m-%d %H:%M:%S"), 236 | ("normal", "denorm")[driver.denormalize], 237 | threads, 238 | ("with", "w/o ")[driver.no_transactions], 239 | driver.warehouses, 240 | round(txn_new_order*60/duration), txn_new_order, duration, 241 | ("off", "on")[driver.batch_writes], total_retries, str(100.0*total_retries/total_cnt)[:5], 242 | ("w/o ", "with")[driver.find_and_modify], 243 | driver.read_preference, 244 | u"%6.2f" % (1000* lat[int(samples/2)]), u"%6.2f" % (1000*lat[int(samples/100.0*75)]), 245 | u"%6.2f" % (1000*lat[int(samples/100.0*90)]), u"%6.2f" % (1000*lat[int(samples/100.0*95)]), 246 | u"%6.2f" % (1000*lat[int(samples/100.0*99)]), 247 | u"%6.2f" % (1000.0*lat[-1]), 248 | str(driver.write_concern), ('false', 'true')[driver.causal_consistency], 249 | ('false', 'true')[driver.all_in_one_txn], ('false', 'true')[driver.retry_writes],total_cnt,total_aborts) 250 | if driver: 251 | driver.save_result(result_doc) 252 | print(result_doc) 253 | return ret 254 | ## CLASS 255 | -------------------------------------------------------------------------------- /pytpcc/util/scaleparameters.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | 3 | # ----------------------------------------------------------------------- 4 | # Copyright (C) 2011 5 | # Andy Pavlo 6 | # http://www.cs.brown.edu/~pavlo/ 7 | # 8 | # Original Java Version: 9 | # Copyright (C) 2008 10 | # Evan Jones 11 | # Massachusetts Institute of Technology 12 | # 13 | # Permission is hereby granted, free of charge, to any person obtaining 14 | # a copy of this software and associated documentation files (the 15 | # "Software"), to deal in the Software without restriction, including 16 | # without limitation the rights to use, copy, modify, merge, publish, 17 | # distribute, sublicense, and/or sell copies of the Software, and to 18 | # permit persons to whom the Software is furnished to do so, subject to 19 | # the following conditions: 20 | # 21 | # The above copyright notice and this permission notice shall be 22 | # included in all copies or substantial portions of the Software. 23 | # 24 | # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, 25 | # EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF 26 | # MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT 27 | # IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR 28 | # OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, 29 | # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR 30 | # OTHER DEALINGS IN THE SOFTWARE. 31 | # ----------------------------------------------------------------------- 32 | 33 | import constants 34 | 35 | def makeDefault(warehouses): 36 | return ScaleParameters(constants.NUM_ITEMS, \ 37 | warehouses, \ 38 | constants.DISTRICTS_PER_WAREHOUSE, \ 39 | constants.CUSTOMERS_PER_DISTRICT, \ 40 | constants.INITIAL_NEW_ORDERS_PER_DISTRICT) 41 | ## DEF 42 | 43 | def makeWithScaleFactor(warehouses, scaleFactor): 44 | assert scaleFactor >= 1.0 45 | 46 | items = int(constants.NUM_ITEMS/scaleFactor) 47 | if items <= 0: items = 1 48 | districts = int(max(constants.DISTRICTS_PER_WAREHOUSE, 1)) 49 | customers = int(max(constants.CUSTOMERS_PER_DISTRICT/scaleFactor, 1)) 50 | newOrders = int(max(constants.INITIAL_NEW_ORDERS_PER_DISTRICT/scaleFactor, 0)) 51 | 52 | return ScaleParameters(items, warehouses, districts, customers, newOrders) 53 | ## DEF 54 | 55 | class ScaleParameters: 56 | 57 | def __init__(self, items, warehouses, districtsPerWarehouse, customersPerDistrict, newOrdersPerDistrict): 58 | assert 1 <= items and items <= constants.NUM_ITEMS 59 | self.items = items 60 | assert warehouses > 0 61 | self.warehouses = warehouses 62 | self.starting_warehouse = 1 63 | assert 1 <= districtsPerWarehouse and districtsPerWarehouse <= constants.DISTRICTS_PER_WAREHOUSE 64 | self.districtsPerWarehouse = districtsPerWarehouse 65 | assert 1 <= customersPerDistrict and customersPerDistrict <= constants.CUSTOMERS_PER_DISTRICT 66 | self.customersPerDistrict = customersPerDistrict 67 | assert 0 <= newOrdersPerDistrict and newOrdersPerDistrict <= constants.CUSTOMERS_PER_DISTRICT 68 | assert newOrdersPerDistrict <= constants.INITIAL_NEW_ORDERS_PER_DISTRICT 69 | self.newOrdersPerDistrict = newOrdersPerDistrict 70 | self.ending_warehouse = (self.warehouses + self.starting_warehouse - 1) 71 | ## DEF 72 | 73 | def __str__(self): 74 | out = "%d items\n" % self.items 75 | out += "%d warehouses\n" % self.warehouses 76 | out += "%d districts/warehouse\n" % self.districtsPerWarehouse 77 | out += "%d customers/district\n" % self.customersPerDistrict 78 | out += "%d initial new orders/district" % self.newOrdersPerDistrict 79 | return out 80 | ## DEF 81 | 82 | ## CLASS 83 | -------------------------------------------------------------------------------- /pytpcc/verify.js: -------------------------------------------------------------------------------- 1 | function checkConsistency(rule, collection, agg, sample=false) { 2 | agg.push({$count:"c"}); 3 | if (sample) { 4 | var c=db.WAREHOUSE.count(); 5 | var start=Math.round(Math.random()*c); 6 | var firstStage={$match:{}}; 7 | var wid="W_ID"; 8 | var did="D_ID"; 9 | if (collection=="NEW_ORDER") { 10 | wid="NO_"+wid; 11 | did="NO_"+did; 12 | } else if (collection=="WAREHOUSE") { 13 | did=collection[0]+"_"+did; 14 | } else if (collection=="DISTRICT") { 15 | wid=collection[0]+"_"+wid; 16 | } else { 17 | wid=collection[0]+"_"+wid; 18 | did=collection[0]+"_"+did; 19 | } 20 | firstStage["$match"][wid]=start; 21 | /* firstStage["$match"][did]=(start%10)+1; */ 22 | agg.unshift(firstStage); 23 | } 24 | var r = db.getCollection(collection).aggregate(agg, {maxTimeMS:60000}).toArray(); 25 | if (r.length != 0) { 26 | print("Consistency rule " + rule + " failed with count " + r[0]["c"]); 27 | } 28 | } 29 | 30 | c=[]; 31 | c[1]={r:"1", c:"WAREHOUSE", pipeline: [ 32 | {$lookup:{ 33 | from:"DISTRICT", 34 | as:"d", 35 | localField:"W_ID", 36 | foreignField:"D_W_ID"}}, 37 | {$match:{$expr:{$ne:[0,{$toLong:{$subtract:["$W_YTD",{$sum:"$d.D_YTD"}]}}]}}}, 38 | ]}; 39 | // consistency check 2 40 | c[2]={r:"2", c:"DISTRICT", pipeline: [ 41 | {$project:{w:"$D_W_ID", d:"$D_ID",next:"$D_NEXT_O_ID"}}, 42 | {$lookup:{from:"ORDERS",as:"maxOID",let:{w:"$w",d:"$d"},pipeline:[ 43 | {$match:{$expr:{$and:[{$eq:["$$w","$O_W_ID"]},{$eq:["$$d","$O_D_ID"]}]}}}, 44 | {$sort:{"O_ID":-1}}, 45 | {$limit:1}, 46 | {$group:{_id:0,maxO:{$first:"$O_ID"}}}]}}, 47 | {$unwind:"$maxOID"}, 48 | {$lookup:{from:"NEW_ORDER",as:"maxNOID",let:{w:"$w",d:"$d"},pipeline:[ 49 | {$match:{$expr:{$and:[{$eq:["$$w","$NO_W_ID"]},{$eq:["$$d","$NO_D_ID"]}]}}}, 50 | {$sort:{"NO_O_ID":-1}}, 51 | {$limit:1}, 52 | {$group:{_id:0,maxO:{$max:"$NO_O_ID"}}}]}}, 53 | {$unwind:"$maxNOID"}, 54 | {$match:{$or:[{$expr:{$ne:["$maxOID.maxO","$maxNOID.maxO"]}}, {$expr:{$ne:[ "$maxOID.maxO",{$subtract:["$next",1]}]}} ]}}, 55 | ]}; 56 | // consistency check 3 57 | c[3]={r:"3", c:"NEW_ORDER", pipeline:[ 58 | {$group:{_id:{w:"$NO_W_ID",d:"$NO_D_ID"},min:{$min:"$NO_O_ID"},max:{$max:"$NO_O_ID"},count:{$sum:1}}}, 59 | {$project:{count:1, diff:{$add:[1,{$subtract:["$max","$min"]}]}}}, 60 | {$match:{$expr:{$ne:["$count","$diff"]}}}, 61 | ]}; 62 | // consistency check 4 63 | c[4]={r:"4", c:"ORDERS", pipeline:[ 64 | {$sort:{O_W_ID:1, O_D_ID:1}}, 65 | {$limit:10000}, 66 | {$group:{_id:{w:"$O_W_ID",d:"$O_D_ID"},O_CLs:{$sum:"$O_OL_CNT"}, OL_CLs:{$sum:{$size:"$ORDER_LINE"}}}}, 67 | {$match:{$expr:{$ne:[ "$O_CLs","$OL_CLs"]}}}, 68 | ]}; 69 | // consistency check 5 70 | c[5]={r:"5", c:"ORDERS", pipeline:[ 71 | {$match:{O_CARRIER_ID:0}}, 72 | {$sort:{O_ID:-1}}, 73 | {$limit:10000}, 74 | {$lookup:{from:"NEW_ORDER", as:"NO_count",let:{w:"$O_W_ID",d:"$O_D_ID",o:"$O_ID"}, pipeline:[ 75 | {$match:{$expr:{$and:[{$eq:["$$w","$NO_W_ID"]},{$eq:["$$d","$NO_D_ID"]},{$eq:["$$o","$NO_O_ID"]} ]}}}, 76 | {$count:"c"}]}}, 77 | {$addFields:{count:{$ifNull:[{$arrayElemAt:["$NO_count.c",0]},0]}}}, 78 | {$match:{"count":{$ne:1}}}, 79 | ]}; 80 | // consistency check 6 81 | c[6]={r:"6", c:"ORDERS", pipeline:[ 82 | {$limit:10000}, 83 | {$match:{$expr:{$ne:[ "$O_OL_CNT",{$size:"$ORDER_LINE"}]}}}, 84 | ]}; 85 | // consistency check 7 86 | c[7]={r:"7", c:"ORDERS", pipeline:[ 87 | {$limit:10000}, 88 | {$match:{O_CARRIER_ID:{$ne:0}, "ORDER_LINE.OL_DELIVERY_D":null}}, 89 | ]}; 90 | // consistency 91 | c[8]={r:"8", c:"HISTORY", pipeline:[ 92 | {$group:{_id:"$H_W_ID",sum:{$sum:{$toDecimal:"$H_AMOUNT"}}}}, 93 | {$lookup:{from:"WAREHOUSE", localField:"_id", foreignField:"W_ID",as:"w"}}, 94 | {$match:{$expr:{$ne:[0, {$toLong:{$subtract:["$sum", {$arrayElemAt:["$w.W_YTD",0]}]}}]}}}, 95 | ]}; 96 | // consistency check 9 97 | c[9]={r:"9", c:"HISTORY", pipeline:[ 98 | {$group:{_id:{w:"$H_W_ID", d:"$H_D_ID"}, sum:{$sum:{$toDecimal:"$H_AMOUNT"}}}}, 99 | {$lookup:{from:"DISTRICT", as:"d", let:{ w: "$_id.w", d:"$_id.d"}, pipeline:[ 100 | {$match: {$expr: {$and: [ {$eq: ["$$w","$D_W_ID"]},{$eq:["$$d","$D_ID" ]}]}}}, 101 | {$group:{_id:0, sum:{$sum:{$toDecimal:"$D_YTD"}}}}]}}, 102 | {$match:{$expr:{$ne:[{$toLong:"$sum"},{$toLong:{$arrayElemAt:["$d.sum",0]}}]}}}, 103 | ]}; 104 | // *** consistency check 10 don't run unless there is an index 105 | /* adding one warehouse filter to limit checking, needed to add index to HISTORY.H_W_ID,etc to make reasonably fast even on one */ 106 | c[10]={r:"10", c:"CUSTOMER", pipeline:[ 107 | {$match:{C_W_ID:1,C_D_ID:1,C_ID:{$lt:100,$gt:80}}}, 108 | {$lookup:{from:"ORDERS", as:"o", let:{ w: "$C_W_ID", d:"$C_D_ID", c:"$C_ID"}, pipeline:[ 109 | {$match: {O_CARRIER_ID:{$ne:0}, $expr: {$and: [ {$eq: ["$$w","$O_W_ID"]},{$eq:["$$d","$O_D_ID"]}, {$eq:["$$c","$O_C_ID"]}]}}}, 110 | {$group:{_id:0, sum:{$sum:{$sum:"$ORDER_LINE.OL_AMOUNT"}}}}]}}, 111 | {$lookup:{from:"HISTORY", as:"h", let:{ w: "$C_W_ID", d:"$C_D_ID", c:"$C_ID"}, pipeline:[ 112 | {$match: {$expr: {$and: [ {$eq: ["$$w","$H_W_ID"]},{$eq:["$$d","$H_D_ID"]}, {$eq:["$$c","$H_C_ID"]}]}}}, 113 | {$group:{_id:0, sum:{$sum:"$H_AMOUNT"}}}]}}, 114 | {$project:{C_BALANCE:1, OSUM:{$ifNull:[{$arrayElemAt:["$o.sum",0]},0]},HSUM:{$arrayElemAt:["$h.sum",0]},_id:0, C_ID:1, C_W_ID:1, C_D_ID:1}}, 115 | {$match:{$expr:{$ne:["$C_BALANCE", {$subtract:["$OSUM","$HSUM"]}]}}}, 116 | ]}; 117 | // *** consistency check 11 Correct when first loaded! 118 | c[11]={r:"11", c:"DISTRICT", pipeline:[ 119 | {$project:{w:"$D_W_ID", d:"$D_ID"}}, 120 | {$lookup:{from:"ORDERS",as:"o",let:{w:"$w",d:"$d"},pipeline:[ 121 | {$match:{$expr:{$and:[{$eq:["$$w","$O_W_ID"]},{$eq:["$$d","$O_D_ID"]}]}}}, 122 | {$count:"c"}]}}, 123 | {$unwind:"$o"}, 124 | {$lookup:{from:"NEW_ORDER",as:"no",let:{w:"$w",d:"$d"},pipeline:[ 125 | {$match:{$expr:{$and:[{$eq:["$$w","$NO_W_ID"]},{$eq:["$$d","$NO_D_ID"]}]}}}, 126 | {$count:"c"}]}}, 127 | {$unwind:"$no"}, 128 | {$match:{$expr:{$ne:[2100, {$subtract:["$o.c","$no.c"]}]}}}, 129 | ]}; 130 | // consistency check 12 131 | c[12]={r:"12", c:"CUSTOMER", pipeline:[ 132 | {$lookup:{from:"ORDERS", as:"o", let:{ w: "$C_W_ID", d:"$C_D_ID", c:"$C_ID"}, pipeline:[ 133 | {$match: {O_CARRIER_ID:{$ne:0},$expr: {$and: [ {$eq: ["$$w","$O_W_ID"]},{$eq:["$$d","$O_D_ID"]}, {$eq:["$$c","$O_C_ID"]}]}}}, 134 | {$group:{_id:0, sum:{$sum:{$sum:"$ORDER_LINE.OL_AMOUNT"}}}}]}}, 135 | {$project:{C_BALANCE:1, C_YTD_PAYMENT:1, OLSUM:{$ifNull:[{$arrayElemAt:["$o.sum",0]},0]}}}, 136 | {$match:{$expr:{$ne:[0,{$toLong:{$subtract:["$OLSUM", {$add:["$C_BALANCE","$C_YTD_PAYMENT"]}]}}]}}}, 137 | ]}; 138 | 139 | for (i=1; i<13; i++) { 140 | print (""+ new ISODate() + " Checking " + i); 141 | if (i==10) continue; 142 | checkConsistency(c[i].r, c[i].c, c[i].pipeline,true); 143 | } 144 | 145 | -------------------------------------------------------------------------------- /pytpcc/worker.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # -*- coding: utf-8 -*- 3 | # ----------------------------------------------------------------------- 4 | # Copyright (C) 2011 5 | # Andy Pavlo & Yang Lu 6 | # http:##www.cs.brown.edu/~pavlo/ 7 | # 8 | # Permission is hereby granted, free of charge, to any person obtaining 9 | # a copy of this software and associated documentation files (the 10 | # "Software"), to deal in the Software without restriction, including 11 | # without limitation the rights to use, copy, modify, merge, publish, 12 | # distribute, sublicense, and/or sell copies of the Software, and to 13 | # permit persons to whom the Software is furnished to do so, subject to 14 | # the following conditions: 15 | # 16 | # The above copyright notice and this permission notice shall be 17 | # included in all copies or substantial portions of the Software. 18 | # 19 | # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, 20 | # EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF 21 | # MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT 22 | # IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR 23 | # OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, 24 | # ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR 25 | # OTHER DEALINGS IN THE SOFTWARE. 26 | # ----------------------------------------------------------------------- 27 | 28 | import sys 29 | import os 30 | import string 31 | import datetime 32 | import logging 33 | import re 34 | import argparse 35 | import glob 36 | import time 37 | import message 38 | import pickle 39 | import traceback 40 | from pprint import pprint,pformat 41 | 42 | from util import * 43 | from runtime import * 44 | import drivers 45 | 46 | ## ============================================== 47 | ## createDriverClass 48 | ## ============================================== 49 | def createDriverClass(name): 50 | full_name = "%sDriver" % name.title() 51 | mod = __import__('drivers.%s' % full_name.lower(), globals(), locals(), [full_name]) 52 | klass = getattr(mod, full_name) 53 | return klass 54 | ## DEF 55 | 56 | 57 | ## ============================================== 58 | ## loaderFunc 59 | ## ============================================== 60 | def loaderFunc(driverClass, scaleParameters, args, config, w_ids, debug): 61 | driver = driverClass(args['ddl']) 62 | assert driver != None 63 | logging.debug("Starting client execution: %s [warehouses=%d]" % (driver, len(w_ids))) 64 | 65 | config['load'] = True 66 | config['execute'] = False 67 | config['reset'] = False 68 | driver.loadConfig(config) 69 | 70 | try: 71 | loadItems = (1 in w_ids) 72 | l = loader.Loader(driver, scaleParameters, w_ids, loadItems) 73 | driver.loadStart() 74 | l.execute() 75 | driver.loadFinish() 76 | except KeyboardInterrupt: 77 | return -1 78 | except (Exception, AssertionError), ex: 79 | logging.warn("Failed to load data: %s" % (ex)) 80 | #if debug: 81 | traceback.print_exc(file=sys.stdout) 82 | raise 83 | 84 | ## DEF 85 | 86 | 87 | ## ============================================== 88 | ## executorFunc 89 | ## ============================================== 90 | def executorFunc(driverClass, scaleParameters, args, config, debug): 91 | driver = driverClass(args['ddl']) 92 | assert driver != None 93 | logging.debug("Starting client execution: %s" % driver) 94 | 95 | config['execute'] = True 96 | config['reset'] = False 97 | driver.loadConfig(config) 98 | 99 | e = executor.Executor(driver, scaleParameters, stop_on_error=args['stop_on_error']) 100 | driver.executeStart() 101 | results = e.execute(args['duration']) 102 | driver.executeFinish() 103 | 104 | return results 105 | ## DEF 106 | 107 | ## MAIN 108 | if __name__=='__channelexec__': 109 | driverClass=None 110 | for item in channel: 111 | command=pickle.loads(item) 112 | if command.header==message.CMD_LOAD: 113 | scaleParameters=command.data[0] 114 | args=command.data[1] 115 | config=command.data[2] 116 | w_ids=command.data[3] 117 | 118 | ## Create a handle to the target client driver at the client side 119 | driverClass = createDriverClass(args['system']) 120 | assert driverClass != None, "Failed to find '%s' class" % args['system'] 121 | driver = driverClass(args['ddl']) 122 | assert driver != None, "Failed to create '%s' driver" % args['system'] 123 | 124 | loaderFunc(driverClass,scaleParameters,args,config,w_ids,True) 125 | m=message.Message(header=message.LOAD_COMPLETED) 126 | channel.send(pickle.dumps(m,-1)) 127 | elif command.header==message.CMD_EXECUTE: 128 | scaleParameters=command.data[0] 129 | args=command.data[1] 130 | config=command.data[2] 131 | 132 | ## Create a handle to the target client driver at the client side 133 | if driverClass==None: 134 | driverClass = createDriverClass(args['system']) 135 | assert driverClass != None, "Failed to find '%s' class" % args['system'] 136 | driver = driverClass(args['ddl']) 137 | assert driver != None, "Failed to create '%s' driver" % args['system'] 138 | 139 | results=executorFunc(driverClass,scaleParameters,args,config,True) 140 | m=message.Message(header=message.EXECUTE_COMPLETED,data=results) 141 | channel.send(pickle.dumps(m,-1)) 142 | 143 | elif command.header==message.CMD_STOP: 144 | pass 145 | else: 146 | pass 147 | 148 | -------------------------------------------------------------------------------- /setup.cfg: -------------------------------------------------------------------------------- 1 | [egg_info] 2 | tag_build = dev 3 | tag_svn_revision = true 4 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | from setuptools import setup, find_packages 2 | import sys, os 3 | 4 | version = '0.0' 5 | 6 | setup(name='py-tpcc', 7 | version=version, 8 | description="Python implementation of the TPC-C benchmark", 9 | long_description="""\ 10 | """, 11 | classifiers=[], # Get strings from http://pypi.python.org/pypi?%3Aaction=list_classifiers 12 | keywords='', 13 | author='Andy Pavlo', 14 | author_email='pavlo@cs.brown.edu', 15 | url='http://www.cs.brown.edu/~pavlo/', 16 | license='BSD', 17 | packages=find_packages(exclude=['ez_setup', 'examples', 'tests']), 18 | include_package_data=True, 19 | zip_safe=False, 20 | install_requires=[ 21 | # -*- Extra requirements: -*- 22 | ], 23 | entry_points=""" 24 | # -*- Entry points: -*- 25 | """, 26 | ) 27 | -------------------------------------------------------------------------------- /vldb2019/paper.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mongodb-labs/py-tpcc/d6d48891547b0e6deb4cfa1ed1321c5765ede520/vldb2019/paper.pdf -------------------------------------------------------------------------------- /vldb2019/poster.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mongodb-labs/py-tpcc/d6d48891547b0e6deb4cfa1ed1321c5765ede520/vldb2019/poster.pdf -------------------------------------------------------------------------------- /vldb2019/posterHiRes.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mongodb-labs/py-tpcc/d6d48891547b0e6deb4cfa1ed1321c5765ede520/vldb2019/posterHiRes.pdf -------------------------------------------------------------------------------- /vldb2019/results/dbSizes.csv: -------------------------------------------------------------------------------- 1 | _id,collections,objects,avgObjSize,dataSize,storageSize,indexes,indexSize,fsUsedSize,fsTotalSize,initial,schema,scale 2 | tpcc_denorm100,8,2.0015349e+07,667.3280676744632,12.439493250101805,11.133750915527344,18,0.7842521667480469,293.30438232421875,1161.84375,true,denorm,100 3 | tpcc_denorm300,8,5.9812523e+07,668.9843211429152,37.26560631301254,28.309009552001953,18,2.2015228271484375,293.30438232421875,1161.84375,true,denorm,300 4 | tpcc_denorm50,8,1.0057136e+07,664.7093958956108,6.225959206931293,5.279247283935547,18,0.386474609375,293.30438232421875,1161.84375,true,denorm,50 5 | tpcc_norm100,9,5.0032327e+07,306.51489873737034,14.282440435141325,8.775493621826172,21,2.2393455505371094,293.30438232421875,1161.84375,true,normal,100 6 | tpcc_norm300,9,1.71357778e+08,293.5935497074431,46.85440874937922,38.29002380371094,21,8.207008361816406,453.9142837524414,1161.84375,true,normal,300 7 | tpcc_norm50,9,2.5077701e+07,306.14241879668316,7.150087544694543,4.682960510253906,21,1.2502708435058594,293.30438232421875,1161.84375,true,normal,50 8 | -------------------------------------------------------------------------------- /vldb2019/results/dbSizes.json: -------------------------------------------------------------------------------- 1 | {"_id":"tpcc_denorm100","collections":8.0,"objects":2.0015349e+07,"avgObjSize":667.3280676744632,"dataSize":12.439493250101805,"storageSize":11.133750915527344,"indexes":18.0,"indexSize":0.7842521667480469,"fsUsedSize":293.30438232421875,"fsTotalSize":1161.84375,"initial":true,"schema":"denorm","scale":100.0} 2 | {"_id":"tpcc_denorm300","collections":8.0,"objects":5.9812523e+07,"avgObjSize":668.9843211429152,"dataSize":37.26560631301254,"storageSize":28.309009552001953,"indexes":18.0,"indexSize":2.2015228271484375,"fsUsedSize":293.30438232421875,"fsTotalSize":1161.84375,"initial":true,"schema":"denorm","scale":300.0} 3 | {"_id":"tpcc_denorm50","collections":8.0,"objects":1.0057136e+07,"avgObjSize":664.7093958956108,"dataSize":6.225959206931293,"storageSize":5.279247283935547,"indexes":18.0,"indexSize":0.386474609375,"fsUsedSize":293.30438232421875,"fsTotalSize":1161.84375,"initial":true,"schema":"denorm","scale":50.0} 4 | {"_id":"tpcc_norm100","collections":9.0,"objects":5.0032327e+07,"avgObjSize":306.51489873737034,"dataSize":14.282440435141325,"storageSize":8.775493621826172,"indexes":21.0,"indexSize":2.2393455505371094,"fsUsedSize":293.30438232421875,"fsTotalSize":1161.84375,"initial":true,"schema":"normal","scale":100.0} 5 | {"_id":"tpcc_norm300","collections":9.0,"objects":1.71357778e+08,"avgObjSize":293.5935497074431,"dataSize":46.85440874937922,"storageSize":38.29002380371094,"indexes":21.0,"indexSize":8.207008361816406,"fsUsedSize":453.9142837524414,"fsTotalSize":1161.84375,"initial":true,"scale":300.0,"schema":"normal"} 6 | {"_id":"tpcc_norm50","collections":9.0,"objects":2.5077701e+07,"avgObjSize":306.14241879668316,"dataSize":7.150087544694543,"storageSize":4.682960510253906,"indexes":21.0,"indexSize":1.2502708435058594,"fsUsedSize":293.30438232421875,"fsTotalSize":1161.84375,"initial":true,"schema":"normal","scale":50.0} 7 | --------------------------------------------------------------------------------