├── .gitignore ├── DEVGUIDE.md ├── README.md ├── buildah ├── README.md ├── buildfio.sh ├── buildfioservice.sh ├── startfio └── startup.sh ├── data └── fio │ └── jobs │ ├── randr.job │ ├── randrlimited.job │ ├── randrw7030.job │ └── seqwrite.job ├── fetchlatency.py ├── fiocli ├── fiodeploy ├── fioservice ├── fiotools ├── __init__.py ├── configuration.py ├── handlers │ ├── __init__.py │ ├── base.py │ ├── debug.py │ ├── kubernetes.py │ ├── local.py │ └── ssh.py ├── reports │ ├── __init__.py │ └── latency.py ├── server │ ├── __init__.py │ ├── db.py │ ├── security.py │ └── web.py └── utils │ ├── __init__.py │ └── utils.py ├── media ├── fioloadgen-demo.gif ├── fioloadgen_1.2_2.gif ├── fioloadgen_banner.png ├── fioloadgen_jobs.png └── fioloadgen_profiles.png ├── react └── app │ ├── .env │ ├── .gitignore │ ├── package-lock.json │ ├── package.json │ ├── src │ ├── app.scss │ ├── common │ │ ├── kebab.jsx │ │ ├── modal.jsx │ │ ├── radioset.jsx │ │ ├── ratioslider.jsx │ │ └── tooltip.jsx │ ├── components │ │ ├── jobs.jsx │ │ ├── main.jsx │ │ ├── masthead.jsx │ │ └── profiles.jsx │ ├── index.jsx │ └── utils │ │ └── utils.js │ └── webpack.config.js ├── requirements.txt ├── setup.py ├── tox.ini ├── www ├── bundle.js ├── css │ ├── patternfly.min.css │ └── style.css ├── fioloadgen.ico └── index.html └── yaml ├── fio.yaml ├── fio_no_pvc.yaml ├── fiomgr.yaml ├── fioservice.yaml ├── fioservice_template.yaml ├── fioworker_statefulset.yaml └── fioworker_statefulset_template.yaml /.gitignore: -------------------------------------------------------------------------------- 1 | ./react/fioweb/node_modules 2 | ./react/fioweb/build 3 | ./react/app/node_modules/ 4 | react/app/dist 5 | *.py[oc] 6 | ./testing/ 7 | ./fixes/ 8 | -------------------------------------------------------------------------------- /DEVGUIDE.md: -------------------------------------------------------------------------------- 1 | 2 | ## DEV Notes 3 | 4 | Getting your environment ready...TODO 5 | 6 | 7 | ### Testing the UI 8 | To test the front end, first ensure your webservice is running (this will sit on port 8080), then run the code under the dev server (normally on 8081) 9 | 10 | ``` 11 | ./fioservice.py --mode=debug start 12 | ``` 13 | This will run the service in the foreground running on port 8080, so you can follow any debug messages emitted. 14 | 15 | ``` 16 | cd react/app 17 | npm start 18 | ``` 19 | This will start the npm dev server (by default on 8080, but since we already have our api on 8080 the dev server is on 8081). 20 | Point your browser at http://localhost:8081 21 | 22 | 23 | ### Building the components for cherrypy 24 | Once your changes have been tested, you need to rebuild the artifacts that cherrypy serves. 25 | ``` 26 | cd react/app 27 | npm run-script build 28 | ``` 29 | this places the updated and compiled content into the react/app/dist directory 30 | 31 | Promote the build to the live location where cherrypy picks it up from 32 | ``` 33 | cd ../.. 34 | cp react/app/dist/bundle.js www/ 35 | cp react/app/dist/css/style.css www/css/ 36 | ``` 37 | 38 | Stop the fioservice, and restart. 39 | 40 | 41 | ## Niggles 42 | When using npm start, if you see "X-Content-Type-Options: nosniff" errors against the patternfly file, check that 43 | it is in the dist folder. If not, copy it there and refresh your browser. 44 | 45 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | 2 | # FIOLoadGen 3 | Project that provides a structured test environment based on fio workload patterns. The project contains a number of tools that promote the following workflow; 4 | 1. Use fiodeploy to create the test environment (builds an fio client/server environment containing a specific number of workers) 5 | 2. run fioservice to provide an API and web interface to manage the tests and view the results 6 | 3. optionally, use fiocli to interact with the API, to run and query job state/results via the CLI 7 | 8 | These components provide the following features; 9 | - standard repeatable deployment of an fio testing framework 10 | - persistent store for job results and profiles for future reference (useful for regression testing, or bake-off's) 11 | - support for predefined fio job profiles and custom profiles defined in the UI 12 | - deployment of fio workers to multiple storageclasses (allows testing against different providers) 13 | - ability to export job results for reuse in other systems 14 | - ability to dump all jobs or a specific job in sqlite format for import in another system 15 | - fio job management through a RESTful API supporting 16 | - cli tool to interact with the API to run jobs, query output, query profiles 17 | - web front end supporting fio profile view/refresh, job submission and visualisation of fio results (using chartjs) 18 | - supported backend - openshift and kubernetes (tested with minikube) 19 | 20 | 21 | ## What does the workflow look like? 22 | Here's a demo against an openshift cluster. It shows the creation of the mgr pod and workers, and illustrates the use of the CLI to run and query jobs. 23 | 24 | ![demo](media/fioloadgen_1.2_2.gif) 25 | 26 | [full video (no sound)](https://youtu.be/YzakBle2afU) 27 | 28 | 29 | ## Installation 30 | The fioloadgen tool currently runs from your local directory, so you just need to download the repo and follow the steps in the "Deploying the FIOLoadgen.." section. 31 | However, the tool provides an API and web interface nd for that there are two options; 32 | either local or remote. 33 | 34 | Choosing 'local' means that the fioservice daemon will try and run on your machine, so 35 | you'll need to satisfy the python dependencies. If this sounds like a hassle, just 36 | choose the 'remote' option to deploy the service in the target kubernetes cluster, 37 | along with the FIO worker pods. 38 | 39 | ### Python Requirements 40 | - python3 41 | - python3-cherrypy 42 | - python3-requests 43 | 44 | ### Notes 45 | Cherrypy can be a pain to install, depending on your distro. Here's a quick table to provide some pointers 46 | 47 | | Distro | Repo | Dev Tested 48 | |----------|---------|----------| 49 | | RHEL8 | ceph (rhcs4) or ocs4 repos | Yes via downstream repo rhceph repo | 50 | | Fedora | in the base repo | Yes via rpm | 51 | | CentOS8 | N/A | Untested | 52 | | OpenSuSE | base repo | Untested | 53 | | Ubuntu | base repo | Untested | 54 | 55 | If all else fails you have two options 56 | - install pip3, and install with ```pip3 install cherrypy``` 57 | - use **remote** mode for the fioservice daemon which runs the API/web interface in the Openshift cluster (no local dependencies!) 58 | 59 | 60 | ## Deploying the FIOLOADGEN environment 61 | Before you deploy, you **must** have a working connection to openshift and the required CLI tool (oc) must be in your path. 62 | Once you have logged in to openshift, you can run the ```fiodeploy.sh``` script. This script is used to standup (```-s```) **and** tear down (```-d```) test environments 63 | ``` 64 | $ ./fiodeploy -h 65 | Usage: fiodeploy [-dsh] 66 | -h ... display usage information 67 | -s ... setup an fio test environment 68 | -d ... destroy the given namespace 69 | -r ... reset - remove the lockfile 70 | e.g. 71 | > ./fiodeploy -s 72 | ``` 73 | Here's an example of a deployment, using the remote fioservice option. 74 | ``` 75 | [paul@rhp1gen3 fioloadgen]$ ./fiodeploy -s 76 | Checking kubernetes CLI is available 77 | ✔ oc command available 78 | Checking access to kubernetes 79 | ✔ access OK 80 | 81 | FIOLoadgen will use a dedicated namespace for the tests 82 | What namespace should be used [fio]? 83 | - checking existing namespaces 84 | ✔ namespace 'fio' will be used 85 | How many fio workers [2]? 86 | ✔ 2 worker pods will be deployed 87 | Checking available storageclasses 88 | - thin 89 | What storageclass should the fio worker pods use [ocs-storagecluster-ceph-rbd]? thin 90 | ✔ storageclass 'thin' will be used 91 | 92 | To manage the tests, FIOLoadgen can use either a local daemon on your machine (local), or 93 | deploy the management daemon to the target environment (remote) 94 | How do you want to manage the tests (local/remote) [local]? remote 95 | 96 | Setting up the environment 97 | 98 | Creating namespace (fio) 99 | ✔ namespace created OK 100 | Deploying the FIO workers statefulset 101 | statefulset.apps/fioworker created 102 | 0 103 | Waiting for pods to reach a running state 104 | - waiting for pods to reach 'Running' state (1/60) 105 | - waiting for pods to reach 'Running' state (2/60) 106 | - waiting for pods to reach 'Running' state (3/60) 107 | - waiting for pods to reach 'Running' state (4/60) 108 | - waiting for pods to reach 'Running' state (5/60) 109 | - waiting for pods to reach 'Running' state (6/60) 110 | - waiting for pods to reach 'Running' state (7/60) 111 | - waiting for pods to reach 'Running' state (8/60) 112 | - waiting for pods to reach 'Running' state (9/60) 113 | ✔ Pods ready 114 | Submitting the fioservice daemon 115 | serviceaccount/fioservice-sa created 116 | role.rbac.authorization.k8s.io/fioservice created 117 | rolebinding.rbac.authorization.k8s.io/fioservice-binding created 118 | deployment.apps/fioservice created 119 | Waiting for fioservice to reach Ready state 120 | - waiting (1/60) 121 | - waiting (2/60) 122 | - waiting (3/60) 123 | ✔ FIOservice pod ready 124 | Adding port-forward rule 125 | 126 | Access the UI at http://localhost:8080. From there you may submit jobs and view 127 | job output and graphs 128 | 129 | 130 | To drop the test environment, use the fiodeploy.sh -d command 131 | ``` 132 | 133 | The pods deployed to the 'fio' namespace vary slightly depending upon 134 | whether you run the fioservice on your local machine, or run it within the target 135 | cluster itself (remote mode). 136 | 137 | In *'local'* mode, you will see an fiomgr pod. This pod provides the focal point for fio job management. The local fioservice daemon will interact with this pod using the 'oc' 138 | command to start test runs. You could 'exec' into the fiomgr pod directly to run fio 139 | jobs, which may provide the most hands-on experience for some users. However, using the 140 | fiomgr directly will not load results into the fioservice database or provide the 141 | analysis charts. 142 | 143 | With *'remote'* mode, fiodeploy will create a deployment in the target environment where the fioservice will run, and establishes a **port-forward** from your local machine to this pod. FIO jobs are managed by this pod, and all interactions between the fioservice and the kubernetes platform is performed using the kubernetes API. 144 | 145 | In either deployment model, the workers that perform the I/O tests are deployed using a 146 | statefulset. The pods created are called 'fioworker-N', with each fioworker pod using a 147 | PVC from the requested storageclass defined during setup. 148 | 149 | ## Using the UI 150 | The fioservice daemon provides an API which supports the web interface and the fiocli 151 | command. The UI is split into 3 main areas 152 | - Page banner/heading 153 | - FIO profiles 154 | - Job Summary and Analysis 155 | 156 | ### Page Banner 157 | ![banner](media/fioloadgen_banner.png) 158 | 159 | 160 | ### FIO Profiles 161 | ![banner](media/fioloadgen_profiles.png) 162 | 163 | 164 | ### Job Summary & Analysis 165 | ![banner](media/fioloadgen_jobs.png) 166 | 167 | Each row in the job table has a menu icon that provides options for managing the job 168 | based on it's state. For example, queued jobs may be deleted and complete jobs rerun. 169 | 170 | 171 | ## Manually starting a local FIO Service (API/UI) 172 | Normally the fioservice is started by the ```fiodeploy``` script. But if you need to manage 173 | things for yourself, this info may help. 174 | 175 | ### Starting the fioservice daemon 176 | ``` 177 | > ./fioservice --mode=dev start 178 | ``` 179 | 1. Defaults to an openshift connection (--type=oc) and namespace of fio (--namespace=fio) 180 | 2. Expects to be run from the root of the project folder (at start up it will attempt 181 | to refresh profiles from the local project folder.) 182 | 183 | ### Stopping the fioservice daemon 184 | ``` 185 | > ./fioservice stop 186 | ``` 187 | 188 | 189 | ## Removing the FIOLoadgen environment 190 | The fiodeploy command provides a **-d** switch to handle the remove of the fio namespace 191 | and all associated resources. 192 | 193 | 194 | ## Using the CLI command to manage tests 195 | 196 | 1. Show the current status of the fioservice 197 | ``` 198 | > ./fiocli status 199 | 200 | Example output: 201 | 202 | Target : Kubernetes 203 | Debug Mode : No 204 | Workers 205 | my-storageclass : 1 206 | standard : 1 207 | Job running : No 208 | Jobs queued : 0 209 | Uptime : 2 days, 1:58:07 210 | 211 | ``` 212 | 213 | 2. List available IO profiles you can test with 214 | ``` 215 | > ./fiocli profile --ls 216 | 217 | Example output: 218 | ./fiocli profile --ls 219 | - randr.job 220 | - randrlimited.job 221 | - randrw7030.job 222 | - seqwrite.job 223 | 224 | ``` 225 | 3. Show the parameters within a profile 226 | ``` 227 | > ./fiocli profile --show 228 | ``` 229 | 4. Add a fio job profile to the fioservice database 230 | ``` 231 | > ./fiocli profile-add --name --file= 232 | 233 | Example output: 234 | ./fiocli profile-add --name new.job --file=/home/test/fioloadgen/newread.job 235 | profile upload successful 236 | ``` 237 | 5. Remove an fio job profile from the fioservice database 238 | ``` 239 | > ./fiocli profile-rm --name 240 | 241 | Example output: 242 | ./fiocli profile-rm --name new.job 243 | profile deleted 244 | ``` 245 | 6. Run an fio job using a given profile 246 | ``` 247 | > ./fiocli run --profile --workers --title 248 | ``` 249 | 7. List jobs stored in the database 250 | ``` 251 | > ./fiocli job --ls 252 | 253 | Example output: 254 | ./fiocli job --ls 255 | Job ID Status End Time Job Title 256 | 91a2c232-1d36-4685-a94d-19ea6a253ae6 complete 2021-03-12 11:38:05 test run - sc=thin 257 | Jobs: 1 258 | 259 | ``` 260 | 8. Show summarized outut from a run 261 | ``` 262 | > ./fiocli.py job --show 263 | 264 | Example output: 265 | ./fiocli.py job --show 91a2c232-1d36-4685-a94d-19ea6a253ae6 266 | 267 | Id : 91a2c232-1d36-4685-a94d-19ea6a253ae6 268 | Title : test run - sc=thin 269 | Run Date : 2021-03-12 11:36:42 270 | Profile : randr.job 271 | Workers : 2 272 | Status : complete 273 | Summary : 274 | Clients: 2 275 | Total_Iops: 23212.479792 276 | Read Ms Min/Avg/Max: 0.50/0.51/0.51 277 | Write Ms Min/Avg/Max: 0.00/0.00/0.00 278 | 279 | ``` 280 | 9. show full json output from a run 281 | ``` 282 | > ./fiocli.py job --show --raw 283 | 284 | Example output: 285 | [paul@rhp1gen3 fioloadgen]$ ./fiocli.py job --show 91a2c232-1d36-4685-a94d-19ea6a253ae6 --raw 286 | 287 | Id : 91a2c232-1d36-4685-a94d-19ea6a253ae6 288 | Title : test run - sc=thin 289 | Run Date : 2021-03-12 11:36:42 290 | Profile : randr.job 291 | Workers : 2 292 | Status : complete 293 | Summary : 294 | Clients: 2 295 | Total_Iops: 23212.479792 296 | Read Ms Min/Avg/Max: 0.50/0.51/0.51 297 | Write Ms Min/Avg/Max: 0.00/0.00/0.00 298 | { 299 | "fio version": "fio-3.25", 300 | "timestamp": 1615502202, 301 | "time": "Thu Mar 11 22:36:42 2021", 302 | "global options": { 303 | "size": "5g", 304 | "directory": "/mnt", 305 | "iodepth": "4", 306 | "direct": "1", 307 | "bs": "4k", 308 | "time_based": "1", 309 | "ioengine": "libaio", 310 | "runtime": "60" 311 | }, 312 | "client_stats": [ 313 | { 314 | "jobname": "workload", 315 | "groupid": 0, 316 | "error": 0, 317 | "job options": { 318 | "rw": "randrw", 319 | "rwmixread": "100", 320 | "numjobs": "1" 321 | }, 322 | 323 | ``` 324 | 325 | 326 | ## The FIOLoadgen Database 327 | The fioservice maintains a sqlite database containing 2 tables - profiles and jobs, which 328 | can be managed using the fiocli command. 329 | 330 | 1. Dump the jobs table to a backup file 331 | ``` 332 | ./fiocli db-dump [-h] [--table {jobs,profiles}] [--out OUT] 333 | 334 | Example output: 335 | ./fiocli db-dump --table jobs --out myjobs.backup 336 | database dump of table 'jobs' written to myjobs.backup 337 | ``` 338 | 2. Export a specific job from the database 339 | ``` 340 | fiocli db-export [-h] [--table {jobs,profiles}] --row ROW [--out OUT] 341 | 342 | Example output: 343 | ./fiocli db-export --table jobs --row id=91a2c232-1d36-4685-a94d-19ea6a253ae6 344 | database table row from 'jobs' written to /home/paul/fioservice-db-jobs-row.sql 345 | ``` 346 | 347 | ## Files used by the environment 348 | Runtime files and a the database are placed in the users home directory 349 | 350 | | filename | Used By | Purpose | 351 | |----------|---------|---------| 352 | | ```fioservice.db``` | web service | sqlite3 database containing profiles and job information | 353 | | ```fioservice.pid``` | web service | pid file for the web service 354 | | ```fioservice.log``` | web service | cherrypy log file - error and generic log messages 355 | | ```fioservice.access.log``` | web service | cherrypy access log messages 356 | | ```fiodeploy.lock``` | deploy script | used as a lock file to prevent multiple deploys running 357 | 358 | 359 | ## TODO List 360 | - [x] implement a wait parameter in the CLI when running an fio job 361 | - [x] UI - define the UI structure and components 362 | - [x] UI - view results from db 363 | - [X] UI - show profiles, submit jobs 364 | - [X] UI - add use chart.js to visualize the results a run 365 | - [X] UI - reload the profiles in the UI with changes in the filesystem 366 | - [ ] extend the 'fiotester' container to include other benchmarking tools 367 | - [X] enable the fioservice to run away from the cli (remote loadgen deployments) 368 | - [X] provide an fioservice container that can be run on the target infrastructure, instead of locally 369 | - [ ] react optimization 370 | - [ ] formalise the code as an installable python package(why not add an rpm too?) 371 | - [X] replace raw information of a profile with widgets to make it more accessible 372 | - [X] use presets and custom 373 | - [X] store the fio parameters used the jobs database record 374 | 375 | -------------------------------------------------------------------------------- /buildah/README.md: -------------------------------------------------------------------------------- 1 | ## fio workload container image 2 | To build the container image, simply run the script 3 | ``` 4 | > ./buildfio.sh 5 | ``` 6 | 7 | The script uses buildah to compose the image, and is based on alpine to keep the image size as small as possible. 8 | 9 | Once built, tag the image and upload to docker.io 10 | 11 | -------------------------------------------------------------------------------- /buildah/buildfio.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/bash 2 | # use buildah to create a container holding fio 3 | TAG=$1 4 | IMAGE="alpine:edge" 5 | 6 | if [ -z "${TAG}" ]; then 7 | TAG="latest" 8 | fi 9 | 10 | echo "Using tag ${TAG}" 11 | 12 | container=$(buildah from $IMAGE) 13 | #mountpoint=$(buildah mount $container) 14 | buildah run $container apk add fio 15 | buildah run $container apk add bash 16 | buildah run $container apk add sysstat 17 | buildah run $container apk add iperf 18 | buildah run $container apk add rsync 19 | #buildah run $container apk add prometheus-node-exporter --repository http://dl-cdn.alpinelinux.org/alpine/edge/testing/ 20 | buildah run $container apk add s3cmd --repository http://dl-cdn.alpinelinux.org/alpine/edge/testing/ 21 | 22 | 23 | buildah run $container mkdir -p /fio/jobs 24 | buildah run $container mkdir /reports 25 | buildah copy $container ./startup.sh /startup.sh 26 | buildah copy $container ./startfio /usr/bin/startfio 27 | 28 | buildah run $container chmod u+x /startup.sh 29 | # expose port 30 | buildah config --port 8765 $container 31 | 32 | # set working dir 33 | #buildah config --workingdir /usr/share/grafana $container 34 | 35 | 36 | 37 | # entrypoint 38 | buildah config --entrypoint "/startup.sh" $container 39 | 40 | # finalize 41 | buildah config --label maintainer="Paul Cuzner " $container 42 | buildah config --label description="fio client/server" $container 43 | buildah config --label summary="fio client/server container - uses environment var MODE=server|client" $container 44 | buildah commit --format docker --squash $container fiotester:$1 45 | 46 | -------------------------------------------------------------------------------- /buildah/buildfioservice.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/bash 2 | # use buildah to create a container holding the fio web service (UI+API) 3 | 4 | if [ ! -z "$1" ]; then 5 | TAG=$1 6 | else 7 | TAG='latest' 8 | fi 9 | 10 | echo "Build image with the tag: $TAG" 11 | 12 | IMAGE="alpine:edge" 13 | 14 | container=$(buildah from $IMAGE) 15 | #mountpoint=$(buildah mount $container) 16 | buildah run $container apk add fio 17 | buildah run $container apk add bash 18 | buildah run $container apk add sysstat 19 | buildah run $container apk add iperf 20 | buildah run $container apk add rsync 21 | buildah run $container apk add python3 22 | buildah run $container apk add --update py3-pip 23 | 24 | # buildah run $container pip3 install --upgrade pip3 25 | # buildah run $container pip3 install jaraco.collections 26 | # buildah run $container pip3 install zc.lockfile 27 | # buildah run $container pip3 install cheroot 28 | # buildah run $container pip3 install portend 29 | buildah run $container pip3 install kubernetes 30 | buildah run $container apk add py3-cherrypy --repository http://dl-cdn.alpinelinux.org/alpine/edge/community/ 31 | buildah run $container apk add py3-more-itertools 32 | 33 | # buildah run $container apk add py3-wheel --repository http://dl-cdn.alpinelinux.org/alpine/edge/main/ 34 | 35 | buildah run $container mkdir -p /var/lib/fioloadgen/{jobs,reports} 36 | buildah run $container mkdir -p /var/log/fioloadgen 37 | buildah run $container mkdir -p /var/run/fioloadgen 38 | 39 | buildah copy $container ../data/fio/jobs/ /var/lib/fioloadgen/jobs 40 | buildah copy $container ../fioservice /fioservice 41 | buildah copy $container ../fiotools /fiotools 42 | buildah copy $container ../www /www 43 | 44 | buildah run $container chmod g+w -R /var/lib/fioloadgen 45 | buildah run $container chmod g+w -R /var/log/fioloadgen 46 | buildah run $container chmod g+w -R /var/run/fioloadgen 47 | 48 | # expose port 49 | # buildah config --port 8080 $container 50 | 51 | # set working dir 52 | #buildah config --workingdir /usr/share/grafana $container 53 | 54 | 55 | 56 | # entrypoint 57 | buildah config --entrypoint "./fioservice start" $container 58 | 59 | # finalize 60 | buildah config --label maintainer="Paul Cuzner " $container 61 | buildah config --label description="fioservice API/UI" $container 62 | buildah config --label summary="fioservice focal point to interact with fiomgr/fioworker pods" $container 63 | buildah commit --format docker --squash $container fioservice:$TAG 64 | -------------------------------------------------------------------------------- /buildah/startfio: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | SCRIPT_NAME=$(basename $0) 4 | 5 | 6 | acquire_lock() { 7 | if [ -f ./fiojob.lock ]; then 8 | exit 16 9 | else 10 | touch ./fiojob.lock 11 | fi 12 | } 13 | 14 | release_lock() { 15 | rm -f ./fiojob.lock 16 | } 17 | 18 | get_max_workers() { 19 | workers_file_name="$1_worker-ip.list" 20 | echo $(grep -c "" ${workers_file_name}) 21 | } 22 | 23 | run_fio() { 24 | acquire_lock 25 | workers_file_name="$1_worker-ip.list" 26 | # build a client list based on the requested worker count 27 | cat ${workers_file_name} | head -n $2 > ./clients 28 | 29 | # run the fio job across the clients 30 | fio --client=./clients fio/jobs/"$3" --output-format=json --output=/reports/"$4" 31 | 32 | release_lock 33 | } 34 | 35 | usage() { 36 | echo -e "Usage: ${SCRIPT_NAME} [-puwh]" 37 | echo -e "\t-h ... display usage information" 38 | echo -e "\t-p ... fio profile name (from /fio/jobs directory)" 39 | echo -e "\t-s ... storageclass to use for the workers" 40 | echo -e "\t-o ... output file name to use (within /reports)" 41 | echo -e "\t-w ... number of workers to run the workload against" 42 | 43 | echo "e.g." 44 | echo -e "> ./${SCRIPT_NAME} -p randrw7030.job -s standard -w 5 -o randrw7030-output.json\n" 45 | } 46 | 47 | main() { 48 | 49 | while getopts "p:s:o:w:h" option; do 50 | case "${option}" in 51 | h) 52 | usage 53 | exit 0 54 | ;; 55 | p) 56 | profile=${OPTARG} 57 | ;; 58 | s) 59 | storageclass=${OPTARG} 60 | ;; 61 | o) 62 | outfile=${OPTARG} 63 | ;; 64 | w) 65 | workers=${OPTARG} 66 | if ! [ "$workers" -eq "$workers" ] 2>/dev/null ; then 67 | echo "invalid -w parameter" 68 | exit 1 69 | fi 70 | ;; 71 | :) 72 | echo "Expected argument for -${OPTARG} missing" 73 | exit 4 74 | ;; 75 | *) 76 | echo "Unsupported option." 77 | usage 78 | exit 8 79 | ;; 80 | esac 81 | args_given=0 82 | done 83 | 84 | # workers is numeric 85 | # if [[ $workers -lt 1 || $workers -gt $max_workers ]]; then 86 | # workers=${max_workers} 87 | # fi 88 | 89 | if [[ ! -v args_given ]]; then 90 | # got to give me something! 91 | usage 92 | exit 4 93 | fi 94 | 95 | if [[ ! -v storageclass ]]; then 96 | # storageclass must be given 97 | echo "Missing storageclass (-s) value" 98 | exit 4 99 | fi 100 | if [[ ! -v profile ]]; then 101 | # profile must be given 102 | echo "Missing profile (-p) value" 103 | exit 4 104 | fi 105 | 106 | max_workers=$(get_max_workers ${storageclass}) 107 | # -v tests needs bash 4.2 or later 108 | if [[ ! -v workers ]]; then 109 | workers=$max_workers 110 | fi 111 | 112 | if [[ $workers -lt 1 || $workers -gt $max_workers ]]; then 113 | workers=${max_workers} 114 | fi 115 | 116 | if [[ ! -v outfile ]]; then 117 | outfile="${profile}-$(date +%s).json" 118 | fi 119 | 120 | run_fio "${storageclass}" "${workers}" "${profile}" "${outfile}" 121 | } 122 | 123 | main $@ 124 | -------------------------------------------------------------------------------- /buildah/startup.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | 3 | server() { 4 | echo "Starting server" 5 | fio --server 6 | } 7 | 8 | client() { 9 | echo "client mode - ready for shell access" 10 | tail -f /dev/null 11 | } 12 | 13 | if [ -z "$FIOMODE" ]; then 14 | server 15 | else 16 | if [ "$FIOMODE" == "server" ]; then 17 | server 18 | else 19 | client 20 | fi 21 | fi 22 | 23 | -------------------------------------------------------------------------------- /data/fio/jobs/randr.job: -------------------------------------------------------------------------------- 1 | [global] 2 | refill_buffers 3 | size=5g 4 | directory=/mnt 5 | direct=1 6 | time_based=1 7 | group_reporting 8 | ioengine=libaio 9 | 10 | [workload] 11 | rw=randrw 12 | rwmixread=100 13 | iodepth=4 14 | blocksize=4KB 15 | numjobs=1 16 | runtime=60 17 | -------------------------------------------------------------------------------- /data/fio/jobs/randrlimited.job: -------------------------------------------------------------------------------- 1 | [global] 2 | refill_buffers 3 | size=5g 4 | directory=/mnt 5 | direct=1 6 | time_based=1 7 | group_reporting 8 | ioengine=libaio 9 | 10 | [workload] 11 | rw=randread 12 | rate_iops=300 13 | blocksize=4KB 14 | iodepth=4 15 | numjobs=1 16 | runtime=60 17 | -------------------------------------------------------------------------------- /data/fio/jobs/randrw7030.job: -------------------------------------------------------------------------------- 1 | [global] 2 | refill_buffers 3 | time_based=1 4 | size=5g 5 | directory=/mnt 6 | direct=1 7 | group_reporting 8 | ioengine=libaio 9 | 10 | [workload] 11 | rw=randrw 12 | rwmixread=70 13 | blocksize=4KB 14 | iodepth=4 15 | numjobs=1 16 | runtime=60 17 | -------------------------------------------------------------------------------- /data/fio/jobs/seqwrite.job: -------------------------------------------------------------------------------- 1 | [global] 2 | refill_buffers 3 | time_based=1 4 | size=5g 5 | directory=/mnt 6 | direct=1 7 | group_reporting 8 | ioengine=sync 9 | 10 | [workload] 11 | rw=write 12 | blocksize=64KB 13 | iodepth=1 14 | numjobs=1 15 | runtime=60 16 | -------------------------------------------------------------------------------- /fetchlatency.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | 3 | import os 4 | import sys 5 | import json 6 | import argparse 7 | import logging 8 | 9 | logger = logging.getLogger() 10 | logger.setLevel(logging.INFO) 11 | sh = logging.StreamHandler(sys.stdout) 12 | logger.addHandler(sh) 13 | 14 | 15 | vars_list = [ 16 | "read/iops", 17 | "read/clat_ns/percentile/95.000000", 18 | "write/iops", 19 | "write/clat_ns/percentile/95.000000", 20 | ] 21 | 22 | 23 | def cmd_parser(): 24 | parser = argparse.ArgumentParser(description="fetch latencies from fio output") 25 | parser.add_argument( 26 | '--debug', 27 | action='store_true', 28 | default=False, 29 | help="provide debug output during processing") 30 | parser.add_argument( 31 | '--file', 32 | type=str, 33 | required=True, 34 | help="filepath") 35 | parser.add_argument( 36 | '--format', 37 | choices=['csv', 'json'], 38 | default='csv', 39 | help='output format') 40 | parser.add_argument( 41 | '--outfile', 42 | type=str, 43 | required=False, 44 | help='output file name') 45 | 46 | return parser 47 | 48 | 49 | def get_item(response, path): 50 | for item in path: 51 | response = response[item] 52 | return response 53 | 54 | 55 | def dump(output): 56 | if args.format == 'csv': 57 | out = format_csv(output) 58 | elif args.format == 'json': 59 | out = format_json(output) 60 | logger.info("dumping to summary file") 61 | with open(args.out, 'w') as f: 62 | f.write(out) 63 | 64 | 65 | def summarize(extracted): 66 | """print some quick summary stats""" 67 | logger.debug("starting summarization") 68 | num_clients = len(extracted) 69 | total_iops = 0 70 | total_read_lat = 0 71 | total_write_lat = 0 72 | for client in extracted: 73 | total_iops += float(client['read/iops']) + float(client['write/iops']) 74 | total_read_lat += float(client['read/clat_ns/percentile/95.000000']) 75 | total_write_lat += float(client['write/clat_ns/percentile/95.000000']) 76 | 77 | logger.info("Clients found : {}".format(num_clients)) 78 | logger.info("Aggregate IOPS : {}".format(int(total_iops))) 79 | logger.info("AVG IOPS per client : {}".format(int(total_iops/num_clients))) 80 | logger.info("AVG Read Latency : {:.2f}ms".format((total_read_lat / num_clients) / 1000000)) 81 | logger.info("AVG Write Latency : {:.2f}ms".format((total_write_lat / num_clients) / 1000000)) 82 | 83 | 84 | def format_json(output): 85 | logger.debug("creating json output") 86 | out = dict() 87 | out['data'] = output 88 | return json.dumps(out,indent=2) 89 | 90 | 91 | def format_csv(output): 92 | logger.debug("creating csv output") 93 | headers = output[0].keys() 94 | csv = list() 95 | csv.append(','.join(headers)) 96 | for data in output: 97 | out = [] 98 | for k in data: 99 | out.append(data[k]) 100 | csv.append(','.join(out)) 101 | return '\n'.join(csv) 102 | 103 | 104 | def main(): 105 | logger.info("Starting..\n") 106 | if not os.path.exists(args.file): 107 | print("Error: file not found") 108 | return 109 | logger.debug("reading file") 110 | with open(args.file) as f: 111 | data = f.read() 112 | 113 | logger.debug("parsing json") 114 | try: 115 | fio_json = json.loads(data) 116 | except ValueError: 117 | print("Error: Invalid file format - must be json") 118 | sys.exit(4) 119 | 120 | if "client_stats" not in fio_json: 121 | print("Error: Invalid fio output - client stats are missing") 122 | sys.exit(8) 123 | client_stats = fio_json['client_stats'] 124 | extract = list() 125 | for item in client_stats: 126 | if item['jobname'].lower() == 'all clients': 127 | continue 128 | hostname = item['hostname'] 129 | logger.debug("- processing host {}".format(hostname)) 130 | hostdata = { 131 | "hostname": hostname, 132 | } 133 | for v in vars_list: 134 | path = v.split('/') 135 | path_value = get_item(item, path) 136 | hostdata[v] = str(path_value) 137 | logger.debug(" {}: {}".format(v, path_value)) 138 | extract.append(hostdata) 139 | 140 | summarize(extract) 141 | if args.outfile: 142 | dump(extract) 143 | logger.info("\nComplete") 144 | 145 | 146 | if __name__ == '__main__': 147 | parser = cmd_parser() 148 | args = parser.parse_args() 149 | if args.debug: 150 | logger.setLevel(logging.DEBUG) 151 | logger.debug("setting debug on") 152 | main() 153 | -------------------------------------------------------------------------------- /fiocli: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | import os 3 | import sys 4 | import argparse 5 | import requests 6 | import datetime 7 | import json 8 | import time 9 | 10 | from fiotools import __version__ 11 | from fiotools import configuration 12 | from fiotools.utils import rfile 13 | 14 | import logging 15 | logger = logging.getLogger() 16 | handler = logging.StreamHandler() 17 | logger.addHandler(handler) 18 | logger.setLevel(logging.INFO) 19 | 20 | REQUEST_TIMEOUT = 2 21 | MAX_PROFILE_NAME = 24 22 | MAX_JOB_TITLE = 60 23 | 24 | 25 | def cmd_parser(): 26 | parser = argparse.ArgumentParser( 27 | description='Interact with the fio web service', 28 | formatter_class=argparse.ArgumentDefaultsHelpFormatter, 29 | ) 30 | 31 | parser.add_argument( 32 | "--version", 33 | action='store_true', 34 | default=False, 35 | help="Show fioloadgen version" 36 | ) 37 | parser.add_argument( 38 | "--mode", 39 | choices=['debug', 'dev', 'prod'], 40 | default='dev', 41 | help="Mode of the CLI" 42 | ) 43 | parser.add_argument( 44 | "-v", "--verbose", 45 | action="store_true", 46 | default=False, 47 | help="show debug messages" 48 | ) 49 | 50 | subparsers = parser.add_subparsers(help="sub-commands") 51 | 52 | parser_status = subparsers.add_parser('status', help="show the status of the web service") 53 | parser_status.set_defaults(func=command_status) 54 | 55 | parser_profile = subparsers.add_parser( 56 | 'profile', 57 | help='view and manage the fio profiles') 58 | parser_profile.set_defaults(func=command_profile) 59 | parser_profile.add_argument( 60 | '--show', 61 | type=str, 62 | metavar='', 63 | help="show content of an fio profile", 64 | ) 65 | parser_profile.add_argument( 66 | '--ls', 67 | action='store_true', 68 | help="list available fio profiles", 69 | ) 70 | parser_profile.add_argument( 71 | '--refresh', 72 | action='store_true', 73 | help="apply fio profiles in data/fio/jobs to the local database and the remote fiomgr pod", 74 | ) 75 | parser_profile_add = subparsers.add_parser( 76 | 'profile-add', 77 | help='Add a profile to the fioservice database') 78 | parser_profile_add.set_defaults(func=command_profile_add) 79 | parser_profile_add.add_argument( 80 | '--name', 81 | type=str, 82 | metavar='', 83 | help=f'name for the fio job profile (1-{MAX_PROFILE_NAME} chars)' 84 | ) 85 | parser_profile_add.add_argument( 86 | '--file', 87 | type=str, 88 | metavar='', 89 | help="local filename containing the fio job to upload" 90 | ) 91 | parser_profile_rm = subparsers.add_parser( 92 | 'profile-rm', 93 | help='Remove a profile from the fioservice database') 94 | parser_profile_rm.set_defaults(func=command_profile_rm) 95 | parser_profile_rm.add_argument( 96 | '--name', 97 | type=str, 98 | metavar='', 99 | help="description name for the fio job profile" 100 | ) 101 | 102 | parser_run = subparsers.add_parser( 103 | 'run', 104 | help="run a given fio profile") 105 | parser_run.set_defaults(func=command_run) 106 | parser_run.add_argument( 107 | '--profile', 108 | required=True, 109 | metavar='', 110 | help="fio profile for the fio workers to execute against", 111 | ) 112 | parser_run.add_argument( 113 | '--workers', 114 | default=9999, 115 | required=False, 116 | metavar='<# of workers>', 117 | help="number of workers to use for the profile", 118 | ) 119 | parser_run.add_argument( 120 | '--provider', 121 | required=False, 122 | default='aws', 123 | type=str, 124 | choices=['aws', 'vmware', 'baremetal', 'azure', 'gcp'], 125 | help="Infrastructure provider where the test is running", 126 | ) 127 | parser_run.add_argument( 128 | '--platform', 129 | required=False, 130 | default='openshift', 131 | type=str, 132 | choices=['openshift', 'kubernetes', 'ssh'], 133 | help="platform running the workload", 134 | ) 135 | parser_run.add_argument( 136 | '--title', 137 | required=True, 138 | type=str, 139 | metavar='', 140 | help=f'Job title for the run (1-{MAX_JOB_TITLE} chars)', 141 | ) 142 | parser_run.add_argument( 143 | '--wait', 144 | action='store_true', 145 | help="wait for the run to complete - NOT IMPLEMENTED YET", 146 | ) 147 | parser_run.add_argument( 148 | '--storageclass', 149 | type=str, 150 | metavar="", 151 | required=True, 152 | help="storageclass to use for the test run", 153 | ) 154 | 155 | parser_job = subparsers.add_parser( 156 | 'job', 157 | help="show fio job information") 158 | parser_job.set_defaults(func=command_job) 159 | parser_job.add_argument( 160 | '--ls', 161 | action="store_true", 162 | help="list all jobs", 163 | ) 164 | parser_job.add_argument( 165 | '--queued', 166 | action="store_true", 167 | help="additional parameter for --ls to limit results to only queued jobs", 168 | ) 169 | parser_job.add_argument( 170 | '--show', 171 | type=str, 172 | metavar='', 173 | help="show content of an job", 174 | ) 175 | parser_job.add_argument( 176 | '--delete', 177 | type=str, 178 | metavar='', 179 | help="delete a queued job", 180 | ) 181 | parser_job.add_argument( 182 | '--raw', 183 | action='store_true', 184 | help="show raw json from a completed job", 185 | ) 186 | parser_job.add_argument( 187 | '--with-spec', 188 | action='store_true', 189 | default=False, 190 | help="show job specification information", 191 | ) 192 | 193 | parser_db_dump = subparsers.add_parser( 194 | 'db-dump', 195 | help="manage the jobs table in the fioservice database") 196 | parser_db_dump.set_defaults(func=command_db_dump) 197 | # example: db-dump --table jobs --row id=0ca72318-c4ed-4a17-b81a-262c44a52fdc 198 | parser_db_dump.add_argument( 199 | '--table', 200 | choices=['jobs', 'profiles'], 201 | default='jobs', 202 | type=str, 203 | help="dump a table (jobs or profiles from the database (default is jobs)", 204 | ) 205 | parser_db_dump.add_argument( 206 | '--out', 207 | type=str, 208 | required=False, 209 | help="filename for the database dump output", 210 | ) 211 | 212 | parser_db_export = subparsers.add_parser( 213 | 'db-export', 214 | help="export a database row to a script file (for import)") 215 | parser_db_export.set_defaults(func=command_db_export) 216 | parser_db_export.add_argument( 217 | '--table', 218 | choices=['jobs', 'profiles'], 219 | default='jobs', 220 | type=str, 221 | help="table name to export a row from", 222 | ) 223 | parser_db_export.add_argument( 224 | '--row', 225 | default='', 226 | required=True, 227 | type=str, 228 | help="query string (key=value) that identifies a specific row in the table", 229 | ) 230 | parser_db_export.add_argument( 231 | '--out', 232 | type=str, 233 | help="filename for the exported row output", 234 | ) 235 | 236 | parser_db_import = subparsers.add_parser( 237 | 'db-import', 238 | help="import a database row export file") 239 | parser_db_import.set_defaults(func=command_db_import) 240 | parser_db_import.add_argument( 241 | '--table', 242 | default='jobs', 243 | choices=['jobs', 'profiles'], 244 | type=str, 245 | help="table to restore the export file to (either jobs or profiles)", 246 | ) 247 | parser_db_import.add_argument( 248 | '--file', 249 | default='', 250 | required=True, 251 | type=str, 252 | help="backup file to import to the database", 253 | ) 254 | parser_db_delete = subparsers.add_parser( 255 | 'db-delete', 256 | help="delete a row from a table") 257 | parser_db_delete.set_defaults(func=command_db_delete) 258 | parser_db_delete.add_argument( 259 | '--table', 260 | choices=['jobs', 'profiles'], 261 | default='jobs', 262 | type=str, 263 | help="table where the row will be deleted from (default is jobs)", 264 | ) 265 | parser_db_delete.add_argument( 266 | '--row', 267 | default='', 268 | required=True, 269 | type=str, 270 | help="query string (key=value) that identifies a specific row in the table", 271 | ) 272 | return parser 273 | 274 | 275 | def _build_qry_string(qs): 276 | try: 277 | args.row.split('=') 278 | except ValueError: 279 | # trigger if >1 '=' sign or no '=' sign at all 280 | return '' 281 | else: 282 | return f'?{qs}' 283 | 284 | 285 | def _extract_API_error(response): 286 | js = json.loads(response._content.decode()) 287 | return "Error: {}".format(js['message']) 288 | 289 | 290 | def command_db_delete(): 291 | qstring = _build_qry_string(args.row) 292 | if not qstring: 293 | print("row must specify a single key=value string i.e. --row id=mykey") 294 | sys.exit(1) 295 | 296 | r = requests.delete("{}/db/{}{}".format(url, args.table, qstring)) 297 | if r.status_code == 200: 298 | print("database table row from '{}' deleted".format(args.table)) 299 | else: 300 | print("database delete API request failed: {}".format(r.status_code)) 301 | print(_extract_API_error(r)) 302 | 303 | 304 | def command_db_export(): 305 | outfile = '' 306 | 307 | qstring = _build_qry_string(args.row) 308 | if not qstring: 309 | print("row must specify a single key=value string i.e. --row id=mykey") 310 | sys.exit(1) 311 | 312 | if args.out: 313 | outfile = args.out 314 | else: 315 | outfile = os.path.join(os.path.expanduser('~'), "fioservice-db-{}-row.sql".format(args.table)) 316 | 317 | r = requests.get("{}/db/{}{}".format(url, args.table, qstring)) 318 | if r.status_code == 200: 319 | with open(outfile, 'wb') as f: 320 | f.write(r.content) 321 | print("database table row from '{}' written to {}".format(args.table, outfile)) 322 | else: 323 | print("database dump API request failed: {}".format(r.status_code)) 324 | print(_extract_API_error(r)) 325 | 326 | 327 | def command_db_import(): 328 | # file must contain a single insert into "" clause 329 | if not os.path.exists(args.file): 330 | print(f"file not found - {args.file}") 331 | sys.exit(1) 332 | 333 | sql_script = rfile(args.file) 334 | if sql_script.count('INSERT INTO "{}"'.format(args.table)) != 1: 335 | print("file invalid format - must contain a single INSERT command") 336 | sys.exit(1) 337 | 338 | headers = {'Content-type': 'application/json'} 339 | r = requests.post( 340 | "{}/db/{}".format(url, args.table), 341 | json={ 342 | "sql_script": sql_script, 343 | }, 344 | headers=headers 345 | ) 346 | if r.status_code == 200: 347 | print("data import successful") 348 | else: 349 | print("database import failed: {}".format(r.status_code)) 350 | print(_extract_API_error(r)) 351 | 352 | 353 | def command_db_dump(): 354 | outfile = '' 355 | 356 | if args.out: 357 | outfile = args.out 358 | else: 359 | outfile = os.path.join(os.path.expanduser('~'), "fioservice-db-{}.sql".format(args.table)) 360 | 361 | r = requests.get("{}/db/{}".format(url, args.table)) 362 | if r.status_code == 200: 363 | with open(outfile, 'wb') as f: 364 | f.write(r.content) 365 | print("database dump of table '{}' written to {}".format(args.table, outfile)) 366 | else: 367 | print("database dump API request failed: {}".format(r.status_code)) 368 | 369 | 370 | def _fetch_status(): 371 | try: 372 | logger.debug(f"Issuing call to {url}/status") 373 | r = requests.get("{}/status".format(url), timeout=REQUEST_TIMEOUT) 374 | except (requests.exceptions.ConnectionError, ConnectionRefusedError, requests.exceptions.ReadTimeout): 375 | print(f'Unable to reach the fioservice daemon at {url}/status') 376 | print('Either start the daemon, or use "fiodeploy -p " to reconnect to the remote daemon port') 377 | sys.exit(1) 378 | 379 | return r 380 | 381 | 382 | def command_status(): 383 | 384 | r = _fetch_status() 385 | if r.status_code == 200: 386 | logger.debug("/status call successful") 387 | js = r.json()['data'] 388 | 389 | job_running = f"Yes ({js['active_job_id']})" if js['task_active'] else 'No' 390 | debug = 'Yes' if js['debug_mode'] else 'No' 391 | print("\nTarget : {}".format(js['target'])) 392 | print("Debug Mode : {}".format(debug)) 393 | print("Workers") 394 | workers = js.get('workers', {}) 395 | max_len = max([len(k) for k in workers.keys()]) 396 | for sc in workers.keys(): 397 | print(f" {sc:<{max_len}} : {workers[sc]}") 398 | 399 | print("Job running : {}".format(job_running)) 400 | print("Jobs queued : {}".format(js['tasks_queued'])) 401 | print("Uptime : {}\n".format(str(datetime.timedelta(seconds=int(js['run_time']))))) 402 | else: 403 | print("Failed to retrieve web service status [{}]".format(r.status_code)) 404 | 405 | 406 | def command_profile(): 407 | 408 | if not args.ls and not args.show and not args.refresh: 409 | print("use -h to view the available profile subcommands") 410 | sys.exit(1) 411 | 412 | logger.debug("Issuing API call to /profile endpoint") 413 | r = requests.get("{}/profile".format(url)) 414 | profiles = [p['name'] for p in r.json()['data']] 415 | 416 | if args.ls: 417 | # logger.debug("Issuing API call to /profile endpoint") 418 | # r = requests.get("{}/profile".format(url)) 419 | # data = r.json()['data'] 420 | for p in profiles: 421 | print(f"- {p}") 422 | elif args.show: 423 | if args.show not in profiles: 424 | print("The server doesn't have a profile called '{}'. Available profiles are: {}".format(args.show, ', '.join(profiles))) 425 | sys.exit(1) 426 | 427 | r = requests.get("{}/profile/{}".format(url, args.show)) 428 | data = r.json()['data'] 429 | print(data) 430 | elif args.refresh: 431 | # refresh the profiles from the local filesystem 432 | r = requests.get("{}/profile?refresh=true".format(url)) 433 | if r.status_code == 200: 434 | print("Profiles refreshed from the filesystem versions") 435 | summary = r.json()['summary'] 436 | for k in summary: 437 | print(" - {:<11s}: {:>2}".format(k, len(summary[k]))) 438 | else: 439 | print("Profile refresh failed: {}".format(r.status_code)) 440 | 441 | 442 | def command_profile_add(): 443 | # file given must exist 444 | if not os.path.exists(args.file): 445 | print("File not found") 446 | exit(1) 447 | if len(args.name) > MAX_PROFILE_NAME: 448 | print(f'Profile name length is too long (maximum allowed is {MAX_PROFILE_NAME}') 449 | exit(1) 450 | 451 | try: 452 | with open(args.file) as f: 453 | data = f.read() 454 | except IOError: 455 | print("Unable to read the file..permissions issue?") 456 | exit(1) 457 | 458 | payload = { 459 | "data": data, 460 | } 461 | r = requests.put(f"{url}/profile/{args.name}", 462 | data=json.dumps(payload), 463 | headers={ 464 | "Content-Type": "application/json" 465 | }) 466 | if r.status_code == 200: 467 | print("profile upload successful") 468 | else: 469 | print(r.json().get('message', 'Unexpected error')) 470 | 471 | 472 | def command_profile_rm(): 473 | r = requests.delete(f"{url}/profile/{args.name}") 474 | if r.status_code == 200: 475 | print("profile deleted") 476 | else: 477 | print(r.json().get('message', 'Unexpected error')) 478 | 479 | 480 | def show_spec(api_response): 481 | data = json.loads(api_response.json()['data']) 482 | if data.get('status', None) == 'complete': 483 | # print the spec 484 | js = json.loads(data.get('raw_json')) 485 | fio_version = js.get('fio version', 'unknown') 486 | global_options = js.get('global options', '') 487 | stats = js.get('client_stats') 488 | print(f'FIO version : {fio_version}') 489 | print("Job Specification") 490 | print('[global]') 491 | for k in global_options: 492 | print(f'{k}={global_options[k]}') 493 | if stats: 494 | for job in stats: 495 | if job.get('jobname') == 'All clients': 496 | continue 497 | print(f'[{job.get("jobname")}]') 498 | job_options = job.get('job options') 499 | for parm in job_options: 500 | print(f'{parm}={job_options.get(parm)}') 501 | 502 | 503 | def show_summary(api_response): 504 | keys_to_show = ['id', 'title', 'started', 'profile', 'workers', 'status'] # NOQA 505 | data = json.loads(api_response.json()['data']) 506 | for k in keys_to_show: 507 | if k == 'started': 508 | if data['started']: 509 | print("Run Date : {}".format(datetime.datetime.fromtimestamp(data[k]).strftime('%Y-%m-%d %H:%M:%S'))) # NOQA 510 | else: 511 | print("Run Date : pending") 512 | else: 513 | print("{:<9}: {}".format(k.title(), data[k])) 514 | if data.get('summary', None): 515 | print("Summary :") 516 | js = json.loads(data['summary']) 517 | for k in js.keys(): 518 | if k == 'total_iops': 519 | name = "Total IOPS" 520 | v = f'{int(js[k]):,}' 521 | else: 522 | name = k.title() 523 | v = js[k] 524 | print(f' {name}: {v}') 525 | else: 526 | print("Summary : Unavailable (missing)") 527 | 528 | 529 | def job_wait(job_uuid): 530 | 531 | try: 532 | while True: 533 | r = requests.get("{}/job/{}".format(url, job_uuid)) 534 | if r.status_code != 200: 535 | break 536 | js = json.loads(r.json()['data']) 537 | if js['status'] in ['complete', 'failed']: 538 | break 539 | sys.stdout.write(".") 540 | sys.stdout.flush() 541 | time.sleep(2) 542 | except KeyboardInterrupt: 543 | print("\nWait aborted") 544 | sys.exit(1) 545 | print("\n") 546 | return r 547 | 548 | 549 | def command_run(): 550 | if len(args.title) > MAX_JOB_TITLE: 551 | print(f'title is too long. Must be 1-{MAX_JOB_TITLE} chars in length') 552 | exit(1) 553 | 554 | print("Run fio workload profile {}".format(args.profile)) 555 | headers = {'Content-type': 'application/json'} 556 | r = requests.post( 557 | '{}/job/{}'.format(url, args.profile), 558 | json={ 559 | "workers": args.workers, 560 | "title": args.title, 561 | "provider": args.provider, 562 | "platform": args.platform, 563 | "storageclass": args.storageclass, 564 | }, 565 | headers=headers) 566 | 567 | if r.status_code == 202: 568 | response = r.json()['data'] 569 | print("- Request queued with uuid = {}".format(response['uuid'])) 570 | if args.wait: 571 | print("Running.", end="") 572 | completion = job_wait(response['uuid']) 573 | if completion.status_code == 200: 574 | show_summary(completion) 575 | else: 576 | print("- Request failed with status code {}".format(completion.status_code)) 577 | else: 578 | print(f'- Job request failed ({r.status_code}). {r.json().get("message", "unknown error")}') 579 | 580 | 581 | def command_job(): 582 | if args.ls: 583 | 584 | # show all jobs in the db 585 | field_list = ['id', 'status', 'title', 'ended'] 586 | r = requests.get("{}/job?fields={}".format(url, ','.join(field_list))) 587 | data = r.json()['data'] 588 | sdata = sorted(data, key=lambda i: i['ended'] if i['ended'] else 9999999999, reverse=True) 589 | print("{:<37} {:<11} {:^19} {}".format('Job ID', 'Status', "End Time", "Job Title")) 590 | row_count = 0 591 | for p in sdata: 592 | if args.queued and p['status'] != 'queued': 593 | continue 594 | 595 | if p['ended']: 596 | end_time = datetime.datetime.fromtimestamp(p['ended']).strftime("%Y-%m-%d %H:%M:%S") 597 | else: 598 | end_time = 'N/A' 599 | 600 | print("{:<37} {:<11} {:^19} {}".format(p['id'], p['status'], end_time, p['title'])) 601 | row_count += 1 602 | print("Jobs: {:>3}".format(row_count)) 603 | 604 | elif args.show: 605 | # show a specific job record 606 | r = requests.get("{}/job/{}".format(url, args.show)) 607 | if r.status_code == 200: 608 | show_summary(r) 609 | if args.with_spec: 610 | show_spec(r) 611 | 612 | if args.raw: 613 | jstr = json.loads(r.json()['data'])['raw_json'] 614 | js = json.loads(jstr) 615 | try: 616 | print(json.dumps(js, indent=2)) 617 | except BrokenPipeError: 618 | pass 619 | elif r.status_code == 404: 620 | print("Job with id '{}', does not exist in the database".format(args.show)) 621 | else: 622 | print("Unknown status returned : {}".format(r.status_code)) 623 | elif args.delete: 624 | # delete a queued job 625 | r = requests.delete("{}/job/{}".format(url, args.delete)) 626 | if r.status_code == 200: 627 | print("Queued job '{}' has been marked for deletion".format(args.delete)) 628 | else: 629 | handle_error(r) 630 | elif args.raw: 631 | print("Syntax error: the --raw parameter can only be used with --show ") 632 | 633 | 634 | def handle_error(response): 635 | js = response.json() 636 | print("{} [{}]".format(js.get('message', "Server didn't return an error description!"), response.status_code)) 637 | 638 | 639 | if __name__ == '__main__': 640 | 641 | # profiles = [] 642 | parser = cmd_parser() 643 | args = parser.parse_args() 644 | 645 | if args.version: 646 | print("fioloadgen version : {}".format(__version__)) 647 | sys.exit(0) 648 | 649 | if args.verbose: 650 | logger.setLevel(logging.DEBUG) 651 | 652 | if 'func' in args: 653 | configuration.init(args) 654 | 655 | api_address = os.environ.get( 656 | 'FIO_API_ADDRESS', 657 | '{}:{}'.format( 658 | configuration.settings.ip_address, 659 | configuration.settings.port 660 | ) 661 | ) 662 | url = 'http://{}/api'.format(api_address) # used by all functions 663 | if args.func.__name__ == 'command_status': 664 | args.func() 665 | else: 666 | try: 667 | logger.debug("Checking API access by querying /ping endpoint") 668 | r = requests.get('{}/ping'.format(url), timeout=REQUEST_TIMEOUT) 669 | except (requests.exceptions.ConnectionError, ConnectionRefusedError, requests.exceptions.ReadTimeout): 670 | print(f'Unable to reach the fioservice daemon @ {url}. Did you run fiodeploy?') 671 | sys.exit(1) 672 | 673 | # profiles = [p['name'] for p in r.json()['data']] 674 | 675 | args.func() 676 | else: 677 | print("Unknown request") 678 | sys.exit(1) 679 | -------------------------------------------------------------------------------- /fiodeploy: -------------------------------------------------------------------------------- 1 | #!/usr/bin/bash 2 | 3 | # TODO 4 | # 1. make the command interaction optional - with oc or kubectl 5 | 6 | PLATFORM='' 7 | DEFAULT_NAMESPACE='fio' 8 | DEFAULT_WORKERS=2 9 | DEFAULT_SC='ocs-storagecluster-ceph-rbd' 10 | DEFAULT_SEED_METHOD='parallel' 11 | DEFAULT_FIOSERVICE_MODE='local' 12 | CLUSTER_CMD='' 13 | SEED_METHOD='' 14 | FIOSERVICE_MODE='' 15 | FIOSERVICE_PORT=8080 16 | WORKER_YAML='fio.yaml' 17 | WORKERS=0 18 | STORAGECLASS=() # array of storageclasses to use for workers 19 | NAMESPACE="" 20 | WORKER_LIMIT=20 21 | SCRIPT_NAME=$(basename $0) 22 | ITERATION_LIMIT=60 # 5 min pod running timeout 23 | ITERATION_DELAY=5 24 | LOCKFILE='./fiodeploy.lock' 25 | POD_READY_LIMIT=120 26 | DELETE_TIMEOUT=300 27 | 28 | NC='\033[0m' # No Color 29 | INFO='\033[0;32m' # green 30 | OK='\033[0;32m' # green 31 | ERROR='\033[0;31m' # red 32 | WARNING='\033[1;33m' # yellow 33 | TICK='\xE2\x9C\x94' # tickmark 34 | CROSS='x' 35 | 36 | trap cleanup INT 37 | 38 | exists () { 39 | which $1 > /dev/null 2>&1 40 | } 41 | 42 | console () { 43 | echo -e "${1}${2}${NC}" 44 | } 45 | 46 | 47 | console_reset() { 48 | if exists "tput"; then 49 | tput init 50 | fi 51 | } 52 | 53 | cleanup() { 54 | # just reset the terminal for now, to avoid any color issues 55 | echo -e "\nBreak..." 56 | console_reset 57 | exit 1 58 | } 59 | 60 | 61 | check_prereq () { 62 | console "\nChecking oc/kubectl CLI is available" 63 | if exists "oc"; then 64 | CLUSTER_CMD='oc' 65 | PLATFORM='Openshift' 66 | console ${OK} "${TICK} oc command available${NC}" 67 | else 68 | if exists "kubectl"; then 69 | CLUSTER_CMD='kubectl' 70 | PLATFORM='Kubernetes' 71 | console ${OK} "${TICK} kubectl command available ${NC}" 72 | else 73 | console ${ERROR} "${CROSS} oc or kubectl commands not found. Unable to continue" 74 | exit 75 | fi 76 | fi 77 | } 78 | 79 | prompt () { 80 | local choices=$2 81 | while true; do 82 | # read -p $(echo -e ${INFO})"$1? "$(echo -e ${NC}) answer 83 | read -p "$1? " answer 84 | if [ -z "$answer" ]; then 85 | continue 86 | fi 87 | 88 | if [[ "${choices}" == *"${answer}"* ]]; then 89 | break 90 | fi 91 | done 92 | echo $answer 93 | } 94 | 95 | check_port() { 96 | console "\nChecking port ${FIOSERVICE_PORT} is free" 97 | cmd=$(lsof -Pi :${FIOSERVICE_PORT} -sTCP:LISTEN -t) 98 | if [ $? -eq 0 ]; then 99 | console ${ERROR} "${CROSS} port ${FIOSERVICE_PORT} is in use${NC}" 100 | exit 101 | fi 102 | console ${OK} "${TICK} port ${FIOSERVICE_PORT} is available${NC}" 103 | } 104 | 105 | check_ready() { 106 | console "\nChecking you are logged in to ${PLATFORM} with kubeadmin" 107 | local login_state 108 | if [ "$CLUSTER_CMD" == "oc" ]; then 109 | login_state=$($CLUSTER_CMD whoami 2>&1) 110 | if [ $? -eq 0 ]; then 111 | if [ "$login_state" = "kube:admin" ]; then 112 | console ${OK} "${TICK} ${PLATFORM} access OK ${NC}" 113 | else 114 | console ${ERROR} "${CROSS} you're logged in, but not with kubeadmin?${NC}" 115 | exit 116 | fi 117 | else 118 | console ${ERROR} "${CROSS} you're not logged in. Login to ${PLATFORM} with the kubeadmin account" 119 | exit 120 | fi 121 | else 122 | console ${WARNING} "? unable to check login state with kubectl" 123 | fi 124 | } 125 | 126 | acquire_lock() { 127 | if [ -f "${LOCKFILE}" ]; then 128 | console ${ERROR} "'fiodeploy.lock' file found. Is the script still running? ${NC}" 129 | exit 130 | else 131 | touch ${LOCKFILE} 132 | fi 133 | } 134 | release_lock() { 135 | rm -f ${LOCKFILE} 2>/dev/null 136 | } 137 | 138 | get_environment() { 139 | console "\nFIOLoadgen will use a create a new namespace to support the test environment" 140 | # read -p $(echo -e ${INFO})"What namespace should be used [${DEFAULT_NAMESPACE}]? "$(echo -e ${NC}) NAMESPACE 141 | read -p "What namespace should be used [${DEFAULT_NAMESPACE}]? " NAMESPACE 142 | if [ -z "$NAMESPACE" ]; then 143 | NAMESPACE=$DEFAULT_NAMESPACE 144 | 145 | fi 146 | console ${OK} "- checking '${NAMESPACE}' namespace is available" 147 | check_ns=$($CLUSTER_CMD get ns $NAMESPACE> /dev/null 2>&1) 148 | if [ $? = 0 ]; then 149 | overwrite "${ERROR}${CROSS} namespace $NAMESPACE already exists. Unable to continue${NC}" 150 | exit 151 | else 152 | overwrite "${OK}${TICK} namespace '$NAMESPACE' is available${NC}\n" 153 | fi 154 | 155 | console "Checking available storageclasses" 156 | sc_names=$($CLUSTER_CMD get sc -o jsonpath='{.items[*].metadata.name}') 157 | sc_array=( $sc_names ) 158 | 159 | for sc_name in ${sc_array[@]}; do 160 | console "- ${sc_name}" 161 | done 162 | 163 | sfx="" 164 | console "You may select multiple storageclasses. Press to end your selection." 165 | while :; do 166 | # read -p $(echo -e ${INFO})"What storageclass should the fio worker pods use [$DEFAULT_SC]? "$(echo -e ${NC}) STORAGECLASS 167 | read -p "Storageclass name: " sc 168 | if [ -z $sc ]; then 169 | break 170 | fi 171 | 172 | if [[ " ${STORAGECLASS[@]} " =~ " ${sc} " ]]; then 173 | console ${WARNING} "- storageclass '${sc}' already selected" 174 | continue 175 | fi 176 | 177 | if [[ ! " ${sc_array[@]} " =~ " ${sc} " ]]; then 178 | console ${ERROR} "${CROSS} storageclass '${sc}' does not exist." 179 | continue 180 | fi 181 | 182 | STORAGECLASS+=( "${sc}" ) 183 | # repeat=$(prompt "Do you want to use another storageclass (y/n)" "y n") 184 | done 185 | 186 | case ${#STORAGECLASS[@]} in 187 | 0) 188 | console ${ERROR} "${CROSS} No storagaeclass provided. Unable to continue" 189 | exit 190 | ;; 191 | 1) 192 | sfx="" 193 | ;; 194 | *) 195 | sfx="es" 196 | ;; 197 | esac 198 | 199 | console ${OK} "${TICK} workers will be deployed to ${#STORAGECLASS[@]} storageclass${sfx}${NC}\n" 200 | # read -p $(echo -e ${INFO})"How many fio workers (1-${WORKER_LIMIT}) [${DEFAULT_WORKERS}]? "$(echo -e ${NC}) WORKERS 201 | read -p "How many fio workers (1-${WORKER_LIMIT}) per storageclass [${DEFAULT_WORKERS}]? " WORKERS 202 | if [ -z "$WORKERS" ]; then 203 | WORKERS=$DEFAULT_WORKERS 204 | else 205 | if [ $WORKERS -eq $WORKERS 2>/dev/null ]; then 206 | # is numeric 207 | if [[ $WORKERS -lt 1 || $WORKERS -gt $WORKER_LIMIT ]]; then 208 | console ${ERROR} "Worker count must be within the range 1-${WORKER_LIMIT}" 209 | exit 210 | fi 211 | else 212 | console ${ERROR} "Invalid input for workers. Must be an integer" 213 | exit 214 | fi 215 | fi 216 | 217 | console ${OK} "${TICK} ${WORKERS} worker pods will be deployed to each required storageclass${NC}" 218 | 219 | # if [ -z "$STORAGECLASS" ]; then 220 | # WORKER_YAML='fio_no_pvc.yaml' 221 | # else 222 | # sed "s/{STORAGECLASS}/${STORAGECLASS}/g" ./yaml/fio.yaml > ./yaml/fioworker.yaml 223 | # WORKER_YAML='fioworker.yaml' 224 | # fi 225 | 226 | console "\nTo manage the tests, FIOLoadgen can use either a local daemon on your machine (local), or" 227 | console "deploy the management daemon to the target environment (remote)" 228 | while [ -z "$FIOSERVICE_MODE" ]; do 229 | # read -p $(echo -e ${INFO})"How do you want to manage the tests (local/remote) [local]? "$(echo -e ${NC}) FIOSERVICE_MODE 230 | read -p "How do you want to manage the tests (local/remote) [local]? " FIOSERVICE_MODE 231 | if [ -z "$FIOSERVICE_MODE" ]; then 232 | FIOSERVICE_MODE=$DEFAULT_FIOSERVICE_MODE 233 | fi 234 | 235 | case "${FIOSERVICE_MODE,,}" in 236 | local) 237 | FIOSERVICE_MODE='local' 238 | ;; 239 | remote) 240 | FIOSERVICE_MODE='remote' 241 | ;; 242 | *) 243 | echo "Unknown response, please try again - local or remote?" 244 | FIOSERVICE_MODE='' 245 | esac 246 | done 247 | console ${OK} "${TICK} fioservice will run in '${FIOSERVICE_MODE}' mode" 248 | 249 | # while [ -z "$SEED_METHOD" ]; do 250 | # read -p $(echo -e ${INFO})"Seed I/O test files in parallel or serial mode [$DEFAULT_SEED_METHOD] ? "$(echo -e ${NC}) SEED_METHOD 251 | # if [ -z "$SEED_METHOD" ]; then 252 | # SEED_METHOD=$DEFAULT_SEED_METHOD 253 | # fi 254 | 255 | # case "${SEED_METHOD,,}" in 256 | # s|serial) 257 | # SEED_METHOD='serial' 258 | # ;; 259 | # p|parallel) 260 | # SEED_METHOD='parallel' 261 | # ;; 262 | # *) 263 | # echo "Unknown response, please try again - parallel or serial (you can shorten this to p or s)" 264 | # SEED_METHOD='' 265 | # esac 266 | # done 267 | 268 | } 269 | 270 | # check_fio_complete() { 271 | # $CLUSTER_CMD -n $NAMESPACE exec fiomgr -- pidof fio 272 | # if [ $? -eq 0 ]; then 273 | # console ${ERROR} "fio tasks still running on the fiomgr pod. These will block/delay workload testing" 274 | # console ${ERROR} "Please login to fiomgr to investigate and check for cluster for errors that could prevent I/O" 275 | # fi 276 | # } 277 | 278 | overwrite() { 279 | echo -e "\r\033[1A\033[0K$@" 280 | } 281 | 282 | wait_for_pod() { 283 | local pod_name=$1 284 | console ${INFO} "- waiting for pod/$pod_name to reach ready status" 285 | local t=1 286 | while [[ $($CLUSTER_CMD get -n $NAMESPACE pod $pod_name -o=jsonpath='{..status.conditions[?(@.type=="Ready")].status}') != "True" ]]; do 287 | sleep 1 288 | t=$((t+1)) 289 | if [[ $t -gt $POD_READY_LIMIT ]]; then 290 | overwrite "${ERROR}x Time out waiting for the pod/$pod_name to reach ready${NC}" 291 | exit 292 | fi 293 | done 294 | overwrite "${OK}${TICK} pod/${pod_name} is ready${NC}" 295 | } 296 | 297 | get_port_fwd_pid() { 298 | echo $(ps -ef | awk '/[p]ort-forward fioservice/ { print $2;}') 299 | } 300 | 301 | setup_port_fwd() { 302 | pod_name=$($CLUSTER_CMD -n $NAMESPACE get pod -l app=fioservice -o jsonpath='{.items[0].metadata.name}') 303 | if [ "$1" == "wait" ]; then 304 | wait_for_pod $pod_name 305 | fi 306 | 307 | $CLUSTER_CMD -n $NAMESPACE port-forward $pod_name 8080:${FIOSERVICE_PORT} > /dev/null 2>&1 & 308 | if [ $? -gt 0 ]; then 309 | console ${ERROR} "${CROSS} failed to add the port-forward to pod ${pod_name}${NC}" 310 | exit 311 | else 312 | # get pid of port-forward 313 | pid=$(get_port_fwd_pid) 314 | console ${OK} "${TICK} port-forward created successfully (pid=${pid})${NC}" 315 | fi 316 | return 0 317 | } 318 | 319 | setup() { 320 | 321 | console "\nDeployment Summary" 322 | console "\n\tNamespace : ${NAMESPACE}" 323 | console "\tStorageclass :" 324 | for sc in ${STORAGECLASS[@]}; do 325 | console "\t - ${sc}" 326 | done 327 | console "\tFIO Workers : $WORKERS" 328 | console "\tFIO service : ${FIOSERVICE_MODE}\n" 329 | ready=$(prompt "Ready to deploy (y/n) " "y n Y N yes no") 330 | case $ready in 331 | [nN] | no) 332 | console ${ERROR} "Aborted${NC}" 333 | exit 334 | esac 335 | 336 | acquire_lock 337 | 338 | console ${INFO} "\nStarting deployment\n" 339 | 340 | # create namespace 341 | console "Creating namespace (${NAMESPACE})" 342 | ns=$($CLUSTER_CMD create namespace ${NAMESPACE}) 343 | if [ $? != 0 ]; then 344 | console ${ERROR} "${CROSS} Unable to create namespace...cannot continue${NC}" 345 | exit 346 | else 347 | console ${OK} "${TICK} namespace created OK${NC}" 348 | fi 349 | 350 | # deploy the worker pods across the requested Storageclass(es) 351 | console "\nDeploying the FIO worker statefulset(s)" 352 | total_workers=0 353 | for sc_name in ${STORAGECLASS[@]}; do 354 | statefulset=$(cat yaml/fioworker_statefulset_template.yaml | \ 355 | sed "s/!WORKERS!/${WORKERS}/" | \ 356 | sed "s/!STORAGECLASS!/${sc_name}/g" | \ 357 | $CLUSTER_CMD -n ${NAMESPACE} create -f - 2>&1) 358 | # $CLUSTER_CMD -n ${NAMESPACE} create -f yaml/fioworker_statefulset.yaml 359 | if [ $? -ne 0 ]; then 360 | console ${ERROR} "${CROSS} Failed to deploy stateful set for storageclass ${sc_name} : '${statefulset}'${NC}" 361 | exit 362 | else 363 | total_workers=$((total_workers+WORKERS)) 364 | console ${OK} "${TICK} ${statefulset}${NC}" 365 | fi 366 | done 367 | 368 | console "\nWaiting for worker pods to reach ready state\n" 369 | t=1 370 | while [ $t -lt $ITERATION_LIMIT ]; do # get pod -o=jsonpath='{..status.conditions[?(@.type=="Ready")].status}' 371 | status=$($CLUSTER_CMD -n ${NAMESPACE} get pod -l app=fioloadgen -o=jsonpath='{..status.conditions[?(@.type=="Ready")].status}') 372 | state_list=( $status ) # convert to array 373 | ready=$(grep -o True <<< ${state_list[*]} | wc -l) 374 | # status=$($CLUSTER_CMD -n ${NAMESPACE} get statefulset fioworker -o jsonpath='{..status.readyReplicas}') 375 | 376 | msg="${INFO}- ${ready}/${total_workers} PODs ready ... (check ${t}/${ITERATION_LIMIT})" 377 | overwrite $msg 378 | if [ $ready -lt $total_workers ]; then 379 | sleep $ITERATION_DELAY 380 | t=$((t+1)) 381 | else 382 | break 383 | fi 384 | done 385 | 386 | if [ $t -ne $ITERATION_LIMIT ]; then 387 | overwrite "${OK}${TICK} All worker pods ready${NC}" 388 | else 389 | console ${ERROR} "\nTimed out waiting too long for worker pods to reach ready state, unable to continue." 390 | for sc in ${STORAGECLASS[@]}; do 391 | console ${ERROR} "Statefulset : fioworker-${sc}" 392 | console ${ERROR} "Storageclass: ${sc}\n" 393 | done 394 | console ${ERROR} "Pods Ready : ${ready}/${total_workers}" 395 | exit 396 | fi 397 | 398 | declare -A lookup 399 | if [ "$FIOSERVICE_MODE" = "local" ]; then 400 | console "\nDeploying the mgr" 401 | fio_mgr_out=$($CLUSTER_CMD -n $NAMESPACE create -f yaml/fiomgr.yaml) 402 | if [ $? == 0 ]; then 403 | console ${OK} "${TICK} ${fio_mgr_out}${NC}" 404 | else 405 | console ${ERROR} "${CROSS} Failed to create the mgr pod ${NC}" 406 | exit 407 | fi 408 | 409 | wait_for_pod 'fiomgr' 410 | 411 | console "\nRemoving residual IP information from local data directory" 412 | rm -fr ./data/*-ip.list 413 | 414 | console "\nFetching the IP information of the worker pods (pod and host)" 415 | for sc in ${STORAGECLASS[@]}; do 416 | 417 | pod_names=$($CLUSTER_CMD -n ${NAMESPACE} get pod -l app=fioloadgen -l storageclass=${sc} -o jsonpath='{.items[*].metadata.name}') 418 | for pod_name in ${pod_names}; do 419 | # podIP_set="NO" 420 | 421 | ip_info=$($CLUSTER_CMD -n ${NAMESPACE} get pod ${pod_name} -o=jsonpath='{.status.podIP} {.status.hostIP}') 422 | podIP=$(echo ${ip_info} | cut -f1 -d ' ') 423 | hostIP=$(echo ${ip_info} | cut -f2 -d ' ') 424 | # if [ "$podIP" != "$hostIP" ]; then 425 | # break 426 | # else 427 | # console ${ERROR} "Unable to retrieve IP information for ${pod_name}" 428 | # exit 429 | # fi 430 | 431 | if [ ${lookup[${hostIP}]+_} ]; then 432 | # add to the entry 433 | ((lookup[${hostIP}]++)) # increment the map 434 | else 435 | # add to the map 436 | lookup[${hostIP}]=1 437 | fi 438 | echo -e "$hostIP" >> ./data/host-ip.list 439 | echo -e "$podIP" >> ./data/${sc}_worker-ip.list 440 | console ${OK} "${TICK} ${pod_name} on ${hostIP} with POD IP ${podIP}${NC}" 441 | done 442 | done 443 | 444 | # transfer the client ip addresses and fio jobs to the mgr 445 | console "\nTransfering worker IP addresses (pod and host), plus fio job specs to the fiomgr pod" 446 | for sc in ${STORAGECLASS[@]}; do 447 | o=$($CLUSTER_CMD -n $NAMESPACE cp data/${sc}_worker-ip.list fiomgr:/) 448 | if [ $? == 0 ]; then 449 | console ${OK} "${TICK} IPs for workers in storageclass ${sc} transferred" 450 | else 451 | console ${ERROR} "${CROSS} transfer of IP inforation for workers using storageclass ${sc} failed" 452 | exit 453 | fi 454 | done 455 | 456 | jobs_out=$($CLUSTER_CMD -n $NAMESPACE cp data/fio fiomgr:/) 457 | if [ $? == 0 ]; then 458 | console ${OK} "${TICK} jobs transfer complete${NC}" 459 | else 460 | console ${ERROR} "${CROSS} CP to the fiomgr pod failed. Unable to continue${NC}" 461 | exit 462 | fi 463 | 464 | console "\nStarting a local instance of the FIOservice daemon (API and UI)\n" 465 | python3 ./fioservice --mode=dev start --namespace ${NAMESPACE} 466 | if [ $? -ne 0 ]; then 467 | console ${ERROR} "${CROSS} failed to start the local fioservice daemon" 468 | exit 469 | fi 470 | console ${OK} "${TICK} fioservice started${NC}" 471 | console ${OK} "\nAccess the UI at http://localhost:8080. From there you may submit jobs and view" 472 | console ${OK} "job output and graphs.\n" 473 | console ${OK} "Daemon logs can be found in $HOME/fioseervice.log\n" 474 | console ${OK} "To stop the daemon, use the './fioservice stop' command\n" 475 | else 476 | # Use remote fioservice deployed to the target cluster 477 | console "\nSubmitting the deployment for the fioservice daemon" 478 | # TODO update the yaml to provide NAMESPACE and ENVIRONMENT 479 | deployment=$(cat yaml/fioservice_template.yaml | \ 480 | sed "s/!NAMESPACE!/${NAMESPACE}/" | \ 481 | sed "s/!ENVIRONMENT!/${PLATFORM}/g" | \ 482 | $CLUSTER_CMD -n ${NAMESPACE} create -f - 2>&1) 483 | 484 | # deployment=$($CLUSTER_CMD -n $NAMESPACE create -f yaml/fioservice.yaml) 485 | if [ $? -gt 0 ]; then 486 | console ${ERROR} "${CROSS} deployment failed, unable to continue${NC}" 487 | exit 488 | else 489 | console ${OK} "${TICK} deployment created${NC}" 490 | fi 491 | 492 | # pod_name=$($CLUSTER_CMD -n $NAMESPACE get pod -l app=fioservice -o jsonpath='{.items[0].metadata.name}') 493 | # status=$($CLUSTER_CMD -n $NAMESPACE get pod $pod_name -o=jsonpath='{..status.conditions[?(@.type=="Ready")].status}') 494 | # console ${INFO} "Waiting for fioservice to reach Ready state" 495 | # wait_for_pod $pod_name 496 | # t=1 497 | # while [ $t -lt $ITERATION_LIMIT ]; do 498 | # status=$($CLUSTER_CMD -n $NAMESPACE get pod $pod_name -o=jsonpath='{..status.conditions[?(@.type=="Ready")].status}') 499 | # if [ "$status" != "True" ]; then 500 | # console ${INFO} "\t - waiting (${t}/${ITERATION_LIMIT})" 501 | # sleep $ITERATION_DELAY 502 | # t=$((t+1)) 503 | # else 504 | # break 505 | # fi 506 | # done 507 | # if [ $t -ne $ITERATION_LIMIT ]; then 508 | # else 509 | # console ${ERROR} "Timed out Waiting too long for fioservice to reach ready, unable to continue." 510 | # console ${ERROR} "Deployment : fioservice" 511 | # console ${ERROR} "Pod : ${pod_name}" 512 | # exit 513 | # fi 514 | console "\nAdding port-forward rule (port ${FIOSERVICE_PORT})" 515 | setup_port_fwd "wait" 516 | 517 | console ${INFO} "\nAccess the UI at http://localhost:8080. From there you may submit jobs and view" 518 | console ${INFO} "job output and graphs" 519 | console ${INFO} "\nTo remove the port-forward, use kill -9 ${pid}\n" 520 | fi 521 | 522 | console ${INFO} "To drop the test environment, use the './fiodeploy -d ' command" 523 | 524 | # # seed the test files 525 | # console ${INFO} "Seeding the test files on the workers (mode=$SEED_METHOD), please wait" 526 | # if [ "$SEED_METHOD" == 'parallel' ]; then 527 | # console ${INFO} "- seeding $WORKERS pods in parallel" 528 | # $CLUSTER_CMD -n $NAMESPACE exec fiomgr -- fio --client=worker-ip.list fio/jobs/randr.job --output=seed.output 529 | # if [ $? != 0 ]; then 530 | # console ${ERROR} " failed to seed the test files on the workers" 531 | # console ${ERROR} "Deployment aborted" 532 | # exit 533 | # fi 534 | # else 535 | # for pod in $(cat ./data/worker-ip.list); do 536 | # console ${INFO} "- seeding $pod" 537 | # $CLUSTER_CMD -n $NAMESPACE exec fiomgr -- fio --client=$pod fio/jobs/randr.job --output=seed.output 538 | # if [ $? != 0 ]; then 539 | # console ${ERROR} " failed to seed the test file on $pod" 540 | # console ${ERROR} "Deployment aborted" 541 | # exit 542 | # fi 543 | # done 544 | # fi 545 | 546 | # check_fio_complete 547 | 548 | # echo -e "\n" 549 | # if [ ${#lookup[@]} -eq 1 ]; then 550 | # console ${WARNING} "${TICK} All workers are on a single host" 551 | # else 552 | # console ${INFO} "${TICK}${NC} ${WORKERS} worker pods running on ${#lookup[@]} hosts" 553 | # fi 554 | # console ${INFO} "${TICK}${NC} test files seeded, workers ready" 555 | # console ${INFO} "${TICK}${NC} ${WORKERS} worker pods ready" 556 | # console ${INFO} "${TICK}${NC} use rsh to login to the fiomgr pod to run a workload or use the fioservice and fiocli commands\n" 557 | 558 | release_lock 559 | } 560 | 561 | destroy() { 562 | echo 563 | ready=$(prompt "Are you sure you want to delete the '${NAMESPACE}' namespace and all it's pods/PVCs? (y/n) " "y n Y N yes no") 564 | case $ready in 565 | [nN] | no) 566 | console ${ERROR} "Aborted$NC" 567 | exit 568 | esac 569 | 570 | console "\nChecking for any port-forward PID to remove" 571 | pid=$(get_port_fwd_pid) 572 | if [ ! -z "$pid" ]; then 573 | kill -9 $pid 574 | if [ $? -gt 0 ]; then 575 | console ${ERROR} "Failed to remove the port-forward (pid=${pid})${NC}" 576 | exit 1 577 | else 578 | console ${OK} "${TICK} removed the port-forward process${NC}" 579 | fi 580 | else 581 | console ${OK} "${TICK} port-forward not present${NC}" 582 | fi 583 | 584 | console "\nChecking pods running in '$NAMESPACE' namespace are only fioloadgen pods" 585 | pod_names=$($CLUSTER_CMD -n $NAMESPACE get pod -o jsonpath='{.items[*].metadata.name}') 586 | pod_array=( $pod_names ) 587 | for pod_name in "${pod_array[@]}"; do 588 | if [[ $pod_name == fioservice* || $pod_name == fioworker* || $pod_name == fiomgr* ]]; then 589 | continue 590 | else 591 | console ${ERROR} "Namespace contains non fioloadgen pods (${pod_name}), unable to cleanup${NC}" 592 | exit 1 593 | fi 594 | done 595 | if [ ${#pod_array[@]} -eq 0 ]; then 596 | console ${OK} "${TICK} namespace is empty${NC}" 597 | else 598 | console ${OK} "${TICK} only fioloadgen related pods present${NC}" 599 | fi 600 | 601 | acquire_lock 602 | 603 | console "\nPlease wait while the '${NAMESPACE}' namespace is deleted (timeout @ ${DELETE_TIMEOUT}s)" 604 | cmd=$(timeout --foreground --kill-after $DELETE_TIMEOUT $DELETE_TIMEOUT $CLUSTER_CMD delete namespace ${NAMESPACE}) 605 | case "$?" in 606 | 0) 607 | console "${OK}${TICK} namespace successfully deleted${NC}" 608 | ;; 609 | 124) 610 | console "${ERROR}${CROSS} namespace delete request timed out${NC}" 611 | ;; 612 | *) 613 | console "${ERROR}${CROSS} namespace delete failed for '${NAMESPACE}' rc=$?${NC}" 614 | ;; 615 | esac 616 | 617 | release_lock 618 | } 619 | 620 | 621 | usage() { 622 | echo -e "Usage: ${SCRIPT_NAME} [-hwsdr]" 623 | echo -e "\t-h ... display usage information" 624 | echo -e "\t-s ... setup an FIOLoadgen test environment" 625 | echo -e "\t-d ... delete the FIOLoadgen namespace" 626 | echo -e "\t-p ... check/configure port-forward to remote fioservice pod" 627 | echo -e "\t-r ... reset (remove the lockfile)" 628 | 629 | echo "e.g." 630 | echo -e "> ./${SCRIPT_NAME} -s\n" 631 | } 632 | 633 | main() { 634 | 635 | args=1 636 | while getopts "hsp:d:r" option; do 637 | case "${option}" in 638 | h) 639 | usage 640 | exit 641 | ;; 642 | # w) 643 | # WORKER_YAML=${OPTARG} 644 | # if [ ! -f "yaml/${WORKER_YAML}" ]; then 645 | # console ${ERROR} "-w provided a file name that does not exist in the yaml directory" 646 | # exit 1 647 | # fi 648 | # ;; 649 | s) 650 | check_prereq 651 | check_port 652 | check_ready 653 | get_environment 654 | setup 655 | exit 656 | ;; 657 | p) 658 | check_prereq 659 | # check_port 660 | check_ready 661 | NAMESPACE=${OPTARG} 662 | echo -e "\nCheck/create port-forward to fioservice daemon in namespace ${NAMESPACE}" 663 | pid=$(get_port_fwd_pid) 664 | if [ -z "$pid" ]; then 665 | setup_port_fwd 666 | else 667 | console "${OK}${TICK} port forward already active using pid ${pid} ${NC}" 668 | fi 669 | exit 670 | ;; 671 | d) 672 | check_prereq 673 | check_ready 674 | NAMESPACE=${OPTARG} 675 | destroy 676 | exit 677 | ;; 678 | 679 | r) 680 | release_lock 681 | exit 682 | ;; 683 | \?) 684 | echo "Unsupported option." 685 | usage 686 | exit 687 | ;; 688 | esac 689 | args=0 690 | done 691 | shift "$((OPTIND-1))" 692 | if [[ $args ]]; then 693 | usage 694 | exit 695 | fi 696 | } 697 | 698 | main $@ 699 | -------------------------------------------------------------------------------- /fioservice: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | 3 | import os 4 | import sys 5 | # import time 6 | # import daemon 7 | # import daemon.pidfile 8 | import signal 9 | import argparse 10 | 11 | from fiotools import __version__ 12 | from fiotools.server import FIOWebService 13 | from fiotools.handlers import ( # NOQA: F401 14 | OpenshiftCMDHandler, 15 | KubernetesCMDHandler, 16 | SSHHandler, 17 | NativeFIOHandler, 18 | DebugHandler, 19 | ) 20 | 21 | from fiotools.utils import rfile, get_pid_file, port_in_use 22 | import fiotools.configuration as configuration 23 | 24 | # settings.init() 25 | import logging 26 | stream = logging.StreamHandler() 27 | logger = logging.getLogger() 28 | logger.addHandler(stream) 29 | 30 | DEFAULT_DIR = os.path.expanduser('~') 31 | DEFAULT_NAMESPACE = 'fio' 32 | 33 | 34 | def cmd_parser(): 35 | parser = argparse.ArgumentParser( 36 | description='Manage the fio web service daemon', 37 | formatter_class=argparse.ArgumentDefaultsHelpFormatter, 38 | ) 39 | 40 | parser.add_argument( 41 | '--version', 42 | action='store_true', 43 | default=False, 44 | help="Show fioloadgen version", 45 | ) 46 | parser.add_argument( 47 | '--mode', 48 | type=str, 49 | choices=['dev', 'prod', 'debug'], 50 | help="mode to run the service", 51 | ) 52 | 53 | subparsers = parser.add_subparsers(help="sub-command") 54 | 55 | parser_start = subparsers.add_parser( 56 | 'start', 57 | help="start the FIO web service") 58 | parser_start.set_defaults(func=command_start) 59 | parser_start.add_argument( 60 | '--type', 61 | required=False, 62 | choices=['oc', 'native', 'kubectl'], 63 | help="type of fioservice target", 64 | ) 65 | parser_start.add_argument( 66 | '--namespace', 67 | required=False, 68 | type=str, 69 | help="Namespace for Openshift based tests", 70 | ) 71 | # parser_start.add_argument( 72 | # '--debug', 73 | # action='store_true', 74 | # help="run standalone without a connection to help debug", 75 | # ) 76 | 77 | parser_stop = subparsers.add_parser( 78 | 'stop', 79 | help="stop the FIO service") 80 | parser_stop.set_defaults(func=command_stop) 81 | 82 | parser_stop = subparsers.add_parser( 83 | 'restart', 84 | help="restart the service") 85 | parser_stop.set_defaults(func=command_restart) 86 | parser_status = subparsers.add_parser( 87 | 'status', 88 | help="show current state of the FIO service") 89 | parser_status.set_defaults(func=command_status) 90 | 91 | return parser 92 | 93 | 94 | def pid_exists(pidfile): 95 | return os.path.exists(pidfile) 96 | 97 | 98 | def command_status(): 99 | pidfile = get_pid_file() 100 | 101 | if os.path.exists(pidfile): 102 | print("PID file : {}".format(pidfile)) 103 | pid = rfile(pidfile) 104 | print("PID : {}".format(pid)) 105 | if os.path.exists('/proc/{}'.format(pid)): 106 | state = "running" 107 | else: 108 | state = "not running" 109 | print("State : {}".format(state)) 110 | else: 111 | print("Not running") 112 | return 113 | 114 | 115 | def command_restart(): 116 | pidfile = get_pid_file() 117 | if pid_exists(pidfile): 118 | command_stop() 119 | command_start() 120 | else: 121 | print("service not running") 122 | 123 | 124 | def command_start(): 125 | if not os.path.isdir(DEFAULT_DIR): 126 | os.makedirs(DEFAULT_DIR) 127 | 128 | if args.mode in ['debug', 'dev']: 129 | logger.setLevel(logging.DEBUG) 130 | 131 | if os.path.exists(get_pid_file()): 132 | logger.error("Already running") 133 | exit(1) 134 | 135 | configuration.init(args) 136 | 137 | if args.mode == 'debug': 138 | logger.info("Using debug handler") 139 | handler = DebugHandler() 140 | elif configuration.settings.type == 'oc': 141 | logger.info("Using 'oc' command handler") 142 | handler = OpenshiftCMDHandler(mgr='fiomgr') 143 | elif configuration.settings.type == 'kubectl': 144 | logger.info("Using 'kubectl' command handler") 145 | handler = KubernetesCMDHandler(mgr='fiomgr') 146 | elif configuration.settings.type == 'native': 147 | logger.info("Using 'native' fio handler") 148 | handler = NativeFIOHandler() # ns=args.namespace) 149 | else: 150 | logger.error(f"'{configuration.settings.type}' handler has not been implemented yet") 151 | sys.exit(1) 152 | 153 | if not handler.has_connection: 154 | logger.error("Handler is not usable: environment or configuration problem") 155 | sys.exit(1) 156 | 157 | logger.info("Checking port {} is free".format(configuration.settings.port)) 158 | if port_in_use(configuration.settings.port): 159 | logger.error("-> port in use, unable to continue") 160 | sys.exit(1) 161 | 162 | server = FIOWebService(handler=handler) 163 | logger.info("Checking connection to {}".format(handler._target)) 164 | if server.ready or configuration.settings.mode == 'debug': 165 | logger.info("Starting the fioservice daemon") 166 | # Call the run handler to start the web service 167 | server.run() 168 | else: 169 | logger.error("{} connection unavailable, or workers not ready".format(handler._target)) 170 | logger.error("stopped") 171 | 172 | # NB. run() forks the daemon, so anything here is never reached 173 | 174 | 175 | def command_stop(): 176 | pidfile = get_pid_file() 177 | if not os.path.exists(pidfile): 178 | print("nothing to do") 179 | return 180 | 181 | with open(pidfile) as p: 182 | pid = p.read().strip() 183 | try: 184 | os.kill(int(pid), signal.SIGTERM) 185 | except Exception: 186 | print("kill for {} caused an exception") 187 | raise 188 | else: 189 | print("engine stopped") 190 | 191 | 192 | if __name__ == '__main__': 193 | parser = cmd_parser() 194 | args = parser.parse_args() 195 | if args.version: 196 | print('fioloadgen version: {}'.format(__version__)) 197 | elif 'func' in args: 198 | args.func() 199 | else: 200 | print("Unknown option(s) provided - try -h to show available options") 201 | -------------------------------------------------------------------------------- /fiotools/__init__.py: -------------------------------------------------------------------------------- 1 | __version__ = '1.5' 2 | -------------------------------------------------------------------------------- /fiotools/configuration.py: -------------------------------------------------------------------------------- 1 | import os 2 | import sys 3 | import shutil 4 | from typing import Optional 5 | 6 | from configparser import ConfigParser, ParsingError 7 | 8 | import logging 9 | logger = logging.getLogger(__name__) 10 | 11 | global settings 12 | 13 | cmd_lookup = { 14 | "oc": "Openshift", 15 | "kubectl": "kubernetes" 16 | } 17 | 18 | 19 | def init(args=None): 20 | global settings 21 | 22 | settings = Config(args) 23 | 24 | 25 | def convert_value(value): 26 | bool_types = { 27 | "TRUE": True, 28 | "FALSE": False, 29 | } 30 | 31 | if value.isdigit(): 32 | value = int(value) 33 | elif value.upper() in bool_types: 34 | value = bool_types[value.upper()] 35 | 36 | return value 37 | 38 | 39 | def cmd_handler() -> Optional[str]: 40 | if shutil.which('oc'): 41 | return 'oc' 42 | elif shutil.which('kubectl'): 43 | return 'kubectl' 44 | else: 45 | return None 46 | 47 | 48 | class Config(object): 49 | 50 | _config_dir_list = { 51 | "prod": [ 52 | "/etc/fioloadgen/fioservice.ini", 53 | os.path.join(os.path.expanduser('~'), 'fioservice.ini'), 54 | ], 55 | "dev": [ 56 | os.path.join(os.path.expanduser('~'), 'fioservice.ini'), 57 | ], 58 | "debug": [ 59 | os.path.join(os.path.expanduser('~'), 'fioservice.ini'), 60 | ] 61 | } 62 | 63 | _global_defaults = { 64 | "prod": { 65 | "db_name": "fioservice.db", 66 | "db_dir": "/var/lib/fioloadgen", 67 | "job_dir": "/var/lib/fioloadgen/jobs", 68 | "job_src": "/var/lib/fioloadgen/jobs", 69 | "log_dir": "/var/log/fioloadgen", 70 | "pid_dir": "/var/run/fioloadgen", 71 | "ssl": False, 72 | "ip_address": "0.0.0.0", 73 | "port": 8080, 74 | "debug": False, 75 | "runtime": "package", 76 | "namespace": "fio", 77 | "type": "native", 78 | "environment": "" 79 | }, 80 | "dev": { 81 | "db_name": "fioservice.db", 82 | "db_dir": os.path.expanduser('~'), 83 | "job_dir": os.path.join("fio", "jobs"), 84 | "job_src": os.path.join(os.getcwd(), "data", "fio", "jobs"), 85 | "log_dir": os.path.expanduser('~'), 86 | "pid_dir": os.path.expanduser('~'), 87 | "ssl": False, 88 | "ip_address": "0.0.0.0", 89 | "port": 8080, 90 | "debug": False, 91 | "runtime": "package", 92 | "namespace": "fio", 93 | "type": cmd_handler(), 94 | "environment": "" 95 | } 96 | } 97 | 98 | _global_defaults.update({"debug": _global_defaults['dev']}) 99 | 100 | _client_defaults = {} 101 | 102 | def __init__(self, args=None): 103 | 104 | if os.getenv('MODE'): 105 | logger.debug("setting mode from environment variable") 106 | # print("setting mode from environment variable") 107 | mode = os.getenv('MODE') 108 | elif args.mode: 109 | logger.debug("settings mode from args") 110 | # print("settings mode from args") 111 | mode = args.mode 112 | else: 113 | logger.debug("using default mode of dev") 114 | # print("using default mode of dev") 115 | mode = 'dev' 116 | 117 | # establish defaults based on the mode 118 | self.mode = mode 119 | self.db_name = Config._global_defaults[mode].get('db_name') 120 | self.db_dir = Config._global_defaults[mode].get('db_dir') 121 | self.log_dir = Config._global_defaults[mode].get('log_dir') 122 | self.pid_dir = Config._global_defaults[mode].get('pid_dir') 123 | self.ssl = Config._global_defaults[mode].get('ssl') 124 | self.port = Config._global_defaults[mode].get('port') 125 | # self.debug = Config._global_defaults[mode].get('debug') 126 | self.job_dir = Config._global_defaults[mode].get('job_dir') 127 | self.job_src = Config._global_defaults[mode].get('job_src') 128 | self.ip_address = Config._global_defaults[mode].get('ip_address') 129 | self.runtime = Config._global_defaults[mode].get('runtime') 130 | self.namespace = Config._global_defaults[mode].get('namespace') 131 | self.type = Config._global_defaults[mode].get('type') 132 | self.environment = cmd_lookup.get(self.type, "Unknown") 133 | 134 | self._apply_file_overrides() 135 | self._apply_env_vars() 136 | self._apply_args(args) 137 | 138 | logger.debug(str(self)) 139 | 140 | @property 141 | def dbpath(self): 142 | return os.path.join(self.db_dir, 'fioservice.db') 143 | 144 | def _apply_args(self, args): 145 | if not args: 146 | return 147 | 148 | for k in args.__dict__.keys(): 149 | if hasattr(self, k): 150 | v = getattr(args, k) 151 | if v is not None: 152 | logger.debug("Applying runtime override : {} = {}".format(k, v)) 153 | setattr(self, k, v) 154 | 155 | def _apply_env_vars(self): 156 | # we'll assume the env vars are all upper case by convention 157 | vars = [v.upper() for v in self.__dict__] 158 | for v in vars: 159 | env_setting = os.getenv(v) 160 | if env_setting: 161 | logger.debug("Using env setting {} = {}".format(v, convert_value(env_setting))) 162 | setattr(self, v.lower(), convert_value(env_setting)) 163 | 164 | def _apply_file_overrides(self): 165 | 166 | # define a list of valid vars 167 | valid_sections = ['global'] 168 | global_vars = set() 169 | global_vars.update(Config._global_defaults['prod'].keys()) 170 | global_vars.update(Config._global_defaults['dev'].keys()) 171 | 172 | # Parse the any config files that are accessible 173 | parser = ConfigParser() 174 | try: 175 | config = parser.read(Config._config_dir_list[self.mode]) 176 | except ParsingError: 177 | logger.error("invalid ini file format, unable to parse") 178 | sys.exit(12) 179 | 180 | if config: 181 | sections = parser.sections() 182 | if not sections or not all(s in valid_sections for s in sections): 183 | logger.error("config file has missing/unsupported sections") 184 | logger.error("valid sections are: {}".format(','.join(valid_sections))) 185 | sys.exit(12) 186 | 187 | # Apply the overrides 188 | for section_name in sections: 189 | if section_name == 'global': 190 | for name, value in parser.items(section_name): 191 | if name in global_vars: 192 | logger.debug("[CONFIG] applying override: {}={}".format(name, value)) 193 | setattr(self, name, convert_value(value)) 194 | else: 195 | logger.warning("-> {} is unsupported, ignoring") 196 | else: 197 | logger.debug("No configuration overrides from local files") 198 | 199 | def __str__(self): 200 | s = '' 201 | vars = self.__dict__ 202 | for k in vars: 203 | s += "{} = {}\n".format(k, vars[k]) 204 | return s 205 | -------------------------------------------------------------------------------- /fiotools/handlers/__init__.py: -------------------------------------------------------------------------------- 1 | from .kubernetes import OpenshiftCMDHandler, KubernetesCMDHandler # NOQA 2 | from .ssh import SSHHandler # NOQA 3 | from .local import NativeFIOHandler # NOQA 4 | from .debug import DebugHandler # NOQA 5 | -------------------------------------------------------------------------------- /fiotools/handlers/base.py: -------------------------------------------------------------------------------- 1 | import shutil 2 | 3 | 4 | class BaseHandler(object): 5 | 6 | _target = "Base" 7 | _cmd = 'missing' 8 | _connection_test = 'missing' 9 | 10 | @property 11 | def _can_run(self) -> bool: 12 | return shutil.which(self._cmd) is not None 13 | 14 | @property 15 | def has_connection(self) -> bool: 16 | return False 17 | 18 | def check(self) -> bool: 19 | """check the stored config against the target environment""" 20 | # can we pick up where we left off? 21 | return True 22 | 23 | def ls(self): 24 | """list the workers""" 25 | raise NotImplementedError 26 | 27 | def fetch(self): 28 | """fetch a file""" 29 | raise NotImplementedError 30 | 31 | def store(self): 32 | """send a file to an fio server target""" 33 | raise NotImplementedError 34 | 35 | def create(self): 36 | """Deploy an fio engine""" 37 | raise NotImplementedError 38 | 39 | def delete(self): 40 | """delete an fio server instance""" 41 | raise NotImplementedError 42 | 43 | def command(self): 44 | """Issue command on remote""" 45 | raise NotImplementedError 46 | 47 | def config(self): 48 | """fetch ceph config""" 49 | raise NotImplementedError 50 | 51 | def execute(self): 52 | """run fio workload""" 53 | raise NotImplementedError 54 | 55 | def reset(self): 56 | """reset the engines state to teardown the test environment""" 57 | raise NotImplementedError 58 | 59 | def fio_valid(self, fiojob): 60 | """check whether an fiojob 'deck' syntax is valid""" 61 | raise NotImplementedError 62 | 63 | def startfio(self, profile, workers, output): 64 | return None 65 | 66 | def fetch_report(self, output) -> int: 67 | return 0 68 | 69 | def copy_file(self, local_file, remote_file, namespace='fio', pod_name='fiomgr'): 70 | return 0 71 | 72 | def scale_workers(self, replica_count): 73 | raise NotImplementedError 74 | -------------------------------------------------------------------------------- /fiotools/handlers/debug.py: -------------------------------------------------------------------------------- 1 | from .base import BaseHandler 2 | 3 | 4 | class DebugHandler(BaseHandler): 5 | _target = "N/A" 6 | 7 | def __init__(self): 8 | self.workers = { 9 | 'MyStorageclass': 10 10 | } 11 | super().__init__() 12 | 13 | @property 14 | def _can_run(self) -> bool: 15 | return True 16 | 17 | @property 18 | def has_connection(self) -> bool: 19 | return True 20 | 21 | def fio_valid(self, fio_job) -> bool: 22 | return True 23 | -------------------------------------------------------------------------------- /fiotools/handlers/kubernetes.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | 3 | from .base import BaseHandler 4 | 5 | # import shutil 6 | import os 7 | import shutil 8 | import subprocess 9 | 10 | from fiotools import configuration 11 | from typing import Dict 12 | 13 | import logging 14 | logger = logging.getLogger(__name__) 15 | 16 | 17 | class OpenshiftCMDHandler(BaseHandler): 18 | 19 | _target = "Openshift" 20 | _cmd = 'oc' 21 | _connection_test = 'oc status' 22 | 23 | def __init__(self, mgr='fiomgr'): 24 | self.ns = configuration.settings.namespace 25 | self.mgr = mgr 26 | 27 | @property 28 | def usable(self) -> bool: 29 | return True 30 | 31 | @property 32 | def workers(self) -> Dict[str, int]: 33 | return self._get_workers() 34 | 35 | @property 36 | def _can_run(self) -> bool: 37 | return shutil.which(self._cmd) is not None 38 | 39 | @property 40 | def has_connection(self) -> bool: 41 | if self._can_run: 42 | r = subprocess.run(self._connection_test.split(' '), capture_output=True) 43 | return r.returncode == 0 44 | else: 45 | return False 46 | 47 | def _get_workers(self) -> Dict[str, int]: 48 | lookup = {} 49 | 50 | o = subprocess.run([ 51 | self._cmd, 52 | '-n', 53 | 'fio', 54 | 'get', 55 | 'pods', 56 | '--selector=app=fioloadgen', 57 | '-o=jsonpath="{range .items[*]}{.metadata.name}{\' \'}{.metadata.labels.storageclass}{\'\\n\'}{end}"'], 58 | capture_output=True) 59 | 60 | if o.returncode == 0: 61 | workers = o.stdout.decode('utf-8').strip('"').split('\n') 62 | for worker in workers: 63 | if worker: 64 | pod_name, storageclass = worker.split() 65 | if storageclass in lookup: 66 | lookup[storageclass] += 1 67 | else: 68 | lookup[storageclass] = 1 69 | return lookup 70 | 71 | def startfio(self, profile, storageclass, workers, output): 72 | cmd = 'startfio' 73 | args = f"-p {profile} -s {storageclass} -o {output} -w {workers}" 74 | cmd_result = subprocess.run([self._cmd, '-n', self.ns, 'exec', self.mgr, '--', cmd, args]) 75 | return cmd_result 76 | 77 | def fetch_report(self, output) -> int: 78 | source_file = os.path.join('/reports/', output) 79 | target_file = os.path.join('/tmp/', output) 80 | o = subprocess.run([self._cmd, 'cp', '{}/{}:{}'.format(self.ns, self.mgr, source_file), target_file]) 81 | # o = subprocess.run(['oc', '-n', self.ns, 'rsync', '{}:/reports/{}'.format(self.mgr, output), '/tmp/.']) 82 | return o.returncode 83 | 84 | def copy_file(self, local_file, remote_file, namespace='fio', pod_name='fiomgr') -> int: 85 | o = subprocess.run([self._cmd, 'cp', local_file, '{}/{}:{}'.format(self.ns, self.mgr, remote_file)]) 86 | return o.returncode 87 | 88 | def runcommand(self, command) -> None: 89 | pass 90 | 91 | def scale_workers(self, replica_count) -> int: 92 | raise NotImplementedError() 93 | # o = subprocess.run([self._cmd, '-n', self.ns, 'statefulsets', 'fioworker', '--replicas', replica_count]) 94 | # return o.returncode 95 | 96 | def fio_valid(self, fiojob) -> bool: 97 | # don't check, just assume it's valid 98 | return True 99 | 100 | 101 | class KubernetesCMDHandler(OpenshiftCMDHandler): 102 | _target = "Kubernetes" 103 | _cmd = 'kubectl' 104 | _connection_test = 'kubectl get ns' 105 | -------------------------------------------------------------------------------- /fiotools/handlers/local.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | import os 3 | import subprocess 4 | import shutil 5 | import tempfile 6 | from typing import Dict 7 | from .base import BaseHandler 8 | from fiotools import configuration 9 | import logging 10 | logger = logging.getLogger('cherrypy.error') 11 | 12 | try: 13 | from kubernetes import client, config 14 | from kubernetes.client.rest import ApiException 15 | from kubernetes.config.config_exception import ConfigException 16 | except ImportError: 17 | kubernetes_imported = False 18 | client = None 19 | config = None 20 | else: 21 | kubernetes_imported = True 22 | 23 | 24 | class NativeFIOHandler(BaseHandler): 25 | _target = "kubernetes" 26 | 27 | def __init__(self): # , namespace): 28 | self.namespace = configuration.settings.namespace 29 | self.job_dir = configuration.settings.job_dir 30 | self.reports = os.path.join(configuration.settings.db_dir, 'reports') 31 | self.replicaset = None 32 | self._target = configuration.settings.environment 33 | 34 | if kubernetes_imported: 35 | try: 36 | config.load_incluster_config() 37 | logger.debug("loaded k8s in cluster config") 38 | except ConfigException: 39 | logger.error("failed to load k8s configuration") 40 | self.k8s = None 41 | self.beta = None 42 | else: 43 | self.k8s = client.CoreV1Api() 44 | self.beta = client.ExtensionsV1beta1Api() 45 | logger.debug("CoreAPI and Extensions API configured and ready") 46 | # self.init() 47 | else: 48 | self.k8s = None 49 | self.beta = None 50 | 51 | super().__init__() 52 | # def init(self): 53 | # # determine the replicaset name for the workers 54 | # pass 55 | 56 | @property 57 | def _can_run(self): 58 | return kubernetes_imported is True 59 | 60 | @property 61 | def has_connection(self): 62 | return self.k8s is not None 63 | 64 | def fio_valid(self, fiojob): 65 | """run fio in parse-only mode to syntax check the job file""" 66 | 67 | tf = tempfile.NamedTemporaryFile(delete=False) 68 | with open(tf.name, 'w') as t: 69 | t.write(fiojob) 70 | 71 | result = subprocess.run(['fio', '--parse-only', tf.name]) 72 | # TODO if the rc is bad, log the problem to the logstream 73 | return True if result.returncode == 0 else False 74 | 75 | def _list_namespaced_pod(self, labels="app=fioloadgen"): 76 | try: 77 | response = self.k8s.list_namespaced_pod(self.namespace, label_selector=labels) 78 | except ApiException: 79 | return [] 80 | else: 81 | return response.items 82 | 83 | @property 84 | def workers(self) -> Dict[str, int]: 85 | lookup = {} 86 | pod_list = self._list_namespaced_pod() 87 | if pod_list: 88 | for pod in pod_list: 89 | sc = pod.metadata.labels.get('storageclass', None) 90 | if sc: 91 | if sc in lookup: 92 | lookup[sc] += 1 93 | else: 94 | lookup[sc] = 1 95 | return lookup 96 | 97 | # def num_workers(self): 98 | # """determine the number of workers""" 99 | # # get pods that have an app=fioworker set 100 | # return len(self._list_namespaced_pod().items) 101 | 102 | def fetch_pods(self, storageclass): 103 | try: 104 | pod_list = self._list_namespaced_pod( 105 | labels=f"app=fioloadgen,storageclass={storageclass}" 106 | ) 107 | except ApiException: 108 | return [] 109 | 110 | return pod_list 111 | 112 | # def whoknows(self): 113 | # pods = self.fetch_pods() 114 | # for pod in pods: 115 | # name = pod.metadata.name 116 | # host_ip = pod.status.host_ip 117 | # pod_ip = pod.status.pod_ip 118 | 119 | def startfio(self, profile, storageclass, workers, output): 120 | """start an fio run""" 121 | pods = self.fetch_pods(storageclass) 122 | if not pods: 123 | raise 124 | working_set = pods[0:workers] 125 | tf = tempfile.NamedTemporaryFile(delete=False) 126 | with open(tf.name, 'w') as t: 127 | for pod in working_set: 128 | logger.info(f"Job using client : {pod.status.pod_ip}") 129 | t.write("{}\n".format(pod.status.pod_ip)) 130 | 131 | fio_cmd = subprocess.run([ 132 | 'fio', 133 | '--client={}'.format(tf.name), 134 | os.path.join(self.job_dir, profile), 135 | '--output-format=json', 136 | '--output={}'.format(os.path.join(self.reports, output))] 137 | ) 138 | return fio_cmd 139 | 140 | def fetch_report(self, output): 141 | """ retrieve report""" 142 | report = os.path.join(self.reports, output) 143 | try: 144 | shutil.copyfile(report, os.path.join('/tmp', output)) 145 | except Exception: 146 | return 4 147 | else: 148 | return 0 149 | 150 | def copy_file(self, local_file, remote_file, namespace='fio', pod_name='fiomgr'): 151 | """copy file""" 152 | try: 153 | shutil.copy2(local_file, remote_file) 154 | except Exception: 155 | return 4 156 | else: 157 | return 0 158 | 159 | def scale_workers(self, new_worker_count): 160 | raise NotImplementedError() 161 | # # beta=client.ExtensionsV1beta1Api() 162 | # # d=beta.list_namespaced_deployment('fio', label_selector='app=fioworker') 163 | # patch = { 164 | # "spec": { 165 | # "replicas": new_worker_count, 166 | # }, 167 | # } 168 | # self.beta.patch_namespaced_deployment('fioworker', self.namespace, body=patch) 169 | -------------------------------------------------------------------------------- /fiotools/handlers/ssh.py: -------------------------------------------------------------------------------- 1 | from .base import BaseHandler 2 | 3 | 4 | class SSHHandler(BaseHandler): 5 | pass 6 | -------------------------------------------------------------------------------- /fiotools/reports/__init__.py: -------------------------------------------------------------------------------- 1 | from .latency import latency_summary # NOQA: F401 2 | -------------------------------------------------------------------------------- /fiotools/reports/latency.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | 3 | 4 | def latency_summary(fio_json, percentile=95): 5 | """determine latency summary stats from fio output (dict/json)""" 6 | 7 | def get_item(response, path): 8 | for item in path: 9 | response = response[item] 10 | return response 11 | 12 | def fmt_latency(latencies, num_clients): 13 | 14 | return "{:.2f}/{:.2f}/{:.2f}".format(min(latencies) / 1000000, 15 | (sum(latencies) / num_clients) / 1000000, 16 | max(latencies) / 1000000) 17 | 18 | vars_list = [ 19 | "read/iops", 20 | "read/clat_ns/percentile/{:.6f}".format(percentile), 21 | "write/iops", 22 | "write/clat_ns/percentile/{:.6f}".format(percentile), 23 | ] 24 | 25 | client_stats = fio_json['client_stats'] 26 | extract = list() 27 | for item in client_stats: 28 | if item['jobname'].lower() == 'all clients': 29 | continue 30 | hostname = item['hostname'] 31 | hostdata = { 32 | "hostname": hostname, 33 | } 34 | for v in vars_list: 35 | path = v.split('/') 36 | try: 37 | path_value = get_item(item, path) 38 | except KeyError: 39 | path_value = 0 40 | hostdata[v] = str(path_value) 41 | extract.append(hostdata) 42 | 43 | total_iops = 0 44 | read_latencies = list() 45 | write_latencies = list() 46 | num_clients = len(extract) 47 | for client in extract: 48 | total_iops += float(client['read/iops']) + float(client['write/iops']) 49 | read_latencies.append(float(client['read/clat_ns/percentile/{:.6f}'.format(percentile)])) 50 | write_latencies.append(float(client['write/clat_ns/percentile/{:.6f}'.format(percentile)])) 51 | 52 | summary = { 53 | "clients": num_clients, 54 | "total_iops": total_iops, 55 | "read ms min/avg/max": fmt_latency(read_latencies, num_clients), 56 | "write ms min/avg/max": fmt_latency(write_latencies, num_clients) 57 | } 58 | return summary 59 | -------------------------------------------------------------------------------- /fiotools/server/__init__.py: -------------------------------------------------------------------------------- 1 | from .web import FIOWebService # NOQA:F401 2 | -------------------------------------------------------------------------------- /fiotools/server/db.py: -------------------------------------------------------------------------------- 1 | import os 2 | import glob 3 | import sqlite3 4 | import datetime 5 | 6 | from ..utils import rfile 7 | from fiotools import configuration 8 | 9 | import logging 10 | logger = logging.getLogger("cherrypy.error") 11 | 12 | 13 | def setup_db(): 14 | 15 | profiles_table = """ CREATE TABLE IF NOT EXISTS profiles ( 16 | name text PRIMARY KEY, 17 | spec text, 18 | created integer, 19 | updated integer 20 | ); """ 21 | jobs_table = """ CREATE TABLE IF NOT EXISTS jobs ( 22 | id text PRIMARY KEY, 23 | title text NOT NULL, 24 | profile text NOT NULL, 25 | profile_spec text, 26 | storageclass text, 27 | workers integer NOT NULL, 28 | status text NOT NULL, 29 | started integer, 30 | ended integer, 31 | raw_json text, 32 | summary text, 33 | type text, 34 | raw_output text, 35 | provider text, 36 | platform text, 37 | FOREIGN KEY(profile) REFERENCES profiles (name) 38 | ); """ 39 | dbpath = configuration.settings.dbpath 40 | 41 | if not os.path.exists(dbpath): 42 | print("Creating results database @ {}".format(dbpath)) 43 | with sqlite3.connect(dbpath) as c: 44 | c.execute(profiles_table) 45 | c.execute(jobs_table) 46 | else: 47 | print("Using existing database @ {}".format(dbpath)) 48 | check_migration(dbpath) 49 | 50 | 51 | def check_migration(dbpath): 52 | def profile_spec(con): 53 | # with sqlite3.connect(dbpath) as con: 54 | cursor = con.cursor() 55 | jobs_table = cursor.execute('select * from jobs') 56 | fields = [desc[0] for desc in jobs_table.description] 57 | if 'profile_spec' not in fields: 58 | print("- updating the database: Adding profile_spec to jobs table") 59 | add_column = "ALTER TABLE jobs ADD COLUMN profile_spec text" 60 | cursor.execute(add_column) 61 | 62 | def storageclass(con): 63 | cursor = con.cursor() 64 | jobs_table = cursor.execute('select * from jobs') 65 | fields = [desc[0] for desc in jobs_table.description] 66 | if 'storageclass' not in fields: 67 | print("- updating the database: Adding storageclass to jobs table") 68 | add_column = "ALTER TABLE jobs ADD COLUMN storageclass text" 69 | cursor.execute(add_column) 70 | pass 71 | 72 | with sqlite3.connect(dbpath) as con: 73 | profile_spec(con) 74 | storageclass(con) 75 | 76 | 77 | def valid_fio_profile(profile_spec): 78 | # TODO validate the profile 79 | # is valid fio syntax 80 | # one workload, called workload 81 | # uses /mnt 82 | return True 83 | 84 | 85 | def message(msg_string, output='console'): 86 | if output == 'console': 87 | print(msg_string) 88 | else: 89 | logger.info(msg_string) 90 | 91 | 92 | # def load_db_profiles(jobdir, dbpath, out='console'): 93 | def load_db_profiles(out='console'): 94 | 95 | changes = { 96 | "processed": [], 97 | "new": [], 98 | "deleted": [], 99 | "changed": [], 100 | "skipped": [], 101 | "errors": [], 102 | } 103 | 104 | dbpath = os.path.join(configuration.settings.db_dir, 'fioservice.db') 105 | message("Refreshing job profiles, syncing the db versions with the local files in {}".format(configuration.settings.job_dir), out) 106 | 107 | profile_paths = glob.glob('{}/*'.format(configuration.settings.job_src)) 108 | fs_profile_names = [os.path.basename(p) for p in profile_paths] 109 | 110 | with sqlite3.connect(dbpath) as c: 111 | c.row_factory = sqlite3.Row 112 | cursor = c.cursor() 113 | for profile in profile_paths: 114 | name = os.path.basename(profile) 115 | changes['processed'].append(profile) 116 | profile_spec = rfile(profile) 117 | 118 | if not valid_fio_profile(profile_spec): 119 | changes['errors'].append(profile) 120 | message("profile '{}' is invalid, and can not be loaded".format(profile)) 121 | continue 122 | 123 | cursor.execute("SELECT * from profiles WHERE name=?;", 124 | (name,)) 125 | data = cursor.fetchone() 126 | if data is None: 127 | message("- loading profile {}".format(name), out) 128 | changes['new'].append(profile) 129 | now = int(datetime.datetime.now().strftime("%s")) 130 | cursor.execute("INSERT INTO profiles VALUES (?,?,?,?);", 131 | (name, profile_spec, now, now)) # NOQA 132 | else: 133 | # if spec is the same - just skip it 134 | if data['spec'] == profile_spec: 135 | message("- skipping identical profile {}".format(name), out) 136 | changes['skipped'].append(profile) 137 | else: 138 | # if not, apply the filesystem copy to the database 139 | message("- refreshing profile '{}' in the db with filesystem copy".format(name), out) 140 | changes['changed'].append(profile) 141 | cursor.execute(""" UPDATE profiles 142 | SET 143 | spec=?, 144 | updated=? 145 | WHERE 146 | name=?;""", 147 | (profile_spec, int(datetime.datetime.now().strftime("%s")), name)) 148 | 149 | stored_profiles = [p['name'] for p in fetch_all('profiles', list(['name']))] 150 | # message("profiles in db are: {}".format(','.join(stored_profiles))) 151 | for db_profile_name in stored_profiles: 152 | if db_profile_name not in fs_profile_names: 153 | message('- deleting db profile {}'.format(db_profile_name)) 154 | changes['deleted'].append(db_profile_name) 155 | # message("changes {}".format(json.dumps(changes))) 156 | cursor.execute(""" DELETE FROM profiles 157 | WHERE name=?;""", 158 | (db_profile_name,)) 159 | 160 | return changes 161 | 162 | 163 | def fetch_all(table, keys): 164 | dbpath = configuration.settings.dbpath 165 | assert isinstance(keys, list) 166 | if not keys: 167 | return list() 168 | 169 | data = list() 170 | with sqlite3.connect(dbpath) as c: 171 | c.row_factory = sqlite3.Row 172 | csr = c.cursor() 173 | fields = ",".join(keys) 174 | csr.execute(""" SELECT {} FROM {};""".format(fields, table)) 175 | 176 | rows = csr.fetchall() 177 | for row in rows: 178 | data.append({k: row[k] for k in keys}) 179 | 180 | return data 181 | 182 | 183 | def fetch_row(table, key=None, content=None): 184 | dbpath = configuration.settings.dbpath 185 | response = dict() 186 | if not key: 187 | return response 188 | available = [row[key] for row in fetch_all(table, list([key]))] 189 | if not content or content not in available: 190 | return response 191 | else: 192 | query = "SELECT * FROM {} WHERE {} = ?;".format(table, key) 193 | with sqlite3.connect(dbpath) as c: 194 | c.row_factory = sqlite3.Row 195 | csr = c.cursor() 196 | csr.execute(query, (content,)) 197 | row = csr.fetchall()[0] 198 | 199 | # convert the row object to a dict 200 | response = {k: row[k] for k in row.keys()} 201 | return response 202 | 203 | 204 | def delete_row(table=None, query=dict()): 205 | dbpath = configuration.settings.dbpath 206 | err = 0 207 | msg = '' 208 | 209 | if not query or len(query) > 1: 210 | logger.info("delete_row called with empty query, or too many parameters - ignoring") 211 | return 'Invalid or missing query' 212 | 213 | k = list(query)[0] 214 | sql = "DELETE FROM {} WHERE {}=?".format(table, k) 215 | 216 | with sqlite3.connect(dbpath) as c: 217 | csr = c.cursor() 218 | try: 219 | csr.execute(sql, (query[k],)) 220 | except sqlite3.Error as e: 221 | err = 1 222 | msg = f"delete_row failed: {str(e)}" 223 | else: 224 | if c.total_changes == 0: 225 | err = 1 226 | msg = "row not found" 227 | c.commit() 228 | 229 | return err, msg 230 | 231 | 232 | def update_job_status(job_uuid, status): 233 | dbpath = configuration.settings.dbpath 234 | with sqlite3.connect(dbpath) as c: 235 | csr = c.cursor() 236 | csr.execute(""" UPDATE jobs 237 | SET status = ?, 238 | started = ? 239 | WHERE 240 | id = ?;""", (status, 241 | int(datetime.datetime.now().strftime("%s")), 242 | job_uuid) 243 | ) 244 | 245 | 246 | def prune_db(): 247 | dbpath = configuration.settings.dbpath 248 | # remove records from the database that represent queued jobs 249 | # cherrypy.log("Pruning jobs still in a queued/started state from the database") 250 | prune_query = "DELETE FROM jobs WHERE status = 'queued' OR status = 'started';" 251 | with sqlite3.connect(dbpath) as c: 252 | csr = c.cursor() 253 | csr.execute(prune_query) 254 | 255 | 256 | def dump_table(table_name='jobs', 257 | query={}): 258 | """ simple iterator function to dump specific row(s) from the job table """ 259 | dbpath = configuration.settings.dbpath 260 | with sqlite3.connect(dbpath) as conn: 261 | csr = conn.cursor() 262 | yield('BEGIN TRANSACTION;') 263 | 264 | # sqlite_master table contains the SQL CREATE statements for the all the tables. 265 | q = """ 266 | SELECT type, sql 267 | FROM sqlite_master 268 | WHERE sql NOT NULL AND 269 | type == 'table' AND 270 | name == :table_name 271 | """ 272 | schema_res = csr.execute(q, {'table_name': table_name}) 273 | 274 | # create the create table syntax (ignore the type value..just using it preserve output) 275 | for _, sql in schema_res.fetchall(): 276 | # yield the create table syntax, if the request doesn't provide a query 277 | if not query: 278 | yield('{};'.format(sql)) 279 | 280 | # fetch the column names of the table 281 | res = csr.execute("PRAGMA table_info('{}')".format(table_name)) 282 | column_names = [str(table_info[1]) for table_info in res.fetchall()] 283 | 284 | # Create the Insert statements to repopulate the table 285 | q = "SELECT 'INSERT INTO \"{}\" VALUES(".format(table_name) 286 | q += ",".join(["'||quote(" + col + ")||'" for col in column_names]) 287 | q += ")' FROM '{}'".format(table_name) 288 | 289 | if query: 290 | # FIXME query assumes the value is a string...is this a problem? 291 | key_values = ["{}='{}'".format(k, query[k]) for k in query.keys()] 292 | q += " WHERE {}".format(','.join(key_values)) 293 | 294 | # Issue the query, then potentially filter to the desired row 295 | query_res = csr.execute(q % {'tbl_name': table_name}) 296 | for row in query_res: 297 | yield("%s;" % row[0]) 298 | 299 | yield('COMMIT;') 300 | 301 | 302 | def run_script(sql_script): 303 | dbpath = configuration.settings.dbpath 304 | err = '' 305 | with sqlite3.connect(dbpath) as conn: 306 | csr = conn.cursor() 307 | try: 308 | csr.executescript(sql_script) 309 | except sqlite3.Error as e: 310 | err = "SQL failure: {}".format(e) 311 | except Exception as e: 312 | err = "generic exception: {}".format(e) 313 | else: 314 | err = '' 315 | 316 | return err 317 | 318 | 319 | def add_profile(name, spec): 320 | 321 | err = 0 322 | msg = '' 323 | 324 | dbpath = os.path.join(configuration.settings.db_dir, 'fioservice.db') 325 | message(f"Adding/updating job profile: {name}", 'log') 326 | 327 | with sqlite3.connect(dbpath) as c: 328 | c.row_factory = sqlite3.Row 329 | cursor = c.cursor() 330 | now = int(datetime.datetime.now().strftime("%s")) 331 | try: 332 | cursor.execute("INSERT INTO profiles VALUES (?,?,?,?);", 333 | (name, spec, now, now)) 334 | message("upload of the profile successful") 335 | except sqlite3.IntegrityError: 336 | err = 1 337 | msg = "profile already exists, ignored" 338 | message(msg) 339 | 340 | return err, msg 341 | 342 | 343 | def delete_profile(profile_name): 344 | return delete_row(table='profiles', query={"name": profile_name}) 345 | -------------------------------------------------------------------------------- /fiotools/server/security.py: -------------------------------------------------------------------------------- 1 | import os 2 | from OpenSSL import crypto # , SSL 3 | 4 | 5 | def cert_gen( 6 | emailAddress="emailAddress", 7 | commonName="commonName", 8 | countryName="NZ", 9 | localityName="localityName", 10 | stateOrProvinceName="NorthIsland", 11 | organizationName="Red Hat", 12 | organizationUnitName="Engineering", 13 | serialNumber=0, 14 | validityStartInSeconds=0, 15 | validityEndInSeconds=10 * 365 * 24 * 60 * 60, 16 | KEY_FILE=os.path.join(os.path.expanduser('~'), 'selfsigned.key'), 17 | CERT_FILE=os.path.join(os.path.expanduser('~'), "selfsigned.crt")): 18 | # can look at generated file using openssl: 19 | # openssl x509 -inform pem -in selfsigned.crt -noout -text 20 | # create a key pair 21 | k = crypto.PKey() 22 | k.generate_key(crypto.TYPE_RSA, 4096) 23 | # create a self-signed cert 24 | cert = crypto.X509() 25 | cert.get_subject().C = countryName 26 | cert.get_subject().ST = stateOrProvinceName 27 | cert.get_subject().L = localityName 28 | cert.get_subject().O = organizationName # noqa 29 | cert.get_subject().OU = organizationUnitName 30 | cert.get_subject().CN = commonName 31 | cert.get_subject().emailAddress = emailAddress 32 | cert.set_serial_number(serialNumber) 33 | cert.gmtime_adj_notBefore(0) 34 | cert.gmtime_adj_notAfter(validityEndInSeconds) 35 | cert.set_issuer(cert.get_subject()) 36 | cert.set_pubkey(k) 37 | cert.sign(k, 'sha512') 38 | with open(CERT_FILE, "wt") as f: 39 | f.write(crypto.dump_certificate(crypto.FILETYPE_PEM, cert).decode("utf-8")) 40 | with open(KEY_FILE, "wt") as f: 41 | f.write(crypto.dump_privatekey(crypto.FILETYPE_PEM, k).decode("utf-8")) 42 | -------------------------------------------------------------------------------- /fiotools/utils/__init__.py: -------------------------------------------------------------------------------- 1 | from .utils import rfile, get_pid_file, port_in_use, generate_fio_profile # NOQA:F401 2 | -------------------------------------------------------------------------------- /fiotools/utils/utils.py: -------------------------------------------------------------------------------- 1 | import os 2 | import socket 3 | from typing import Dict, Any 4 | 5 | 6 | def rfile(file_path): 7 | with open(file_path, 'r') as f: 8 | data = f.read().strip() 9 | return data 10 | 11 | 12 | def get_pid_file(prefix=None): 13 | if not prefix: 14 | prefix = os.path.expanduser('~') 15 | return os.path.join(prefix, 'fioservice.pid') 16 | 17 | 18 | def port_in_use(port_num): 19 | """Detect whether a port is in use on the local machine - IPv4 only""" 20 | 21 | try: 22 | s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) 23 | s.bind(('0.0.0.0', port_num)) 24 | except OSError: 25 | return True 26 | else: 27 | return False 28 | 29 | 30 | def generate_fio_profile(spec: Dict[str, Any]) -> str: 31 | global_section = """ 32 | [global] 33 | refill_buffers 34 | size=5g 35 | directory=/mnt 36 | direct=1 37 | time_based=1 38 | ioengine=libaio 39 | group_reporting 40 | """ 41 | workload_section = f""" 42 | [workload] 43 | blocksize={spec['ioBlockSize']} 44 | runtime={spec['runTime']} 45 | iodepth={spec['ioDepth']} 46 | numjobs=1 47 | """ 48 | if spec['ioType'].lower() == 'random': 49 | if spec['ioPattern'] == 0: 50 | # 100% random read 51 | workload_section += "rw=randread\n" 52 | elif spec['ioPattern'] == 100: 53 | # 100% random write 54 | workload_section += "rw=randwrite\n" 55 | else: 56 | # mixed random 57 | workload_section += "rw=randrw\n" 58 | workload_section += f"rwmixwrite={spec['ioPattern']}\n" 59 | else: 60 | # sequential workloads 61 | if spec['ioPattern'] == 0: 62 | # 100% seqential read 63 | workload_section += "rw=read\n" 64 | elif spec['ioPattern'] == 100: 65 | # 100% seqential writes 66 | workload_section += "rw=write\n" 67 | else: 68 | # mixed sequential 69 | workload_section += "rw=readwrite\n" 70 | workload_section += f"rwmixwrite={spec['ioPattern']}\n" 71 | 72 | return global_section + workload_section 73 | -------------------------------------------------------------------------------- /media/fioloadgen-demo.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/pcuzner/fioloadgen/b9d7654ec57e9ab0513208f14263846141f6c9e4/media/fioloadgen-demo.gif -------------------------------------------------------------------------------- /media/fioloadgen_1.2_2.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/pcuzner/fioloadgen/b9d7654ec57e9ab0513208f14263846141f6c9e4/media/fioloadgen_1.2_2.gif -------------------------------------------------------------------------------- /media/fioloadgen_banner.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/pcuzner/fioloadgen/b9d7654ec57e9ab0513208f14263846141f6c9e4/media/fioloadgen_banner.png -------------------------------------------------------------------------------- /media/fioloadgen_jobs.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/pcuzner/fioloadgen/b9d7654ec57e9ab0513208f14263846141f6c9e4/media/fioloadgen_jobs.png -------------------------------------------------------------------------------- /media/fioloadgen_profiles.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/pcuzner/fioloadgen/b9d7654ec57e9ab0513208f14263846141f6c9e4/media/fioloadgen_profiles.png -------------------------------------------------------------------------------- /react/app/.env: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/pcuzner/fioloadgen/b9d7654ec57e9ab0513208f14263846141f6c9e4/react/app/.env -------------------------------------------------------------------------------- /react/app/.gitignore: -------------------------------------------------------------------------------- 1 | /node_modules 2 | -------------------------------------------------------------------------------- /react/app/package.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "app", 3 | "version": "1.0.0", 4 | "description": "", 5 | "main": "index.jsx", 6 | "scripts": { 7 | "start": "webpack-dev-server --config ./webpack.config.js --mode development", 8 | "build": "webpack", 9 | "clean": "rm -fr dist/*", 10 | "test": "echo 'no test defined' && exit 1" 11 | }, 12 | "keywords": [], 13 | "author": "paul cuzner", 14 | "license": "ISC", 15 | "babel": { 16 | "presets": [ 17 | "@babel/preset-env", 18 | "@babel/preset-react" 19 | ], 20 | "plugins": [ 21 | "@babel/plugin-proposal-class-properties" 22 | ] 23 | }, 24 | "devDependencies": { 25 | "@babel/core": "^7.13.10", 26 | "@babel/node": "^7.13.10", 27 | "@babel/plugin-proposal-class-properties": "^7.13.0", 28 | "@babel/preset-env": "^7.13.10", 29 | "@babel/preset-react": "^7.12.13", 30 | "babel-loader": "^8.2.2", 31 | "css-loader": "^3.6.0", 32 | "dotenv": "^8.2.0", 33 | "mini-css-extract-plugin": "^0.9.0", 34 | "node-sass": "^4.14.1", 35 | "nodemon": "^2.0.7", 36 | "sass-loader": "^8.0.2", 37 | "style-loader": "^1.3.0", 38 | "webpack": "^4.46.0", 39 | "webpack-cli": "^3.3.12", 40 | "webpack-dev-server": "^3.11.2" 41 | }, 42 | "dependencies": { 43 | "@patternfly/patternfly": "^2.71.7", 44 | "@patternfly/react-core": "^3.158.4", 45 | "chart.js": "^2.9.4", 46 | "chartjs-plugin-datalabels": "^0.7.0", 47 | "copy-webpack-plugin": "^5.1.2", 48 | "extract-text-webpack-plugin": "^3.0.2", 49 | "node-sass-chokidar": "^1.5.0", 50 | "react": "^16.14.0", 51 | "react-bootstrap": "^1.5.1", 52 | "react-chartjs-2": "^2.11.1", 53 | "react-dom": "^16.14.0" 54 | } 55 | } 56 | -------------------------------------------------------------------------------- /react/app/src/app.scss: -------------------------------------------------------------------------------- 1 | body { 2 | font-family: sans-serif; 3 | } 4 | 5 | h1 { 6 | font-weight: bold; 7 | font-size: 18px; 8 | padding-top:10px; 9 | margin: 0 0 0 0;} 10 | .profile-select select { 11 | border: 1px solid transparent; 12 | scrollbar-width: none; /*For Firefox*/; 13 | -ms-overflow-style: none; /*For Internet Explorer 10+*/; 14 | width: 220px; 15 | padding: 10px 0px 5px 0px; } 16 | .profile-select select::-webkit-scrollbar { /* for webkit browsers */ 17 | width: 0 ; 18 | height: 0; 19 | } 20 | .profile-select select:focus { 21 | outline: 0; /* remove the normal select box outline */ 22 | } 23 | .profile-select option { 24 | padding: 5px; 25 | border-bottom: 1px solid lightgray; } 26 | 27 | .profile-select option:hover { 28 | background-color: #ebf8ff; } 29 | .profile-select option:checked { 30 | background:#006998 !important; // linear-gradient(#a0a0a0, #B8B8B8); } 31 | } 32 | #profile-content-container { 33 | border: 1px solid lightgray; 34 | width: 490px; 35 | height: 510px; 36 | display:block; /*flex: 1; */ 37 | margin-top: 8px; 38 | } 39 | .italic { 40 | font-style:italic; 41 | } 42 | .job_table tbody tr.selectedRow:hover { 43 | background-color: #006998; color:white;} 44 | .job_table tbody tr:not(.selectedRow):hover { 45 | background-color: #ebf8ff; } 46 | .job_table { 47 | text-align:left; 48 | min-width: 1410px; } 49 | 50 | .job_table td,th{ 51 | padding: 5px; 52 | } 53 | .job_table tbody { 54 | width:100%; 55 | display: block; 56 | height:300px; 57 | overflow:auto; 58 | } 59 | .job_table thead tr { 60 | display: block; 61 | border-bottom: 2px solid black;} 62 | .job_table tr { 63 | border-bottom: 1px solid lightgray; 64 | } 65 | .job_table tfoot tr{ 66 | display: block; 67 | font-style: italic; 68 | border-bottom: none; 69 | } 70 | /* masthead menu items */ 71 | li.menu { 72 | display: block; 73 | font-size: 1.2em; 74 | color: white; 75 | padding-bottom: 7px; 76 | } 77 | 78 | li.menu:hover { 79 | border-bottom: 5px solid lightskyblue; 80 | text-decoration: none; 81 | cursor:pointer; 82 | } 83 | .menu { 84 | display: block; 85 | font-size: 1.2em; 86 | float: left; 87 | padding-left: 10px; 88 | padding-right: 10px; 89 | padding-top: 7px; 90 | } 91 | .menu_active { 92 | border-bottom: 5px solid lightskyblue; 93 | color: lightskyblue; 94 | } 95 | .menu_inactive { 96 | position:relative; 97 | color: white; 98 | border-bottom: 5px solid #333; 99 | } 100 | ul { 101 | list-style-type: none; 102 | margin: 0; 103 | padding: 0; 104 | overflow: hidden; 105 | background-color: #333; } 106 | /* masthead definitions */ 107 | #masthead { 108 | background-color: #333; 109 | color: white; 110 | } 111 | .page-heading { 112 | color: white; 113 | float:left; 114 | width: 25%; 115 | font-weight: bold; 116 | font-size: 24px; 117 | padding-left: 10px; 118 | padding-top:10px; } 119 | 120 | .page-heading::before { 121 | font-family: FontAwesome; /* using fontawesome 4 */ 122 | content: "\f0e4"; 123 | display: inline-block; 124 | padding-right: 5px; 125 | vertical-align: top; 126 | font-weight: 900; 127 | } 128 | .status-area { 129 | float:right; 130 | width:75%; 131 | height:44px; 132 | padding-top:20px; 133 | padding-right:20px 134 | } 135 | .status-spacing { 136 | padding-right:30px; 137 | } 138 | 139 | #jobs { 140 | position:absolute; 141 | } 142 | #profiles { 143 | position:absolute; 144 | max-height: 290px ; 145 | } 146 | .active { 147 | z-index: 1; 148 | background-color: white; 149 | width:100%; 150 | height:100%;} 151 | 152 | .inactive { 153 | z-index: -1; 154 | } 155 | 156 | /* Job table widths */ 157 | .job_selector { 158 | min-width: 40px; 159 | text-align: center; 160 | } 161 | .job_id { 162 | min-width:80px; 163 | } 164 | .job_title { 165 | min-width:465px; 166 | } 167 | .job_clients { 168 | min-width: 80px; 169 | text-align: center; 170 | } 171 | .job_storageclass { 172 | width: 200px; 173 | text-align:left; 174 | } 175 | .job_profile { 176 | min-width:150px; 177 | text-align: left; 178 | } 179 | .job_provider { 180 | min-width: 100px; 181 | } 182 | .job_platform { 183 | min-width: 100px; 184 | } 185 | .job_start { 186 | min-width:180px; 187 | } 188 | .job_status { 189 | min-width:100px; 190 | } 191 | .job-actions { 192 | min-width:40px; 193 | } 194 | .job-content pre { 195 | height:400px; 196 | border: 1px solid grey; 197 | margin-bottom: 20px; 198 | overflow-y:scroll; 199 | } 200 | 201 | .offset-right { 202 | margin-right:15px; 203 | margin-bottom:10px; 204 | } 205 | .inline-block { 206 | display: inline-block; 207 | } 208 | .display-block { 209 | display: block; 210 | } 211 | .display-none { 212 | display: none; 213 | } 214 | .align-right { 215 | text-align:right; 216 | } 217 | .align-center { 218 | text-align:center; 219 | } 220 | .float-left { 221 | float: left; 222 | } 223 | .float-right { 224 | float: right; 225 | } 226 | .float-none { 227 | float: none; 228 | } 229 | pre { 230 | border: 0px; background: transparent; 231 | } 232 | .profile-select { 233 | width: 220px; 234 | } 235 | .profile-info { 236 | background: transparent; 237 | // border: 1px solid lightgray; 238 | // width: 470px; 239 | // height: 510px; 240 | // display:block; /*flex: 1; */ 241 | // margin-top: 8px; 242 | } 243 | .profile-msg { 244 | padding-top:40px; 245 | text-align: center; 246 | font-size: 1.4em; 247 | } 248 | .profile-msg::before { 249 | font-family: FontAwesome; 250 | /* using fontawesome 4 */ 251 | content: "\f0d9"; 252 | position:relative; 253 | } 254 | .profile-reload { 255 | position:relative; 256 | top: 310px; 257 | } 258 | .profile-run { 259 | position: relative; 260 | left: 672px; 261 | top: 10px; 262 | } 263 | .profile-container { 264 | padding-left:10px; 265 | } 266 | .modal { 267 | position: fixed; 268 | top: 0; 269 | left: 0; 270 | width: 100%; 271 | height: 100%; 272 | background: rgba(0, 0, 0, 0.6); } 273 | 274 | .modal-main { 275 | position: relative; 276 | background: white; 277 | width: 70%; 278 | max-height: 700px; 279 | top: 50%; 280 | left: 0; 281 | margin: 0 auto; 282 | overflow-y: hidden; 283 | transform: translateY(-50%); } 284 | .modal-content-container { 285 | padding: 10px; 286 | 287 | } 288 | .modal-inner { 289 | margin: 0px 10px 0px 10px; 290 | overflow-y: auto; 291 | max-height: 480px; 292 | } 293 | .modal-title-bar { 294 | width: 100%; 295 | background-color: whitesmoke; 296 | height: 3em; 297 | margin-bottom: 20px;} 298 | .modal-title { 299 | font-size: 1.3em; 300 | color: #1C2833; 301 | padding-left: 10px; 302 | padding-top:5px; 303 | } 304 | .modal-close { 305 | float: right; 306 | margin-right: 20px; 307 | margin-top: 20px; 308 | margin-bottom: 20px; } 309 | .close-symbol:after { 310 | content: "\2716"; 311 | color: #1C2833;; 312 | margin-right:10px; 313 | font-size:1.8em; 314 | } 315 | .close-symbol:hover:after { 316 | cursor:pointer; 317 | color:black; 318 | } 319 | input[type=range].workers-slider { 320 | width: 200px; 321 | display: inline-block;} 322 | 323 | .workers-slider { 324 | position: relative; 325 | top: 6px; 326 | } 327 | 328 | textarea { 329 | padding:10px; 330 | } 331 | .selectedRow { 332 | background-color:#006998; 333 | color:white; 334 | } 335 | .notSelectedRow { 336 | background-color: transparent; 337 | } 338 | input[type=checkbox] { 339 | cursor: pointer; 340 | } 341 | #jobsContainer { 342 | padding:10px; 343 | // height: 45%; 344 | overflow-y:auto; 345 | } 346 | #jobTableArea { 347 | background-color: white; 348 | min-width: 1500px; 349 | } 350 | .job-details-container { 351 | margin-bottom:10px; 352 | padding-bottom: 10px; 353 | border-bottom: 1px solid lightgray; 354 | } 355 | .chart-item { 356 | margin-left: 20px; 357 | border-left: 1px solid #e8e8e8; 358 | padding-left: 15px; } 359 | 360 | .align-top { 361 | 362 | vertical-align: top; 363 | } 364 | .job-summary { 365 | width: 320px; 366 | } 367 | .state-failed { 368 | color: red; 369 | } 370 | .state-ok { 371 | color: #33cc33; 372 | } 373 | .state-warning { 374 | color: yellow; 375 | } 376 | 377 | .spacer-left { 378 | margin-left: 5px; 379 | } 380 | .job-panel { 381 | width: 100px; 382 | background-color: #fafafa; 383 | border: 1px solid #d8d8d8; 384 | border-radius: 8px; 385 | height: 100px; 386 | margin-right: 10px; } 387 | .job-panel:hover { 388 | border-color: #c0c0c0; 389 | } 390 | .lat-table-op { 391 | width: 120px; } 392 | 393 | .lat-table-head { 394 | width: 60px; 395 | text-align: right; } 396 | 397 | .lat-table td { 398 | text-align: right; 399 | padding: 0px 5px 0px 0px; } 400 | 401 | .lat-table tr:last-of-type { 402 | border-bottom: 1px solid lightgray; } 403 | .lat-table thead tr:first-of-type th { 404 | padding: 0px; 405 | } 406 | .lat-table thead tr:last-of-type th { 407 | padding: 0px 5px 0px 0px; 408 | } 409 | .lat-table caption { 410 | color: #404040; 411 | font-weight: bold; 412 | padding: 10px 0 0 0; } 413 | .bold { 414 | font-weight: bold; 415 | } 416 | .divider { 417 | width:100%; 418 | height:3px; 419 | border-bottom: 2px solid lightgray; 420 | } 421 | .divider:hover { 422 | cursor: ns-resize; 423 | } 424 | 425 | .zoom-in { 426 | // display: inline-block; 427 | float: right; 428 | margin-top:3px; 429 | height: 25px; 430 | width: 25px; 431 | background-color: lightgray; 432 | color: black; 433 | font-size:24px; 434 | } 435 | .zoom-in::before { 436 | font-family: FontAwesome; 437 | /* using fontawesome 4 */ 438 | content: "\f106"; 439 | position:relative; top:-40%; left:15%; 440 | } 441 | .zoom-in:hover { 442 | cursor: pointer; 443 | color:white; 444 | } 445 | 446 | .zoom-out { 447 | // display: inline-block; 448 | float: right; 449 | margin-top:3px; 450 | height: 25px; 451 | width: 25px; 452 | background-color: lightgray; 453 | color:black; 454 | font-size: 24px; 455 | } 456 | .zoom-out::before { 457 | font-family: FontAwesome; 458 | /* using fontawesome 4 */ 459 | content: "\f107"; 460 | position:relative; top:-40%; left:15%; 461 | } 462 | .zoom-out:hover { 463 | cursor: pointer; color: white; 464 | } 465 | .dropdown-menu-tbl-right { 466 | margin-left: -80px; 467 | top: 16px; 468 | min-width: 70px; } 469 | ul.dropdown-menu-tbl-right::before { 470 | content:none !important; 471 | } 472 | ul.dropdown-menu-tbl-right::after { 473 | content:none !important; 474 | } 475 | .hidden { 476 | display: none; 477 | } 478 | .visible { 479 | display: block; 480 | } 481 | .radio-common { 482 | margin-left: 10px; 483 | width: 120px; 484 | margin-top: 0px; } 485 | 486 | .radio-label-horizontal { 487 | width: 100px; 488 | margin-right: -15px; 489 | margin-left: 20px; 490 | text-align: left; 491 | vertical-align: top; } 492 | 493 | .radio-label-vertical { 494 | position: relative; } 495 | 496 | .radio-info { 497 | width: 750px; 498 | margin-left: 20px; 499 | display: inline-flex; 500 | } 501 | 502 | .radio-spacer { 503 | background-color: black; 504 | margin-left: 0px; 505 | margin-top:5px; 506 | width: 0px; 507 | content: ''; } 508 | 509 | .radio-container { 510 | margin-top:5px; } 511 | .textInfo { 512 | position: relative; 513 | display: inline-block; 514 | } 515 | .textInfo .tooltipContent { 516 | visibility: hidden; 517 | white-space: nowrap; 518 | background-color: black; 519 | opacity: 0.7; 520 | color: #fff; 521 | text-align: left; 522 | border-radius: 3px; 523 | padding: 5px 10px; 524 | position: absolute; 525 | z-index: 1; 526 | margin-left: 10px; 527 | line-height: 1.5em; 528 | top: -3.4em; 529 | left: -17px; 530 | font-weight: normal; } 531 | 532 | .textInfo .tooltipContent::after { 533 | content: ""; 534 | position: absolute; 535 | top: 1.3em; 536 | /* right: 93%; */ 537 | margin-top: 12px; 538 | /* margin-left:190px; */ 539 | border-width: 10px; 540 | border-style: solid; 541 | border-color: black transparent transparent transparent; } 542 | 543 | .textInfo:hover .tooltipContent { 544 | visibility: visible; 545 | } 546 | .icon-info { 547 | float: right; 548 | margin-top: 3px; 549 | height: 25px; 550 | width: 10px; 551 | color: black; 552 | font-size: 1.1em; } 553 | 554 | .icon-info::before { 555 | font-family: FontAwesome; 556 | /* using fontawesome 4 */ 557 | content: "\f05a"; 558 | position:relative; top:-40%; 559 | } 560 | 561 | .ratio-slider { 562 | position:relative; 563 | width: 200px !important; 564 | top:4px; 565 | margin-left: 15px; 566 | margin-right: 10px; } 567 | 568 | .ratio-text { 569 | margin-left: 0px; } 570 | 571 | .ratio-value { 572 | margin-left: 5px; 573 | padding-right: 10px; 574 | display:inline-block; 575 | width:20px; } 576 | 577 | .option-title { 578 | width: 120px; 579 | display:inline-block; 580 | text-align: right; 581 | padding-right:10px } 582 | .custom-profile-box { 583 | padding-top:0px; 584 | padding-left:5px; 585 | } 586 | .custom-profile-options-container { 587 | margin-top:20px; 588 | height:370px; 589 | display: grid; 590 | 591 | } 592 | 593 | ::-webkit-input-placeholder { /* Edge */ 594 | font-style: italic; 595 | } 596 | 597 | :-ms-input-placeholder { /* Internet Explorer 10-11 */ 598 | font-style: italic; 599 | } 600 | 601 | ::placeholder { 602 | font-style: italic; 603 | } 604 | 605 | 606 | label:after { 607 | padding-right: 10px; } 608 | 609 | label { 610 | width:120px; 611 | } 612 | 613 | #masthead-container { 614 | z-index:99; 615 | position: fixed; 616 | top: 0; 617 | width: 100%; 618 | } 619 | 620 | #container { 621 | position: relative; 622 | top:80px; 623 | } -------------------------------------------------------------------------------- /react/app/src/common/kebab.jsx: -------------------------------------------------------------------------------- 1 | import React from 'react'; 2 | import '../app.scss'; 3 | 4 | export class Kebab extends React.Component { 5 | // 6 | // Kebab based on the Patternfly spec @ https://www.patternfly.org/pattern-library/widgets/#kebabs 7 | constructor (props) { 8 | super(props); 9 | this.state = { 10 | menu: "dropdown-menu hidden dropdown-menu-tbl-right" 11 | }; 12 | } 13 | 14 | toggle = () => { 15 | if (this.state.menu === "dropdown-menu hidden dropdown-menu-tbl-right") { 16 | this.setState({ 17 | menu: "dropdown-menu visible dropdown-menu-tbl-right" 18 | }); 19 | } else { 20 | this.setState({ 21 | menu: "dropdown-menu hidden dropdown-menu-tbl-right" 22 | }); 23 | } 24 | } 25 | 26 | clickHandler = (value, callback) => { 27 | this.setState({ 28 | menu: "dropdown-menu hidden dropdown-menu-tbl-right" 29 | }); 30 | callback(value); 31 | } 32 | 33 | render () { 34 | let actions; 35 | 36 | // must use mousedown on the li components to prevent the button onclick sequence clash 37 | if (this.props.actions) { 38 | actions = this.props.actions.map((item, idx) => { 39 | return
  • { this.clickHandler(this.props.value, item.callback) }} >{ item.action }
    • ; 40 | }); 41 | } else { 42 | actions = (
    ); 43 | } 44 | console.log("rendering kebab"); 45 | return ( 46 |
    47 | 53 |
      54 | { actions } 55 |
    56 |
    57 | ); 58 | } 59 | } 60 | -------------------------------------------------------------------------------- /react/app/src/common/modal.jsx: -------------------------------------------------------------------------------- 1 | import React from 'react'; 2 | import '../app.scss'; 3 | 4 | 5 | export class GenericModal extends React.Component { 6 | // 7 | // generic modal component, with title bar and close button 8 | constructor(props) { 9 | super(props); 10 | this.state = {}; 11 | } 12 | 13 | render() { 14 | let showHideClass = this.props.show ? 'modal display-block' : 'modal display-none'; 15 | let closeButton = (
    ); 16 | if (this.props.withClose) { 17 | closeButton = (
    18 | 20 |
    ); 21 | } 22 | return ( 23 |
    24 |
    25 | 26 |
    27 |
    28 | { this.props.content } 29 |
    30 |
    31 | {closeButton} 32 |
    33 |
    34 |
    35 | ); 36 | } 37 | } 38 | 39 | export class WindowTitle extends React.Component { 40 | render () { 41 | return ( 42 |
    43 |
    {this.props.title}
    44 |
    { this.props.closeHandler() }} /> 45 |
    46 | ); 47 | } 48 | } -------------------------------------------------------------------------------- /react/app/src/common/radioset.jsx: -------------------------------------------------------------------------------- 1 | import React from 'react'; 2 | import '../app.scss'; 3 | import { Tooltip } from './tooltip.jsx'; 4 | 5 | export class RadioSet extends React.Component { 6 | // 7 | // radio button group component 8 | constructor(props) { 9 | super(props); 10 | 11 | this.state = { 12 | default: this.props.default, 13 | name: props.config.name, 14 | selected: props.default 15 | }; 16 | } 17 | 18 | componentDidUpdate(prevProps, prevState) { 19 | console.log("props changed in radioset"); 20 | if (prevProps.default != this.props.default) { 21 | console.log("sendng change of default :" + this.props.default); 22 | this.updateDefault(this.props.default); 23 | } 24 | // if (!prevProps.visible) { 25 | // this.refs.hostInputField.focus(); 26 | // console.log("with props " + JSON.stringify(prevProps)); 27 | // } 28 | } 29 | 30 | updateDefault = (option) => { 31 | console.log("updating " + this.props.config.name + " as default changed"); 32 | this.setState({default: option}); 33 | } 34 | 35 | changeHandler = (event) => { 36 | console.log("radio set " + this.props.config.name + " changed"); 37 | console.log("value is " + event.target.value); 38 | this.setState({ 39 | selected: event.target.value, 40 | default: event.target.value 41 | }); 42 | this.props.callback(event); 43 | } 44 | 45 | render() { 46 | console.log("in radioset render for " + this.props.config.name); 47 | var radioGrpClass; 48 | var labelClass; 49 | var toolTip; 50 | var radioGrp; 51 | var buttons; 52 | if (this.props.config.horizontal) { 53 | radioGrpClass = "radio radio-common inline-block"; 54 | labelClass = "radio-label-horizontal inline-block"; 55 | } else { 56 | radioGrpClass = "radio radio-common display-block"; 57 | labelClass = "radio-label-vertical"; 58 | } 59 | 60 | if (this.props.config.tooltip) { 61 | toolTip = (); 62 | } else { 63 | toolTip = ( 64 | 65 | ); 66 | } 67 | buttons = this.props.config.options.map((text, i) => { 68 | return ( 69 |
    70 | 78 |
    ); 79 | }); 80 | 81 | if (this.props.config.info) { 82 | radioGrp = ( 83 |
    84 |
    {this.props.config.info}
    85 |
    86 |
     
    87 | {buttons} 88 |
    89 |
    90 | ); 91 | } else { 92 | radioGrp = buttons; 93 | } 94 | 95 | return ( 96 |
    97 |
    98 |
    {this.props.config.description}{ toolTip }
    99 | {radioGrp} 100 |
    101 |
    102 | ); 103 | } 104 | } 105 | -------------------------------------------------------------------------------- /react/app/src/common/ratioslider.jsx: -------------------------------------------------------------------------------- 1 | import React from 'react'; 2 | import '../app.scss'; 3 | 4 | export class RatioSlider extends React.Component { 5 | constructor(props) { 6 | super(props); 7 | this.state = { 8 | value: props.value, 9 | prefixValue: props.value, 10 | suffixValue: props.value, 11 | }; 12 | } 13 | // 14 | // Simple tooltip widget 15 | // constructor(props) { 16 | // super(props); 17 | 18 | // this.state = { 19 | // timer: 0, 20 | // active: false 21 | // }; 22 | // this.loadInterval = 0; 23 | // } 24 | 25 | componentDidUpdate(prevProps, prevState) { 26 | if (prevProps.value != this.props.value) { 27 | console.log("value changed! :" + this.props.value); 28 | this.setState({ 29 | value: this.props.value, 30 | prefixValue: 100 - this.props.value, 31 | suffixValue: this.props.value, 32 | }); 33 | // this.updateDefault(this.props.default); 34 | } 35 | } 36 | updateHandler = (event) => { 37 | console.log("slider changed " + event.target.value); 38 | 39 | this.setState({ 40 | value: event.target.value, 41 | prefixValue: 100 - event.target.value, 42 | suffixValue: event.target.value, 43 | }); 44 | this.props.callback(event); 45 | } 46 | 47 | render() { 48 | console.debug("in ratioslider render") 49 | return ( 50 |
    51 |
    52 | {this.props.title} 53 |
    54 |
    55 | {this.props.prefix} 56 | {this.state.prefixValue}% 57 |
    58 |
    59 | {this.updateHandler(event);}}> 66 | 67 |
    68 |
    69 | {this.props.suffix} 70 | {this.state.suffixValue}% 71 |
    72 |
    ); 73 | } 74 | } 75 | -------------------------------------------------------------------------------- /react/app/src/common/tooltip.jsx: -------------------------------------------------------------------------------- 1 | import React from 'react'; 2 | import '../app.scss'; 3 | 4 | export class Tooltip extends React.Component { 5 | // 6 | // Simple tooltip widget 7 | // constructor(props) { 8 | // super(props); 9 | 10 | // this.state = { 11 | // timer: 0, 12 | // active: false 13 | // }; 14 | // this.loadInterval = 0; 15 | // } 16 | 17 | render() { 18 | let tooltipText = this.props.text.split('\n').map((text, key) => { 19 | let out; 20 | if (text.includes('!Link:')) { 21 | let [prefix, therest] = text.split('!Link:'); 22 | let [protocol, urlPath, linkName, remainingText] = therest.split(':'); 23 | let url = protocol + ":" + urlPath; 24 | out = ({prefix}{linkName}{remainingText}); 25 | } else { 26 | out = text; 27 | } 28 | return
    {out}
    ; 29 | }); 30 | return ( 31 |
      32 | 33 | { tooltipText } 34 |
    ); 35 | } 36 | } 37 | -------------------------------------------------------------------------------- /react/app/src/components/main.jsx: -------------------------------------------------------------------------------- 1 | import React from 'react'; 2 | import '../app.scss'; 3 | import { MastHead } from './masthead.jsx'; 4 | import { Profiles } from './profiles.jsx'; 5 | import { Jobs } from './jobs.jsx'; 6 | 7 | 8 | export class App extends React.Component { 9 | constructor(props) { 10 | super(props); 11 | this.state = { 12 | profiles: 'active', 13 | jobs: 'inactive', 14 | current: 'profiles', 15 | workers: 0, 16 | activeJobId: undefined, 17 | }; 18 | // this.workers = 2; 19 | }; 20 | 21 | menuSelect = (item) => { 22 | if (item != this.state.current) { 23 | let newState = { 24 | current: item, 25 | profiles: 'inactive', 26 | jobs:'inactive' 27 | }; 28 | newState[item] = 'active'; 29 | console.log(item); 30 | this.setState(newState); 31 | console.log(JSON.stringify(newState)); 32 | } 33 | } 34 | 35 | // updateWorker = (workerCount) => { 36 | // this.workers = workerCount; 37 | // this.setState({ 38 | // workers: workerCount 39 | // }); 40 | // } 41 | 42 | updateWorkers = (count) => { 43 | console.debug("updating workers to ", count); 44 | this.setState({ 45 | workers: count, 46 | }); 47 | } 48 | 49 | jobStateChange = (jobID) => { 50 | console.debug("job state has changed ", jobID); 51 | this.setState({ 52 | activeJobId: jobID, 53 | }); 54 | } 55 | 56 | render() { 57 | console.log("render main. env vars: " + JSON.stringify(process.env)); 58 | return ( 59 |
    60 |
    61 | 62 | 66 |
    67 |
    68 | 69 | 70 |
    71 | 72 |
    73 | ); 74 | } 75 | } 76 | 77 | 78 | export default App; 79 | 80 | -------------------------------------------------------------------------------- /react/app/src/components/masthead.jsx: -------------------------------------------------------------------------------- 1 | import React from 'react'; 2 | import '../app.scss'; 3 | import {setAPIURL} from '../utils/utils.js'; 4 | 5 | /* Masthead will contain a couple of items from the webservice status api 6 | to show mode, task active, job queue size 7 | */ 8 | var api_url = setAPIURL(); 9 | 10 | export class MastHead extends React.Component { 11 | constructor(props) { 12 | super(props); 13 | this.interval = 0; 14 | this.state = { 15 | task_active: false, 16 | active_job_id: '', 17 | tasks_queued: 0, 18 | task_type: 'N/A', 19 | target: '', 20 | run_time: 0, 21 | workers: 0, 22 | apiAvailable: true, 23 | debug_mode: false, 24 | }; 25 | }; 26 | 27 | getStatus() { 28 | fetch(api_url + "/api/status") 29 | .then((response) => { 30 | console.debug("status fetch : ", response.status); 31 | if (response.status == 200) { 32 | return response.json(); 33 | } else {} 34 | throw Error(`status API call failed with HTTP status: ${response.status}`); 35 | }) 36 | .then((status) => { 37 | /* Happy path */ 38 | let state = status.data; 39 | state['apiAvailable'] = true; 40 | // console.debug("state returned " + JSON.stringify(state)); 41 | if (state.workers != this.state.workers) { 42 | console.debug('worker count changed telling parent :', state.workers, this.state.workers); 43 | this.props.workersCallback(state.workers); 44 | } 45 | if (state.active_job_id != this.state.active_job_id) { 46 | console.debug("job change"); 47 | this.props.jobChangeCallback(state.active_job_id); 48 | } 49 | this.setState(state); 50 | // console.debug("masthead status returned worker count of " + state.workers); 51 | // this.props.workerCB(this.state.workers); 52 | // console.log(JSON.stringify(state)); 53 | }) 54 | .catch((error) => { 55 | console.error("Error:", error); 56 | console.error("killing interval based status checking"); 57 | this.setState({ 58 | apiAvailable: false 59 | }); 60 | clearInterval(this.interval); 61 | }); 62 | 63 | } 64 | 65 | intervalHandler = () => { 66 | this.getStatus(); 67 | } 68 | 69 | componentDidMount() { 70 | console.log("starting interval based call to the /status API endpoint"); 71 | console.log("here's the env var setting " + api_url); 72 | this.getStatus(); 73 | this.interval = setInterval(this.intervalHandler, 5000); 74 | } 75 | 76 | render() { 77 | return ( 78 |
    79 |
    80 |
    FIOLoadGen
    81 | 82 |
    83 |
    84 |
    85 | ); 86 | } 87 | } 88 | 89 | class ServiceState extends React.Component { 90 | constructor(props) { 91 | super(props); 92 | self.state = { 93 | dummy: false 94 | }; 95 | } 96 | render() { 97 | let taskText = this.props.state.task_active ? "Yes" : "no"; 98 | let apiSymbol = this.props.state.apiAvailable ? "fa fa-lg fa-check-circle-o state-ok" : "fa fa-lg fa-times-circle-o state-failed"; 99 | let debugFlag; 100 | if (this.props.state.debug_mode) { 101 | debugFlag = (DEBUG); 102 | } else { 103 | debugFlag = (); 104 | } 105 | 106 | return ( 107 |
    108 |
    API:
    109 |
    Job Active:{taskText}
    110 |
    Queued:{this.props.state.tasks_queued}
    111 |
    Target Platform:{this.props.state.target}
    112 |
    {debugFlag}
    113 |
    114 | ) 115 | } 116 | } 117 | 118 | export default MastHead; 119 | 120 | -------------------------------------------------------------------------------- /react/app/src/components/profiles.jsx: -------------------------------------------------------------------------------- 1 | import React from 'react'; 2 | 3 | import {GenericModal} from '../common/modal.jsx'; 4 | import '../app.scss'; 5 | import { handleAPIErrors, setAPIURL } from '../utils/utils.js'; 6 | import { RadioSet} from '../common/radioset.jsx'; 7 | import { Tooltip } from '../common/tooltip.jsx'; 8 | import { RatioSlider } from '../common/ratioslider.jsx'; 9 | 10 | var api_url = setAPIURL(); 11 | const ioDepthTip="Changing the IO depth, varies the number of OS queues the FIO tool uses to drive I/O" 12 | const ioTypeTip="Databases typical exhibit random I/O, whereas logging is sequential" 13 | const runTimeTip="Required run time for the test (in minutes)" 14 | 15 | export class Profiles extends React.Component { 16 | constructor(props) { 17 | super(props); 18 | this.state = { 19 | profiles: [], 20 | activeProfile: undefined, 21 | profileContent: '', 22 | modalOpen: false, 23 | workers: {}, 24 | }; 25 | this.spec = { 26 | runTime: 60, 27 | ioType: "Random", 28 | ioBlockSize: "4KB", 29 | ioPattern: 50, 30 | ioDepth: 4, 31 | profileName: '', 32 | }; 33 | }; 34 | 35 | // shouldComponentUpdate(nextProps, PrevProps) { 36 | // if (nextProps.visibility == "active"){ 37 | // console.debug("profiles should render"); 38 | // return true; 39 | // } else { 40 | // return false; 41 | // } 42 | // } 43 | 44 | openModal() { 45 | this.setState({ 46 | modalOpen: true 47 | }); 48 | } 49 | closeModal = () => { 50 | this.setState({ 51 | modalOpen: false 52 | }); 53 | } 54 | 55 | checkProfile = (profileName) => { 56 | return this.state.profiles.includes(profileName); 57 | } 58 | 59 | fetchAllProfiles() { //refresh = false) { 60 | // let endpoint = (refresh == true) ? '/api/profile?refresh=true' : '/api/profile'; 61 | fetch(api_url + '/api/profile') 62 | .then((response) => { 63 | console.debug("Profile fetch : ", response.status); 64 | if (response.status == 200) { 65 | return response.json(); 66 | } else {} 67 | throw Error(`Profile fetch failed with HTTP status: ${response.status}`); 68 | }) 69 | .then((profiles) => { 70 | /* Happy path */ 71 | let profileNames = []; 72 | profiles.data.forEach(profile => { 73 | profileNames.push(profile.name); 74 | }); 75 | profileNames.push('custom'); 76 | this.setState({ 77 | profiles: profileNames, 78 | }); 79 | console.log(profiles); 80 | }) 81 | .catch((error) => { 82 | console.error("Error:", error); 83 | }); 84 | } 85 | 86 | componentDidMount() { 87 | 88 | this.fetchAllProfiles(); 89 | 90 | fetch(api_url + "/api/status") 91 | .then((response) => { 92 | console.log("Initial status call ", response.status); 93 | // console.log(JSON.stringify(response.json())); 94 | return response.json(); 95 | }) 96 | .then((status) => { 97 | console.log(status.data); 98 | this.setState({ 99 | workers: status.data.workers, 100 | }); 101 | }) 102 | .catch((error) => { 103 | console.error("initial status call failure, unable to fetch worker info, using default of 0"); 104 | this.setState({ 105 | workers: {}, 106 | }); 107 | }) 108 | } 109 | 110 | selectProfile(event) { 111 | this.setState({ 112 | activeProfile: event.target.value 113 | }); 114 | 115 | if (event.target.value == "custom") { 116 | console.log("selected the custom profile type") 117 | } else { 118 | this.fetchProfile(event.target.value); 119 | } 120 | 121 | } 122 | 123 | fetchProfile(profileName) { 124 | console.debug("fetching profile " + profileName); 125 | fetch(api_url + "/api/profile/" + profileName) 126 | .then((response) => { 127 | console.debug("Profile fetch : ", response.status); 128 | if (response.status == 200) { 129 | return response.json(); 130 | } else {} 131 | throw Error(`Fetch failed with HTTP status: ${response.status}`); 132 | }) 133 | .then((profile) => { 134 | /* Happy path */ 135 | this.setState({ 136 | profileContent: profile.data 137 | }); 138 | }) 139 | .catch((error) => { 140 | console.error("Profile fetch error:", error); 141 | }); 142 | } 143 | runJob = (parms) => { 144 | console.debug("run the job - issue a POST request to the API"); 145 | // remove specific attributes from parms object 146 | delete parms.titleBorder; 147 | console.debug(JSON.stringify(this.spec)) 148 | if (this.state.activeProfile == 'custom') { 149 | parms['spec'] = this.spec 150 | } else { 151 | parms['spec'] = this.state.profileContent; 152 | } 153 | // parms['spec'] = this.state.profileContent; 154 | console.debug("in runJob handler " + JSON.stringify(parms)); 155 | console.debug("profile is " + this.state.activeProfile); 156 | 157 | fetch(api_url + "/api/job/" + this.state.activeProfile, { 158 | method: 'post', 159 | headers: { 160 | "Accept": "application/json", 161 | "Content-Type": "application/json" 162 | }, 163 | body: JSON.stringify(parms) 164 | }) 165 | .then((e) => { 166 | if (e.status == 202) { 167 | console.debug("request accepted (" + e.status + ")") 168 | this.props.changeMenuCallback('jobs'); 169 | } else { 170 | console.error("POST request to submit job failed with http status " + e.status); 171 | } 172 | }) 173 | .catch((e) => { 174 | console.log(JSON.stringify(e)); 175 | console.error("Post to /api/job failed " + e.status); 176 | }); 177 | 178 | } 179 | 180 | getJobDetails() { 181 | if (this.state.activeProfile) { 182 | this.openModal(); 183 | } 184 | } 185 | submitHandler = (parms) => { 186 | console.debug("in submit handler " + JSON.stringify(parms)); 187 | this.closeModal() 188 | this.runJob(parms) 189 | } 190 | 191 | refreshProfiles= () => { 192 | console.debug("in refresh profiles"); 193 | this.fetchAllProfiles(); // true); 194 | // clear content in the profile textarea 195 | // this.setState({ 196 | // profileContent: '' 197 | // }); 198 | } 199 | updateSpec = (kv) => { 200 | console.log("updating spec ", JSON.stringify(kv)) 201 | Object.assign(this.spec, kv); 202 | } 203 | 204 | render() { 205 | console.debug("render profiles called with visibility: ", this.props.visibility); 206 | if (this.props.visibility != 'active') { 207 | return ( 208 |
    209 | ); 210 | } 211 | console.log("workers is set to :" + JSON.stringify(this.props.workers)) 212 | let profileSelector; 213 | // console.debug("client limit is " + this.props.clientLimit); 214 | if (this.state.profiles.length > 0) { 215 | let profileList = this.state.profiles.map((profile, i) => { 216 | return () 217 | }); 218 | profileSelector = ( 219 |
    220 | {/* */} 221 | 224 |
    225 |
    226 | ); 227 | } 228 | let jobDefinition; 229 | if (this.state.modalOpen) { 230 | jobDefinition = (); 231 | } else { 232 | jobDefinition = (
    ); 233 | } 234 | 235 | return ( 236 |
    237 | 242 |
    243 |
    244 |
    245 | {profileSelector} 246 |
    247 | 248 | 249 |
    250 |
    251 |
    252 |
    253 |
    254 | ); 255 | } 256 | } 257 | 258 | class CustomProfile extends React.Component { 259 | constructor(props) { 260 | super(props); 261 | this.state = { 262 | ioType: "Random", 263 | ioPattern: 50, 264 | ioBlockSize: "4KB", 265 | runTime: 1, 266 | ioDepth: 4, 267 | profileName: '', 268 | profileStyle: {}, 269 | }; 270 | this.defaults = Object.assign({}, this.state); 271 | this.ioType = { 272 | description: "I/O Type:", 273 | options: ["Random", "Sequential"], 274 | name: "ioType", 275 | info: "", // Disk I/O can be issued in a sequential or random manner", 276 | tooltip: ioTypeTip, 277 | horizontal: true 278 | }; 279 | 280 | } 281 | 282 | resetButtonHandler = () => { 283 | // event.preventDefault(); 284 | console.log("defaults are " + JSON.stringify(this.defaults)); 285 | this.setState({ 286 | profileName: '', 287 | ioPattern: this.defaults.ioPattern, 288 | ioType: this.defaults.ioType, 289 | ioBlockSize: this.defaults.ioBlockSize, 290 | ioDepth: this.defaults.ioDepth, 291 | runTime: this.defaults.runTime, 292 | profileStyle:{}, 293 | }); 294 | 295 | } 296 | radioButtonHandler = (event) => { 297 | // if name is not set this is a select widget 298 | console.log("in option handler " + event.target.name + " / " + event.target.value); 299 | this.setState({ioType: event.target.value}) 300 | this.props.callback({ioType: event.target.value}); 301 | 302 | } 303 | sliderHandler = (event) => { 304 | console.log("in slider handler " + event.target.value); 305 | this.setState({ioPattern: event.target.value}); 306 | this.props.callback({ioPattern: event.target.value}); 307 | } 308 | selectHandler = (event) => { 309 | console.log("in select handler with " + event.target.value); 310 | this.setState({ioBlockSize: event.target.value}); 311 | this.props.callback({ioBlockSize: event.target.value}) 312 | } 313 | runtimeHandler = (event) => { 314 | console.log("in runtime handler " + event.target.name + " / " + event.target.value); 315 | this.setState({runTime: event.target.value}) 316 | let mins = event.target.value * 60; 317 | console.log("runtime handler sending runtime of ", mins, "to parent"); 318 | this.props.callback({runTime: mins}) 319 | } 320 | ioDepthHandler = (event) => { 321 | console.log("in iodepth handler " + event.target.name + " / " + event.target.value); 322 | this.setState({ioDepth: event.target.value}) 323 | this.props.callback({ioDepth: event.target.value}) 324 | } 325 | profileNameUpdater = (event) => { 326 | let newState = {}; 327 | newState.profileName = event.target.value; 328 | 329 | if (this.props.checkProfileCallback(event.target.value)) { 330 | console.error("profile exists!") 331 | newState.profileStyle = {borderColor: "red"}; 332 | } else { 333 | if (Object.keys(this.state.profileStyle).length > 0) { 334 | newState.profileStyle = {} 335 | } 336 | } 337 | this.setState(newState); 338 | } 339 | profileNameHandler = (event) => { 340 | 341 | if (event.target.value) { 342 | if (event.target.value != this.state.profileName) { 343 | console.debug("Profile name updated to " + event.target.value); 344 | // 345 | // TODO check that the name doesn't conflict with an existing name 346 | // 347 | this.setState({ 348 | profileName: event.target.value, 349 | profileStyle: {}, 350 | }); 351 | this.props.callback({profileName: event.target.value}); 352 | } 353 | 354 | } else { 355 | if (Object.keys(this.state.profileStyle).length > 0) { 356 | this.setState({profileStyle: {}}); 357 | } 358 | 359 | } 360 | } 361 | saveProfile() { 362 | let localState = JSON.parse(JSON.stringify(this.state)) 363 | if (this.state.profileName == '') { 364 | console.debug("profile save requested but no name given") 365 | this.setState({profileStyle: { 366 | borderColor: "red"} 367 | // borderRadius: "5px"} 368 | }); 369 | return 370 | } 371 | 372 | if (Object.keys(this.state.profileStyle).length > 0) { 373 | console.error("Can't save a profile which has been flagged as an error"); 374 | return 375 | } 376 | 377 | delete localState.profileStyle; 378 | localState.runTime = localState.runTime * 60; 379 | console.log("save me " + JSON.stringify(localState)) 380 | fetch(api_url + "/api/profile/" + this.state.profileName, { 381 | method: 'put', 382 | headers: { 383 | "Accept": "application/json", 384 | "Content-Type": "application/json" 385 | }, 386 | body: JSON.stringify({spec: localState}) 387 | }) 388 | .then((e) => { 389 | if (e.status == 200) { 390 | console.debug("request accepted (" + e.status + ")") 391 | this.props.refreshCallback() 392 | } else { 393 | console.error("PUT request to store profile failed with http status " + e.status); 394 | } 395 | }) 396 | .catch((e) => { 397 | console.error("PUT to /api/profile failed :" + e.message); 398 | }); 399 | 400 | } 401 | 402 | render () { 403 | if (!this.props.visible) { 404 | return ( 405 |
    406 | ) 407 | } 408 | 409 | return ( 410 |
    411 |

    Custom Profile

    412 |

    In addition to the IO profiles listed on the left, the custom option allows 413 | you to create and then execute one-off I/O profiles.

    414 |

    Use the options below to define a profile.

    415 |
    416 |
    417 | 418 | 430 | 431 |
    432 | 433 | 434 |
    435 | 436 | 447 | 451 | 460 |
    461 |
    462 | 466 | 475 | 476 |
    477 |
    478 | 479 | 480 |
    481 |
    482 |
    483 | ) 484 | } 485 | } 486 | class ProfileContent extends React.Component { 487 | constructor(props) { 488 | super(props); 489 | this.state = { 490 | readonly: true, 491 | profileContent: '', 492 | }; 493 | } 494 | 495 | render () { 496 | if (!this.props.visible) { 497 | return ( 498 |
    499 | ); 500 | } 501 | 502 | let content; 503 | if (this.props.profileContent == '') { 504 | content = (
     Choose a profile to view the FIO specification
    ); 505 | } else { 506 | content = (
    {this.props.profileContent}
    ); 507 | } 508 | 509 | return ( 510 |
    511 | {content} 512 | {/*