├── .gitignore ├── README.md ├── alembic.ini ├── alembic ├── README ├── env.py ├── script.py.mako └── versions │ └── f2b6313d4618_initial_create.py ├── frag_tests ├── buffered-append-vs-fallocate.fio ├── buffered-append-vs-fallocate.py ├── correlated-lifetimes.fio ├── correlated-lifetimes.py ├── four-sizes.fio ├── four-sizes.py ├── funny-sizes-high.fio ├── funny-sizes-high.py ├── funny-sizes-low.fio ├── funny-sizes-low.py ├── mixed-lifetimes.fio └── mixed-lifetimes.py ├── fsperf ├── fsperf-clean-results ├── fsperf-compare ├── fsperf-generate-graph ├── fsperf-generate-results ├── fsperf-sqlite.sql ├── local-cfg-example ├── manage.py ├── src ├── .gitignore ├── FioCompare.py ├── FioResultDecoder.py ├── PerfTest.py ├── ResultData.py ├── clean-results.py ├── compare.py ├── frag │ ├── .gitignore │ ├── Cargo.toml │ ├── bg-dump.jinja │ ├── cleanup.sh │ ├── src │ │ └── main.rs │ └── tests │ │ └── buffered-append-vs-fallocate.py ├── fsperf.py ├── generate-graph.py ├── generate-results-page.py ├── generate-schema.py ├── index.jinja ├── nullblk.py ├── test.jinja └── utils.py ├── tests ├── btrfsbgscalability.py ├── buffered-append-sync.py ├── buffered-randwrite-16g.py ├── dbench-60.py ├── dio-4kbs-16threads.py ├── dio-randread.py ├── empty-files-500k.py ├── randwrite-2xram.py ├── small-files-100k.py └── untar-firefox.py └── www └── style.css /.gitignore: -------------------------------------------------------------------------------- 1 | local.cfg 2 | results 3 | fsperf-results.db 4 | tests/__pycache__/ 5 | *.swp 6 | firefox*.tar* 7 | www/* 8 | __pycache__/ 9 | *.py[cod] 10 | src/frag/bg-dump.btrd 11 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # fsperf 2 | 3 | fsperf is a performance testing framework built around 4 | [fio](https://github.com/axboe/fio). The goal is to provide a simple to run 5 | framework for file system developers to be able to check their patches and make 6 | sure they do not regress in performance. In addition to `fio` tests, `fsperf` 7 | supports basic timing tests and `dbench` tests. 8 | 9 | # Requirements 10 | The following python packages are required: 11 | * alembic 12 | * SQLAlchemy 13 | * sqlalchemy-migrate 14 | * texttable 15 | * psutil 16 | * jinja2 17 | * matplotlib 18 | * numpy 19 | 20 | For fragmentation analysis (-F), rust is required. Run 21 | `cargo build -r` in `src/frag/` 22 | and it will pull the necessary dependencies. In addition, btrd is required to 23 | be installed on PATH to collect btrfs bg/extent data. 24 | 25 | # Configuration 26 | 27 | In order to configure the suite you need to create a `local.cfg` file in the 28 | base of the fsperf directory. This file takes the format normal INI files. You 29 | must specify a `[main]` section with the following required options 30 | 31 | * `directory` - the directory to run the performance jobs in. 32 | * `cpugovernor` - the cpu governor to use for the run. 33 | 34 | Then subsequently you must specify configs, which whatever name you wish to use. 35 | These take the following optional options 36 | 37 | * `mkfs` - if specified this will be run between every test. This is the full 38 | mkfs command needed for your test environment. 39 | * `mount` - the command to mount the fs. If specified the fs will be 40 | unmounted between each test. 41 | * `device` - the device that will be used for this section. 42 | * `iosched` - Must match one of the options in 43 | `/sys/block/{device}/queue/scheduler`. 44 | 45 | ``` 46 | [main] 47 | directory=/mnt/test 48 | 49 | [btrfs] 50 | device=/dev/nvme0n1 51 | mkfs=mkfs.btrfs -f 52 | mount=mount -o noatime 53 | ``` 54 | 55 | You can specify multiple configurations per file, and switch between them with 56 | the `-c` option for fsperf. 57 | 58 | # How to run 59 | 60 | Once you've setup your `local.cfg` you simply run 61 | 62 | ``` 63 | ./fsperf 64 | ``` 65 | 66 | and wait for the suite to finish. This will run the tests found under the 67 | tests/ directory and store the results. The blank invocation is meant for 68 | continuous performance testing. 69 | 70 | ## A/B testing 71 | 72 | If you wish to do A/B testing you can do the following 73 | 74 | ``` 75 | ./fsperf -p "myabtest" 76 | 77 | ./fsperf -p "myabtest" -t 78 | ./fsperf-clean-results myabtest 79 | ``` 80 | 81 | This will store the base results under a specific heading, "myabtest", so it 82 | doesn't muddy any other unrelated performance results. Then using the `-t` 83 | option it will run the tests, and spit out a comparison table between the 84 | baseline and the current run, and then discard the current run's results in 85 | order to not pollute the baseline results. 86 | 87 | The comparison compares all of the saved results, so things like `fio`'s max 88 | latencies may be a little noisy. In order to reduce the noise of these sort of 89 | metrics you can do something like the following 90 | 91 | ``` 92 | ./fsperf -p "myabtest" -n 5 93 | 94 | ./fsperf -p "myabtest" -n 5 -t 95 | ./fsperf-clean-results myabtest 96 | ``` 97 | 98 | This will run each test 5 times, which means the baseline and new results will 99 | be averaged, and then the averages will be compared against eachother. 100 | 101 | Finally the `fsperf-clean-results` script will delete anything that matches your 102 | special results, so you can re-use the label in the future. 103 | 104 | # Understanding the comparisons 105 | 106 | We only compare the last run of the given test with the given configuration. So 107 | if you have multiple sections in your configuration file, such as the following 108 | 109 | ``` 110 | [btrfs] 111 | device=/dev/nvme0n1 112 | mkfs=mkfs.btrfs -f 113 | mount=mount -o noatime 114 | 115 | [xfs] 116 | device=/dev/nvme0n1 117 | iosched=none 118 | mkfs=mkfs.xfs -f 119 | mount=mount -o noatime 120 | ``` 121 | 122 | Only tests in the same configuration will be compared against each other. 123 | Future work will include the ability to compare with other configurations, but 124 | currently you can just change your local.cfg if you wish to compare runs of 125 | different configurations. 126 | 127 | # Disabling tests 128 | 129 | Sometimes you may need to disable a test, so simply add the test name to it's 130 | own line in the `disabled-tests` file in the root of the project directory. 131 | 132 | # Fragmentation tests 133 | 134 | A second set of tests which challenge the btrfs block_group/extent allocator 135 | are found in frag_tests/. Running fsperf with -F will add them to the list of 136 | eligible tests to run. 137 | -------------------------------------------------------------------------------- /alembic.ini: -------------------------------------------------------------------------------- 1 | # A generic, single database configuration. 2 | 3 | [alembic] 4 | # path to migration scripts 5 | # Use forward slashes (/) also on windows to provide an os agnostic path 6 | script_location = alembic 7 | 8 | # template used to generate migration file names; The default value is %%(rev)s_%%(slug)s 9 | # Uncomment the line below if you want the files to be prepended with date and time 10 | # see https://alembic.sqlalchemy.org/en/latest/tutorial.html#editing-the-ini-file 11 | # for all available tokens 12 | # file_template = %%(year)d_%%(month).2d_%%(day).2d_%%(hour).2d%%(minute).2d-%%(rev)s_%%(slug)s 13 | 14 | # sys.path path, will be prepended to sys.path if present. 15 | # defaults to the current working directory. 16 | prepend_sys_path = . 17 | 18 | # timezone to use when rendering the date within the migration file 19 | # as well as the filename. 20 | # If specified, requires the python>=3.9 or backports.zoneinfo library. 21 | # Any required deps can installed by adding `alembic[tz]` to the pip requirements 22 | # string value is passed to ZoneInfo() 23 | # leave blank for localtime 24 | # timezone = 25 | 26 | # max length of characters to apply to the "slug" field 27 | # truncate_slug_length = 40 28 | 29 | # set to 'true' to run the environment during 30 | # the 'revision' command, regardless of autogenerate 31 | # revision_environment = false 32 | 33 | # set to 'true' to allow .pyc and .pyo files without 34 | # a source .py file to be detected as revisions in the 35 | # versions/ directory 36 | # sourceless = false 37 | 38 | # version location specification; This defaults 39 | # to alembic/versions. When using multiple version 40 | # directories, initial revisions must be specified with --version-path. 41 | # The path separator used here should be the separator specified by "version_path_separator" below. 42 | # version_locations = %(here)s/bar:%(here)s/bat:alembic/versions 43 | 44 | # version path separator; As mentioned above, this is the character used to split 45 | # version_locations. The default within new alembic.ini files is "os", which uses os.pathsep. 46 | # If this key is omitted entirely, it falls back to the legacy behavior of splitting on spaces and/or commas. 47 | # Valid values for version_path_separator are: 48 | # 49 | # version_path_separator = : 50 | # version_path_separator = ; 51 | # version_path_separator = space 52 | version_path_separator = os # Use os.pathsep. Default configuration used for new projects. 53 | 54 | # set to 'true' to search source files recursively 55 | # in each "version_locations" directory 56 | # new in Alembic version 1.10 57 | # recursive_version_locations = false 58 | 59 | # the output encoding used when revision files 60 | # are written from script.py.mako 61 | # output_encoding = utf-8 62 | 63 | sqlalchemy.url = sqlite:///fsperf-results.db 64 | 65 | [post_write_hooks] 66 | # post_write_hooks defines scripts or Python functions that are run 67 | # on newly generated revision scripts. See the documentation for further 68 | # detail and examples 69 | 70 | # format using "black" - use the console_scripts runner, against the "black" entrypoint 71 | # hooks = black 72 | # black.type = console_scripts 73 | # black.entrypoint = black 74 | # black.options = -l 79 REVISION_SCRIPT_FILENAME 75 | 76 | # lint with attempts to fix using "ruff" - use the exec runner, execute a binary 77 | # hooks = ruff 78 | # ruff.type = exec 79 | # ruff.executable = %(here)s/.venv/bin/ruff 80 | # ruff.options = --fix REVISION_SCRIPT_FILENAME 81 | 82 | # Logging configuration 83 | [loggers] 84 | keys = root,sqlalchemy,alembic 85 | 86 | [handlers] 87 | keys = console 88 | 89 | [formatters] 90 | keys = generic 91 | 92 | [logger_root] 93 | level = WARN 94 | handlers = console 95 | qualname = 96 | 97 | [logger_sqlalchemy] 98 | level = WARN 99 | handlers = 100 | qualname = sqlalchemy.engine 101 | 102 | [logger_alembic] 103 | level = INFO 104 | handlers = 105 | qualname = alembic 106 | 107 | [handler_console] 108 | class = StreamHandler 109 | args = (sys.stderr,) 110 | level = NOTSET 111 | formatter = generic 112 | 113 | [formatter_generic] 114 | format = %(levelname)-5.5s [%(name)s] %(message)s 115 | datefmt = %H:%M:%S 116 | -------------------------------------------------------------------------------- /alembic/README: -------------------------------------------------------------------------------- 1 | Generic single-database configuration. -------------------------------------------------------------------------------- /alembic/env.py: -------------------------------------------------------------------------------- 1 | from logging.config import fileConfig 2 | 3 | from sqlalchemy import engine_from_config 4 | from sqlalchemy import pool 5 | 6 | from alembic import context 7 | 8 | # this is the Alembic Config object, which provides 9 | # access to the values within the .ini file in use. 10 | config = context.config 11 | 12 | # Interpret the config file for Python logging. 13 | # This line sets up loggers basically. 14 | if config.config_file_name is not None: 15 | fileConfig(config.config_file_name) 16 | 17 | # add your model's MetaData object here 18 | # for 'autogenerate' support 19 | # from myapp import mymodel 20 | # target_metadata = mymodel.Base.metadata 21 | from src import ResultData 22 | target_metadata = ResultData.Base.metadata 23 | 24 | # other values from the config, defined by the needs of env.py, 25 | # can be acquired: 26 | # my_important_option = config.get_main_option("my_important_option") 27 | # ... etc. 28 | 29 | 30 | def run_migrations_offline() -> None: 31 | """Run migrations in 'offline' mode. 32 | 33 | This configures the context with just a URL 34 | and not an Engine, though an Engine is acceptable 35 | here as well. By skipping the Engine creation 36 | we don't even need a DBAPI to be available. 37 | 38 | Calls to context.execute() here emit the given string to the 39 | script output. 40 | 41 | """ 42 | url = config.get_main_option("sqlalchemy.url") 43 | context.configure( 44 | url=url, 45 | target_metadata=target_metadata, 46 | literal_binds=True, 47 | dialect_opts={"paramstyle": "named"}, 48 | ) 49 | 50 | with context.begin_transaction(): 51 | context.run_migrations() 52 | 53 | 54 | def run_migrations_online() -> None: 55 | """Run migrations in 'online' mode. 56 | 57 | In this scenario we need to create an Engine 58 | and associate a connection with the context. 59 | 60 | """ 61 | connectable = engine_from_config( 62 | config.get_section(config.config_ini_section, {}), 63 | prefix="sqlalchemy.", 64 | poolclass=pool.NullPool, 65 | ) 66 | 67 | with connectable.connect() as connection: 68 | context.configure( 69 | connection=connection, target_metadata=target_metadata 70 | ) 71 | 72 | with context.begin_transaction(): 73 | context.run_migrations() 74 | 75 | 76 | if context.is_offline_mode(): 77 | run_migrations_offline() 78 | else: 79 | run_migrations_online() 80 | -------------------------------------------------------------------------------- /alembic/script.py.mako: -------------------------------------------------------------------------------- 1 | """${message} 2 | 3 | Revision ID: ${up_revision} 4 | Revises: ${down_revision | comma,n} 5 | Create Date: ${create_date} 6 | 7 | """ 8 | from typing import Sequence, Union 9 | 10 | from alembic import op 11 | import sqlalchemy as sa 12 | ${imports if imports else ""} 13 | 14 | # revision identifiers, used by Alembic. 15 | revision: str = ${repr(up_revision)} 16 | down_revision: Union[str, None] = ${repr(down_revision)} 17 | branch_labels: Union[str, Sequence[str], None] = ${repr(branch_labels)} 18 | depends_on: Union[str, Sequence[str], None] = ${repr(depends_on)} 19 | 20 | 21 | def upgrade() -> None: 22 | ${upgrades if upgrades else "pass"} 23 | 24 | 25 | def downgrade() -> None: 26 | ${downgrades if downgrades else "pass"} 27 | -------------------------------------------------------------------------------- /alembic/versions/f2b6313d4618_initial_create.py: -------------------------------------------------------------------------------- 1 | """initial create 2 | 3 | Revision ID: f2b6313d4618 4 | Revises: 5 | Create Date: 2024-10-04 16:05:59.876022 6 | 7 | """ 8 | from typing import Sequence, Union 9 | 10 | from alembic import op 11 | from sqlalchemy import Column, DateTime, Float, ForeignKey, Integer, String 12 | 13 | 14 | # revision identifiers, used by Alembic. 15 | revision: str = 'f2b6313d4618' 16 | down_revision: Union[str, None] = None 17 | branch_labels: Union[str, Sequence[str], None] = None 18 | depends_on: Union[str, Sequence[str], None] = None 19 | 20 | 21 | def upgrade() -> None: 22 | op.create_table( 23 | "runs", 24 | Column('id', Integer, primary_key=True), 25 | Column('kernel', String), 26 | Column('config', String), 27 | Column('name', String), 28 | Column('hostname', String), 29 | Column('purpose', String), 30 | Column('time', DateTime), 31 | ) 32 | 33 | op.create_table( 34 | "fio_results", 35 | Column('id', Integer, primary_key=True), 36 | Column('run_id', Integer, ForeignKey('runs.id', ondelete="CASCADE"), nullable=False), 37 | Column('read_io_bytes', Integer, default=0), 38 | Column('elapsed', Integer, default=0), 39 | Column('sys_cpu', Float, default=0.0), 40 | Column('read_lat_ns_min', Integer, default=0), 41 | Column('read_lat_ns_max', Integer, default=0), 42 | Column('read_clat_ns_p50', Integer, default=0), 43 | Column('read_clat_ns_p99', Integer, default=0), 44 | Column('read_iops', Float, default=0), 45 | Column('read_io_kbytes', Integer, default=0), 46 | Column('read_bw_bytes', Integer, default=0), 47 | Column('write_lat_ns_min', Integer, default=0), 48 | Column('write_lat_ns_max', Integer, default=0), 49 | Column('write_iops', Float, default=0.0), 50 | Column('write_io_kbytes', Integer, default=0), 51 | Column('write_bw_bytes', Integer, default=0), 52 | Column('write_clat_ns_p50', Integer, default=0), 53 | Column('write_clat_ns_p99', Integer, default=0), 54 | Column('read_lat_ns_mean', Integer, default=0), 55 | Column('read_clat_ns_mean', Integer, default=0), 56 | Column('write_lat_ns_mean', Integer, default=0), 57 | Column('write_clat_ns_mean', Integer, default=0), 58 | ) 59 | 60 | op.create_table( 61 | "time_results", 62 | Column('id', Integer, primary_key=True), 63 | Column('run_id', Integer, ForeignKey('runs.id', ondelete="CASCADE"), nullable=False), 64 | Column('elapsed', Float, default=0.0), 65 | ) 66 | 67 | op.create_table( 68 | "dbench_results", 69 | Column('id', Integer, primary_key=True), 70 | Column('run_id', Integer, ForeignKey('runs.id', ondelete="CASCADE"), nullable=False), 71 | Column('throughput', Float, default=0.0), 72 | Column('ntcreatex', Float, default=0.0), 73 | Column('close', Float, default=0.0), 74 | Column('rename', Float, default=0.0), 75 | Column('unlink', Float, default=0.0), 76 | Column('deltree', Float, default=0.0), 77 | Column('mkdir', Float, default=0.0), 78 | Column('qpathinfo', Float, default=0.0), 79 | Column('qfileinfo', Float, default=0.0), 80 | Column('qfsinfo', Float, default=0.0), 81 | Column('sfileinfo', Float, default=0.0), 82 | Column('find', Float, default=0.0), 83 | Column('writex', Float, default=0.0), 84 | Column('readx', Float, default=0.0), 85 | Column('lockx', Float, default=0.0), 86 | Column('unlockx', Float, default=0.0), 87 | Column('flush', Float, default=0.0), 88 | ) 89 | 90 | op.create_table( 91 | "fragmentation", 92 | Column('id', Integer, primary_key=True), 93 | Column('run_id', Integer, ForeignKey('runs.id', ondelete="CASCADE"), nullable=False), 94 | Column('bg_count', Integer, default=0), 95 | Column('fragmented_bg_count', Integer, default=0), 96 | Column('frag_pct_mean', Float, default=0.0), 97 | Column('frag_pct_min', Float, default=0.0), 98 | Column('frag_pct_p50', Float, default=0.0), 99 | Column('frag_pct_p95', Float, default=0.0), 100 | Column('frag_pct_p99', Float, default=0.0), 101 | Column('frag_pct_max', Float, default=0.0), 102 | ) 103 | 104 | op.create_table( 105 | "latency_traces", 106 | Column('id', Integer, primary_key=True), 107 | Column('run_id', Integer, ForeignKey('runs.id', ondelete="CASCADE"), nullable=False), 108 | Column('function', String), 109 | Column('ns_mean', Float, default=0.0), 110 | Column('ns_min', Float, default=0.0), 111 | Column('ns_p50', Float, default=0.0), 112 | Column('ns_p95', Float, default=0.0), 113 | Column('ns_p99', Float, default=0.0), 114 | Column('ns_max', Float, default=0.0), 115 | Column('calls', Integer, default=0), 116 | ) 117 | 118 | op.create_table( 119 | "btrfs_commit_stats", 120 | Column('id', Integer, primary_key=True), 121 | Column('run_id', Integer, ForeignKey('runs.id', ondelete="CASCADE"), nullable=False), 122 | Column('commits', Integer, default=0), 123 | Column('avg_commit_ms', Float, default=0.0), 124 | Column('max_commit_ms', Integer, default=0), 125 | ) 126 | 127 | op.create_table( 128 | "mount_timings", 129 | Column('id', Integer, primary_key=True), 130 | Column('run_id', Integer, ForeignKey('runs.id', ondelete="CASCADE"), nullable=False), 131 | Column('end_state_umount_ns', Integer, default=0), 132 | Column('end_state_mount_ns', Integer, default=0), 133 | ) 134 | 135 | op.create_table( 136 | "io_stats", 137 | Column('id', Integer, primary_key=True), 138 | Column('run_id', Integer, ForeignKey('runs.id', ondelete="CASCADE"), nullable=False), 139 | Column('dev_read_iops', Integer, default=0), 140 | Column('dev_read_kbytes', Integer, default=0), 141 | Column('dev_write_iops', Integer, default=0), 142 | Column('dev_write_kbytes', Integer, default=0), 143 | ) 144 | pass 145 | 146 | 147 | def downgrade() -> None: 148 | pass 149 | -------------------------------------------------------------------------------- /frag_tests/buffered-append-vs-fallocate.fio: -------------------------------------------------------------------------------- 1 | [fallocate] 2 | ioengine=falloc 3 | rw=write 4 | blocksize=256M 5 | filesize=256M 6 | numjobs=4 7 | unlink_each_loop=1 8 | unlink=1 9 | loops=400 10 | thinktime=500ms 11 | runtime=20s 12 | stats=0 13 | 14 | [buffered-append] 15 | ioengine=sync 16 | rw=write 17 | blocksize=64K 18 | fdatasync=1 19 | filesize=128M 20 | file_append=1 21 | numjobs=32 22 | create_on_open=1 23 | thinktime=1ms 24 | new_group=1 25 | -------------------------------------------------------------------------------- /frag_tests/buffered-append-vs-fallocate.py: -------------------------------------------------------------------------------- 1 | from PerfTest import FioTest 2 | import os.path 3 | 4 | class BufferedAppendVsFallocate(FioTest): 5 | name = "bufferedappendvsfallocate" 6 | command = os.path.join(os.path.dirname(__file__), "buffered-append-vs-fallocate.fio") 7 | trace_fns = "find_free_extent" 8 | -------------------------------------------------------------------------------- /frag_tests/correlated-lifetimes.fio: -------------------------------------------------------------------------------- 1 | [large-short] 2 | ioengine=falloc 3 | rw=write 4 | blocksize=128M 5 | filesize=1G 6 | numjobs=1 7 | unlink_each_loop=1 8 | unlink=1 9 | loops=100 10 | 11 | [small-short] 12 | ioengine=falloc 13 | rw=write 14 | blocksize=128K 15 | filesize=128M 16 | numjobs=1 17 | unlink_each_loop=1 18 | loops=1000 19 | -------------------------------------------------------------------------------- /frag_tests/correlated-lifetimes.py: -------------------------------------------------------------------------------- 1 | from PerfTest import FioTest 2 | import os.path 3 | 4 | class CorrelatedLifetimes(FioTest): 5 | name = "correlatedlifetimes" 6 | command = os.path.join(os.path.dirname(__file__), "correlated-lifetimes.fio") 7 | trace_fns = "find_free_extent" 8 | -------------------------------------------------------------------------------- /frag_tests/four-sizes.fio: -------------------------------------------------------------------------------- 1 | [huge] 2 | ioengine=falloc 3 | rw=write 4 | blocksize=2G 5 | filesize=2G 6 | numjobs=1 7 | unlink_each_loop=1 8 | unlink=1 9 | loops=100 10 | stats=0 11 | 12 | [large] 13 | ioengine=falloc 14 | rw=write 15 | blocksize=128M 16 | filesize=1G 17 | numjobs=1 18 | unlink_each_loop=1 19 | unlink=1 20 | loops=100 21 | stats=0 22 | 23 | [medium] 24 | ioengine=falloc 25 | rw=write 26 | blocksize=8M 27 | filesize=1G 28 | numjobs=1 29 | unlink_each_loop=1 30 | unlink=1 31 | loops=100 32 | stats=0 33 | 34 | [small] 35 | ioengine=sync 36 | rw=write 37 | blocksize=128K 38 | filesize=128M 39 | numjobs=32 40 | create_on_open=1 41 | file_append=1 42 | fdatasync=1 43 | thinktime=1ms 44 | new_group=1 45 | -------------------------------------------------------------------------------- /frag_tests/four-sizes.py: -------------------------------------------------------------------------------- 1 | from PerfTest import FioTest 2 | import os.path 3 | 4 | class FourSizes(FioTest): 5 | name = "foursizes" 6 | command = os.path.join(os.path.dirname(__file__), "four-sizes.fio") 7 | trace_fns = "find_free_extent" 8 | -------------------------------------------------------------------------------- /frag_tests/funny-sizes-high.fio: -------------------------------------------------------------------------------- 1 | [large] 2 | ioengine=falloc 3 | rw=write 4 | blocksize=129M 5 | filesize=1G 6 | numjobs=1 7 | unlink_each_loop=1 8 | unlink=1 9 | loops=100 10 | stats=0 11 | 12 | [medium] 13 | ioengine=falloc 14 | rw=write 15 | blocksize=9M 16 | filesize=1G 17 | numjobs=1 18 | unlink_each_loop=1 19 | unlink=1 20 | loops=100 21 | stats=0 22 | 23 | [small] 24 | ioengine=sync 25 | rw=write 26 | blocksize=129K 27 | filesize=128M 28 | numjobs=32 29 | create_on_open=1 30 | file_append=1 31 | fdatasync=1 32 | thinktime=1ms 33 | new_group=1 34 | -------------------------------------------------------------------------------- /frag_tests/funny-sizes-high.py: -------------------------------------------------------------------------------- 1 | from PerfTest import FioTest 2 | import os.path 3 | 4 | class FunnySizesHigh(FioTest): 5 | name = "funnysizeshigh" 6 | command = os.path.join(os.path.dirname(__file__), "funny-sizes-high.fio") 7 | trace_fns = "find_free_extent" 8 | -------------------------------------------------------------------------------- /frag_tests/funny-sizes-low.fio: -------------------------------------------------------------------------------- 1 | [large] 2 | ioengine=falloc 3 | rw=write 4 | blocksize=127M 5 | filesize=1G 6 | numjobs=1 7 | unlink_each_loop=1 8 | unlink=1 9 | loops=100 10 | stats=0 11 | 12 | [medium] 13 | ioengine=falloc 14 | rw=write 15 | blocksize=7M 16 | filesize=1G 17 | numjobs=1 18 | unlink_each_loop=1 19 | unlink=1 20 | loops=100 21 | stats=0 22 | 23 | [small] 24 | ioengine=sync 25 | rw=write 26 | blocksize=127K 27 | filesize=128M 28 | numjobs=32 29 | create_on_open=1 30 | file_append=1 31 | fdatasync=1 32 | thinktime=1ms 33 | new_group=1 34 | -------------------------------------------------------------------------------- /frag_tests/funny-sizes-low.py: -------------------------------------------------------------------------------- 1 | from PerfTest import FioTest 2 | import os.path 3 | 4 | class FunnySizesLow(FioTest): 5 | name = "funnysizeslow" 6 | command = os.path.join(os.path.dirname(__file__), "funny-sizes-low.fio") 7 | trace_fns = "find_free_extent" 8 | -------------------------------------------------------------------------------- /frag_tests/mixed-lifetimes.fio: -------------------------------------------------------------------------------- 1 | [large-long] 2 | ioengine=falloc 3 | rw=write 4 | blocksize=128M 5 | filesize=1G 6 | numjobs=1 7 | unlink_each_loop=1 8 | unlink=1 9 | loops=100 10 | thinktime=10ms 11 | stats=0 12 | 13 | [large-short] 14 | ioengine=falloc 15 | rw=write 16 | blocksize=128M 17 | filesize=1G 18 | numjobs=1 19 | unlink_each_loop=1 20 | unlink=1 21 | loops=100 22 | stats=0 23 | 24 | [small-short] 25 | ioengine=falloc 26 | rw=write 27 | blocksize=128K 28 | filesize=128M 29 | numjobs=1 30 | unlink_each_loop=1 31 | unlink=1 32 | loops=1000 33 | stats=0 34 | 35 | [small-long] 36 | ioengine=sync 37 | rw=write 38 | blocksize=128K 39 | filesize=128M 40 | numjobs=32 41 | create_on_open=1 42 | file_append=1 43 | fdatasync=1 44 | thinktime=1ms 45 | new_group=1 46 | -------------------------------------------------------------------------------- /frag_tests/mixed-lifetimes.py: -------------------------------------------------------------------------------- 1 | from PerfTest import FioTest 2 | import os.path 3 | 4 | class MixedLifetimes(FioTest): 5 | name = "mixedlifetimes" 6 | command = os.path.join(os.path.dirname(__file__), "mixed-lifetimes.fio") 7 | trace_fns = "find_free_extent" 8 | -------------------------------------------------------------------------------- /fsperf: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | MYPATH=$(realpath $0) 4 | cd $(dirname $MYPATH) 5 | 6 | alembic upgrade head 7 | python3 src/fsperf.py "$@" 8 | -------------------------------------------------------------------------------- /fsperf-clean-results: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | python3 src/clean-results.py "$@" 4 | -------------------------------------------------------------------------------- /fsperf-compare: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | MYPATH=$(realpath $0) 4 | cd $(dirname $MYPATH) 5 | 6 | if [ ! -f "fsperf-results.db" ]; then 7 | echo "Need an fsperf-results database. Run fsperf!" 8 | exit 1 9 | fi 10 | 11 | python3 src/compare.py "$@" 12 | -------------------------------------------------------------------------------- /fsperf-generate-graph: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | python3 src/generate-graph.py "$@" 4 | -------------------------------------------------------------------------------- /fsperf-generate-results: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | python3 src/generate-results-page.py "$@" 4 | -------------------------------------------------------------------------------- /fsperf-sqlite.sql: -------------------------------------------------------------------------------- 1 | CREATE TABLE IF NOT EXISTS `fio_runs` ( 2 | `id` INTEGER PRIMARY KEY AUTOINCREMENT, 3 | `kernel` datetime NOT NULL, 4 | `config` varchar(256) NOT NULL, 5 | `name` varchar(256) NOT NULL, 6 | `time` datetime NOT NULL 7 | ); 8 | CREATE TABLE IF NOT EXISTS `fio_jobs` ( 9 | `run_id` int NOT NULL, 10 | `latency_window` int NOT NULL, 11 | `trim_lat_ns_mean` float NOT NULL, 12 | `read_iops_min` int NOT NULL, 13 | `read_bw_dev` float NOT NULL, 14 | `trim_runtime` int NOT NULL, 15 | `read_io_bytes` int NOT NULL, 16 | `read_short_ios` int NOT NULL, 17 | `read_iops_samples` int NOT NULL, 18 | `minf` int NOT NULL, 19 | `read_drop_ios` int NOT NULL, 20 | `trim_iops_samples` int NOT NULL, 21 | `trim_iops_max` int NOT NULL, 22 | `trim_bw_agg` float NOT NULL, 23 | `write_bw_min` int NOT NULL, 24 | `write_iops_mean` float NOT NULL, 25 | `read_bw_max` int NOT NULL, 26 | `read_bw_min` int NOT NULL, 27 | `trim_bw_dev` float NOT NULL, 28 | `read_iops_max` int NOT NULL, 29 | `read_total_ios` int NOT NULL, 30 | `read_lat_ns_mean` float NOT NULL, 31 | `write_iops` float NOT NULL, 32 | `latency_target` int NOT NULL, 33 | `trim_bw` int NOT NULL, 34 | `eta` int NOT NULL, 35 | `read_bw_samples` int NOT NULL, 36 | `trim_io_kbytes` int NOT NULL, 37 | `write_iops_max` int NOT NULL, 38 | `write_drop_ios` int NOT NULL, 39 | `trim_iops_min` int NOT NULL, 40 | `write_bw_samples` int NOT NULL, 41 | `read_iops_stddev` float NOT NULL, 42 | `write_io_kbytes` int NOT NULL, 43 | `trim_bw_mean` float NOT NULL, 44 | `write_bw_agg` float NOT NULL, 45 | `write_bw_dev` float NOT NULL, 46 | `write_lat_ns_stddev` float NOT NULL, 47 | `trim_lat_ns_stddev` float NOT NULL, 48 | `groupid` int NOT NULL, 49 | `latency_depth` int NOT NULL, 50 | `trim_short_ios` int NOT NULL, 51 | `read_lat_ns_stddev` float NOT NULL, 52 | `write_iops_min` int NOT NULL, 53 | `write_iops_stddev` float NOT NULL, 54 | `read_io_kbytes` int NOT NULL, 55 | `trim_bw_samples` int NOT NULL, 56 | `trim_lat_ns_min` int NOT NULL, 57 | `error` int NOT NULL, 58 | `read_bw_mean` float NOT NULL, 59 | `trim_iops_mean` float NOT NULL, 60 | `elapsed` int NOT NULL, 61 | `write_bw_mean` float NOT NULL, 62 | `write_short_ios` int NOT NULL, 63 | `ctx` int NOT NULL, 64 | `write_io_bytes` int NOT NULL, 65 | `usr_cpu` float NOT NULL, 66 | `trim_drop_ios` int NOT NULL, 67 | `write_bw` int NOT NULL, 68 | `jobname` varchar(256) NOT NULL, 69 | `trim_bw_min` int NOT NULL, 70 | `read_runtime` int NOT NULL, 71 | `sys_cpu` float NOT NULL, 72 | `trim_lat_ns_max` int NOT NULL, 73 | `read_iops_mean` float NOT NULL, 74 | `write_lat_ns_min` int NOT NULL, 75 | `trim_iops_stddev` float NOT NULL, 76 | `write_lat_ns_max` int NOT NULL, 77 | `majf` int NOT NULL, 78 | `write_total_ios` int NOT NULL, 79 | `read_bw` int NOT NULL, 80 | `read_lat_ns_min` int NOT NULL, 81 | `trim_bw_max` int NOT NULL, 82 | `write_iops_samples` int NOT NULL, 83 | `write_runtime` int NOT NULL, 84 | `trim_io_bytes` int NOT NULL, 85 | `latency_percentile` float NOT NULL, 86 | `read_iops` float NOT NULL, 87 | `trim_total_ios` int NOT NULL, 88 | `write_lat_ns_mean` float NOT NULL, 89 | `write_bw_max` int NOT NULL, 90 | `read_bw_agg` float NOT NULL, 91 | `read_lat_ns_max` int NOT NULL, 92 | `trim_iops` float NOT NULL 93 | ); 94 | -------------------------------------------------------------------------------- /local-cfg-example: -------------------------------------------------------------------------------- 1 | [main] 2 | directory=/mnt/test 3 | cpugovernor=performance 4 | 5 | [btrfs] 6 | device=/dev/nvme0n1 7 | mkfs=mkfs.btrfs -f 8 | mount=mount -o noatime 9 | 10 | [btrfs-32k] 11 | device=/dev/nvme0n1 12 | mkfs=mkfs.btrfs -f 13 | mount=mount -o noatime 14 | blocksize=32k 15 | 16 | [btrfs-raid1] 17 | device=/dev/vg/scratch0 18 | mkfs=mkfs.btrfs -f -draid1 -mraid1 /dev/vg/scratch1 19 | mount=mount -o noatime 20 | readpolicy=pid 21 | 22 | [xfs] 23 | device=/dev/nvme0n1 24 | iosched=none 25 | mkfs=mkfs.xfs -f 26 | mount=mount -o noatime 27 | -------------------------------------------------------------------------------- /manage.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | from migrate.versioning.shell import main 3 | 4 | if __name__ == '__main__': 5 | main(repository='fsperf-db', url='sqlite:///fsperf-results.db', debug='False') 6 | -------------------------------------------------------------------------------- /src/.gitignore: -------------------------------------------------------------------------------- 1 | *.pyc 2 | -------------------------------------------------------------------------------- /src/FioCompare.py: -------------------------------------------------------------------------------- 1 | OK = '\033[92m' 2 | FAIL = '\033[91m' 3 | ENDC = '\033[0m' 4 | 5 | default_keys = [ 'iops', 'io_kbytes', 'bw' ] 6 | latency_keys = [ 'lat_ns_min', 'lat_ns_max' ] 7 | main_job_keys = [ 'sys_cpu', 'elapsed' ] 8 | io_ops = ['read', 'write', 'trim' ] 9 | 10 | def _fuzzy_compare(a, b, fuzzy): 11 | if a == b: 12 | return 0 13 | if a == 0: 14 | return 100 15 | a = float(a) 16 | b = float(b) 17 | fuzzy = float(fuzzy) 18 | val = ((b - a) / a) * 100 19 | if val > fuzzy or val < -fuzzy: 20 | return val; 21 | return 0 22 | 23 | def _compare_jobs(ijob, njob, latency, fuzz): 24 | failed = 0 25 | for k in default_keys: 26 | for io in io_ops: 27 | key = "{}_{}".format(io, k) 28 | comp = _fuzzy_compare(ijob[key], njob[key], fuzz) 29 | if comp < 0: 30 | outstr = " {} regressed: old {} new {} {}%".format(key, 31 | ijob[key], njob[key], comp) 32 | print(FAIL + outstr + ENDC) 33 | failed += 1 34 | elif comp > 0: 35 | outstr = " {} improved: old {} new {} {}%".format(key, 36 | ijob[key], njob[key], comp) 37 | print(OK + outstr + ENDC) 38 | for k in latency_keys: 39 | if not latency: 40 | break 41 | for io in io_ops: 42 | key = "{}_{}".format(io, k) 43 | comp = _fuzzy_compare(ijob[key], njob[key], fuzz) 44 | if comp > 0: 45 | outstr = " {} regressed: old {} new {} {}%".format(key, 46 | ijob[key], njob[key], comp) 47 | print(FAIL + outstr + ENDC) 48 | failed += 1 49 | elif comp < 0: 50 | outstr = " {} improved: old {} new {} {}%".format(key, 51 | ijob[key], njob[key], comp) 52 | print(OK + outstr + ENDC) 53 | for k in main_job_keys: 54 | comp = _fuzzy_compare(ijob[k], njob[k], fuzz) 55 | if comp > 0: 56 | outstr = " {} regressed: old {} new {} {}%".format(k, ijob[k], 57 | njob[k], comp) 58 | print(FAIL + outstr + ENDC) 59 | failed += 1 60 | elif comp < 0: 61 | outstr = " {} improved: old {} new {} {}%".format(k, ijob[k], 62 | njob[k], comp) 63 | print(OK + outstr + ENDC) 64 | return failed 65 | 66 | def compare_individual_jobs(initial, data, fuzz): 67 | failed = 0; 68 | initial_jobs = initial['jobs'][:] 69 | for njob in data['jobs']: 70 | for ijob in initial_jobs: 71 | if njob['jobname'] == ijob['jobname']: 72 | print(" Checking results for {}".format(njob['jobname'])) 73 | failed += _compare_jobs(ijob, njob, fuzz) 74 | initial_jobs.remove(ijob) 75 | break 76 | return failed 77 | 78 | def default_merge(data): 79 | '''Default merge function for multiple jobs in one run 80 | 81 | For runs that include multiple threads we will have a lot of variation 82 | between the different threads, which makes comparing them to eachother 83 | across multiple runs less that useful. Instead merge the jobs into a single 84 | job. This function does that by adding up 'iops', 'io_kbytes', and 'bw' for 85 | read/write/trim in the merged job, and then taking the maximal values of the 86 | latency numbers. 87 | ''' 88 | merge_job = {} 89 | for job in data['jobs']: 90 | for k in main_job_keys: 91 | if k not in merge_job: 92 | merge_job[k] = job[k] 93 | else: 94 | merge_job[k] += job[k] 95 | for io in io_ops: 96 | for k in default_keys: 97 | key = "{}_{}".format(io, k) 98 | if key not in merge_job: 99 | merge_job[key] = job[key] 100 | else: 101 | merge_job[key] += job[key] 102 | for k in latency_keys: 103 | key = "{}_{}".format(io, k) 104 | if key not in merge_job: 105 | merge_job[key] = job[key] 106 | elif merge_job[key] < job[key]: 107 | merge_job[key] = job[key] 108 | return merge_job 109 | 110 | def compare_fiodata(initial, data, latency, merge_func=default_merge, fuzz=5): 111 | failed = 0 112 | if merge_func is None: 113 | return compare_individual_jobs(initial, data, fuzz) 114 | ijob = merge_func(initial) 115 | njob = merge_func(data) 116 | return _compare_jobs(ijob, njob, latency, fuzz) 117 | -------------------------------------------------------------------------------- /src/FioResultDecoder.py: -------------------------------------------------------------------------------- 1 | import json 2 | 3 | class FioResultDecoder(json.JSONDecoder): 4 | """Decoder for decoding fio result json to an object for our database 5 | 6 | This decodes the json output from fio into an object that can be directly 7 | inserted into our database. This just strips out the fields we don't care 8 | about and collapses the read/write/trim classes into a flat value structure 9 | inside of the jobs object. 10 | 11 | For example 12 | "write" : { 13 | "io_bytes" : 313360384, 14 | "bw" : 1016, 15 | } 16 | 17 | Get's collapsed to 18 | 19 | "write_io_bytes" : 313360384, 20 | "write_bw": 1016, 21 | 22 | Currently any dict under 'jobs' get's dropped, with the exception of 'read', 23 | 'write', and 'trim'. For those sub sections we drop any dict's under those. 24 | 25 | Attempt to keep this as generic as possible, we don't want to break every 26 | time fio changes it's json output format. 27 | """ 28 | _ignore_types = ['dict', 'list'] 29 | _override_keys = ['lat_ns', 'clat_ns'] 30 | _io_ops = ['read', 'write', 'trim'] 31 | 32 | def _extract_percentiles(self, new_job, iotype, key, percentiles): 33 | for p,pval in percentiles.items(): 34 | p = float(p) 35 | if p.is_integer(): 36 | p = int(p) 37 | collapsed_key = "{}_{}_p{}".format(iotype, key, p) 38 | new_job[collapsed_key] = pval 39 | 40 | def decode(self, json_string): 41 | """This does the dirty work of converting everything""" 42 | default_obj = super(FioResultDecoder, self).decode(json_string) 43 | obj = {} 44 | obj['jobs'] = [] 45 | for job in default_obj['jobs']: 46 | new_job = {} 47 | for key,value in job.items(): 48 | if key not in self._io_ops: 49 | if value.__class__.__name__ in self._ignore_types: 50 | continue 51 | new_job[key] = value 52 | continue 53 | for k,v in value.items(): 54 | if k in self._override_keys: 55 | for subk,subv in v.items(): 56 | if subk == "percentile": 57 | self._extract_percentiles(new_job, key, k, subv) 58 | continue 59 | collapsed_key = "{}_{}_{}".format(key, k, subk) 60 | new_job[collapsed_key] = subv 61 | continue 62 | if v.__class__.__name__ in self._ignore_types: 63 | continue 64 | collapsed_key = "{}_{}".format(key, k) 65 | new_job[collapsed_key] = v 66 | obj['jobs'].append(new_job) 67 | return obj 68 | -------------------------------------------------------------------------------- /src/PerfTest.py: -------------------------------------------------------------------------------- 1 | import FioResultDecoder 2 | import ResultData 3 | import utils 4 | import json 5 | from timeit import default_timer as timer 6 | import contextlib 7 | 8 | RESULTS_DIR = "results" 9 | FRAG_DIR = "src/frag" 10 | 11 | class PerfTest: 12 | name = "" 13 | command = "" 14 | trace_fns = "" 15 | need_remount_after_setup = False 16 | skip_mkfs_and_mount = False 17 | end_state_umount_s = 0 18 | end_state_mount_s = 0 19 | 20 | # Set this if the test does something specific and isn't going to use the 21 | # configuration options to change how the test is run. 22 | oneoff = False 23 | 24 | def maybe_cycle_mount(self, mnt): 25 | if self.need_remount_after_setup: 26 | mnt.cycle_mount() 27 | 28 | def run(self, run, config, section, results): 29 | with self.test_context(config, section): 30 | self.maybe_cycle_mount(self.mnt) 31 | with utils.IOStats(self.dev) as ios: 32 | with utils.LatencyTracing(self.what_latency_traces(config, section)) as lt: 33 | self.test(run, config, results) 34 | self.io_stats = ios.results() 35 | self.latency_traces = lt.results() 36 | self.commit_stats = utils.collect_commit_stats(self.dev) 37 | self.end_state_umount_s, self.end_state_mount_s = self.mnt.timed_cycle_mount() 38 | self.collect_fragmentation(run, config) 39 | self.record_results(run) 40 | 41 | # do generic setup (mkfs/mount), then test-specific setup. 42 | # use ExitStack to ensure we call umount/teardown appropriately 43 | def test_context(self, config, section): 44 | self.dev = utils.mkfs(self, config, section) 45 | stack = contextlib.ExitStack() 46 | if utils.want_mnt(self, config, section): 47 | self.mnt = utils.Mount( 48 | config.get(section, 'mount'), 49 | config.get(section, 'device'), 50 | config.get('main', 'directory')) 51 | stack.enter_context(self.mnt) 52 | self.setup(config, section) 53 | stack.callback(self.teardown, config, RESULTS_DIR) 54 | return stack 55 | 56 | # override for special per-test setup 57 | def setup(self, config, section): 58 | pass 59 | 60 | def record_results(self, run): 61 | for lt in self.latency_traces: 62 | ltr = ResultData.LatencyTrace() 63 | ltr.load_from_dict(lt) 64 | run.latency_traces.append(ltr) 65 | ios = ResultData.IOStats() 66 | ios.load_from_dict(self.io_stats) 67 | run.io_stats.append(ios) 68 | mt = ResultData.MountTiming(self.end_state_umount_s, self.end_state_mount_s) 69 | run.mount_timings.append(mt) 70 | f = ResultData.Fragmentation() 71 | f.load_from_dict(self.fragmentation) 72 | run.fragmentation.append(f) 73 | if self.commit_stats and 'commits' in self.commit_stats: 74 | stats = ResultData.BtrfsCommitStats() 75 | stats.load_from_dict(self.commit_stats) 76 | run.btrfs_commit_stats.append(stats) 77 | 78 | # must override, this is the actual test logic! 79 | def test(self, config): 80 | raise NotImplementedError 81 | 82 | # override for special per-test teardown (to undo setup) 83 | def teardown(self, config, results): 84 | pass 85 | 86 | def collect_fragmentation(self, run, config): 87 | bg_dump_filename = f"{RESULTS_DIR}/bgs.txt" 88 | utils.generate_bg_dump(config, FRAG_DIR) 89 | with open(bg_dump_filename, 'w') as f: 90 | try: 91 | utils.run_command(f"btrd {FRAG_DIR}/bg-dump.btrd", f) 92 | except Exception as e: 93 | print(f"failed to collect fragmentation data: {e}. (Likely, running the btrd script OOMed)") 94 | self.fragmentation = {} 95 | return 96 | frag_filename = f"{RESULTS_DIR}/{self.name}.frag" 97 | with open(frag_filename, 'w') as f: 98 | try: 99 | utils.run_command(f"{FRAG_DIR}/target/release/btrfs-frag-view {RESULTS_DIR}/bgs.txt", f) 100 | except Exception as e: 101 | print(f"failed to analyze fragmentation data: {e}.") 102 | self.fragmentation = {} 103 | return 104 | self.fragmentation = json.load(open(frag_filename)) 105 | 106 | def what_latency_traces(self, config, section): 107 | trace_fns = "" 108 | if self.trace_fns: 109 | trace_fns = self.trace_fns 110 | elif config.has_option(section, 'trace_fns'): 111 | trace_fns = config.get(section, 'trace_fns') 112 | return [fn for fn in trace_fns.split(",") if fn] 113 | 114 | class FioTest(PerfTest): 115 | def record_results(self, run): 116 | PerfTest.record_results(self, run) 117 | json_data = open("{}/{}.json".format(RESULTS_DIR, self.name)) 118 | data = json.load(json_data, cls=FioResultDecoder.FioResultDecoder) 119 | for j in data['jobs']: 120 | r = ResultData.FioResult() 121 | r.load_from_dict(j) 122 | run.fio_results.append(r) 123 | 124 | def default_cmd(self, results): 125 | command = "fio --output-format=json" 126 | command += " --output={}/{}.json".format(RESULTS_DIR, self.name) 127 | command += " --alloc-size 98304 --allrandrepeat=1 --randseed=12345 --group_reporting=1" 128 | return command 129 | 130 | def test(self, run, config, results): 131 | directory = config.get('main', 'directory') 132 | command = self.default_cmd(results) 133 | command += " --directory {} ".format(directory) 134 | command += self.command 135 | utils.run_command(command) 136 | 137 | class TimeTest(PerfTest): 138 | def record_results(self, run): 139 | PerfTest.record_results(self, run) 140 | r = ResultData.TimeResult() 141 | r.elapsed = self.elapsed 142 | run.time_results.append(r) 143 | 144 | def test(self, run, config, results): 145 | directory = config.get('main', 'directory') 146 | command = self.command.replace('DIRECTORY', directory) 147 | start = timer() 148 | utils.run_command(command) 149 | self.elapsed = timer() - start 150 | 151 | class DbenchTest(PerfTest): 152 | def record_results(self, run): 153 | PerfTest.record_results(self, run) 154 | r = ResultData.DbenchResult() 155 | r.load_from_dict(self.results) 156 | run.dbench_results.append(r) 157 | 158 | def test(self, run, config, results): 159 | directory = config.get('main', 'directory') 160 | command = "dbench " + self.command + " -D {}".format(directory) 161 | fd = open("{}/{}.txt".format(RESULTS_DIR, self.name), "w+") 162 | utils.run_command(command, fd) 163 | fd.seek(0) 164 | parse = False 165 | self.results = {} 166 | for line in fd: 167 | if not parse: 168 | if "----" in line: 169 | parse = True 170 | continue 171 | vals = line.split() 172 | if len(vals) == 4: 173 | key = vals[0].lower() 174 | self.results[key] = vals[3] 175 | elif len(vals) > 4: 176 | self.results['throughput'] = vals[1] 177 | -------------------------------------------------------------------------------- /src/ResultData.py: -------------------------------------------------------------------------------- 1 | import datetime 2 | import numbers 3 | import socket 4 | from sqlalchemy.ext.declarative import declarative_base 5 | from sqlalchemy import (Table, Column, Integer, String, ForeignKey, DateTime, 6 | Float) 7 | from sqlalchemy.orm import relationship 8 | 9 | Base = declarative_base() 10 | 11 | class Run(Base): 12 | __tablename__ = "runs" 13 | 14 | id = Column(Integer, primary_key=True) 15 | kernel = Column(String) 16 | config = Column(String) 17 | name = Column(String) 18 | hostname = Column(String, default=socket.gethostname()) 19 | purpose = Column(String, default="continuous") 20 | time = Column(DateTime, default=datetime.datetime.utcnow) 21 | 22 | time_results = relationship("TimeResult", backref="runs", 23 | order_by="TimeResult.id", 24 | cascade="all,delete") 25 | fio_results = relationship("FioResult", backref="runs", 26 | order_by="FioResult.id", 27 | cascade="all,delete") 28 | dbench_results = relationship("DbenchResult", backref="runs", 29 | order_by="DbenchResult.id", 30 | cascade="all,delete") 31 | fragmentation = relationship("Fragmentation", backref="runs", 32 | order_by="Fragmentation.id", 33 | cascade="all,delete") 34 | latency_traces = relationship("LatencyTrace", backref="runs", 35 | order_by="LatencyTrace.id", 36 | cascade="all,delete") 37 | io_stats = relationship("IOStats", backref="runs", 38 | order_by="IOStats.id", 39 | cascade="all,delete") 40 | btrfs_commit_stats = relationship("BtrfsCommitStats", 41 | backref="runs", 42 | order_by="BtrfsCommitStats.id", 43 | cascade="all,delete") 44 | mount_timings = relationship("MountTiming", backref="runs", 45 | order_by="MountTiming.id", 46 | cascade="all,delete") 47 | 48 | def is_stat(key, value): 49 | return not "id" in key and isinstance(value, numbers.Number) 50 | 51 | def result_to_dict(result): 52 | return { k: v for (k, v) in vars(result).items() if is_stat(k, v) } 53 | 54 | class FioResult(Base): 55 | __tablename__ = 'fio_results' 56 | 57 | id = Column(Integer, primary_key=True) 58 | run_id = Column(ForeignKey('runs.id', ondelete="CASCADE"), nullable=False) 59 | read_io_bytes = Column(Integer, default=0) 60 | elapsed = Column(Integer, default=0) 61 | sys_cpu = Column(Float, default=0.0) 62 | read_lat_ns_min = Column(Integer, default=0) 63 | read_lat_ns_max = Column(Integer, default=0) 64 | read_lat_ns_mean = Column(Integer, default=0) 65 | read_clat_ns_p50 = Column(Integer, default=0) 66 | read_clat_ns_p99 = Column(Integer, default=0) 67 | read_clat_ns_mean = Column(Integer, default=0) 68 | read_iops = Column(Float, default=0) 69 | read_io_kbytes = Column(Integer, default=0) 70 | read_bw_bytes = Column(Integer, default=0) 71 | write_lat_ns_min = Column(Integer, default=0) 72 | write_lat_ns_max = Column(Integer, default=0) 73 | write_lat_ns_mean = Column(Integer, default=0) 74 | write_clat_ns_p50 = Column(Integer, default=0) 75 | write_clat_ns_p99 = Column(Integer, default=0) 76 | write_clat_ns_mean = Column(Integer, default=0) 77 | write_iops = Column(Float, default=0.0) 78 | write_io_kbytes = Column(Integer, default=0) 79 | write_bw_bytes = Column(Integer, default=0) 80 | 81 | def load_from_dict(self, inval): 82 | for k in dir(self): 83 | if k not in inval: 84 | continue 85 | setattr(self, k, inval[k]) 86 | 87 | def to_dict(self): 88 | return result_to_dict(self) 89 | 90 | class TimeResult(Base): 91 | __tablename__ = 'time_results' 92 | id = Column(Integer, primary_key=True) 93 | run_id = Column(ForeignKey('runs.id', ondelete="CASCADE"), nullable=False) 94 | elapsed = Column(Float, default=0.0) 95 | 96 | def to_dict(self): 97 | return result_to_dict(self) 98 | 99 | class DbenchResult(Base): 100 | __tablename__ = 'dbench_results' 101 | id = Column(Integer, primary_key=True) 102 | run_id = Column(ForeignKey('runs.id', ondelete="CASCADE"), nullable=False) 103 | throughput = Column(Float, default=0.0) 104 | ntcreatex = Column(Float, default=0.0) 105 | close = Column(Float, default=0.0) 106 | rename = Column(Float, default=0.0) 107 | unlink = Column(Float, default=0.0) 108 | deltree = Column(Float, default=0.0) 109 | mkdir = Column(Float, default=0.0) 110 | qpathinfo = Column(Float, default=0.0) 111 | qfileinfo = Column(Float, default=0.0) 112 | qfsinfo = Column(Float, default=0.0) 113 | sfileinfo = Column(Float, default=0.0) 114 | find = Column(Float, default=0.0) 115 | writex = Column(Float, default=0.0) 116 | readx = Column(Float, default=0.0) 117 | lockx = Column(Float, default=0.0) 118 | unlockx = Column(Float, default=0.0) 119 | flush = Column(Float, default=0.0) 120 | 121 | def load_from_dict(self, inval): 122 | for k in dir(self): 123 | if k not in inval: 124 | continue 125 | setattr(self, k, inval[k]) 126 | 127 | def to_dict(self): 128 | return result_to_dict(self) 129 | 130 | class Fragmentation(Base): 131 | __tablename__ = 'fragmentation' 132 | id = Column(Integer, primary_key=True) 133 | run_id = Column(ForeignKey('runs.id', ondelete="CASCADE"), nullable=False) 134 | bg_count = Column(Integer, default=0) 135 | fragmented_bg_count = Column(Integer, default=0) 136 | frag_pct_mean = Column(Float, default=0.0) 137 | frag_pct_min = Column(Float, default=0.0) 138 | frag_pct_p50 = Column(Float, default=0.0) 139 | frag_pct_p95 = Column(Float, default=0.0) 140 | frag_pct_p99 = Column(Float, default=0.0) 141 | frag_pct_max = Column(Float, default=0.0) 142 | 143 | def load_from_dict(self, inval): 144 | for k in dir(self): 145 | if k not in inval: 146 | continue 147 | setattr(self, k, inval[k]) 148 | 149 | def to_dict(self): 150 | return result_to_dict(self) 151 | 152 | class IOStats(Base): 153 | __tablename__ = 'io_stats' 154 | id = Column(Integer, primary_key=True) 155 | run_id = Column(ForeignKey('runs.id', ondelete="CASCADE"), nullable=False) 156 | dev_read_iops = Column(Integer, default=0) 157 | dev_read_kbytes = Column(Integer, default=0) 158 | dev_write_iops = Column(Integer, default=0) 159 | dev_write_kbytes = Column(Integer, default=0) 160 | 161 | def load_from_dict(self, inval): 162 | for k in dir(self): 163 | if k not in inval: 164 | continue 165 | setattr(self, k, inval[k]) 166 | 167 | def to_dict(self): 168 | return result_to_dict(self) 169 | 170 | class LatencyTrace(Base): 171 | __tablename__ = 'latency_traces' 172 | id = Column(Integer, primary_key=True) 173 | run_id = Column(ForeignKey('runs.id', ondelete="CASCADE"), nullable=False) 174 | function = Column(String) 175 | ns_mean = Column(Float, default=0.0) 176 | ns_min = Column(Float, default=0.0) 177 | ns_p50 = Column(Float, default=0.0) 178 | ns_p95 = Column(Float, default=0.0) 179 | ns_p99 = Column(Float, default=0.0) 180 | ns_max = Column(Float, default=0.0) 181 | calls = Column(Integer, default=0) 182 | 183 | def load_from_dict(self, inval): 184 | for k in dir(self): 185 | if k not in inval: 186 | continue 187 | setattr(self, k, inval[k]) 188 | 189 | def to_dict(self): 190 | items = result_to_dict(self).items() 191 | return { f"{self.function}_{k}": v for (k, v) in items } 192 | 193 | class BtrfsCommitStats(Base): 194 | __tablename__ = 'btrfs_commit_stats' 195 | id = Column(Integer, primary_key=True) 196 | run_id = Column(ForeignKey('runs.id', ondelete="CASCADE"), nullable=False) 197 | commits = Column(Integer, default=0) 198 | avg_commit_ms = Column(Float, default=0.0) 199 | max_commit_ms = Column(Integer, default=0) 200 | 201 | def load_from_dict(self, inval): 202 | for k in dir(self): 203 | if k not in inval: 204 | continue 205 | setattr(self, k, inval[k]) 206 | 207 | def to_dict(self): 208 | return result_to_dict(self) 209 | 210 | class MountTiming(Base): 211 | __tablename__ = 'mount_timings' 212 | id = Column(Integer, primary_key=True) 213 | run_id = Column(ForeignKey('runs.id', ondelete="CASCADE"), nullable=False) 214 | end_state_umount_ns = Column(Integer, default=0) 215 | end_state_mount_ns = Column(Integer, default=0) 216 | 217 | def __init__(self, umount, mount): 218 | self.end_state_umount_ns = umount 219 | self.end_state_mount_ns = mount 220 | 221 | def to_dict(self): 222 | return result_to_dict(self) 223 | -------------------------------------------------------------------------------- /src/clean-results.py: -------------------------------------------------------------------------------- 1 | from ResultData import * 2 | from sqlalchemy import create_engine 3 | from sqlalchemy.orm import sessionmaker 4 | import argparse 5 | import sys 6 | 7 | parser = argparse.ArgumentParser() 8 | parser.add_argument("--labels", nargs='*', type=str, default=[], 9 | help="Labels to delete the results for") 10 | parser.add_argument("--config", type=str, 11 | help="Configs to delete the results for") 12 | parser.add_argument("--test", type=str, 13 | help="Test name to delete results for") 14 | args = parser.parse_args() 15 | 16 | if not args.labels and not args.config and not args.test: 17 | print("Must specify either labels or configs to delete from") 18 | sys.exit(1) 19 | 20 | engine = create_engine('sqlite:///fsperf-results.db') 21 | Session = sessionmaker() 22 | Session.configure(bind=engine) 23 | session = Session() 24 | 25 | for p in args.labels: 26 | results = session.query(Run).\ 27 | outerjoin(FioResult).\ 28 | outerjoin(DbenchResult).\ 29 | outerjoin(TimeResult).\ 30 | outerjoin(Fragmentation).\ 31 | outerjoin(LatencyTrace).\ 32 | outerjoin(IOStats).\ 33 | outerjoin(BtrfsCommitStats).\ 34 | outerjoin(MountTiming).\ 35 | filter(Run.purpose == p).all() 36 | for r in results: 37 | session.delete(r) 38 | session.commit() 39 | 40 | if args.test is not None: 41 | results = session.query(Run).\ 42 | outerjoin(FioResult).\ 43 | outerjoin(DbenchResult).\ 44 | outerjoin(TimeResult).\ 45 | outerjoin(Fragmentation).\ 46 | outerjoin(LatencyTrace).\ 47 | outerjoin(IOStats).\ 48 | outerjoin(BtrfsCommitStats).\ 49 | outerjoin(MountTiming).\ 50 | filter(Run.name == args.test).all() 51 | for r in results: 52 | session.delete(r) 53 | session.commit() 54 | 55 | if args.config is not None: 56 | results = session.query(Run).\ 57 | outerjoin(FioResult).\ 58 | outerjoin(DbenchResult).\ 59 | outerjoin(TimeResult).\ 60 | outerjoin(Fragmentation).\ 61 | outerjoin(LatencyTrace).\ 62 | outerjoin(IOStats).\ 63 | outerjoin(BtrfsCommitStats).\ 64 | outerjoin(MountTiming).\ 65 | filter(Run.config == args.config).all() 66 | for r in results: 67 | session.delete(r) 68 | session.commit() 69 | -------------------------------------------------------------------------------- /src/compare.py: -------------------------------------------------------------------------------- 1 | import ResultData 2 | 3 | import argparse 4 | import configparser 5 | import datetime 6 | from sqlalchemy.orm import sessionmaker 7 | from sqlalchemy import create_engine 8 | import utils 9 | 10 | def compare_results(session, section_A, section_B, test, purpose_A, purpose_B, age): 11 | results_A = utils.get_results(session, test.name, section_A, purpose_A, age) 12 | results_B = utils.get_results(session, test.name, section_B, purpose_B, age) 13 | avg_A = utils.avg_results(results_A) 14 | avg_B = utils.avg_results(results_B) 15 | if not (avg_A and avg_B): 16 | return 17 | print(f"{test.name} results") 18 | utils.print_comparison_table(avg_A, avg_B) 19 | print("") 20 | 21 | if __name__ == "__main__": 22 | engine = create_engine('sqlite:///fsperf-results.db') 23 | ResultData.Base.metadata.create_all(engine) 24 | Session = sessionmaker() 25 | Session.configure(bind=engine) 26 | session = Session() 27 | 28 | parser = argparse.ArgumentParser() 29 | parser.add_argument('A', type=str, help='purpose A for a comparison') 30 | parser.add_argument('B', type=str, help='purpose B for a comparison') 31 | parser.add_argument('-F', '--fragmentation', action='store_true', help='include fragmentation tests') 32 | 33 | args = parser.parse_args() 34 | 35 | config = configparser.ConfigParser() 36 | config.read('local.cfg') 37 | sections = config.sections() 38 | sections.remove("main") 39 | 40 | tests, oneoffs = utils.get_tests("tests/") 41 | if args.fragmentation: 42 | frag_tests, _ = utils.get_tests("frag_tests/") 43 | tests.extend(frag_tests) 44 | 45 | age = datetime.date.today() - datetime.timedelta(days=365) 46 | 47 | for section in sections: 48 | print(f"{section} test results") 49 | for test in tests: 50 | compare_results(session, section, section, test, args.A, args.B, age) 51 | -------------------------------------------------------------------------------- /src/frag/.gitignore: -------------------------------------------------------------------------------- 1 | *.txt 2 | .png 3 | Data-* 4 | /target 5 | -------------------------------------------------------------------------------- /src/frag/Cargo.toml: -------------------------------------------------------------------------------- 1 | [package] 2 | name = "btrfs-frag-view" 3 | version = "0.1.0" 4 | edition = "2018" 5 | 6 | # See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html 7 | 8 | [dependencies] 9 | image = "0.23.14" 10 | fast_hilbert = "1.0.0" 11 | clap = { version = "3.2.16", features = ["derive"] } 12 | statrs = "0.15.0" 13 | serde_json = "1.0" 14 | -------------------------------------------------------------------------------- /src/frag/bg-dump.jinja: -------------------------------------------------------------------------------- 1 | filesystem "{{ testdir }}"; 2 | k = key(0, BTRFS_BLOCK_GROUP_ITEM_KEY, 0, 0); 3 | k.max_type = BTRFS_BLOCK_GROUP_ITEM_KEY; 4 | bgs = search(BTRFS_EXTENT_TREE_OBJECTID, k); 5 | 6 | for bg in bgs { 7 | bg_key = keyof(bg); 8 | if bg_key.type != BTRFS_BLOCK_GROUP_ITEM_KEY { 9 | continue; 10 | } 11 | 12 | bg_start = bg_key.objectid; 13 | bg_len = bg_key.offset; 14 | print("INS BLOCK-GROUP " + str(bg_start) + " " + str(bg_len)); 15 | 16 | k2 = key(bg_start, BTRFS_EXTENT_ITEM_KEY, 0, 0); 17 | k2.max_objectid = bg_start + bg_len - 1; 18 | k2.max_type = BTRFS_EXTENT_ITEM_KEY; 19 | extents = search(BTRFS_EXTENT_TREE_OBJECTID, k2); 20 | 21 | for extent in extents { 22 | extent_key = keyof(extent); 23 | extent_start = extent_key.objectid; 24 | extent_len = extent_key.offset; 25 | type = ""; 26 | 27 | if extent_key.type != BTRFS_EXTENT_ITEM_KEY && extent_key.type != BTRFS_METADATA_ITEM_KEY { 28 | continue; 29 | } 30 | 31 | if extent_start >= bg_start + bg_len { 32 | break; 33 | } 34 | if extent_key.type == BTRFS_EXTENT_ITEM_KEY { 35 | type = "DATA-EXTENT"; 36 | } else { 37 | type = "METADATA-EXTENT"; 38 | extent_len = 16384; 39 | } 40 | print("INS " + type + " " + str(extent_start) + " " + str(extent_len)); 41 | } 42 | } 43 | -------------------------------------------------------------------------------- /src/frag/cleanup.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | 3 | rm -rf Data-* 4 | rm -rf Metadata-* 5 | rm -rf Empty-* 6 | rm *.txt 7 | -------------------------------------------------------------------------------- /src/frag/src/main.rs: -------------------------------------------------------------------------------- 1 | extern crate image; 2 | 3 | use clap::Parser; 4 | use core::ops::{Deref, DerefMut}; 5 | use image::{ImageBuffer, Pixel, Rgb, RgbImage}; 6 | use fast_hilbert::h2xy; 7 | use std::collections::{BTreeMap, HashSet}; 8 | use std::collections::Bound::{Included, Unbounded}; 9 | use std::error; 10 | use std::fmt; 11 | use std::fs; 12 | use statrs::statistics::Data; 13 | use statrs::statistics::Max; 14 | use statrs::statistics::Min; 15 | use statrs::statistics::Distribution; 16 | use statrs::statistics::OrderStatistics; 17 | use serde_json::json; 18 | 19 | const K: u64 = 1 << 10; 20 | const BLOCK: u64 = 4 * K; 21 | const WHITE_PIXEL: Rgb = Rgb([255, 255, 255]); 22 | const RED_PIXEL: Rgb = Rgb([255, 0, 0]); 23 | const GREEN_PIXEL: Rgb = Rgb([0, 255, 0]); 24 | const BLUE_PIXEL: Rgb = Rgb([0, 0, 255]); 25 | 26 | #[derive(Debug, Hash, Eq, PartialEq)] 27 | enum ExtentType { 28 | Data, 29 | Metadata 30 | } 31 | #[derive(Debug, PartialEq)] 32 | enum AllocType { 33 | BlockGroup, 34 | Extent(ExtentType) 35 | } 36 | 37 | #[derive(Debug)] 38 | enum FragViewError { 39 | BeforeStart(u64, u64), 40 | PastEnd(u64, u64), 41 | MissingBg(u64), 42 | MissingExtent(u64, u64), 43 | Parse(String), 44 | TooMuchFree(u64, u64), 45 | } 46 | 47 | impl error::Error for FragViewError { } 48 | 49 | impl fmt::Display for FragViewError { 50 | fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { 51 | match self { 52 | FragViewError::BeforeStart(e, bg) => write!(f, "extent start {} before bg start {}", e, bg), 53 | FragViewError::PastEnd(e, bg) => write!(f, "extent end {} past bg end {}", e, bg), 54 | FragViewError::MissingBg(bg) => write!(f, "missing bg {}", bg), 55 | FragViewError::MissingExtent(e, bg) => write!(f, "missing extent {} in bg {}", e, bg), 56 | FragViewError::Parse(s) => write!(f, "invalid allocation change {}", s), 57 | FragViewError::TooMuchFree(free, len) => write!(f, "bg has more free space {} than size {}", free, len), 58 | } 59 | } 60 | } 61 | 62 | type BoxResult = Result>; 63 | 64 | impl AllocType { 65 | fn from_str(type_str: &str) -> BoxResult { 66 | if type_str == "BLOCK-GROUP" { 67 | Ok(AllocType::BlockGroup) 68 | } else if type_str == "METADATA-EXTENT" { 69 | Ok(AllocType::Extent(ExtentType::Metadata)) 70 | } else if type_str == "DATA-EXTENT" { 71 | Ok(AllocType::Extent(ExtentType::Data)) 72 | } else { 73 | Err(FragViewError::Parse(String::from(type_str)))? 74 | } 75 | } 76 | } 77 | 78 | #[derive(Debug, PartialEq)] 79 | struct AllocId { 80 | alloc_type: AllocType, 81 | offset: u64, 82 | } 83 | 84 | #[derive(Debug, PartialEq)] 85 | enum AllocChange { 86 | Insert(AllocId, u64), 87 | Delete(AllocId), 88 | } 89 | 90 | impl AllocChange { 91 | fn from_dump(dump_line: &str) -> BoxResult { 92 | let vec: Vec<&str> = dump_line.split(" ").collect(); 93 | let change_str = vec[0]; 94 | let type_str = vec[1]; 95 | let alloc_type = AllocType::from_str(type_str)?; 96 | let offset: u64 = vec[2].parse().unwrap(); 97 | let eid = AllocId { alloc_type, offset }; 98 | if change_str == "INS" { 99 | let len: u64 = vec[3].parse().unwrap(); 100 | Ok(AllocChange::Insert(eid, len)) 101 | } else if change_str == "DEL" { 102 | Ok(AllocChange::Delete(eid)) 103 | } else { 104 | Err(FragViewError::Parse(String::from(change_str)))? 105 | } 106 | } 107 | } 108 | 109 | #[derive(Debug)] 110 | struct BlockGroupFragmentation { 111 | len: u64, // block group len 112 | total_free: u64, // sum of all free extents 113 | max_free: u64, // largest free extent 114 | } 115 | 116 | impl BlockGroupFragmentation { 117 | fn new(len: u64) -> Self { 118 | Self { len: len, total_free: 0, max_free: 0 } 119 | } 120 | fn add_free(&mut self, len: u64) -> BoxResult<()>{ 121 | self.total_free = self.total_free + len; 122 | if self.total_free > self.len { 123 | Err(FragViewError::TooMuchFree(self.total_free, self.len))? 124 | } 125 | if len > self.max_free { 126 | self.max_free = len; 127 | } 128 | Ok(()) 129 | } 130 | fn percentage(&self) -> f64 { 131 | if self.total_free == 0 { 132 | return 0.0; 133 | } 134 | 100.0 * (1.0 - ((self.max_free as f64) / (self.total_free as f64))) 135 | } 136 | } 137 | 138 | #[derive(Debug)] 139 | struct BlockGroup { 140 | offset: u64, 141 | len: u64, 142 | extents: BTreeMap, 143 | extent_types: HashSet, 144 | img: RgbImage, 145 | next_extent_color: Rgb, 146 | dump: bool, 147 | dump_count: usize, 148 | } 149 | 150 | // fast_hilbert outputs 4 regions of size 256x256 151 | // TODO: why??? 152 | fn bg_dim(_bg_len: u64) -> u32 { 153 | 512 154 | } 155 | 156 | fn bg_block_to_coord(_dim: u32, block_offset: u64) -> (u32, u32) { 157 | h2xy::(block_offset) 158 | } 159 | 160 | fn global_to_bg(bg_start: u64, offset: u64) -> u64 { 161 | offset - bg_start 162 | } 163 | 164 | fn byte_to_block(offset: u64) -> u64 { 165 | offset / BLOCK 166 | } 167 | 168 | // RUST BS: 169 | // illegal double borrow for: 170 | // for ext in self.extents { // immutable borrow 171 | // self.draw_extent(ext) // mutable borrow, doesn't touch extents 172 | // } 173 | // to fix it without adding a copy, need to pull out this free function 174 | fn draw_extent( 175 | img: &mut ImageBuffer, 176 | bg_start: u64, 177 | extent_offset: u64, 178 | extent_len: u64, 179 | dim: u32, 180 | pixel: P, 181 | ) where 182 | P: Pixel + 'static, 183 | C: Deref + DerefMut, 184 | { 185 | let ext_bg_off = global_to_bg(bg_start, extent_offset); 186 | let ext_block_bg_off = byte_to_block(ext_bg_off); 187 | let nr_blocks = byte_to_block(extent_len); 188 | let ext_block_bg_end = ext_block_bg_off + nr_blocks; 189 | for bg_block in ext_block_bg_off..ext_block_bg_end { 190 | let (x, y) = bg_block_to_coord(dim, bg_block); 191 | img.put_pixel(x, y, pixel); 192 | } 193 | } 194 | 195 | impl BlockGroup { 196 | fn new(offset: u64, len: u64, dump: bool) -> Self { 197 | let dim = bg_dim(len); 198 | BlockGroup { 199 | offset: offset, 200 | len: len, 201 | extent_types: HashSet::new(), 202 | extents: BTreeMap::new(), 203 | img: ImageBuffer::from_pixel(dim, dim, WHITE_PIXEL), 204 | next_extent_color: RED_PIXEL, 205 | dump: dump, 206 | dump_count: 0, 207 | } 208 | } 209 | 210 | fn get_next_extent_color(&mut self) -> Rgb { 211 | match self.next_extent_color { 212 | RED_PIXEL => self.next_extent_color = GREEN_PIXEL, 213 | GREEN_PIXEL => self.next_extent_color = BLUE_PIXEL, 214 | BLUE_PIXEL => self.next_extent_color = RED_PIXEL, 215 | _ => panic!("invalid extent color!"), 216 | } 217 | self.next_extent_color 218 | } 219 | 220 | fn ins_extent(&mut self, offset: u64, len: u64) -> BoxResult<()> { 221 | if offset < self.offset { 222 | return Err(FragViewError::BeforeStart(offset, self.offset))?; 223 | } 224 | if offset + len > self.offset + self.len { 225 | return Err(FragViewError::PastEnd(offset+len, self.offset+self.len))?; 226 | } 227 | self.extents.insert(offset, len); 228 | let color = self.get_next_extent_color(); 229 | self.draw_extent(offset, len, color); 230 | if self.dump { 231 | self.dump_next()?; 232 | } 233 | Ok(()) 234 | } 235 | 236 | fn del_extent(&mut self, offset: u64) -> BoxResult<()> { 237 | let extent = self.extents.remove(&offset); 238 | match extent { 239 | Some(len) => { 240 | self.draw_extent(offset, len, WHITE_PIXEL); 241 | if self.dump { 242 | self.dump_next()?; 243 | } 244 | Ok(()) 245 | }, 246 | None => { 247 | Err(FragViewError::MissingExtent(offset, self.offset))? 248 | } 249 | } 250 | } 251 | 252 | fn fragmentation(&self) -> BoxResult { 253 | let mut bg_frag = BlockGroupFragmentation::new(self.len); 254 | let mut last_extent_end = self.offset; 255 | for (off, len) in &self.extents { 256 | if *off > last_extent_end { 257 | let free_len = off - last_extent_end; 258 | bg_frag.add_free(free_len)?; 259 | } 260 | last_extent_end = off + len; 261 | } 262 | let bg_end = self.offset + self.len; 263 | if last_extent_end < bg_end { 264 | let free_len = bg_end - last_extent_end; 265 | bg_frag.add_free(free_len)?; 266 | } 267 | Ok(bg_frag) 268 | } 269 | 270 | fn draw_extent(&mut self, extent_offset: u64, len: u64, pixel: Rgb) { 271 | let dim = bg_dim(self.len); 272 | draw_extent(&mut self.img, self.offset, extent_offset, len, dim, pixel) 273 | } 274 | 275 | fn name(&self) -> String { 276 | let types: Vec = self.extent_types.iter().map(|et| format!("{:?}", et)).collect(); 277 | let type_names = if types.is_empty() { String::from("Empty") } else { types.join("-") }; 278 | format!("{}-{}", type_names, self.offset) 279 | } 280 | 281 | fn dump_img(&self, f: &str) -> BoxResult<()> { 282 | let d = self.name(); 283 | if d.contains("Meta") { 284 | return Ok(()); 285 | } 286 | if d.contains("Empty") { 287 | return Ok(()); 288 | } 289 | let _ = fs::create_dir_all(&d)?; 290 | let path = format!("{}/{}.png", d, f); 291 | Ok(self.img.save(path)?) 292 | } 293 | 294 | fn dump_frag(&self) -> BoxResult<()> { 295 | if !self.name().contains("Data") { 296 | return Ok(()); 297 | } 298 | let frag = self.fragmentation()?; 299 | println!("{}: {} {:?}", self.offset, frag.percentage(), frag); 300 | Ok(()) 301 | } 302 | 303 | fn dump_next(&mut self) -> BoxResult<()> { 304 | let f = format!("{}", self.dump_count); 305 | self.dump_img(&f)?; 306 | self.dump_count = self.dump_count + 1; 307 | Ok(()) 308 | } 309 | } 310 | 311 | #[derive(Debug)] 312 | struct SpaceInfo { 313 | block_groups: BTreeMap, 314 | dump: bool, 315 | } 316 | 317 | impl SpaceInfo { 318 | fn new() -> Self { 319 | SpaceInfo { 320 | block_groups: BTreeMap::new(), 321 | dump: false, 322 | } 323 | } 324 | fn ins_block_group(&mut self, offset: u64, len: u64) { 325 | self.block_groups 326 | .insert(offset, BlockGroup::new(offset, len, self.dump)); 327 | } 328 | fn del_block_group(&mut self, offset: u64) { 329 | self.block_groups.remove(&offset); 330 | } 331 | fn find_block_group(&mut self, offset: u64) -> BoxResult<&mut BlockGroup> { 332 | let r = self.block_groups.range_mut((Unbounded, Included(offset))); 333 | match r.last() { 334 | Some((_, bg)) => Ok(bg), 335 | None => Err(FragViewError::MissingBg(offset))?, 336 | } 337 | } 338 | fn ins_extent(&mut self, extent_type: ExtentType, offset: u64, len: u64) -> BoxResult<()> { 339 | let offset = offset; 340 | let bg = self.find_block_group(offset)?; 341 | bg.ins_extent(offset, len)?; 342 | bg.extent_types.insert(extent_type); 343 | Ok(()) 344 | } 345 | fn del_extent(&mut self, offset: u64) -> BoxResult<()> { 346 | let bg = self.find_block_group(offset)?; 347 | bg.del_extent(offset)?; 348 | Ok(()) 349 | } 350 | 351 | fn handle_alloc_change(&mut self, alloc_change: AllocChange) -> BoxResult<()> { 352 | match alloc_change { 353 | AllocChange::Insert(AllocId { alloc_type, offset }, len) => match alloc_type { 354 | AllocType::BlockGroup => { 355 | self.ins_block_group(offset, len); 356 | } 357 | AllocType::Extent(extent_type) => { 358 | self.ins_extent(extent_type, offset, len)?; 359 | } 360 | }, 361 | AllocChange::Delete(AllocId { alloc_type, offset }) => match alloc_type { 362 | AllocType::BlockGroup => { 363 | self.del_block_group(offset); 364 | } 365 | _ => { 366 | self.del_extent(offset)?; 367 | } 368 | }, 369 | } 370 | Ok(()) 371 | } 372 | 373 | fn dump_imgs(&self, name: &str) -> BoxResult<()> { 374 | for (_, bg) in &self.block_groups { 375 | bg.dump_img(name)?; 376 | } 377 | Ok(()) 378 | } 379 | 380 | fn dump_raw(&self) -> BoxResult<()> { 381 | for (_, bg) in &self.block_groups { 382 | bg.dump_frag()?; 383 | } 384 | Ok(()) 385 | } 386 | 387 | fn dump_stats(&self) -> BoxResult<()> { 388 | let mut pctv: Vec = Vec::new(); 389 | let mut total_bgs = 0; 390 | for (_, bg) in &self.block_groups { 391 | total_bgs = total_bgs + 1; 392 | if !bg.name().contains("Data") { 393 | continue; 394 | } 395 | let frag = bg.fragmentation()?; 396 | if frag.percentage() == 0.0 { 397 | continue; 398 | } 399 | pctv.push(frag.percentage()); 400 | } 401 | let mut d = Data::new(pctv); 402 | let mean = match d.mean() { 403 | Some(m) => m, 404 | None => 0.0 405 | }; 406 | let mut min = 0.0; 407 | let mut p50 = 0.0; 408 | let mut p95 = 0.0; 409 | let mut p99 = 0.0; 410 | let mut max = 0.0; 411 | if d.len() > 0 { 412 | min = d.min(); 413 | p50 = d.median(); 414 | p95 = d.percentile(95); 415 | p99 = d.percentile(99); 416 | max = d.max(); 417 | } 418 | let json = json!({ 419 | "bg_count": total_bgs, 420 | "fragmented_bg_count": d.len(), 421 | "frag_pct_mean": mean, 422 | "frag_pct_min": min, 423 | "frag_pct_p50": p50, 424 | "frag_pct_p95": p95, 425 | "frag_pct_p99": p99, 426 | "frag_pct_max": max 427 | }); 428 | println!("{}", json); 429 | Ok(()) 430 | } 431 | 432 | fn handle_file(&mut self, f: &str) -> BoxResult<()> { 433 | let contents = fs::read_to_string(&f)?; 434 | for line in contents.split("\n") { 435 | if line.is_empty() { 436 | continue; 437 | } 438 | let ac = AllocChange::from_dump(line)?; 439 | self.handle_alloc_change(ac)?; 440 | } 441 | Ok(()) 442 | } 443 | } 444 | 445 | /// Analyze and visualize btrfs block group fragmentation 446 | #[derive(Parser, Debug)] 447 | #[clap(author, version, about, long_about = None)] 448 | struct Args { 449 | /// btrd frag dump file name 450 | #[clap(value_parser)] 451 | file: String, 452 | 453 | /// Whether or not to dump fragmentation stats 454 | #[clap(short, long, value_parser, default_value_t = true)] 455 | stats: bool, 456 | 457 | /// Whether or not to dump fragmentation images 458 | #[clap(short, long, value_parser, default_value_t = false)] 459 | images: bool, 460 | 461 | /// Whether or not to dump raw bg fragmentation data 462 | #[clap(short, long, value_parser, default_value_t = false)] 463 | raw: bool, 464 | } 465 | 466 | fn main() -> BoxResult<()> { 467 | let args = Args::parse(); 468 | let mut si = SpaceInfo::new(); 469 | si.handle_file(&args.file)?; 470 | if args.stats { 471 | si.dump_stats()?; 472 | } 473 | if args.images { 474 | si.dump_imgs(&args.file)?; 475 | } 476 | if args.raw { 477 | si.dump_raw()?; 478 | } 479 | Ok(()) 480 | } 481 | 482 | #[cfg(test)] 483 | mod test { 484 | const M: u64 = 1 << 20; 485 | const G: u64 = 1 << 30; 486 | 487 | 488 | use super::*; 489 | #[test] 490 | fn parse_dump_lines() { 491 | let dummy_dump_line = "INS BLOCK-GROUP 420 42"; 492 | let ac = AllocChange::from_dump(dummy_dump_line).unwrap(); 493 | assert_eq!( 494 | ac, 495 | AllocChange::Insert( 496 | AllocId { 497 | alloc_type: AllocType::BlockGroup, 498 | offset: 420 499 | }, 500 | 42 501 | ) 502 | ); 503 | 504 | let dummy_dump_line = "DEL BLOCK-GROUP 420"; 505 | let ac = AllocChange::from_dump(dummy_dump_line).unwrap(); 506 | assert_eq!( 507 | ac, 508 | AllocChange::Delete(AllocId { 509 | alloc_type: AllocType::BlockGroup, 510 | offset: 420 511 | }) 512 | ); 513 | 514 | let dummy_dump_line = "INS DATA-EXTENT 420 42"; 515 | let ac = AllocChange::from_dump(dummy_dump_line).unwrap(); 516 | assert_eq!( 517 | ac, 518 | AllocChange::Insert( 519 | AllocId { 520 | alloc_type: AllocType::Data, 521 | offset: 420 522 | }, 523 | 42 524 | ) 525 | ); 526 | 527 | let dummy_dump_line = "DEL DATA-EXTENT 420"; 528 | let ac = AllocChange::from_dump(dummy_dump_line).unwrap(); 529 | assert_eq!( 530 | ac, 531 | AllocChange::Delete(AllocId { 532 | alloc_type: AllocType::Data, 533 | offset: 420 534 | }) 535 | ); 536 | 537 | let dummy_dump_line = "INS METADATA-EXTENT 420 42"; 538 | let ac = AllocChange::from_dump(dummy_dump_line).unwrap(); 539 | assert_eq!( 540 | ac, 541 | AllocChange::Insert( 542 | AllocId { 543 | alloc_type: AllocType::Extent(ExtentType::Metadata), 544 | offset: 420 545 | }, 546 | 42 547 | ) 548 | ); 549 | 550 | let dummy_dump_line = "DEL METADATA-EXTENT 420"; 551 | let ac = AllocChange::from_dump(dummy_dump_line).unwrap(); 552 | assert_eq!( 553 | ac, 554 | AllocChange::Delete(AllocId { 555 | alloc_type: AllocType::Extent(ExtentType::Metadata), 556 | offset: 420 557 | }) 558 | ); 559 | } 560 | #[test] 561 | fn ins_extents() { 562 | let mut si = SpaceInfo::new(); 563 | si.ins_block_group(G, G); 564 | si.ins_block_group(2 * G, G); 565 | si.ins_extent(G + K, 4 * K); 566 | si.ins_extent(2 * G + 10 * K, 256 * M); 567 | assert_eq!(si.block_groups.len(), 2); 568 | for bg in si.block_groups.values() { 569 | assert_eq!(bg.extents.len(), 1); 570 | } 571 | } 572 | #[test] 573 | fn del_extents() { 574 | let mut si = SpaceInfo::new(); 575 | si.ins_block_group(G, G); 576 | si.ins_block_group(2 * G, G); 577 | si.ins_extent(G + K, 4 * K); 578 | si.ins_extent(G + 10 * K, 256 * M); 579 | si.ins_extent(2 * G + 10 * K, 256 * M); 580 | assert_eq!(si.block_groups.len(), 2); 581 | si.del_extent(G + 10 * K).unwrap(); 582 | for bg in si.block_groups.values() { 583 | assert_eq!(bg.extents.len(), 1); 584 | } 585 | } 586 | // various scenarios with missing block group 587 | #[test] 588 | fn test_no_bg() {} 589 | 590 | // various scenarios with invalid overlapping block_groups 591 | #[test] 592 | fn test_bg_overlap() {} 593 | 594 | // various scenarios with invalid overlapping extents 595 | #[test] 596 | fn test_extent_overlap() {} 597 | } 598 | -------------------------------------------------------------------------------- /src/frag/tests/buffered-append-vs-fallocate.py: -------------------------------------------------------------------------------- 1 | from FragTest import FragTest 2 | import os.path 3 | 4 | class BufferedAppendVsFallocate(FragTest): 5 | name = "bufferedappendvsfallocate" 6 | command = os.path.join(os.path.dirname(__file__), "buffered-append-vs-fallocate.fio") 7 | -------------------------------------------------------------------------------- /src/fsperf.py: -------------------------------------------------------------------------------- 1 | import argparse 2 | import configparser 3 | import os 4 | import sys 5 | from subprocess import Popen 6 | import FioCompare 7 | import ResultData 8 | from utils import run_command,Mount,setup_device,setup_cpu_governor,mkfs,NotRunException 9 | from sqlalchemy import create_engine 10 | from sqlalchemy.orm import sessionmaker 11 | import datetime 12 | import utils 13 | import compare 14 | import platform 15 | 16 | TEST_ONLY = "TMP-TEST-ONLY" 17 | 18 | def clean_testonly(session, sections, tests): 19 | today = datetime.date.today() 20 | age = today - datetime.timedelta(days=365) 21 | for section in sections: 22 | for test in tests: 23 | results = utils.get_results(session, test.name, section, TEST_ONLY, age) 24 | for r in results: 25 | session.delete(r) 26 | session.commit() 27 | 28 | def want_run_test(run_tests, disabled_tests, t): 29 | names = [t.name, t.__class__.__name__] 30 | if disabled_tests: 31 | for name in names: 32 | if name in disabled_tests: 33 | return False 34 | if run_tests: 35 | for name in names: 36 | if name in run_tests: 37 | return True 38 | return False 39 | return True 40 | 41 | def run_test(args, session, config, section, purpose, test): 42 | for i in range(0, args.numruns): 43 | try: 44 | run = ResultData.Run(kernel=platform.release(), config=section, 45 | name=test.name, purpose=purpose) 46 | test.run(run, config, section, "results") 47 | session.add(run) 48 | session.commit() 49 | except NotRunException as e: 50 | print("Not run: {}".format(e)) 51 | return 0 52 | 53 | parser = argparse.ArgumentParser() 54 | parser.add_argument('-c', '--config', type=str, 55 | help="Configuration to use to run the tests") 56 | parser.add_argument('-l', '--latency', action='store_true', 57 | help="Compare latency values of the current run to old runs") 58 | parser.add_argument('-t', '--testonly', action='store_true', 59 | help="Compare this run to previous runs, but do not store this run.") 60 | parser.add_argument('-F', '--fragmentation', action='store_true', 61 | help="include fragmentation tests in run") 62 | parser.add_argument('-n', '--numruns', type=int, default=1, 63 | help="Run each test N number of times") 64 | parser.add_argument('-p', '--purpose', type=str, default="continuous", 65 | help="Set the specific purpose for this run, useful for A/B testing") 66 | parser.add_argument('-C', '--compare', type=str, 67 | help="Configuration to compare this run to, used with -t") 68 | parser.add_argument('tests', nargs='*', 69 | help="Specific test[s] to run.") 70 | parser.add_argument('--list', action='store_true', help="List all available tests") 71 | 72 | args = parser.parse_args() 73 | 74 | config = configparser.ConfigParser() 75 | config.read('local.cfg') 76 | if not config.has_section('main'): 77 | print("Must have a [main] section in local.cfg") 78 | sys.exit(1) 79 | 80 | if not config.get('main', 'directory'): 81 | print("Must specify 'directory' in [main]") 82 | sys.exit(1) 83 | 84 | setup_cpu_governor(config) 85 | 86 | disabled_tests = [] 87 | failed_tests = [] 88 | 89 | try: 90 | with open('disabled-tests') as f: 91 | for line in f: 92 | disabled_tests.append(line.rstrip()) 93 | print("Disabled {}".format(line.rstrip())) 94 | except FileNotFoundError: 95 | pass 96 | 97 | sections = [args.config] 98 | if args.config is None: 99 | sections = config.sections() 100 | sections.remove('main') 101 | elif not config.has_section(args.config): 102 | print("No section '{}' in local.cfg".format(args.config)) 103 | sys.exit(1) 104 | 105 | compare_config = None 106 | if args.compare is not None: 107 | compare_config = args.compare 108 | if not config.has_section(compare_config): 109 | print("No section '{}' in local.cfg".format(compare_config)) 110 | sys.exit(1) 111 | 112 | engine = create_engine('sqlite:///fsperf-results.db') 113 | ResultData.Base.metadata.create_all(engine) 114 | Session = sessionmaker() 115 | Session.configure(bind=engine) 116 | session = Session() 117 | 118 | utils.mkdir_p("results/") 119 | 120 | tests, oneoffs = utils.get_tests("tests/") 121 | 122 | # In case we changed our test directory, delete bg-dump.btrd so it can be 123 | # re-generated with the appropriate directory 124 | try: 125 | os.unlink('src/frag/bg-dump.btrd') 126 | except FileNotFoundError: 127 | pass 128 | 129 | if args.fragmentation: 130 | frag_tests, frag_oneoffs = utils.get_tests("frag_tests/") 131 | tests.extend(frag_tests) 132 | oneoffs.extend(frag_oneoffs) 133 | 134 | if args.list: 135 | print("Normal tests") 136 | for t in tests: 137 | print("\t{}".format(t.__class__.__name__)) 138 | print("Oneoff tests") 139 | for t in oneoffs: 140 | print("\t{}".format(t.__class__.__name__)) 141 | sys.exit(1) 142 | 143 | if args.testonly: 144 | run_purpose = TEST_ONLY 145 | # We might have exited uncleanly and left behind testonly results 146 | clean_testonly(session, sections, tests) 147 | else: 148 | run_purpose = args.purpose 149 | 150 | # Run the normal tests 151 | for section in sections: 152 | setup_device(config, section) 153 | for t in tests: 154 | if not want_run_test(args.tests, disabled_tests, t): 155 | continue 156 | print("Running {}".format(t.__class__.__name__)) 157 | run_test(args, session, config, section, run_purpose, t) 158 | 159 | for t in oneoffs: 160 | if not want_run_test(args.tests, disabled_tests, t): 161 | continue 162 | print("Running {}".format(t.__class__.__name__)) 163 | run_test(args, session, config, "oneoff", run_purpose, t) 164 | 165 | if args.testonly: 166 | today = datetime.date.today() 167 | if args.purpose == "continuous": 168 | age = today - datetime.timedelta(days=7) 169 | else: 170 | age = today - datetime.timedelta(days=365) 171 | for section in sections: 172 | print(f"{section} test results") 173 | for t in tests: 174 | if not want_run_test(args.tests, disabled_tests, t): 175 | continue 176 | compare_section = compare_config if compare_config else section 177 | compare.compare_results(session, compare_section, section, t, args.purpose, TEST_ONLY, age) 178 | if oneoffs: 179 | print(f"oneoff test results") 180 | for t in oneoffs: 181 | if not want_run_test(args.tests, disabled_tests, t): 182 | continue 183 | compare_section = compare_config if compare_config else section 184 | compare.compare_results(session, "oneoff", "oneoff", t, args.purpose, TEST_ONLY, age) 185 | # We use the db to uniformly access test results. Clean up testonly results 186 | clean_testonly(session, sections, tests) 187 | -------------------------------------------------------------------------------- /src/generate-graph.py: -------------------------------------------------------------------------------- 1 | from ResultData import * 2 | from sqlalchemy import create_engine 3 | from sqlalchemy.orm import sessionmaker 4 | import argparse 5 | import matplotlib.pyplot as plt 6 | import numpy as np 7 | import datetime 8 | import utils 9 | import numbers 10 | 11 | def get_all_results(session, purpose, test): 12 | results = session.query(Run).\ 13 | outerjoin(FioResult).\ 14 | outerjoin(DbenchResult).\ 15 | outerjoin(TimeResult).\ 16 | filter(Run.name == test).\ 17 | filter(Run.purpose == purpose).\ 18 | order_by(Run.time).all() 19 | ret = [] 20 | for r in results: 21 | ret.append(utils.results_to_dict(r, include_time=True)) 22 | return ret 23 | 24 | def get_all_purposes(session, purposes): 25 | r = session.query(Run.purpose).distinct().all() 26 | results = [] 27 | for i in r: 28 | if len(purposes) == 0 or i[0] in purposes: 29 | results.append(i[0]) 30 | return results 31 | 32 | def get_values_for_key(results_array, key): 33 | runs = [] 34 | values = [] 35 | found_nonzero = False 36 | count = 0 37 | for run in results_array: 38 | runs.append(count) 39 | count += 1 40 | values.append(run[key]) 41 | if run[key] > 0 or run[key] < 0: 42 | found_nonzero = True 43 | if found_nonzero: 44 | return (runs, values) 45 | return (None, None) 46 | 47 | parser = argparse.ArgumentParser() 48 | parser.add_argument('-t', '--test', type=str, required=True, 49 | help="Test to generate the graph for") 50 | parser.add_argument('-d', '--dir', type=str, default=".", 51 | help="Directory to write the graphs to") 52 | parser.add_argument('-p', '--purposes', nargs="*", type=str, default=[], 53 | help="Purposes to graph") 54 | 55 | args = parser.parse_args() 56 | 57 | engine = create_engine('sqlite:///fsperf-results.db') 58 | Session = sessionmaker() 59 | Session.configure(bind=engine) 60 | session = Session() 61 | 62 | purposes = get_all_purposes(session, args.purposes) 63 | 64 | last = utils.get_last_test(session, args.test) 65 | for k,v in last.items(): 66 | if not isinstance(v, numbers.Number): 67 | continue 68 | if "id" in k: 69 | continue 70 | if v == 0: 71 | continue 72 | 73 | print(f'Generating graph for {args.test}_{k}') 74 | # Start a new figure 75 | plt.figure() 76 | fig, ax = plt.subplots() 77 | 78 | # format the ticks 79 | #ax.xaxis.set_major_locator(locator) 80 | #ax.xaxis.set_major_formatter(formatter) 81 | 82 | for p in purposes: 83 | print(f'getting results for {p}') 84 | results = get_all_results(session, p, args.test) 85 | (runs, values) = get_values_for_key(results, k) 86 | if runs is None: 87 | continue 88 | 89 | plt.plot(runs, values, label=p) 90 | 91 | plt.title(f"{args.test} {k} results over time") 92 | plt.legend(bbox_to_anchor=(1.04, 1), borderaxespad=0) 93 | plt.show() 94 | plt.savefig(f"{args.dir}/{args.test}_{k}.png", bbox_inches="tight") 95 | plt.close('all') 96 | -------------------------------------------------------------------------------- /src/generate-results-page.py: -------------------------------------------------------------------------------- 1 | from ResultData import * 2 | from sqlalchemy import create_engine 3 | from sqlalchemy.orm import sessionmaker 4 | from jinja2 import Template,Environment,FileSystemLoader 5 | import matplotlib.pyplot as plt 6 | import matplotlib.dates as mdates 7 | import numpy as np 8 | import datetime 9 | import utils 10 | import numbers 11 | import multiprocessing 12 | import statistics 13 | 14 | def get_avgs(session, config, test, days): 15 | today = datetime.date.today() 16 | thresh = today - datetime.timedelta(days=days) 17 | results = session.query(Run).\ 18 | outerjoin(FioResult).\ 19 | outerjoin(DbenchResult).\ 20 | outerjoin(TimeResult).\ 21 | outerjoin(Fragmentation).\ 22 | outerjoin(LatencyTrace).\ 23 | outerjoin(IOStats).\ 24 | outerjoin(BtrfsCommitStats).\ 25 | outerjoin(MountTiming).\ 26 | filter(Run.time >= thresh).\ 27 | filter(Run.config == config).\ 28 | filter(Run.name == test).\ 29 | filter(Run.purpose == "continuous").\ 30 | order_by(Run.time).all() 31 | newest = None 32 | if len(results) > 1: 33 | newest = results.pop() 34 | avgs = utils.avg_results(results) 35 | if newest is None: 36 | return avgs 37 | 38 | newest_dict = utils.results_to_dict(newest) 39 | for k,vs in newest_dict.items(): 40 | if k in avgs: 41 | continue 42 | avgs[k] = {'mean': 0.0, 'stdev': 0} 43 | return avgs 44 | 45 | def get_last(session, config, test): 46 | result = session.query(Run).\ 47 | outerjoin(FioResult).\ 48 | outerjoin(DbenchResult).\ 49 | outerjoin(TimeResult).\ 50 | outerjoin(Fragmentation).\ 51 | outerjoin(LatencyTrace).\ 52 | outerjoin(IOStats).\ 53 | outerjoin(BtrfsCommitStats).\ 54 | outerjoin(MountTiming).\ 55 | filter(Run.name == test).\ 56 | filter(Run.config == config).\ 57 | filter(Run.purpose == "continuous").\ 58 | order_by(Run.id.desc()).first() 59 | if result is None: 60 | return result 61 | results = utils.results_to_dict(result) 62 | ret = {} 63 | for k,v in results.items(): 64 | ret[k] = {'value': v} 65 | return ret 66 | 67 | def get_all_results(session, config, test): 68 | results = session.query(Run).\ 69 | outerjoin(FioResult).\ 70 | outerjoin(DbenchResult).\ 71 | outerjoin(TimeResult).\ 72 | outerjoin(Fragmentation).\ 73 | outerjoin(LatencyTrace).\ 74 | outerjoin(IOStats).\ 75 | outerjoin(BtrfsCommitStats).\ 76 | outerjoin(MountTiming).\ 77 | filter(Run.name == test).\ 78 | filter(Run.config == config).\ 79 | filter(Run.purpose == "continuous").\ 80 | order_by(Run.time).all() 81 | ret = [] 82 | for r in results: 83 | ret.append(utils.results_to_dict(r, include_time=True)) 84 | return ret 85 | 86 | def get_values_for_key(results_array, key): 87 | dates = [] 88 | values = [] 89 | found_nonzero = False 90 | for run in results_array: 91 | dates.append(run['time']) 92 | if key not in run: 93 | values.append(0) 94 | continue 95 | values.append(run[key]) 96 | if run[key] > 0 or run[key] < 0: 97 | found_nonzero = True 98 | if not found_nonzero: 99 | return (None, None) 100 | 101 | mean = statistics.mean(values) 102 | stdev = statistics.stdev(values) 103 | 104 | loop = True 105 | while loop: 106 | loop = False 107 | for i in range(0, len(values)): 108 | if stdev == 0: 109 | break 110 | zval = (values[i] - mean) / stdev 111 | if zval > 3 or zval < -3: 112 | del values[i] 113 | del dates[i] 114 | loop = True 115 | break 116 | if len(values) == 0: 117 | return (None, None) 118 | return (dates, values) 119 | 120 | def generate_graph(session, test, config): 121 | last = utils.get_last_test(session, test) 122 | results = get_all_results(session, config, test) 123 | if len(results) == 0: 124 | return 125 | 126 | for k,v in last.items(): 127 | if not isinstance(v, numbers.Number): 128 | continue 129 | if "id" in k: 130 | continue 131 | if v == 0: 132 | continue 133 | 134 | configname = config.replace(" ", "_") 135 | print(f'Generating graph for {test}_{configname}_{k}') 136 | # Start a new figure 137 | plt.figure() 138 | fig, ax = plt.subplots() 139 | 140 | # format the ticks 141 | ax.xaxis.set_major_locator(locator) 142 | ax.xaxis.set_major_formatter(formatter) 143 | 144 | (dates, values) = get_values_for_key(results, k) 145 | if dates is None: 146 | continue 147 | 148 | # figure out the range 149 | datemin = np.datetime64(dates[0], 'D') 150 | datemax = np.datetime64(dates[-1], 'D') + 1 151 | 152 | plt.plot(dates, values, label=config) 153 | 154 | ax.set_xlim(datemin, datemax) 155 | fig.autofmt_xdate() 156 | plt.title(f"{test} {k} {config} results over time") 157 | plt.legend(bbox_to_anchor=(1.04, 1), borderaxespad=0) 158 | plt.savefig(f"www/{test}_{configname}_{k}.png", bbox_inches="tight") 159 | plt.close('all') 160 | 161 | def generate_graphs(session, tests, configs): 162 | tasks = [] 163 | for t in tests: 164 | for c in configs: 165 | tasks.append(multiprocessing.Process(target=generate_graph, args=([session, t, c]))) 166 | for t in tasks: 167 | t.start() 168 | for t in tasks: 169 | t.join() 170 | 171 | engine = create_engine('sqlite:///fsperf-results.db') 172 | Session = sessionmaker() 173 | Session.configure(bind=engine) 174 | session = Session() 175 | 176 | tests = [] 177 | for tname in session.query(Run.name).distinct(): 178 | tests.append(tname[0]) 179 | 180 | configs = [] 181 | for config in session.query(Run.config).distinct(): 182 | configs.append(config[0]) 183 | 184 | week_avgs = {} 185 | two_week_avgs = {} 186 | three_week_avgs = {} 187 | four_week_avgs = {} 188 | recent = {} 189 | 190 | for c in configs: 191 | recent[c] = {} 192 | week_avgs[c] = {} 193 | two_week_avgs[c] = {} 194 | three_week_avgs[c] = {} 195 | four_week_avgs[c] = {} 196 | 197 | for t in tests: 198 | # Not all configs can run all tests, so if we don't have the test 199 | # results for the given config simply skip the test 200 | run = get_last(session, c, t) 201 | if run is None: 202 | print(f'no run for {t} in config {c}') 203 | recent[c][t] = None 204 | continue 205 | recent[c][t] = get_last(session, c, t) 206 | week_avgs[c][t] = get_avgs(session, c, t, 7) 207 | two_week_avgs[c][t] = get_avgs(session, c, t, 14) 208 | three_week_avgs[c][t] = get_avgs(session, c, t, 21) 209 | four_week_avgs[c][t] = get_avgs(session, c, t, 28) 210 | recent[c][t]['regression'] = False 211 | if (utils.check_regression(week_avgs[c][t], recent[c][t]) or 212 | utils.check_regression(two_week_avgs[c][t], recent[c][t]) or 213 | utils.check_regression(three_week_avgs[c][t], recent[c][t]) or 214 | utils.check_regression(four_week_avgs[c][t], recent[c][t])): 215 | recent[c][t]['regression'] = True 216 | 217 | env = Environment(loader=FileSystemLoader('src')) 218 | index_template = env.get_template('index.jinja') 219 | test_template = env.get_template('test.jinja') 220 | 221 | for t in tests: 222 | f = open(f'www/{t}.html', 'w') 223 | print(f'Writing {t}.html') 224 | f.write(test_template.render(test=t, configs=configs, 225 | avgs=[week_avgs, two_week_avgs, 226 | three_week_avgs, four_week_avgs], 227 | recent=recent)) 228 | f.close() 229 | 230 | f = open(f'www/index.html', 'w') 231 | print(f'Writing index.html') 232 | f.write(index_template.render(tests=tests, configs=configs, recent=recent)) 233 | f.close() 234 | 235 | locator = mdates.AutoDateLocator(minticks=3, maxticks=7) 236 | formatter = mdates.ConciseDateFormatter(locator) 237 | 238 | generate_graphs(session, tests, configs) 239 | -------------------------------------------------------------------------------- /src/generate-schema.py: -------------------------------------------------------------------------------- 1 | import json 2 | import argparse 3 | import FioResultDecoder 4 | from dateutil.parser import parse 5 | 6 | def is_date(string): 7 | try: 8 | parse(string) 9 | return True 10 | except ValueError: 11 | return False 12 | 13 | def print_schema_def(key, value): 14 | typestr = value.__class__.__name__ 15 | if typestr == 'str' or typestr == 'unicode': 16 | if (is_date(value)): 17 | typestr = "datetime" 18 | else: 19 | typestr = "varchar(256)" 20 | return ",\n `{}` {} NOT NULL".format(key, typestr) 21 | 22 | parser = argparse.ArgumentParser() 23 | parser.add_argument('infile', help="The json file to strip") 24 | args = parser.parse_args() 25 | 26 | json_data = open(args.infile) 27 | data = json.load(json_data, cls=FioResultDecoder.FioResultDecoder) 28 | 29 | # These get populated by the test runner, not fio, so add them so their 30 | # definitions get populated in the schema properly 31 | data['global']['config'] = 'default' 32 | data['global']['kernel'] = '4.14' 33 | 34 | print("CREATE TABLE `fio_runs` (") 35 | outstr = " `id` int(11) PRIMARY KEY" 36 | for key,value in data['global'].items(): 37 | outstr += print_schema_def(key, value) 38 | print(outstr) 39 | print(");") 40 | 41 | job = data['jobs'][0] 42 | job['run_id'] = 0 43 | 44 | print("CREATE TABLE `fio_jobs` (") 45 | outstr = " `id` int PRIMARY KEY" 46 | for key,value in job.items(): 47 | outstr += print_schema_def(key, value) 48 | print(outstr) 49 | print(");") 50 | -------------------------------------------------------------------------------- /src/index.jinja: -------------------------------------------------------------------------------- 1 | 2 | 3 | Performance results 4 | 5 | 6 | 7 | {% for c in configs %} 8 | 9 | 10 | 11 | {% for t in tests if recent[c][t]%} 12 | 13 | 14 | {% if recent[c][t]['regression'] %} 15 | 16 | {% else %} 17 | 18 | {% endif %} 19 | 20 | {% endfor %} 21 |
{{ c }}
TestStatus
{{ t }}FAILOK
22 | {% endfor %} 23 | 24 | 25 | -------------------------------------------------------------------------------- /src/nullblk.py: -------------------------------------------------------------------------------- 1 | import os 2 | import utils 3 | 4 | class NullBlock(): 5 | def __del__(self): 6 | if not self._started: 7 | return 8 | dname = f"/sys/kernel/config/nullb/{self.name}" 9 | with open(f'{dname}/power', 'w') as power: 10 | power.write('0') 11 | os.rmdir(dname) 12 | 13 | def __init__(self, name="nullb0"): 14 | self._started = False 15 | self.name = name 16 | self.config_values = {} 17 | 18 | def start(self): 19 | if not os.path.isdir('/sys/kernel/config/nullb'): 20 | utils.run_command('modprobe null_blk nr_devices=0') 21 | 22 | if os.path.exists('/dev/nullb0'): 23 | utils.run_command('rmmod null_blk') 24 | utils.run_command('modprobe null_blk nr_devices=0') 25 | 26 | dname = f"/sys/kernel/config/nullb/{self.name}" 27 | os.makedirs(dname) 28 | for key in self.config_values: 29 | with open(f"{dname}/{key}", 'w') as writer: 30 | writer.write(self.config_values[key]) 31 | with open(f"{dname}/power", 'w') as power: 32 | power.write('1') 33 | with open(f'/sys/block/{self.name}/queue/scheduler', 'w') as f: 34 | f.write('none') 35 | self._started = True 36 | -------------------------------------------------------------------------------- /src/test.jinja: -------------------------------------------------------------------------------- 1 | 2 | 3 | {{ test }} results 4 | 5 | 6 | 7 | {% for c in configs if recent[c][test] %} 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | {% for m in avgs[0][c][test].keys() %} 19 | 20 | 21 | 22 | 23 | 24 | 25 | {% if recent[c][test][m]['regression'] %} 26 | 31 | 32 | {% endfor %} 33 |
{{ c }}
Metric4 week avg3 week avg2 week avg1 week avgLast
{{ m }}{{ "%0.2f" | format(avgs[0][c][test][m]['mean']|float) }}{{ "%0.2f" | format(avgs[1][c][test][m]['mean']|float) }}{{ "%0.2f" | format(avgs[2][c][test][m]['mean']|float) }}{{ "%0.2f" | format(avgs[3][c][test][m]['mean']|float) }} 27 | {% else %} 28 | 29 | {% endif %} 30 | {{ "%0.2f" | format(recent[c][test][m]['value']|float) }}
34 | {% endfor %} 35 | 36 | 37 | -------------------------------------------------------------------------------- /src/utils.py: -------------------------------------------------------------------------------- 1 | from subprocess import Popen, PIPE, DEVNULL, CalledProcessError 2 | import PerfTest 3 | import ResultData 4 | import sys 5 | import os 6 | import errno 7 | import texttable 8 | import itertools 9 | import datetime 10 | import statistics 11 | import subprocess 12 | import re 13 | import shlex 14 | import collections 15 | import importlib.util 16 | import inspect 17 | import numpy 18 | import signal 19 | import time 20 | import jinja2 21 | import stat 22 | 23 | LOWER_IS_BETTER = 0 24 | HIGHER_IS_BETTER = 1 25 | 26 | METRIC_DIRECTIONS = { 27 | 'elapsed': LOWER_IS_BETTER, 28 | 'sys_cpu': LOWER_IS_BETTER, 29 | 'throughput': HIGHER_IS_BETTER, 30 | 31 | # These are all latency values from dbench 32 | 'ntcreatex': LOWER_IS_BETTER, 33 | 'close': LOWER_IS_BETTER, 34 | 'rename': LOWER_IS_BETTER, 35 | 'unlink': LOWER_IS_BETTER, 36 | 'deltree': LOWER_IS_BETTER, 37 | 'mkdir': LOWER_IS_BETTER, 38 | 'qpathinfo': LOWER_IS_BETTER, 39 | 'qfileinfo': LOWER_IS_BETTER, 40 | 'qfsinfo': LOWER_IS_BETTER, 41 | 'sfileinfo': LOWER_IS_BETTER, 42 | 'find': LOWER_IS_BETTER, 43 | 'writex': LOWER_IS_BETTER, 44 | 'readx': LOWER_IS_BETTER, 45 | 'lockx': LOWER_IS_BETTER, 46 | 'unlockx': LOWER_IS_BETTER, 47 | 'flush': LOWER_IS_BETTER, 48 | } 49 | 50 | def metric_direction(metric): 51 | if "bytes" in metric: 52 | return HIGHER_IS_BETTER 53 | if "calls" in metric: 54 | return LOWER_IS_BETTER 55 | if "_iops" in metric: 56 | return HIGHER_IS_BETTER 57 | if "_ns_" in metric: 58 | return LOWER_IS_BETTER 59 | if metric in METRIC_DIRECTIONS: 60 | return METRIC_DIRECTIONS[metric] 61 | return LOWER_IS_BETTER 62 | 63 | 64 | # We only mark the whole test as failed if these keys regress 65 | test_regression_keys = [ 66 | 'read_bw_bytes', 67 | 'write_bw_bytes', 68 | 'throughput', 69 | 'elapsed' 70 | ] 71 | 72 | class NotRunException(Exception): 73 | def __init__(self, m): 74 | super().__init__(m) 75 | self.m = m 76 | 77 | def get_last_test(session, test): 78 | result = session.query(ResultData.Run).\ 79 | outerjoin(ResultData.FioResult).\ 80 | outerjoin(ResultData.DbenchResult).\ 81 | outerjoin(ResultData.TimeResult).\ 82 | filter(ResultData.Run.name == test).\ 83 | order_by(ResultData.Run.id.desc()).first() 84 | return results_to_dict(result) 85 | 86 | def get_results(session, name, config, purpose, time): 87 | return session.query(ResultData.Run).\ 88 | outerjoin(ResultData.FioResult).\ 89 | outerjoin(ResultData.DbenchResult).\ 90 | outerjoin(ResultData.TimeResult).\ 91 | outerjoin(ResultData.Fragmentation).\ 92 | outerjoin(ResultData.LatencyTrace).\ 93 | outerjoin(ResultData.IOStats).\ 94 | outerjoin(ResultData.BtrfsCommitStats).\ 95 | outerjoin(ResultData.MountTiming).\ 96 | filter(ResultData.Run.time >= time).\ 97 | filter(ResultData.Run.name == name).\ 98 | filter(ResultData.Run.purpose == purpose).\ 99 | filter(ResultData.Run.config == config).\ 100 | order_by(ResultData.Run.id).all() 101 | 102 | # Shamelessly copied from stackoverflow 103 | def mkdir_p(path): 104 | try: 105 | os.makedirs(path) 106 | except OSError as exc: 107 | if exc.errno == errno.EEXIST and os.path.isdir(path): 108 | pass 109 | else: 110 | raise 111 | 112 | def run_command(cmd, outputfile=None): 113 | print(f" running cmd '{cmd}'") 114 | if not outputfile: 115 | outputfile = PIPE 116 | p = Popen(shlex.split(cmd), stdout=outputfile, stderr=outputfile) 117 | out, err = p.communicate() 118 | if p.returncode: 119 | print(f"{cmd} failed. out: {out}, err: {err}") 120 | raise CalledProcessError(p.returncode, cmd) 121 | 122 | def setup_cpu_governor(config): 123 | if not config.has_option('main', 'cpugovernor'): 124 | return 125 | governor = config.get('main', 'cpugovernor') 126 | dirpath = "/sys/devices/system/cpu" 127 | for filename in os.listdir(dirpath): 128 | if re.match("cpu\d+", filename): 129 | try: 130 | with open(f'{dirpath}/{filename}/cpufreq/scaling_governor', 'w') as f: 131 | f.write(governor) 132 | except OSError as exc: 133 | print("cpu governor isn't enabled, skipping") 134 | return 135 | 136 | def setup_device(config, section): 137 | device = os.path.basename(os.path.realpath(config.get(section, 'device'))) 138 | if config.has_option(section, 'iosched'): 139 | with open(f'/sys/block/{device}/queue/scheduler', 'w') as f: 140 | f.write(config.get(section, 'iosched')) 141 | 142 | def want_mkfs(test, config, section): 143 | return not test.skip_mkfs_and_mount and config.has_option(section, 'mkfs') 144 | 145 | def mkfs(test, config, section): 146 | if not want_mkfs(test, config, section): 147 | return None 148 | device = config.get(section, 'device') 149 | mkfs_cmd = config.get(section, 'mkfs') 150 | run_command(f'{mkfs_cmd} {device}') 151 | return device 152 | 153 | def want_mnt(test, config, section): 154 | return config.has_option(section, 'mount') and not test.skip_mkfs_and_mount 155 | 156 | class Mount: 157 | def __init__(self, command, device, mount_point): 158 | self.live = False 159 | self.device = device 160 | self.mount_point = mount_point 161 | self.mount_cmd = command 162 | self.mount() 163 | 164 | def mount(self): 165 | run_command(f'{self.mount_cmd} {self.device} {self.mount_point}') 166 | self.live = True 167 | 168 | def umount(self): 169 | if self.live: 170 | run_command(f"umount {self.mount_point}") 171 | self.live = False 172 | # umount sometimes results in asynchronous downstream cleanup, 173 | # e.g. for implicit loop devices. If our device isn't a block 174 | # device, we have to sleep to give that a chance to happen. 175 | if self.is_on_block_device(): 176 | time.sleep(1) 177 | 178 | def cycle_mount(self): 179 | self.umount() 180 | self.mount() 181 | 182 | def timed_cycle_mount(self): 183 | t1 = time.perf_counter_ns() 184 | self.umount() 185 | t2 = time.perf_counter_ns() 186 | self.mount() 187 | t3 = time.perf_counter_ns() 188 | return (t2 - t1, t3 - t2) 189 | 190 | def __enter__(self): 191 | return self 192 | 193 | def __exit__(self, et, ev, etb): 194 | self.umount() 195 | # re-raise 196 | if et is not None: 197 | return False 198 | 199 | def is_on_block_device(self): 200 | st_mode = os.stat(self.device).st_mode 201 | return stat.S_ISBLK(st_mode) 202 | 203 | class IOStats: 204 | def __init__(self, dev): 205 | self.dev_read_iops = 0 206 | self.dev_written_ios = 0 207 | self.dev_read_kbytes = 0 208 | self.dev_written_bytes = 0 209 | self.device = os.path.basename(os.path.realpath(dev)) 210 | 211 | def get_dev_stats(self): 212 | with open(f"/sys/block/{self.device}/stat") as file: 213 | stats = file.readline() 214 | fields = stats.split() 215 | dev_stats = {"dev_read_iops": int(fields[0]), 216 | "dev_read_kbytes": int(fields[2]) * 512 / 1024, 217 | "dev_write_iops": int(fields[4]), 218 | "dev_write_kbytes": int(fields[6]) * 512 / 1024} 219 | return dev_stats 220 | 221 | def __enter__(self): 222 | self.stats_start = self.get_dev_stats() 223 | return self 224 | 225 | def __exit__(self, et, ev, etb): 226 | self.stats_end = self.get_dev_stats() 227 | 228 | def results(self): 229 | # return stats for ios performed between enter <-> end 230 | return {k: self.stats_end[k] - self.stats_start.get(k, 0) for k in self.stats_start} 231 | 232 | class LatencyTracing: 233 | def __init__(self, fns): 234 | self.ps = {} 235 | self.latencies = {} 236 | self.calls = {} 237 | self.fns = fns 238 | 239 | def start_latency_trace(self, fn): 240 | # this sets the max size of a map in bpftrace 241 | # in our case, this is a bound on the number of unique delays we trace 242 | os.environ["BPFTRACE_MAP_KEYS_MAX"] = "65536" 243 | kprobe = f"kprobe:{fn} {{ @start[tid] = nsecs; }}" 244 | kretprobe = f"kretprobe:{fn} {{ if(@start[tid]) {{ $delay = nsecs - @start[tid]; @delays[$delay]++; }} delete(@start[tid]); }}" 245 | end = "END { clear(@start); }" 246 | self.ps[fn] = Popen(["bpftrace", "-e", f"{kprobe} {kretprobe} {end}"], text=True, stdout=PIPE, stderr=PIPE) 247 | 248 | def collect_latency_trace(self, fn): 249 | bt_p = self.ps[fn] 250 | bt_p.send_signal(signal.SIGINT) 251 | # ignore errors in latency tracing; better to let the whole run still complete. 252 | try: 253 | stdout, stderr = bt_p.communicate(timeout=15) 254 | except subprocess.TimeoutExpired: 255 | print("Couldn't interrupt {fn} bpftrace. Kill it and move on.") 256 | bt_p.kill() 257 | return 258 | if bt_p.returncode: 259 | print(f"{fn} bpftrace had an error {bt_p.returncode}. stderr: {stderr.strip()}") 260 | return 261 | self.latencies[fn] = [] 262 | out = stdout.split('\n') 263 | if len(out) == 65536: 264 | raise OverflowError(f"too many unique delay values: {len(out)} while tracing {fn}. Increase BPFTRACE_MAP_KEYS_MAX above") 265 | for l in out: 266 | if not l: 267 | continue 268 | if 'Attaching' in l: 269 | continue 270 | lat_str, count_str = l.split(':') 271 | lat_re = r"@delays\[(\d+)\]" 272 | m = re.match(lat_re, lat_str) 273 | if not m: 274 | continue 275 | lat = int(m.groups()[0]) 276 | count = int(count_str) 277 | self.latencies[fn] += [lat] * count 278 | 279 | def results(self): 280 | r = [] 281 | for fn, lats in self.latencies.items(): 282 | if not lats: 283 | continue 284 | t = {} 285 | t["function"] = fn 286 | t["ns_mean"] = statistics.mean(lats) 287 | t["ns_min"] = min(lats) 288 | t["ns_p50"] = numpy.percentile(lats, 50) 289 | t["ns_p95"] = numpy.percentile(lats, 95) 290 | t["ns_p99"] = numpy.percentile(lats, 99) 291 | t["ns_max"] = max(lats) 292 | t["calls"] = len(lats) 293 | r.append(t) 294 | return r 295 | 296 | def __enter__(self): 297 | for fn in self.fns: 298 | self.start_latency_trace(fn) 299 | return self 300 | 301 | def __exit__(self, et, ev, etb): 302 | for fn in self.fns: 303 | self.collect_latency_trace(fn) 304 | 305 | def results_to_dict(run, include_time=False): 306 | ret_dict = {} 307 | sub_results = list(itertools.chain(run.time_results, run.fio_results, 308 | run.dbench_results, run.fragmentation, 309 | run.latency_traces, 310 | run.io_stats, 311 | run.btrfs_commit_stats, 312 | run.mount_timings)) 313 | for r in sub_results: 314 | ret_dict.update(r.to_dict()) 315 | if include_time: 316 | ret_dict['time'] = run.time 317 | return ret_dict 318 | 319 | def filter_outliers(vs, mean, stdev): 320 | def z(v, mean, stdev): 321 | if not stdev or not mean: 322 | return 0 323 | return (v - mean) / stdev 324 | return [v for v in vs if abs(z(v, mean, stdev)) > 3] 325 | 326 | def avg_results(results): 327 | ret_dict = {} 328 | vals_dict = collections.defaultdict(list) 329 | for run in results: 330 | run_results = results_to_dict(run) 331 | for k,v in run_results.items(): 332 | vals_dict[k].append(v) 333 | for k,vs in vals_dict.items(): 334 | ret_dict[k] = {} 335 | if len(vs) == 1: 336 | ret_dict[k]['mean'] = vs[0] 337 | ret_dict[k]['stdev'] = 0 338 | continue 339 | mean = statistics.mean(vs) 340 | stdev = statistics.stdev(vs) 341 | filtered = filter_outliers(vs, mean, stdev) 342 | ret_dict[k]['mean'] = statistics.mean(vs) 343 | ret_dict[k]['stdev'] = statistics.stdev(vs) 344 | return ret_dict 345 | 346 | def pct_diff(a, b): 347 | if a == 0 and b == 0: 348 | return 0 349 | # kind of silly, but renders reasonably well 350 | if a == 0: 351 | return 100 352 | return ((b - a) / a) * 100 353 | 354 | def color_str(s, color): 355 | ENDC = '\033[0m' 356 | return color + s + ENDC 357 | 358 | def diff_string(a, b, better): 359 | DEFAULT = '\033[99m' 360 | GREEN = '\033[92m' 361 | RED = '\033[91m' 362 | sig_delta = a['stdev'] * 1.96 363 | lo_thresh = a['mean'] - sig_delta 364 | hi_thresh = a['mean'] + sig_delta 365 | below = b['mean'] < lo_thresh 366 | above = b['mean'] > hi_thresh 367 | higher_better = better == HIGHER_IS_BETTER 368 | 369 | bad = (below and higher_better) or (above and not higher_better) 370 | good = (above and higher_better) or (below and not higher_better) 371 | if bad: 372 | color = RED 373 | elif good: 374 | color = GREEN 375 | else: 376 | color = DEFAULT 377 | 378 | diff = pct_diff(a['mean'], b['mean']) 379 | diff_str = "{:.2f}%".format(diff) 380 | return color_str(diff_str, color) 381 | 382 | def check_regression(baseline, recent): 383 | nr_regress_keys = 0 384 | nr_fail = 0 385 | for k,v in baseline.items(): 386 | better = metric_direction(k) 387 | if k in test_regression_keys: 388 | nr_regress_keys += 1 389 | if k not in recent: 390 | return False 391 | if better == HIGHER_IS_BETTER: 392 | threshold = v['mean'] - (v['stdev'] * 1.96) 393 | if recent[k]['value'] < threshold: 394 | recent[k]['regression'] = True 395 | if k in test_regression_keys: 396 | nr_fail += 1 397 | else: 398 | recent[k]['regression'] = False 399 | else: 400 | threshold = v['mean'] + (v['stdev'] * 1.96) 401 | if recent[k]['value'] > threshold: 402 | recent[k]['regression'] = True 403 | if k in test_regression_keys: 404 | nr_fail += 1 405 | else: 406 | recent[k]['regression'] = False 407 | fail_thresh = nr_regress_keys * 10 / 100 408 | if fail_thresh == 0: 409 | fail_thresh = 1 410 | return nr_fail >= fail_thresh 411 | 412 | def print_comparison_table(baseline, results): 413 | table = texttable.Texttable(max_width=100) 414 | table.set_precision(2) 415 | table.set_deco(texttable.Texttable.HEADER) 416 | table.set_cols_dtype(['t', 'a', 'a', 'a', 't']) 417 | table.set_cols_align(['l', 'r', 'r', 'r', 'r']) 418 | table_rows = [["metric", "baseline", "current", "stdev", "diff"]] 419 | for k,v in sorted(baseline.items()): 420 | if k not in results: 421 | continue 422 | if not v['mean'] and not results[k]['mean']: 423 | continue 424 | better = metric_direction(k) 425 | diff_str = diff_string(v, results[k], better) 426 | cur = [k, v['mean'], results[k]['mean'], v['stdev'], diff_str] 427 | table_rows.append(cur) 428 | table.add_rows(table_rows) 429 | print(table.draw()) 430 | 431 | def get_fstype(device): 432 | fstype = subprocess.check_output("blkid -s TYPE -o value "+device, shell=True) 433 | # strip the output b'btrfs\n' 434 | return (str(fstype).removesuffix("\\n'")).removeprefix("b'") 435 | 436 | def get_fsid(device): 437 | cmd = shlex.split(f"blkid -s UUID -o value {device}") 438 | return subprocess.check_output(cmd, text=True).strip() 439 | 440 | def get_readpolicies(device): 441 | fsid = get_fsid(device) 442 | sysfs = open("/sys/fs/btrfs/"+fsid+"/read_policy", "r") 443 | # Strip '[ ]' around the active policy 444 | policies = (((sysfs.read()).strip()).strip("[")).strip("]") 445 | sysfs.close() 446 | return policies 447 | 448 | def get_active_readpolicy(device): 449 | fsid = get_fsid(device) 450 | sysfs = open("/sys/fs/btrfs/"+fsid+"/read_policy", "r") 451 | policies = (sysfs.read()).strip() 452 | # Output is as below, pick the policy within '[ ]' 453 | # device [pid] latency 454 | active = re.search(r"\[([A-Za-z0-9_]+)\]", policies) 455 | sysfs.close() 456 | return active.group(1) 457 | 458 | def set_readpolicy(device, policy="pid"): 459 | if not policy in get_readpolicies(device): 460 | print("Read policy '{}' is invalid".format(policy)) 461 | sys.exit(1) 462 | return 463 | fsid = get_fsid(device) 464 | # Ran out of ideas why run_command fails. 465 | # command = "echo "+policy+" > /sys/fs/btrfs/"+fsid+"/read_policy" 466 | # run_command(command) 467 | sysfs = open("/sys/fs/btrfs/"+fsid+"/read_policy", "w") 468 | ret = sysfs.write(policy) 469 | sysfs.close() 470 | 471 | def has_readpolicy(device): 472 | fsid = get_fsid(device) 473 | return os.path.exists("/sys/fs/btrfs/"+fsid+"/read_policy") 474 | 475 | def collect_commit_stats(device): 476 | fsid = get_fsid(device) 477 | if not os.path.exists(f"/sys/fs/btrfs/{fsid}/commit_stats"): 478 | return {} 479 | commits = 0 480 | total_commit_ms = 0 481 | max_commit_ms = 0 482 | avg_commit_ms = 0.0 483 | with open(f"/sys/fs/btrfs/{fsid}/commit_stats") as f: 484 | for line in f: 485 | if "commits " in line: 486 | commits = int(line.split(" ")[1]) 487 | elif "max_commit_ms " in line: 488 | max_commit_ms = int(line.split(" ")[1]) 489 | elif "total_commit_ms " in line: 490 | total_commit_ms = int(line.split(" ")[1]) 491 | 492 | if commits > 0: 493 | avg_commit_ms = total_commit_ms / commits 494 | return { 'commits': commits, 'avg_commit_ms': avg_commit_ms, 495 | 'max_commit_ms': max_commit_ms } 496 | 497 | def get_tests(test_dir): 498 | tests = [] 499 | oneoffs = [] 500 | for (dirpath, dirnames, filenames) in os.walk(test_dir): 501 | for f in filenames: 502 | if not f.endswith(".py"): 503 | continue 504 | p = dirpath + '/' + f 505 | spec = importlib.util.spec_from_file_location('module.name', p) 506 | m = importlib.util.module_from_spec(spec) 507 | spec.loader.exec_module(m) 508 | attrs = set(dir(m)) - set(dir(PerfTest)) 509 | for cname in attrs: 510 | c = getattr(m, cname) 511 | if inspect.isclass(c) and issubclass(c, PerfTest.PerfTest): 512 | t = c() 513 | if t.oneoff: 514 | oneoffs.append(t) 515 | else: 516 | tests.append(t) 517 | return tests, oneoffs 518 | 519 | def generate_bg_dump(config, frag_dir): 520 | if os.path.exists(f"{frag_dir}/bg-dump.btrd"): 521 | return 522 | env = jinja2.Environment(loader=jinja2.FileSystemLoader(frag_dir)) 523 | template = env.get_template('bg-dump.jinja') 524 | f = open(f'{frag_dir}/bg-dump.btrd', 'w') 525 | f.write(template.render(testdir=config.get('main', 'directory'))) 526 | f.close() 527 | -------------------------------------------------------------------------------- /tests/btrfsbgscalability.py: -------------------------------------------------------------------------------- 1 | from PerfTest import FioTest 2 | from nullblk import NullBlock 3 | import utils 4 | 5 | class BtrfsBgScalability(FioTest): 6 | name = "btrfsbgscalability" 7 | command = ("--name btrfsbgscalability --rw=randwrite --fsync=0 " 8 | "--fallocate=posix --direct=1 " 9 | "--ioengine=io_uring --iodepth=64 --bs=64k --filesize=1g " 10 | "--runtime=300 --time_based --numjobs=8 --thread") 11 | oneoff = True 12 | skip_mkfs_and_mount = True 13 | 14 | def teardown(self, config, results): 15 | directory = config.get('main', 'directory') 16 | self.mnt.umount() 17 | self.nullb_mnt.umount() 18 | self.nullblk = None 19 | 20 | def setup(self, config, section): 21 | directory = config.get('main', 'directory') 22 | self.nullblk = NullBlock() 23 | config_values = { 'submit_queues': '2', 24 | 'size': '16384', 25 | 'memory_backed': '1', 26 | } 27 | self.nullblk.config_values = config_values 28 | try: 29 | self.nullblk.start() 30 | except Exception as e: 31 | raise utils.NotRunException(f"We don't have nullblk support loaded {e}") 32 | 33 | mkfsopts = "-f -R free-space-tree -O no-holes" 34 | mntcmd = "mount -o ssd,nodatacow" 35 | 36 | # First create the nullblk fs to load the loop device onto 37 | command = f'mkfs.btrfs {mkfsopts} /dev/nullb0' 38 | utils.run_command(command) 39 | self.nullb_mnt = utils.Mount(mntcmd, '/dev/nullb0', directory) 40 | 41 | # Now create the loop device 42 | loopdir = f'{directory}/loop' 43 | loopfile = f'{directory}/loopfile' 44 | utils.mkdir_p(f'{directory}/loop') 45 | utils.run_command(f'truncate -s 4T {loopfile}') 46 | utils.run_command(f'mkfs.btrfs {mkfsopts} {loopfile}') 47 | self.dev = loopfile 48 | self.mnt = utils.Mount(mntcmd, loopfile, loopdir) 49 | 50 | # Trigger the allocation of about 3500 data block groups, without 51 | # actually consuming space on the underlying filesystem, just to make 52 | # the tree of block groups large 53 | utils.run_command(f'fallocate -l 3500G {loopdir}/filler') 54 | 55 | # We override test here just because we create a loopback device ontop of 56 | # the directory and want to use a different directory than the one we 57 | # mounted the nullblk ontop of 58 | def test(self, run, config, results): 59 | directory = self.mnt.mount_point 60 | command = self.default_cmd(results) 61 | command += f' --directory {directory} ' 62 | command += self.command 63 | utils.run_command(command) 64 | -------------------------------------------------------------------------------- /tests/buffered-append-sync.py: -------------------------------------------------------------------------------- 1 | from PerfTest import FioTest 2 | 3 | class Bufferedappendsync(FioTest): 4 | name = "bufferedappenddatasync" 5 | command = ("--name bufferedappendsync --direct=0 --size=1g --rw=write " 6 | "--fdatasync=1 --ioengine=sync") 7 | -------------------------------------------------------------------------------- /tests/buffered-randwrite-16g.py: -------------------------------------------------------------------------------- 1 | from PerfTest import FioTest 2 | 3 | class BufferedRandwrite16g(FioTest): 4 | name = "bufferedrandwrite16g" 5 | command = ("--name bufferedrandwrite16g " 6 | "--readwrite randwrite --size 16G --ioengine psync " 7 | "--end_fsync 1 --fallocate none") 8 | -------------------------------------------------------------------------------- /tests/dbench-60.py: -------------------------------------------------------------------------------- 1 | from PerfTest import DbenchTest 2 | 3 | class Dbench60(DbenchTest): 4 | name = "dbench60" 5 | command = "60" 6 | -------------------------------------------------------------------------------- /tests/dio-4kbs-16threads.py: -------------------------------------------------------------------------------- 1 | from PerfTest import FioTest 2 | 3 | class Dio4kbs16threads(FioTest): 4 | name = "dio4kbs16threads" 5 | command = ("--name dio4kbs16threads --direct=1 --size=1g --rw=randwrite " 6 | "--norandommap --runtime=60 --iodepth=1024 --nrfiles=16 " 7 | "--numjobs=16") 8 | -------------------------------------------------------------------------------- /tests/dio-randread.py: -------------------------------------------------------------------------------- 1 | import sys 2 | from PerfTest import FioTest 3 | from utils import get_fstype,set_readpolicy,get_active_readpolicy,has_readpolicy 4 | from utils import NotRunException 5 | 6 | class DioRandread(FioTest): 7 | name = "diorandread" 8 | command = ("--name diorandread --direct=1 --size=1g --rw=randread " 9 | "--runtime=60 --iodepth=1024 --nrfiles=16 " 10 | "--numjobs=16") 11 | 12 | def setup(self, config, section): 13 | device = config.get(section, 'device') 14 | 15 | if not get_fstype(device) == "btrfs": 16 | return 17 | 18 | if config.has_option(section, 'readpolicy'): 19 | policy = config.get(section, 'readpolicy') 20 | if not has_readpolicy(device): 21 | raise NotRunException("Kernel does not support readpolicy") 22 | 23 | set_readpolicy(config.get(section, 'device'), policy) 24 | policy = get_active_readpolicy(config.get(section, 'device')) 25 | print("\tReadpolicy is set to '{}'".format(policy)) 26 | -------------------------------------------------------------------------------- /tests/empty-files-500k.py: -------------------------------------------------------------------------------- 1 | from PerfTest import FioTest 2 | 3 | class EmptyFiles500k(FioTest): 4 | name = "emptyfiles500k" 5 | command = ("--name emptyfiles500k --create_on_open=1 --nrfiles=31250 " 6 | "--readwrite=write --ioengine=filecreate --fallocate=none " 7 | "--filesize=4k --openfiles=1") 8 | -------------------------------------------------------------------------------- /tests/randwrite-2xram.py: -------------------------------------------------------------------------------- 1 | from PerfTest import FioTest 2 | import psutil 3 | import configparser 4 | 5 | class Randwrite2xRam(FioTest): 6 | name = "randwrite2xram" 7 | command = ("--name randwrite2xram --direct=0 --ioengine=sync --thread " 8 | "--invalidate=1 --runtime=300 " 9 | "--fallocate=none --ramp_time=10 --new_group --rw=randwrite " 10 | "--size=SIZE --numjobs=4 --bs=BLOCKSIZE --fsync_on_close=0 " 11 | "--end_fsync=0") 12 | 13 | def setup(self, config, section): 14 | bs = config.get(section, "blocksize", fallback="4k") 15 | self.command = Randwrite2xRam.command.replace('BLOCKSIZE', bs) 16 | 17 | mem = psutil.virtual_memory() 18 | self.command = self.command.replace('SIZE', str((mem.total*2)//4)) 19 | 20 | -------------------------------------------------------------------------------- /tests/small-files-100k.py: -------------------------------------------------------------------------------- 1 | from PerfTest import FioTest 2 | 3 | class SmallFiles100k(FioTest): 4 | name = "smallfiles100k" 5 | command = ("--name=smallfiles100k --nrfiles=100000 --blocksize_unaligned=1 " 6 | "--filesize=10:1m --readwrite=write --fallocate=none --numjobs=4 " 7 | "--create_on_open=1 --openfiles=500") 8 | -------------------------------------------------------------------------------- /tests/untar-firefox.py: -------------------------------------------------------------------------------- 1 | from PerfTest import TimeTest 2 | import utils 3 | 4 | class UntarFirefox(TimeTest): 5 | name = "untarfirefox" 6 | command = "tar -xf firefox-87.0b5.source.tar.xz -C DIRECTORY" 7 | 8 | def setup(self, config, section): 9 | utils.run_command("wget -nc https://archive.mozilla.org/pub/firefox/releases/87.0b5/source/firefox-87.0b5.source.tar.xz") 10 | -------------------------------------------------------------------------------- /www/style.css: -------------------------------------------------------------------------------- 1 | a {color:blue} 2 | a:link{color:inherit} 3 | a:active{color:inherit} 4 | a:visited{color:inherit} 5 | a:hover{color:inherit} 6 | 7 | table.summary th { 8 | text-align: center; 9 | } 10 | 11 | table.summary td { 12 | text-align: left; 13 | padding: 5px; 14 | } 15 | 16 | table.results_time { 17 | float: left; 18 | max-width: 500px; 19 | } 20 | 21 | table.results { 22 | float: left; 23 | max-width: 300px; 24 | } 25 | 26 | table.results_output { 27 | float: left; 28 | max-width: 700px; 29 | } 30 | 31 | table.runs { 32 | float: left; 33 | max-width: 900px; 34 | } 35 | 36 | table.results { 37 | float: left; 38 | max-width: 900px; 39 | } 40 | 41 | th { 42 | height: 50px; 43 | } 44 | 45 | table.runs th, td { 46 | padding: 15px; 47 | text-align: left; 48 | } 49 | 50 | table.results th, td { 51 | padding: 5px; 52 | text-align: left; 53 | } 54 | 55 | td.passing { 56 | background-color: #4CAF50; 57 | color: white; 58 | } 59 | 60 | td.failing { 61 | background-color: Tomato; 62 | color: white; 63 | } 64 | 65 | th.runs { 66 | background-color: Pink; 67 | color: black; 68 | } 69 | --------------------------------------------------------------------------------