├── .gitignore ├── CHANGES ├── COPYING ├── README.md ├── btrfs-heatmap ├── doc ├── README │ ├── animated-balance-small.gif │ ├── example-238gib.png │ ├── hilbert_256.png │ └── mekker.png ├── curves.md ├── curves │ ├── curves.py │ ├── hilbert-1.dia │ ├── hilbert-1.png │ ├── hilbert-2.dia │ ├── hilbert-2.png │ ├── hilbert-fs.png │ ├── hilbert-order-05-size-08.gif │ ├── hilbert.dia │ ├── hilbert.png │ ├── line.dia │ ├── line.png │ ├── linear-fs.png │ ├── linear.dia │ ├── linear.png │ ├── snake-fs.png │ ├── snake.dia │ ├── snake.png │ └── ubuishilbert.jpg ├── extent.md ├── extent │ ├── CHUNK_TREE.png │ ├── CSUM_TREE.png │ ├── DATA_RELOC_TREE.png │ ├── DEV_TREE.png │ ├── EXTENT_TREE.png │ ├── FREE_SPACE_TREE.png │ ├── FS_TREE.png │ ├── QUOTA_TREE.png │ ├── ROOT_TREE.png │ ├── UUID_TREE.png │ ├── example-data.png │ ├── example-metadata.png │ └── update_color_map.py ├── scripting.md ├── scripting │ ├── 4-highest.png │ ├── all-bg.png │ ├── device_1.png │ ├── device_2.png │ ├── full-fs.png │ ├── stripes_on_device_1.png │ └── stripes_on_device_2.png ├── sort.md └── sort │ ├── physical-dev-extents.png │ └── virtual-chunks.png └── man └── btrfs-heatmap.1 /.gitignore: -------------------------------------------------------------------------------- 1 | /*.png 2 | /btrfs 3 | -------------------------------------------------------------------------------- /CHANGES: -------------------------------------------------------------------------------- 1 | btrfs-heatmap v9, Oct 14, 2020 2 | * Support image output to stdout! This is an awesome feature, because 3 | we can pipe the output into a tool like catimg to show pictures in a 4 | text based terminal instead of having to copy png files around! 5 | * Rename heatmap.py to btrfs-heatmap. It should be shipped like that 6 | in the distro packaging, and it makes documentation etc. more 7 | straightforward. 8 | * Move the manual page from the debian branch into the normal source 9 | tree. 10 | * Allow setting colors for specific bg flags. In order to use this 11 | feature, it's still necessary to edit the dev_extent_colors 12 | dictionary in the program code. There's no command line option or 13 | anything. But, at least it can be done. 14 | * Fixes: 15 | - Actually use the python-btrfs FileSystem object context manager. 16 | - Default to white color for unknown tree number. This prevents the 17 | program from crashing when a yet unknown metadata tree number is 18 | encountered. 19 | * Various small documentation and bug fixes. 20 | 21 | btrfs-heatmap v8, Jan 19, 2019 22 | * This program now needs at least python-btrfs v10 23 | * Make sorting on virtual address actually produce correct output 24 | for RAID0, RAID10, RAID5 and RAID6. 25 | * Change license to MIT (Expat) 26 | -------------------------------------------------------------------------------- /COPYING: -------------------------------------------------------------------------------- 1 | Copyright (C) 2016 Hans van Kranenburg 2 | 3 | Permission is hereby granted, free of charge, to any person obtaining 4 | a copy of this software and associated documentation files (the 5 | "Software"), to deal in the Software without restriction, including 6 | without limitation the rights to use, copy, modify, merge, publish, 7 | distribute, sublicense, and/or sell copies of the Software, and to 8 | permit persons to whom the Software is furnished to do so, subject to 9 | the following conditions: 10 | 11 | The above copyright notice and this permission notice shall be included 12 | in all copies or substantial portions of the Software. 13 | 14 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, 15 | EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF 16 | MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. 17 | IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY 18 | CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, 19 | TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE 20 | SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 21 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | Btrfs Heatmap 2 | ============= 3 | 4 | The btrfs heatmap script creates a visualization of how a btrfs filesystem is 5 | using the underlying disk space of the block devices that are added to it. 6 | 7 | ## What does it look like? 8 | 9 | 238GiB file system | 10 | :--------------------------:| 11 | ![filesystem](doc/README/example-238gib.png)| 12 | 13 | 14 | This picture shows the 238GiB filesystem in my computer at work. It was 15 | generated using the command `btrfs-heatmap --size 9 /`, resulting in a 512x512 16 | pixel png. The black parts are unallocated disk space. Raw disk space that is 17 | allocated to be used for data (white), metadata (blue) or system (red) gets 18 | brighter if the fill factor of block groups is higher. 19 | 20 | ## How do I create a picture like this of my own computer? 21 | 22 | Well, first install the program. It's probably available as package 23 | 'btrfs-heatmap' in your favourite Linux distro. 24 | 25 | When pointing btrfs-heatmap to a mounted btrfs filesystem location, it will ask 26 | the linux kernel for usage information and build a png picture reflecting that 27 | low level information. Because the needed information is retrieved using the 28 | btrfs kernel API, it has to be run as root user: 29 | 30 | ``` 31 | -$ sudo btrfs-heatmap /mountpoint 32 | ``` 33 | 34 | ## I have a picture now, with quite a long filename, why? 35 | 36 | The filename of the png picture is a combination of the filesystem ID and a 37 | timestamp by default, so that if you create multiple of them, they nicely pile 38 | up as input for creating a timelapse video. 39 | 40 | Creating multiple ones is as easy as doing `watch 'btrfs-heatmap /mountpoint'` 41 | 42 | ## Can I directly view the resulting image? 43 | 44 | Yes! For this, output to stdout can be used. When using a dash as output 45 | filename (`-o -`), the resulting png data is written to stdout directly, which 46 | can be connected to an image viewer like `catimg` for displaying the image 47 | right away in a terminal screen using ansi color codes. For a pop up window, a 48 | program like `feh` can be used. 49 | 50 | A typical use case for using this functionality and `catimg` is to conveniently 51 | view results on a remote server without having to copy resulting png files 52 | around. 53 | 54 | ## Where's what? In what corner is the first or last byte located? 55 | 56 | By default, the ordering inside the picture is based on a [Hilbert 57 | Curve](doc/curves.md). The lowest physical address of the block devices is 58 | located in the bottom left corner. From there it walks up, to the right and 59 | down again. 60 | 61 | Hilbert Curve | Example Image 62 | :------------:|:----------: 63 | [![Hilbert Curve](doc/README/hilbert_256.png)](doc/curves.md)|[![Mekker](doc/README/mekker.png)](doc/curves.md) 64 | 65 | ## In btrfs technical terms speaking, what does it display? 66 | 67 | The picture that is generated by default shows the physical address space of a 68 | filesystem, by walking all dev extents of all devices in the filesystem using 69 | the search ioctl and concatenating all information into a single big image. The 70 | usage values are computed by looking up usage counters in the block group items 71 | from the extent tree. 72 | 73 | It's also possible to have the picture sorted by btrfs virtual address space 74 | instead, or to create pictures of the contents of block groups, on extent 75 | level. For more information, see links to additional documentation below. 76 | 77 | ## How do I create a timelapse movie out of this? 78 | 79 | Here's an example command to create an mp4 video out of all the png files if 80 | you create multiple ones over time: 81 | ``` 82 | ffmpeg -framerate 2 -pattern_type glob -i '*.png' -c:v libx264 -r 25 -pix_fmt yuv420p btrfs-heatmap.mp4 83 | ``` 84 | By varying the `-framerate` option, you can make the video go faster or slower. 85 | 86 | Another option is to create an animated gif. By varying the `-delay` option, 87 | you change the speed. 88 | ``` 89 | convert -layers optimize-frame -loop 0 -delay 20 *.png btrfs-heatmap.gif 90 | ``` 91 | 92 | The next picture is an animated gif of running `btrfs balance` on the data of 93 | the filesystem which the first picture was also taken of. You can see how all 94 | free space is defragmented by packing data together: 95 | 96 | btrfs balance | 97 | :--------------------------:| 98 | ![animated gif balance](doc/README/animated-balance-small.gif)| 99 | 100 | ## More documentation and advanced usage 101 | 102 | * The built-in `--help` option will show all functionality that is available 103 | through the command line. 104 | * Different ways to walk the pixel grid: [Hilbert, Snake, Linear](doc/curves.md). 105 | * Sorting the picture on [virtual instead of physical address space](doc/sort.md). 106 | * [Extent level pictures](doc/extent.md) show detailed usage of the virtual 107 | address space inside block groups. 108 | * By [scripting btrfs-heatmap](doc/scripting.md) it's possible to make pictures 109 | of single devices, or any combination of block groups. 110 | 111 | ## Feedback 112 | 113 | Let me know if this program was useful for you, or if you have brilliant ideas 114 | about how to improve it. 115 | 116 | You can reach me on IRC in #btrfs on Freenode (I'm Knorrie), use the github 117 | issue system or send me an email on hans@knorrie.org 118 | -------------------------------------------------------------------------------- /btrfs-heatmap: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python3 2 | # 3 | # Copyright (C) 2016 Hans van Kranenburg 4 | # 5 | # Permission is hereby granted, free of charge, to any person obtaining 6 | # a copy of this software and associated documentation files (the 7 | # "Software"), to deal in the Software without restriction, including 8 | # without limitation the rights to use, copy, modify, merge, publish, 9 | # distribute, sublicense, and/or sell copies of the Software, and to 10 | # permit persons to whom the Software is furnished to do so, subject to 11 | # the following conditions: 12 | # 13 | # The above copyright notice and this permission notice shall be included 14 | # in all copies or substantial portions of the Software. 15 | # 16 | # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, 17 | # EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF 18 | # MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. 19 | # IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY 20 | # CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, 21 | # TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE 22 | # SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 23 | 24 | import argparse 25 | import btrfs 26 | import errno 27 | import os 28 | import struct 29 | import sys 30 | import types 31 | import zlib 32 | 33 | 34 | class HeatmapError(Exception): 35 | pass 36 | 37 | 38 | def parse_args(): 39 | parser = argparse.ArgumentParser() 40 | parser.add_argument( 41 | "--order", 42 | type=int, 43 | help="Hilbert curve order (default: automatically chosen)", 44 | ) 45 | parser.add_argument( 46 | "--size", 47 | type=int, 48 | help="Image size (default: 10). Height/width is 2^size", 49 | ) 50 | parser.add_argument( 51 | "--sort", 52 | choices=['physical', 'virtual'], 53 | default='physical', 54 | help="Show disk usage sorted on dev_extent (physical) or chunk/stripe (virtual)" 55 | ) 56 | parser.add_argument( 57 | "--blockgroup", 58 | type=int, 59 | help="Instead of a filesystem overview, show extents in a block group", 60 | ) 61 | parser.add_argument( 62 | "-v", 63 | "--verbose", 64 | action="count", 65 | help="increase debug output verbosity (-v, -vv, -vvv, etc)", 66 | ) 67 | parser.add_argument( 68 | "-q", 69 | "--quiet", 70 | action="count", 71 | help="decrease debug output verbosity (-q, -qq, -qqq, etc)", 72 | ) 73 | parser.add_argument( 74 | "-o", 75 | "--output", 76 | dest="output", 77 | help="Output png file name or directory (default: filename automatically chosen)", 78 | ) 79 | parser.add_argument( 80 | "--curve", 81 | choices=['hilbert', 'linear', 'snake'], 82 | default='hilbert', 83 | help="Space filling curve type or alternative. Default is hilbert.", 84 | ) 85 | parser.add_argument( 86 | "mountpoint", 87 | help="Btrfs filesystem mountpoint", 88 | ) 89 | return parser.parse_args() 90 | 91 | 92 | struct_color = struct.Struct('!BBB') 93 | 94 | black = (0x00, 0x00, 0x00) 95 | white = (0xff, 0xff, 0xff) 96 | 97 | p_red = (0xca, 0x53, 0x5c) 98 | fuchsia = (0xde, 0x5d, 0x94) 99 | curry = (0xf9, 0xe1, 0x7e) 100 | clover = (0x6e, 0xa6, 0x34) 101 | moss = (0x81, 0x88, 0x3c) 102 | bluebell = (0xaa, 0xcc, 0xeb) 103 | pool = (0x8f, 0xdd, 0xea) 104 | beet = (0x9d, 0x54, 0x9c) 105 | aubergine = (0x6a, 0x5a, 0x7f) 106 | plum = (0xdb, 0xc9, 0xea) 107 | slate = (0x75, 0x77, 0x7b) 108 | chocolate = (0x6f, 0x5e, 0x55) 109 | 110 | red = (0xff, 0x00, 0x33) 111 | blue = (0x00, 0x00, 0xff) 112 | blue_white = (0x99, 0xcc, 0xff) # for mixed bg 113 | 114 | dev_extent_colors = { 115 | btrfs.BLOCK_GROUP_DATA: white, 116 | btrfs.BLOCK_GROUP_METADATA: blue, 117 | btrfs.BLOCK_GROUP_SYSTEM: red, 118 | btrfs.BLOCK_GROUP_DATA | btrfs.BLOCK_GROUP_METADATA: blue_white, 119 | } 120 | 121 | metadata_extent_colors = { 122 | btrfs.ctree.ROOT_TREE_OBJECTID: p_red, 123 | btrfs.ctree.EXTENT_TREE_OBJECTID: beet, 124 | btrfs.ctree.CHUNK_TREE_OBJECTID: moss, 125 | btrfs.ctree.DEV_TREE_OBJECTID: aubergine, 126 | btrfs.ctree.FS_TREE_OBJECTID: bluebell, 127 | btrfs.ctree.CSUM_TREE_OBJECTID: clover, 128 | btrfs.ctree.QUOTA_TREE_OBJECTID: fuchsia, 129 | btrfs.ctree.UUID_TREE_OBJECTID: chocolate, 130 | btrfs.ctree.FREE_SPACE_TREE_OBJECTID: plum, 131 | btrfs.ctree.DATA_RELOC_TREE_OBJECTID: slate, 132 | } 133 | 134 | 135 | def hilbert(order): 136 | U = (-1, 0) 137 | R = (0, 1) 138 | D = (1, 0) 139 | L = (0, -1) 140 | 141 | URDR = (U, R, D, R) 142 | RULU = (R, U, L, U) 143 | URDD = (U, R, D, D) 144 | LDRR = (L, D, R, R) 145 | RULL = (R, U, L, L) 146 | DLUU = (D, L, U, U) 147 | LDRD = (L, D, R, D) 148 | DLUL = (D, L, U, L) 149 | 150 | inception = { 151 | URDR: (RULU, URDR, URDD, LDRR), 152 | RULU: (URDR, RULU, RULL, DLUU), 153 | URDD: (RULU, URDR, URDD, LDRD), 154 | LDRR: (DLUL, LDRD, LDRR, URDR), 155 | RULL: (URDR, RULU, RULL, DLUL), 156 | DLUU: (LDRD, DLUL, DLUU, RULU), 157 | LDRD: (DLUL, LDRD, LDRR, URDD), 158 | DLUL: (LDRD, DLUL, DLUU, RULL) 159 | } 160 | 161 | pos = [(2 ** order) - 1, 0, 0] # y, x, linear 162 | 163 | def walk(steps, level): 164 | if level > 1: 165 | for substeps in inception[steps]: 166 | for subpos in walk(substeps, level - 1): 167 | yield subpos 168 | else: 169 | for step in steps: 170 | yield pos 171 | pos[0] += step[0] # y 172 | pos[1] += step[1] # x 173 | pos[2] += 1 # linear 174 | 175 | return walk(URDR, order) 176 | 177 | 178 | def linear(order): 179 | edge_len = 2 ** order 180 | l = 0 181 | for y in range(0, edge_len): 182 | for x in range(0, edge_len): 183 | yield (y, x, l) 184 | l += 1 185 | 186 | 187 | def snake(order): 188 | edge_len = 2 ** order 189 | l = 0 190 | for y in range(0, edge_len, 2): 191 | for x in range(0, edge_len): 192 | yield (y, x, l) 193 | l += 1 194 | y += 1 195 | for x in range(edge_len - 1, -1, -1): 196 | yield (y, x, l) 197 | l += 1 198 | 199 | 200 | curves = { 201 | 'hilbert': hilbert, 202 | 'linear': linear, 203 | 'snake': snake, 204 | } 205 | 206 | 207 | class Grid(object): 208 | def __init__(self, order, size, total_bytes, default_granularity, verbose, 209 | min_brightness=None, curve=None): 210 | self.order, self.size = choose_order_size(order, size, total_bytes, default_granularity) 211 | self.verbose = verbose 212 | if curve is None: 213 | curve = 'hilbert' 214 | self.curve = curves.get(curve)(self.order) 215 | self._pixel_mix = [] 216 | self._pixel_dirty = False 217 | self._next_pixel() 218 | self.height = 2 ** self.order 219 | self.width = 2 ** self.order 220 | self.num_steps = (2 ** self.order) ** 2 221 | self.total_bytes = total_bytes 222 | self.bytes_per_pixel = total_bytes / self.num_steps 223 | self._color_cache = {} 224 | self._add_color_cache(black) 225 | self._grid = [[self._color_cache[black] 226 | for x in range(self.width)] 227 | for y in range(self.height)] 228 | self._finished = False 229 | if min_brightness is None: 230 | self._min_brightness = 0.1 231 | else: 232 | if min_brightness < 0 or min_brightness > 1: 233 | raise ValueError("min_brightness out of range (need >= 0 and <= 1)") 234 | self._min_brightness = min_brightness 235 | if self.verbose >= 0: 236 | print("grid curve {} order {} size {} height {} width {} total_bytes {} " 237 | "bytes_per_pixel {}".format(curve, self.order, self.size, 238 | self.height, self.width, total_bytes, 239 | self.bytes_per_pixel, self.num_steps)) 240 | 241 | def _next_pixel(self): 242 | if self._pixel_dirty is True: 243 | self._finish_pixel() 244 | self.y, self.x, self.linear = next(self.curve) 245 | 246 | def _add_to_pixel_mix(self, color, used_pct, pixel_pct): 247 | self._pixel_mix.append((color, used_pct, pixel_pct)) 248 | self._pixel_dirty = True 249 | 250 | def _pixel_mix_to_rgbytes(self): 251 | R_composite = sum(color[0] * pixel_pct for color, _, pixel_pct in self._pixel_mix) 252 | G_composite = sum(color[1] * pixel_pct for color, _, pixel_pct in self._pixel_mix) 253 | B_composite = sum(color[2] * pixel_pct for color, _, pixel_pct in self._pixel_mix) 254 | 255 | weighted_usage = sum(used_pct * pixel_pct 256 | for _, used_pct, pixel_pct in self._pixel_mix) 257 | weighted_usage_min_bright = self._min_brightness + \ 258 | weighted_usage * (1 - self._min_brightness) 259 | 260 | RGB = ( 261 | int(round(R_composite * weighted_usage_min_bright)), 262 | int(round(G_composite * weighted_usage_min_bright)), 263 | int(round(B_composite * weighted_usage_min_bright)), 264 | ) 265 | 266 | if RGB in self._color_cache: 267 | return self._color_cache[RGB] 268 | return self._add_color_cache(RGB) 269 | 270 | def _add_color_cache(self, color): 271 | rgbytes = struct_color.pack(*color) 272 | self._color_cache[color] = rgbytes 273 | return rgbytes 274 | 275 | def _set_pixel(self, rgbytes): 276 | self._grid[self.y][self.x] = rgbytes 277 | 278 | def _finish_pixel(self): 279 | rgbytes = self._pixel_mix_to_rgbytes() 280 | self._set_pixel(rgbytes) 281 | if self.verbose >= 3: 282 | print(" pixel y {} x{} linear {} rgb #{:02x}{:02x}{:02x}".format( 283 | self.y, self.x, self.linear, *[byte for byte in rgbytes])) 284 | self._pixel_mix = [] 285 | self._pixel_dirty = False 286 | 287 | def fill(self, first_byte, length, used_pct, color=white): 288 | if self._finished is True: 289 | raise Exception("Cannot change grid any more after retrieving the result once!") 290 | first_pixel = int(first_byte / self.bytes_per_pixel) 291 | last_pixel = int((first_byte + length - 1) / self.bytes_per_pixel) 292 | 293 | while self.linear < first_pixel: 294 | self._next_pixel() 295 | 296 | if first_pixel == last_pixel: 297 | pct_of_pixel = length / self.bytes_per_pixel 298 | if self.verbose >= 2: 299 | print(" in_pixel {0} {1:.2f}%".format(first_pixel, pct_of_pixel * 100)) 300 | self._add_to_pixel_mix(color, used_pct, pct_of_pixel) 301 | else: 302 | pct_of_first_pixel = \ 303 | (self.bytes_per_pixel - (first_byte % self.bytes_per_pixel)) / self.bytes_per_pixel 304 | pct_of_last_pixel = \ 305 | ((first_byte + length) % self.bytes_per_pixel) / self.bytes_per_pixel 306 | if pct_of_last_pixel == 0: 307 | pct_of_last_pixel = 1 308 | if self.verbose >= 2: 309 | print(" first_pixel {0} {1:.2f}% last_pixel {2} {3:.2f}%".format( 310 | first_pixel, pct_of_first_pixel * 100, last_pixel, pct_of_last_pixel * 100)) 311 | # add our part of the first pixel, may be shared with previous fill 312 | self._add_to_pixel_mix(color, used_pct, pct_of_first_pixel) 313 | # all intermediate pixels are ours, set brightness directly 314 | if self.linear < last_pixel - 1: 315 | self._next_pixel() 316 | self._add_to_pixel_mix(color, used_pct, pixel_pct=1) 317 | rgbytes = self._pixel_mix_to_rgbytes() 318 | self._set_pixel(rgbytes) 319 | if self.verbose >= 3: 320 | print(" pixel range linear {} to {} rgb #{:02x}{:02x}{:02x}".format( 321 | self.linear, last_pixel - 1, *[byte for byte in rgbytes])) 322 | while self.linear < last_pixel - 1: 323 | self._next_pixel() 324 | self._set_pixel(rgbytes) 325 | self._next_pixel() 326 | # add our part of the last pixel, may be shared with next fill 327 | self._add_to_pixel_mix(color, used_pct, pct_of_last_pixel) 328 | 329 | def write_png(self, pngfile): 330 | if self.verbose >= 0: 331 | print("pngfile {}".format(pngfile)) 332 | if self._finished is False: 333 | if self._pixel_dirty is True: 334 | self._finish_pixel() 335 | self._finished = True 336 | if self.size > self.order: 337 | scale = 2 ** (self.size - self.order) 338 | rows = ((pix for pix in row for _ in range(scale)) 339 | for row in self._grid for _ in range(scale)) 340 | _write_png(pngfile, self.width * scale, self.height * scale, rows) 341 | else: 342 | _write_png(pngfile, self.width, self.height, self._grid) 343 | 344 | 345 | def walk_chunks(fs, devices=None, order=None, size=None, 346 | default_granularity=33554432, verbose=0, min_brightness=None, curve=None): 347 | if devices is None: 348 | devices = list(fs.devices()) 349 | devids = None 350 | if verbose >= 0: 351 | print("scope chunks") 352 | else: 353 | if isinstance(devices, types.GeneratorType): 354 | devices = list(devices) 355 | devids = [device.devid for device in devices] 356 | if verbose >= 0: 357 | print("scope chunk stripes on devices {}".format(' '.join(map(str, devids)))) 358 | 359 | total_bytes = sum(device.total_bytes for device in devices) 360 | 361 | grid = Grid(order, size, total_bytes, default_granularity, verbose, min_brightness, curve) 362 | byte_offset = 0 363 | for chunk in fs.chunks(): 364 | if devids is None: 365 | stripes = chunk.stripes 366 | else: 367 | stripes = [stripe for stripe in chunk.stripes if stripe.devid in devids] 368 | if len(stripes) == 0: 369 | continue 370 | try: 371 | block_group = fs.block_group(chunk.vaddr, chunk.length) 372 | except btrfs.ctree.ItemNotFoundError: 373 | continue 374 | used_pct = block_group.used / block_group.length 375 | length = btrfs.volumes.chunk_to_dev_extent_length(chunk) * len(stripes) 376 | if verbose >= 1: 377 | print(block_group) 378 | print(chunk) 379 | print("allocated physical space for chunk at {}: {}".format( 380 | chunk.vaddr, btrfs.utils.pretty_size(length))) 381 | if verbose >= 2: 382 | for stripe in stripes: 383 | print(" {}".format(stripe)) 384 | if block_group.flags in dev_extent_colors: 385 | color = dev_extent_colors[block_group.flags] 386 | else: 387 | color = dev_extent_colors[block_group.flags & btrfs.BLOCK_GROUP_TYPE_MASK] 388 | grid.fill(byte_offset, length, used_pct, color) 389 | byte_offset += length 390 | return grid 391 | 392 | 393 | def walk_dev_extents(fs, devices=None, order=None, size=None, 394 | default_granularity=33554432, verbose=0, min_brightness=None, curve=None): 395 | if devices is None: 396 | devices = list(fs.devices()) 397 | dev_extents = fs.dev_extents() 398 | else: 399 | if isinstance(devices, types.GeneratorType): 400 | devices = list(devices) 401 | dev_extents = (dev_extent 402 | for device in devices 403 | for dev_extent in fs.dev_extents(device.devid, device.devid)) 404 | 405 | if verbose >= 0: 406 | print("scope device {}".format(' '.join([str(device.devid) for device in devices]))) 407 | total_bytes = 0 408 | device_grid_offset = {} 409 | for device in devices: 410 | device_grid_offset[device.devid] = total_bytes 411 | total_bytes += device.total_bytes 412 | 413 | grid = Grid(order, size, total_bytes, default_granularity, verbose, min_brightness, curve) 414 | block_group_cache = {} 415 | for dev_extent in dev_extents: 416 | if dev_extent.vaddr in block_group_cache: 417 | block_group = block_group_cache[dev_extent.vaddr] 418 | else: 419 | try: 420 | block_group = fs.block_group(dev_extent.vaddr) 421 | except IndexError: 422 | continue 423 | if block_group.flags & btrfs.BLOCK_GROUP_PROFILE_MASK != 0: 424 | block_group_cache[dev_extent.vaddr] = block_group 425 | used_pct = block_group.used / block_group.length 426 | if verbose >= 1: 427 | print("dev_extent devid {0} paddr {1} length {2} pend {3} type {4} " 428 | "used_pct {5:.2f}".format(dev_extent.devid, dev_extent.paddr, dev_extent.length, 429 | dev_extent.paddr + dev_extent.length - 1, 430 | btrfs.utils.block_group_flags_str(block_group.flags), 431 | used_pct * 100)) 432 | first_byte = device_grid_offset[dev_extent.devid] + dev_extent.paddr 433 | if block_group.flags in dev_extent_colors: 434 | color = dev_extent_colors[block_group.flags] 435 | else: 436 | color = dev_extent_colors[block_group.flags & btrfs.BLOCK_GROUP_TYPE_MASK] 437 | grid.fill(first_byte, dev_extent.length, used_pct, color) 438 | return grid 439 | 440 | 441 | def _get_metadata_root(extent): 442 | if extent.refs > 1: 443 | return btrfs.ctree.FS_TREE_OBJECTID 444 | if len(extent.shared_block_refs) > 0: 445 | return btrfs.ctree.FS_TREE_OBJECTID 446 | root = extent.tree_block_refs[0].root 447 | if root >= btrfs.ctree.FIRST_FREE_OBJECTID and root <= btrfs.ctree.LAST_FREE_OBJECTID: 448 | return btrfs.ctree.FS_TREE_OBJECTID 449 | return root 450 | 451 | 452 | def walk_extents(fs, block_groups, order=None, size=None, default_granularity=None, verbose=0, 453 | curve=None): 454 | if isinstance(block_groups, types.GeneratorType): 455 | block_groups = list(block_groups) 456 | fs_info = fs.fs_info() 457 | nodesize = fs_info.nodesize 458 | 459 | if default_granularity is None: 460 | default_granularity = fs_info.sectorsize 461 | 462 | if verbose >= 0: 463 | print("scope block_group {}".format(' '.join([str(b.vaddr) for b in block_groups]))) 464 | total_bytes = 0 465 | block_group_grid_offset = {} 466 | for block_group in block_groups: 467 | block_group_grid_offset[block_group] = total_bytes - block_group.vaddr 468 | total_bytes += block_group.length 469 | 470 | grid = Grid(order, size, total_bytes, default_granularity, verbose, curve=curve) 471 | 472 | tree = btrfs.ctree.EXTENT_TREE_OBJECTID 473 | for block_group in block_groups: 474 | if verbose > 0: 475 | print(block_group) 476 | if block_group.flags & btrfs.BLOCK_GROUP_TYPE_MASK == btrfs.BLOCK_GROUP_DATA: 477 | # Only DATA, so also not DATA|METADATA (mixed). In this case we 478 | # take a shortcut. Since we know that all extents are data extents, 479 | # which get their usual white color, we don't need to load the 480 | # actual extent objects. 481 | min_key = btrfs.ctree.Key(block_group.vaddr, 0, 0) 482 | max_key = btrfs.ctree.Key(block_group.vaddr + block_group.length, 0, 0) - 1 483 | for header, _ in btrfs.ioctl.search_v2(fs.fd, tree, min_key, max_key, buf_size=65536): 484 | if header.type == btrfs.ctree.EXTENT_ITEM_KEY: 485 | length = header.offset 486 | first_byte = block_group_grid_offset[block_group] + header.objectid 487 | if verbose >= 1: 488 | print("extent vaddr {0} first_byte {1} type {2} length {3}".format( 489 | header.objectid, first_byte, 490 | btrfs.ctree.key_type_str(header.type), length)) 491 | grid.fill(first_byte, length, 1, white) 492 | 493 | else: 494 | # The block group is METADATA or DATA|METADATA or SYSTEM (chunk 495 | # tree metadata). We load all extent info to figure out which 496 | # btree root metadata extents belong to. 497 | min_vaddr = block_group.vaddr 498 | max_vaddr = block_group.vaddr + block_group.length - 1 499 | for extent in fs.extents(min_vaddr, max_vaddr, 500 | load_data_refs=True, load_metadata_refs=True): 501 | if isinstance(extent, btrfs.ctree.ExtentItem): 502 | length = extent.length 503 | if extent.flags & btrfs.ctree.EXTENT_FLAG_DATA: 504 | color = white 505 | elif extent.flags & btrfs.ctree.EXTENT_FLAG_TREE_BLOCK: 506 | color = metadata_extent_colors.get(_get_metadata_root(extent), white) 507 | else: 508 | raise Exception("BUG: expected either DATA or TREE_BLOCK flag, but got " 509 | "{}".format(btrfs.utils.extent_flags_str(extent.flags))) 510 | elif isinstance(extent, btrfs.ctree.MetaDataItem): 511 | length = nodesize 512 | color = metadata_extent_colors.get(_get_metadata_root(extent), white) 513 | first_byte = block_group_grid_offset[block_group] + extent.vaddr 514 | if verbose >= 1: 515 | print("extent vaddr {0} first_byte {1} type {2} length {3}".format( 516 | extent.vaddr, first_byte, 517 | btrfs.ctree.key_type_str(extent.key.type), length)) 518 | grid.fill(first_byte, length, 1, color) 519 | return grid 520 | 521 | 522 | def choose_order_size(order=None, size=None, total_bytes=None, default_granularity=None): 523 | order_was_none = order is None 524 | if order_was_none: 525 | import math 526 | order = min(10, int(math.ceil(math.log(math.sqrt(total_bytes/default_granularity), 2)))) 527 | if size is None: 528 | if order > 10: 529 | size = order 530 | else: 531 | size = 10 532 | if size < order: 533 | if order_was_none: 534 | order = size 535 | else: 536 | raise HeatmapError("size ({}) cannot be smaller than order ({})".format(size, order)) 537 | return order, size 538 | 539 | 540 | def generate_png_file_name(output=None, parts=None): 541 | if output is not None and os.path.isdir(output): 542 | output_dir = output 543 | output_file = None 544 | else: 545 | output_dir = None 546 | output_file = output 547 | if output_file is None: 548 | if parts is None: 549 | parts = [] 550 | else: 551 | parts.append('at') 552 | import time 553 | parts.append(str(int(time.time()))) 554 | output_file = '_'.join([str(part) for part in parts]) + '.png' 555 | if output_dir is None: 556 | return output_file 557 | return os.path.join(output_dir, output_file) 558 | 559 | 560 | class StdoutWriter: 561 | def __init__(self): 562 | self.pos = 0 563 | self.bytelist = [] 564 | 565 | def write(self, data): 566 | self.bytelist[self.pos:self.pos+len(data)] = data 567 | self.pos += len(data) 568 | 569 | def tell(self): 570 | return self.pos 571 | 572 | def seek(self, pos): 573 | self.pos = pos 574 | 575 | def close(self): 576 | sys.stdout.buffer.write(bytes(self.bytelist)) 577 | 578 | 579 | def _write_png(pngfile, width, height, rows, color_type=2): 580 | struct_len = struct_crc = struct.Struct('!I') 581 | if pngfile == '-': 582 | out = StdoutWriter() 583 | else: 584 | out = open(pngfile, 'wb') 585 | out.write(b'\x89PNG\r\n\x1a\n') 586 | # IHDR 587 | out.write(struct_len.pack(13)) 588 | ihdr = struct.Struct('!4s2I5B').pack(b'IHDR', width, height, 8, color_type, 0, 0, 0) 589 | out.write(ihdr) 590 | out.write(struct_crc.pack(zlib.crc32(ihdr) & 0xffffffff)) 591 | # IDAT 592 | length_pos = out.tell() 593 | out.write(b'\x00\x00\x00\x00IDAT') 594 | crc = zlib.crc32(b'IDAT') 595 | datalen = 0 596 | compress = zlib.compressobj() 597 | for row in rows: 598 | for uncompressed in (b'\x00', b''.join(row)): 599 | compressed = compress.compress(uncompressed) 600 | if len(compressed) > 0: 601 | crc = zlib.crc32(compressed, crc) 602 | datalen += len(compressed) 603 | out.write(compressed) 604 | compressed = compress.flush() 605 | if len(compressed) > 0: 606 | crc = zlib.crc32(compressed, crc) 607 | datalen += len(compressed) 608 | out.write(compressed) 609 | out.write(struct_crc.pack(crc & 0xffffffff)) 610 | # IEND 611 | out.write(b'\x00\x00\x00\x00IEND\xae\x42\x60\x82') 612 | # Go back and write length of the IDAT 613 | out.seek(length_pos) 614 | out.write(struct_len.pack(datalen)) 615 | out.close() 616 | 617 | 618 | def main(): 619 | args = parse_args() 620 | path = args.mountpoint 621 | 622 | verbose = 0 623 | if args.verbose is not None: 624 | verbose += args.verbose 625 | if args.quiet is not None: 626 | verbose -= args.quiet 627 | if args.output == '-': 628 | verbose = -1 629 | 630 | try: 631 | with btrfs.FileSystem(path) as fs: 632 | filename_parts = ['fsid', fs.fsid] 633 | if args.curve != 'hilbert': 634 | filename_parts.append(args.curve) 635 | bg_vaddr = args.blockgroup 636 | if bg_vaddr is None: 637 | if args.sort == 'physical': 638 | grid = walk_dev_extents(fs, order=args.order, size=args.size, 639 | verbose=verbose, curve=args.curve) 640 | elif args.sort == 'virtual': 641 | filename_parts.append('chunks') 642 | grid = walk_chunks(fs, order=args.order, size=args.size, 643 | verbose=verbose, curve=args.curve) 644 | else: 645 | raise HeatmapError("Invalid sort option {}".format(args.sort)) 646 | else: 647 | try: 648 | block_group = fs.block_group(bg_vaddr) 649 | except IndexError: 650 | raise HeatmapError("No block group at vaddr {}!".format(bg_vaddr)) 651 | grid = walk_extents(fs, [block_group], order=args.order, size=args.size, 652 | verbose=verbose, curve=args.curve) 653 | filename_parts.extend(['blockgroup', block_group.vaddr]) 654 | except OSError as e: 655 | if e.errno == errno.EPERM: 656 | raise HeatmapError("Insufficient permissions to use the btrfs kernel API. " 657 | "Hint: Try running the script as root user.".format(e)) 658 | elif e.errno == errno.ENOTTY: 659 | raise HeatmapError("Unable to retrieve data. Hint: Not a btrfs file system?") 660 | raise 661 | try: 662 | filename = generate_png_file_name(args.output, filename_parts) 663 | grid.write_png(filename) 664 | except Exception as e: 665 | raise HeatmapError("Unable to write output file {}: {}".format(filename, e)) 666 | 667 | 668 | if __name__ == '__main__': 669 | try: 670 | main() 671 | except HeatmapError as e: 672 | print("Error: {0}".format(e), file=sys.stderr) 673 | sys.exit(1) 674 | -------------------------------------------------------------------------------- /doc/README/animated-balance-small.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/knorrie/btrfs-heatmap/381fad6a0e38d8273fab1f9a7f90eb28007811e9/doc/README/animated-balance-small.gif -------------------------------------------------------------------------------- /doc/README/example-238gib.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/knorrie/btrfs-heatmap/381fad6a0e38d8273fab1f9a7f90eb28007811e9/doc/README/example-238gib.png -------------------------------------------------------------------------------- /doc/README/hilbert_256.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/knorrie/btrfs-heatmap/381fad6a0e38d8273fab1f9a7f90eb28007811e9/doc/README/hilbert_256.png -------------------------------------------------------------------------------- /doc/README/mekker.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/knorrie/btrfs-heatmap/381fad6a0e38d8273fab1f9a7f90eb28007811e9/doc/README/mekker.png -------------------------------------------------------------------------------- /doc/curves.md: -------------------------------------------------------------------------------- 1 | Btrfs Heatmap - Curves 2 | ====================== 3 | 4 | When displaying the layout of data in a filesystem, we have to keep in mind 5 | that, either when following the physical or virtual address space, we're actually 6 | following a linear address space. 7 | 8 | Linear address space | 9 | :----------: | 10 | ![hilbert](curves/line.png) | 11 | 12 | Because it's quite impractical to generate an image that is only one pixel 13 | high, and really really wide, we want to make it a bit more rectangular. 14 | 15 | ### What's this Hilbert thing about? 16 | 17 | The [Hilbert Curve](https://en.wikipedia.org/wiki/Hilbert_curve) is a space filling curve 18 | that has a characteristic of folding the long linear line into a square picture in a 19 | way that tries to keep as much idea of locality of data as possible. 20 | 21 | Hilbert | Snake | Linear | 22 | :------:|:------:|:-------: 23 | ![hilbert](curves/hilbert.png) | ![snake](curves/snake.png) | ![linear](curves/linear.png) 24 | 25 | Here's an example of a filesystem level picture generated with all three. If 26 | you want to have pictures generated with a `linear` or `snake` way of walking 27 | the pixel grid, then you can specify the option `--curve linear` or `--curve 28 | snake` to the btrfs-heatmap program. 29 | 30 | Hilbert | Snake | Linear | 31 | :------:|:------:|:------: 32 | ![hilbert](curves/hilbert-fs.png) | ![linear](curves/snake-fs.png) | ![linear](curves/linear-fs.png) 33 | 34 | I personally like the Hilbert picture better. It's showing blocks of data with 35 | their usage gradient in a much more convenient way. 36 | 37 | Besides this, changing the order of detail of snake or linear pictures also 38 | behaves different than the Hilbert one, When changing the order of the hilbert 39 | curve, to make it more detailed, the picture also looks the same, but with more 40 | detail. For snake and linear, data starts moving around much more. 41 | 42 | Someone else made a very nice visual explanation of this, which you can see in 43 | a video that's available on Youtube, done by 3Blue1Brown. At least watch it 44 | from 5'18" (where it starts, from the external link below), to 7'8". 45 | 46 | [![A video about the Hilbert Curve](curves/ubuishilbert.jpg)](https://www.youtube.com/watch?v=DuiryHHTrjU&t=5m18s) 47 | 48 | ### Hilbert curve 'orders' 49 | 50 | 1st order | 2nd order | 3rd order | 51 | :------:|:------:|:------: 52 | ![hilbert](curves/hilbert-1.png) | ![linear](curves/hilbert-2.png) | ![linear](curves/hilbert.png) 53 | 54 | Well, the easiest way to explain is to show it. When increasing the order of the curve, 55 | you get a curve that walks the pixel grid which we're filling in more detail by 56 | replicating itself in more detail. 57 | 58 | The video linked above also explains the concept of curve orders, in the beginning. 59 | 60 | Hilbert Curve, order 5 | 61 | :------:| 62 | ![hilbert](curves/hilbert-order-05-size-08.gif) | 63 | 64 | While this animation already looks quite complex, it's only a 32 by 32 pixel 65 | area. 66 | 67 | But, the fun thing is, even when looking at an order 10 picture, which has 1024 68 | by 1024 pixels, the basic idea about the curve going up, to the right and down, 69 | and then only getting more detailed in between keeps standing. That's what I 70 | like about it. 71 | -------------------------------------------------------------------------------- /doc/curves/curves.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python3 2 | 3 | # Run this to create pictures to create animated gifs for the first five 4 | # hilbert order curves and linear and snake pictures as a bonus 5 | # 6 | # In the end, I only used one of them in the documentation (hilbert 5) 7 | 8 | import heatmap 9 | import os 10 | 11 | size = 8 12 | white = b'\xff' 13 | 14 | 15 | def dump_grid(filename, grid, scale, width, height): 16 | rows = ((pix for pix in row for _ in range(scale)) 17 | for row in grid for _ in range(scale)) 18 | heatmap._write_png(filename, width * scale, height * scale, rows, color_type=0) 19 | 20 | 21 | for curve_name in ('hilbert', 'snake', 'linear'): 22 | for order in range(1, 6): 23 | width = height = 2 ** order 24 | total_bytes = width * height 25 | grid = [[b'\x00' 26 | for x in range(width)] 27 | for y in range(height)] 28 | 29 | scale = 2 ** (size - order) 30 | curve = heatmap.curves[curve_name](order) 31 | png_dir = 'png/{}/{:0>2}/{:0>2}'.format(curve_name, order, size) 32 | try: 33 | os.makedirs(png_dir) 34 | except: 35 | pass 36 | dump_grid('{}/{:0>6}.png'.format(png_dir, 0), grid, scale, width, height) 37 | delay = max(1, int(100/(order*order))) 38 | print('convert -loop 0 -delay {} -alpha on -coalesce -deconstruct {}/*.png ' 39 | '{}-order-{:0>2}-size-{:0>2}.gif'.format( 40 | delay, png_dir, curve_name, order, size)) 41 | for frame in range(1, total_bytes+1): 42 | y, x, _ = next(curve) 43 | grid[y][x] = white 44 | dump_grid('{}/{:0>6}.png'.format(png_dir, frame), grid, scale, width, height) 45 | -------------------------------------------------------------------------------- /doc/curves/hilbert-1.dia: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/knorrie/btrfs-heatmap/381fad6a0e38d8273fab1f9a7f90eb28007811e9/doc/curves/hilbert-1.dia -------------------------------------------------------------------------------- /doc/curves/hilbert-1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/knorrie/btrfs-heatmap/381fad6a0e38d8273fab1f9a7f90eb28007811e9/doc/curves/hilbert-1.png -------------------------------------------------------------------------------- /doc/curves/hilbert-2.dia: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/knorrie/btrfs-heatmap/381fad6a0e38d8273fab1f9a7f90eb28007811e9/doc/curves/hilbert-2.dia -------------------------------------------------------------------------------- /doc/curves/hilbert-2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/knorrie/btrfs-heatmap/381fad6a0e38d8273fab1f9a7f90eb28007811e9/doc/curves/hilbert-2.png -------------------------------------------------------------------------------- /doc/curves/hilbert-fs.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/knorrie/btrfs-heatmap/381fad6a0e38d8273fab1f9a7f90eb28007811e9/doc/curves/hilbert-fs.png -------------------------------------------------------------------------------- /doc/curves/hilbert-order-05-size-08.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/knorrie/btrfs-heatmap/381fad6a0e38d8273fab1f9a7f90eb28007811e9/doc/curves/hilbert-order-05-size-08.gif -------------------------------------------------------------------------------- /doc/curves/hilbert.dia: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/knorrie/btrfs-heatmap/381fad6a0e38d8273fab1f9a7f90eb28007811e9/doc/curves/hilbert.dia -------------------------------------------------------------------------------- /doc/curves/hilbert.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/knorrie/btrfs-heatmap/381fad6a0e38d8273fab1f9a7f90eb28007811e9/doc/curves/hilbert.png -------------------------------------------------------------------------------- /doc/curves/line.dia: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/knorrie/btrfs-heatmap/381fad6a0e38d8273fab1f9a7f90eb28007811e9/doc/curves/line.dia -------------------------------------------------------------------------------- /doc/curves/line.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/knorrie/btrfs-heatmap/381fad6a0e38d8273fab1f9a7f90eb28007811e9/doc/curves/line.png -------------------------------------------------------------------------------- /doc/curves/linear-fs.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/knorrie/btrfs-heatmap/381fad6a0e38d8273fab1f9a7f90eb28007811e9/doc/curves/linear-fs.png -------------------------------------------------------------------------------- /doc/curves/linear.dia: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/knorrie/btrfs-heatmap/381fad6a0e38d8273fab1f9a7f90eb28007811e9/doc/curves/linear.dia -------------------------------------------------------------------------------- /doc/curves/linear.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/knorrie/btrfs-heatmap/381fad6a0e38d8273fab1f9a7f90eb28007811e9/doc/curves/linear.png -------------------------------------------------------------------------------- /doc/curves/snake-fs.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/knorrie/btrfs-heatmap/381fad6a0e38d8273fab1f9a7f90eb28007811e9/doc/curves/snake-fs.png -------------------------------------------------------------------------------- /doc/curves/snake.dia: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/knorrie/btrfs-heatmap/381fad6a0e38d8273fab1f9a7f90eb28007811e9/doc/curves/snake.dia -------------------------------------------------------------------------------- /doc/curves/snake.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/knorrie/btrfs-heatmap/381fad6a0e38d8273fab1f9a7f90eb28007811e9/doc/curves/snake.png -------------------------------------------------------------------------------- /doc/curves/ubuishilbert.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/knorrie/btrfs-heatmap/381fad6a0e38d8273fab1f9a7f90eb28007811e9/doc/curves/ubuishilbert.jpg -------------------------------------------------------------------------------- /doc/extent.md: -------------------------------------------------------------------------------- 1 | Btrfs Heatmap - Extent level pictures 2 | ===================================== 3 | 4 | The filesystem level only displays a certain greyscale value for entire block 5 | groups. By specifying a block group address, we can also get a view on the 6 | distribution of data and metadata inside a single block group. This shows for 7 | example how fragmented the free space inside the block group is. 8 | 9 | 1GiB DATA block group | 512MiB DUP METADATA block group 10 | :-------------------------:|:-------:| 11 | ![data](extent/example-data.png) | ![metadata](extent/example-metadata.png) 12 | 13 | Metadata tree blocks get colored depending on the tree they belong to: 14 | 15 | Color | Tree | Color | Tree | 16 | :-----:|:----:|:-----:|:----:| 17 | ![ROOT](extent/ROOT_TREE.png) | ROOT (1) | ![CSUM](extent/CSUM_TREE.png) | CSUM (7) | 18 | ![EXTENT](extent/EXTENT_TREE.png) | EXTENT (2) | ![QUOTA](extent/QUOTA_TREE.png) | QUOTA (8) | 19 | ![CHUNK](extent/CHUNK_TREE.png) | CHUNK (3) | ![UUID](extent/UUID_TREE.png) | UUID (9) | 20 | ![DEV](extent/DEV_TREE.png) | DEV (4) | ![FREE SPACE](extent/FREE_SPACE_TREE.png) | FREE SPACE (10) | 21 | ![FS](extent/FS_TREE.png) | FS (5, 256+) | ![DATA RELOC](extent/DATA_RELOC_TREE.png) | DATA RELOC (-9) | 22 | 23 | The `btrfs-heatmap` program can take a `--blockgroup` argument, which needs a 24 | vaddr of a block group as argument. 25 | 26 | In order to list all block groups, we can use the `btrfs-search-metadata` 27 | program that is included with python-btrfs since v12. Example part of the 28 | output: 29 | 30 | ``` 31 | -# btrfs-search-metadata block_groups / 32 | [...] 33 | block group vaddr 722187845632 transid 1871184 length 536870912 flags METADATA|DUP used 409714688 used_pct 76 34 | block group vaddr 783391129600 transid 1851697 length 1073741824 flags DATA used 573911040 used_pct 53 35 | [...] 36 | ``` 37 | 38 | Then I created the images using the following commands: 39 | 40 | ``` 41 | -# btrfs-heatmap --blockgroup 722187845632 --size 8 / 42 | max_id 1 num_devices 1 fsid 64ac42f5-4ff7-4be0-b94a-90def45e6c1e nodesize 16384 sectorsize 4096 clone_alignment 4096 43 | scope block_group 722187845632 44 | grid order 8 size 8 height 256 width 256 total_bytes 536870912 bytes_per_pixel 8192.0 45 | pngfile fsid_64ac42f5-4ff7-4be0-b94a-90def45e6c1e_blockgroup_722187845632_at_1484322493.png 46 | 47 | -# btrfs-heatmap --blockgroup 783391129600 --size 8 / 48 | max_id 1 num_devices 1 fsid 64ac42f5-4ff7-4be0-b94a-90def45e6c1e nodesize 16384 sectorsize 4096 clone_alignment 4096 49 | scope block_group 783391129600 50 | grid order 8 size 8 height 256 width 256 total_bytes 1073741824 bytes_per_pixel 16384.0 51 | pngfile fsid_64ac42f5-4ff7-4be0-b94a-90def45e6c1e_blockgroup_783391129600_at_1484322668.png 52 | ``` 53 | 54 | Also note: 55 | * These are pictures from the virtual address space. The DUP in the header of 56 | the metdata picture doesn't mean much. 57 | * By default, size of pictures is 10 (2^10=1024 width/height). I used 8 to make 58 | them smaller for the documentation page. 59 | 60 | Next: [Scripting btrfs-heatmap](scripting.md) 61 | -------------------------------------------------------------------------------- /doc/extent/CHUNK_TREE.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/knorrie/btrfs-heatmap/381fad6a0e38d8273fab1f9a7f90eb28007811e9/doc/extent/CHUNK_TREE.png -------------------------------------------------------------------------------- /doc/extent/CSUM_TREE.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/knorrie/btrfs-heatmap/381fad6a0e38d8273fab1f9a7f90eb28007811e9/doc/extent/CSUM_TREE.png -------------------------------------------------------------------------------- /doc/extent/DATA_RELOC_TREE.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/knorrie/btrfs-heatmap/381fad6a0e38d8273fab1f9a7f90eb28007811e9/doc/extent/DATA_RELOC_TREE.png -------------------------------------------------------------------------------- /doc/extent/DEV_TREE.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/knorrie/btrfs-heatmap/381fad6a0e38d8273fab1f9a7f90eb28007811e9/doc/extent/DEV_TREE.png -------------------------------------------------------------------------------- /doc/extent/EXTENT_TREE.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/knorrie/btrfs-heatmap/381fad6a0e38d8273fab1f9a7f90eb28007811e9/doc/extent/EXTENT_TREE.png -------------------------------------------------------------------------------- /doc/extent/FREE_SPACE_TREE.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/knorrie/btrfs-heatmap/381fad6a0e38d8273fab1f9a7f90eb28007811e9/doc/extent/FREE_SPACE_TREE.png -------------------------------------------------------------------------------- /doc/extent/FS_TREE.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/knorrie/btrfs-heatmap/381fad6a0e38d8273fab1f9a7f90eb28007811e9/doc/extent/FS_TREE.png -------------------------------------------------------------------------------- /doc/extent/QUOTA_TREE.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/knorrie/btrfs-heatmap/381fad6a0e38d8273fab1f9a7f90eb28007811e9/doc/extent/QUOTA_TREE.png -------------------------------------------------------------------------------- /doc/extent/ROOT_TREE.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/knorrie/btrfs-heatmap/381fad6a0e38d8273fab1f9a7f90eb28007811e9/doc/extent/ROOT_TREE.png -------------------------------------------------------------------------------- /doc/extent/UUID_TREE.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/knorrie/btrfs-heatmap/381fad6a0e38d8273fab1f9a7f90eb28007811e9/doc/extent/UUID_TREE.png -------------------------------------------------------------------------------- /doc/extent/example-data.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/knorrie/btrfs-heatmap/381fad6a0e38d8273fab1f9a7f90eb28007811e9/doc/extent/example-data.png -------------------------------------------------------------------------------- /doc/extent/example-metadata.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/knorrie/btrfs-heatmap/381fad6a0e38d8273fab1f9a7f90eb28007811e9/doc/extent/example-metadata.png -------------------------------------------------------------------------------- /doc/extent/update_color_map.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python3 2 | 3 | import heatmap 4 | import btrfs 5 | 6 | height = 25 7 | width = 100 8 | for color in heatmap.metadata_extent_colors: 9 | pngfile = "doc/%s.png" % btrfs.ctree._key_objectid_str_map[color] 10 | print(pngfile) 11 | bcolor = heatmap.struct_color.pack(*heatmap.metadata_extent_colors[color]) 12 | rows = ((bcolor for _ in range(width)) 13 | for _ in range(height)) 14 | heatmap._write_png(pngfile, width, height, rows) 15 | -------------------------------------------------------------------------------- /doc/scripting.md: -------------------------------------------------------------------------------- 1 | Btrfs Heatmap - Scripting 2 | ========================= 3 | 4 | When doing something more sophisticated than just creating a picture of a 5 | complete filesystem or a single block group, it's better to do it from python 6 | and use the btrfs library to find the objects that we want to display. 7 | 8 | The btrfs-heatmap code is not a full-blown library, but that doesn't prevent us 9 | from importing the code from another script in the same directory and then 10 | using functions from it. In order to be able to do this, I recommend just 11 | making a symlink to the btrfs-heatmap program in the location where you're 12 | writing the new script. This is a bit clumsy, but it's what it is, for now. 13 | 14 | ``` 15 | -$ ln -s $(which btrfs-heatmap) heatmap.py 16 | ``` 17 | 18 | After doing this, we can simply `import heatmap` in our new script. 19 | 20 | First, we'll have a look at some interesting fuctions inside the code, 21 | after which I'll show some examples of how to use them. 22 | 23 | ## 1. Internal btrfs-heatmap functions 24 | 25 | ### 1.1 Working with devices, dev extent level picture 26 | 27 | ```python 28 | walk_dev_extents(fs, devices=None, order=None, size=None, 29 | default_granularity=33554432, verbose=0, 30 | min_brightness=None, curve=None) 31 | ``` 32 | 33 | * `fs` is a btrfs.FileSystem object. 34 | * `devices` is a list of one or more device objects (`btrfs.ctree.DevItem`), 35 | or `None` to automatically use all of them. 36 | * `order` defines the hilbert curve order. Most of the time it's best to let 37 | it be determined automatically. A higher number will result in a more 38 | detailed picture, with less bytes per pixel. 39 | * `size` defines the size of the output image. By default, this is 10, which 40 | means a picture with a height and width of 1024 (2^10). If the size argument 41 | is higher than the curve order, pixels are simply duplicated when writing 42 | out the png picture. 43 | * `default_granularity` defines the amount of bytes that should be mapped 44 | (approximately) on a single pixel in the output to determine the hilbert 45 | curve order to use. By default, this is 32MiB. 46 | * A higher number for `verbose` makes the output more verbose (like `-vvv` on 47 | the command line of `btrfs-heatmap` would be `verbose=3`). 48 | * `min_brightness` (0 <= `min_brightness` <= 1, default 0.1) sets the minimal 49 | brightness of pixels that are part of allocated space, to be able to 50 | distinguish them from unallocated space when usage is really low. 51 | * `curve` is either 'hilbert' (the default), 'snake', or 'linear' 52 | 53 | ### 1.2 The virtual address space, chunk level picture 54 | 55 | ```python 56 | walk_chunks(fs, devices=None, order=None, size=None, default_granularity=33554432, 57 | verbose=0, min_brightness=None, curve=None) 58 | ``` 59 | 60 | * for all options, see above 61 | 62 | ### 1.3 Working with block groups, extent level picture 63 | 64 | ```python 65 | walk_extents(fs, block_groups, order=None, size=None, 66 | default_granularity=None, verbose=0, curve=None) 67 | ``` 68 | 69 | * `block_groups` is a list of one or multiple block group objects. 70 | * For block group internals, `default_granularity` defaults to the sector size 71 | of the filesystem, which is often 4096 bytes. 72 | * for other options, see above 73 | 74 | ### 1.4 A helper for generating file names 75 | 76 | ```python 77 | generate_png_file_name(output=None, parts=None) 78 | ``` 79 | 80 | * `parts` is a list of filename parts that will be concatenated, after which a 81 | timestamp is also added. e.g. `parts=['foo', 'bar']` results in 82 | `foo_bar_at_1482095269.png` 83 | * `output` can be a filename, in which case the function just returns that 84 | filename again 85 | * `output` can be a directory, in which case the function will return a path 86 | to an autogenerated filename using parts in that directory 87 | 88 | ## 2. Examples 89 | 90 | ### 2.1 Full filesystem image 91 | 92 | The following is the equivalent of doing `btrfs-heatmap --size 8 -o full-fs.png 93 | /`: 94 | 95 | ```python 96 | #!/usr/bin/python3 97 | import btrfs 98 | import heatmap 99 | fs = btrfs.FileSystem('/') 100 | heatmap.walk_dev_extents(fs, size=8).write_png('full-fs.png') 101 | ``` 102 | 103 | output: 104 | ``` 105 | scope device 1 106 | grid order 5 size 8 height 32 width 32 total_bytes 21474836480 bytes_per_pixel 20971520.0 107 | pngfile full-fs.png 108 | ``` 109 | 110 | 20 GiB file system | 111 | :-------------------------:| 112 | ![Full FS](scripting/full-fs.png) | 113 | 114 | ### 2.2 The four newest DATA block groups together 115 | 116 | ```python 117 | #!/usr/bin/python3 118 | import btrfs 119 | import heatmap 120 | fs = btrfs.FileSystem('/') 121 | four_newest_bg = [fs.block_group(chunk.vaddr, chunk.length) 122 | for chunk in fs.chunks() 123 | if chunk.type & btrfs.BLOCK_GROUP_DATA][-4:] 124 | grid = heatmap.walk_extents(fs, four_newest_bg) 125 | parts = ['fsid', fs.fsid, 'startat', four_newest_bg[0].vaddr] 126 | png_filename = heatmap.generate_png_file_name('/output/directory/', parts=parts) 127 | grid.write_png(png_filename) 128 | ``` 129 | 130 | output: 131 | ``` 132 | scope block_group 154696417280 155770159104 156843900928 157917642752 133 | grid order 8 size 8 height 256 width 256 total_bytes 4294967296 bytes_per_pixel 65536.0 134 | pngfile fsid_9881fc30-8f69-4069-a8c8-c057b842b0c4_startat_154696417280_at_1484406586.png 135 | ``` 136 | 137 | This example uses the png file name generator helper which adds a timestamp so 138 | we can easily repeat it to get images which can be put together into a timelapse. 139 | 140 | 4 newest DATA block groups | 141 | :-------------------------:| 142 | ![Four block groups](scripting/4-highest.png) | 143 | 144 | ### 2.3 Show usage, separate image per device, more verbose output 145 | 146 | The following script generates a separate picture per physical device. Since this is 147 | a very small filesystem, we use mode linear to make it look a bit more like norton 148 | disk defragmenter. 149 | 150 | ```python 151 | #!/usr/bin/python3 152 | import btrfs 153 | import heatmap 154 | fs = btrfs.FileSystem('/mnt/raid0') 155 | for device in fs.devices(): 156 | grid = heatmap.walk_dev_extents(fs, [device], curve='linear', size=8, verbose=1) 157 | grid.write_png('device_%s.png' % device.devid) 158 | ``` 159 | 160 | output: 161 | ``` 162 | scope device 1 163 | grid curve hilbert order 4 size 8 height 16 width 16 total_bytes 5368709120 bytes_per_pixel 20971520.0 164 | dev_extent devid 1 paddr 20971520 length 8388608 pend 29360127 type SYSTEM|RAID1 used_pct 0.20 165 | dev_extent devid 1 paddr 29360128 length 1073741824 pend 1103101951 type METADATA|RAID1 used_pct 0.50 166 | dev_extent devid 1 paddr 1103101952 length 1073741824 pend 2176843775 type DATA used_pct 96.10 167 | dev_extent devid 1 paddr 3250585600 length 1073741824 pend 4324327423 type DATA used_pct 87.01 168 | dev_extent devid 1 paddr 4324327424 length 1044381696 pend 5368709119 type DATA used_pct 76.09 169 | pngfile device_1.png 170 | 171 | scope device 2 172 | grid curve hilbert order 4 size 8 height 16 width 16 total_bytes 5368709120 bytes_per_pixel 20971520.0 173 | dev_extent devid 2 paddr 1048576 length 8388608 pend 9437183 type SYSTEM|RAID1 used_pct 0.20 174 | dev_extent devid 2 paddr 9437184 length 1073741824 pend 1083179007 type METADATA|RAID1 used_pct 0.50 175 | dev_extent devid 2 paddr 1083179008 length 536870912 pend 1620049919 type DATA used_pct 50.00 176 | dev_extent devid 2 paddr 1620049920 length 1073741824 pend 2693791743 type DATA used_pct 96.03 177 | dev_extent devid 2 paddr 2693791744 length 1073741824 pend 3767533567 type DATA used_pct 50.00 178 | dev_extent devid 2 paddr 3767533568 length 1073741824 pend 4841275391 type DATA used_pct 25.00 179 | dev_extent devid 2 paddr 4841275392 length 527433728 pend 5368709119 type DATA used_pct 18.58 180 | pngfile device_2.png 181 | ``` 182 | 183 | Dev Extents on device 1 | Dev Extents on device 2 184 | :------------------:|:-------------------: 185 | |![Device 1](scripting/device_1.png) | ![Device 2](scripting/device_2.png) 186 | 187 | ### 2.4 Show virtual address space, separate image per device 188 | 189 | This one is very similar to the previous example, but it shows the amount of used space 190 | on each device that is attached to the filesystem sorted on virtual address space. 191 | 192 | In addition, it lowers the resolution a bit from what would be chosen 193 | automatically (4 instead of 5), chooses a snake curve instead of hilbert and 194 | bumps the minimal brightness a bit, since the metadata part is almost 195 | completely empty. 196 | 197 | ```python 198 | #!/usr/bin/python3 199 | import btrfs 200 | import heatmap 201 | fs = btrfs.FileSystem('/mnt/btrfs') 202 | for device in fs.devices(): 203 | grid = heatmap.walk_chunks(fs, [device], order=4, size=8, curve='snake', 204 | verbose=2, min_brightness=0.3) 205 | grid.write_png('stripes_on_device_%s.png' % device.devid) 206 | ``` 207 | 208 | output: 209 | ``` 210 | scope chunk stripes on devices 1 211 | grid curve snake order 4 size 8 height 16 width 16 total_bytes 5368709120 bytes_per_pixel 20971520.0 212 | block group vaddr 20971520 transid 23 length 8388608 flags SYSTEM|RAID1 used 16384 used_pct 0 213 | chunk vaddr 20971520 type SYSTEM|RAID1 length 8388608 num_stripes 2 214 | stripe devid 1 offset 20971520 215 | in_pixel 0 40.00% 216 | block group vaddr 29360128 transid 23 length 1073741824 flags METADATA|RAID1 used 7684096 used_pct 1 217 | chunk vaddr 29360128 type METADATA|RAID1 length 1073741824 num_stripes 2 218 | stripe devid 1 offset 29360128 219 | first_pixel 0 60.00% last_pixel 51 60.00% 220 | block group vaddr 3250585600 transid 23 length 1073741824 flags DATA used 1031831552 used_pct 96 221 | chunk vaddr 3250585600 type DATA length 1073741824 num_stripes 1 222 | stripe devid 1 offset 1103101952 223 | first_pixel 51 40.00% last_pixel 102 80.00% 224 | block group vaddr 5398069248 transid 23 length 1073741824 flags DATA used 1071894528 used_pct 100 225 | chunk vaddr 5398069248 type DATA length 1073741824 num_stripes 1 226 | stripe devid 1 offset 2176843776 227 | first_pixel 102 20.00% last_pixel 153 100.00% 228 | block group vaddr 7545552896 transid 23 length 1073741824 flags DATA used 934223872 used_pct 87 229 | chunk vaddr 7545552896 type DATA length 1073741824 num_stripes 1 230 | stripe devid 1 offset 3250585600 231 | first_pixel 154 100.00% last_pixel 205 20.00% 232 | block group vaddr 8619294720 transid 23 length 1044381696 flags DATA used 794705920 used_pct 76 233 | chunk vaddr 8619294720 type DATA length 1044381696 num_stripes 1 234 | stripe devid 1 offset 4324327424 235 | first_pixel 205 80.00% last_pixel 254 100.00% 236 | pngfile stripes_on_device_1.png 237 | 238 | scope chunk stripes on devices 2 239 | grid curve snake order 4 size 8 height 16 width 16 total_bytes 5368709120 bytes_per_pixel 20971520.0 240 | block group vaddr 20971520 transid 23 length 8388608 flags SYSTEM|RAID1 used 16384 used_pct 0 241 | chunk vaddr 20971520 type SYSTEM|RAID1 length 8388608 num_stripes 2 242 | stripe devid 2 offset 1048576 243 | in_pixel 0 40.00% 244 | block group vaddr 29360128 transid 23 length 1073741824 flags METADATA|RAID1 used 7684096 used_pct 1 245 | chunk vaddr 29360128 type METADATA|RAID1 length 1073741824 num_stripes 2 246 | stripe devid 2 offset 9437184 247 | first_pixel 0 60.00% last_pixel 51 60.00% 248 | block group vaddr 2176843776 transid 23 length 1073741824 flags DATA used 1030082560 used_pct 96 249 | chunk vaddr 2176843776 type DATA length 1073741824 num_stripes 1 250 | stripe devid 2 offset 1620049920 251 | first_pixel 51 40.00% last_pixel 102 80.00% 252 | block group vaddr 4324327424 transid 23 length 1073741824 flags DATA used 1040887808 used_pct 97 253 | chunk vaddr 4324327424 type DATA length 1073741824 num_stripes 1 254 | stripe devid 2 offset 2693791744 255 | first_pixel 102 20.00% last_pixel 153 100.00% 256 | block group vaddr 6471811072 transid 23 length 1073741824 flags DATA used 871952384 used_pct 81 257 | chunk vaddr 6471811072 type DATA length 1073741824 num_stripes 1 258 | stripe devid 2 offset 3767533568 259 | first_pixel 154 100.00% last_pixel 205 20.00% 260 | block group vaddr 9663676416 transid 23 length 536870912 flags DATA used 268435456 used_pct 50 261 | chunk vaddr 9663676416 type DATA length 536870912 num_stripes 1 262 | stripe devid 2 offset 1083179008 263 | first_pixel 205 80.00% last_pixel 230 80.00% 264 | block group vaddr 10200547328 transid 23 length 527433728 flags DATA used 98013184 used_pct 19 265 | chunk vaddr 10200547328 type DATA length 527433728 num_stripes 1 266 | stripe devid 2 offset 4841275392 267 | first_pixel 230 20.00% last_pixel 255 95.00% 268 | pngfile stripes_on_device_2.png 269 | ``` 270 | 271 | Stripes on device 1 | Stripes on device 2 272 | :------------------:|:-------------------: 273 | |![Device 1](scripting/stripes_on_device_1.png) | ![Device 2](scripting/stripes_on_device_2.png) 274 | 275 | ### 2.5 Detailed picture of a full filesystem 276 | 277 | ```python 278 | #!/usr/bin/python3 279 | 280 | import btrfs 281 | import heatmap 282 | 283 | fs = btrfs.FileSystem('/') 284 | bgs = [fs.block_group(chunk.vaddr, chunk.length) 285 | for chunk in fs.chunks()] 286 | grid = heatmap.walk_extents(fs, bgs, size=9) 287 | parts = ['fsid', fs.fsid, 'all_bg'] 288 | png_filename = heatmap.generate_png_file_name(parts=parts) 289 | grid.write_png(png_filename) 290 | ``` 291 | 292 | output: 293 | ``` 294 | scope block_group 87285563392 87319117824 89466601472 90540343296 90808778752 91077214208 295 | 91345649664 91614085120 91882520576 92150956032 122215727104 123289468928 130000355328 296 | 154696417280 155770159104 156843900928 157917642752 297 | grid order 9 size 9 height 512 width 512 total_bytes 11576279040 bytes_per_pixel 44160.0 298 | pngfile fsid_9881fc30-8f69-4069-a8c8-c057b842b0c4_all_bg_at_1484409171.png 299 | ``` 300 | 301 | Note that the following picture, while being taken of the same filesystem as the 'full fs' 302 | picture in the first example, has a very different ordering. The block groups in the virtual 303 | address space do not have to map to the same order as allocated chunks of disk. 304 | 305 | And, because the picture shows all block groups, it does not show any 306 | unallocated raw disk space. 307 | 308 | I used size 9 in this example, to get a picture that would fit into this page. 309 | When specifying a higher hilbert curve order (like 11, or maybe even 12), a 310 | very detailed, but very big picture can be made. 311 | 312 | All block groups | 313 | :-------------------------:| 314 | ![All block groups](scripting/all-bg.png) | 315 | -------------------------------------------------------------------------------- /doc/scripting/4-highest.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/knorrie/btrfs-heatmap/381fad6a0e38d8273fab1f9a7f90eb28007811e9/doc/scripting/4-highest.png -------------------------------------------------------------------------------- /doc/scripting/all-bg.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/knorrie/btrfs-heatmap/381fad6a0e38d8273fab1f9a7f90eb28007811e9/doc/scripting/all-bg.png -------------------------------------------------------------------------------- /doc/scripting/device_1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/knorrie/btrfs-heatmap/381fad6a0e38d8273fab1f9a7f90eb28007811e9/doc/scripting/device_1.png -------------------------------------------------------------------------------- /doc/scripting/device_2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/knorrie/btrfs-heatmap/381fad6a0e38d8273fab1f9a7f90eb28007811e9/doc/scripting/device_2.png -------------------------------------------------------------------------------- /doc/scripting/full-fs.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/knorrie/btrfs-heatmap/381fad6a0e38d8273fab1f9a7f90eb28007811e9/doc/scripting/full-fs.png -------------------------------------------------------------------------------- /doc/scripting/stripes_on_device_1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/knorrie/btrfs-heatmap/381fad6a0e38d8273fab1f9a7f90eb28007811e9/doc/scripting/stripes_on_device_1.png -------------------------------------------------------------------------------- /doc/scripting/stripes_on_device_2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/knorrie/btrfs-heatmap/381fad6a0e38d8273fab1f9a7f90eb28007811e9/doc/scripting/stripes_on_device_2.png -------------------------------------------------------------------------------- /doc/sort.md: -------------------------------------------------------------------------------- 1 | Btrfs Heatmap - Physical and virtual address space 2 | ================================================== 3 | 4 | By default, the heatmap script produces a picture that shows allocated and used 5 | space sorted in the order in which the allocated chunks are present on actual 6 | underlying disk storage. 7 | 8 | By specifying the option `--sort virtual`, we can sort the same information 9 | based on the internal virtual addressing instead. 10 | 11 | When quickly wanting to find out how filled up a filesystem is, this can be 12 | quite useful, since unallocated space is not scattered around the whole 13 | picture, but pushed to the back. 14 | 15 | The longest living block groups are sorted near the beginning, which gives an 16 | idea about how much fragmented free space that is not being reused by btrfs is 17 | left behind over time. 18 | 19 | Physical | Virtual 20 | :---------------:|:----: 21 | ![Physical](sort/physical-dev-extents.png) | ![Virtual](sort/virtual-chunks.png) 22 | 23 | ## Virtual address space 24 | 25 | The virtual address space of a btrfs filesystem is a space which gets extended 26 | every time a chunk of raw disk space gets allocated to be used for data or 27 | metadata. While unallocted physical space gets reused all the time, virtual 28 | address space is never reused and will only grow further into higher numbers. 29 | 30 | When btrfs wants to allocate free physical space into new virtual address 31 | space, it will always choose the first free part of physical unallocated space 32 | on the attached block device that has the most unallocated raw space of all of 33 | them. 34 | 35 | When looking closely at the 'btrfs balance' animated picture on the frontpage 36 | of this documentation, you can actually see this happen. 37 | 38 | ## Mappings between physical and virtual space 39 | 40 | A simple way to determine what the physical and virtual address space of a 41 | filesystem looks like, is to dump the information of either `dev_extents` or 42 | `chunks`. 43 | 44 | ### dev\_extent 45 | 46 | The `dev_extent` objects live in tree number 4, the device tree. For each piece 47 | of raw disk space that is in use, on each device, there's an object in the tree 48 | that points to a device, shows an address and length and lists the associated 49 | chunk in the virtual address space this piece of physical disk space belongs 50 | to. 51 | 52 | The following small filesystem (which is not the filesystem the pictures above 53 | have been taken from) only has one block device attached, and you can see that 54 | the physical addresses and the length of the allocated space maps to chunks 55 | that vary wildy all around the virtual address space. 56 | 57 | ``` 58 | -# python3 59 | >>> import btrfs 60 | >>> fs = btrfs.FileSystem('/') 61 | >>> for dev_extent in fs.dev_extents(): 62 | ... print(dev_extent) 63 | ... 64 | dev extent devid 1 paddr 1048576 length 268435456 chunk 21663580160 65 | dev extent devid 1 paddr 269484032 length 268435456 chunk 21932015616 66 | dev extent devid 1 paddr 537919488 length 268435456 chunk 22200451072 67 | dev extent devid 1 paddr 806354944 length 1073741824 chunk 86982524928 68 | dev extent devid 1 paddr 1880096768 length 268435456 chunk 36427530240 69 | dev extent devid 1 paddr 2148532224 length 268435456 chunk 84298170368 70 | dev extent devid 1 paddr 2416967680 length 268435456 chunk 85640347648 71 | dev extent devid 1 paddr 2896691200 length 1073741824 chunk 32132562944 72 | dev extent devid 1 paddr 3970433024 length 1073741824 chunk 89130008576 73 | dev extent devid 1 paddr 5256511488 length 33554432 chunk 21630025728 74 | dev extent devid 1 paddr 5290065920 length 1073741824 chunk 85908783104 75 | dev extent devid 1 paddr 6363807744 length 1073741824 chunk 90203750400 76 | dev extent devid 1 paddr 7549222912 length 1073741824 chunk 22468886528 77 | dev extent devid 1 paddr 8622964736 length 1073741824 chunk 91277492224 78 | dev extent devid 1 paddr 9696706560 length 1073741824 chunk 88056266752 79 | dev extent devid 1 paddr 13275496448 length 1073741824 chunk 24616370176 80 | dev extent devid 1 paddr 14349238272 length 1073741824 chunk 25690112000 81 | dev extent devid 1 paddr 17570463744 length 1073741824 chunk 75708235776 82 | ``` 83 | 84 | ### chunk 85 | 86 | The `chunk` tree, tree number 3, provides a mapping back from virtual address 87 | space back to the physical address space. This is the tree that gets stored 88 | into the mysterious SYSTEM part of the filesystem space. For each `chunk` of 89 | virtual address space, there can be one or more stripes which point back to 90 | the device numbers and physical addressing of each `dev_extent`: 91 | 92 | ``` 93 | -# python3 94 | >>> import btrfs 95 | >>> fs = btrfs.FileSystem('/') 96 | >>> for chunk in fs.chunks(): 97 | ... print(chunk) 98 | ... for stripe in chunk.stripes: 99 | ... print(" %s" % stripe) 100 | ... 101 | chunk vaddr 21630025728 type SYSTEM length 33554432 num_stripes 1 102 | stripe devid 1 offset 5256511488 103 | chunk vaddr 21663580160 type METADATA length 268435456 num_stripes 1 104 | stripe devid 1 offset 1048576 105 | chunk vaddr 21932015616 type METADATA length 268435456 num_stripes 1 106 | stripe devid 1 offset 269484032 107 | chunk vaddr 22200451072 type METADATA length 268435456 num_stripes 1 108 | stripe devid 1 offset 537919488 109 | chunk vaddr 22468886528 type DATA length 1073741824 num_stripes 1 110 | stripe devid 1 offset 7549222912 111 | chunk vaddr 24616370176 type DATA length 1073741824 num_stripes 1 112 | stripe devid 1 offset 13275496448 113 | chunk vaddr 25690112000 type DATA length 1073741824 num_stripes 1 114 | stripe devid 1 offset 14349238272 115 | chunk vaddr 32132562944 type DATA length 1073741824 num_stripes 1 116 | stripe devid 1 offset 2896691200 117 | chunk vaddr 36427530240 type METADATA length 268435456 num_stripes 1 118 | stripe devid 1 offset 1880096768 119 | chunk vaddr 75708235776 type DATA length 1073741824 num_stripes 1 120 | stripe devid 1 offset 17570463744 121 | chunk vaddr 84298170368 type METADATA length 268435456 num_stripes 1 122 | stripe devid 1 offset 2148532224 123 | chunk vaddr 85640347648 type METADATA length 268435456 num_stripes 1 124 | stripe devid 1 offset 2416967680 125 | chunk vaddr 85908783104 type DATA length 1073741824 num_stripes 1 126 | stripe devid 1 offset 5290065920 127 | chunk vaddr 86982524928 type DATA length 1073741824 num_stripes 1 128 | stripe devid 1 offset 806354944 129 | chunk vaddr 88056266752 type DATA length 1073741824 num_stripes 1 130 | stripe devid 1 offset 9696706560 131 | chunk vaddr 89130008576 type DATA length 1073741824 num_stripes 1 132 | stripe devid 1 offset 3970433024 133 | chunk vaddr 90203750400 type DATA length 1073741824 num_stripes 1 134 | stripe devid 1 offset 6363807744 135 | chunk vaddr 91277492224 type DATA length 1073741824 num_stripes 1 136 | stripe devid 1 offset 8622964736 137 | ``` 138 | 139 | While this is a very boring filesystem, your one might show multiple stripes 140 | when for example using DUP for METADATA. 141 | -------------------------------------------------------------------------------- /doc/sort/physical-dev-extents.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/knorrie/btrfs-heatmap/381fad6a0e38d8273fab1f9a7f90eb28007811e9/doc/sort/physical-dev-extents.png -------------------------------------------------------------------------------- /doc/sort/virtual-chunks.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/knorrie/btrfs-heatmap/381fad6a0e38d8273fab1f9a7f90eb28007811e9/doc/sort/virtual-chunks.png -------------------------------------------------------------------------------- /man/btrfs-heatmap.1: -------------------------------------------------------------------------------- 1 | .TH BTRFS\-HEATMAP 1 " 2017" "" "Btrfs Heatmap" 2 | .nh 3 | .ad l 4 | 5 | .SH "NAME" 6 | btrfs\-heatmap \- visualize the layout of data on your btrfs filesystem 7 | 8 | .SH SYNOPSIS 9 | .B btrfs\-heatmap 10 | [\fIargs\fR] 11 | .IR mountpoint 12 | 13 | .SH DESCRIPTION 14 | The \fBbtrfs\-heatmap\fR script creates a visualization of how a btrfs 15 | filesystem is using the underlying disk space of the block devices that are 16 | added to it. 17 | 18 | The resulting PNG image will show unallocated disk space as black pixels. Raw 19 | disk space that is allocated to be used for data (white), metadata (blue) or 20 | system (red) gets brighter if the fill factor of block groups is higher. 21 | 22 | Because the needed information is retrieved using the btrfs kernel API, it has 23 | to be run as root. 24 | 25 | By default, the filename of the PNG image is a combination of the filesystem ID 26 | and a timestamp, so that if you create multiple of them, they nicely pile up as 27 | input for creating a timelapse video. 28 | 29 | .SS About the ordering of data in the picture 30 | 31 | By default, the ordering inside the picture is based on a Hilbert Curve. The 32 | lowest physical address of the block devices is located in the bottom left 33 | corner. From there it walks up, to the right and down again. 34 | 35 | .SS In btrfs technical terms speaking... 36 | 37 | The picture that is generated by default shows the physical address space of a 38 | filesystem, by walking all dev extents of all devices in the filesystem using 39 | the search ioctl and concatenating all information into a single big image. The 40 | usage values are computed by looking up usage counters in the block group items 41 | from the extent tree. 42 | 43 | It's also possible to have the picture sorted by btrfs virtual address space 44 | instead, or to create pictures of the contents of block groups, on extent 45 | level by using the \-\-blockgroup option. 46 | 47 | .SH OPTIONS 48 | .TP 49 | .BR \-h ", " \-\-help 50 | Show the built\-in help message and exit. 51 | .TP 52 | .BR "\-\-order " \fIorder 53 | Hilbert curve order (default: automatically chosen) 54 | .TP 55 | .BR "\-\-size " \fIsize 56 | Image size (default: 10). Height/width is 2^size 57 | .TP 58 | .BR "\-\-sort " { \fBphysical | \fBvirtual } 59 | Show disk usage sorted on dev_extent (physical, default) or chunk/stripe 60 | (virtual). 61 | .TP 62 | .BR "\-\-blockgroup " \fIvaddr 63 | Instead of a filesystem overview, show extents in the block group starting at 64 | virtual address \fIvaddr\fR. 65 | .TP 66 | .BR "\-\-curve " { \fBhilbert | \fBlinear | \fBsnake } 67 | Space filling curve type or alternative. Default is hilbert. 68 | .TP 69 | .BR \-v ", " \-\-verbose 70 | Increase debug output verbosity. May be specified multiple times to increase 71 | verbosity. (\fB\-v\fR, \fB\-vv\fR, \fB\-vvv\fR, etc...) 72 | .TP 73 | .BR \-o ", " "\-\-output " { \fIfilename | \fIdirectory | \fI- } 74 | Output png file name or directory (default: filename automatically chosen). 75 | When using the special value '-' as file, the png data will be written to the 76 | standard output instead of to a file on disk, so it can be directly viewed by 77 | an image viewer, e.g. with catimg. 78 | 79 | .SH "SEE ALSO" 80 | Source and documentation on github: https://github.com/knorrie/btrfs-heatmap 81 | --------------------------------------------------------------------------------