├── files ├── 999-aws-ebs-nvme.rules └── ebs-nvme-mapping.sh ├── meta └── main.yml ├── LICENSE ├── tasks └── main.yml └── README.md /files/999-aws-ebs-nvme.rules: -------------------------------------------------------------------------------- 1 | ACTION=="add", SUBSYSTEM=="block", KERNEL=="nvme[1-26]n1", ATTRS{model}=="Amazon Elastic Block Store ", RUN+="/usr/local/bin/ebs-nvme-mapping" 2 | -------------------------------------------------------------------------------- /meta/main.yml: -------------------------------------------------------------------------------- 1 | galaxy_info: 2 | author: James Bach 3 | description: A role for installing and running a hack that makes NVME automation possible 4 | license: Apache 2.0 5 | min_ansible_version: 2.4 6 | platforms: 7 | - name: Ubuntu 8 | versions: 9 | - 16.04 10 | - 18.04 11 | -------------------------------------------------------------------------------- /files/ebs-nvme-mapping.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | PATH="${PATH}:/usr/sbin" 4 | 5 | for blkdev in $( nvme list | awk '/^\/dev/ { print $1 }' ) ; do 6 | mapping=$(nvme id-ctrl --raw-binary "${blkdev}" | cut -c3073-3104 | tr -s ' ' | sed 's/ $//g' | sed 's/dev//g' | sed 's/\///g' | sed 's@/dev/@@g') 7 | if [[ "/dev/${mapping}" == /dev/* ]]; then 8 | ( test -b "${blkdev}" && test -L "/dev/${mapping}" ) || ln -s "${blkdev}" "/dev/${mapping}" 9 | fi 10 | done 11 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2018 Omachonu Ogali 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /tasks/main.yml: -------------------------------------------------------------------------------- 1 | --- 2 | 3 | - name: install nvme tools 4 | apt: 5 | name: nvme-cli 6 | state: present 7 | 8 | - name: check if any nvme devices exist 9 | command: nvme list 10 | register: nvme_exist 11 | changed_when: False 12 | check_mode: no 13 | 14 | - name: Install files 15 | block: 16 | - name: Install ebs mapper script 17 | copy: 18 | src: "{{ role_path }}/files/ebs-nvme-mapping.sh" 19 | dest: "/usr/local/bin/ebs-nvme-mapping" 20 | owner: root 21 | group: root 22 | mode: 0755 23 | 24 | - name: Install udev rules 25 | copy: 26 | src: "{{ role_path }}/files/999-aws-ebs-nvme.rules" 27 | dest: "/etc/udev/rules.d/999-aws-ebs-nvme.rules" 28 | owner: root 29 | group: root 30 | mode: 0644 31 | 32 | - name: Ensure script runs once 33 | command: /usr/local/bin/ebs-nvme-mapping 34 | changed_when: False 35 | 36 | when: not(nvme_exist.stdout | regex_search('No NVMe devices detected.')) 37 | 38 | - name: remove nvme tool 39 | apt: 40 | name: nvme-cli 41 | state: absent 42 | when: nvme_exist.stdout | regex_search('No NVMe devices detected.') 43 | 44 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Automatic Mapping of NVMe-style EBS Volums to Standard Block Device Paths 2 | 3 | 1. [Introduction](#introduction) 4 | 2. [The Problem](#the-problem) 5 | 3. [Brainstorming](#brainstorming) 6 | 4. [The Hacky Solution](#the-hacky-solution) 7 | 5. [The Less Hacky Solution](#the-less-hacky-solution) 8 | 6. [A Test Run](#a-test-run) 9 | 10 | ## Introduction 11 | 12 | So you've decided to use the new `c5.large` or `m5.large` instance type in EC2. 13 | 14 | Congratulations! You get more IOPS! 15 | 16 | ``` 17 | $ aws --profile=personal-aws-testing ec2 --region=us-east-1 \ 18 | run-instances \ 19 | --count=1 \ 20 | --instance-type=m5.large \ 21 | --image-id=ami-8d3fd59b \ 22 | --key-name=aws-testing \ 23 | --subnet-id=subnet-bbbbbbbb \ 24 | --security-group-ids=sg-77777777 \ 25 | --ebs-optimized \ 26 | --instance-initiated-shutdown-behavior=terminate \ 27 | --tag-specifications='ResourceType=instance,Tags=[{Key=Name,Value=nvme-mapping-example}]' 28 | ``` 29 | But wait, when you go to look at mountpoints (and disk space), what is this new `/dev/nvme*` device 30 | you see? 31 | 32 | ``` 33 | [ec2-user@ip-10-81-64-224 ~]$ df -h 34 | Filesystem Size Used Avail Use% Mounted on 35 | ... 36 | /dev/nvme0n1p1 7.8G 956M 6.8G 13% / 37 | ``` 38 | 39 | Oh, ok. That's a NVMe block device per the AWS documentation[1]. That's cool. 40 | 41 | ## The Problem 42 | 43 | Now, let's create and attach a 10GB EBS volume. 44 | 45 | ``` 46 | $ aws --profile=personal-aws-testing ec2 --region=us-east-1 \ 47 | create-volume \ 48 | --availability-zone=us-east-1a \ 49 | --size=10 \ 50 | --volume-type=gp2 \ 51 | --tag-specifications='ResourceType=volume,Tags=[{Key=Name,Value="nvme-mapping-example, Example EBS Volume"}]' 52 | ... 53 | 54 | $ aws --profile=personal-aws-testing ec2 --region=us-east-1 \ 55 | attach-volume \ 56 | --device=/dev/xvdt \ 57 | --instance-id=i-44444444444444444 \ 58 | --volume-id=vol-22222222222222222 59 | ``` 60 | 61 | Right, now let's verify our volume was attached. 62 | 63 | ``` 64 | [ec2-user@ip-10-81-66-128 ~]$ dmesg | grep xvdt 65 | [ec2-user@ip-10-81-66-128 ~]$ 66 | ``` 67 | 68 | Uh, there's no mention of our block device (`xvdt`) in the kernel output. What about NVMe devices? 69 | 70 | ``` 71 | [ 504.204889] nvme 0000:00:1f.0: enabling device (0000 -> 0002) 72 | ``` 73 | 74 | That's not quite what I'm expecting from the old days. 75 | 76 | ## Brainstorming 77 | 78 | There's more to this NVMe business. Let's get to it. 79 | 80 | ### nvme-cli 81 | 82 | We'll start by install the NVMe tools, and requesting a list of all NVMe devices in the system. 83 | 84 | ``` 85 | [ec2-user@ip-10-81-66-128 ~]$ sudo yum install nvme-cli 86 | ... 87 | Installed: 88 | nvme-cli.x86_64 0:0.7-1.3.amzn1 89 | 90 | Complete! 91 | 92 | [ec2-user@ip-10-81-66-128 ~]$ sudo nvme list 93 | Node SN Model Version Namespace Usage Format FW Rev 94 | ---------------- -------------------- ---------------------------------------- -------- --------- -------------------------- ---------------- -------- 95 | /dev/nvme0n1 vol11111111111111111 Amazon Elastic Block Store 1.0 1 0.00 B / 8.59 GB 512 B + 0 B 1.0 96 | /dev/nvme1n1 vol22222222222222222 Amazon Elastic Block Store 1.0 1 0.00 B / 10.74 GB 512 B + 0 B 1.0 97 | ``` 98 | 99 | The summary output shows our root EBS device (`/dev/nvme0n1`) and our newly created-and-attached 100 | device (`/dev/nvme1n1`). 101 | 102 | #### Can we get more information about our device? Yes. 103 | 104 | ``` 105 | [ec2-user@ip-10-81-66-128 ~]$ sudo nvme id-ctrl /dev/nvme1n1 106 | NVME Identify Controller: 107 | vid : 0x1d0f 108 | ssvid : 0x1d0f 109 | sn : vol22222222222222222 110 | mn : Amazon Elastic Block Store 111 | fr : 1.0 112 | rab : 32 113 | ieee : dc02a0 114 | cmic : 0 115 | mdts : 6 116 | cntlid : 0 117 | ver : 0 118 | rtd3r : 0 119 | rtd3e : 0 120 | oaes : 0 121 | oacs : 0 122 | acl : 4 123 | aerl : 0 124 | frmw : 0x3 125 | lpa : 0 126 | elpe : 0 127 | npss : 1 128 | avscc : 0x1 129 | apsta : 0 130 | wctemp : 0 131 | cctemp : 0 132 | mtfa : 0 133 | hmpre : 0 134 | hmmin : 0 135 | tnvmcap : 0 136 | unvmcap : 0 137 | rpmbs : 0 138 | sqes : 0x66 139 | cqes : 0x44 140 | nn : 1 141 | oncs : 0 142 | fuses : 0 143 | fna : 0 144 | vwc : 0x1 145 | awun : 0 146 | awupf : 0 147 | nvscc : 0 148 | acwu : 0 149 | sgls : 0 150 | ps 0 : mp:0.01W operational enlat:1000000 exlat:1000000 rrt:0 rrl:0 151 | rwt:0 rwl:0 idle_power:- active_power:- 152 | ps 1 : mp:0.00W operational enlat:0 exlat:0 rrt:0 rrl:0 153 | rwt:0 rwl:0 idle_power:- active_power:- 154 | ``` 155 | 156 | #### How about MORE information? Yes. 157 | 158 | ``` 159 | [ec2-user@ip-10-81-66-128 ~]$ sudo nvme id-ctrl --vendor-specific /dev/nvme1n1 160 | ... 161 | vs[]: 162 | 0 1 2 3 4 5 6 7 8 9 a b c d e f 163 | 0000: 2f 64 65 76 2f 78 76 64 74 20 20 20 20 20 20 20 "/dev/xvdt......." 164 | 0010: 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 "................" 165 | 0020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "................" 166 | 0030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "................" 167 | 0040: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "................" 168 | 0050: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "................" 169 | 0060: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "................" 170 | 0070: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "................" 171 | 0080: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "................" 172 | 0090: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "................" 173 | 00a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "................" 174 | 00b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "................" 175 | 00c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "................" 176 | 00d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "................" 177 | 00e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "................" 178 | 00f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "................" 179 | 0100: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "................" 180 | 0110: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "................" 181 | 0120: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "................" 182 | 0130: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "................" 183 | 0140: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "................" 184 | 0150: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "................" 185 | 0160: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "................" 186 | 0170: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "................" 187 | 0180: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "................" 188 | 0190: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "................" 189 | 01a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "................" 190 | 01b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "................" 191 | 01c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "................" 192 | 01d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "................" 193 | 01e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "................" 194 | 01f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "................" 195 | 0200: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "................" 196 | 0210: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "................" 197 | 0220: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "................" 198 | 0230: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "................" 199 | 0240: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "................" 200 | 0250: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "................" 201 | 0260: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "................" 202 | 0270: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "................" 203 | 0280: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "................" 204 | 0290: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "................" 205 | 02a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "................" 206 | 02b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "................" 207 | 02c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "................" 208 | 02d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "................" 209 | 02e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "................" 210 | 02f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "................" 211 | 0300: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "................" 212 | 0310: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "................" 213 | 0320: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "................" 214 | 0330: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "................" 215 | 0340: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "................" 216 | 0350: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "................" 217 | 0360: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "................" 218 | 0370: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "................" 219 | 0380: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "................" 220 | 0390: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "................" 221 | 03a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "................" 222 | 03b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "................" 223 | 03c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "................" 224 | 03d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "................" 225 | 03e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "................" 226 | 03f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 "................" 227 | ``` 228 | 229 | Whaddya know? There's our requested block device name, at the beginning of the vendor specific 230 | information. 231 | 232 | Let's extract it. How? The `nvme id-ctrl` command takes an option called `--raw-binary`, which dumps 233 | out the information block inclusive of the vendor-specific data. 234 | 235 | ``` 236 | [ec2-user@ip-10-81-66-128 ~]$ sudo nvme id-ctrl --raw-binary /dev/nvme1n1 | hexdump -C 237 | 00000000 0f 1d 0f 1d 76 6f 6c 30 32 34 32 39 34 33 34 62 |....vol222222222| 238 | 00000010 38 61 35 35 37 66 66 32 41 6d 61 7a 6f 6e 20 45 |22222222Amazon E| 239 | 00000020 6c 61 73 74 69 63 20 42 6c 6f 63 6b 20 53 74 6f |lastic Block Sto| 240 | 00000030 72 65 20 20 20 20 20 20 20 20 20 20 20 20 20 20 |re | 241 | 00000040 31 2e 30 20 20 20 20 20 20 a0 02 dc 00 06 00 00 |1.0 .......| 242 | 00000050 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 243 | * 244 | 00000100 00 00 04 00 03 00 00 01 01 00 00 00 00 00 00 00 |................| 245 | 00000110 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 246 | * 247 | 00000200 66 44 00 00 01 00 00 00 00 00 00 00 00 01 00 00 |fD..............| 248 | 00000210 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 249 | * 250 | 00000800 01 00 00 00 40 42 0f 00 40 42 0f 00 00 00 00 00 |....@B..@B......| 251 | 00000810 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 252 | * 253 | 00000c00 2f 64 65 76 2f 78 76 64 74 20 20 20 20 20 20 20 |/dev/xvdt | 254 | 00000c10 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 | | 255 | 00000c20 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 256 | * 257 | 00001000 258 | ``` 259 | 260 | The information within the vendor-specific data we are looking for appears to start at an offset of 261 | 3,072 bytes, is a 32-byte record, and is padded with spaces (`0x20` == 32 dec == ``). 262 | 263 | ``` 264 | [ec2-user@ip-10-81-66-128 ~]$ sudo nvme id-ctrl --raw-binary /dev/nvme1n1 | cut -c3073-3104 265 | /dev/xvdt 266 | ``` 267 | 268 | It looks fine... but it has trailing spaces. How to verify that? Count the characters. 269 | 270 | ``` 271 | [ec2-user@ip-10-81-66-128 ~]$ sudo nvme id-ctrl --raw-binary /dev/nvme1n1 | cut -c3073-3104 | wc -c 272 | 33 273 | ``` 274 | 275 | (It says 33 characters: 32 characters from our original block, plus one byte for a newline added by 276 | `cut`) 277 | 278 | So, let's trim the trailing spaces for a viable block device name. 279 | 280 | ``` 281 | [ec2-user@ip-10-81-66-128 ~]$ sudo nvme id-ctrl --raw-binary /dev/nvme1n1 | cut -c3073-3104 | tr -s ' ' | sed 's/ $//g' 282 | /dev/xvdt 283 | ``` 284 | 285 | We now have our desired block device name. 286 | 287 | ## The Hacky Solution 288 | 289 | We can create a symbolic link from the origin NVMe device, to the desired block device name. 290 | 291 | ``` 292 | [ec2-user@ip-10-81-66-128 ~]$ sudo ln -s /dev/nvme1n1 /dev/xvdt 293 | [ec2-user@ip-10-81-66-128 ~]$ 294 | ``` 295 | 296 | However, the original block device can change with each reboot of the EC2 instance, e.g. reboot an 297 | EC2 instance with two volumes attached, and AWS can attach the two EBS volumes in different order, 298 | resulting in `nvme1n1` and `nvme2n1` swapping places. 299 | 300 | But it's not limited to reboots. The same behavior can happen whenever an EBS volume is detached or 301 | attached. 302 | 303 | ## The Less Hacky Solution 304 | 305 | To make this a bit more resilient to attach-detach events, we want to trigger our shell script upon 306 | each of those events. This is what udev[2] was made for. 307 | 308 | ### udev 309 | 310 | `udev`, the userspace /dev manager, operates as a daemon, that receives events each time a device is 311 | attached/detached from the host. It reads each event, compares the attributes of that event to a set 312 | of rules (located in `/etc/udev/rules.d`), and executes the specified actions of the rule. 313 | 314 | What are the attributes of our NVMe device that we can match on? 315 | 316 | ``` 317 | [ec2-user@ip-10-81-66-128 ~]$ sudo udevadm info --query=all --attribute-walk --path=/sys/block/nvme1n1 318 | 319 | Udevadm info starts with the device specified by the devpath and then 320 | walks up the chain of parent devices. It prints for every device 321 | found, all possible attributes in the udev rules key format. 322 | A rule to match, can be composed by the attributes of the device 323 | and the attributes from one single parent device. 324 | 325 | looking at device '/devices/pci0000:00/0000:00:1f.0/nvme/nvme1/nvme1n1': 326 | KERNEL=="nvme1n1" 327 | SUBSYSTEM=="block" 328 | DRIVER=="" 329 | ATTR{ro}=="0" 330 | ATTR{size}=="20971520" 331 | ATTR{stat}==" 61 0 920 108 0 0 0 0 0 108 108" 332 | ATTR{range}=="0" 333 | ATTR{discard_alignment}=="0" 334 | ATTR{ext_range}=="256" 335 | ATTR{alignment_offset}=="0" 336 | ATTR{inflight}==" 0 0" 337 | ATTR{removable}=="0" 338 | ATTR{capability}=="50" 339 | 340 | looking at parent device '/devices/pci0000:00/0000:00:1f.0/nvme/nvme1': 341 | KERNELS=="nvme1" 342 | SUBSYSTEMS=="nvme" 343 | DRIVERS=="" 344 | ATTRS{model}=="Amazon Elastic Block Store " 345 | ATTRS{serial}=="vol22222222222222222" 346 | ATTRS{firmware_rev}=="1.0 " 347 | 348 | looking at parent device '/devices/pci0000:00/0000:00:1f.0': 349 | KERNELS=="0000:00:1f.0" 350 | SUBSYSTEMS=="pci" 351 | DRIVERS=="nvme" 352 | ATTRS{irq}=="10" 353 | ATTRS{subsystem_vendor}=="0x1d0f" 354 | ATTRS{broken_parity_status}=="0" 355 | ATTRS{class}=="0x010802" 356 | ATTRS{driver_override}=="(null)" 357 | ATTRS{consistent_dma_mask_bits}=="64" 358 | ATTRS{dma_mask_bits}=="64" 359 | ATTRS{local_cpus}=="3" 360 | ATTRS{device}=="0x8061" 361 | ATTRS{enable}=="1" 362 | ATTRS{msi_bus}=="1" 363 | ATTRS{local_cpulist}=="0-1" 364 | ATTRS{vendor}=="0x1d0f" 365 | ATTRS{subsystem_device}=="0x8061" 366 | ATTRS{numa_node}=="0" 367 | ATTRS{d3cold_allowed}=="0" 368 | 369 | looking at parent device '/devices/pci0000:00': 370 | KERNELS=="pci0000:00" 371 | SUBSYSTEMS=="" 372 | DRIVERS=="" 373 | ``` 374 | 375 | In order to identify EBS volumes, we want something that is stable across reboots. But not too 376 | specific to this volume, otherwise you're flying in the face of automation and manually hard-coding 377 | your configuation. 378 | 379 | I've picked `ATTRS{model}`. 380 | 381 | Let's combine what we've found into a shell script... 382 | 383 | ``` 384 | [ec2-user@ip-10-81-66-128 ~]$ cat < ebs-nvme-mapping 385 | > #!/bin/bash 386 | > 387 | > PATH="${PATH}:/usr/sbin" 388 | > 389 | > for blkdev in $( nvme list | awk '/^\/dev/ { print $1 }' ) ; do 390 | > mapping=$(nvme id-ctrl --raw-binary "${blkdev}" | cut -c3073-3104 | tr -s ' ' | sed 's/ $//g') 391 | > if [[ "${mapping}" == /dev/* ]]; then 392 | > ( test -b "${blkdev}" && test -L "${mapping}" ) || ln -s "${blkdev}" "${mapping}" 393 | > fi 394 | > done 395 | > EOF 396 | [ec2-user@ip-10-81-66-128 ~]$ sudo install -m 0755 ebs-nvme-mapping /usr/local/bin/ 397 | [ec2-user@ip-10-81-66-128 ~]$ 398 | ``` 399 | 400 | ...and a udev rule... 401 | 402 | ``` 403 | [ec2-user@ip-10-81-66-128 ~]$ cat < 999-aws-ebs-nvme.rules 404 | > ACTION=="add", SUBSYSTEM=="block", KERNEL=="nvme[1-26]n1", ATTRS{model}=="Amazon Elastic Block Store ", RUN+="/usr/local/bin/ebs-nvme-mapping" 405 | > EOF 406 | [ec2-user@ip-10-81-66-128 ~]$ sudo install -m 0644 999-aws-ebs-nvme.rules /etc/udev/rules.d/ 407 | [ec2-user@ip-10-81-66-128 ~]$ 408 | ``` 409 | 410 | `udev` will automatically reload rules upon changes to files in the rules directory. So we're locked 411 | and loaded. 412 | 413 | Now, when we attach and detach EBS volumes, our shell script will run. 414 | 415 | ## A Test Run 416 | 417 | ### Dry run 418 | 419 | ``` 420 | [ec2-user@ip-10-81-66-128 ~]$ sudo udevadm test /sys/block/nvme1n1 421 | run_command: calling: test 422 | adm_test: version 173 423 | This program is for debugging only, it does not run any program, 424 | specified by a RUN key. It may show incorrect results, because 425 | some values may be different, or not available at a simulation run. 426 | 427 | parse_file: reading '/lib/udev/rules.d/10-console.rules' as rules file 428 | parse_file: reading '/lib/udev/rules.d/10-dm.rules' as rules file 429 | parse_file: reading '/lib/udev/rules.d/11-dm-lvm.rules' as rules file 430 | parse_file: reading '/lib/udev/rules.d/13-dm-disk.rules' as rules file 431 | parse_file: reading '/lib/udev/rules.d/42-qemu-usb.rules' as rules file 432 | parse_file: reading '/lib/udev/rules.d/50-firmware.rules' as rules file 433 | parse_file: reading '/lib/udev/rules.d/50-udev-default.rules' as rules file 434 | parse_file: reading '/etc/udev/rules.d/51-ec2-hvm-devices.rules' as rules file 435 | parse_file: reading '/etc/udev/rules.d/52-ec2-vcpu.rules' as rules file 436 | parse_file: reading '/etc/udev/rules.d/53-ec2-network-interfaces.rules' as rules file 437 | parse_file: reading '/etc/udev/rules.d/60-cdrom_id.rules' as rules file 438 | parse_file: reading '/lib/udev/rules.d/60-floppy.rules' as rules file 439 | parse_file: reading '/lib/udev/rules.d/60-net.rules' as rules file 440 | parse_file: reading '/lib/udev/rules.d/60-persistent-alsa.rules' as rules file 441 | parse_file: reading '/lib/udev/rules.d/60-persistent-input.rules' as rules file 442 | parse_file: reading '/lib/udev/rules.d/60-persistent-serial.rules' as rules file 443 | parse_file: reading '/lib/udev/rules.d/60-persistent-storage-tape.rules' as rules file 444 | parse_file: reading '/lib/udev/rules.d/60-persistent-storage.rules' as rules file 445 | parse_file: reading '/lib/udev/rules.d/60-persistent-v4l.rules' as rules file 446 | parse_file: reading '/etc/udev/rules.d/60-raw.rules' as rules file 447 | parse_file: reading '/lib/udev/rules.d/61-accelerometer.rules' as rules file 448 | parse_file: reading '/lib/udev/rules.d/64-md-raid.rules' as rules file 449 | parse_file: reading '/lib/udev/rules.d/65-md-incremental.rules' as rules file 450 | parse_file: reading '/lib/udev/rules.d/69-dm-lvm-metad.rules' as rules file 451 | parse_file: reading '/etc/udev/rules.d/70-ec2-nvme-devices.rules' as rules file 452 | parse_file: reading '/lib/udev/rules.d/75-cd-aliases-generator.rules' as rules file 453 | parse_file: reading '/lib/udev/rules.d/75-net-description.rules' as rules file 454 | parse_file: reading '/etc/udev/rules.d/75-persistent-net-generator.rules' as rules file 455 | parse_file: reading '/lib/udev/rules.d/75-probe_mtd.rules' as rules file 456 | parse_file: reading '/lib/udev/rules.d/75-tty-description.rules' as rules file 457 | parse_file: reading '/lib/udev/rules.d/78-sound-card.rules' as rules file 458 | parse_file: reading '/lib/udev/rules.d/80-drivers.rules' as rules file 459 | parse_file: reading '/lib/udev/rules.d/81-kvm-rhel.rules' as rules file 460 | parse_file: reading '/lib/udev/rules.d/88-clock.rules' as rules file 461 | parse_file: reading '/lib/udev/rules.d/95-dm-notify.rules' as rules file 462 | parse_file: reading '/lib/udev/rules.d/95-keyboard-force-release.rules' as rules file 463 | parse_file: reading '/lib/udev/rules.d/95-keymap.rules' as rules file 464 | parse_file: reading '/lib/udev/rules.d/95-udev-late.rules' as rules file 465 | parse_file: reading '/dev/.udev/rules.d/99-root.rules' as rules file 466 | parse_file: reading '/etc/udev/rules.d/999-aws-ebs-nvme.rules' as rules file 467 | udev_rules_new: rules use 23184 bytes tokens (1932 * 12 bytes), 15869 bytes buffer 468 | udev_rules_new: temporary index used 15140 bytes (757 * 20 bytes) 469 | udev_device_new_from_syspath: device 0x55e241effc90 has devpath '/devices/pci0000:00/0000:00:1f.0/nvme/nvme1/nvme1n1' 470 | udev_device_new_from_syspath: device 0x55e241effdc0 has devpath '/devices/pci0000:00/0000:00:1f.0/nvme/nvme1/nvme1n1' 471 | udev_device_read_db: device 0x55e241effdc0 filled with db file data 472 | udev_rules_apply_to_event: GROUP 6 /lib/udev/rules.d/50-udev-default.rules:68 473 | udev_rules_apply_to_event: IMPORT 'path_id /devices/pci0000:00/0000:00:1f.0/nvme/nvme1/nvme1n1' /lib/udev/rules.d/60-persistent-storage.rules:61 474 | udev_event_spawn: starting 'path_id /devices/pci0000:00/0000:00:1f.0/nvme/nvme1/nvme1n1' 475 | spawn_read: 'path_id /devices/pci0000:00/0000:00:1f.0/nvme/nvme1/nvme1n1'(out) 'ID_PATH=pci-0000:00:1f.0' 476 | spawn_read: 'path_id /devices/pci0000:00/0000:00:1f.0/nvme/nvme1/nvme1n1'(out) 'ID_PATH_TAG=pci-0000_00_1f_0' 477 | spawn_wait: 'path_id /devices/pci0000:00/0000:00:1f.0/nvme/nvme1/nvme1n1' [22764] exit with return code 0 478 | udev_rules_apply_to_event: LINK 'disk/by-path/pci-0000:00:1f.0' /lib/udev/rules.d/60-persistent-storage.rules:62 479 | udev_rules_apply_to_event: IMPORT '/sbin/blkid -o udev -p /dev/nvme1n1' /lib/udev/rules.d/60-persistent-storage.rules:74 480 | udev_event_spawn: starting '/sbin/blkid -o udev -p /dev/nvme1n1' 481 | spawn_wait: '/sbin/blkid -o udev -p /dev/nvme1n1' [22765] exit with return code 2 482 | udev_device_new_from_syspath: device 0x55e241efff80 has devpath '/devices/pci0000:00/0000:00:1f.0/nvme/nvme1' 483 | udev_rules_apply_to_event: 3 character(s) replaced 484 | udev_rules_apply_to_event: LINK 'disk/by-id/nvme-Amazon_Elastic_Block_Store_vol22222222222222222-ns-1' /etc/udev/rules.d/70-ec2-nvme-devices.rules:17 485 | udev_rules_apply_to_event: RUN '/usr/local/bin/ebs-nvme-mapping' /etc/udev/rules.d/999-aws-ebs-nvme.rules:1 486 | udev_event_execute_rules: no node name set, will use kernel supplied name 'nvme1n1' 487 | udev_node_add: creating device node '/dev/nvme1n1', devnum=259:3, mode=0660, uid=0, gid=6 488 | udev_node_mknod: preserve file '/dev/nvme1n1', because it has correct dev_t 489 | udev_node_mknod: preserve permissions /dev/nvme1n1, 060660, uid=0, gid=6 490 | node_symlink: preserve already existing symlink '/dev/block/259:3' to '../nvme1n1' 491 | link_find_prioritized: found 'b259:3' claiming '/dev/.udev/links/disk\x2fby-path\x2fpci-0000:00:1f.0' 492 | link_update: creating link '/dev/disk/by-path/pci-0000:00:1f.0' to '/dev/nvme1n1' 493 | node_symlink: preserve already existing symlink '/dev/disk/by-path/pci-0000:00:1f.0' to '../../nvme1n1' 494 | link_find_prioritized: found 'b259:3' claiming '/dev/.udev/links/disk\x2fby-id\x2fnvme-Amazon_Elastic_Block_Store_vol22222222222222222-ns-1' 495 | link_update: creating link '/dev/disk/by-id/nvme-Amazon_Elastic_Block_Store_vol22222222222222222-ns-1' to '/dev/nvme1n1' 496 | node_symlink: preserve already existing symlink '/dev/disk/by-id/nvme-Amazon_Elastic_Block_Store_vol22222222222222222-ns-1' to '../../nvme1n1' 497 | udev_device_update_db: created db file '/dev/.udev/data/b259:3' for '/devices/pci0000:00/0000:00:1f.0/nvme/nvme1/nvme1n1' 498 | UDEV_LOG=6 499 | DEVPATH=/devices/pci0000:00/0000:00:1f.0/nvme/nvme1/nvme1n1 500 | MAJOR=259 501 | MINOR=3 502 | DEVNAME=/dev/nvme1n1 503 | DEVTYPE=disk 504 | ACTION=add 505 | SUBSYSTEM=block 506 | ID_PATH=pci-0000:00:1f.0 507 | ID_PATH_TAG=pci-0000_00_1f_0 508 | DEVLINKS=/dev/disk/by-path/pci-0000:00:1f.0 /dev/disk/by-id/nvme-Amazon_Elastic_Block_Store_vol22222222222222222-ns-1 509 | .ID_FS_TYPE_NEW= 510 | ID_FS_TYPE= 511 | run: '/usr/local/bin/ebs-nvme-mapping' 512 | ``` 513 | 514 | ### Live run 515 | 516 | We have no block device (symlink) before... 517 | 518 | ``` 519 | [ec2-user@ip-10-81-66-128 ~]$ ls -la /dev/xvdf 520 | ls: cannot access /dev/xvdf: No such file or directory 521 | ``` 522 | 523 | Now let's create and attach a new volume... 524 | 525 | ``` 526 | $ aws --profile=personal-aws-testing ec2 --region=us-east-1 \ 527 | create-volume \ 528 | --availability-zone=us-east-1a \ 529 | --size=30 \ 530 | --volume-type=gp2 \ 531 | --tag-specifications='ResourceType=volume,Tags=[{Key=Name,Value="nvme-mapping-example, Example EBS Volume #2"}]' 532 | ... 533 | 534 | $ aws --profile=personal-aws-testing ec2 --region=us-east-1 \ 535 | attach-volume \ 536 | --device=/dev/xvdf \ 537 | --instance-id=i-44444444444444444 \ 538 | --volume-id=vol-22222222222222222 539 | ... 540 | ``` 541 | 542 | Et voila! 543 | 544 | ``` 545 | [ec2-user@ip-10-81-66-128 ~]$ ls -la /dev/xvdf 546 | lrwxrwxrwx 1 root root 12 Jan 19 21:43 /dev/xvdf -> /dev/nvme2n1 547 | ``` 548 | 549 | 1: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/nvme-ebs-volumes.html 550 | 551 | 2: https://en.wikipedia.org/wiki/Udev 552 | --------------------------------------------------------------------------------