├── LICENSE ├── blocksizechange.md ├── README.md └── removeintegrityprotection.md /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2024 GMS 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /blocksizechange.md: -------------------------------------------------------------------------------- 1 | # Symptoms and Issue 2 | After installing a hard disk drive, the disk is not recognized by the system despite the disk being functional and correctly seated. 3 | 4 | # Warnings 5 | :exclamation: This guide is solely for SAS or SCSI disks and does not apply to SATA disks. 6 | 7 | :exclamation: An interuption of the process can destroy your disk, assure correct functioning of all other components, stable energy supply and. 8 | 9 | :exclamation: Avoid using remote access to shells As soon as you close the shell the process might be (prematurely) terminated. 10 | 11 | :exclamation: Study the drives you want to reformat carefully to avoid data loss. 12 | 13 | ## Strategy to reformat the block size of a physical disk for one drive 14 | 15 | ### Hardware Requirements 16 | 17 | :exclamation: Only use HBA / PCI Mode controllers that allow to loop through all information of the disk and all data streams directly to the operating system. 18 | Don't use under any circumstance Raid Controllers to perform these operations. 19 | You can find more information about HBA and Raid Controllers [here](https://www.servethehome.com/buyers-guides/top-hardware-components-for-truenas-freenas-nas-servers/top-picks-truenas-freenas-hbas/) 20 | 21 | ### Software Requierements 22 | * [sg3_utils]([url](https://sg.danny.cz/sg/sg3_utils.html)) 23 | 24 | ### Instructions 25 | 1. List disks using lsblk 26 | 2. Identify the device with the wrong block size by executing `sg_format ` 27 | 3. Run `sg_format -f --format --size=<512 / 4096> --quick` 28 | 4. Wait till the process terminated and check by executing `sg_format ` again. 29 | 30 | ## Strategy to reformat the block size of a physical disk for multiple drives at the same time 31 | 32 | ### Requierements 33 | * [sg3_utils]([url](https://sg.danny.cz/sg/sg3_utils.html)) 34 | * [parallel]([url](https://www.gnu.org/software/parallel/)) 35 | 36 | ### Instructions 37 | 1. List disks using lsblk 38 | 2. Identify the device with the wrong block size by executing `sg_format ` 39 | 3. Run `for i in {1..28}; do echo "sg_format --format --size=512 /dev/sg$i"; done | parallel -j40` 40 | 41 | ### Additional Information for systems using letters to denominate drives 42 | The same command would work also with systems using a drive letter system: 43 | `for i in {a..z}; do echo "sg_format --format --size=512 /dev/sg$i"; done | parallel -j40` 44 | 45 | ### Additional Information for formating multiple drives that are not in a sequence 46 | Given that the nomenclation of the drives is not always sequential, for i in can be adapted to that use case: 47 | If I want to format drives /dev/sga to /dev/sgc and the drives /dev/sge/ to /dev/sgf/ the command could look like this: 48 | `for i in {{a..c},{e..f}}; do echo "sg_format --format --size=512 /dev/sg$i"; done | parallel -j40` 49 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Disk Formating Guide 2 | This guide covers an array of formatting related issues with SAS / SATA hard disk for usage in general purpose server environments. 3 | 4 | # Should I boot from disk? 5 | Largely speaking all instructions here should perform reasonably well in all linux environments. Some systems though such as TrueNas do not allow the installation of additional packages needed to fix your issues effectively. For that reason I would consider to boot from an iso to avoid any issues. For the purpose of reformating disks [ShredOS](https://github.com/PartialVolume/shredos.x86_64/) is a good choice. 6 | * it is lightweight; 7 | * it comes with support for most sas / raid controllers; 8 | * it comes with support for plenty of network cards; 9 | * it comes prebundled with the parallel package to allow you executing all procedures in this guides on multiple disks at the same time. 10 | 11 | # How do I start? 12 | 1. Download the iso from the [ShredOS Repository](https://github.com/PartialVolume/shredos.x86_64/). 13 | 2. [Configure Telnet on ShredOS](https://github.com/PartialVolume/shredos.x86_64?tab=readme-ov-file#how-to-wipe-drives-on-headless-systems-or-systems-with-faulty-display-hardware-for-use-on-secure-lans-only) to access headless systems if neccessary, 14 | 3. Switch after the boot of ShredOS to a virtual terminal by pressing ALT + F3, 15 | 4. Execute the instructions [Change Block size](https://github.com/gms-electronics/formatingguide/blob/main/blocksizechange.md) or [Remove the Data Integrity Feature from disks](https://github.com/gms-electronics/formatingguide/blob/main/removeintegrityprotection.md). 16 | 17 | # Contained Guides 18 | 1. [Change Block size from 3Par or EMC Storrage solution disks from 520 to 512 / 4096 bytes](https://github.com/gms-electronics/formatingguide/blob/main/blocksizechange.md) 19 | 2. [Remove the Data Integrity Feature from disks](https://github.com/gms-electronics/formatingguide/blob/main/removeintegrityprotection.md) 20 | 21 | # Sources 22 | * [Reformating 520k Drives](https://forum.level1techs.com/t/how-to-reformat-520-byte-drives-to-512-bytes-usually/133021), Level 1 Techs Forum, 2018 23 | * [Troubleshooting Disk Format Warnings in TrueNas Scale](https://www.truenas.com/community/threads/troubleshooting-disk-format-warnings-in-truenas-scale.106051/), Truenas Forum, constantly updated 24 | * [ShredOS](https://github.com/PartialVolume/shredos.x86_64/) 25 | 26 | # Attributions 27 | * Special thanks got to @PartialVolume and his tool [ShredOS](https://github.com/PartialVolume/shredos.x86_64/). It has been immensely helpful for disk sanitization for quiet some time now and is the Open Source reference in that field; 28 | * TrueNas Community Member [Daisuke](https://www.truenas.com/community/members/daisuke.1017/) for giving the most dettailed description and solution approach; 29 | * Wendell Wilson from Level 1 Tech who was the first one describing the problem and the solution. 30 | 31 | -------------------------------------------------------------------------------- /removeintegrityprotection.md: -------------------------------------------------------------------------------- 1 | # Symptoms and Issue 2 | Your system rejects the adoption of a drive with a warning message such as the following example on TrueNas: 3 | `Disk(s): sda ... are formatted with Data Integrity Feature (DIF) which is unsupported.` 4 | The data integrity Feature or also called T10 Protection Information was a feature first enabled on enterprise SAS HDDs and SSDs that has been included in the NVMe specification. T10 Protection Information allows for additional metadata to reside with user data which contains additional fields to help protect against cases of misdirected writes and silent data corruption, to name a few scenarios. This feature is not universally supported and might cause the rejection of the disk in certain systems. 5 | You can identify disks by running `sg_format ` looking for `PROTECT=1` or 6 | # Warnings 7 | :exclamation: This guide is solely for SAS or SCSI disks and does not apply to SATA disks. 8 | 9 | :exclamation: An interuption of the process can destroy your disk, assure correct functioning of all other components and stable energy supply. 10 | 11 | :exclamation: Avoid using remote access to shells As soon as you close the shell the process might be (prematurely) terminated. 12 | 13 | :exclamation: Study the drives you want to reformat carefully to avoid data loss by formatting the wrong drive. 14 | 15 | ### Hardware Requirements 16 | 17 | :exclamation: Only use HBA / PCI Mode controllers that allow to loop through all information of the disk and all data streams directly to the operating system. 18 | Don't use under any circumstance Raid Controllers to perform these operations. 19 | You can find more information about HBA and Raid Controllers [here](https://www.servethehome.com/buyers-guides/top-hardware-components-for-truenas-freenas-nas-servers/top-picks-truenas-freenas-hbas/) 20 | 21 | ## Strategy to reformat the block size of a physical disk for one drive 22 | 23 | ### Requierements 24 | * [sg3_utils]([url](https://sg.danny.cz/sg/sg3_utils.html)) 25 | 26 | ### Instructions 27 | 1. List disks using lsblk 28 | 2. Identify the device with the wrong block size by executing `sg_format ` 29 | 3. Run `sg_format -v --fmtpinfo=0 --format --size=<512 / 4096> --quick` 30 | 4. Wait till the process terminated and check by executing `sg_format ` again. 31 | 32 | ## Strategy to reformat the block size of a physical disk for multiple drives at the same time 33 | 34 | ### Requierements 35 | * [sg3_utils]([url](https://sg.danny.cz/sg/sg3_utils.html)) 36 | * [parallel]([url](https://www.gnu.org/software/parallel/)) 37 | 38 | ### Instructions 39 | 1. List disks using lsblk 40 | 2. Identify the device with the wrong block size by executing `sg_format ` 41 | 3. Run `for i in {1..28}; do echo "sg_format -v --fmtpinfo=0 --format --size=<512 / 4096> /dev/sg$i --quick"; done | parallel -j40` 42 | 43 | ### Additional Information for systems using letters to denominate drives 44 | The same command would work also with systems using a drive letter system: 45 | `for i in {a..z}; do echo "sg_format -v --fmtpinfo=0 --format --size=<512 / 4096> /dev/sg$i --quick"; done | parallel -j40` 46 | 47 | 48 | ### Additional Information for formating multiple drives that are not in a sequence 49 | Given that the nomenclation of the drives is not always sequential, for i in can be adapted to that use case: 50 | If I want to format drives /dev/sga to /dev/sgc and the drives /dev/sge/ to /dev/sgf/ the command could look like this: 51 | `for i in {{a..c},{e..f}}; do echo "sg_format -v --fmtpinfo=0 --format --size=<512 / 4096> /dev/sg$i --quick"; done | parallel -j40` 52 | --------------------------------------------------------------------------------