├── CONTRIBUTING.md ├── assets └── README.md ├── docs ├── data-types.md ├── use-cases.md ├── best-practices.md ├── tools-and-libraries.md └── compression-techniques.md ├── scripts └── README.md ├── examples ├── lossless-raster │ └── README.md ├── lossy-visual-image │ └── README.md └── lossy-numerical-raster │ └── README.md ├── LICENSE └── README.md /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /assets/README.md: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /docs/data-types.md: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /docs/use-cases.md: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /scripts/README.md: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /docs/best-practices.md: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /docs/tools-and-libraries.md: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /docs/compression-techniques.md: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /examples/lossless-raster/README.md: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /examples/lossy-visual-image/README.md: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /examples/lossy-numerical-raster/README.md: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Creative Commons Legal Code 2 | 3 | CC0 1.0 Universal 4 | 5 | CREATIVE COMMONS CORPORATION IS NOT A LAW FIRM AND DOES NOT PROVIDE 6 | LEGAL SERVICES. DISTRIBUTION OF THIS DOCUMENT DOES NOT CREATE AN 7 | ATTORNEY-CLIENT RELATIONSHIP. CREATIVE COMMONS PROVIDES THIS 8 | INFORMATION ON AN "AS-IS" BASIS. CREATIVE COMMONS MAKES NO WARRANTIES 9 | REGARDING THE USE OF THIS DOCUMENT OR THE INFORMATION OR WORKS 10 | PROVIDED HEREUNDER, AND DISCLAIMS LIABILITY FOR DAMAGES RESULTING FROM 11 | THE USE OF THIS DOCUMENT OR THE INFORMATION OR WORKS PROVIDED 12 | HEREUNDER. 13 | 14 | Statement of Purpose 15 | 16 | The laws of most jurisdictions throughout the world automatically confer 17 | exclusive Copyright and Related Rights (defined below) upon the creator 18 | and subsequent owner(s) (each and all, an "owner") of an original work of 19 | authorship and/or a database (each, a "Work"). 20 | 21 | Certain owners wish to permanently relinquish those rights to a Work for 22 | the purpose of contributing to a commons of creative, cultural and 23 | scientific works ("Commons") that the public can reliably and without fear 24 | of later claims of infringement build upon, modify, incorporate in other 25 | works, reuse and redistribute as freely as possible in any form whatsoever 26 | and for any purposes, including without limitation commercial purposes. 27 | These owners may contribute to the Commons to promote the ideal of a free 28 | culture and the further production of creative, cultural and scientific 29 | works, or to gain reputation or greater distribution for their Work in 30 | part through the use and efforts of others. 31 | 32 | For these and/or other purposes and motivations, and without any 33 | expectation of additional consideration or compensation, the person 34 | associating CC0 with a Work (the "Affirmer"), to the extent that he or she 35 | is an owner of Copyright and Related Rights in the Work, voluntarily 36 | elects to apply CC0 to the Work and publicly distribute the Work under its 37 | terms, with knowledge of his or her Copyright and Related Rights in the 38 | Work and the meaning and intended legal effect of CC0 on those rights. 39 | 40 | 1. Copyright and Related Rights. A Work made available under CC0 may be 41 | protected by copyright and related or neighboring rights ("Copyright and 42 | Related Rights"). Copyright and Related Rights include, but are not 43 | limited to, the following: 44 | 45 | i. the right to reproduce, adapt, distribute, perform, display, 46 | communicate, and translate a Work; 47 | ii. moral rights retained by the original author(s) and/or performer(s); 48 | iii. publicity and privacy rights pertaining to a person's image or 49 | likeness depicted in a Work; 50 | iv. rights protecting against unfair competition in regards to a Work, 51 | subject to the limitations in paragraph 4(a), below; 52 | v. rights protecting the extraction, dissemination, use and reuse of data 53 | in a Work; 54 | vi. database rights (such as those arising under Directive 96/9/EC of the 55 | European Parliament and of the Council of 11 March 1996 on the legal 56 | protection of databases, and under any national implementation 57 | thereof, including any amended or successor version of such 58 | directive); and 59 | vii. other similar, equivalent or corresponding rights throughout the 60 | world based on applicable law or treaty, and any national 61 | implementations thereof. 62 | 63 | 2. Waiver. To the greatest extent permitted by, but not in contravention 64 | of, applicable law, Affirmer hereby overtly, fully, permanently, 65 | irrevocably and unconditionally waives, abandons, and surrenders all of 66 | Affirmer's Copyright and Related Rights and associated claims and causes 67 | of action, whether now known or unknown (including existing as well as 68 | future claims and causes of action), in the Work (i) in all territories 69 | worldwide, (ii) for the maximum duration provided by applicable law or 70 | treaty (including future time extensions), (iii) in any current or future 71 | medium and for any number of copies, and (iv) for any purpose whatsoever, 72 | including without limitation commercial, advertising or promotional 73 | purposes (the "Waiver"). Affirmer makes the Waiver for the benefit of each 74 | member of the public at large and to the detriment of Affirmer's heirs and 75 | successors, fully intending that such Waiver shall not be subject to 76 | revocation, rescission, cancellation, termination, or any other legal or 77 | equitable action to disrupt the quiet enjoyment of the Work by the public 78 | as contemplated by Affirmer's express Statement of Purpose. 79 | 80 | 3. Public License Fallback. Should any part of the Waiver for any reason 81 | be judged legally invalid or ineffective under applicable law, then the 82 | Waiver shall be preserved to the maximum extent permitted taking into 83 | account Affirmer's express Statement of Purpose. In addition, to the 84 | extent the Waiver is so judged Affirmer hereby grants to each affected 85 | person a royalty-free, non transferable, non sublicensable, non exclusive, 86 | irrevocable and unconditional license to exercise Affirmer's Copyright and 87 | Related Rights in the Work (i) in all territories worldwide, (ii) for the 88 | maximum duration provided by applicable law or treaty (including future 89 | time extensions), (iii) in any current or future medium and for any number 90 | of copies, and (iv) for any purpose whatsoever, including without 91 | limitation commercial, advertising or promotional purposes (the 92 | "License"). The License shall be deemed effective as of the date CC0 was 93 | applied by Affirmer to the Work. Should any part of the License for any 94 | reason be judged legally invalid or ineffective under applicable law, such 95 | partial invalidity or ineffectiveness shall not invalidate the remainder 96 | of the License, and in such case Affirmer hereby affirms that he or she 97 | will not (i) exercise any of his or her remaining Copyright and Related 98 | Rights in the Work or (ii) assert any associated claims and causes of 99 | action with respect to the Work, in either case contrary to Affirmer's 100 | express Statement of Purpose. 101 | 102 | 4. Limitations and Disclaimers. 103 | 104 | a. No trademark or patent rights held by Affirmer are waived, abandoned, 105 | surrendered, licensed or otherwise affected by this document. 106 | b. Affirmer offers the Work as-is and makes no representations or 107 | warranties of any kind concerning the Work, express, implied, 108 | statutory or otherwise, including without limitation warranties of 109 | title, merchantability, fitness for a particular purpose, non 110 | infringement, or the absence of latent or other defects, accuracy, or 111 | the present or absence of errors, whether or not discoverable, all to 112 | the greatest extent permissible under applicable law. 113 | c. Affirmer disclaims responsibility for clearing rights of other persons 114 | that may apply to the Work or any use thereof, including without 115 | limitation any person's Copyright and Related Rights in the Work. 116 | Further, Affirmer disclaims responsibility for obtaining any necessary 117 | consents, permissions or other rights required for any use of the 118 | Work. 119 | d. Affirmer understands and acknowledges that Creative Commons is not a 120 | party to this document and has no duty or obligation with respect to 121 | this CC0 or use of the Work. 122 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Best Practices for Cloud-Optimized GeoTIFFs (COGs) for the use in Swiss Geospatial Infastructures 2 | 3 | Our goal is to establish and share optimal command line snippets for creating [Cloud-Optimized GeoTIFFs (COGs)](https://cogeo.org/) within the Swiss geospatial context (aka EPSG 2056 users) via a decision tree for various use cases. 4 | COGs are standard GeoTIFF files optimized for cloud environments, allowing for efficient access to specific parts of large datasets without downloading entire files. 5 | 6 | *Pull Requests Accepted:* This Best Practice Guide is fully community driven. If something is not right, or if you want to contribute a use case, add your tutorial to the implementation list or tell ~~Switzerland~~ the world about the awesome gdal / r snippets you are making available then please [contribute](#contributing). 7 | 8 | # Choose your case 9 | ```mermaid 10 | graph TD 11 | start[*Start*: Generate
Cloud Optimized GeoTIFF] --> purpose{What's the main purpose?} 12 | 13 | purpose -->|View Data| view{Data Type?} 14 | purpose -->|Analyze Data| analyze{limited loss acceptable? e.g. cm instead of µm} 15 | 16 | view -->|8-bit
RGB/RGBa/grey| view8{Transparency Required?} 17 | view -->|16-bit
RGB/RGBa/grey| view16{Transparency Required?} 18 | view -->|Multi-Band| analyze 19 | 20 | view8 -->|Yes| jpeg-mask[Use JPEG compression
while creating alpha mask on-the-fly
-co COMPRESS=JPEG] 21 | view8 -->|No| jpeg[Use JPEG compression
-co COMPRESS=JPEG] 22 | 23 | view16 -->|No| jpeg[Use JPEG compression
-co COMPRESS=JPEG] 24 | view16 -->|Yes| lerc_lossy[Use LERC_ZSTD compression
-co COMPRESS=LERC_ZSTD] 25 | 26 | analyze -->|Yes| lerc_lossy[Use LERC_ZSTD compression
-co COMPRESS=LERC_ZSTD] 27 | analyze -->|No| lerc_lossless[Use LERC_ZSTD compression
-co COMPRESS=LERC_ZSTD] 28 | 29 | click jpeg "https://github.com/geostandards-ch/cog-best-practices#lossy-visual-image" 30 | click jpeg-mask "https://github.com/geostandards-ch/cog-best-practices#lossy-visual-image-with-transparency" 31 | click lerc_lossy "https://github.com/geostandards-ch/cog-best-practices#lossy-numerical-raster" 32 | click lerc_lossless "https://github.com/geostandards-ch/cog-best-practices#lossless-raster" 33 | ``` 34 | 35 | # Workflows 36 | ## Virtual Raster Tiles (VRT) 37 | When dealing with a large number of input files, it's often more efficient to use a VRT (Virtual Raster) approach. This method allows you to create a virtual dataset that combines multiple raster files without actually merging the data. Here's how you can do this: 38 | 39 | 1. First, set up the options for gdalbuildvrt and gdal_translate: 40 | 41 | ```bash 42 | gdalbuildvrt_options=( 43 | --config NUM_THREADS ALL_CPUS 44 | --config GDAL_DISABLE_READDIR_ON_OPEN EMPTY_DIR 45 | # Add any other gdalbuildvrt options here 46 | ) 47 | 48 | gdal_translate_options=( 49 | --config NUM_THREADS ALL_CPUS 50 | --config GDAL_DISABLE_READDIR_ON_OPEN EMPTY_DIR 51 | 52 | # Add any other gdal_translate options from the decision tree here 53 | ) 54 | ``` 55 | 2. Create the VRT file and then translate it to a COG: 56 | 57 | ```bash 58 | echo "Creating VRT and translating to COG..." 59 | gdalbuildvrt "${gdalbuildvrt_options[@]}" -input_file_list "${input_file_list}" "${temp_vrt}" 60 | gdal_translate "${gdal_translate_options[@]}" "${temp_vrt}" "${output_file}" 61 | ``` 62 | It's important to set the `--config GDAL_DISABLE_READDIR_ON_OPEN EMPTY_DIR` option when creating the VRT file. This prevents GDAL from attempting to search for additional files (like world files) for each of the input images, which can significantly speed up the process when dealing with a large number of files. 63 | 64 | This approach is particularly useful when you're working with thousands of input files, as it allows GDAL to efficiently handle the data without trying to load everything into memory at once 65 | 66 | ## Batch conversion 67 | 68 | ### Windows: cmd.exe 69 | 70 | ```cmd 71 | FOR %f IN ("C:\path\to\folder\*.tif") DO ( 72 | gdal_translate -of COG [... params ...] %f C:\path\to\output\%~nf.tif 73 | ) 74 | ``` 75 | 76 | ### Linux: bash 77 | 78 | ```bash 79 | for f in /path/to/folder/*.tif 80 | do gdal_translate -of COG [... params ...] $f /path/to/output/$(basename -s .tif $f).tif 81 | done 82 | ``` 83 | # Raster 84 | Apply the optimization steps according to the use case and create Cloud Optimized GeoTIFF [COG](https://cogeo.org) using a recent version of [gdal](https://gdal.org). Windows users may use the OSGeo4WShell provided by [QGIS](https://qgis.org). 85 | 86 | ## lossless raster 87 | 88 | Apply optimization steps for raster data 89 | 1. prepare input data at a reasonable precision (i.e. `cm` instead of `µm` or `nm` i.e. in the data source) 90 | 2. compress your data using 91 | ``` 92 | gdal_translate -a_srs EPSG:2056 -of COG -co COMPRESS=LERC_ZSTD -co LEVEL=22 -co NUM_THREADS=ALL_CPUS -co BIGTIFF=YES -co STATISTICS=YES -tr -r Cubic -a_nodata -ot 93 | ``` 94 | while 95 | - choosing the appropriate scale and offset for the data if feasible (i.e. scale `cm` instead of `m` using [-a_scale](https://gdal.org/en/stable/programs/gdal_translate.html#cmdoption-gdal_translate-a_scale)) and / or offset (using [-a_offset](https://gdal.org/en/stable/programs/gdal_translate.html#cmdoption-gdal_translate-a_offset)) to better fit the numbers 96 | - rescaling the data on the fly if not preprocessed already using [-scale](https://gdal.org/en/stable/programs/gdal_translate.html#cmdoption-gdal_translate-scale) 97 | - choosing to the most appropriate data type ([Byte/Int8/Int16/UInt16/UInt32/Int32/Float32](https://gdal.org/en/stable/programs/gdal_translate.html#cmdoption-gdal_translate-ot)) using the least amount of bits possible. 98 | 99 | #### _Notes on `LERC_ZSTD` (lossy/lossless) compression_ 100 | - _`MAX_Z_ERROR` defaults to `0` (lossless)_ 101 | - _supported since [GDAL](https://gdal.org) v2.4+_ thus also in QGIS, etc. 102 | - _supported by [geotiff.js](https://geotiffjs.github.io/) v2.1+_ thus also in openlayers, etc. 103 | - _has been [developed and published by ESRI](https://github.com/esri/lerc/) and is supported in their products_ 104 | - _supported in STAC-browser v3.4_ 105 | 106 | ## lossy numerical raster 107 | 108 | 1. compress your data using 109 | ``` 110 | gdal_translate -a_srs EPSG:2056 -of COG -co COMPRESS=LERC_ZSTD -co LEVEL=22 -co NUM_THREADS=ALL_CPUS -co BIGTIFF=YES -co STATISTICS=YES -co MAX_Z_ERROR= -tr -r Cubic -a_nodata -ot 111 | ``` 112 | while 113 | - choosing a [threshold](https://gdal.org/en/stable/drivers/raster/cog.html#general-creation-options:~:text=MAX_Z_ERROR) (limited error tolerance i.e. `0.01` for `cm`) for better lossy compression ratio 114 | 115 | ## lossy visual image 116 | 117 | 1. compress your data using 118 | ``` 119 | gdal_translate -a_srs EPSG:2056 -of COG -co COMPRESS=JPEG -co NUM_THREADS=ALL_CPUS -co BIGTIFF=YES -co STATISTICS=YES -co QUALITY=70 -tr -r Cubic 120 | ``` 121 | 122 | ## lossy visual image with transparency 123 | 124 | 1. compress your data using 125 | ``` 126 | gdalwarp -s_srs EPSG:2056 -of COG -co COMPRESS=JPEG -co NUM_THREADS=ALL_CPUS -co BIGTIFF=YES -co STATISTICS=YES -co QUALITY=70 -tr -r Cubic -multi -wo NUM_THREADS=ALL_CPUS -dstalpha -srcnodata 127 | ``` 128 | while 129 | - using [`-dstalpha`](https://gdal.org/en/stable/programs/gdalwarp.html#cmdoption-gdalwarp-dstalpha) 130 | - setting [`-srcnodata`](https://gdal.org/en/stable/programs/gdalwarp.html#cmdoption-gdalwarp-srcnodata) to the nodata value per band as eg `"0 0 0"` see [explanation below](#notes-on-alpha-channel) 131 | - or if the source data already has an alpha band: 132 | - setting [`-srcalpha`](https://gdal.org/en/stable/programs/gdalwarp.html#cmdoption-gdalwarp-srcalpha) 133 | 134 | result: 135 | - grey+alpha or RGB+alpha image representing the data 136 | 137 | #### _Notes on alpha channel_ 138 | - transparency is stored in the extra alpha channel 139 | - it can be created on the fly from _nodata_ at the source (i.e. [`-srcnodata "0 0 0"`](https://gdal.org/en/stable/programs/gdalwarp.html#cmdoption-gdalwarp-srcnodata) for black in RGB source image) 140 | - unused input bands (NIR, etc.) may be dropped (i.e. [`-nosrcalpha`](https://gdal.org/en/stable/programs/gdalwarp.html#cmdoption-gdalwarp-nosrcalpha) [`-srcband 2 -srcband 3 -srcband 4`](https://gdal.org/en/stable/programs/gdalwarp.html#cmdoption-gdalwarp-srcband) to generate RGB from NRGB) 141 | - it can also be created by clipping using a [`cutline`](https://gdal.org/en/stable/programs/gdalwarp.html#cmdoption-gdalwarp-cutline) (i.e. polygon from a file) 142 | - even clip offsetting is possible (see [`-cblend`](https://gdal.org/en/stable/programs/gdalwarp.html#cmdoption-gdalwarp-cblend)) 143 | 144 | ## Contributing 145 | 146 | We strongly encourage and welcome contributions from all members / enthusiasts of the cloud native geospatial community! Your expertise and experiences are invaluable in refining these best practices and ensuring they meet the unique needs of our diverse geographical and technological landscape. 147 | 148 | Here's how you can contribute: 149 | 150 | 1. **Share Your Experience**: If you've implemented COGs in your projects, we'd love to hear about your approach and lessons learned. 151 | 2. **Suggest Improvements**: Found a way to optimize COG creation or usage? Submit a pull request with your suggestions. 152 | 3. **Report Issues**: If you encounter problems or have questions, please open an issue. It helps everyone learn and improve. 153 | 4. **Review and Discuss**: Participate in discussions on open pull requests and issues. Your perspective matters! 154 | 155 | To contribute: 156 | 1. Fork this repository 157 | 2. Create a new branch for your changes 158 | 3. Make your changes and commit them to the 'dev' branch 159 | 4. Submit a pull request with a clear description of your modifications 160 | 161 | Remember, no contribution is too small. Whether it's fixing a typo, adding a use case, or proposing a new best practice, every input helps build a stronger, more knowledgeable Swiss geospatial community. 162 | 163 | Let's work together to optimize our geospatial workflows and make Swiss geographic data more accessible and efficient for everyone! 164 | 165 | ## References 166 | 167 | For further information and more in-depth explanations of Cloud Optimized GeoTIFFs in context with gdal and related topics, please see the following resources: 168 | 169 | * GDAL COG Driver: [https://gdal.org/drivers/raster/cog.html](https://gdal.org/drivers/raster/cog.html) 170 | * gdal_translate: [https://gdal.org/en/stable/programs/gdal_translate.html](https://gdal.org/en/stable/programs/gdal_translate.html) 171 | * GDAL Cloud Optimized GeoTIFF Wiki: [https://trac.osgeo.org/gdal/wiki/CloudOptimizedGeoTIFF](https://trac.osgeo.org/gdal/wiki/CloudOptimizedGeoTIFF) 172 | * GeoTIFF Compression Optimization Guide: [https://kokoalberti.com/articles/geotiff-compression-optimization-guide](https://kokoalberti.com/articles/geotiff-compression-optimization-guide) 173 | * COG Recipes: [https://github.com/Doctor-Who/cog-recipes?tab=readme-ov-file](https://github.com/Doctor-Who/cog-recipes?tab=readme-ov-file) 174 | * Image Compression: [https://wiesehahn.github.io/posts/image_compression/](https://wiesehahn.github.io/posts/image_compression/) 175 | --------------------------------------------------------------------------------