├── .github ├── build-script │ ├── create_readme.py │ └── requirements.txt └── workflows │ └── build.yml ├── HOWTO.md ├── README.md ├── README.md.jinja └── env.yml /.github/build-script/create_readme.py: -------------------------------------------------------------------------------- 1 | from pathlib import Path 2 | from datetime import datetime 3 | from jinja2 import Template 4 | import logging 5 | import requests 6 | import sys 7 | 8 | logger = logging.getLogger(__name__) 9 | 10 | def main() -> bool: 11 | logger.info(f"Reading list of repositories") 12 | with requests.get("https://fiboa.org/stac/catalog.json") as response: 13 | catalog = response.json() 14 | data = list(filter(lambda link: link["rel"] == "child", catalog["links"])) 15 | 16 | data.sort(key = lambda x: x["title"]) 17 | count = len(data) 18 | now = datetime.utcnow().strftime("%b %d %Y, %H:%M %Z") 19 | template = Template(Path("./README.md.jinja").read_text()) 20 | 21 | with Path("./README.md") as f: 22 | f.write_text(template.render(data=data, updated=now, count=count)) 23 | 24 | sys.exit(0) 25 | 26 | 27 | if __name__ == "__main__": 28 | main() 29 | -------------------------------------------------------------------------------- /.github/build-script/requirements.txt: -------------------------------------------------------------------------------- 1 | requests 2 | jinja2 -------------------------------------------------------------------------------- /.github/workflows/build.yml: -------------------------------------------------------------------------------- 1 | name: Build README 2 | on: 3 | schedule: 4 | - cron: '0 0 * * *' # at 00:00 everyday 5 | push: 6 | branches: 7 | - main 8 | jobs: 9 | build: 10 | runs-on: ubuntu-latest 11 | steps: 12 | - name: Checkout repo 13 | uses: actions/checkout@v4 14 | - name: Setup Python 15 | uses: actions/setup-python@v5 16 | - name: Install dependencies 17 | run: | 18 | python -m pip install --upgrade pip 19 | pip install -r .github/build-script/requirements.txt 20 | - name: Update README 21 | run: |- 22 | python .github/build-script/create_readme.py 23 | cat README.md 24 | env: 25 | GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} 26 | - name: Commit changes 27 | run: | 28 | git config user.name "fiboa CI" 29 | git config user.email "actions@users.noreply.github.com" 30 | git add -A 31 | timestamp=$(date -u) 32 | git commit -m ":robot: Automated update README: ${timestamp} [skip ci]" || exit 0 33 | git push -------------------------------------------------------------------------------- /HOWTO.md: -------------------------------------------------------------------------------- 1 | # How to create a cloud-native fiboa dataset using fiboa CLI 2 | 3 | **WORK IN PROGRESS** 4 | 5 | ## Once when starting with fiboa 6 | 7 | The following steps you usually only need to do once: 8 | 9 | 1. Install Python 3.9 or later, GDAL 3.8 or later, [tippecanoe](https://github.com/felt/tippecanoe), and AWS CLI 10 | - If you have trouble installing Python, GDAL, etc., consider using the conda environment. 11 | Install Anaconda and run the following commands using the env.yml file provided in this repository: 12 | - `conda env create --file="https://raw.githubusercontent.com/fiboa/data/refs/heads/main/env.yml"` 13 | - `conda activate fiboa` 14 | 2. Clone the [fiboa CLI repository](https://github.com/fiboa/cli): 15 | `git clone https://github.com/fiboa/cli` 16 | and `cd cli` to switch into the new folder. 17 | 3. Install the dependencies of the fiboa cli repo and the CLI itself: 18 | `pip install -e .` 19 | 4. Check whether the CLI works: `fiboa --version` 20 | 21 | ## Once when creating a new dataset 22 | 23 | 5. Create a converter using fiboa CLI converters, see the 24 | [template](https://github.com/fiboa/cli/blob/main/fiboa_cli/datasets/template.py). 25 | 6. If the dataset is available under an open license, we also want to create a test assuming the converter is named `xx_yy`: 26 | 1. Install development dependencies: `pip install -r /path/to/requirements.txt` 27 | 2. Create a new folder for the test data: `mkdir tests/data-files/convert/xx_yy` 28 | 3. Create a subset of the dataset: `ogr2ogr tests/data-files/convert/xx_yy/input_file.gpkg -limit 100 /path/to/input_file.gpkg` 29 | 4. Update the test file `tests/test_convert.py` to include your converter 30 | 5. Run the tests: `pytest`. Iterate on 4 and 5 until the tests succeed. 31 | 7. Register at Source Cooperative and email `hello@source.coop` for permission to publish in the fiboa organization. 32 | 8. [Create a new repository](https://source.coop/repositories/new/) in the fiboa organization, e.g. `@fiboa/xx-yy`. 33 | You'll find it at `https://source.coop/fiboa/xx-yy/` 34 | 9. Create a new folder for your data, e.g. data 35 | `mkdir data` 36 | 10. Create a README file at `data/README.md` and a license file at `data/LICENSE.txt` 37 | An example repository with a README etc. can be found here: 38 | 39 | 40 | ## Each time you update the dataset 41 | 42 | 11. Go to the parent folder of the folder that contains your data (e.g. `data`) in CLI 43 | 12. Run the converter, e.g. `xx_yy`: 44 | `fiboa convert xx_yy -o data/xx-yy.parquet -h https://source.coop/fiboa/xx-yy/ --collection` 45 | Make sure there are no errors (usually in red) or warnings (usually in yellow) 46 | 13. Validate the result, e.g. `fiboa validate data/xx-yy.parquet --data` and run the tests `pytest` 47 | 14. Move the collection.json into a stac folder: 48 | `mkdir data/stac` and `mv data/collection.json data/stac` 49 | 15. Update the README file at `data/README.md` 50 | 16. Create PMTiles file: 51 | `ogr2ogr -t_srs EPSG:4326 geo.json data/xx-yy.parquet` 52 | and 53 | `tippecanoe -zg --projection=EPSG:4326 -o data/xx-yy.pmtiles -l xx-yy geo.json --drop-densest-as-needed` 54 | 17. Edit the STAC Collection, update the paths, and everything else that you want to customize. 55 | Also don't forget to add a link to the PMTiles file using the 56 | [corresponding STAC extension](https://github.com/stac-extensions/web-map-links?tab=readme-ov-file#pmtiles). 57 | 18. Create a new API key, at: `https://source.coop/repositories/fiboa/xx-yy/manage` 58 | 19. Set the environment variables as follows (Linux/MacOS): 59 | 60 | ```bash 61 | export AWS_ENDPOINT_URL=https://data.source.coop 62 | export AWS_ACCESS_KEY_ID= 63 | export AWS_SECRET_ACCESS_KEY= 64 | ``` 65 | 66 | Note: Windows users may need to change the commands slightly. Use e.g. 67 | `$env:AWS_ENDPOINT_URL="[us-west-2](https://data.source.coop)"` instead of `export AWS_ENDPOINT_URL=https://data.source.coop`. 68 | 20. Upload to AWS: 69 | `aws s3 sync data s3://fiboa/xx-yy/` 70 | 71 | If you've created and published a STAC Collection for your dataset, make sure to add it to the 72 | STAC catalog that combines all datasets into a single STAC catalog 73 | It will also publish your dataset to the 74 | [fiboa data overview page](https://github.com/fiboa/data/blob/main/README.md). 75 | Create a PR to add your STAC Collection as child link to the following file: 76 | 77 | See also the [README](README.md#add-your-dataset) for an alternative. 78 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | 2 | # fiboa Datasets 3 | 4 | This repository provides you with all information about existing fiboa datasets. 5 | 6 | ## List of datasets 7 | 8 | | Title | License | Provider | 9 | | ----- | ------- | -------- | 10 | | [Austria](https://source.coop/repositories/fiboa/austria/description/) | CC-BY-4.0 | Agrarmarkt Austria | 11 | | [BRP Crop Field Boundaries for The Netherlands (CAP-based)](https://source.coop/repositories/fiboa/nl-crop/description/) | CC0-1.0 | | 12 | | [Denmark Crop Fields (Marker)](https://source.coop/repositories/fiboa/denmark/description/) | CC-0 | | 13 | | [Field boundaries for Belgium Wallonia](https://source.coop/repositories/fiboa/be-wa/description/) | No conditions apply to access and use | | 14 | | [Field boundaries for California Statewide Crop Mapping](https://source.coop/repositories/fiboa/us-ca-scm/description/) | CC-0 | | 15 | | [Field boundaries for Cambodia and Vietnam (AI4SmallFarms)](https://source.coop/repositories/fiboa/ai4sf/description/) | CC-BY-4.0 | | 16 | | [Field boundaries for Catalonia, Spain](https://source.coop/repositories/fiboa/es-cat/description/) | Open Information Use License - Catalonia | Catalonia Department of Agriculture, Livestock, Fisheries and Food | 17 | | [Field boundaries for Croatia](https://source.coop/repositories/fiboa/croatia/description/) | Publicly available data | | 18 | | [Field boundaries for Czech](https://source.coop/repositories/fiboa/czech/description/) | CC-0 | | 19 | | [Field boundaries for Estonia](https://source.coop/repositories/fiboa/estonia-ec/description/) | CC-BY-SA-3.0 | Põllumajanduse Registrite ja Informatsiooni Amet | 20 | | [Field boundaries for Flanders, Belgium](https://source.coop/repositories/fiboa/be-vlg/description/) | Licentie modellicentie-gratis-hergebruik/v1.0 | | 21 | | [Field boundaries for France from Eurocrops (2018)](https://source.coop/repositories/fiboa/france-ec/description/) | CC-BY-SA-4.0 | Institut National de l'Information Géographique et Forestière | 22 | | [Field boundaries for Latvia 2024](https://source.coop/repositories/fiboa/latvia/description/) | CC-BY-4.0 | Lauku atbalsta dienests | 23 | | [Field boundaries for Luxembourg](https://source.coop/repositories/fiboa/luxembourg/description/) | CC-BY-4.0 | Luxembourg ministry of Agriculture | 24 | | [Field boundaries for Mecklenburg-Western Pomerania, Germany](https://source.coop/repositories/fiboa/de-mv/description/) | No restrictions apply | | 25 | | [Field boundaries for Portugal](https://source.coop/repositories/fiboa/portugal/description/) | No conditions apply | | 26 | | [Field boundaries for Saarland, Germany](https://source.coop/repositories/fiboa/de-sl/description/) | cc-by-4.0 | © GDI-SL 2024 | 27 | | [Field boundaries for Saxony, Germany](https://source.coop/repositories/fiboa/de-sax/description/) | dl-de/by-2-0 | Sächsisches Landesamt für Umwelt, Landwirtschaft und Geologie | 28 | | [Field boundaries for Slovakia](https://source.coop/repositories/fiboa/slovakia/description/) | Publicly available data | | 29 | | [Field boundaries for Slovenia](https://source.coop/repositories/fiboa/slovenia/description/) | Publicly available data | | 30 | | [Field boundaries for Slovenia from EuroCrops (2021)](https://source.coop/repositories/fiboa/slovenia-ec/description/) | CC-BY-SA-4.0 | Ministrstvo za kmetijstvo, gozdarstvo in prehrano | 31 | | [Field boundaries for Spain Andalucia](https://source.coop/repositories/fiboa/es-cl/description/) | CC-SA-BY-ND | ©Junta de Andalucía | 32 | | [Field boundaries for Spain Aragon](https://source.coop/repositories/fiboa/es-ar/description/) | CC-BY-4.0 | (c) Gobierno de Aragon | 33 | | [Field boundaries for Spain Balearic Islands](https://source.coop/repositories/fiboa/es-pm/description/) | CC-BY-4.0 | Govern de les Illes Balears | 34 | | [Field boundaries for Spain Basque Country](https://source.coop/repositories/fiboa/es-pv/description/) | CC-BY-4.0 | Basque Government / Gobierno Vasco | 35 | | [Field boundaries for Spain Canary Islands](https://source.coop/repositories/fiboa/es-cn/description/) | CC-BY-4.0 | Gobierno de Canarias | 36 | | [Field boundaries for Spain Cantabria](https://source.coop/repositories/fiboa/es-cb/description/) | CC-BY-NC-4.0 | ©Government of Cantabria. Free information available at https://mapas.cantabria.es | 37 | | [Field boundaries for Spain Castilla y León](https://source.coop/repositories/fiboa/es-cl/description/) | Non Commercial | Commercial exploitation is prohibited | 38 | | [Field boundaries for Spain Castilla-La Mancha](https://source.coop/repositories/fiboa/es-cm/description/) | CC-BY-SA-4.0 | Unidad de Cartografía. Secretaría General. Consejería de Agricultura , Ganadería y Desarrollo Rural. | 39 | | [Field boundaries for Spain Extremadura](https://source.coop/repositories/fiboa/es-ex/description/) | CC-BY-4.0 | Junta de Extremadura | 40 | | [Field boundaries for Spain Galicia](https://source.coop/repositories/fiboa/es-ga/description/) | CC-BY-4.0 | Información procedente do FOGGA | 41 | | [Field boundaries for Spain Madrid](https://source.coop/repositories/fiboa/es-md/description/) | CC-0 | Comunidad de Madrid | 42 | | [Field boundaries for Spain Navarra](https://source.coop/repositories/fiboa/es-nc/description/) | CC-BY-4.0 | Comunidad Foral de Navarra | 43 | | [Field boundaries for Spain Valencia](https://source.coop/repositories/fiboa/es-vc/description/) | CC-BY-4.0 | 1403_2024PAC0050 CC BY 4.0 © Institut Cartogràfic Valencià, Generalitat | 44 | | [Field boundaries for Sweden](https://source.coop/repositories/fiboa/sweden/description/) | CC0-1.0 | Jordbruksverket | 45 | | [Field boundaries for Switzerland](https://source.coop/repositories/fiboa/switzerland/description/) | Open BY | | 46 | | [Field boundaries for The Netherlands](https://source.coop/repositories/fiboa/nl-ref/description/) | CC0-1.0 | | 47 | | [Field boundaries for ireland](https://source.coop/repositories/fiboa/ireland/description/) | CC-BY-4.0 | Ireland Department of Agriculture, Food and the Marine | 48 | | [Field boundaries for the west of Bahia state, Brazil](https://source.coop/repositories/fiboa/br-ba-lem/description/) | CC-BY-4.0 | | 49 | | [Fields of the World](https://source.coop/repositories/kerner-lab/fields-of-the-world/description/) | various | | 50 | | [Finnish Crop Fields (Maatalousmaa)](https://source.coop/repositories/fiboa/finland/description/) | CC-BY-4.0 | Finnish Food Authority | 51 | | [Germany, Berlin / Brandenburg](https://source.coop/repositories/fiboa/de-bb/description/) | dl-de/by-2-0 | Land Brandenburg | 52 | | [Germany, Lower Saxony](https://source.coop/repositories/fiboa/de-nds/description/) | dl-de/by-2-0 | Land Niedersachsen | 53 | | [Germany, North Rhine-Westphalia](https://source.coop/repositories/fiboa/de-nrw/description/) | dl-de/by-2-0 | Land Nordrhein-Westfalen / Open.NRW | 54 | | [Germany, Schleswig-Holstein](https://source.coop/repositories/fiboa/de-sh/description/) | dl-de/zero-2-0 | Land Schleswig-Holstein | 55 | | [Germany, Thuringia](https://source.coop/repositories/fiboa/de-th/description/) | dl-de/by-2-0 | Thüringer Landesamt für Landwirtschaft und Ländlichen Raum | 56 | | [Japan Fude Parcels](https://source.coop/repositories/fiboa/japan/description/) | CC-BY-4.0 | Fude Polygon Data (2021-2024). Japanese Ministry of Agriculture, Forestry and Fisheries. Processed by Pacific Spatial Solutions, Inc | 57 | | [Lacuna labels](https://source.coop/repositories/fiboa/lacunalabels/description/) | Participant license agreement for the NICFI contract | | 58 | | [New Zealand National irrigated land](https://source.coop/repositories/fiboa/newzealand/description/) | CC-BY-4.0 | New Zealand Ministry for the Environment | 59 | | [UKFields](https://zenodo.org/records/11110206) | CC-BY-4.0 | Bancroft S, Wilkins J | 60 | | [USDA Crop Sequence Boundaries](https://source.coop/repositories/fiboa/us-usda-cropland/description/) | Freely available at https://www.nass.usda.gov/Research_and_Science/Crop-Sequence-Boundaries | | 61 | 62 | * **Last updated:** Jun 06 2025, 01:42 63 | * **Count:** 51 64 | 65 | ## Add your dataset 66 | 67 | If you want, follow the [How To](HOWTO.md) to create a new dataset using fiboa CLI. 68 | 69 | Once your dataset is ready, you have two options to add your dataset: 70 | 1. If it has a STAC Collection for it, edit [this file](https://github.com/fiboa/fiboa.github.io/edit/main/stac/catalog.json), add your Collection as a child to the STAC catalog, and lastly create a PR. 71 | 2. In all other cases, edit [this file](https://github.com/fiboa/data/edit/main/README.md.jinja), add your dataset to the table (after the `endfor %}`), and lastly create a PR. 72 | 73 | Once the PR is merged your data will be listed here shortly after (it's updated every night). -------------------------------------------------------------------------------- /README.md.jinja: -------------------------------------------------------------------------------- 1 | 2 | # fiboa Datasets 3 | 4 | This repository provides you with all information about existing fiboa datasets. 5 | 6 | ## List of datasets 7 | 8 | | Title | License | Provider | 9 | | ----- | ------- | -------- | 10 | {%- for d in data %} 11 | | [{{d.title}}]({{d.source}}) | {{d.license}} | {{ d.attribution }} | 12 | {%- endfor %} 13 | 14 | * **Last updated:** {{updated}} 15 | * **Count:** {{count}} 16 | 17 | ## Add your dataset 18 | 19 | If you want, follow the [How To](HOWTO.md) to create a new dataset using fiboa CLI. 20 | 21 | Once your dataset is ready, you have two options to add your dataset: 22 | 1. If it has a STAC Collection for it, edit [this file](https://github.com/fiboa/fiboa.github.io/edit/main/stac/catalog.json), add your Collection as a child to the STAC catalog, and lastly create a PR. 23 | 2. In all other cases, edit [this file](https://github.com/fiboa/data/edit/main/README.md.jinja), add your dataset to the table (after the `endfor %}`), and lastly create a PR. 24 | 25 | Once the PR is merged your data will be listed here shortly after (it's updated every night). 26 | -------------------------------------------------------------------------------- /env.yml: -------------------------------------------------------------------------------- 1 | name: fiboa 2 | channels: 3 | - conda-forge 4 | dependencies: 5 | - python=3.10 6 | - pip 7 | - gdal 8 | - libgdal-arrow-parquet 9 | - tippecanoe 10 | - awscli 11 | --------------------------------------------------------------------------------