├── csvz_logo.png ├── contributing.md ├── code-of-conduct.md ├── license ├── cc-zero.svg └── README.md /csvz_logo.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/secretGeek/csvz/HEAD/csvz_logo.png -------------------------------------------------------------------------------- /contributing.md: -------------------------------------------------------------------------------- 1 | # Contribution Guidelines 2 | 3 | Please note that this project is released with a [Contributor Code of Conduct](code-of-conduct.md). By participating in this project you agree to abide by its terms. 4 | 5 | ## Adding something to this repository 6 | 7 | If you have something to contribute to this repository, this is how you do it. 8 | 9 | You'll need a [GitHub account](https://github.com/join)! 10 | 11 | 1. Access the repos GitHub page. For example: https://github.com/secretgeek/csvz 12 | 2. Click on the `readme.md` file: ![Step 2 Click on Readme.md](https://cloud.githubusercontent.com/assets/170270/9402920/53a7e3ea-480c-11e5-9d81-aecf64be55eb.png) 13 | 3. Now click on the edit icon. ![Step 3 - Click on Edit](https://cloud.githubusercontent.com/assets/170270/9402927/6506af22-480c-11e5-8c18-7ea823530099.png) 14 | 4. You can start editing the text of the file in the in-browser editor. Make sure you follow guidelines above. You can use [GitHub Flavored Markdown](https://help.github.com/articles/github-flavored-markdown/). ![Step 4 - Edit the file](https://cloud.githubusercontent.com/assets/170270/9402932/7301c3a0-480c-11e5-81f5-7e343b71674f.png) 15 | 5. Say why you're proposing the changes, and then click on "Propose file change". ![Step 5 - Propose Changes](https://cloud.githubusercontent.com/assets/170270/9402937/7dd0652a-480c-11e5-9138-bd14244593d5.png) 16 | 6. Submit the [pull request](https://help.github.com/articles/using-pull-requests/)! 17 | 18 | ## Content of changes 19 | 20 | Please be consistent with other content in the repository. 21 | 22 | 23 | ## Updating your Pull Request 24 | 25 | Sometimes, a maintainer of repository will ask you to edit your Pull Request before it is included. This is normally due to spelling errors or because your PR didn't match the expected guidelines. 26 | 27 | [Here](https://github.com/RichardLitt/knowledge/blob/master/github/amending-a-commit-guide.md) is a write up on how to change a Pull Request, and the different ways you can do that. -------------------------------------------------------------------------------- /code-of-conduct.md: -------------------------------------------------------------------------------- 1 | # Contributor Covenant Code of Conduct 2 | 3 | ## Our Pledge 4 | 5 | In the interest of fostering an open and welcoming environment, we as 6 | contributors and maintainers pledge to making participation in our project and 7 | our community a harassment-free experience for everyone, regardless of age, body 8 | size, disability, ethnicity, sex characteristics, gender identity and expression, 9 | level of experience, education, socio-economic status, nationality, personal 10 | appearance, race, religion, or sexual identity and orientation. 11 | 12 | ## Our Standards 13 | 14 | Examples of behavior that contributes to creating a positive environment 15 | include: 16 | 17 | * Using welcoming and inclusive language 18 | * Being respectful of differing viewpoints and experiences 19 | * Gracefully accepting constructive criticism 20 | * Focusing on what is best for the community 21 | * Showing empathy towards other community members 22 | 23 | Examples of unacceptable behavior by participants include: 24 | 25 | * The use of sexualized language or imagery and unwelcome sexual attention or 26 | advances 27 | * Trolling, insulting/derogatory comments, and personal or political attacks 28 | * Public or private harassment 29 | * Publishing others' private information, such as a physical or electronic 30 | address, without explicit permission 31 | * Other conduct which could reasonably be considered inappropriate in a 32 | professional setting 33 | 34 | ## Our Responsibilities 35 | 36 | Project maintainers are responsible for clarifying the standards of acceptable 37 | behavior and are expected to take appropriate and fair corrective action in 38 | response to any instances of unacceptable behavior. 39 | 40 | Project maintainers have the right and responsibility to remove, edit, or 41 | reject comments, commits, code, wiki edits, issues, and other contributions 42 | that are not aligned to this Code of Conduct, or to ban temporarily or 43 | permanently any contributor for other behaviors that they deem inappropriate, 44 | threatening, offensive, or harmful. 45 | 46 | ## Scope 47 | 48 | This Code of Conduct applies both within project spaces and in public spaces 49 | when an individual is representing the project or its community. Examples of 50 | representing a project or community include using an official project e-mail 51 | address, posting via an official social media account, or acting as an appointed 52 | representative at an online or offline event. Representation of a project may be 53 | further defined and clarified by project maintainers. 54 | 55 | ## Enforcement 56 | 57 | Instances of abusive, harassing, or otherwise unacceptable behavior may be 58 | reported by contacting the project team at leon.bambrick+csvz at gmail.com. All 59 | complaints will be reviewed and investigated and will result in a response that 60 | is deemed necessary and appropriate to the circumstances. The project team is 61 | obligated to maintain confidentiality with regard to the reporter of an incident. 62 | Further details of specific enforcement policies may be posted separately. 63 | 64 | Project maintainers who do not follow or enforce the Code of Conduct in good 65 | faith may face temporary or permanent repercussions as determined by other 66 | members of the project's leadership. 67 | 68 | ## Attribution 69 | 70 | This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4, 71 | available at https://www.contributor-covenant.org/version/1/4/code-of-conduct.html 72 | 73 | [homepage]: https://www.contributor-covenant.org 74 | 75 | For answers to common questions about this code of conduct, see 76 | https://www.contributor-covenant.org/faq 77 | -------------------------------------------------------------------------------- /license: -------------------------------------------------------------------------------- 1 | CC0 1.0 Universal 2 | 3 | Statement of Purpose 4 | 5 | The laws of most jurisdictions throughout the world automatically confer 6 | exclusive Copyright and Related Rights (defined below) upon the creator and 7 | subsequent owner(s) (each and all, an "owner") of an original work of 8 | authorship and/or a database (each, a "Work"). 9 | 10 | Certain owners wish to permanently relinquish those rights to a Work for the 11 | purpose of contributing to a commons of creative, cultural and scientific 12 | works ("Commons") that the public can reliably and without fear of later 13 | claims of infringement build upon, modify, incorporate in other works, reuse 14 | and redistribute as freely as possible in any form whatsoever and for any 15 | purposes, including without limitation commercial purposes. These owners may 16 | contribute to the Commons to promote the ideal of a free culture and the 17 | further production of creative, cultural and scientific works, or to gain 18 | reputation or greater distribution for their Work in part through the use and 19 | efforts of others. 20 | 21 | For these and/or other purposes and motivations, and without any expectation 22 | of additional consideration or compensation, the person associating CC0 with a 23 | Work (the "Affirmer"), to the extent that he or she is an owner of Copyright 24 | and Related Rights in the Work, voluntarily elects to apply CC0 to the Work 25 | and publicly distribute the Work under its terms, with knowledge of his or her 26 | Copyright and Related Rights in the Work and the meaning and intended legal 27 | effect of CC0 on those rights. 28 | 29 | 1. Copyright and Related Rights. A Work made available under CC0 may be 30 | protected by copyright and related or neighboring rights ("Copyright and 31 | Related Rights"). Copyright and Related Rights include, but are not limited 32 | to, the following: 33 | 34 | i. the right to reproduce, adapt, distribute, perform, display, communicate, 35 | and translate a Work; 36 | 37 | ii. moral rights retained by the original author(s) and/or performer(s); 38 | 39 | iii. publicity and privacy rights pertaining to a person's image or likeness 40 | depicted in a Work; 41 | 42 | iv. rights protecting against unfair competition in regards to a Work, 43 | subject to the limitations in paragraph 4(a), below; 44 | 45 | v. rights protecting the extraction, dissemination, use and reuse of data in 46 | a Work; 47 | 48 | vi. database rights (such as those arising under Directive 96/9/EC of the 49 | European Parliament and of the Council of 11 March 1996 on the legal 50 | protection of databases, and under any national implementation thereof, 51 | including any amended or successor version of such directive); and 52 | 53 | vii. other similar, equivalent or corresponding rights throughout the world 54 | based on applicable law or treaty, and any national implementations thereof. 55 | 56 | 2. Waiver. To the greatest extent permitted by, but not in contravention of, 57 | applicable law, Affirmer hereby overtly, fully, permanently, irrevocably and 58 | unconditionally waives, abandons, and surrenders all of Affirmer's Copyright 59 | and Related Rights and associated claims and causes of action, whether now 60 | known or unknown (including existing as well as future claims and causes of 61 | action), in the Work (i) in all territories worldwide, (ii) for the maximum 62 | duration provided by applicable law or treaty (including future time 63 | extensions), (iii) in any current or future medium and for any number of 64 | copies, and (iv) for any purpose whatsoever, including without limitation 65 | commercial, advertising or promotional purposes (the "Waiver"). Affirmer makes 66 | the Waiver for the benefit of each member of the public at large and to the 67 | detriment of Affirmer's heirs and successors, fully intending that such Waiver 68 | shall not be subject to revocation, rescission, cancellation, termination, or 69 | any other legal or equitable action to disrupt the quiet enjoyment of the Work 70 | by the public as contemplated by Affirmer's express Statement of Purpose. 71 | 72 | 3. Public License Fallback. Should any part of the Waiver for any reason be 73 | judged legally invalid or ineffective under applicable law, then the Waiver 74 | shall be preserved to the maximum extent permitted taking into account 75 | Affirmer's express Statement of Purpose. In addition, to the extent the Waiver 76 | is so judged Affirmer hereby grants to each affected person a royalty-free, 77 | non transferable, non sublicensable, non exclusive, irrevocable and 78 | unconditional license to exercise Affirmer's Copyright and Related Rights in 79 | the Work (i) in all territories worldwide, (ii) for the maximum duration 80 | provided by applicable law or treaty (including future time extensions), (iii) 81 | in any current or future medium and for any number of copies, and (iv) for any 82 | purpose whatsoever, including without limitation commercial, advertising or 83 | promotional purposes (the "License"). The License shall be deemed effective as 84 | of the date CC0 was applied by Affirmer to the Work. Should any part of the 85 | License for any reason be judged legally invalid or ineffective under 86 | applicable law, such partial invalidity or ineffectiveness shall not 87 | invalidate the remainder of the License, and in such case Affirmer hereby 88 | affirms that he or she will not (i) exercise any of his or her remaining 89 | Copyright and Related Rights in the Work or (ii) assert any associated claims 90 | and causes of action with respect to the Work, in either case contrary to 91 | Affirmer's express Statement of Purpose. 92 | 93 | 4. Limitations and Disclaimers. 94 | 95 | a. No trademark or patent rights held by Affirmer are waived, abandoned, 96 | surrendered, licensed or otherwise affected by this document. 97 | 98 | b. Affirmer offers the Work as-is and makes no representations or warranties 99 | of any kind concerning the Work, express, implied, statutory or otherwise, 100 | including without limitation warranties of title, merchantability, fitness 101 | for a particular purpose, non infringement, or the absence of latent or 102 | other defects, accuracy, or the present or absence of errors, whether or not 103 | discoverable, all to the greatest extent permissible under applicable law. 104 | 105 | c. Affirmer disclaims responsibility for clearing rights of other persons 106 | that may apply to the Work or any use thereof, including without limitation 107 | any person's Copyright and Related Rights in the Work. Further, Affirmer 108 | disclaims responsibility for obtaining any necessary consents, permissions 109 | or other rights required for any use of the Work. 110 | 111 | d. Affirmer understands and acknowledges that Creative Commons is not a 112 | party to this document and has no duty or obligation with respect to this 113 | CC0 or use of the Work. 114 | 115 | For more information, please see 116 | -------------------------------------------------------------------------------- /cc-zero.svg: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 6 | 7 | 9 | 10 | 11 | 12 | 13 | 14 | 21 | 25 | 35 | 36 | 37 | 48 | 55 | 67 | 69 | 71 | 72 | 74 | 75 | 78 | 79 | 80 | 86 | 96 | 97 | 98 | 99 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # csvz 2 | 3 | `csvz` is the hot new open database standard that is taking the entire technological world by storm. 4 | 5 | A `csvz` file is literally just a bunch of `csv` files, in a zip file, that has been renamed to have a ".csvz" file extension. 6 | 7 | ----- 8 | 9 | > Are you using `csvz` ? Why not? `csvz` is the brave technology that unites the worlds of data science, sql and no-sql. Is it no-sql's answer to the rdbms? Or is it the rdbms answer to no-sql? You decide. 10 | 11 | ----- 12 | 13 | ## Contents 14 | 15 | - [The csvz specification](#the-csvz-specification) 16 | - [`csvz-0` A csvz file is literally just a bunch of `csv` files, in a zip file with a file name that ends with ".csvz"](#csvz-0-a-csvz-file-is-literally-just-a-bunch-of-csv-files-in-a-zip-file-with-a-file-name-that-ends-with-csvz) 17 | - [`csvz-meta-tables` A csvz file can contain a file called `tables.csv` describing the contents of the file](#csvz-meta-tables-a-csvz-file-can-contain-a-file-called-tablescsv-describing-the-contents-of-the-file) 18 | - [`csvz-meta-columns` A csvz file can contain a file called `columns.csv`](#csvz-meta-columns-a-csvz-file-can-contain-a-file-called-columnscsv) 19 | - [`csvz-meta-relations` A csvz file can contain a file called `relations.csv`](#csvz-meta-relations-a-csvz-file-can-contain-a-file-called-relationscsv) 20 | - [`csvz-meta-csv` A csvz file can contain a file called `csv.csv`](#csvz-meta-csv-A-csvz-file-can-contain-a-file-called-csvcsv) 21 | - [`csvz-meta-per-file` The ability to include individual meta-files per csv file](#csvz-meta-per-file-the-ability-to-include-individual-meta-files-per-csv-file) 22 | - [Suggested specification fragments](#suggested-specification-fragments) 23 | - [A list of `csvz-compliant` Tools and Libraries](#a-list-of-csvz-compliant-tools-and-libraries) 24 | - [Contribute](#contribute) 25 | - [License](#license) 26 | 27 | ----- 28 | 29 | ## The csvz specification 30 | 31 | The `csvz` specification is broken into meaningful fragments. 32 | 33 | Files can call themselves `csvz-compliant` if they only comply with the first fragment of the specification, [`csvz-0`](#csvz-0-a-csvz-file-is-literally-just-a-bunch-of-csv-files-in-a-zip-file-with-a-file-name-that-ends-with-csvz). 34 | 35 | They can also indicate other fragments of the specification that they have implemented, such as [`csvz-meta-tables`](#csvz-meta-tables-a-csvz-file-can-contain-a-file-called-tablescsv-describing-the-contents-of-the-file), [`csv-meta-relations`](#csvz-meta-relations-a-csvz-file-can-contain-a-file-called-relations.csv) etc. 36 | 37 | ----- 38 | 39 | ### `csvz-0` A csvz file is literally just a bunch of `csv` files, in a zip file with a file name that ends with ".csvz" 40 | 41 | A csvz file is compliant with `csvz-0` if it is literally just a bunch of `csv` files, in a zip file, that has been renamed to have a ".csvz" file extension. 42 | 43 | (Note that each fragment has a fragment identifier written at the beginning of the fragment. For example this is `csvz-0` and the next fragment is `csvz-meta-tables`. Fragments are optional, but it is good to know which fragments you do or do not comply with.) 44 | 45 | The `csv` files themselves should be parseable with most csv reading software. 46 | 47 | (Anywhere that this spec refers to "a csv file" it means a file that complies with [`RFC 4180`](http://www.faqs.org/rfcs/rfc4180.html) or a compatible [dialect](https://www.w3.org/TR/2015/REC-tabular-metadata-20151217/#dialect-descriptions) as described by the [CSV on the Web Working Group](http://www.w3.org/2013/csvw/), unless a stricter definition is explicitly given.) 48 | 49 | (Anywhere that `the csvz specification` refers to "this spec" it means `the csvz specification`.) 50 | 51 | ----- 52 | 53 | ### **`csvz-meta-tables`** A `csvz` file can contain a file called `tables.csv` describing the contents of the file 54 | 55 | Metadata about the contents of the csvz file is contained in a directory called "_meta". The file `tables.csv`, if present, is inside this directory. 56 | 57 | (Assume that the csvz reserves the right to create other .csv files under the _meta folder, and to create more folders under it. Details appear in subsequent spec fragments.) 58 | 59 | The file `tables.csv` contains metadata about all of the csv files included in the `csvz` file. 60 | 61 | (The file `tables.csv` is a csv file.) 62 | 63 | (Anywhere that this spec refers to a file with a name that ends with ".csv" it means the file is a "csv file", as described in [`csvz-0`](#csvz-0-a-csvz-file-is-literally-just-a-bunch-of-csv-files-in-a-zip-file-with-a-file-name-that-ends-with-csvz).) 64 | 65 | The file `tables.csv` meets the following description: 66 | 67 | - There is a header row naming the columns in this file 68 | - Each data row describes a different csv file within this `csvz` file 69 | - The columns must include a column called "filename" 70 | - There may be more columns. 71 | - Here are some suggestions: 72 | - `bytes` - the size of the file in bytes 73 | - `rows` - the number of rows in the file 74 | - `columns` - the number of columns in the file 75 | - `description` - a description of the file 76 | - `published` - the date the data in the file was first published 77 | - `source` - information about the source of the data in the file 78 | - `has-column-names` - a `true/false` value indicating if the file has a header row containing column names 79 | - `skip-rows` - How many rows need to be skipped, before the data begins? (Rarely need to specify this, but when you need it, you need it!) 80 | - (todo: where information in table.csv conflicts with information in `csv.csv`, then `tables.csv` has precedence over `csv.csv`, for the file it describes. For example `csv.csv` may indicate that all files have header rows, but a specific file may not, and this would be indicated in `tables.csv`) 81 | - The file `tables.csv` may also describe itself. See [Russell](http://wiki.secretgeek.net/paradox#bertrand-russell-making-life-impossible-for-frege-since-1902). Note that `bytes` (for example) might cause a paradox. 82 | 83 | (The word "must" is used for parts of the specification that are required for a file or tool to claim compliance with the standards described in this spec. The word "may" is used for parts which are not required; Optional sections may be covered in more detail, as required elements in a subsequent fragment of this spec.) 84 | 85 | (Whenever suggestions are provided, they are not required for conformance with the current spec fragment. These suggestion may be described more fully in later spec fragments, in which they may be required.) 86 | 87 | (Expectations around the encoding of `true/false` values, and other fundamental `data-types`, are not currently defined.) 88 | 89 | ----- 90 | 91 | ### `csvz-meta-columns` A csvz file can contain a file called `columns.csv` 92 | 93 | Metadata about the contents of the csvz file is contained in a directory called "_meta". The file `columns.csv`, if present, is inside this directory. 94 | 95 | The file `columns.csv` contains metadata about all of the columns in all of the csv files included in the `csvz` file. 96 | 97 | The file `columns.csv` meets the following description: 98 | 99 | - There is a header row naming the columns in this file 100 | - Each data row describes a different column in a different file 101 | - The columns must include a column called "filename" and a column called "column". 102 | - It is expected that the columns "filename" and "column" are unique. 103 | - If the columns "filename" and "column" are not unique, then any meta data about that file may not be correctly interpreted. This may cause difficulties 104 | - There should be more columns than just the "filename" and "column" column. Some suggestions: 105 | - `data-type` - the type of the column. (Data-types are not described in this spec fragment, and will be covered in later spec fragments.) 106 | - `nullable` - a `true/false` value indicating if the column can be null 107 | - `max-length` - a nullable column, that describes the maximum length of the column, in cases where the data-type supports a maximum length 108 | - `unique` - a `true/false` value indicating if the values in the column should be unique 109 | - `primary-key` - a `true/false` value indicating if the column can serve as (part or whole of) the primary key of the table. 110 | - `description` - a description of the column 111 | - `units` - a nullable name description of the unit of measure 112 | - `ordinal` - the order in which the columns have been written to the file. In cases where there is no header row, or where columns are re-ordered, this can be helpful. 113 | - `published` - the date the data in the file was first published 114 | - `source` - information about the source of the data in the file 115 | 116 | (The word "should" is used for parts of the specification that are not required, but which will lead to difficulty for users of the data or the tools if they are not complied with.) 117 | 118 | ----- 119 | 120 | ### `csvz-meta-relations` A csvz file can contain a file called `relations.csv` 121 | 122 | Metadata about the contents of the csvz file is contained in a directory called "_meta". The file `relations.csv`, if present, is inside this directory. 123 | 124 | The file `relations.csv` contains metadata about all of the relationships between any of the columns in any of the files in the `csvz` file. 125 | 126 | The file `relations.csv` meets the following description: 127 | 128 | - There is a header row naming the columns in this file 129 | - Each data row describes a different foreign key relationship within the `csvz` file. 130 | - The columns must include these columns: 131 | - called "table", "key-column", "foreign-table","foreign-key-column" 132 | - There may also be a column called "key-name". In the case of a composite keys, there would be multiple rows with the same "key-name". 133 | - todo: There may be more columns needed to describe the relationships. 134 | 135 | ----- 136 | 137 | ### `csvz-meta-csv` A csvz file can contain a file called `csv.csv` 138 | 139 | (todo: this section is still very much a draft) 140 | 141 | Metadata about the rules of the csvz file are contained in a directory called "_meta". The file `csv.csv`, if present, is inside this directory. 142 | 143 | The file `csv.csv` contains metadata about how the csv files in this `csvz` file are formatted, from a general csv standards point of view. 144 | 145 | (Later spec fragments will give exact definitions for the expected columns and supported columns, their possible values and the meanings of those values.) 146 | 147 | But to comply with `csvz-meta-csv` the file `csv.csv` must: 148 | 149 | - Be formatted as [`strict-4180`](https://tools.ietf.org/html/rfc4180) 150 | - Have a header row naming the columns in this file 151 | - Data rows, each of which describes a different and very specific but fundamental aspect of the csv format used by all other csv files in this csvz file. 152 | - Suggested aspects that can be described (and which should be described in subsequent fragments) 153 | - `encoding - what file encoding is used for the csv files (utf-x, BOM, etc.) 154 | - `field separator` - examples comma, tab, semicolon, space, various emoji 155 | - `row separator` - examples CRLF, LF, CR, semicolon, exclamation point, backtick 156 | - `qualifier` - what qualifiers (if any) are used for embedding delimiters. perhaps qualifiers are not used. Can single/double/mixed/other be used? 157 | - `escaping` - Are qualifiers doubled or escaped? (If escaped, escaped with what?) 158 | - `null-values` - how are `null-values` represented? e.g. the literal string `null` with no quotes? or `NULL`, or `nil`? Or are empty strings, unquoted, to be treated as NULLs? 159 | - `has-column-names` - a `true/false` value indicating if every csv file (other than this one) file has a header row containing column names. (Can be over-ridden by a `has-column-names` value in the `tables.csv` file, if present.) 160 | - user defined `data-types` can be handled elsewhere, but a limited number of common fundamental `data-types` could be most expediently described in `csv.csv`, such as: 161 | - date formats - e.g. ISO 8601 162 | - boolean formats 163 | - binary data (hint: base 64 coded) 164 | - integer ranges. 165 | - floats 166 | - Later spec fragments may further describe "sensible defaults" for these things 167 | - Later spec fragments may describe "sensible heuristics" for detecting delimiters/qualifiers/quoting and escaping rules, etc. 168 | 169 | 170 | (todo: See also [csvw dialect descriptions](https://www.w3.org/TR/2015/REC-tabular-metadata-20151217/#dialect-descriptions)) 171 | 172 | ----- 173 | 174 | ### `csvz-meta-per-file` The ability to include individual meta-files per csv file 175 | 176 | This fragment extends all other `csvz-meta-*` fragments. 177 | 178 | Consider an example where a single csv file, `people.csv` inside the csvz follows different standards to the other files. 179 | 180 | It's csv conventions could be described in a file: `_meta/csv/people.csv` and those would be taken to override the conventions in `_meta/csv.csv` 181 | 182 | Similarly, a file can have its own `_meta/tables/{filename}.csv` file, `_meta/columns/{filename}.csv` and `_meta/relations/{filename}.csv`. 183 | 184 | This methods can be assumed to extend for all other `_meta/*.csv` files. 185 | 186 | A `per-file` meta file is assumed to have higher **precedence** than the files directly contained in `_meta/*.csv`. 187 | 188 | For example: if `_meta/columns.csv` decribed the columns of `states.csv` in one way, but `_meta/columns/states.csv` described those columns in another way, all details for `states.csv` in `_meta/columns.csv` should be ignored, and those in `_meta/columns/states.csv` used instead. (i.e. they are not *combined*). 189 | 190 | (Note - combining might be more interesting, useful. Would let you build up/inherit attributes. But would also need a way to "erase" a rule, and I can't think of a way to do that so let's stick with "no combining") 191 | 192 | (Suggestion for authors of Tooling that reads these files: they may want to provide optional debug information that describes where meta data was sourced from, highlighting situations where precedence rules needed to be applied.) 193 | 194 | You can also mix and match `_meta/*.csv` with `per-file` meta information, without loss of meaning. 195 | 196 | For example the table `states.csv` may be described in `_meta/tables.csv` while it's columns may be described in `_meta/columns/states.csv` 197 | 198 | ----- 199 | 200 | ### Unwritten meta fragments 201 | 202 | More `meta-*` spec fragments may be needed to describe other meta files. 203 | 204 | For example: 205 | 206 | - `indexes` - what indexes can/should be built on the tables (if the data) 207 | - `data-types` - what types are used, how are they encoded (e.g. dates: how? binary data base-64 encoded? etc), what ranges exist for numbers etc. 208 | - `user-defined-types` - how can types be extended? 209 | - `schemas` - consider situations where directories are used to describe separate schemas\databases (or other namespacing concepts) 210 | - `directories` - instead of defining schemas (or other namespaces) perhaps the concept of directories could be directly described, a kind of set of routing rules/conventions. in the directories.csv you might in effect say, the directories in the root directory (other than _meta) are to be treated as "server" names. the next level are to be treated as "share" names... or perhaps you will say, "under "/databases" the directory names in there are treated as "database" names, and the names under that are "schema" names. 211 | - `naming` - perhaps you will define naming conventions, e.g. ways to pull data from names, or use names to know which files can be combined into one logical unit later. 212 | - `deltas` - csv files may hold operations on data, instead of data itself, i.e. details of insert,update,delete,(upsert) operations 213 | - `constraints` - what other constraints exist on the data 214 | - `partitions` - consider situations where a single table is split across multiple files, and or a `csvz` itself is split amongst multiple files, including 215 | - `formulas` - are there calculated columns? what form do the calculations take? 216 | 217 | ## A list of `csvz-compliant` Tools and Libraries 218 | 219 | The following tools and libraries are able to read, write or process `.csvz` files. 220 | 221 | | Tool | Actions | Compliance |Description | 222 | |-----------|---------|------------|-----------------------------------------------------------------------------------------------------| 223 | |[Sylvan.Data.CsvZip](https://github.com/MarkPflug/Sylvan.Data.CsvZip) | Create / Read | `csvz-0` `csvz-meta-tables` `csvz-meta-columns` | Library for programatically creating and reading .csvz files | 224 | |[Sylvan.Tools.CsvZip](https://github.com/MarkPflug/Sylvan.Data.CsvZip#sylvantoolscsvzip-tool) | Create | `csvz-0` `csvz-meta-tables` `csvz-meta-columns` | .NET global tool for creating .csvz files from the commandline | 225 | | | | | Packs a set of csv files into a new csvz file, and generates a `tables.csv` and `columns.csv` | 226 | | | | | Converts a `.csvz` file into a `.xlsx` file, that can be opened by Excel. | 227 | | | | | Converts a `.csvz` file into a `.xlsx` file, that can be opened by Excel. | 228 | | | | | Converts a `.xlsx` file into a `.csvz` file (note that not all of Excel's features are respected.) | 229 | | | | | Exports a `sqlite` database into a new `.csvz` file | 230 | | | | | Creates a new `sqlite` database from a `.csvz` file | 231 | | | | | Exports a `mysql` database into a new `.csvz` file | 232 | | | | | Creates a new `PostgreSQL` database from a `.csvz` file | 233 | | | | | Exports a `PostgreSQL` database into a new `.csvz` file | 234 | | | | | Save a JSON file as a series of csv files and _meta files (ready for zipping) | 235 | | | | | Load some or all of an unzipped csvz as a single json object (limited filtering ability) | 236 | | | | | Validates which spec fragments a `csvz` file complies with | 237 | | | | | (More tools...) | 238 | 239 | If you know of a `csvz` compliant tool, or you have created one (hint hint), a pull request is welcome. 240 | 241 | Suggestion: You can **use existing `csvz` or `csv` libraries** to build a new type of connection (e.g. A tool to create/read csvz files from an Oracle database, using existing libraries, would take some Oracle knowledge, and not much else.) 242 | 243 | ----- 244 | 245 | ## Contribute 246 | 247 | To experience the fun of contributing, see [Contributing](contributing.md) 248 | 249 | Contributors definitely includes people who raise issues. **Raising issues is the quickest way to contribute.** Also look for issues marked `good first issue` or `help wanted` 250 | 251 | A community forum for discussion/ideas for implementors and tool builders is much needed, following [issue #14](https://github.com/secretGeek/csvz/issues/14) to find where the community will be built. 252 | 253 | ## License 254 | 255 | [![CC0](cc-zero.svg)](https://creativecommons.org/publicdomain/zero/1.0/) 256 | 257 | To the extent possible under law, [Leon Bambrick](http://secretgeek.net) has waived all copyright and related or neighboring rights to this work. 258 | 259 | ----- 260 | 261 | > Some ideas are too smart to live; other ideas are too dumb to die. 262 | --------------------------------------------------------------------------------