├── LICENSE ├── README.md ├── bigquery ├── .gitignore ├── LICENSE ├── README.md ├── doc │ └── intro.md ├── project.clj └── src │ ├── clojure │ └── gclouj │ │ └── bigquery.clj │ └── java │ └── gclouj │ └── BigQueryOptionsFactory.java ├── datastore ├── .gitignore ├── LICENSE ├── README.md ├── project.clj ├── src │ ├── clojure │ │ └── gclouj │ │ │ └── datastore.clj │ └── java │ │ └── gclouj │ │ └── DatastoreOptionsFactory.java └── test │ └── datastore_test.clj └── storage ├── .gitignore ├── project.clj └── src ├── clojure └── gclouj │ └── storage.clj └── java └── gclouj └── StorageOptionsFactory.java /LICENSE: -------------------------------------------------------------------------------- 1 | Eclipse Public License - v 1.0 2 | 3 | THE ACCOMPANYING PROGRAM IS PROVIDED UNDER THE TERMS OF THIS ECLIPSE PUBLIC 4 | LICENSE ("AGREEMENT"). ANY USE, REPRODUCTION OR DISTRIBUTION OF THE PROGRAM 5 | CONSTITUTES RECIPIENT'S ACCEPTANCE OF THIS AGREEMENT. 6 | 7 | 1. DEFINITIONS 8 | 9 | "Contribution" means: 10 | 11 | a) in the case of the initial Contributor, the initial code and documentation 12 | distributed under this Agreement, and 13 | b) in the case of each subsequent Contributor: 14 | i) changes to the Program, and 15 | ii) additions to the Program; 16 | 17 | where such changes and/or additions to the Program originate from and are 18 | distributed by that particular Contributor. A Contribution 'originates' 19 | from a Contributor if it was added to the Program by such Contributor 20 | itself or anyone acting on such Contributor's behalf. Contributions do not 21 | include additions to the Program which: (i) are separate modules of 22 | software distributed in conjunction with the Program under their own 23 | license agreement, and (ii) are not derivative works of the Program. 24 | 25 | "Contributor" means any person or entity that distributes the Program. 26 | 27 | "Licensed Patents" mean patent claims licensable by a Contributor which are 28 | necessarily infringed by the use or sale of its Contribution alone or when 29 | combined with the Program. 30 | 31 | "Program" means the Contributions distributed in accordance with this 32 | Agreement. 33 | 34 | "Recipient" means anyone who receives the Program under this Agreement, 35 | including all Contributors. 36 | 37 | 2. GRANT OF RIGHTS 38 | a) Subject to the terms of this Agreement, each Contributor hereby grants 39 | Recipient a non-exclusive, worldwide, royalty-free copyright license to 40 | reproduce, prepare derivative works of, publicly display, publicly 41 | perform, distribute and sublicense the Contribution of such Contributor, 42 | if any, and such derivative works, in source code and object code form. 43 | b) Subject to the terms of this Agreement, each Contributor hereby grants 44 | Recipient a non-exclusive, worldwide, royalty-free patent license under 45 | Licensed Patents to make, use, sell, offer to sell, import and otherwise 46 | transfer the Contribution of such Contributor, if any, in source code and 47 | object code form. This patent license shall apply to the combination of 48 | the Contribution and the Program if, at the time the Contribution is 49 | added by the Contributor, such addition of the Contribution causes such 50 | combination to be covered by the Licensed Patents. The patent license 51 | shall not apply to any other combinations which include the Contribution. 52 | No hardware per se is licensed hereunder. 53 | c) Recipient understands that although each Contributor grants the licenses 54 | to its Contributions set forth herein, no assurances are provided by any 55 | Contributor that the Program does not infringe the patent or other 56 | intellectual property rights of any other entity. Each Contributor 57 | disclaims any liability to Recipient for claims brought by any other 58 | entity based on infringement of intellectual property rights or 59 | otherwise. As a condition to exercising the rights and licenses granted 60 | hereunder, each Recipient hereby assumes sole responsibility to secure 61 | any other intellectual property rights needed, if any. For example, if a 62 | third party patent license is required to allow Recipient to distribute 63 | the Program, it is Recipient's responsibility to acquire that license 64 | before distributing the Program. 65 | d) Each Contributor represents that to its knowledge it has sufficient 66 | copyright rights in its Contribution, if any, to grant the copyright 67 | license set forth in this Agreement. 68 | 69 | 3. REQUIREMENTS 70 | 71 | A Contributor may choose to distribute the Program in object code form under 72 | its own license agreement, provided that: 73 | 74 | a) it complies with the terms and conditions of this Agreement; and 75 | b) its license agreement: 76 | i) effectively disclaims on behalf of all Contributors all warranties 77 | and conditions, express and implied, including warranties or 78 | conditions of title and non-infringement, and implied warranties or 79 | conditions of merchantability and fitness for a particular purpose; 80 | ii) effectively excludes on behalf of all Contributors all liability for 81 | damages, including direct, indirect, special, incidental and 82 | consequential damages, such as lost profits; 83 | iii) states that any provisions which differ from this Agreement are 84 | offered by that Contributor alone and not by any other party; and 85 | iv) states that source code for the Program is available from such 86 | Contributor, and informs licensees how to obtain it in a reasonable 87 | manner on or through a medium customarily used for software exchange. 88 | 89 | When the Program is made available in source code form: 90 | 91 | a) it must be made available under this Agreement; and 92 | b) a copy of this Agreement must be included with each copy of the Program. 93 | Contributors may not remove or alter any copyright notices contained 94 | within the Program. 95 | 96 | Each Contributor must identify itself as the originator of its Contribution, 97 | if 98 | any, in a manner that reasonably allows subsequent Recipients to identify the 99 | originator of the Contribution. 100 | 101 | 4. COMMERCIAL DISTRIBUTION 102 | 103 | Commercial distributors of software may accept certain responsibilities with 104 | respect to end users, business partners and the like. While this license is 105 | intended to facilitate the commercial use of the Program, the Contributor who 106 | includes the Program in a commercial product offering should do so in a manner 107 | which does not create potential liability for other Contributors. Therefore, 108 | if a Contributor includes the Program in a commercial product offering, such 109 | Contributor ("Commercial Contributor") hereby agrees to defend and indemnify 110 | every other Contributor ("Indemnified Contributor") against any losses, 111 | damages and costs (collectively "Losses") arising from claims, lawsuits and 112 | other legal actions brought by a third party against the Indemnified 113 | Contributor to the extent caused by the acts or omissions of such Commercial 114 | Contributor in connection with its distribution of the Program in a commercial 115 | product offering. The obligations in this section do not apply to any claims 116 | or Losses relating to any actual or alleged intellectual property 117 | infringement. In order to qualify, an Indemnified Contributor must: 118 | a) promptly notify the Commercial Contributor in writing of such claim, and 119 | b) allow the Commercial Contributor to control, and cooperate with the 120 | Commercial Contributor in, the defense and any related settlement 121 | negotiations. The Indemnified Contributor may participate in any such claim at 122 | its own expense. 123 | 124 | For example, a Contributor might include the Program in a commercial product 125 | offering, Product X. That Contributor is then a Commercial Contributor. If 126 | that Commercial Contributor then makes performance claims, or offers 127 | warranties related to Product X, those performance claims and warranties are 128 | such Commercial Contributor's responsibility alone. Under this section, the 129 | Commercial Contributor would have to defend claims against the other 130 | Contributors related to those performance claims and warranties, and if a 131 | court requires any other Contributor to pay any damages as a result, the 132 | Commercial Contributor must pay those damages. 133 | 134 | 5. NO WARRANTY 135 | 136 | EXCEPT AS EXPRESSLY SET FORTH IN THIS AGREEMENT, THE PROGRAM IS PROVIDED ON AN 137 | "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, EITHER EXPRESS OR 138 | IMPLIED INCLUDING, WITHOUT LIMITATION, ANY WARRANTIES OR CONDITIONS OF TITLE, 139 | NON-INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Each 140 | Recipient is solely responsible for determining the appropriateness of using 141 | and distributing the Program and assumes all risks associated with its 142 | exercise of rights under this Agreement , including but not limited to the 143 | risks and costs of program errors, compliance with applicable laws, damage to 144 | or loss of data, programs or equipment, and unavailability or interruption of 145 | operations. 146 | 147 | 6. DISCLAIMER OF LIABILITY 148 | 149 | EXCEPT AS EXPRESSLY SET FORTH IN THIS AGREEMENT, NEITHER RECIPIENT NOR ANY 150 | CONTRIBUTORS SHALL HAVE ANY LIABILITY FOR ANY DIRECT, INDIRECT, INCIDENTAL, 151 | SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING WITHOUT LIMITATION 152 | LOST PROFITS), HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN 153 | CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) 154 | ARISING IN ANY WAY OUT OF THE USE OR DISTRIBUTION OF THE PROGRAM OR THE 155 | EXERCISE OF ANY RIGHTS GRANTED HEREUNDER, EVEN IF ADVISED OF THE POSSIBILITY 156 | OF SUCH DAMAGES. 157 | 158 | 7. GENERAL 159 | 160 | If any provision of this Agreement is invalid or unenforceable under 161 | applicable law, it shall not affect the validity or enforceability of the 162 | remainder of the terms of this Agreement, and without further action by the 163 | parties hereto, such provision shall be reformed to the minimum extent 164 | necessary to make such provision valid and enforceable. 165 | 166 | If Recipient institutes patent litigation against any entity (including a 167 | cross-claim or counterclaim in a lawsuit) alleging that the Program itself 168 | (excluding combinations of the Program with other software or hardware) 169 | infringes such Recipient's patent(s), then such Recipient's rights granted 170 | under Section 2(b) shall terminate as of the date such litigation is filed. 171 | 172 | All Recipient's rights under this Agreement shall terminate if it fails to 173 | comply with any of the material terms or conditions of this Agreement and does 174 | not cure such failure in a reasonable period of time after becoming aware of 175 | such noncompliance. If all Recipient's rights under this Agreement terminate, 176 | Recipient agrees to cease use and distribution of the Program as soon as 177 | reasonably practicable. However, Recipient's obligations under this Agreement 178 | and any licenses granted by Recipient relating to the Program shall continue 179 | and survive. 180 | 181 | Everyone is permitted to copy and distribute copies of this Agreement, but in 182 | order to avoid inconsistency the Agreement is copyrighted and may only be 183 | modified in the following manner. The Agreement Steward reserves the right to 184 | publish new versions (including revisions) of this Agreement from time to 185 | time. No one other than the Agreement Steward has the right to modify this 186 | Agreement. The Eclipse Foundation is the initial Agreement Steward. The 187 | Eclipse Foundation may assign the responsibility to serve as the Agreement 188 | Steward to a suitable separate entity. Each new version of the Agreement will 189 | be given a distinguishing version number. The Program (including 190 | Contributions) may always be distributed subject to the version of the 191 | Agreement under which it was received. In addition, after a new version of the 192 | Agreement is published, Contributor may elect to distribute the Program 193 | (including its Contributions) under the new version. Except as expressly 194 | stated in Sections 2(a) and 2(b) above, Recipient receives no rights or 195 | licenses to the intellectual property of any Contributor under this Agreement, 196 | whether expressly, by implication, estoppel or otherwise. All rights in the 197 | Program not expressly granted under this Agreement are reserved. 198 | 199 | This Agreement is governed by the laws of the State of New York and the 200 | intellectual property laws of the United States of America. No party to this 201 | Agreement will bring a legal action under this Agreement more than one year 202 | after the cause of action arose. Each party waives its rights to a jury trial in 203 | any resulting litigation. 204 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # gclouj 2 | 3 | Idiomatic Clojure to [https://github.com/GoogleCloudPlatform/gcloud-java](https://github.com/GoogleCloudPlatform/gcloud-java). 4 | 5 | ## License 6 | 7 | Copyright © Paul Ingles, 2016. 8 | 9 | Distributed under the Eclipse Public License, the same as Clojure. -------------------------------------------------------------------------------- /bigquery/.gitignore: -------------------------------------------------------------------------------- 1 | /target 2 | /classes 3 | /checkouts 4 | pom.xml 5 | pom.xml.asc 6 | *.jar 7 | *.class 8 | /.lein-* 9 | /.nrepl-port 10 | .hgignore 11 | .hg/ 12 | /.idea 13 | /bigquery.iml -------------------------------------------------------------------------------- /bigquery/LICENSE: -------------------------------------------------------------------------------- 1 | THE ACCOMPANYING PROGRAM IS PROVIDED UNDER THE TERMS OF THIS ECLIPSE PUBLIC 2 | LICENSE ("AGREEMENT"). ANY USE, REPRODUCTION OR DISTRIBUTION OF THE PROGRAM 3 | CONSTITUTES RECIPIENT'S ACCEPTANCE OF THIS AGREEMENT. 4 | 5 | 1. DEFINITIONS 6 | 7 | "Contribution" means: 8 | 9 | a) in the case of the initial Contributor, the initial code and 10 | documentation distributed under this Agreement, and 11 | 12 | b) in the case of each subsequent Contributor: 13 | 14 | i) changes to the Program, and 15 | 16 | ii) additions to the Program; 17 | 18 | where such changes and/or additions to the Program originate from and are 19 | distributed by that particular Contributor. A Contribution 'originates' from 20 | a Contributor if it was added to the Program by such Contributor itself or 21 | anyone acting on such Contributor's behalf. Contributions do not include 22 | additions to the Program which: (i) are separate modules of software 23 | distributed in conjunction with the Program under their own license 24 | agreement, and (ii) are not derivative works of the Program. 25 | 26 | "Contributor" means any person or entity that distributes the Program. 27 | 28 | "Licensed Patents" mean patent claims licensable by a Contributor which are 29 | necessarily infringed by the use or sale of its Contribution alone or when 30 | combined with the Program. 31 | 32 | "Program" means the Contributions distributed in accordance with this 33 | Agreement. 34 | 35 | "Recipient" means anyone who receives the Program under this Agreement, 36 | including all Contributors. 37 | 38 | 2. GRANT OF RIGHTS 39 | 40 | a) Subject to the terms of this Agreement, each Contributor hereby grants 41 | Recipient a non-exclusive, worldwide, royalty-free copyright license to 42 | reproduce, prepare derivative works of, publicly display, publicly perform, 43 | distribute and sublicense the Contribution of such Contributor, if any, and 44 | such derivative works, in source code and object code form. 45 | 46 | b) Subject to the terms of this Agreement, each Contributor hereby grants 47 | Recipient a non-exclusive, worldwide, royalty-free patent license under 48 | Licensed Patents to make, use, sell, offer to sell, import and otherwise 49 | transfer the Contribution of such Contributor, if any, in source code and 50 | object code form. This patent license shall apply to the combination of the 51 | Contribution and the Program if, at the time the Contribution is added by the 52 | Contributor, such addition of the Contribution causes such combination to be 53 | covered by the Licensed Patents. The patent license shall not apply to any 54 | other combinations which include the Contribution. No hardware per se is 55 | licensed hereunder. 56 | 57 | c) Recipient understands that although each Contributor grants the licenses 58 | to its Contributions set forth herein, no assurances are provided by any 59 | Contributor that the Program does not infringe the patent or other 60 | intellectual property rights of any other entity. Each Contributor disclaims 61 | any liability to Recipient for claims brought by any other entity based on 62 | infringement of intellectual property rights or otherwise. As a condition to 63 | exercising the rights and licenses granted hereunder, each Recipient hereby 64 | assumes sole responsibility to secure any other intellectual property rights 65 | needed, if any. For example, if a third party patent license is required to 66 | allow Recipient to distribute the Program, it is Recipient's responsibility 67 | to acquire that license before distributing the Program. 68 | 69 | d) Each Contributor represents that to its knowledge it has sufficient 70 | copyright rights in its Contribution, if any, to grant the copyright license 71 | set forth in this Agreement. 72 | 73 | 3. REQUIREMENTS 74 | 75 | A Contributor may choose to distribute the Program in object code form under 76 | its own license agreement, provided that: 77 | 78 | a) it complies with the terms and conditions of this Agreement; and 79 | 80 | b) its license agreement: 81 | 82 | i) effectively disclaims on behalf of all Contributors all warranties and 83 | conditions, express and implied, including warranties or conditions of title 84 | and non-infringement, and implied warranties or conditions of merchantability 85 | and fitness for a particular purpose; 86 | 87 | ii) effectively excludes on behalf of all Contributors all liability for 88 | damages, including direct, indirect, special, incidental and consequential 89 | damages, such as lost profits; 90 | 91 | iii) states that any provisions which differ from this Agreement are offered 92 | by that Contributor alone and not by any other party; and 93 | 94 | iv) states that source code for the Program is available from such 95 | Contributor, and informs licensees how to obtain it in a reasonable manner on 96 | or through a medium customarily used for software exchange. 97 | 98 | When the Program is made available in source code form: 99 | 100 | a) it must be made available under this Agreement; and 101 | 102 | b) a copy of this Agreement must be included with each copy of the Program. 103 | 104 | Contributors may not remove or alter any copyright notices contained within 105 | the Program. 106 | 107 | Each Contributor must identify itself as the originator of its Contribution, 108 | if any, in a manner that reasonably allows subsequent Recipients to identify 109 | the originator of the Contribution. 110 | 111 | 4. COMMERCIAL DISTRIBUTION 112 | 113 | Commercial distributors of software may accept certain responsibilities with 114 | respect to end users, business partners and the like. While this license is 115 | intended to facilitate the commercial use of the Program, the Contributor who 116 | includes the Program in a commercial product offering should do so in a 117 | manner which does not create potential liability for other Contributors. 118 | Therefore, if a Contributor includes the Program in a commercial product 119 | offering, such Contributor ("Commercial Contributor") hereby agrees to defend 120 | and indemnify every other Contributor ("Indemnified Contributor") against any 121 | losses, damages and costs (collectively "Losses") arising from claims, 122 | lawsuits and other legal actions brought by a third party against the 123 | Indemnified Contributor to the extent caused by the acts or omissions of such 124 | Commercial Contributor in connection with its distribution of the Program in 125 | a commercial product offering. The obligations in this section do not apply 126 | to any claims or Losses relating to any actual or alleged intellectual 127 | property infringement. In order to qualify, an Indemnified Contributor must: 128 | a) promptly notify the Commercial Contributor in writing of such claim, and 129 | b) allow the Commercial Contributor tocontrol, and cooperate with the 130 | Commercial Contributor in, the defense and any related settlement 131 | negotiations. The Indemnified Contributor may participate in any such claim 132 | at its own expense. 133 | 134 | For example, a Contributor might include the Program in a commercial product 135 | offering, Product X. That Contributor is then a Commercial Contributor. If 136 | that Commercial Contributor then makes performance claims, or offers 137 | warranties related to Product X, those performance claims and warranties are 138 | such Commercial Contributor's responsibility alone. Under this section, the 139 | Commercial Contributor would have to defend claims against the other 140 | Contributors related to those performance claims and warranties, and if a 141 | court requires any other Contributor to pay any damages as a result, the 142 | Commercial Contributor must pay those damages. 143 | 144 | 5. NO WARRANTY 145 | 146 | EXCEPT AS EXPRESSLY SET FORTH IN THIS AGREEMENT, THE PROGRAM IS PROVIDED ON 147 | AN "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, EITHER 148 | EXPRESS OR IMPLIED INCLUDING, WITHOUT LIMITATION, ANY WARRANTIES OR 149 | CONDITIONS OF TITLE, NON-INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A 150 | PARTICULAR PURPOSE. Each Recipient is solely responsible for determining the 151 | appropriateness of using and distributing the Program and assumes all risks 152 | associated with its exercise of rights under this Agreement , including but 153 | not limited to the risks and costs of program errors, compliance with 154 | applicable laws, damage to or loss of data, programs or equipment, and 155 | unavailability or interruption of operations. 156 | 157 | 6. DISCLAIMER OF LIABILITY 158 | 159 | EXCEPT AS EXPRESSLY SET FORTH IN THIS AGREEMENT, NEITHER RECIPIENT NOR ANY 160 | CONTRIBUTORS SHALL HAVE ANY LIABILITY FOR ANY DIRECT, INDIRECT, INCIDENTAL, 161 | SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING WITHOUT LIMITATION 162 | LOST PROFITS), HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN 163 | CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) 164 | ARISING IN ANY WAY OUT OF THE USE OR DISTRIBUTION OF THE PROGRAM OR THE 165 | EXERCISE OF ANY RIGHTS GRANTED HEREUNDER, EVEN IF ADVISED OF THE POSSIBILITY 166 | OF SUCH DAMAGES. 167 | 168 | 7. GENERAL 169 | 170 | If any provision of this Agreement is invalid or unenforceable under 171 | applicable law, it shall not affect the validity or enforceability of the 172 | remainder of the terms of this Agreement, and without further action by the 173 | parties hereto, such provision shall be reformed to the minimum extent 174 | necessary to make such provision valid and enforceable. 175 | 176 | If Recipient institutes patent litigation against any entity (including a 177 | cross-claim or counterclaim in a lawsuit) alleging that the Program itself 178 | (excluding combinations of the Program with other software or hardware) 179 | infringes such Recipient's patent(s), then such Recipient's rights granted 180 | under Section 2(b) shall terminate as of the date such litigation is filed. 181 | 182 | All Recipient's rights under this Agreement shall terminate if it fails to 183 | comply with any of the material terms or conditions of this Agreement and 184 | does not cure such failure in a reasonable period of time after becoming 185 | aware of such noncompliance. If all Recipient's rights under this Agreement 186 | terminate, Recipient agrees to cease use and distribution of the Program as 187 | soon as reasonably practicable. However, Recipient's obligations under this 188 | Agreement and any licenses granted by Recipient relating to the Program shall 189 | continue and survive. 190 | 191 | Everyone is permitted to copy and distribute copies of this Agreement, but in 192 | order to avoid inconsistency the Agreement is copyrighted and may only be 193 | modified in the following manner. The Agreement Steward reserves the right to 194 | publish new versions (including revisions) of this Agreement from time to 195 | time. No one other than the Agreement Steward has the right to modify this 196 | Agreement. The Eclipse Foundation is the initial Agreement Steward. The 197 | Eclipse Foundation may assign the responsibility to serve as the Agreement 198 | Steward to a suitable separate entity. Each new version of the Agreement will 199 | be given a distinguishing version number. The Program (including 200 | Contributions) may always be distributed subject to the version of the 201 | Agreement under which it was received. In addition, after a new version of 202 | the Agreement is published, Contributor may elect to distribute the Program 203 | (including its Contributions) under the new version. Except as expressly 204 | stated in Sections 2(a) and 2(b) above, Recipient receives no rights or 205 | licenses to the intellectual property of any Contributor under this 206 | Agreement, whether expressly, by implication, estoppel or otherwise. All 207 | rights in the Program not expressly granted under this Agreement are 208 | reserved. 209 | 210 | This Agreement is governed by the laws of the State of New York and the 211 | intellectual property laws of the United States of America. No party to this 212 | Agreement will bring a legal action under this Agreement more than one year 213 | after the cause of action arose. Each party waives its rights to a jury trial 214 | in any resulting litigation. 215 | -------------------------------------------------------------------------------- /bigquery/README.md: -------------------------------------------------------------------------------- 1 | # bigquery 2 | 3 | A Clojure library designed to ... well, that part is up to you. 4 | 5 | ## Usage 6 | 7 | FIXME 8 | 9 | ## License 10 | 11 | Copyright © 2016 Paul Ingles 12 | 13 | Distributed under the Eclipse Public License either version 1.0 or (at 14 | your option) any later version. 15 | -------------------------------------------------------------------------------- /bigquery/doc/intro.md: -------------------------------------------------------------------------------- 1 | # Introduction to bigquery 2 | 3 | TODO: write [great documentation](http://jacobian.org/writing/what-to-write/) 4 | -------------------------------------------------------------------------------- /bigquery/project.clj: -------------------------------------------------------------------------------- 1 | (defproject gclouj/bigquery "0.2.7.0" 2 | :description "Google Cloud BigQuery" 3 | :url "https://github.com/pingles/gclouj" 4 | :license {:name "Eclipse Public License" 5 | :url "http://www.eclipse.org/legal/epl-v10.html"} 6 | :dependencies [[org.clojure/clojure "1.8.0"] 7 | [com.google.cloud/gcloud-java-bigquery "0.2.8" :exclusions [io.netty/netty-codec-http2 io.grpc/grpc-core]] 8 | [clj-time "0.14.0"]] 9 | :source-paths ["src/clojure"] 10 | :java-source-paths ["src/java"] 11 | :profiles {:dev {:global-vars {*warn-on-reflection* true}}}) 12 | -------------------------------------------------------------------------------- /bigquery/src/clojure/gclouj/bigquery.clj: -------------------------------------------------------------------------------- 1 | (ns gclouj.bigquery 2 | (:require [clojure.walk :as walk] 3 | [clj-time.coerce :as tc]) 4 | (:import [com.google.cloud.bigquery BigQueryOptions BigQuery$DatasetListOption DatasetInfo DatasetId BigQuery$TableListOption StandardTableDefinition TableId BigQuery$DatasetOption BigQuery$TableOption Schema Field Field$Type Field$Mode StandardTableDefinition$StreamingBuffer InsertAllRequest InsertAllRequest$RowToInsert InsertAllResponse BigQueryError BigQuery$DatasetDeleteOption QueryRequest QueryResponse QueryResult JobId Field Field$Type$Value FieldValue FieldValue$Attribute LoadConfiguration BigQuery$JobOption JobInfo$CreateDisposition JobInfo$WriteDisposition JobStatistics JobStatistics$LoadStatistics JobStatus JobStatus$State FormatOptions UserDefinedFunction JobInfo ExtractJobConfiguration LoadJobConfiguration QueryJobConfiguration QueryJobConfiguration$Priority Table BigQuery$QueryResultsOption TableInfo ViewDefinition CsvOptions CopyJobConfiguration] 5 | [gclouj BigQueryOptionsFactory] 6 | [com.google.common.hash Hashing] 7 | [java.util List Collections] 8 | [java.util.concurrent TimeUnit])) 9 | 10 | 11 | (defmulti field-value->clojure (fn [attribute val] 12 | attribute)) 13 | (defmethod field-value->clojure FieldValue$Attribute/PRIMITIVE [_ ^FieldValue val] 14 | (.value val)) 15 | 16 | (defprotocol ToClojure 17 | (to-clojure [_])) 18 | (extend-protocol ToClojure 19 | DatasetId 20 | (to-clojure [x] {:dataset-id (.dataset x) 21 | :project-id (.project x)}) 22 | DatasetInfo 23 | (to-clojure [x] {:creation-time (.creationTime x) 24 | :dataset-id (to-clojure (.datasetId x)) 25 | :description (.description x) 26 | :friendly-name (.friendlyName x) 27 | :location (.location x) 28 | :last-modified (.lastModified x)}) 29 | TableId 30 | (to-clojure [x] {:dataset-id (.dataset x) 31 | :project-id (.project x) 32 | :table-id (.table x)}) 33 | StandardTableDefinition$StreamingBuffer 34 | (to-clojure [x] {:estimated-bytes (.estimatedBytes x) 35 | :estimated-rows (.estimatedRows x) 36 | :oldest-entry-time (.oldestEntryTime x)}) 37 | StandardTableDefinition 38 | (to-clojure [x] {:location (.location x) 39 | :bytes (.numBytes x) 40 | :rows (.numRows x) 41 | :streaming-buffer (when-let [sb (.streamingBuffer x)] (to-clojure sb)) 42 | :schema (when-let [schema (.schema x)] (to-clojure schema))}) 43 | ViewDefinition 44 | (to-clojure [x] {:schema (when-let [schema (.schema x)] 45 | (to-clojure schema))}) 46 | Table 47 | (to-clojure [x] {:creation-time (.creationTime x) 48 | :description (.description x) 49 | :friendly-name (.friendlyName x) 50 | :table-id (to-clojure (.tableId x)) 51 | :definition (to-clojure (.definition x))}) 52 | BigQueryError 53 | (to-clojure [x] {:reason (.reason x) 54 | :location (.location x) 55 | :message (.message x)}) 56 | InsertAllResponse 57 | (to-clojure [x] {:errors (->> (.insertErrors x) 58 | (map (fn [[idx errors]] 59 | {:index idx 60 | :errors (map to-clojure errors)})) 61 | (seq))}) 62 | JobId 63 | (to-clojure [x] {:project-id (.project x) 64 | :job-id (.job x)}) 65 | Field 66 | (to-clojure [x] (let [type (.. x type value)] 67 | {:name (.name x) 68 | :mode ({Field$Mode/NULLABLE :nullable 69 | Field$Mode/REPEATED :repeated 70 | Field$Mode/REQUIRED :required} (.mode x)) 71 | :type ({Field$Type$Value/BOOLEAN :bool 72 | Field$Type$Value/FLOAT :float 73 | Field$Type$Value/INTEGER :integer 74 | Field$Type$Value/RECORD :record 75 | Field$Type$Value/STRING :string 76 | Field$Type$Value/TIMESTAMP :timestamp} type) 77 | :fields (when (= Field$Type$Value/RECORD type) 78 | (map to-clojure (.fields x)))})) 79 | FieldValue 80 | (to-clojure [x] (field-value->clojure (.attribute x) x)) 81 | Schema 82 | (to-clojure [x] (map to-clojure (.fields x))) 83 | QueryResponse 84 | (to-clojure [x] (let [completed (.jobCompleted x)] 85 | {:completed? completed 86 | :errors (->> (.executionErrors x) (map to-clojure) (seq)) 87 | :job-id (to-clojure (.jobId x)) 88 | :results (when completed 89 | (map (fn [fields] (map to-clojure fields)) 90 | (iterator-seq (.. x result iterateAll)))) 91 | :schema (when completed 92 | (to-clojure (.. x result schema))) 93 | :cache-hit (when completed 94 | (.. x result cacheHit))})) 95 | JobStatistics$LoadStatistics 96 | (to-clojure [x] {:input-bytes (.inputBytes x) 97 | :input-files (.inputFiles x) 98 | :output-bytes (.outputBytes x) 99 | :output-rows (.outputRows x)}) 100 | JobStatistics 101 | (to-clojure [x] {:created (.creationTime x) 102 | :end (.endTime x) 103 | :started (.startTime x)}) 104 | JobStatus 105 | (to-clojure [x] {:state ({JobStatus$State/DONE :done 106 | JobStatus$State/PENDING :pending 107 | JobStatus$State/RUNNING :running} (.state x)) 108 | :errors (seq (map to-clojure (.executionErrors x)))}) 109 | JobInfo 110 | (to-clojure [x] {:job-id (to-clojure (.jobId x)) 111 | :statistics (to-clojure (.statistics x)) 112 | :email (.userEmail x) 113 | :status (to-clojure (.status x))})) 114 | 115 | (defn service 116 | ([] (.service (BigQueryOptions/defaultInstance))) 117 | ([{:keys [project-id] :as options}] 118 | (.service (BigQueryOptionsFactory/create project-id)))) 119 | 120 | (defn datasets [service] 121 | (let [it (-> service 122 | (.listDatasets (into-array BigQuery$DatasetListOption [])) 123 | (.iterateAll))] 124 | (map to-clojure (iterator-seq it)))) 125 | 126 | (defn dataset [service {:keys [project-id dataset-id] :as dataset}] 127 | (when-let [dataset (.getDataset service (DatasetId/of project-id dataset-id) (into-array BigQuery$DatasetOption []))] 128 | (to-clojure dataset))) 129 | 130 | (defn tables 131 | "Returns a sequence of table-ids. For complete table 132 | information (schema, location, size etc.) you'll need to also use the 133 | `table` function." 134 | [service {:keys [project-id dataset-id] :as dataset}] 135 | (let [it (-> service 136 | (.listTables (DatasetId/of project-id dataset-id) 137 | (into-array BigQuery$TableListOption [])) 138 | (.iterateAll))] 139 | (map to-clojure (iterator-seq it)))) 140 | 141 | (defn table [service {:keys [project-id dataset-id table-id] :as table}] 142 | (to-clojure (.getTable service 143 | (TableId/of project-id dataset-id table-id) 144 | (into-array BigQuery$TableOption [])))) 145 | 146 | (defn create-dataset [service {:keys [project-id dataset-id friendly-name location description table-lifetime-millis] :as dataset}] 147 | (let [locations {:eu "EU" 148 | :us "US"} 149 | builder (DatasetInfo/builder project-id dataset-id)] 150 | (when friendly-name 151 | (.friendlyName builder friendly-name)) 152 | (when description 153 | (.description builder description)) 154 | (when table-lifetime-millis 155 | (.defaultTableLifetime builder table-lifetime-millis)) 156 | (.location builder (or (locations location) "US")) 157 | (to-clojure (.create service (.build builder) (into-array BigQuery$DatasetOption []))))) 158 | 159 | (defn delete-dataset [service {:keys [project-id dataset-id delete-contents?] :as dataset}] 160 | (let [options (if delete-contents? 161 | [(BigQuery$DatasetDeleteOption/deleteContents)] 162 | [])] 163 | (.delete service 164 | (DatasetId/of project-id dataset-id) 165 | (into-array BigQuery$DatasetDeleteOption options)))) 166 | 167 | (defn- mkfield [{:keys [name type description mode fields]}] 168 | (let [field-type (condp = type 169 | :bool (Field$Type/bool) 170 | :float (Field$Type/floatingPoint) 171 | :integer (Field$Type/integer) 172 | :string (Field$Type/string) 173 | :timestamp (Field$Type/timestamp) 174 | :record (Field$Type/record ^List (map mkfield fields))) 175 | builder (Field/builder name field-type) 176 | field-mode ({:nullable (Field$Mode/NULLABLE) 177 | :repeated (Field$Mode/REPEATED) 178 | :required (Field$Mode/REQUIRED)} (or mode :nullable))] 179 | (.mode builder field-mode) 180 | (.build builder))) 181 | 182 | (defn- mkschema 183 | [fields] 184 | (let [builder (Schema/builder)] 185 | (.fields builder (into-array Field (->> fields (map mkfield)))) 186 | (.build builder))) 187 | 188 | (defn create-table 189 | "Fields: sequence of fields representing the table schema. 190 | e.g. [{:name \"foo\" :type :record :fields [{:name \"bar\" :type :integer}]}]" 191 | [service {:keys [project-id dataset-id table-id] :as table} fields] 192 | (let [builder (TableInfo/builder (TableId/of project-id dataset-id table-id) 193 | (StandardTableDefinition/of (mkschema fields))) 194 | table-info (.build builder)] 195 | (to-clojure (.create service table-info (into-array BigQuery$TableOption []))))) 196 | 197 | (defn delete-table 198 | [service {:keys [project-id dataset-id table-id] :as table}] 199 | (.delete service (TableId/of project-id dataset-id table-id))) 200 | 201 | (defn row-hash 202 | "Creates a hash suitable for identifying duplicate rows, useful when 203 | streaming to avoid inserting duplicate rows." 204 | [m & {:keys [bits] :or {bits 128}}] 205 | (-> (Hashing/goodFastHash bits) (.hashUnencodedChars (pr-str m)) (.toString))) 206 | 207 | (defn- row-value [m] 208 | ;; the google client incorrectly interprets clojure maps as arrays so 209 | ;; we wrap in an unmodifiableMap to ensure the client interprets 210 | ;; correctly. 211 | (letfn [(wrap-map [x] 212 | (if (map? x) 213 | (Collections/unmodifiableMap x) 214 | x))] 215 | (walk/postwalk wrap-map m))) 216 | 217 | (defn- insert-row [row-id row] 218 | (if row-id 219 | (InsertAllRequest$RowToInsert/of (row-id row) (row-value row)) 220 | (InsertAllRequest$RowToInsert/of (row-value row)))) 221 | 222 | (defn insert-all 223 | "Performs a streaming insert of rows. row-id can be a function to 224 | return the unique identity of the row (e.g. row-hash). Template suffix 225 | can be used to create tables according to a template." 226 | [service {:keys [project-id dataset-id table-id skip-invalid? template-suffix row-id] :as table} rows] 227 | (let [builder (InsertAllRequest/builder (TableId/of project-id dataset-id table-id) 228 | ^Iterable (map (partial insert-row row-id) rows))] 229 | (when template-suffix 230 | (.templateSuffix builder template-suffix)) 231 | (->> builder 232 | (.build) 233 | (.insertAll service) 234 | (to-clojure)))) 235 | 236 | 237 | 238 | (defn query 239 | "Executes a query. BigQuery will create a Query Job and block for the 240 | specified timeout. If the query returns within the time the results 241 | will be returned. Otherwise, results need to be retrieved separately 242 | using query-results. Status of the job can be checked using the job 243 | function, and checking completed?" 244 | [service query {:keys [max-results dry-run? max-wait-millis use-cache? use-legacy-sql? default-dataset] :as query-opts}] 245 | (let [builder (QueryRequest/builder query)] 246 | (when default-dataset 247 | (let [{:keys [project-id dataset-id]} default-dataset] 248 | (.defaultDataset builder (DatasetId/of project-id dataset-id)))) 249 | (when max-results 250 | (.maxResults builder max-results)) 251 | (when-not (nil? dry-run?) 252 | (.dryRun builder dry-run?)) 253 | (when max-wait-millis 254 | (.maxWaitTime builder max-wait-millis)) 255 | (when-not use-legacy-sql? 256 | (.useLegacySql builder use-legacy-sql?)) 257 | (.useQueryCache builder use-cache?) 258 | (let [q (.build builder)] 259 | (to-clojure (.query service q))))) 260 | 261 | 262 | 263 | 264 | (defmulti query-option (fn [[type _]] type)) 265 | (defmethod query-option :max-wait-millis [[_ val]] (BigQuery$QueryResultsOption/maxWaitTime val)) 266 | 267 | (defn query-results 268 | "Retrieves results for a Query job. Will throw exceptions unless Job 269 | has completed successfully. Check using job and completed? functions." 270 | [service {:keys [project-id job-id] :as job} & {:keys [max-wait-millis] :as opts}] 271 | {:pre [(string? job-id) (string? project-id)]} 272 | (to-clojure (.getQueryResults service 273 | (JobId/of project-id job-id) 274 | (->> opts (map query-option) (into-array BigQuery$QueryResultsOption))))) 275 | 276 | (defn- parse-timestamp [val] 277 | (let [seconds (long (Double/valueOf val)) 278 | millis (.toMillis (TimeUnit/SECONDS) seconds)] 279 | (tc/from-long millis))) 280 | 281 | (def cell-coercions {:integer #(when % (Long/valueOf %)) 282 | :bool #(when % (Boolean/valueOf %)) 283 | :float #(when % (Double/valueOf %)) 284 | :string identity 285 | :timestamp #(when % (parse-timestamp %))}) 286 | 287 | (defn- coerce-result 288 | [schema] 289 | (let [coercions (map cell-coercions (map :type schema)) 290 | names (map :name schema)] 291 | (fn [row] 292 | (->> (for [[name coerce value] (partition 3 (interleave names coercions row))] 293 | [name (coerce value)]) 294 | (into {}))))) 295 | 296 | (defn query-results-seq 297 | "Takes a query result and coerces the results from being raw sequences 298 | into maps according to the schema and coercing values according to 299 | their type. e.g.: 300 | converts from query results of: {:results ((\"500\")) :schema ({:name \"col\" :type :integer})} into... 301 | ({\"col\" 500} ...)" 302 | [{:keys [results schema] :as query-results}] 303 | (map (coerce-result schema) results)) 304 | 305 | (defn job [service {:keys [project-id job-id] :as job}] 306 | (to-clojure (.getJob service (JobId/of project-id job-id) (into-array BigQuery$JobOption [])))) 307 | 308 | (defn successful? [job] 309 | (and (= :done (get-in job [:status :state])) 310 | (empty? (get-in job [:status :errors])))) 311 | 312 | (defn running? [job] 313 | (= :running (get-in job [:status :state]))) 314 | 315 | (def create-dispositions {:needed JobInfo$CreateDisposition/CREATE_IF_NEEDED 316 | :never JobInfo$CreateDisposition/CREATE_NEVER}) 317 | 318 | (def write-dispositions {:append JobInfo$WriteDisposition/WRITE_APPEND 319 | :empty JobInfo$WriteDisposition/WRITE_EMPTY 320 | :truncate JobInfo$WriteDisposition/WRITE_TRUNCATE}) 321 | 322 | (defn table-id [{:keys [project-id dataset-id table-id]}] 323 | (TableId/of project-id dataset-id table-id)) 324 | 325 | (defn execute-job [service job] 326 | (to-clojure (.create service (.build (JobInfo/builder job)) (into-array BigQuery$JobOption [])))) 327 | 328 | (defn load-job 329 | "Loads data from Cloud Storage URIs into the specified table. 330 | Table argument needs to be a map with project-id, dataset-id and table-id. 331 | Options: 332 | `create-disposition` controls whether tables are created if 333 | necessary, or assume to have been created already (default). 334 | `write-disposition` controls whether data should :append (default), 335 | :truncate or :empty to fail if table exists. 336 | :format :json or :csv 337 | :schema sequence describing the table schema.[{:name \"foo\" :type :record :fields [{:name \"bar\" :type :integer}]}]" 338 | [service table {:keys [format create-disposition write-disposition max-bad-records schema]} uris] 339 | (let [builder (LoadJobConfiguration/builder (table-id table) 340 | uris 341 | ({:json (FormatOptions/json) 342 | :csv (FormatOptions/csv)} (or format :json)))] 343 | (.createDisposition builder (create-dispositions (or create-disposition :never))) 344 | (.writeDisposition builder (write-dispositions (or write-disposition :append))) 345 | (.maxBadRecords builder (int (or max-bad-records 0))) 346 | (when schema 347 | (.schema builder (mkschema schema))) 348 | (execute-job service (.build builder)))) 349 | 350 | (def extract-format {:json "NEWLINE_DELIMITED_JSON" 351 | :csv "CSV" 352 | :avro "AVRO"}) 353 | 354 | (def extract-compression {:gzip "GZIP" 355 | :none "NONE"}) 356 | 357 | (defn extract-job 358 | "Extracts data from BigQuery into a Google Cloud Storage location. 359 | Table argument needs to be a map with project-id, dataset-id and table-id." 360 | [service table destination-uri & {:keys [format compression] 361 | :or {format :json 362 | compression :gzip}}] 363 | (let [builder (ExtractJobConfiguration/builder (table-id table) destination-uri)] 364 | (.format builder (extract-format format)) 365 | (.compression builder (extract-compression compression)) 366 | (execute-job service (.build builder)))) 367 | 368 | (defn copy-job 369 | [service sources destination & {:keys [create-disposition write-disposition] 370 | :or {create-disposition :needed 371 | write-disposition :empty}}] 372 | (let [builder (CopyJobConfiguration/builder (table-id destination) (map table-id sources))] 373 | (.createDisposition builder (create-dispositions create-disposition)) 374 | (.writeDisposition builder (write-dispositions write-disposition)) 375 | (execute-job service (.build builder)))) 376 | 377 | (defn user-defined-function 378 | "Creates a User Defined Function suitable for use in BigQuery queries. Can be a Google Cloud Storage uri (e.g. gs://bucket/path), or an inline JavaScript code blob." 379 | [udf] 380 | (if (.startsWith udf "gs://") 381 | (UserDefinedFunction/fromUri udf) 382 | (UserDefinedFunction/inline udf))) 383 | 384 | (defn query-job 385 | [service query {:keys [create-disposition write-disposition large-results? dry-run? destination-table default-dataset use-cache? flatten-results? use-legacy-sql? priority udfs]}] 386 | (let [priorities {:batch (QueryJobConfiguration$Priority/BATCH) 387 | :interactive (QueryJobConfiguration$Priority/INTERACTIVE)} 388 | builder (QueryJobConfiguration/builder query)] 389 | (when default-dataset 390 | (let [{:keys [project-id dataset-id]} default-dataset] 391 | (.defaultDataset builder (DatasetId/of project-id dataset-id)))) 392 | (.createDisposition builder (create-dispositions (or create-disposition :never))) 393 | (.writeDisposition builder (write-dispositions (or write-disposition :append))) 394 | (.useLegacySql builder use-legacy-sql?) 395 | (.allowLargeResults builder large-results?) 396 | (.useQueryCache builder use-cache?) 397 | (.flattenResults builder flatten-results?) 398 | (.priority builder (priorities (or priority :batch))) 399 | (when udfs 400 | (.userDefinedFunctions builder udfs)) 401 | (when destination-table 402 | (let [{:keys [project-id dataset-id table-id]} destination-table] 403 | (.destinationTable builder (TableId/of project-id dataset-id table-id)))) 404 | (when-not (nil? dry-run?) 405 | (.dryRun builder dry-run?)) 406 | (execute-job service (.build builder)))) 407 | -------------------------------------------------------------------------------- /bigquery/src/java/gclouj/BigQueryOptionsFactory.java: -------------------------------------------------------------------------------- 1 | package gclouj; 2 | 3 | import com.google.cloud.bigquery.BigQueryOptions; 4 | 5 | public class BigQueryOptionsFactory { 6 | public static BigQueryOptions create(String projectId) { 7 | BigQueryOptions.Builder builder = BigQueryOptions.builder() 8 | .projectId(projectId); 9 | return builder.build(); 10 | } 11 | } -------------------------------------------------------------------------------- /datastore/.gitignore: -------------------------------------------------------------------------------- 1 | /target 2 | /classes 3 | /checkouts 4 | pom.xml 5 | pom.xml.asc 6 | *.jar 7 | *.class 8 | /.lein-* 9 | /.nrepl-port 10 | .hgignore 11 | .hg/ 12 | -------------------------------------------------------------------------------- /datastore/LICENSE: -------------------------------------------------------------------------------- 1 | THE ACCOMPANYING PROGRAM IS PROVIDED UNDER THE TERMS OF THIS ECLIPSE PUBLIC 2 | LICENSE ("AGREEMENT"). ANY USE, REPRODUCTION OR DISTRIBUTION OF THE PROGRAM 3 | CONSTITUTES RECIPIENT'S ACCEPTANCE OF THIS AGREEMENT. 4 | 5 | 1. DEFINITIONS 6 | 7 | "Contribution" means: 8 | 9 | a) in the case of the initial Contributor, the initial code and 10 | documentation distributed under this Agreement, and 11 | 12 | b) in the case of each subsequent Contributor: 13 | 14 | i) changes to the Program, and 15 | 16 | ii) additions to the Program; 17 | 18 | where such changes and/or additions to the Program originate from and are 19 | distributed by that particular Contributor. A Contribution 'originates' from 20 | a Contributor if it was added to the Program by such Contributor itself or 21 | anyone acting on such Contributor's behalf. Contributions do not include 22 | additions to the Program which: (i) are separate modules of software 23 | distributed in conjunction with the Program under their own license 24 | agreement, and (ii) are not derivative works of the Program. 25 | 26 | "Contributor" means any person or entity that distributes the Program. 27 | 28 | "Licensed Patents" mean patent claims licensable by a Contributor which are 29 | necessarily infringed by the use or sale of its Contribution alone or when 30 | combined with the Program. 31 | 32 | "Program" means the Contributions distributed in accordance with this 33 | Agreement. 34 | 35 | "Recipient" means anyone who receives the Program under this Agreement, 36 | including all Contributors. 37 | 38 | 2. GRANT OF RIGHTS 39 | 40 | a) Subject to the terms of this Agreement, each Contributor hereby grants 41 | Recipient a non-exclusive, worldwide, royalty-free copyright license to 42 | reproduce, prepare derivative works of, publicly display, publicly perform, 43 | distribute and sublicense the Contribution of such Contributor, if any, and 44 | such derivative works, in source code and object code form. 45 | 46 | b) Subject to the terms of this Agreement, each Contributor hereby grants 47 | Recipient a non-exclusive, worldwide, royalty-free patent license under 48 | Licensed Patents to make, use, sell, offer to sell, import and otherwise 49 | transfer the Contribution of such Contributor, if any, in source code and 50 | object code form. This patent license shall apply to the combination of the 51 | Contribution and the Program if, at the time the Contribution is added by the 52 | Contributor, such addition of the Contribution causes such combination to be 53 | covered by the Licensed Patents. The patent license shall not apply to any 54 | other combinations which include the Contribution. No hardware per se is 55 | licensed hereunder. 56 | 57 | c) Recipient understands that although each Contributor grants the licenses 58 | to its Contributions set forth herein, no assurances are provided by any 59 | Contributor that the Program does not infringe the patent or other 60 | intellectual property rights of any other entity. Each Contributor disclaims 61 | any liability to Recipient for claims brought by any other entity based on 62 | infringement of intellectual property rights or otherwise. As a condition to 63 | exercising the rights and licenses granted hereunder, each Recipient hereby 64 | assumes sole responsibility to secure any other intellectual property rights 65 | needed, if any. For example, if a third party patent license is required to 66 | allow Recipient to distribute the Program, it is Recipient's responsibility 67 | to acquire that license before distributing the Program. 68 | 69 | d) Each Contributor represents that to its knowledge it has sufficient 70 | copyright rights in its Contribution, if any, to grant the copyright license 71 | set forth in this Agreement. 72 | 73 | 3. REQUIREMENTS 74 | 75 | A Contributor may choose to distribute the Program in object code form under 76 | its own license agreement, provided that: 77 | 78 | a) it complies with the terms and conditions of this Agreement; and 79 | 80 | b) its license agreement: 81 | 82 | i) effectively disclaims on behalf of all Contributors all warranties and 83 | conditions, express and implied, including warranties or conditions of title 84 | and non-infringement, and implied warranties or conditions of merchantability 85 | and fitness for a particular purpose; 86 | 87 | ii) effectively excludes on behalf of all Contributors all liability for 88 | damages, including direct, indirect, special, incidental and consequential 89 | damages, such as lost profits; 90 | 91 | iii) states that any provisions which differ from this Agreement are offered 92 | by that Contributor alone and not by any other party; and 93 | 94 | iv) states that source code for the Program is available from such 95 | Contributor, and informs licensees how to obtain it in a reasonable manner on 96 | or through a medium customarily used for software exchange. 97 | 98 | When the Program is made available in source code form: 99 | 100 | a) it must be made available under this Agreement; and 101 | 102 | b) a copy of this Agreement must be included with each copy of the Program. 103 | 104 | Contributors may not remove or alter any copyright notices contained within 105 | the Program. 106 | 107 | Each Contributor must identify itself as the originator of its Contribution, 108 | if any, in a manner that reasonably allows subsequent Recipients to identify 109 | the originator of the Contribution. 110 | 111 | 4. COMMERCIAL DISTRIBUTION 112 | 113 | Commercial distributors of software may accept certain responsibilities with 114 | respect to end users, business partners and the like. While this license is 115 | intended to facilitate the commercial use of the Program, the Contributor who 116 | includes the Program in a commercial product offering should do so in a 117 | manner which does not create potential liability for other Contributors. 118 | Therefore, if a Contributor includes the Program in a commercial product 119 | offering, such Contributor ("Commercial Contributor") hereby agrees to defend 120 | and indemnify every other Contributor ("Indemnified Contributor") against any 121 | losses, damages and costs (collectively "Losses") arising from claims, 122 | lawsuits and other legal actions brought by a third party against the 123 | Indemnified Contributor to the extent caused by the acts or omissions of such 124 | Commercial Contributor in connection with its distribution of the Program in 125 | a commercial product offering. The obligations in this section do not apply 126 | to any claims or Losses relating to any actual or alleged intellectual 127 | property infringement. In order to qualify, an Indemnified Contributor must: 128 | a) promptly notify the Commercial Contributor in writing of such claim, and 129 | b) allow the Commercial Contributor tocontrol, and cooperate with the 130 | Commercial Contributor in, the defense and any related settlement 131 | negotiations. The Indemnified Contributor may participate in any such claim 132 | at its own expense. 133 | 134 | For example, a Contributor might include the Program in a commercial product 135 | offering, Product X. That Contributor is then a Commercial Contributor. If 136 | that Commercial Contributor then makes performance claims, or offers 137 | warranties related to Product X, those performance claims and warranties are 138 | such Commercial Contributor's responsibility alone. Under this section, the 139 | Commercial Contributor would have to defend claims against the other 140 | Contributors related to those performance claims and warranties, and if a 141 | court requires any other Contributor to pay any damages as a result, the 142 | Commercial Contributor must pay those damages. 143 | 144 | 5. NO WARRANTY 145 | 146 | EXCEPT AS EXPRESSLY SET FORTH IN THIS AGREEMENT, THE PROGRAM IS PROVIDED ON 147 | AN "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, EITHER 148 | EXPRESS OR IMPLIED INCLUDING, WITHOUT LIMITATION, ANY WARRANTIES OR 149 | CONDITIONS OF TITLE, NON-INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A 150 | PARTICULAR PURPOSE. Each Recipient is solely responsible for determining the 151 | appropriateness of using and distributing the Program and assumes all risks 152 | associated with its exercise of rights under this Agreement , including but 153 | not limited to the risks and costs of program errors, compliance with 154 | applicable laws, damage to or loss of data, programs or equipment, and 155 | unavailability or interruption of operations. 156 | 157 | 6. DISCLAIMER OF LIABILITY 158 | 159 | EXCEPT AS EXPRESSLY SET FORTH IN THIS AGREEMENT, NEITHER RECIPIENT NOR ANY 160 | CONTRIBUTORS SHALL HAVE ANY LIABILITY FOR ANY DIRECT, INDIRECT, INCIDENTAL, 161 | SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING WITHOUT LIMITATION 162 | LOST PROFITS), HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN 163 | CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) 164 | ARISING IN ANY WAY OUT OF THE USE OR DISTRIBUTION OF THE PROGRAM OR THE 165 | EXERCISE OF ANY RIGHTS GRANTED HEREUNDER, EVEN IF ADVISED OF THE POSSIBILITY 166 | OF SUCH DAMAGES. 167 | 168 | 7. GENERAL 169 | 170 | If any provision of this Agreement is invalid or unenforceable under 171 | applicable law, it shall not affect the validity or enforceability of the 172 | remainder of the terms of this Agreement, and without further action by the 173 | parties hereto, such provision shall be reformed to the minimum extent 174 | necessary to make such provision valid and enforceable. 175 | 176 | If Recipient institutes patent litigation against any entity (including a 177 | cross-claim or counterclaim in a lawsuit) alleging that the Program itself 178 | (excluding combinations of the Program with other software or hardware) 179 | infringes such Recipient's patent(s), then such Recipient's rights granted 180 | under Section 2(b) shall terminate as of the date such litigation is filed. 181 | 182 | All Recipient's rights under this Agreement shall terminate if it fails to 183 | comply with any of the material terms or conditions of this Agreement and 184 | does not cure such failure in a reasonable period of time after becoming 185 | aware of such noncompliance. If all Recipient's rights under this Agreement 186 | terminate, Recipient agrees to cease use and distribution of the Program as 187 | soon as reasonably practicable. However, Recipient's obligations under this 188 | Agreement and any licenses granted by Recipient relating to the Program shall 189 | continue and survive. 190 | 191 | Everyone is permitted to copy and distribute copies of this Agreement, but in 192 | order to avoid inconsistency the Agreement is copyrighted and may only be 193 | modified in the following manner. The Agreement Steward reserves the right to 194 | publish new versions (including revisions) of this Agreement from time to 195 | time. No one other than the Agreement Steward has the right to modify this 196 | Agreement. The Eclipse Foundation is the initial Agreement Steward. The 197 | Eclipse Foundation may assign the responsibility to serve as the Agreement 198 | Steward to a suitable separate entity. Each new version of the Agreement will 199 | be given a distinguishing version number. The Program (including 200 | Contributions) may always be distributed subject to the version of the 201 | Agreement under which it was received. In addition, after a new version of 202 | the Agreement is published, Contributor may elect to distribute the Program 203 | (including its Contributions) under the new version. Except as expressly 204 | stated in Sections 2(a) and 2(b) above, Recipient receives no rights or 205 | licenses to the intellectual property of any Contributor under this 206 | Agreement, whether expressly, by implication, estoppel or otherwise. All 207 | rights in the Program not expressly granted under this Agreement are 208 | reserved. 209 | 210 | This Agreement is governed by the laws of the State of New York and the 211 | intellectual property laws of the United States of America. No party to this 212 | Agreement will bring a legal action under this Agreement more than one year 213 | after the cause of action arose. Each party waives its rights to a jury trial 214 | in any resulting litigation. 215 | -------------------------------------------------------------------------------- /datastore/README.md: -------------------------------------------------------------------------------- 1 | # datastore 2 | 3 | A Clojure library designed to ... well, that part is up to you. 4 | 5 | ## Usage 6 | 7 | FIXME 8 | 9 | ## License 10 | 11 | Copyright © 2016 FIXME 12 | 13 | Distributed under the Eclipse Public License either version 1.0 or (at 14 | your option) any later version. 15 | -------------------------------------------------------------------------------- /datastore/project.clj: -------------------------------------------------------------------------------- 1 | (defproject gclouj/datastore "0.2.1" 2 | :description "Google Cloud Datastore" 3 | :url "https://github.com/pingles/gclouj" 4 | :license {:name "Eclipse Public License" 5 | :url "http://www.eclipse.org/legal/epl-v10.html"} 6 | :dependencies [[org.clojure/clojure "1.8.0"] 7 | [com.google.cloud/gcloud-java-datastore "0.2.6" :exclusions [com.google.guava/guava com.google.guava/guava-jdk5]] 8 | [com.google.guava/guava "19.0"]] 9 | :source-paths ["src/clojure"] 10 | :java-source-paths ["src/java"] 11 | :profiles {:dev {:global-vars {*warn-on-reflection* true}}}) 12 | -------------------------------------------------------------------------------- /datastore/src/clojure/gclouj/datastore.clj: -------------------------------------------------------------------------------- 1 | (ns gclouj.datastore 2 | (:require [clojure.java.io :as io] 3 | [clojure.walk :as w]) 4 | (:import [gclouj DatastoreOptionsFactory] 5 | [java.io InputStream] 6 | [java.nio ByteBuffer] 7 | [com.google.cloud.datastore DatastoreOptions FullEntity Entity FullEntity DatastoreOptions$DefaultDatastoreFactory Transaction Key IncompleteKey DatastoreBatchWriter EntityValue ValueType StringValue LongValue DoubleValue DateTime DateTimeValue BooleanValue BlobValue Blob ListValue NullValue Value KeyValue FullEntity$Builder Query StructuredQuery$PropertyFilter StructuredQuery$CompositeFilter StructuredQuery$Filter StructuredQuery$OrderBy] 8 | [com.google.cloud AuthCredentials])) 9 | 10 | (defn credential-options [project-id namespace json-key] 11 | (DatastoreOptionsFactory/create project-id namespace (AuthCredentials/createForJson (io/input-stream json-key)))) 12 | 13 | (defn service 14 | "Creates the Datastore service to be used with other functions. Use 15 | options when connecting outside of the Google Cloud Platform, if 16 | you're running on GCE/GAE you can use the no-args version." 17 | ([] (.service (DatastoreOptions/defaultInstance))) 18 | ([options] (.service options))) 19 | 20 | (defmulti complete-key (fn [project-id kind name-or-id] (cond (string? name-or-id) :name 21 | (number? name-or-id) :id))) 22 | (defmethod complete-key :name [project-id kind name] (.build (Key/builder project-id kind ^String name))) 23 | (defmethod complete-key :id [project-id kind id] (.build (Key/builder project-id kind ^long id))) 24 | 25 | (defn incomplete-key [project-id kind] 26 | (.build (IncompleteKey/builder project-id kind))) 27 | 28 | (defn get-entity 29 | ([^Transaction txn key] 30 | (.get txn key)) 31 | ([^Transaction txn key1 key2] 32 | (iterator-seq (.get txn (into-array Key [key1 key2])))) 33 | ([^Transaction txn key1 key2 & keys] 34 | (iterator-seq (.get txn (into-array Key (concat [key1 key2] keys)))))) 35 | 36 | (defn add-entity [^Transaction txn entity] 37 | (.add txn ^FullEntity entity)) 38 | 39 | (defn delete-entity [^Transaction txn & keys] 40 | (.delete txn (into-array Key keys))) 41 | 42 | (defn update-entity [^Transaction txn & entities] 43 | (.update txn (into-array Entity entities))) 44 | 45 | (defn byte-array? [x] 46 | (= (Class/forName "[B") (class x))) 47 | 48 | 49 | 50 | 51 | (declare entity) 52 | 53 | (defn to-value [x] 54 | (cond (nil? x) (NullValue. ) 55 | (float? x) (DoubleValue/of x) 56 | (number? x) (LongValue/of x) 57 | (string? x) (StringValue/of x) 58 | (instance? DateTime x) (DateTimeValue/of x) 59 | (instance? Boolean x) (BooleanValue/of x) 60 | (instance? Key x) (KeyValue/of x) 61 | (or (byte-array? x) 62 | (instance? InputStream x) 63 | (instance? ByteBuffer x)) (BlobValue/of (Blob/copyFrom x)) 64 | (sequential? x) (ListValue/of (map to-value x)) 65 | (instance? FullEntity x) (EntityValue/of x) 66 | (map? x) (EntityValue/of (entity x)) 67 | :else x)) 68 | 69 | (defn- map->entity 70 | [builder m] 71 | (doseq [[k v] m] 72 | (if (map? v) 73 | (.set builder k ^FullEntity (entity v)) 74 | (.set builder k ^Value (to-value v)))) 75 | builder) 76 | 77 | (defn entity 78 | "Converts a Clojure map into a Datastore Entity. Recursively converts 79 | all values into an entity or value using property-value. Keys must all 80 | be strings." 81 | ([m] (let [b (FullEntity/builder)] 82 | (.build (map->entity b m)))) 83 | ([k m] (let [b (cond 84 | (instance? Key k) (Entity/builder k) 85 | (instance? IncompleteKey k) (FullEntity/builder k))] 86 | (.build (map->entity b m))))) 87 | 88 | (defprotocol ToClojure 89 | (to-clojure [x])) 90 | (extend-protocol ToClojure 91 | BlobValue 92 | (to-clojure [value] (.. value asReadOnlyByteBuffer)) 93 | BooleanValue 94 | (to-clojure [value] (.get value)) 95 | DateTimeValue 96 | (to-clojure [value] (.get value)) 97 | DoubleValue 98 | (to-clojure [value] (.get value)) 99 | KeyValue 100 | (to-clojure [value] (.get value)) 101 | NullValue 102 | (to-clojure [value] nil) 103 | LongValue 104 | (to-clojure [value] (.get value)) 105 | StringValue 106 | (to-clojure [value] (.get value)) 107 | ListValue 108 | (to-clojure [value] (map to-clojure (.get value))) 109 | EntityValue 110 | (to-clojure [value] (to-clojure (.get value))) 111 | FullEntity 112 | (to-clojure [entity] 113 | (let [names (-> entity (.names) (.iterator) (iterator-seq))] 114 | (into {} (map (fn [attr] [attr (to-clojure (.getValue entity attr))]) names))))) 115 | 116 | (defn transaction [service] 117 | (.newTransaction service)) 118 | 119 | (defmulti efilter (fn [type & filter-expression] type)) 120 | (defmethod efilter :ancestor [_ key] 121 | (StructuredQuery$PropertyFilter/hasAncestor key)) 122 | (defmethod efilter := [_ property value] 123 | (StructuredQuery$PropertyFilter/eq property value)) 124 | (defmethod efilter :>= [_ property value] 125 | (StructuredQuery$PropertyFilter/ge property value)) 126 | (defmethod efilter :> [_ property value] 127 | (StructuredQuery$PropertyFilter/gt property value)) 128 | (defmethod efilter :nil [_ property] 129 | (StructuredQuery$PropertyFilter/isNull property)) 130 | (defmethod efilter :<= [_ property value] 131 | (StructuredQuery$PropertyFilter/le property value)) 132 | (defmethod efilter :< [_ property value] 133 | (StructuredQuery$PropertyFilter/lt property value)) 134 | 135 | (defn query-filters 136 | ([f1] f1) 137 | ([f1 f2 & more] 138 | (StructuredQuery$CompositeFilter/and f1 (into-array StructuredQuery$Filter (conj more f2))))) 139 | 140 | (defn order-by [direction property] 141 | (condp = direction 142 | :desc (StructuredQuery$OrderBy/desc property) 143 | :asc (StructuredQuery$OrderBy/asc property))) 144 | 145 | (defn query 146 | "Query can specify: kind (string) and a sequence of filters. Runner 147 | can be a service or transaction. Only ancestor filters are able to be 148 | run by transactions. 149 | (query service {:kind \"Foo\" 150 | :order [[:desc \"Age\"]] 151 | :filters ['(:= \"Name\" \"Paul\")]}" 152 | [runner {:keys [kind filters order limit offset]} & {:keys [query-type] 153 | :or {query-type :entity}}] 154 | {:pre [(contains? #{:entity :key} query-type)]} 155 | (let [builder (condp = query-type 156 | :entity (Query/entityQueryBuilder) 157 | :key (Query/keyQueryBuilder))] 158 | (when kind 159 | (.kind builder kind)) 160 | (when (seq filters) 161 | (let [fs (->> filters 162 | (map (fn [[expr & args]] 163 | (apply efilter expr args)))) 164 | query-filter (apply query-filters fs)] 165 | (.filter builder query-filter))) 166 | (when-let [os (seq (->> order (map (fn [[direction property]] 167 | (order-by direction property)))))] 168 | (cond (= (count os) 1) (.orderBy builder (first os) (into-array StructuredQuery$OrderBy [])) 169 | :else (.orderBy builder (first os) (into-array StructuredQuery$OrderBy (rest os))))) 170 | (when limit 171 | (.limit builder (int limit))) 172 | (when offset 173 | (.offset builder (int offset))) 174 | (let [query (.build builder)] 175 | (iterator-seq (.run runner query))))) 176 | -------------------------------------------------------------------------------- /datastore/src/java/gclouj/DatastoreOptionsFactory.java: -------------------------------------------------------------------------------- 1 | package gclouj; 2 | 3 | import com.google.cloud.AuthCredentials; 4 | import com.google.cloud.datastore.DatastoreOptions; 5 | 6 | public class DatastoreOptionsFactory { 7 | public static DatastoreOptions create(String projectId, String namespace, AuthCredentials credentials) { 8 | DatastoreOptions.Builder builder = DatastoreOptions.builder() 9 | .projectId(projectId) 10 | .namespace(namespace) 11 | .authCredentials(credentials); 12 | return builder.build(); 13 | 14 | } 15 | 16 | public static DatastoreOptions createTestOptions(String projectId, Integer port) { 17 | String host = "localhost:"+port; 18 | 19 | DatastoreOptions.Builder builder = DatastoreOptions.builder() 20 | .projectId(projectId) 21 | .host(host); 22 | 23 | return builder.build(); 24 | } 25 | } 26 | -------------------------------------------------------------------------------- /datastore/test/datastore_test.clj: -------------------------------------------------------------------------------- 1 | (ns datastore-test 2 | (:import [com.google.cloud.datastore.testing LocalDatastoreHelper] 3 | [com.google.cloud.datastore NullValue Entity Key]) 4 | (:require [gclouj.datastore :refer :all] 5 | [clojure.test :refer :all] 6 | [clojure.walk :as w])) 7 | 8 | (defrecord Person [name]) 9 | 10 | (deftest record-mapping 11 | (let [r (Person. "Paul") 12 | e (entity (incomplete-key "project-id" "Person") (w/stringify-keys r))] 13 | (is (= "Paul" (.getString e "name"))))) 14 | 15 | (defn datastore-helper [] 16 | (let [datastore (doto (LocalDatastoreHelper/create 1.0) 17 | (.start))] 18 | {:datastore datastore 19 | :project-id (.projectId datastore) 20 | :options (.options datastore)})) 21 | 22 | (deftest entity-mapping 23 | (let [testbytes (byte-array [1 2 3 4 5]) 24 | e (entity (incomplete-key "project-id" "Foo") 25 | {"FirstName" "Paul" 26 | "Age" 35 27 | "Clojurist" true 28 | "NilValue" nil 29 | "TestBytes" testbytes 30 | "Intelligence" 50.5 31 | "Address" {"City" "London"} 32 | "Tags" ["Foo" "Bar"]})] 33 | (is (= "Paul" (.getString e "FirstName"))) 34 | (is (= 35 (.getLong e "Age"))) 35 | (is (= 50.5 (.getDouble e "Intelligence"))) 36 | (is (true? (.getBoolean e "Clojurist"))) 37 | (is (instance? NullValue (.getValue e "NilValue"))) 38 | (is (= (seq testbytes) (seq (.toByteArray (.getBlob e "TestBytes"))))) 39 | (is (= "London" (-> e (.getEntity "Address") (.getString "City")))) 40 | (is (= ["Foo" "Bar"] (map to-clojure (.getList e "Tags")))))) 41 | 42 | (deftest put-and-retrieve-entity 43 | (let [{:keys [datastore options project-id]} (datastore-helper)] 44 | (let [s (service options) 45 | t (transaction s)] 46 | (let [added (add-entity t (entity (incomplete-key project-id "Foo") 47 | {"FirstName" "Paul"})) 48 | added-key (.key added) 49 | added2 (add-entity t (entity (incomplete-key project-id "Foo") 50 | {"FirstName" "Peter"})) 51 | added-key2 (.key added2) 52 | added3 (add-entity t (entity (incomplete-key project-id "Foo") 53 | {"FirstName" "Arthur"})) 54 | added-key3 (.key added3)] 55 | (.commit t) 56 | (let [t (transaction s) 57 | fetched (get-entity t added-key)] 58 | (is (= "Paul" (.getString fetched "FirstName"))) 59 | (let [new-entity (Entity/builder fetched)] 60 | (.set new-entity "FirstName" "Arthur")) 61 | (is (= 3 (count (get-entity t added-key added-key2 added-key3)))) 62 | (is (= "Arthur" (-> (get-entity t added-key added-key2 added-key3) 63 | (last) 64 | (.getString "FirstName"))))))) 65 | (.stop datastore))) 66 | 67 | (deftest querying 68 | (let [{:keys [datastore options project-id]} (datastore-helper) 69 | s (-> options (service))] 70 | ;; create some entities to query for 71 | (let [t (transaction s)] 72 | (add-entity t (entity (incomplete-key project-id "QueryFoo") {"Name" "Paul" 73 | "Age" 35})) 74 | (add-entity t (entity (incomplete-key project-id "QueryFoo") {"Name" "Pat" 75 | "Age" 30})) 76 | (.commit t)) 77 | 78 | (let [nothing (query s {:kind "BadKind"}) 79 | found (query s {:kind "QueryFoo" 80 | :filters ['(:= "Name" "Paul")]})] 81 | (is (= 0 (count nothing))) 82 | (is (= 1 (count found))) 83 | (is (= "Paul" (.getString (first found) "Name")))) 84 | 85 | (let [both (query s {:kind "QueryFoo" 86 | :order [[:desc "Age"] [:asc "Name"]]})] 87 | (is (= "Paul" (.getString (first both) "Name"))) 88 | (is (= "Pat" (.getString (last both) "Name")))) 89 | 90 | (is (= "Paul" (-> (query s {:kind "QueryFoo" 91 | :order [[:desc "Age"]] 92 | :limit 1}) 93 | (first) 94 | (.getString "Name")))) 95 | (is (= "Pat" (-> (query s {:kind "QueryFoo" 96 | :order [[:desc "Age"]] 97 | :offset 1 98 | :limit 1}) 99 | (first) 100 | (.getString "Name")))) 101 | (is (instance? Key 102 | (first (query s 103 | {:kind "QueryFoo" 104 | :order [[:desc "Age"]] 105 | :limit 1} 106 | :query-type :key)))) 107 | (.stop datastore))) 108 | 109 | (deftest add-update-delete-entity 110 | (let [{:keys [datastore options project-id]} (datastore-helper) 111 | s (-> options (service))] 112 | 113 | (let [t (transaction s) 114 | added (add-entity t (entity (incomplete-key project-id "QueryFoo") {"Name" "Paul" 115 | "Age" 35 116 | "Location" {"City" "London"}}))] 117 | (.commit t) 118 | (let [t2 (transaction s)] 119 | (is (= {"Name" "Paul" 120 | "Age" 35 121 | "Location" {"City" "London"}} (to-clojure added))) 122 | (update-entity t2 (entity (.key added) 123 | (-> (to-clojure added) 124 | (assoc "Age" 36) 125 | (update-in ["Location" "City"] (constantly "Hong Kong"))))) 126 | (.commit t2)) 127 | 128 | (let [updated (get-entity (transaction s) (.key added))] 129 | (is (= 36 (.getLong updated "Age"))) 130 | (is (= "Hong Kong" (-> updated (.getEntity "Location") (.getString "City"))))) 131 | 132 | (let [t3 (transaction s)] 133 | (delete-entity t3 (.key added)) 134 | (.commit t3))) 135 | 136 | (is (= 0 (count (query s {:kind "QueryFoo"})))) 137 | (.stop datastore))) 138 | -------------------------------------------------------------------------------- /storage/.gitignore: -------------------------------------------------------------------------------- 1 | /target 2 | /classes 3 | /checkouts 4 | pom.xml 5 | pom.xml.asc 6 | *.jar 7 | *.class 8 | /.lein-* 9 | /.nrepl-port 10 | .hgignore 11 | .hg/ 12 | /.idea/ 13 | /storage.iml -------------------------------------------------------------------------------- /storage/project.clj: -------------------------------------------------------------------------------- 1 | (defproject gclouj/storage "0.2.6" 2 | :description "Google Cloud Storage" 3 | :url "https://github.com/pingles/gclouj" 4 | :license {:name "Eclipse Public License" 5 | :url "http://www.eclipse.org/legal/epl-v10.html"} 6 | :dependencies [[org.clojure/clojure "1.8.0"] 7 | [com.google.cloud/gcloud-java-storage "0.2.8" :exclusions [io.netty/netty-codec-http2 io.grpc/grpc-core]] 8 | [clj-time "0.14.0"]] 9 | :source-paths ["src/clojure"] 10 | :java-source-paths ["src/java"] 11 | :profiles {:dev {:global-vars {*warn-on-reflection* true}}}) 12 | -------------------------------------------------------------------------------- /storage/src/clojure/gclouj/storage.clj: -------------------------------------------------------------------------------- 1 | (ns gclouj.storage 2 | (:import [com.google.cloud AuthCredentials] 3 | [com.google.cloud.storage StorageOptions Storage$BlobListOption Storage$BucketListOption Bucket Blob BlobId Storage$SignUrlOption Storage$BlobSourceOption BlobInfo Storage$BlobWriteOption] 4 | [gclouj StorageOptionsFactory] 5 | [java.util.concurrent TimeUnit] 6 | [java.nio.channels Channels]) 7 | (:require [clojure.java.io :as io] 8 | [clj-time.coerce :as tc] 9 | [clj-time.core :as tcc])) 10 | 11 | 12 | (defprotocol ToClojure 13 | (to-clojure [x])) 14 | 15 | (extend-protocol ToClojure 16 | BlobId 17 | (to-clojure [id] {:bucket (.bucket id) 18 | :name (.name id)}) 19 | BlobInfo 20 | (to-clojure [blob] {:id (to-clojure (.blobId blob)) 21 | :cache-control (.cacheControl blob) 22 | :content-disposition (.contentDisposition blob) 23 | :content-encoding (.contentEncoding blob) 24 | :content-language (.contentLanguage blob) 25 | :content-type (.contentType blob) 26 | :crc32c (.crc32c blob) 27 | :generation (.generation blob) 28 | :md5 (.md5 blob) 29 | :metadata (.metadata blob) 30 | :name (.name blob) 31 | :bytes (.size blob) 32 | :media-link (.mediaLink blob) 33 | :updated-at (tc/from-long (.updateTime blob)) 34 | :directory (.isDirectory blob)}) 35 | Bucket 36 | (to-clojure [bucket] {:name (.name bucket) 37 | :created (tc/from-long (.createTime bucket)) 38 | :storage-class (.storageClass bucket) 39 | :location (.location bucket)})) 40 | 41 | (defn credential-options [project-id json-key] 42 | (StorageOptionsFactory/create project-id (AuthCredentials/createForJson (io/input-stream json-key)))) 43 | 44 | (defn service 45 | ([] (.service (StorageOptions/defaultInstance))) 46 | ([^StorageOptions options] (.service options))) 47 | 48 | 49 | (defn buckets [service] 50 | (->> (.iterateAll (.list service (into-array Storage$BucketListOption []))) 51 | (iterator-seq) 52 | (map to-clojure))) 53 | 54 | (defn- blobs* [service bucket options] 55 | (->> (.iterateAll (.list service bucket (into-array Storage$BlobListOption options))) 56 | (iterator-seq) 57 | (map to-clojure))) 58 | 59 | (defn blobs 60 | ([service bucket] 61 | (blobs* service bucket [])) 62 | ([service bucket prefix] 63 | (blobs* service bucket [(Storage$BlobListOption/prefix prefix)]))) 64 | 65 | 66 | (defn signed-url 67 | "Generates a signed URL for the blob, valid for the length of the 68 | period specified. Period should be a Joda 69 | period,e.g.: (clj-time.core/minutes 5)" 70 | [service {:keys [bucket name] :as blob-id} period] 71 | (when-let [blob (.get service (BlobId/of bucket name))] 72 | (.signUrl blob (tcc/in-seconds period) TimeUnit/SECONDS (into-array Storage$SignUrlOption [])))) 73 | 74 | (defn content-stream 75 | [service {:keys [bucket name] :as blob-id}] 76 | (when-let [reader (.reader service (BlobId/of bucket name) (into-array Storage$BlobSourceOption []))] 77 | (Channels/newInputStream reader))) 78 | 79 | 80 | (defn- blob-info [{:keys [bucket name cache-control content-disposition content-encoding content-language content-type metadata]}] 81 | (let [b (BlobInfo/builder (BlobId/of bucket name))] 82 | (when cache-control 83 | (.cacheControl b cache-control)) 84 | (when content-disposition 85 | (.contentDisposition b content-disposition)) 86 | (when content-encoding 87 | (.contentEncoding b content-encoding)) 88 | (when content-language 89 | (.contentLanguage b content-language)) 90 | (when content-type 91 | (.contentType b content-type)) 92 | (when metadata 93 | (.metadata b metadata)) 94 | (.build b))) 95 | 96 | (defn- write-options [options] 97 | (let [opts {:does-not-exist (Storage$BlobWriteOption/doesNotExist)}] 98 | (->> options (map opts) (into-array Storage$BlobWriteOption)))) 99 | 100 | (defn create-blob 101 | "Creates a blob. blob should be a map with at least :bucket 102 | and :name (for other keys, see blob-info fn). 103 | Will succeed even if blob already exists- to fail if the blob already exists, " 104 | [service blob input-stream & opts] 105 | (to-clojure (.create service (blob-info blob) input-stream (write-options opts)))) 106 | 107 | 108 | (defn delete-blob 109 | "Deletes a blob. Returns boolean indicating if it was deleted 110 | successfully." 111 | [service {:keys [bucket name] :as blob-id}] 112 | (.delete service (BlobId/of bucket name))) 113 | -------------------------------------------------------------------------------- /storage/src/java/gclouj/StorageOptionsFactory.java: -------------------------------------------------------------------------------- 1 | package gclouj; 2 | 3 | import com.google.cloud.AuthCredentials; 4 | import com.google.cloud.storage.StorageOptions; 5 | 6 | public class StorageOptionsFactory { 7 | public static StorageOptions create(String projectId, AuthCredentials credentials) { 8 | StorageOptions.Builder builder = StorageOptions.builder() 9 | .projectId(projectId) 10 | .authCredentials(credentials); 11 | return builder.build(); 12 | } 13 | } 14 | --------------------------------------------------------------------------------