├── .gitignore ├── .travis.yml ├── LICENSE ├── README.md ├── changelog ├── pom.xml ├── src ├── main │ ├── java │ │ └── edu │ │ │ └── jhuapl │ │ │ └── tinkerpop │ │ │ ├── AccumuloBulkIngester.java │ │ │ ├── AccumuloByteSerializer.java │ │ │ ├── AccumuloEdge.java │ │ │ ├── AccumuloElement.java │ │ │ ├── AccumuloFeatures.java │ │ │ ├── AccumuloGraph.java │ │ │ ├── AccumuloGraphConfiguration.java │ │ │ ├── AccumuloGraphException.java │ │ │ ├── AccumuloGraphUtils.java │ │ │ ├── AccumuloIndex.java │ │ │ ├── AccumuloRexsterGraphConfiguration.java │ │ │ ├── AccumuloVertex.java │ │ │ ├── Constants.java │ │ │ ├── GlobalInstances.java │ │ │ ├── ScannerIterable.java │ │ │ ├── cache │ │ │ ├── ElementCache.java │ │ │ ├── ElementCaches.java │ │ │ └── PropertyCache.java │ │ │ ├── mapreduce │ │ │ ├── EdgeInputFormat.java │ │ │ ├── ElementOutputFormat.java │ │ │ ├── MapReduceEdge.java │ │ │ ├── MapReduceElement.java │ │ │ ├── MapReduceVertex.java │ │ │ └── VertexInputFormat.java │ │ │ ├── mutator │ │ │ ├── Mutator.java │ │ │ ├── Mutators.java │ │ │ ├── edge │ │ │ │ ├── BaseEdgeMutator.java │ │ │ │ ├── EdgeEndpointsMutator.java │ │ │ │ └── EdgeMutator.java │ │ │ ├── index │ │ │ │ ├── IndexMetadataMutator.java │ │ │ │ └── IndexValueMutator.java │ │ │ ├── property │ │ │ │ ├── BasePropertyMutator.java │ │ │ │ ├── ClearPropertyMutator.java │ │ │ │ └── WritePropertyMutator.java │ │ │ └── vertex │ │ │ │ └── AddVertexMutator.java │ │ │ ├── parser │ │ │ ├── EdgeIndexParser.java │ │ │ ├── EdgeParser.java │ │ │ ├── ElementIndexParser.java │ │ │ ├── ElementParser.java │ │ │ ├── EntryParser.java │ │ │ ├── IndexedItem.java │ │ │ ├── IndexedItemsListParser.java │ │ │ ├── PropertyParser.java │ │ │ ├── VertexIndexParser.java │ │ │ └── VertexParser.java │ │ │ └── tables │ │ │ ├── BaseTableWrapper.java │ │ │ ├── core │ │ │ ├── EdgeTableWrapper.java │ │ │ ├── ElementTableWrapper.java │ │ │ └── VertexTableWrapper.java │ │ │ └── index │ │ │ ├── BaseIndexValuesTableWrapper.java │ │ │ ├── BaseKeyIndexTableWrapper.java │ │ │ ├── EdgeKeyIndexTableWrapper.java │ │ │ ├── IndexMetadataTableWrapper.java │ │ │ ├── NamedIndexTableWrapper.java │ │ │ └── VertexKeyIndexTableWrapper.java │ └── resources │ │ ├── eclipse-codestyle.xml │ │ └── log4j.properties └── test │ └── java │ └── edu │ └── jhuapl │ └── tinkerpop │ ├── AccumuloBulkIngesterTest.java │ ├── AccumuloElementTest.java │ ├── AccumuloGraphConfigurationTest.java │ ├── AccumuloGraphTest.java │ ├── AccumuloGraphTestUtils.java │ ├── AutoIndexTest.java │ ├── DistributedInstanceTest.java │ ├── ElementCacheTest.java │ ├── ElementPropertyCachingTest.java │ ├── ExtendedAccumuloGraphTest.java │ ├── ExtendedElementTest.java │ ├── MockInstanceTest.java │ ├── PropertyCacheTest.java │ └── mapreduce │ ├── ElementOutputFormatTest.java │ └── InputFormatsTest.java └── table-structure.md /.gitignore: -------------------------------------------------------------------------------- 1 | target/ 2 | .settings/ 3 | .classpath 4 | .project 5 | /target/ 6 | *.swp 7 | -------------------------------------------------------------------------------- /.travis.yml: -------------------------------------------------------------------------------- 1 | language: java 2 | jdk: 3 | - openjdk7 4 | 5 | -------------------------------------------------------------------------------- /changelog: -------------------------------------------------------------------------------- 1 | [2.2 Snapshot] 2 | -- 113 Applied instance name when mock instance is used 3 | -- 114 Made map reduce elements serializable 4 | -- 116 Made MapReduceElement serializable with a tranisent graph 5 | [2.1] 6 | Change Log started 7 | --109,103 Merged Metadata tables 8 | --97 Added range queries 9 | --95 Table structure documentation 10 | --79 All flag for preloading 11 | -------------------------------------------------------------------------------- /pom.xml: -------------------------------------------------------------------------------- 1 | 3 | 4.0.0 4 | edu.jhuapl.tinkerpop 5 | blueprints-accumulo-graph 6 | 0.2.2-SNAPSHOT 7 | blueprints-accumulo-graph 8 | An Accumulo-backed implementation of the Tinkerpop Blueprints graph API. 9 | jar 10 | https://github.com/JHUAPL/AccumuloGraph 11 | 12 | 13 | Apache License, Version 2.0 14 | http://www.apache.org/licenses/LICENSE-2.0.txt 15 | repo 16 | 17 | 18 | 19 | 20 | scm:git:git@github.com:JHUAPL/AccumuloGraph.git 21 | scm:git:git@github.com:JHUAPL/AccumuloGraph.git 22 | git@github.com:JHUAPL/AccumuloGraph.git 23 | 24 | 25 | 26 | 27 | Ryan Webb 28 | webbrl@live.com 29 | https://github.com/webbrl1 30 | 31 | 32 | Sandy Hider 33 | sandy@hider.us 34 | https://github.com/sandyhider 35 | 36 | 37 | 38 | 39 | 1.7 40 | 1.7 41 | UTF-8 42 | 43 | 44 | 45 | 46 | org.apache.accumulo 47 | accumulo-core 48 | 1.5.1 49 | 50 | 51 | org.apache.accumulo 52 | accumulo-minicluster 53 | 1.5.0 54 | 55 | 56 | com.tinkerpop.blueprints 57 | blueprints-core 58 | 2.6.0 59 | 60 | 61 | commons-configuration 62 | commons-configuration 63 | 1.10 64 | 65 | 66 | commons-collections 67 | commons-collections 68 | 3.2.2 69 | 70 | 71 | org.apache.hadoop 72 | hadoop-client 73 | 2.2.0 74 | provided 75 | 76 | 77 | com.tinkerpop.rexster 78 | rexster-server 79 | 2.6.0 80 | provided 81 | 82 | 83 | com.tinkerpop.blueprints 84 | blueprints-test 85 | 2.6.0 86 | test 87 | 88 | 89 | org.apache.hadoop 90 | hadoop-minicluster 91 | 2.2.0 92 | test 93 | 94 | 95 | junit 96 | junit 97 | 4.11 98 | test 99 | 100 | 101 | commons-httpclient 102 | commons-httpclient 103 | 3.1 104 | test 105 | 106 | 107 | 108 | 109 | 110 | ossrh 111 | https://oss.sonatype.org/content/repositories/snapshots 112 | 113 | 114 | ossrh 115 | https://oss.sonatype.org/service/local/staging/deploy/maven2/ 116 | 117 | 118 | 119 | 120 | 121 | release 122 | 123 | 124 | 125 | org.apache.maven.plugins 126 | maven-source-plugin 127 | 2.2.1 128 | 129 | 130 | attach-sources 131 | 132 | jar-no-fork 133 | 134 | 135 | 136 | 137 | 138 | org.apache.maven.plugins 139 | maven-javadoc-plugin 140 | 2.9.1 141 | 142 | 143 | attach-javadocs 144 | 145 | jar 146 | 147 | 148 | 149 | 150 | 151 | org.apache.maven.plugins 152 | maven-gpg-plugin 153 | 1.5 154 | 155 | 156 | sign-artifacts 157 | verify 158 | 159 | sign 160 | 161 | 162 | 163 | 164 | 165 | 166 | 167 | 168 | javadoc 169 | 170 | 171 | 172 | org.apache.maven.plugins 173 | maven-source-plugin 174 | 2.2.1 175 | 176 | 177 | attach-sources 178 | 179 | jar-no-fork 180 | 181 | 182 | 183 | 184 | 185 | org.apache.maven.plugins 186 | maven-javadoc-plugin 187 | 2.9.1 188 | 189 | 190 | attach-javadocs 191 | 192 | jar 193 | 194 | 195 | 196 | 197 | 198 | 199 | 200 | 201 | fatjar 202 | 203 | 204 | 205 | org.apache.maven.plugins 206 | maven-shade-plugin 207 | 2.3 208 | 209 | 210 | package 211 | 212 | shade 213 | 214 | 215 | 216 | 217 | 218 | 219 | 220 | 221 | 222 | 223 | 224 | 225 | org.sonatype.plugins 226 | nexus-staging-maven-plugin 227 | 1.6.2 228 | true 229 | 230 | ossrh 231 | https://oss.sonatype.org/ 232 | true 233 | 234 | 235 | 236 | 237 | 238 | -------------------------------------------------------------------------------- /src/main/java/edu/jhuapl/tinkerpop/AccumuloByteSerializer.java: -------------------------------------------------------------------------------- 1 | /* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory 2 | * 3 | * Licensed under the Apache License, Version 2.0 (the "License"); 4 | * you may not use this file except in compliance with the License. 5 | * You may obtain a copy of the License at 6 | * 7 | * http://www.apache.org/licenses/LICENSE-2.0 8 | * 9 | * Unless required by applicable law or agreed to in writing, software 10 | * distributed under the License is distributed on an "AS IS" BASIS, 11 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | * See the License for the specific language governing permissions and 13 | * limitations under the License. 14 | */ 15 | package edu.jhuapl.tinkerpop; 16 | 17 | import java.io.ByteArrayInputStream; 18 | import java.io.ByteArrayOutputStream; 19 | import java.io.IOException; 20 | import java.io.ObjectInputStream; 21 | import java.io.ObjectOutputStream; 22 | import java.io.Serializable; 23 | import java.util.Date; 24 | 25 | import javax.xml.namespace.QName; 26 | 27 | public final class AccumuloByteSerializer { 28 | 29 | public static final int NULL = 'n'; 30 | 31 | public static final int BYTE = 'b'; 32 | public static final int SHORT = 's'; 33 | public static final int CHARACTER = 'c'; 34 | public static final int INTEGER = 'i'; 35 | public static final int LONG = 'l'; 36 | public static final int FLOAT = 'f'; 37 | public static final int DOUBLE = 'd'; 38 | public static final int BOOLEAN = 'o'; 39 | public static final int DATE = 't'; 40 | public static final int ENUM = 'e'; 41 | public static final int STRING = 'a'; 42 | public static final int SERIALIZABLE = 'x'; 43 | public static final int QNAME = 'q'; 44 | 45 | private AccumuloByteSerializer() { 46 | 47 | } 48 | 49 | private static final ThreadLocal BOUTS = new ThreadLocal() { 50 | @Override 51 | public ByteArrayOutputStream initialValue() { 52 | return new ByteArrayOutputStream(32); 53 | } 54 | }; 55 | 56 | @SuppressWarnings("unchecked") 57 | public static T deserialize(byte[] target) { 58 | if (target[0] == NULL) { 59 | return null; 60 | } 61 | 62 | switch (target[0]) { 63 | case BYTE: 64 | return (T) (Byte) Byte.parseByte(new String(target, 1, target.length - 1)); 65 | case SHORT: 66 | return (T) (Short) Short.parseShort(new String(target, 1, target.length - 1)); 67 | case CHARACTER: 68 | return (T) (Character) new String(target, 1, target.length - 1).charAt(0); 69 | case INTEGER: 70 | return (T) (Integer) Integer.parseInt(new String(target, 1, target.length - 1)); 71 | case LONG: 72 | return (T) (Long) Long.parseLong(new String(target, 1, target.length - 1)); 73 | case FLOAT: 74 | return (T) (Float) Float.parseFloat(new String(target, 1, target.length - 1)); 75 | case DOUBLE: 76 | return (T) (Double) Double.parseDouble(new String(target, 1, target.length - 1)); 77 | case BOOLEAN: 78 | switch (new String(target, 1, 1)) { 79 | case "t": 80 | return (T) Boolean.TRUE; 81 | case "f": 82 | return (T) Boolean.FALSE; 83 | default: 84 | throw new RuntimeException("Unexpected boolean value: " + target[1]); 85 | } 86 | case DATE: 87 | long millis = Long.parseLong(new String(target, 1, target.length - 1)); 88 | return (T) new Date(millis); 89 | case STRING: 90 | return (T) new String(target, 1, target.length - 1); 91 | case QNAME: 92 | return (T) QName.valueOf(new String(target, 1, target.length - 1)); 93 | case ENUM: 94 | try { 95 | String[] s = new String(target, 1, target.length - 1).split(":"); 96 | @SuppressWarnings("rawtypes") 97 | Class clz = (Class) Class.forName(s[0]); 98 | return (T) Enum.valueOf(clz, s[1]); 99 | } catch (ClassNotFoundException cnfe) { 100 | throw new RuntimeException("Unexpected error deserializing object.", cnfe); 101 | } 102 | case SERIALIZABLE: 103 | try { 104 | ByteArrayInputStream bin = new ByteArrayInputStream(target, 1, target.length); 105 | ObjectInputStream ois = new ObjectInputStream(bin); 106 | return (T) ois.readObject(); 107 | } catch (IOException io) { 108 | throw new RuntimeException("Unexpected error deserializing object.", io); 109 | } catch (ClassNotFoundException cnfe) { 110 | throw new RuntimeException("Unexpected error deserializing object.", cnfe); 111 | } 112 | case NULL: 113 | return null; 114 | default: 115 | throw new RuntimeException("Unexpected data type: " + (char) target[0]); 116 | } 117 | } 118 | 119 | public static byte[] serialize(Object o) { 120 | ByteArrayOutputStream bout = BOUTS.get(); 121 | 122 | try { 123 | if (o == null) { 124 | bout.write(NULL); 125 | return bout.toByteArray(); 126 | } 127 | 128 | String val = o.toString(); 129 | int type = -1; 130 | String cls = o.getClass().getSimpleName(); 131 | switch (cls) { 132 | case "Byte": 133 | type = BYTE; 134 | break; 135 | case "Short": 136 | type = SHORT; 137 | break; 138 | case "Character": 139 | type = CHARACTER; 140 | break; 141 | case "Integer": 142 | type = INTEGER; 143 | break; 144 | case "Long": 145 | type = LONG; 146 | break; 147 | case "Float": 148 | type = FLOAT; 149 | break; 150 | case "Double": 151 | type = DOUBLE; 152 | break; 153 | case "Boolean": 154 | type = BOOLEAN; 155 | if ((Boolean) o) { 156 | val = "t"; 157 | } else { 158 | val = "f"; 159 | } 160 | break; 161 | case "Date": 162 | val = Long.toString(((Date) o).getTime()); 163 | type = DATE; 164 | break; 165 | case "String": 166 | type = STRING; 167 | break; 168 | case "QName": 169 | type = QNAME; 170 | break; 171 | default: 172 | if (o instanceof Enum) { 173 | type = ENUM; 174 | val = o.getClass().getName() + ":" + val; 175 | } else if (o instanceof Serializable) { 176 | bout.write(SERIALIZABLE); 177 | ObjectOutputStream oos = new ObjectOutputStream(bout); 178 | oos.writeObject(o); 179 | oos.close(); 180 | return bout.toByteArray(); 181 | } else { 182 | throw new RuntimeException("Unsupported data type: " + o.getClass()); 183 | } 184 | } 185 | 186 | bout.write(type); 187 | bout.write(val.getBytes()); 188 | return bout.toByteArray(); 189 | } catch (IOException io) { 190 | throw new RuntimeException("Unexpected error writing to byte array.", io); 191 | } finally { 192 | bout.reset(); 193 | } 194 | } 195 | } 196 | -------------------------------------------------------------------------------- /src/main/java/edu/jhuapl/tinkerpop/AccumuloEdge.java: -------------------------------------------------------------------------------- 1 | /* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory 2 | * 3 | * Licensed under the Apache License, Version 2.0 (the "License"); 4 | * you may not use this file except in compliance with the License. 5 | * You may obtain a copy of the License at 6 | * 7 | * http://www.apache.org/licenses/LICENSE-2.0 8 | * 9 | * Unless required by applicable law or agreed to in writing, software 10 | * distributed under the License is distributed on an "AS IS" BASIS, 11 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | * See the License for the specific language governing permissions and 13 | * limitations under the License. 14 | */ 15 | package edu.jhuapl.tinkerpop; 16 | 17 | import java.util.Map; 18 | import java.util.Map.Entry; 19 | 20 | import org.apache.log4j.Logger; 21 | 22 | import com.tinkerpop.blueprints.Direction; 23 | import com.tinkerpop.blueprints.Edge; 24 | import com.tinkerpop.blueprints.Vertex; 25 | import com.tinkerpop.blueprints.util.StringFactory; 26 | 27 | /** 28 | * TODO 29 | */ 30 | public class AccumuloEdge extends AccumuloElement implements Edge { 31 | 32 | private static final Logger log = Logger.getLogger(AccumuloEdge.class); 33 | 34 | private String label; 35 | private Vertex inVertex; 36 | private Vertex outVertex; 37 | 38 | public AccumuloEdge(GlobalInstances globals, String id) { 39 | this(globals, id, null, null, null); 40 | } 41 | 42 | public AccumuloEdge(GlobalInstances globals, String id, 43 | Vertex inVertex, Vertex outVertex, String label) { 44 | super(globals, id, Edge.class); 45 | this.label = label; 46 | this.inVertex = inVertex; 47 | this.outVertex = outVertex; 48 | } 49 | 50 | @Override 51 | public Vertex getVertex(Direction direction) throws IllegalArgumentException { 52 | if (!Direction.IN.equals(direction) && !Direction.OUT.equals(direction)) { 53 | throw new IllegalArgumentException("Invalid direction: "+direction); 54 | } 55 | 56 | // The vertex information needs to be loaded. 57 | if (inVertex == null || outVertex == null || label == null) { 58 | log.debug("Loading information for edge: "+this); 59 | globals.getEdgeWrapper().loadEndpointsAndLabel(this); 60 | } 61 | 62 | return Direction.IN.equals(direction) ? inVertex : outVertex; 63 | } 64 | 65 | @Override 66 | public String getLabel() { 67 | // TODO less special treatment for "LABEL" property... 68 | if (label != null) { 69 | return label; 70 | } 71 | return getProperty(StringFactory.LABEL); 72 | } 73 | 74 | @Override 75 | public void remove() { 76 | // Remove from named indexes. 77 | super.removeElementFromNamedIndexes(); 78 | 79 | // If edge was removed already, forget it. 80 | // This may happen due to self-loops... 81 | if (!globals.getEdgeWrapper().elementExists(id)) { 82 | return; 83 | } 84 | 85 | // Remove properties from key/value indexes. 86 | Map props = globals.getEdgeWrapper() 87 | .readAllProperties(this); 88 | 89 | for (Entry ents : props.entrySet()) { 90 | globals.getEdgeKeyIndexWrapper().removePropertyFromIndex(this, 91 | ents.getKey(), ents.getValue()); 92 | } 93 | 94 | // Get rid of the endpoints and edge themselves. 95 | globals.getVertexWrapper().deleteEdgeEndpoints(this); 96 | globals.getEdgeWrapper().deleteEdge(this); 97 | 98 | // Remove element from cache. 99 | globals.getCaches().remove(id, Edge.class); 100 | 101 | globals.checkedFlush(); 102 | } 103 | 104 | public void setVertices(AccumuloVertex inVertex, AccumuloVertex outVertex) { 105 | this.inVertex = inVertex; 106 | this.outVertex = outVertex; 107 | } 108 | 109 | public void setLabel(String label) { 110 | this.label = label; 111 | } 112 | 113 | @Override 114 | public String toString() { 115 | return "[" + getId() + ":" + inVertex + " -> " + label + " -> " + outVertex + "]"; 116 | } 117 | } 118 | -------------------------------------------------------------------------------- /src/main/java/edu/jhuapl/tinkerpop/AccumuloElement.java: -------------------------------------------------------------------------------- 1 | /* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory 2 | * 3 | * Licensed under the Apache License, Version 2.0 (the "License"); 4 | * you may not use this file except in compliance with the License. 5 | * You may obtain a copy of the License at 6 | * 7 | * http://www.apache.org/licenses/LICENSE-2.0 8 | * 9 | * Unless required by applicable law or agreed to in writing, software 10 | * distributed under the License is distributed on an "AS IS" BASIS, 11 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | * See the License for the specific language governing permissions and 13 | * limitations under the License. 14 | */ 15 | package edu.jhuapl.tinkerpop; 16 | 17 | import java.util.Set; 18 | 19 | import com.tinkerpop.blueprints.Element; 20 | import com.tinkerpop.blueprints.Index; 21 | import com.tinkerpop.blueprints.util.StringFactory; 22 | 23 | import edu.jhuapl.tinkerpop.cache.PropertyCache; 24 | 25 | /** 26 | * TODO 27 | */ 28 | public abstract class AccumuloElement implements Element { 29 | 30 | protected GlobalInstances globals; 31 | protected String id; 32 | 33 | private Class type; 34 | 35 | private PropertyCache propertyCache; 36 | 37 | protected AccumuloElement(GlobalInstances globals, 38 | String id, Class type) { 39 | this.globals = globals; 40 | this.id = id; 41 | this.type = type; 42 | } 43 | 44 | /** 45 | * Create properties cache if it doesn't exist, 46 | * and preload any properties. 47 | */ 48 | private void makeCache() { 49 | if (propertyCache == null) { 50 | propertyCache = new PropertyCache(globals.getConfig()); 51 | 52 | // Preload any keys, if needed. 53 | String[] preloadKeys = globals.getConfig().getPreloadedProperties(); 54 | if (preloadKeys != null) { 55 | propertyCache.putAll(globals.getElementWrapper(type) 56 | .readProperties(this, preloadKeys)); 57 | } 58 | } 59 | } 60 | 61 | @Override 62 | public T getProperty(String key) { 63 | makeCache(); 64 | 65 | // Get from property cache. 66 | T value = propertyCache.get(key); 67 | 68 | // If not cached, get it from the backing table. 69 | if (value == null) { 70 | value = globals.getElementWrapper(type).readProperty(this, key); 71 | } 72 | 73 | // Cache the new value. 74 | if (value != null) { 75 | propertyCache.put(key, value); 76 | } 77 | 78 | return value; 79 | } 80 | 81 | @Override 82 | public Set getPropertyKeys() { 83 | return globals.getElementWrapper(type).readPropertyKeys(this); 84 | } 85 | 86 | @Override 87 | public void setProperty(String key, Object value) { 88 | makeCache(); 89 | globals.getKeyIndexTableWrapper(type).setPropertyForIndex(this, key, value); 90 | // MDL 31 Dec 2014: The above calls getProperty, so this 91 | // order is important (for now). 92 | globals.getElementWrapper(type).writeProperty(this, key, value); 93 | globals.checkedFlush(); 94 | setPropertyInMemory(key, value); 95 | } 96 | 97 | /** 98 | * Set a property but only in the instantiated object, 99 | * not in the backing store. 100 | * @param key 101 | * @param value 102 | */ 103 | public void setPropertyInMemory(String key, Object value) { 104 | makeCache(); 105 | propertyCache.put(key, value); 106 | } 107 | 108 | @Override 109 | public T removeProperty(String key) { 110 | if (StringFactory.LABEL.equals(key) || 111 | Constants.LABEL.equals(key)) { 112 | throw new AccumuloGraphException("Cannot remove the " + StringFactory.LABEL + " property."); 113 | } 114 | 115 | makeCache(); 116 | T value = getProperty(key); 117 | if (value != null) { 118 | globals.getElementWrapper(type).clearProperty(this, key); 119 | globals.checkedFlush(); 120 | } 121 | globals.getKeyIndexTableWrapper(type).removePropertyFromIndex(this, key, value); 122 | // MDL 31 Dec 2014: AccumuloGraph.removeProperty 123 | // calls getProperty which populates the cache. 124 | // So the order here is important (for now). 125 | removePropertyInMemory(key); 126 | return value; 127 | } 128 | 129 | /** 130 | * Remove element from all named indexes. 131 | * @param element 132 | */ 133 | protected void removeElementFromNamedIndexes() { 134 | for (Index index : globals.getIndexMetadataWrapper().getIndices()) { 135 | ((AccumuloIndex) index).getWrapper().removeElementFromIndex(this); 136 | } 137 | } 138 | 139 | /** 140 | * Remove a property but only in the instantiated 141 | * object, not the backing store. 142 | * @param key 143 | */ 144 | public void removePropertyInMemory(String key) { 145 | makeCache(); 146 | propertyCache.remove(key); 147 | } 148 | 149 | /** 150 | * Return the properties currently cached in memory. 151 | * @return 152 | */ 153 | public Iterable getPropertyKeysInMemory() { 154 | makeCache(); 155 | return propertyCache.keySet(); 156 | } 157 | 158 | /** 159 | * Retrieve a property from memory. 160 | * @param key 161 | * @return 162 | */ 163 | public Object getPropertyInMemory(String key) { 164 | makeCache(); 165 | return propertyCache.get(key); 166 | } 167 | 168 | @Override 169 | public Object getId() { 170 | return id; 171 | } 172 | 173 | @Override 174 | public boolean equals(Object obj) { 175 | if (obj == null) { 176 | return false; 177 | } else if (obj == this) { 178 | return true; 179 | } else if (!obj.getClass().equals(getClass())) { 180 | return false; 181 | } else { 182 | return id.equals(((AccumuloElement) obj).id); 183 | } 184 | } 185 | 186 | @Override 187 | public int hashCode() { 188 | return getClass().hashCode() ^ id.hashCode(); 189 | } 190 | 191 | /** 192 | * Internal method for unit tests. 193 | * @return 194 | */ 195 | PropertyCache getPropertyCache() { 196 | return propertyCache; 197 | } 198 | } 199 | -------------------------------------------------------------------------------- /src/main/java/edu/jhuapl/tinkerpop/AccumuloFeatures.java: -------------------------------------------------------------------------------- 1 | /* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory 2 | * 3 | * Licensed under the Apache License, Version 2.0 (the "License"); 4 | * you may not use this file except in compliance with the License. 5 | * You may obtain a copy of the License at 6 | * 7 | * http://www.apache.org/licenses/LICENSE-2.0 8 | * 9 | * Unless required by applicable law or agreed to in writing, software 10 | * distributed under the License is distributed on an "AS IS" BASIS, 11 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | * See the License for the specific language governing permissions and 13 | * limitations under the License. 14 | */ 15 | package edu.jhuapl.tinkerpop; 16 | 17 | import com.tinkerpop.blueprints.Features; 18 | 19 | /** 20 | * {@link Features} creator. 21 | */ 22 | public class AccumuloFeatures { 23 | 24 | public static Features get() { 25 | Features f = new Features(); 26 | 27 | // For simplicity, I accept all property types. They are handled in not the 28 | // best way. To be fixed later. 29 | f.ignoresSuppliedIds = true; 30 | f.isPersistent = true; 31 | f.isWrapper = false; 32 | f.supportsBooleanProperty = true; 33 | f.supportsDoubleProperty = true; 34 | f.supportsDuplicateEdges = true; 35 | f.supportsEdgeIndex = true; 36 | f.supportsEdgeIteration = true; 37 | f.supportsEdgeRetrieval = true; 38 | f.supportsEdgeKeyIndex = true; 39 | f.supportsEdgeProperties = true; 40 | f.supportsFloatProperty = true; 41 | f.supportsIndices = true; 42 | f.supportsIntegerProperty = true; 43 | f.supportsKeyIndices = true; 44 | f.supportsLongProperty = true; 45 | f.supportsMapProperty = true; 46 | f.supportsMixedListProperty = true; 47 | f.supportsPrimitiveArrayProperty = true; 48 | f.supportsSelfLoops = true; 49 | f.supportsSerializableObjectProperty = true; 50 | f.supportsStringProperty = true; 51 | f.supportsThreadedTransactions = false; 52 | f.supportsTransactions = false; 53 | f.supportsUniformListProperty = true; 54 | f.supportsVertexIndex = true; 55 | f.supportsVertexIteration = true; 56 | f.supportsVertexKeyIndex = true; 57 | f.supportsVertexProperties = true; 58 | f.supportsThreadIsolatedTransactions = false; 59 | 60 | return f; 61 | } 62 | } 63 | -------------------------------------------------------------------------------- /src/main/java/edu/jhuapl/tinkerpop/AccumuloGraphException.java: -------------------------------------------------------------------------------- 1 | /* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory 2 | * 3 | * Licensed under the Apache License, Version 2.0 (the "License"); 4 | * you may not use this file except in compliance with the License. 5 | * You may obtain a copy of the License at 6 | * 7 | * http://www.apache.org/licenses/LICENSE-2.0 8 | * 9 | * Unless required by applicable law or agreed to in writing, software 10 | * distributed under the License is distributed on an "AS IS" BASIS, 11 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | * See the License for the specific language governing permissions and 13 | * limitations under the License. 14 | */ 15 | package edu.jhuapl.tinkerpop; 16 | 17 | public class AccumuloGraphException extends RuntimeException { 18 | 19 | private static final long serialVersionUID = 2179662107592532517L; 20 | 21 | public AccumuloGraphException() { 22 | 23 | } 24 | 25 | public AccumuloGraphException(String reason) { 26 | super(reason); 27 | } 28 | 29 | public AccumuloGraphException(Throwable cause) { 30 | super(cause); 31 | } 32 | 33 | public AccumuloGraphException(String reason, Throwable cause) { 34 | super(reason, cause); 35 | } 36 | } 37 | -------------------------------------------------------------------------------- /src/main/java/edu/jhuapl/tinkerpop/AccumuloGraphUtils.java: -------------------------------------------------------------------------------- 1 | /* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory 2 | * 3 | * Licensed under the Apache License, Version 2.0 (the "License"); 4 | * you may not use this file except in compliance with the License. 5 | * You may obtain a copy of the License at 6 | * 7 | * http://www.apache.org/licenses/LICENSE-2.0 8 | * 9 | * Unless required by applicable law or agreed to in writing, software 10 | * distributed under the License is distributed on an "AS IS" BASIS, 11 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | * See the License for the specific language governing permissions and 13 | * limitations under the License. 14 | */ 15 | package edu.jhuapl.tinkerpop; 16 | 17 | import java.util.SortedSet; 18 | import java.util.UUID; 19 | 20 | import org.apache.accumulo.core.client.admin.TableOperations; 21 | import org.apache.hadoop.io.Text; 22 | 23 | import com.tinkerpop.blueprints.util.ExceptionFactory; 24 | import com.tinkerpop.blueprints.util.StringFactory; 25 | 26 | public final class AccumuloGraphUtils { 27 | 28 | /** 29 | * Create and/or clear existing graph tables for the given configuration. 30 | * 31 | * @param cfg 32 | */ 33 | static void handleCreateAndClear(AccumuloGraphConfiguration cfg) { 34 | try { 35 | TableOperations tableOps = cfg.getConnector().tableOperations(); 36 | 37 | // Track whether tables existed before we do anything. 38 | boolean existedBeforeClear = false; 39 | for (String table : cfg.getTableNames()) { 40 | if (tableOps.exists(table)) { 41 | existedBeforeClear = true; 42 | break; 43 | } 44 | } 45 | 46 | // Check edge cases. 47 | // No tables exist, and we are not allowed to create. 48 | if (!existedBeforeClear && !cfg.getCreate()) { 49 | throw new IllegalArgumentException("Graph does not exist, and create option is disabled"); 50 | } 51 | // Tables exist, and we are not clearing them. 52 | else if (existedBeforeClear && !cfg.getClear()) { 53 | // Do nothing. 54 | return; 55 | } 56 | 57 | // We want to clear tables, so do it. 58 | if (cfg.getClear()) { 59 | for (String table : cfg.getTableNames()) { 60 | if (tableOps.exists(table)) { 61 | tableOps.delete(table); 62 | } 63 | } 64 | } 65 | 66 | // Tables existed, or we want to create them. So do it. 67 | if (existedBeforeClear || cfg.getCreate()) { 68 | for (String table : cfg.getTableNames()) { 69 | if (!tableOps.exists(table)) { 70 | tableOps.create(table); 71 | SortedSet splits = cfg.getSplits(); 72 | if (splits != null) { 73 | tableOps.addSplits(table, splits); 74 | } 75 | } 76 | } 77 | } 78 | 79 | } catch (Exception e) { 80 | throw new IllegalArgumentException(e); 81 | } 82 | } 83 | 84 | /** 85 | * Generate an element id. 86 | * @return 87 | */ 88 | public static String generateId() { 89 | return UUID.randomUUID().toString(); 90 | } 91 | 92 | /** 93 | * Ensure that the given key/value don't conflict with 94 | * Blueprints reserved words. 95 | * @param key 96 | * @param value 97 | */ 98 | public static void validateProperty(String key, Object value) { 99 | nullCheckProperty(key, value); 100 | if (key.equals(StringFactory.ID)) { 101 | throw ExceptionFactory.propertyKeyIdIsReserved(); 102 | } else if (key.equals(StringFactory.LABEL)) { 103 | throw ExceptionFactory.propertyKeyLabelIsReservedForEdges(); 104 | } else if (value == null) { 105 | throw ExceptionFactory.propertyValueCanNotBeNull(); 106 | } 107 | } 108 | 109 | /** 110 | * Disallow null keys/values and throw appropriate 111 | * Blueprints exceptions. 112 | * @param key 113 | * @param value 114 | */ 115 | public static void nullCheckProperty(String key, Object value) { 116 | if (key == null) { 117 | throw ExceptionFactory.propertyKeyCanNotBeNull(); 118 | } else if (value == null) { 119 | throw ExceptionFactory.propertyValueCanNotBeNull(); 120 | } else if (key.trim().equals(StringFactory.EMPTY_STRING)) { 121 | throw ExceptionFactory.propertyKeyCanNotBeEmpty(); 122 | } 123 | } 124 | } 125 | -------------------------------------------------------------------------------- /src/main/java/edu/jhuapl/tinkerpop/AccumuloIndex.java: -------------------------------------------------------------------------------- 1 | /* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory 2 | * 3 | * Licensed under the Apache License, Version 2.0 (the "License"); 4 | * you may not use this file except in compliance with the License. 5 | * You may obtain a copy of the License at 6 | * 7 | * http://www.apache.org/licenses/LICENSE-2.0 8 | * 9 | * Unless required by applicable law or agreed to in writing, software 10 | * distributed under the License is distributed on an "AS IS" BASIS, 11 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | * See the License for the specific language governing permissions and 13 | * limitations under the License. 14 | */ 15 | package edu.jhuapl.tinkerpop; 16 | 17 | import java.util.Iterator; 18 | 19 | import com.tinkerpop.blueprints.CloseableIterable; 20 | import com.tinkerpop.blueprints.Element; 21 | import com.tinkerpop.blueprints.Index; 22 | import com.tinkerpop.blueprints.IndexableGraph; 23 | 24 | import edu.jhuapl.tinkerpop.tables.index.NamedIndexTableWrapper; 25 | 26 | /** 27 | * Accumulo-based implementation for {@link IndexableGraph}. 28 | * @param 29 | */ 30 | public class AccumuloIndex implements Index { 31 | private final GlobalInstances globals; 32 | private final Class indexedType; 33 | private final String indexName; 34 | private final NamedIndexTableWrapper indexWrapper; 35 | 36 | public AccumuloIndex(GlobalInstances globals, String indexName, Class indexedType) { 37 | this.globals = globals; 38 | this.indexName = indexName; 39 | this.indexedType = indexedType; 40 | 41 | try { 42 | if (!globals.getConfig().getConnector() 43 | .tableOperations().exists(getTableName())) { 44 | globals.getConfig().getConnector() 45 | .tableOperations().create(getTableName()); 46 | } 47 | } catch (Exception e) { 48 | throw new AccumuloGraphException(e); 49 | } 50 | 51 | indexWrapper = new NamedIndexTableWrapper(globals, indexedType, indexName); 52 | } 53 | 54 | @Override 55 | public String getIndexName() { 56 | return indexName; 57 | } 58 | 59 | public String getTableName() { 60 | return globals.getConfig().getNamedIndexTableName(indexName); 61 | } 62 | 63 | public NamedIndexTableWrapper getWrapper() { 64 | return indexWrapper; 65 | } 66 | 67 | @Override 68 | public Class getIndexClass() { 69 | return indexedType; 70 | } 71 | 72 | @Override 73 | public void put(String key, Object value, Element element) { 74 | indexWrapper.setPropertyForIndex(element, key, value, true); 75 | } 76 | 77 | @Override 78 | public CloseableIterable get(String key, Object value) { 79 | return indexWrapper.readElementsFromIndex(key, value); 80 | } 81 | 82 | @Override 83 | public CloseableIterable query(String key, Object query) { 84 | throw new UnsupportedOperationException(); 85 | } 86 | 87 | @Override 88 | public long count(String key, Object value) { 89 | CloseableIterable iterable = get(key, value); 90 | Iterator iter = iterable.iterator(); 91 | int count = 0; 92 | while (iter.hasNext()) { 93 | count++; 94 | iter.next(); 95 | } 96 | iterable.close(); 97 | return count; 98 | } 99 | 100 | @Override 101 | public void remove(String key, Object value, Element element) { 102 | indexWrapper.removePropertyFromIndex(element, key, value); 103 | } 104 | } 105 | -------------------------------------------------------------------------------- /src/main/java/edu/jhuapl/tinkerpop/AccumuloRexsterGraphConfiguration.java: -------------------------------------------------------------------------------- 1 | /* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory 2 | * 3 | * Licensed under the Apache License, Version 2.0 (the "License"); 4 | * you may not use this file except in compliance with the License. 5 | * You may obtain a copy of the License at 6 | * 7 | * http://www.apache.org/licenses/LICENSE-2.0 8 | * 9 | * Unless required by applicable law or agreed to in writing, software 10 | * distributed under the License is distributed on an "AS IS" BASIS, 11 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | * See the License for the specific language governing permissions and 13 | * limitations under the License. 14 | */ 15 | package edu.jhuapl.tinkerpop; 16 | 17 | import org.apache.commons.configuration.Configuration; 18 | import org.apache.commons.configuration.HierarchicalConfiguration; 19 | 20 | import com.tinkerpop.blueprints.Graph; 21 | import com.tinkerpop.blueprints.GraphFactory; 22 | import com.tinkerpop.rexster.Tokens; 23 | import com.tinkerpop.rexster.config.GraphConfiguration; 24 | import com.tinkerpop.rexster.config.GraphConfigurationContext; 25 | import com.tinkerpop.rexster.config.GraphConfigurationException; 26 | 27 | import edu.jhuapl.tinkerpop.AccumuloGraphConfiguration.InstanceType; 28 | 29 | public class AccumuloRexsterGraphConfiguration implements GraphConfiguration { 30 | 31 | @Override 32 | public Graph configureGraphInstance(GraphConfigurationContext context) throws GraphConfigurationException { 33 | 34 | Configuration conf = context.getProperties(); 35 | try { 36 | conf = ((HierarchicalConfiguration) conf).configurationAt(Tokens.REXSTER_GRAPH_PROPERTIES); 37 | } catch (IllegalArgumentException iae) { 38 | throw new GraphConfigurationException("Check graph configuration. Missing or empty configuration element: " + Tokens.REXSTER_GRAPH_PROPERTIES); 39 | } 40 | 41 | AccumuloGraphConfiguration cfg = new AccumuloGraphConfiguration(conf); 42 | if (cfg.getInstanceType().equals(InstanceType.Mock)) { 43 | cfg.setPassword("".getBytes()); 44 | cfg.setUser("root"); 45 | } 46 | cfg.setGraphName(context.getProperties().getString(Tokens.REXSTER_GRAPH_NAME)); 47 | return GraphFactory.open(cfg.getConfiguration()); 48 | } 49 | 50 | } 51 | -------------------------------------------------------------------------------- /src/main/java/edu/jhuapl/tinkerpop/AccumuloVertex.java: -------------------------------------------------------------------------------- 1 | /* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory 2 | * 3 | * Licensed under the Apache License, Version 2.0 (the "License"); 4 | * you may not use this file except in compliance with the License. 5 | * You may obtain a copy of the License at 6 | * 7 | * http://www.apache.org/licenses/LICENSE-2.0 8 | * 9 | * Unless required by applicable law or agreed to in writing, software 10 | * distributed under the License is distributed on an "AS IS" BASIS, 11 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | * See the License for the specific language governing permissions and 13 | * limitations under the License. 14 | */ 15 | package edu.jhuapl.tinkerpop; 16 | 17 | import java.util.Map; 18 | import java.util.Map.Entry; 19 | 20 | import com.tinkerpop.blueprints.CloseableIterable; 21 | import com.tinkerpop.blueprints.Direction; 22 | import com.tinkerpop.blueprints.Edge; 23 | import com.tinkerpop.blueprints.Vertex; 24 | import com.tinkerpop.blueprints.VertexQuery; 25 | import com.tinkerpop.blueprints.util.DefaultVertexQuery; 26 | import com.tinkerpop.blueprints.util.ExceptionFactory; 27 | 28 | /** 29 | * TODO 30 | */ 31 | public class AccumuloVertex extends AccumuloElement implements Vertex { 32 | 33 | public AccumuloVertex(GlobalInstances globals, String id) { 34 | super(globals, id, Vertex.class); 35 | } 36 | 37 | @Override 38 | public Iterable getEdges(Direction direction, String... labels) { 39 | return globals.getVertexWrapper().getEdges(this, direction, labels); 40 | } 41 | 42 | @Override 43 | public Iterable getVertices(Direction direction, String... labels) { 44 | return globals.getVertexWrapper().getVertices(this, direction, labels); 45 | } 46 | 47 | @Override 48 | public VertexQuery query() { 49 | return new DefaultVertexQuery(this); 50 | } 51 | 52 | @Override 53 | public Edge addEdge(String label, Vertex inVertex) { 54 | return addEdge(null, label, inVertex); 55 | } 56 | 57 | /** 58 | * Add an edge as with {@link #addEdge(String, Vertex)}, 59 | * but with a specified edge id. 60 | * @param id 61 | * @param label 62 | * @param inVertex 63 | * @return 64 | */ 65 | public Edge addEdge(Object id, String label, Vertex inVertex) { 66 | if (label == null) { 67 | throw ExceptionFactory.edgeLabelCanNotBeNull(); 68 | } 69 | if (id == null) { 70 | id = AccumuloGraphUtils.generateId(); 71 | } 72 | 73 | String myID = id.toString(); 74 | 75 | AccumuloEdge edge = new AccumuloEdge(globals, myID, inVertex, this, label); 76 | 77 | // TODO we arent suppose to make sure the given edge ID doesn't already 78 | // exist? 79 | 80 | globals.getEdgeWrapper().writeEdge(edge); 81 | globals.getVertexWrapper().writeEdgeEndpoints(edge); 82 | 83 | globals.checkedFlush(); 84 | 85 | globals.getCaches().cache(edge, Edge.class); 86 | 87 | return edge; 88 | } 89 | 90 | @Override 91 | public void remove() { 92 | globals.getCaches().remove(getId(), Vertex.class); 93 | 94 | super.removeElementFromNamedIndexes(); 95 | 96 | // Throw exception if the element does not exist. 97 | if (!globals.getVertexWrapper().elementExists(id)) { 98 | throw ExceptionFactory.vertexWithIdDoesNotExist(getId()); 99 | } 100 | 101 | // Remove properties from key/value indexes. 102 | Map props = globals.getVertexWrapper() 103 | .readAllProperties(this); 104 | 105 | for (Entry ent : props.entrySet()) { 106 | globals.getVertexKeyIndexWrapper().removePropertyFromIndex(this, 107 | ent.getKey(), ent.getValue()); 108 | } 109 | 110 | // Remove edges incident to this vertex. 111 | CloseableIterable iter = (CloseableIterable)getEdges(Direction.BOTH); 112 | for (Edge edge : iter) { 113 | edge.remove(); 114 | } 115 | iter.close(); 116 | 117 | globals.checkedFlush(); 118 | 119 | // Get rid of the vertex. 120 | globals.getVertexWrapper().deleteVertex(this); 121 | globals.checkedFlush(); 122 | } 123 | 124 | @Override 125 | public String toString() { 126 | return "[" + getId() + "]"; 127 | } 128 | 129 | } 130 | -------------------------------------------------------------------------------- /src/main/java/edu/jhuapl/tinkerpop/Constants.java: -------------------------------------------------------------------------------- 1 | /* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory 2 | * 3 | * Licensed under the Apache License, Version 2.0 (the "License"); 4 | * you may not use this file except in compliance with the License. 5 | * You may obtain a copy of the License at 6 | * 7 | * http://www.apache.org/licenses/LICENSE-2.0 8 | * 9 | * Unless required by applicable law or agreed to in writing, software 10 | * distributed under the License is distributed on an "AS IS" BASIS, 11 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | * See the License for the specific language governing permissions and 13 | * limitations under the License. 14 | */ 15 | package edu.jhuapl.tinkerpop; 16 | 17 | /** 18 | * Collect up various constants here. 19 | * @author Michael Lieberman 20 | * 21 | */ 22 | public class Constants { 23 | 24 | private Constants() { } 25 | 26 | /** 27 | * Separate element ids in Accumulo entries. 28 | */ 29 | public static final String ID_DELIM = "__DELIM__"; 30 | 31 | public static final byte[] EMPTY = new byte[0]; 32 | 33 | /** 34 | * Prefixes for various Accumulo entries. 35 | */ 36 | public static final String LABEL = "__LABEL__"; 37 | public static final String IN_EDGE = "__IN_EDGE__"; 38 | public static final String OUT_EDGE = "__OUT_EDGE__"; 39 | public static final String EXISTS = "__EXISTS__"; 40 | 41 | /** 42 | * Type of metadata to distinguish between 43 | * entries in the metadata table. 44 | */ 45 | public static enum IndexMetadataEntryType {__INDEX_KEY__, __INDEX_NAME__}; 46 | } 47 | -------------------------------------------------------------------------------- /src/main/java/edu/jhuapl/tinkerpop/GlobalInstances.java: -------------------------------------------------------------------------------- 1 | /* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory 2 | * 3 | * Licensed under the Apache License, Version 2.0 (the "License"); 4 | * you may not use this file except in compliance with the License. 5 | * You may obtain a copy of the License at 6 | * 7 | * http://www.apache.org/licenses/LICENSE-2.0 8 | * 9 | * Unless required by applicable law or agreed to in writing, software 10 | * distributed under the License is distributed on an "AS IS" BASIS, 11 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | * See the License for the specific language governing permissions and 13 | * limitations under the License. 14 | */ 15 | package edu.jhuapl.tinkerpop; 16 | 17 | import org.apache.accumulo.core.client.MultiTableBatchWriter; 18 | import org.apache.accumulo.core.client.MutationsRejectedException; 19 | 20 | import com.tinkerpop.blueprints.Edge; 21 | import com.tinkerpop.blueprints.Element; 22 | import com.tinkerpop.blueprints.Vertex; 23 | 24 | import edu.jhuapl.tinkerpop.cache.ElementCaches; 25 | import edu.jhuapl.tinkerpop.tables.core.EdgeTableWrapper; 26 | import edu.jhuapl.tinkerpop.tables.core.ElementTableWrapper; 27 | import edu.jhuapl.tinkerpop.tables.core.VertexTableWrapper; 28 | import edu.jhuapl.tinkerpop.tables.index.BaseKeyIndexTableWrapper; 29 | import edu.jhuapl.tinkerpop.tables.index.EdgeKeyIndexTableWrapper; 30 | import edu.jhuapl.tinkerpop.tables.index.IndexMetadataTableWrapper; 31 | import edu.jhuapl.tinkerpop.tables.index.VertexKeyIndexTableWrapper; 32 | 33 | /** 34 | * Internal class gathering together instances of 35 | * objects needed for various AccumuloGraph components. 36 | */ 37 | public class GlobalInstances { 38 | 39 | private final AccumuloGraphConfiguration config; 40 | private final MultiTableBatchWriter mtbw; 41 | private final ElementCaches caches; 42 | 43 | public GlobalInstances(AccumuloGraphConfiguration config, 44 | MultiTableBatchWriter mtbw, ElementCaches caches) { 45 | this.config = config; 46 | this.mtbw = mtbw; 47 | this.caches = caches; 48 | } 49 | 50 | public AccumuloGraphConfiguration getConfig() { 51 | return config; 52 | } 53 | 54 | public MultiTableBatchWriter getMtbw() { 55 | return mtbw; 56 | } 57 | 58 | public VertexTableWrapper getVertexWrapper() { 59 | return new VertexTableWrapper(this); 60 | } 61 | 62 | public EdgeTableWrapper getEdgeWrapper() { 63 | return new EdgeTableWrapper(this); 64 | } 65 | 66 | public VertexKeyIndexTableWrapper getVertexKeyIndexWrapper() { 67 | return new VertexKeyIndexTableWrapper(this); 68 | } 69 | 70 | public EdgeKeyIndexTableWrapper getEdgeKeyIndexWrapper() { 71 | return new EdgeKeyIndexTableWrapper(this); 72 | } 73 | 74 | public IndexMetadataTableWrapper getIndexMetadataWrapper() { 75 | return new IndexMetadataTableWrapper(this); 76 | } 77 | 78 | public ElementTableWrapper getElementWrapper(Class clazz) { 79 | if (Vertex.class.equals(clazz)) { 80 | return getVertexWrapper(); 81 | } else if (Edge.class.equals(clazz)) { 82 | return getEdgeWrapper(); 83 | } else { 84 | throw new AccumuloGraphException("Unrecognized class: "+clazz); 85 | } 86 | } 87 | 88 | public BaseKeyIndexTableWrapper getKeyIndexTableWrapper(Class clazz) { 89 | if (Vertex.class.equals(clazz)) { 90 | return getVertexKeyIndexWrapper(); 91 | } else if (Edge.class.equals(clazz)) { 92 | return getEdgeKeyIndexWrapper(); 93 | } else { 94 | throw new AccumuloGraphException("Unrecognized class: "+clazz); 95 | } 96 | } 97 | 98 | public ElementCaches getCaches() { 99 | return caches; 100 | } 101 | 102 | /** 103 | * Flush the writer, if autoflush is enabled. 104 | */ 105 | public void checkedFlush() { 106 | if (config.getAutoFlush()) { 107 | try { 108 | mtbw.flush(); 109 | } catch (MutationsRejectedException e) { 110 | throw new AccumuloGraphException(e); 111 | } 112 | } 113 | } 114 | } 115 | -------------------------------------------------------------------------------- /src/main/java/edu/jhuapl/tinkerpop/ScannerIterable.java: -------------------------------------------------------------------------------- 1 | /* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory 2 | * 3 | * Licensed under the Apache License, Version 2.0 (the "License"); 4 | * you may not use this file except in compliance with the License. 5 | * You may obtain a copy of the License at 6 | * 7 | * http://www.apache.org/licenses/LICENSE-2.0 8 | * 9 | * Unless required by applicable law or agreed to in writing, software 10 | * distributed under the License is distributed on an "AS IS" BASIS, 11 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | * See the License for the specific language governing permissions and 13 | * limitations under the License. 14 | */ 15 | package edu.jhuapl.tinkerpop; 16 | 17 | import java.util.Iterator; 18 | import java.util.Map.Entry; 19 | 20 | import org.apache.accumulo.core.client.ScannerBase; 21 | import org.apache.accumulo.core.data.Key; 22 | import org.apache.accumulo.core.data.Value; 23 | import org.apache.accumulo.core.util.PeekingIterator; 24 | 25 | import com.tinkerpop.blueprints.CloseableIterable; 26 | import com.tinkerpop.blueprints.Element; 27 | 28 | /** 29 | * TODO 30 | */ 31 | public abstract class ScannerIterable implements CloseableIterable { 32 | 33 | private ScannerBase scanner; 34 | 35 | public ScannerIterable(ScannerBase scanner) { 36 | this.scanner = scanner; 37 | } 38 | 39 | @Override 40 | public Iterator iterator() { 41 | return new ScannerIterator(new PeekingIterator>(scanner.iterator())); 42 | } 43 | 44 | public abstract T next(PeekingIterator> iterator); 45 | 46 | @Override 47 | public void close() { 48 | if (scanner != null) { 49 | scanner.close(); 50 | scanner = null; 51 | } 52 | } 53 | 54 | @Override 55 | protected void finalize() { 56 | close(); 57 | } 58 | 59 | private class ScannerIterator implements Iterator { 60 | private PeekingIterator> iterator; 61 | 62 | private ScannerIterator(PeekingIterator> iterator) { 63 | this.iterator = iterator; 64 | } 65 | 66 | @Override 67 | public boolean hasNext() { 68 | return iterator.hasNext(); 69 | } 70 | 71 | @Override 72 | public T next() { 73 | return ScannerIterable.this.next(iterator); 74 | } 75 | 76 | @Override 77 | public void remove() { 78 | throw new UnsupportedOperationException(); 79 | } 80 | } 81 | } 82 | -------------------------------------------------------------------------------- /src/main/java/edu/jhuapl/tinkerpop/cache/ElementCache.java: -------------------------------------------------------------------------------- 1 | /* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory 2 | * 3 | * Licensed under the Apache License, Version 2.0 (the "License"); 4 | * you may not use this file except in compliance with the License. 5 | * You may obtain a copy of the License at 6 | * 7 | * http://www.apache.org/licenses/LICENSE-2.0 8 | * 9 | * Unless required by applicable law or agreed to in writing, software 10 | * distributed under the License is distributed on an "AS IS" BASIS, 11 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | * See the License for the specific language governing permissions and 13 | * limitations under the License. 14 | */ 15 | package edu.jhuapl.tinkerpop.cache; 16 | 17 | import java.util.concurrent.TimeUnit; 18 | 19 | import com.google.common.cache.Cache; 20 | import com.google.common.cache.CacheBuilder; 21 | import com.tinkerpop.blueprints.Element; 22 | 23 | /** 24 | * Simple cache for retrieved graph elements, 25 | * backed by Guava's cache implementation. 26 | */ 27 | public class ElementCache { 28 | 29 | private Cache cache; 30 | 31 | public ElementCache(int size, int timeout) { 32 | cache = CacheBuilder.newBuilder() 33 | .maximumSize(size) 34 | .expireAfterAccess(timeout, TimeUnit.MILLISECONDS) 35 | .build(); 36 | } 37 | 38 | public void cache(T element) { 39 | cache.put(element.getId(), element); 40 | } 41 | 42 | public T retrieve(Object id) { 43 | return cache.getIfPresent(id); 44 | } 45 | 46 | public void remove(Object id) { 47 | cache.invalidate(id); 48 | } 49 | 50 | public void clear() { 51 | cache.invalidateAll(); 52 | } 53 | } 54 | -------------------------------------------------------------------------------- /src/main/java/edu/jhuapl/tinkerpop/cache/ElementCaches.java: -------------------------------------------------------------------------------- 1 | /* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory 2 | * 3 | * Licensed under the Apache License, Version 2.0 (the "License"); 4 | * you may not use this file except in compliance with the License. 5 | * You may obtain a copy of the License at 6 | * 7 | * http://www.apache.org/licenses/LICENSE-2.0 8 | * 9 | * Unless required by applicable law or agreed to in writing, software 10 | * distributed under the License is distributed on an "AS IS" BASIS, 11 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | * See the License for the specific language governing permissions and 13 | * limitations under the License. 14 | */ 15 | package edu.jhuapl.tinkerpop.cache; 16 | 17 | import com.tinkerpop.blueprints.Edge; 18 | import com.tinkerpop.blueprints.Element; 19 | import com.tinkerpop.blueprints.Vertex; 20 | 21 | import edu.jhuapl.tinkerpop.AccumuloGraphConfiguration; 22 | import edu.jhuapl.tinkerpop.AccumuloGraphException; 23 | 24 | /** 25 | * Utility class wrapping element caches. 26 | */ 27 | public class ElementCaches { 28 | private ElementCache vertexCache; 29 | private ElementCache edgeCache; 30 | 31 | public ElementCaches(AccumuloGraphConfiguration config) { 32 | if (config.getVertexCacheEnabled()) { 33 | vertexCache = new ElementCache(config.getVertexCacheSize(), 34 | config.getVertexCacheTimeout()); 35 | } 36 | 37 | if (config.getEdgeCacheEnabled()) { 38 | edgeCache = new ElementCache(config.getEdgeCacheSize(), 39 | config.getEdgeCacheTimeout()); 40 | } 41 | } 42 | 43 | public void cache(T element, Class clazz) { 44 | if (pick(clazz) != null) { 45 | pick(clazz).cache(element); 46 | } 47 | } 48 | 49 | public T retrieve(Object id, Class clazz) { 50 | return pick(clazz) != null ? pick(clazz).retrieve(id) : null; 51 | } 52 | 53 | public void remove(Object id, Class clazz) { 54 | if (pick(clazz) != null) { 55 | pick(clazz).remove(id); 56 | } 57 | } 58 | 59 | public void clear(Class clazz) { 60 | if (pick(clazz) != null) { 61 | pick(clazz).clear(); 62 | } 63 | } 64 | 65 | @SuppressWarnings("unchecked") 66 | private ElementCache pick(Class clazz) { 67 | if (Vertex.class.equals(clazz)) { 68 | return (ElementCache) vertexCache; 69 | } 70 | else if (Edge.class.equals(clazz)) { 71 | return (ElementCache) edgeCache; 72 | } 73 | else { 74 | throw new AccumuloGraphException("Unknown element class: "+clazz); 75 | } 76 | } 77 | } -------------------------------------------------------------------------------- /src/main/java/edu/jhuapl/tinkerpop/cache/PropertyCache.java: -------------------------------------------------------------------------------- 1 | /* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory 2 | * 3 | * Licensed under the Apache License, Version 2.0 (the "License"); 4 | * you may not use this file except in compliance with the License. 5 | * You may obtain a copy of the License at 6 | * 7 | * http://www.apache.org/licenses/LICENSE-2.0 8 | * 9 | * Unless required by applicable law or agreed to in writing, software 10 | * distributed under the License is distributed on an "AS IS" BASIS, 11 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | * See the License for the specific language governing permissions and 13 | * limitations under the License. 14 | */ 15 | package edu.jhuapl.tinkerpop.cache; 16 | 17 | import java.util.HashMap; 18 | import java.util.Map; 19 | import java.util.Map.Entry; 20 | import java.util.Set; 21 | 22 | import edu.jhuapl.tinkerpop.AccumuloGraphConfiguration; 23 | 24 | /** 25 | * Cache for storing element properties. 26 | * This supports a global timeout for key evictions, 27 | * as well as per-property eviction. 28 | * If caching is disabled for the given property, 29 | * this does nothing (stores no values). 30 | */ 31 | public class PropertyCache { 32 | 33 | private final AccumuloGraphConfiguration cfg; 34 | private final Map values; 35 | 36 | public PropertyCache(AccumuloGraphConfiguration cfg) { 37 | this.cfg = cfg; 38 | this.values = new HashMap(); 39 | } 40 | 41 | public boolean containsKey(String key) { 42 | return values.containsKey(key); 43 | } 44 | 45 | public Set keySet() { 46 | return values.keySet(); 47 | } 48 | 49 | public void put(String key, Object value) { 50 | Integer timeout = getTimeout(key); 51 | 52 | // Don't cache anything without a specified timeout. 53 | if (timeout == null) { 54 | return; 55 | } 56 | values.put(key, new TimedValue(value, 57 | System.currentTimeMillis() + timeout)); 58 | } 59 | 60 | public void putAll(Map entries) { 61 | if(entries==null)return; 62 | for (Entry ent : entries.entrySet()) { 63 | put(ent.getKey(), ent.getValue()); 64 | } 65 | } 66 | 67 | @SuppressWarnings("unchecked") 68 | public T get(String key) { 69 | long now = System.currentTimeMillis(); 70 | 71 | TimedValue val = values.get(key); 72 | if (val != null) { 73 | if (val.getExpiry() != null && 74 | val.getExpiry() <= now) { 75 | remove(key); 76 | return null; 77 | } 78 | else { 79 | return (T) val.getValue(); 80 | } 81 | } 82 | 83 | return null; 84 | } 85 | 86 | public void remove(String key) { 87 | values.remove(key); 88 | } 89 | 90 | public void clear() { 91 | values.clear(); 92 | } 93 | 94 | @Override 95 | public String toString() { 96 | return values.toString(); 97 | } 98 | 99 | /** 100 | * Return the timeout for the given key. 101 | * Checks for a key-specific timeout 102 | * first, then the default, if any. 103 | * Return null if no timeout. 104 | * @param key 105 | * @return 106 | */ 107 | private Integer getTimeout(String key) { 108 | int timeout = cfg.getPropertyCacheTimeout(key); 109 | if (timeout <= 0) { 110 | timeout = cfg.getPropertyCacheTimeout(null); 111 | } 112 | return timeout > 0 ? timeout : null; 113 | } 114 | 115 | /** 116 | * Value with associated expiry time. 117 | */ 118 | private static class TimedValue { 119 | private final Object value; 120 | private final Long expiry; 121 | 122 | public TimedValue(Object value, Long expiry) { 123 | this.value = value; 124 | this.expiry = expiry; 125 | } 126 | 127 | public Object getValue() { 128 | return value; 129 | } 130 | 131 | public Long getExpiry() { 132 | return expiry; 133 | } 134 | 135 | @Override 136 | public String toString() { 137 | return "[" + value + ", " + expiry + "]"; 138 | } 139 | } 140 | } 141 | -------------------------------------------------------------------------------- /src/main/java/edu/jhuapl/tinkerpop/mapreduce/EdgeInputFormat.java: -------------------------------------------------------------------------------- 1 | package edu.jhuapl.tinkerpop.mapreduce; 2 | 3 | import java.io.ByteArrayInputStream; 4 | import java.io.DataInputStream; 5 | import java.io.IOException; 6 | import java.util.Iterator; 7 | import java.util.Map.Entry; 8 | 9 | import org.apache.accumulo.core.client.AccumuloException; 10 | import org.apache.accumulo.core.client.AccumuloSecurityException; 11 | import org.apache.accumulo.core.client.RowIterator; 12 | import org.apache.accumulo.core.client.mapreduce.InputFormatBase; 13 | import org.apache.accumulo.core.client.mock.MockInstance; 14 | import org.apache.accumulo.core.client.security.tokens.PasswordToken; 15 | import org.apache.accumulo.core.data.Key; 16 | import org.apache.accumulo.core.data.Value; 17 | import org.apache.hadoop.io.Text; 18 | import org.apache.hadoop.mapreduce.InputSplit; 19 | import org.apache.hadoop.mapreduce.Job; 20 | import org.apache.hadoop.mapreduce.RecordReader; 21 | import org.apache.hadoop.mapreduce.TaskAttemptContext; 22 | 23 | import com.tinkerpop.blueprints.Edge; 24 | 25 | import edu.jhuapl.tinkerpop.AccumuloByteSerializer; 26 | import edu.jhuapl.tinkerpop.AccumuloGraph; 27 | import edu.jhuapl.tinkerpop.AccumuloGraphConfiguration; 28 | import edu.jhuapl.tinkerpop.AccumuloGraphConfiguration.InstanceType; 29 | import edu.jhuapl.tinkerpop.AccumuloGraphException; 30 | import edu.jhuapl.tinkerpop.Constants; 31 | 32 | public class EdgeInputFormat extends InputFormatBase { 33 | 34 | private static final String PREFIX = EdgeInputFormat.class.getSimpleName()+"."; 35 | private static final String GRAPH_NAME = PREFIX+"graph.name"; 36 | 37 | static AccumuloGraphConfiguration conf; 38 | 39 | @Override 40 | public RecordReader createRecordReader(InputSplit split, TaskAttemptContext attempt) throws IOException, InterruptedException { 41 | return new EdgeRecordReader(); 42 | } 43 | 44 | private class EdgeRecordReader extends RecordReaderBase { 45 | 46 | RowIterator rowIterator; 47 | AccumuloGraph parent; 48 | 49 | EdgeRecordReader() {} 50 | 51 | @Override 52 | public void initialize(InputSplit inSplit, TaskAttemptContext attempt) throws IOException { 53 | 54 | super.initialize(inSplit, attempt); 55 | rowIterator = new RowIterator(scannerIterator); 56 | 57 | currentK = new Text(); 58 | 59 | try { 60 | conf = new AccumuloGraphConfiguration(); 61 | conf.setZooKeeperHosts(EdgeInputFormat.getInstance(attempt).getZooKeepers()); 62 | conf.setInstanceName(EdgeInputFormat.getInstance(attempt).getInstanceName()); 63 | conf.setUser(EdgeInputFormat.getPrincipal(attempt)); 64 | conf.setTokenWithFallback(EdgeInputFormat.getToken(attempt)); 65 | conf.setGraphName(attempt.getConfiguration().get(GRAPH_NAME)); 66 | if (EdgeInputFormat.getInstance(attempt) instanceof MockInstance) { 67 | conf.setInstanceType(InstanceType.Mock); 68 | } 69 | parent = AccumuloGraph.open(conf.getConfiguration()); 70 | } catch (AccumuloException e) { 71 | throw new AccumuloGraphException(e); 72 | } 73 | 74 | } 75 | 76 | @Override 77 | public boolean nextKeyValue() throws IOException, InterruptedException { 78 | if (rowIterator.hasNext()) { 79 | Iterator> it = rowIterator.next(); 80 | 81 | MapReduceEdge edge = new MapReduceEdge(parent); 82 | while (it.hasNext()) { 83 | Entry entry = it.next(); 84 | numKeysRead++; 85 | 86 | currentKey = entry.getKey(); 87 | String eid = currentKey.getRow().toString(); 88 | String colf = currentKey.getColumnFamily().toString(); 89 | switch (colf) { 90 | case Constants.LABEL: 91 | currentK.set(eid); 92 | edge.prepareId(eid); 93 | String[] ids = currentKey.getColumnQualifier().toString().split(Constants.ID_DELIM); 94 | edge.setSourceId(ids[1]); 95 | edge.setDestId(ids[0]); 96 | edge.setLabel(AccumuloByteSerializer.deserialize(entry.getValue().get()).toString()); 97 | break; 98 | default: 99 | String propertyKey = currentKey.getColumnFamily().toString(); 100 | Object propertyValue = AccumuloByteSerializer.deserialize(entry.getValue().get()); 101 | edge.prepareProperty(propertyKey, propertyValue); 102 | } 103 | } 104 | currentV = edge; 105 | return true; 106 | } 107 | return false; 108 | } 109 | 110 | } 111 | 112 | public static void setAccumuloGraphConfiguration(Job job, AccumuloGraphConfiguration cfg) throws AccumuloSecurityException { 113 | 114 | EdgeInputFormat.setConnectorInfo(job, cfg.getUser(), new PasswordToken(cfg.getPassword())); 115 | EdgeInputFormat.setInputTableName(job, cfg.getEdgeTableName()); 116 | if (cfg.getInstanceType().equals(InstanceType.Mock)) { 117 | EdgeInputFormat.setMockInstance(job, cfg.getInstanceName()); 118 | } else { 119 | EdgeInputFormat.setZooKeeperInstance(job, cfg.getInstanceName(), cfg.getZooKeeperHosts()); 120 | } 121 | job.getConfiguration().set(GRAPH_NAME, cfg.getGraphName()); 122 | 123 | } 124 | 125 | } 126 | -------------------------------------------------------------------------------- /src/main/java/edu/jhuapl/tinkerpop/mapreduce/ElementOutputFormat.java: -------------------------------------------------------------------------------- 1 | package edu.jhuapl.tinkerpop.mapreduce; 2 | 3 | import java.io.IOException; 4 | import java.io.Serializable; 5 | import java.util.Map.Entry; 6 | 7 | import org.apache.accumulo.core.client.AccumuloException; 8 | import org.apache.accumulo.core.client.AccumuloSecurityException; 9 | import org.apache.accumulo.core.client.BatchWriter; 10 | import org.apache.accumulo.core.client.MutationsRejectedException; 11 | import org.apache.accumulo.core.client.TableNotFoundException; 12 | import org.apache.accumulo.core.client.mapreduce.AccumuloOutputFormat; 13 | import org.apache.accumulo.core.data.Mutation; 14 | import org.apache.hadoop.conf.Configuration; 15 | import org.apache.hadoop.io.NullWritable; 16 | import org.apache.hadoop.io.Text; 17 | import org.apache.hadoop.mapreduce.Job; 18 | import org.apache.hadoop.mapreduce.JobContext; 19 | import org.apache.hadoop.mapreduce.OutputCommitter; 20 | import org.apache.hadoop.mapreduce.OutputFormat; 21 | import org.apache.hadoop.mapreduce.RecordWriter; 22 | import org.apache.hadoop.mapreduce.TaskAttemptContext; 23 | import org.apache.hadoop.mapreduce.lib.output.NullOutputFormat; 24 | 25 | import com.tinkerpop.blueprints.Element; 26 | 27 | import edu.jhuapl.tinkerpop.AccumuloByteSerializer; 28 | import edu.jhuapl.tinkerpop.AccumuloGraphConfiguration; 29 | import edu.jhuapl.tinkerpop.AccumuloGraphException; 30 | import edu.jhuapl.tinkerpop.AccumuloGraphConfiguration.InstanceType; 31 | 32 | public class ElementOutputFormat extends OutputFormat { 33 | 34 | private static final String PREFIX = ElementOutputFormat.class.getSimpleName()+"."; 35 | private static final String USER = "username"; 36 | private static final String PASSWORD = PREFIX+"password"; 37 | private static final String GRAPH_NAME = PREFIX+"graphName"; 38 | private static final String INSTANCE = PREFIX+"instanceName"; 39 | private static final String INSTANCE_TYPE = PREFIX+"instanceType"; 40 | private static final String ZK_HOSTS = PREFIX+"zookeeperHosts"; 41 | 42 | @Override 43 | public RecordWriter getRecordWriter(TaskAttemptContext context) throws IOException, InterruptedException { 44 | return new ElementRecordWriter(context); 45 | } 46 | 47 | @Override 48 | public void checkOutputSpecs(JobContext context) throws IOException, InterruptedException { 49 | 50 | } 51 | 52 | public static void setAccumuloGraphConfiguration(Job job, AccumuloGraphConfiguration acc) { 53 | acc.validate(); 54 | Configuration jobconf = job.getConfiguration(); 55 | 56 | jobconf.set(USER, acc.getUser()); 57 | jobconf.set(PASSWORD, acc.getPassword()); 58 | jobconf.set(GRAPH_NAME, acc.getGraphName()); 59 | jobconf.set(INSTANCE, acc.getInstanceName()); 60 | jobconf.set(INSTANCE_TYPE, acc.getInstanceType().toString()); 61 | if(acc.getInstanceType().equals(InstanceType.Distributed)) 62 | jobconf.set(ZK_HOSTS, acc.getZooKeeperHosts()); 63 | } 64 | 65 | /** 66 | * @see AccumuloOutputFormat 67 | */ 68 | // TODO I think we can implement this to provide a little more robustness. 69 | @Override 70 | public OutputCommitter getOutputCommitter(TaskAttemptContext context) { 71 | return new NullOutputFormat().getOutputCommitter(context); 72 | } 73 | 74 | static class ElementRecordWriter extends RecordWriter implements Serializable{ 75 | AccumuloGraphConfiguration config; 76 | 77 | public ElementRecordWriter(TaskAttemptContext context) { 78 | config = new AccumuloGraphConfiguration(); 79 | Configuration jobconf = context.getConfiguration(); 80 | config.setUser(jobconf.get(USER)); 81 | config.setPassword(jobconf.get(PASSWORD)); 82 | config.setGraphName(jobconf.get(GRAPH_NAME)); 83 | config.setInstanceName(jobconf.get(INSTANCE)); 84 | config.setInstanceType(InstanceType.valueOf(jobconf.get(INSTANCE_TYPE))); 85 | config.setZooKeeperHosts(jobconf.get(ZK_HOSTS)); 86 | 87 | } 88 | 89 | transient BatchWriter bw; 90 | 91 | @Override 92 | public void write(NullWritable key, Element value) throws IOException, InterruptedException { 93 | MapReduceElement ele = (MapReduceElement) value; 94 | try { 95 | if (bw == null) { 96 | if (ele instanceof MapReduceVertex) { 97 | bw = config.getConnector().createBatchWriter(config.getVertexTableName(), config.getBatchWriterConfig()); 98 | } else { 99 | bw = config.getConnector().createBatchWriter(config.getEdgeTableName(), config.getBatchWriterConfig()); 100 | } 101 | } 102 | 103 | Mutation mut = new Mutation(ele.id); 104 | for (Entry map : ele.getNewProperties().entrySet()) { 105 | mut.put(map.getKey().getBytes(), "".getBytes(), AccumuloByteSerializer.serialize(map.getValue())); 106 | } 107 | 108 | bw.addMutation(mut); 109 | } catch (TableNotFoundException | AccumuloException | AccumuloSecurityException e) { 110 | throw new AccumuloGraphException(e); 111 | } 112 | } 113 | 114 | @Override 115 | public void close(TaskAttemptContext context) throws IOException, InterruptedException { 116 | if (bw != null) { 117 | try { 118 | bw.close(); 119 | } catch (MutationsRejectedException e) { 120 | throw new AccumuloGraphException(e); 121 | } 122 | } 123 | } 124 | 125 | } 126 | 127 | } 128 | -------------------------------------------------------------------------------- /src/main/java/edu/jhuapl/tinkerpop/mapreduce/MapReduceEdge.java: -------------------------------------------------------------------------------- 1 | /* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory 2 | * 3 | * Licensed under the Apache License, Version 2.0 (the "License"); 4 | * you may not use this file except in compliance with the License. 5 | * You may obtain a copy of the License at 6 | * 7 | * http://www.apache.org/licenses/LICENSE-2.0 8 | * 9 | * Unless required by applicable law or agreed to in writing, software 10 | * distributed under the License is distributed on an "AS IS" BASIS, 11 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | * See the License for the specific language governing permissions and 13 | * limitations under the License. 14 | */ 15 | package edu.jhuapl.tinkerpop.mapreduce; 16 | 17 | import java.io.Serializable; 18 | 19 | import com.tinkerpop.blueprints.Direction; 20 | import com.tinkerpop.blueprints.Edge; 21 | import com.tinkerpop.blueprints.Vertex; 22 | import com.tinkerpop.blueprints.util.ExceptionFactory; 23 | 24 | import edu.jhuapl.tinkerpop.AccumuloGraph; 25 | 26 | public class MapReduceEdge extends MapReduceElement implements Edge, Serializable { 27 | 28 | String sourceId; 29 | String label; 30 | String destinationId; 31 | 32 | MapReduceEdge(AccumuloGraph parent) { 33 | super(parent); 34 | } 35 | 36 | MapReduceEdge() { super(); } 37 | 38 | void setSourceId(String id) { 39 | sourceId = id; 40 | } 41 | 42 | void setDestId(String id) { 43 | destinationId = id; 44 | } 45 | 46 | MapReduceEdge(AccumuloGraph parent, String id, String src, String label, String dest) { 47 | super(parent); 48 | prepareId(id); 49 | this.sourceId = src; 50 | this.label = label; 51 | this.destinationId = dest; 52 | } 53 | 54 | @Override 55 | public String getLabel() { 56 | return label; 57 | } 58 | 59 | @Override 60 | public Vertex getVertex(Direction direction) throws IllegalArgumentException { 61 | return parent.getVertex(getVertexId(direction)); 62 | } 63 | 64 | public String getVertexId(Direction direction) throws IllegalArgumentException { 65 | switch (direction) { 66 | case IN: 67 | return destinationId; 68 | case OUT: 69 | return sourceId; 70 | case BOTH: 71 | default: 72 | throw ExceptionFactory.bothIsNotSupported(); 73 | } 74 | } 75 | 76 | public void setLabel(String label) { 77 | this.label = label; 78 | } 79 | 80 | } 81 | -------------------------------------------------------------------------------- /src/main/java/edu/jhuapl/tinkerpop/mapreduce/MapReduceElement.java: -------------------------------------------------------------------------------- 1 | /* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory 2 | * 3 | * Licensed under the Apache License, Version 2.0 (the "License"); 4 | * you may not use this file except in compliance with the License. 5 | * You may obtain a copy of the License at 6 | * 7 | * http://www.apache.org/licenses/LICENSE-2.0 8 | * 9 | * Unless required by applicable law or agreed to in writing, software 10 | * distributed under the License is distributed on an "AS IS" BASIS, 11 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | * See the License for the specific language governing permissions and 13 | * limitations under the License. 14 | */ 15 | package edu.jhuapl.tinkerpop.mapreduce; 16 | 17 | import java.io.DataInput; 18 | import java.io.DataOutput; 19 | import java.io.IOException; 20 | import java.io.Serializable; 21 | import java.util.HashMap; 22 | import java.util.HashSet; 23 | import java.util.Map; 24 | import java.util.Set; 25 | 26 | import org.apache.hadoop.io.WritableComparable; 27 | 28 | import com.google.common.collect.Sets; 29 | import com.tinkerpop.blueprints.Element; 30 | import com.tinkerpop.blueprints.Graph; 31 | 32 | import edu.jhuapl.tinkerpop.AccumuloByteSerializer; 33 | import edu.jhuapl.tinkerpop.AccumuloGraph; 34 | 35 | public abstract class MapReduceElement implements Serializable, Element, WritableComparable { 36 | 37 | protected String id; 38 | 39 | protected Map properties; 40 | 41 | protected Map newProperties; 42 | 43 | transient AccumuloGraph parent; 44 | 45 | MapReduceElement(AccumuloGraph parent) { 46 | this(); 47 | this.parent = parent; 48 | } 49 | 50 | MapReduceElement(){ 51 | properties = new HashMap(); 52 | newProperties = new HashMap(); 53 | } 54 | 55 | void prepareId(String id) { 56 | this.id = id; 57 | } 58 | 59 | void prepareProperty(String key, Object property) { 60 | properties.put(key, property); 61 | } 62 | 63 | Map getNewProperties() { 64 | return newProperties; 65 | } 66 | 67 | @Override 68 | public Object getId() { 69 | return id; 70 | } 71 | 72 | @SuppressWarnings("unchecked") 73 | @Override 74 | public T getProperty(String key) { 75 | 76 | Object newProp = newProperties.get(key); 77 | if (newProp != null) 78 | return (T) newProp; 79 | return (T) properties.get(key); 80 | } 81 | 82 | @Override 83 | public Set getPropertyKeys() { 84 | return Sets.union(new HashSet(properties.keySet()), new HashSet(newProperties.keySet())); 85 | } 86 | 87 | @Override 88 | public void remove() { 89 | throw new UnsupportedOperationException("You cannot remove an element during a MapReduce job."); 90 | } 91 | 92 | @Override 93 | public T removeProperty(String key) { 94 | throw new UnsupportedOperationException("You cannot modify an element during a MapReduce job."); 95 | } 96 | 97 | @Override 98 | public void setProperty(String key, Object value) { 99 | newProperties.put(key, value); 100 | } 101 | 102 | protected Graph getParent() { 103 | return parent; 104 | } 105 | 106 | @Override 107 | public void readFields(DataInput in) throws IOException { 108 | prepareId(in.readUTF()); 109 | 110 | properties.clear(); 111 | int count = in.readInt(); 112 | for (int i = 0; i < count; i++) { 113 | String key = in.readUTF(); 114 | byte[] data = new byte[in.readInt()]; 115 | in.readFully(data); 116 | Object val = AccumuloByteSerializer.deserialize(data); 117 | properties.put(key, val); 118 | } 119 | 120 | count = in.readInt(); 121 | for (int i = 0; i < count; i++) { 122 | String key = in.readUTF(); 123 | byte[] data = new byte[in.readInt()]; 124 | in.readFully(data); 125 | Object val = AccumuloByteSerializer.deserialize(data); 126 | newProperties.put(key, val); 127 | } 128 | 129 | } 130 | 131 | @Override 132 | public void write(DataOutput out) throws IOException { 133 | out.writeUTF(id); 134 | out.writeInt(properties.size()); 135 | for (String key : properties.keySet()) { 136 | out.writeUTF(key); 137 | byte[] data = AccumuloByteSerializer.serialize(properties.get(key)); 138 | out.writeInt(data.length); 139 | out.write(data); 140 | } 141 | 142 | for (String key : newProperties.keySet()) { 143 | out.writeUTF(key); 144 | byte[] data = AccumuloByteSerializer.serialize(newProperties.get(key)); 145 | out.writeInt(data.length); 146 | out.write(data); 147 | } 148 | } 149 | 150 | @Override 151 | public int compareTo(MapReduceElement other) { 152 | int val = this.id.compareTo(other.id); 153 | if (val != 0) { 154 | return val; 155 | } 156 | return getClass().getSimpleName().compareTo(other.getClass().getSimpleName()); 157 | } 158 | 159 | public boolean equals(Object object) { 160 | if (object == this) { 161 | return true; 162 | } else if (object == null) { 163 | return false; 164 | } else if (!object.getClass().equals(getClass())) { 165 | return false; 166 | } else { 167 | Element element = (Element) object; 168 | if (id == null) { 169 | return element.getId() == null; 170 | } else { 171 | return id.equals(element.getId()); 172 | } 173 | } 174 | } 175 | 176 | public int hashCode() { 177 | int hash = 31 * getClass().hashCode(); 178 | if (id != null) { 179 | hash ^= id.hashCode(); 180 | } 181 | return hash; 182 | } 183 | 184 | } 185 | -------------------------------------------------------------------------------- /src/main/java/edu/jhuapl/tinkerpop/mapreduce/MapReduceVertex.java: -------------------------------------------------------------------------------- 1 | /* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory 2 | * 3 | * Licensed under the Apache License, Version 2.0 (the "License"); 4 | * you may not use this file except in compliance with the License. 5 | * You may obtain a copy of the License at 6 | * 7 | * http://www.apache.org/licenses/LICENSE-2.0 8 | * 9 | * Unless required by applicable law or agreed to in writing, software 10 | * distributed under the License is distributed on an "AS IS" BASIS, 11 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | * See the License for the specific language governing permissions and 13 | * limitations under the License. 14 | */ 15 | package edu.jhuapl.tinkerpop.mapreduce; 16 | 17 | import java.io.DataInput; 18 | import java.io.DataOutput; 19 | import java.io.IOException; 20 | import java.io.Serializable; 21 | import java.util.Arrays; 22 | import java.util.Collection; 23 | import java.util.HashSet; 24 | import java.util.Iterator; 25 | import java.util.LinkedList; 26 | import java.util.List; 27 | import java.util.Set; 28 | 29 | import com.tinkerpop.blueprints.Direction; 30 | import com.tinkerpop.blueprints.Edge; 31 | import com.tinkerpop.blueprints.Vertex; 32 | import com.tinkerpop.blueprints.VertexQuery; 33 | import com.tinkerpop.blueprints.util.DefaultVertexQuery; 34 | 35 | import edu.jhuapl.tinkerpop.AccumuloGraph; 36 | 37 | public class MapReduceVertex extends MapReduceElement implements Vertex, Serializable { 38 | 39 | List inEdges; 40 | List outEdges; 41 | 42 | MapReduceVertex(AccumuloGraph parent) { 43 | super(parent); 44 | inEdges = new LinkedList(); 45 | outEdges = new LinkedList(); 46 | } 47 | 48 | MapReduceVertex(){ 49 | //super(); 50 | inEdges = new LinkedList(); 51 | outEdges = new LinkedList(); 52 | } 53 | 54 | MapReduceEdge prepareEdge(String id, String src, String label, String dest) { 55 | MapReduceEdge mre = new MapReduceEdge(parent, id, src, label, dest); 56 | if (src.equals(getId())) { 57 | outEdges.add(mre); 58 | } 59 | // maybe this could be an "else if"-- do we support self-referential 60 | // edges?? 61 | if (dest.equals(getId())) { 62 | inEdges.add(mre); 63 | } 64 | return mre; 65 | } 66 | 67 | @Override 68 | public Edge addEdge(String label, Vertex inVertex) { 69 | throw new UnsupportedOperationException("You cannot modify a vertex during a MapReduce job."); 70 | } 71 | 72 | @Override 73 | public Iterable getEdges(Direction direction, String... labels) { 74 | switch (direction) { 75 | case IN: 76 | return getEdges(inEdges, labels); 77 | case OUT: 78 | return getEdges(outEdges, labels); 79 | case BOTH: 80 | Set all = new HashSet(inEdges.size() + outEdges.size()); 81 | all.addAll(inEdges); 82 | all.addAll(outEdges); 83 | return getEdges(all, labels); 84 | default: 85 | throw new RuntimeException("Unexpected direction: " + direction); 86 | } 87 | } 88 | 89 | private Iterable getEdges(Collection edges, String... labels) { 90 | if (labels.length > 0) { 91 | List filters = Arrays.asList(labels); 92 | LinkedList filteredEdges = new LinkedList(); 93 | for (Edge e : edges) { 94 | if (filters.contains(e.getLabel())) { 95 | filteredEdges.add(e); 96 | } 97 | } 98 | return filteredEdges; 99 | } else { 100 | return edges; 101 | } 102 | } 103 | 104 | @Override 105 | public Iterable getVertices(final Direction direction, final String... labels) { 106 | return new Iterable() { 107 | @Override 108 | public Iterator iterator() { 109 | final Iterator edges = getEdges(direction, labels).iterator(); 110 | 111 | return new Iterator() { 112 | 113 | @Override 114 | public boolean hasNext() { 115 | return edges.hasNext(); 116 | } 117 | 118 | @Override 119 | public Vertex next() { 120 | Edge e = edges.next(); 121 | switch (direction) { 122 | case IN: 123 | // no break intentional 124 | case OUT: 125 | return e.getVertex(direction); 126 | case BOTH: 127 | Vertex v = e.getVertex(Direction.IN); 128 | if (this.equals(v)) { 129 | return e.getVertex(Direction.OUT); 130 | } else { 131 | return v; 132 | } 133 | default: 134 | throw new RuntimeException("Unexpected direction: " + direction); 135 | } 136 | } 137 | 138 | @Override 139 | public void remove() { 140 | throw new UnsupportedOperationException(); 141 | } 142 | }; 143 | }; 144 | }; 145 | } 146 | 147 | @Override 148 | public VertexQuery query() { 149 | return new DefaultVertexQuery(getParent().getVertex(getId())); 150 | } 151 | 152 | @Override 153 | public void readFields(DataInput in) throws IOException { 154 | super.readFields(in); 155 | 156 | inEdges.clear(); 157 | int count = in.readInt(); 158 | for (int i = 0; i < count; i++) { 159 | String eid = in.readUTF(); 160 | String label = in.readUTF(); 161 | String src = in.readUTF(); 162 | MapReduceEdge mre = new MapReduceEdge(parent, eid, src, label, id); 163 | inEdges.add(mre); 164 | } 165 | 166 | outEdges.clear(); 167 | count = in.readInt(); 168 | for (int i = 0; i < count; i++) { 169 | String eid = in.readUTF(); 170 | String label = in.readUTF(); 171 | String dest = in.readUTF(); 172 | MapReduceEdge mre = new MapReduceEdge(parent, eid, id, label, dest); 173 | outEdges.add(mre); 174 | } 175 | } 176 | 177 | @Override 178 | public void write(DataOutput out) throws IOException { 179 | super.write(out); 180 | 181 | out.writeInt(inEdges.size()); 182 | for (Edge e : inEdges) { 183 | out.writeUTF(e.getId().toString()); 184 | out.writeUTF(e.getLabel()); 185 | MapReduceEdge mre = (MapReduceEdge) e; 186 | out.writeUTF(mre.getVertexId(Direction.OUT)); 187 | } 188 | out.writeInt(outEdges.size()); 189 | for (Edge e : outEdges) { 190 | out.writeUTF(e.getId().toString()); 191 | out.writeUTF(e.getLabel()); 192 | MapReduceEdge mre = (MapReduceEdge) e; 193 | out.writeUTF(mre.getVertexId(Direction.IN)); 194 | } 195 | } 196 | 197 | } 198 | -------------------------------------------------------------------------------- /src/main/java/edu/jhuapl/tinkerpop/mapreduce/VertexInputFormat.java: -------------------------------------------------------------------------------- 1 | package edu.jhuapl.tinkerpop.mapreduce; 2 | 3 | import java.io.ByteArrayInputStream; 4 | import java.io.DataInputStream; 5 | import java.io.IOException; 6 | import java.util.Iterator; 7 | import java.util.Map.Entry; 8 | 9 | import org.apache.accumulo.core.client.AccumuloException; 10 | import org.apache.accumulo.core.client.AccumuloSecurityException; 11 | import org.apache.accumulo.core.client.RowIterator; 12 | import org.apache.accumulo.core.client.mapreduce.InputFormatBase; 13 | import org.apache.accumulo.core.client.mock.MockInstance; 14 | import org.apache.accumulo.core.client.security.tokens.PasswordToken; 15 | import org.apache.accumulo.core.data.Key; 16 | import org.apache.accumulo.core.data.Value; 17 | import org.apache.hadoop.io.Text; 18 | import org.apache.hadoop.mapreduce.InputSplit; 19 | import org.apache.hadoop.mapreduce.Job; 20 | import org.apache.hadoop.mapreduce.RecordReader; 21 | import org.apache.hadoop.mapreduce.TaskAttemptContext; 22 | 23 | import com.tinkerpop.blueprints.Vertex; 24 | 25 | import edu.jhuapl.tinkerpop.AccumuloByteSerializer; 26 | import edu.jhuapl.tinkerpop.AccumuloGraph; 27 | import edu.jhuapl.tinkerpop.AccumuloGraphConfiguration; 28 | import edu.jhuapl.tinkerpop.AccumuloGraphConfiguration.InstanceType; 29 | import edu.jhuapl.tinkerpop.AccumuloGraphException; 30 | import edu.jhuapl.tinkerpop.Constants; 31 | 32 | public class VertexInputFormat extends InputFormatBase { 33 | static AccumuloGraphConfiguration conf; 34 | 35 | private static final String PREFIX = VertexInputFormat.class.getSimpleName()+"."; 36 | private static final String GRAPH_NAME = PREFIX+"graph.name"; 37 | 38 | @Override 39 | public RecordReader createRecordReader(InputSplit split, 40 | TaskAttemptContext attempt) throws IOException, InterruptedException { 41 | return new VertexRecordReader(); 42 | } 43 | 44 | private class VertexRecordReader extends RecordReaderBase { 45 | 46 | RowIterator rowIterator; 47 | AccumuloGraph parent; 48 | 49 | VertexRecordReader() {} 50 | 51 | @Override 52 | public void initialize(InputSplit inSplit, TaskAttemptContext attempt) throws IOException { 53 | 54 | super.initialize(inSplit, attempt); 55 | rowIterator = new RowIterator(scannerIterator); 56 | 57 | currentK = new Text(); 58 | 59 | try { 60 | conf = new AccumuloGraphConfiguration(); 61 | conf.setZooKeeperHosts(VertexInputFormat.getInstance(attempt).getZooKeepers()); 62 | conf.setInstanceName(VertexInputFormat.getInstance(attempt).getInstanceName()); 63 | conf.setUser(VertexInputFormat.getPrincipal(attempt)); 64 | conf.setTokenWithFallback(VertexInputFormat.getToken(attempt)); 65 | conf.setGraphName(attempt.getConfiguration().get(GRAPH_NAME)); 66 | if (VertexInputFormat.getInstance(attempt) instanceof MockInstance) { 67 | conf.setInstanceType(InstanceType.Mock); 68 | } 69 | 70 | parent = AccumuloGraph.open(conf.getConfiguration()); 71 | } catch (AccumuloException e) { 72 | throw new AccumuloGraphException(e); 73 | } 74 | } 75 | 76 | 77 | 78 | @Override 79 | public boolean nextKeyValue() throws IOException, InterruptedException { 80 | if (rowIterator.hasNext()) { 81 | Iterator> it = rowIterator.next(); 82 | 83 | MapReduceVertex vertex = new MapReduceVertex(parent); 84 | while (it.hasNext()) { 85 | Entry entry = it.next(); 86 | numKeysRead++; 87 | 88 | currentKey = entry.getKey(); 89 | String vid = currentKey.getRow().toString(); 90 | String colf = currentKey.getColumnFamily().toString(); 91 | switch (colf) { 92 | case Constants.LABEL: 93 | currentK.set(vid); 94 | vertex.prepareId(vid); 95 | break; 96 | case Constants.IN_EDGE: 97 | String[] parts = currentKey.getColumnQualifier().toString().split(Constants.ID_DELIM); 98 | String label = new String(entry.getValue().get()); 99 | vertex.prepareEdge(parts[1], parts[0], label, vid); 100 | break; 101 | case Constants.OUT_EDGE: 102 | parts = currentKey.getColumnQualifier().toString().split(Constants.ID_DELIM); 103 | label = new String(entry.getValue().get()); 104 | vertex.prepareEdge(parts[1], vid, label, parts[0]); 105 | break; 106 | default: 107 | String propertyKey = currentKey.getColumnFamily().toString(); 108 | Object propertyValue = AccumuloByteSerializer.deserialize(entry.getValue().get()); 109 | vertex.prepareProperty(propertyKey, propertyValue); 110 | } 111 | } 112 | currentV = vertex; 113 | return true; 114 | } 115 | return false; 116 | } 117 | 118 | } 119 | 120 | public static void setAccumuloGraphConfiguration(Job job, AccumuloGraphConfiguration cfg) throws AccumuloSecurityException { 121 | 122 | VertexInputFormat.setConnectorInfo(job, cfg.getUser(), new PasswordToken(cfg.getPassword())); 123 | VertexInputFormat.setInputTableName(job, cfg.getVertexTableName()); 124 | if (cfg.getInstanceType().equals(InstanceType.Mock)) { 125 | VertexInputFormat.setMockInstance(job, cfg.getInstanceName()); 126 | } else { 127 | VertexInputFormat.setZooKeeperInstance(job, cfg.getInstanceName(), cfg.getZooKeeperHosts()); 128 | } 129 | job.getConfiguration().set(GRAPH_NAME, cfg.getGraphName()); 130 | } 131 | 132 | } 133 | -------------------------------------------------------------------------------- /src/main/java/edu/jhuapl/tinkerpop/mutator/Mutator.java: -------------------------------------------------------------------------------- 1 | /* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory 2 | * 3 | * Licensed under the Apache License, Version 2.0 (the "License"); 4 | * you may not use this file except in compliance with the License. 5 | * You may obtain a copy of the License at 6 | * 7 | * http://www.apache.org/licenses/LICENSE-2.0 8 | * 9 | * Unless required by applicable law or agreed to in writing, software 10 | * distributed under the License is distributed on an "AS IS" BASIS, 11 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | * See the License for the specific language governing permissions and 13 | * limitations under the License. 14 | */ 15 | package edu.jhuapl.tinkerpop.mutator; 16 | 17 | import org.apache.accumulo.core.data.Mutation; 18 | 19 | public interface Mutator { 20 | 21 | public Iterable create(); 22 | } 23 | -------------------------------------------------------------------------------- /src/main/java/edu/jhuapl/tinkerpop/mutator/Mutators.java: -------------------------------------------------------------------------------- 1 | /* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory 2 | * 3 | * Licensed under the Apache License, Version 2.0 (the "License"); 4 | * you may not use this file except in compliance with the License. 5 | * You may obtain a copy of the License at 6 | * 7 | * http://www.apache.org/licenses/LICENSE-2.0 8 | * 9 | * Unless required by applicable law or agreed to in writing, software 10 | * distributed under the License is distributed on an "AS IS" BASIS, 11 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | * See the License for the specific language governing permissions and 13 | * limitations under the License. 14 | */ 15 | package edu.jhuapl.tinkerpop.mutator; 16 | 17 | import java.util.LinkedList; 18 | import java.util.List; 19 | 20 | import org.apache.accumulo.core.client.BatchDeleter; 21 | import org.apache.accumulo.core.client.BatchWriter; 22 | import org.apache.accumulo.core.client.MutationsRejectedException; 23 | import org.apache.accumulo.core.data.Range; 24 | 25 | import com.tinkerpop.blueprints.Element; 26 | 27 | import edu.jhuapl.tinkerpop.AccumuloGraphException; 28 | 29 | public class Mutators { 30 | 31 | public static void apply(BatchWriter writer, Mutator mut) { 32 | try { 33 | writer.addMutations(mut.create()); 34 | } catch (MutationsRejectedException e) { 35 | throw new AccumuloGraphException(e); 36 | } 37 | } 38 | 39 | public static void deleteElementRanges(BatchDeleter deleter, Element... elements) { 40 | List ranges = new LinkedList(); 41 | 42 | for (Element element : elements) { 43 | ranges.add(new Range(element.getId().toString())); 44 | } 45 | deleter.setRanges(ranges); 46 | 47 | try { 48 | deleter.delete(); 49 | } catch (Exception e) { 50 | throw new AccumuloGraphException(e); 51 | } 52 | } 53 | } 54 | -------------------------------------------------------------------------------- /src/main/java/edu/jhuapl/tinkerpop/mutator/edge/BaseEdgeMutator.java: -------------------------------------------------------------------------------- 1 | /* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory 2 | * 3 | * Licensed under the Apache License, Version 2.0 (the "License"); 4 | * you may not use this file except in compliance with the License. 5 | * You may obtain a copy of the License at 6 | * 7 | * http://www.apache.org/licenses/LICENSE-2.0 8 | * 9 | * Unless required by applicable law or agreed to in writing, software 10 | * distributed under the License is distributed on an "AS IS" BASIS, 11 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | * See the License for the specific language governing permissions and 13 | * limitations under the License. 14 | */ 15 | package edu.jhuapl.tinkerpop.mutator.edge; 16 | 17 | import com.tinkerpop.blueprints.Direction; 18 | import com.tinkerpop.blueprints.Edge; 19 | 20 | import edu.jhuapl.tinkerpop.mutator.Mutator; 21 | 22 | public abstract class BaseEdgeMutator implements Mutator { 23 | 24 | protected final String id; 25 | protected final String outVertexId; 26 | protected final String inVertexId; 27 | protected final String label; 28 | 29 | public BaseEdgeMutator(Edge edge) { 30 | this(edge.getId().toString(), 31 | edge.getVertex(Direction.OUT).getId().toString(), 32 | edge.getVertex(Direction.IN).getId().toString(), 33 | edge.getLabel()); 34 | } 35 | public BaseEdgeMutator(String id, String outVertexId, String inVertexId, String label) { 36 | this.id = id; 37 | this.outVertexId = outVertexId; 38 | this.inVertexId = inVertexId; 39 | this.label = label; 40 | } 41 | } 42 | -------------------------------------------------------------------------------- /src/main/java/edu/jhuapl/tinkerpop/mutator/edge/EdgeEndpointsMutator.java: -------------------------------------------------------------------------------- 1 | /* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory 2 | * 3 | * Licensed under the Apache License, Version 2.0 (the "License"); 4 | * you may not use this file except in compliance with the License. 5 | * You may obtain a copy of the License at 6 | * 7 | * http://www.apache.org/licenses/LICENSE-2.0 8 | * 9 | * Unless required by applicable law or agreed to in writing, software 10 | * distributed under the License is distributed on an "AS IS" BASIS, 11 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | * See the License for the specific language governing permissions and 13 | * limitations under the License. 14 | */ 15 | package edu.jhuapl.tinkerpop.mutator.edge; 16 | 17 | import org.apache.accumulo.core.data.Mutation; 18 | 19 | import com.google.common.collect.Lists; 20 | import com.tinkerpop.blueprints.Edge; 21 | 22 | import edu.jhuapl.tinkerpop.Constants; 23 | 24 | public class EdgeEndpointsMutator { 25 | 26 | private EdgeEndpointsMutator() { 27 | } 28 | 29 | public static class Add extends BaseEdgeMutator { 30 | 31 | public Add(Edge edge) { 32 | super(edge); 33 | } 34 | 35 | public Add(String id, String outVertexId, String inVertexId, String label) { 36 | super(id, outVertexId, inVertexId, label); 37 | } 38 | 39 | @Override 40 | public Iterable create() { 41 | Mutation in = new Mutation(inVertexId); 42 | in.put(Constants.IN_EDGE.getBytes(), 43 | (outVertexId + Constants.ID_DELIM + id).getBytes(), 44 | (Constants.ID_DELIM + label).getBytes()); 45 | 46 | Mutation out = new Mutation(outVertexId); 47 | out.put(Constants.OUT_EDGE.getBytes(), 48 | (inVertexId + Constants.ID_DELIM + id).getBytes(), 49 | (Constants.ID_DELIM + label).getBytes()); 50 | 51 | return Lists.newArrayList(in, out); 52 | } 53 | } 54 | 55 | public static class Delete extends BaseEdgeMutator { 56 | 57 | public Delete(Edge edge) { 58 | super(edge); 59 | } 60 | 61 | public Delete(String id, String outVertexId, String inVertexId, String label) { 62 | super(id, outVertexId, inVertexId, label); 63 | } 64 | 65 | @Override 66 | public Iterable create() { 67 | Mutation in = new Mutation(inVertexId); 68 | in.putDelete(Constants.IN_EDGE.getBytes(), 69 | (outVertexId + Constants.ID_DELIM + id).getBytes()); 70 | 71 | Mutation out = new Mutation(outVertexId); 72 | out.putDelete(Constants.OUT_EDGE.getBytes(), 73 | (inVertexId + Constants.ID_DELIM + id).getBytes()); 74 | 75 | return Lists.newArrayList(in, out); 76 | } 77 | } 78 | } 79 | -------------------------------------------------------------------------------- /src/main/java/edu/jhuapl/tinkerpop/mutator/edge/EdgeMutator.java: -------------------------------------------------------------------------------- 1 | /* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory 2 | * 3 | * Licensed under the Apache License, Version 2.0 (the "License"); 4 | * you may not use this file except in compliance with the License. 5 | * You may obtain a copy of the License at 6 | * 7 | * http://www.apache.org/licenses/LICENSE-2.0 8 | * 9 | * Unless required by applicable law or agreed to in writing, software 10 | * distributed under the License is distributed on an "AS IS" BASIS, 11 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | * See the License for the specific language governing permissions and 13 | * limitations under the License. 14 | */ 15 | package edu.jhuapl.tinkerpop.mutator.edge; 16 | 17 | import org.apache.accumulo.core.data.Mutation; 18 | 19 | import com.google.common.collect.Lists; 20 | import com.tinkerpop.blueprints.Edge; 21 | 22 | import edu.jhuapl.tinkerpop.AccumuloByteSerializer; 23 | import edu.jhuapl.tinkerpop.Constants; 24 | 25 | public final class EdgeMutator { 26 | 27 | public static class Add extends BaseEdgeMutator { 28 | 29 | public Add(Edge edge) { 30 | super(edge); 31 | } 32 | 33 | public Add(String id, String outVertexId, String inVertexId, String label) { 34 | super(id, outVertexId, inVertexId, label); 35 | } 36 | 37 | @Override 38 | public Iterable create() { 39 | Mutation m = new Mutation(id); 40 | m.put(Constants.LABEL.getBytes(), 41 | (inVertexId + Constants.ID_DELIM + outVertexId).getBytes(), 42 | AccumuloByteSerializer.serialize(label)); 43 | 44 | return Lists.newArrayList(m); 45 | } 46 | } 47 | 48 | public static class Delete extends BaseEdgeMutator { 49 | 50 | public Delete(Edge edge) { 51 | super(edge); 52 | } 53 | 54 | public Delete(String id, String outVertexId, String inVertexId, String label) { 55 | super(id, outVertexId, inVertexId, label); 56 | } 57 | 58 | @Override 59 | public Iterable create() { 60 | Mutation m = new Mutation(id); 61 | m.putDelete(Constants.LABEL.getBytes(), 62 | (inVertexId + Constants.ID_DELIM + outVertexId).getBytes()); 63 | 64 | return Lists.newArrayList(m); 65 | } 66 | } 67 | } -------------------------------------------------------------------------------- /src/main/java/edu/jhuapl/tinkerpop/mutator/index/IndexMetadataMutator.java: -------------------------------------------------------------------------------- 1 | /* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory 2 | * 3 | * Licensed under the Apache License, Version 2.0 (the "License"); 4 | * you may not use this file except in compliance with the License. 5 | * You may obtain a copy of the License at 6 | * 7 | * http://www.apache.org/licenses/LICENSE-2.0 8 | * 9 | * Unless required by applicable law or agreed to in writing, software 10 | * distributed under the License is distributed on an "AS IS" BASIS, 11 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | * See the License for the specific language governing permissions and 13 | * limitations under the License. 14 | */ 15 | package edu.jhuapl.tinkerpop.mutator.index; 16 | 17 | import org.apache.accumulo.core.data.Mutation; 18 | 19 | import com.google.common.collect.Lists; 20 | import com.tinkerpop.blueprints.Element; 21 | 22 | import edu.jhuapl.tinkerpop.Constants; 23 | import edu.jhuapl.tinkerpop.Constants.IndexMetadataEntryType; 24 | import edu.jhuapl.tinkerpop.mutator.Mutator; 25 | 26 | /** 27 | * Mutators for index metadata table entries. 28 | */ 29 | public class IndexMetadataMutator { 30 | 31 | private IndexMetadataMutator() { } 32 | 33 | public static class Add implements Mutator { 34 | 35 | private final String key; 36 | private final Class elementClass; 37 | private final IndexMetadataEntryType entryType; 38 | 39 | public Add(String key, Class elementClass, 40 | IndexMetadataEntryType entryType) { 41 | this.key = key; 42 | this.elementClass = elementClass; 43 | this.entryType = entryType; 44 | } 45 | 46 | @Override 47 | public Iterable create() { 48 | Mutation m = new Mutation(key); 49 | m.put(entryType.name().getBytes(), 50 | elementClass.getName().getBytes(), Constants.EMPTY); 51 | return Lists.newArrayList(m); 52 | } 53 | } 54 | 55 | public static class Delete implements Mutator { 56 | 57 | private final String key; 58 | private final Class elementClass; 59 | private final IndexMetadataEntryType entryType; 60 | 61 | public Delete(String indexName, Class elementClass, 62 | IndexMetadataEntryType entryType) { 63 | this.key = indexName; 64 | this.elementClass = elementClass; 65 | this.entryType = entryType; 66 | } 67 | 68 | @Override 69 | public Iterable create() { 70 | Mutation m = new Mutation(key); 71 | m.putDelete(entryType.name().getBytes(), 72 | elementClass.getName().getBytes()); 73 | return Lists.newArrayList(m); 74 | } 75 | } 76 | } 77 | -------------------------------------------------------------------------------- /src/main/java/edu/jhuapl/tinkerpop/mutator/index/IndexValueMutator.java: -------------------------------------------------------------------------------- 1 | /* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory 2 | * 3 | * Licensed under the Apache License, Version 2.0 (the "License"); 4 | * you may not use this file except in compliance with the License. 5 | * You may obtain a copy of the License at 6 | * 7 | * http://www.apache.org/licenses/LICENSE-2.0 8 | * 9 | * Unless required by applicable law or agreed to in writing, software 10 | * distributed under the License is distributed on an "AS IS" BASIS, 11 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | * See the License for the specific language governing permissions and 13 | * limitations under the License. 14 | */ 15 | package edu.jhuapl.tinkerpop.mutator.index; 16 | 17 | import org.apache.accumulo.core.data.Mutation; 18 | 19 | import com.google.common.collect.Lists; 20 | import com.tinkerpop.blueprints.Element; 21 | 22 | import edu.jhuapl.tinkerpop.AccumuloByteSerializer; 23 | import edu.jhuapl.tinkerpop.Constants; 24 | import edu.jhuapl.tinkerpop.mutator.Mutator; 25 | 26 | /** 27 | * Mutators for vertex/edge index tables. 28 | */ 29 | public class IndexValueMutator { 30 | 31 | private IndexValueMutator() { } 32 | 33 | public static class Add implements Mutator { 34 | 35 | private final Element element; 36 | private final String key; 37 | private final Object value; 38 | 39 | public Add(Element element, String key, Object value) { 40 | this.element = element; 41 | this.key = key; 42 | this.value = value; 43 | } 44 | 45 | @Override 46 | public Iterable create() { 47 | byte[] bytes = AccumuloByteSerializer.serialize(value); 48 | Mutation m = new Mutation(bytes); 49 | m.put(key.getBytes(), element.getId().toString() 50 | .getBytes(), Constants.EMPTY); 51 | return Lists.newArrayList(m); 52 | } 53 | } 54 | 55 | public static class Delete implements Mutator { 56 | 57 | private final Element element; 58 | private final String key; 59 | private final Object value; 60 | 61 | public Delete(Element element, String key, Object value) { 62 | this.element = element; 63 | this.key = key; 64 | this.value = value; 65 | } 66 | 67 | @Override 68 | public Iterable create() { 69 | byte[] bytes = AccumuloByteSerializer.serialize(value); 70 | Mutation m = new Mutation(bytes); 71 | m.putDelete(key, element.getId().toString()); 72 | return Lists.newArrayList(m); 73 | } 74 | } 75 | } 76 | -------------------------------------------------------------------------------- /src/main/java/edu/jhuapl/tinkerpop/mutator/property/BasePropertyMutator.java: -------------------------------------------------------------------------------- 1 | /* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory 2 | * 3 | * Licensed under the Apache License, Version 2.0 (the "License"); 4 | * you may not use this file except in compliance with the License. 5 | * You may obtain a copy of the License at 6 | * 7 | * http://www.apache.org/licenses/LICENSE-2.0 8 | * 9 | * Unless required by applicable law or agreed to in writing, software 10 | * distributed under the License is distributed on an "AS IS" BASIS, 11 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | * See the License for the specific language governing permissions and 13 | * limitations under the License. 14 | */ 15 | package edu.jhuapl.tinkerpop.mutator.property; 16 | 17 | import edu.jhuapl.tinkerpop.mutator.Mutator; 18 | 19 | public abstract class BasePropertyMutator implements Mutator { 20 | 21 | protected final String id; 22 | protected final String key; 23 | 24 | public BasePropertyMutator(String id, String key) { 25 | this.id = id; 26 | this.key = key; 27 | } 28 | } 29 | -------------------------------------------------------------------------------- /src/main/java/edu/jhuapl/tinkerpop/mutator/property/ClearPropertyMutator.java: -------------------------------------------------------------------------------- 1 | /* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory 2 | * 3 | * Licensed under the Apache License, Version 2.0 (the "License"); 4 | * you may not use this file except in compliance with the License. 5 | * You may obtain a copy of the License at 6 | * 7 | * http://www.apache.org/licenses/LICENSE-2.0 8 | * 9 | * Unless required by applicable law or agreed to in writing, software 10 | * distributed under the License is distributed on an "AS IS" BASIS, 11 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | * See the License for the specific language governing permissions and 13 | * limitations under the License. 14 | */ 15 | package edu.jhuapl.tinkerpop.mutator.property; 16 | 17 | import org.apache.accumulo.core.data.Mutation; 18 | 19 | import com.google.common.collect.Lists; 20 | import edu.jhuapl.tinkerpop.Constants; 21 | 22 | public class ClearPropertyMutator extends BasePropertyMutator { 23 | 24 | public ClearPropertyMutator(String id, String key) { 25 | super(id, key); 26 | } 27 | 28 | @Override 29 | public Iterable create() { 30 | Mutation m = new Mutation(id); 31 | m.putDelete(key.getBytes(), Constants.EMPTY); 32 | return Lists.newArrayList(m); 33 | } 34 | } 35 | -------------------------------------------------------------------------------- /src/main/java/edu/jhuapl/tinkerpop/mutator/property/WritePropertyMutator.java: -------------------------------------------------------------------------------- 1 | /* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory 2 | * 3 | * Licensed under the Apache License, Version 2.0 (the "License"); 4 | * you may not use this file except in compliance with the License. 5 | * You may obtain a copy of the License at 6 | * 7 | * http://www.apache.org/licenses/LICENSE-2.0 8 | * 9 | * Unless required by applicable law or agreed to in writing, software 10 | * distributed under the License is distributed on an "AS IS" BASIS, 11 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | * See the License for the specific language governing permissions and 13 | * limitations under the License. 14 | */ 15 | package edu.jhuapl.tinkerpop.mutator.property; 16 | 17 | import org.apache.accumulo.core.data.Mutation; 18 | 19 | import com.google.common.collect.Lists; 20 | import edu.jhuapl.tinkerpop.AccumuloByteSerializer; 21 | import edu.jhuapl.tinkerpop.Constants; 22 | 23 | public class WritePropertyMutator extends BasePropertyMutator { 24 | 25 | private final Object value; 26 | 27 | public WritePropertyMutator(String id, String key, Object value) { 28 | super(id, key); 29 | this.value = value; 30 | } 31 | 32 | @Override 33 | public Iterable create() { 34 | byte[] bytes = AccumuloByteSerializer.serialize(value); 35 | Mutation m = new Mutation(id); 36 | m.put(key.getBytes(), Constants.EMPTY, bytes); 37 | return Lists.newArrayList(m); 38 | } 39 | } 40 | -------------------------------------------------------------------------------- /src/main/java/edu/jhuapl/tinkerpop/mutator/vertex/AddVertexMutator.java: -------------------------------------------------------------------------------- 1 | /* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory 2 | * 3 | * Licensed under the Apache License, Version 2.0 (the "License"); 4 | * you may not use this file except in compliance with the License. 5 | * You may obtain a copy of the License at 6 | * 7 | * http://www.apache.org/licenses/LICENSE-2.0 8 | * 9 | * Unless required by applicable law or agreed to in writing, software 10 | * distributed under the License is distributed on an "AS IS" BASIS, 11 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | * See the License for the specific language governing permissions and 13 | * limitations under the License. 14 | */ 15 | package edu.jhuapl.tinkerpop.mutator.vertex; 16 | 17 | import org.apache.accumulo.core.data.Mutation; 18 | 19 | import com.google.common.collect.Lists; 20 | import edu.jhuapl.tinkerpop.Constants; 21 | import edu.jhuapl.tinkerpop.mutator.Mutator; 22 | 23 | public final class AddVertexMutator implements Mutator { 24 | 25 | private final String id; 26 | 27 | public AddVertexMutator(String id) { 28 | this.id = id; 29 | } 30 | 31 | @Override 32 | public Iterable create() { 33 | Mutation m = new Mutation(id); 34 | m.put(Constants.LABEL.getBytes(), 35 | Constants.EXISTS.getBytes(), Constants.EMPTY); 36 | return Lists.newArrayList(m); 37 | } 38 | } 39 | -------------------------------------------------------------------------------- /src/main/java/edu/jhuapl/tinkerpop/parser/EdgeIndexParser.java: -------------------------------------------------------------------------------- 1 | /* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory 2 | * 3 | * Licensed under the Apache License, Version 2.0 (the "License"); 4 | * you may not use this file except in compliance with the License. 5 | * You may obtain a copy of the License at 6 | * 7 | * http://www.apache.org/licenses/LICENSE-2.0 8 | * 9 | * Unless required by applicable law or agreed to in writing, software 10 | * distributed under the License is distributed on an "AS IS" BASIS, 11 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | * See the License for the specific language governing permissions and 13 | * limitations under the License. 14 | */ 15 | package edu.jhuapl.tinkerpop.parser; 16 | 17 | import edu.jhuapl.tinkerpop.AccumuloEdge; 18 | import edu.jhuapl.tinkerpop.GlobalInstances; 19 | 20 | /** 21 | * Edge-specific index parser. 22 | */ 23 | public class EdgeIndexParser extends ElementIndexParser { 24 | 25 | public EdgeIndexParser(GlobalInstances globals) { 26 | super(globals); 27 | } 28 | 29 | @Override 30 | protected AccumuloEdge instantiate(String id) { 31 | return new AccumuloEdge(globals, id); 32 | } 33 | } 34 | -------------------------------------------------------------------------------- /src/main/java/edu/jhuapl/tinkerpop/parser/EdgeParser.java: -------------------------------------------------------------------------------- 1 | /* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory 2 | * 3 | * Licensed under the Apache License, Version 2.0 (the "License"); 4 | * you may not use this file except in compliance with the License. 5 | * You may obtain a copy of the License at 6 | * 7 | * http://www.apache.org/licenses/LICENSE-2.0 8 | * 9 | * Unless required by applicable law or agreed to in writing, software 10 | * distributed under the License is distributed on an "AS IS" BASIS, 11 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | * See the License for the specific language governing permissions and 13 | * limitations under the License. 14 | */ 15 | package edu.jhuapl.tinkerpop.parser; 16 | 17 | import java.util.Map.Entry; 18 | 19 | import org.apache.accumulo.core.data.Key; 20 | import org.apache.accumulo.core.data.Value; 21 | 22 | import edu.jhuapl.tinkerpop.AccumuloByteSerializer; 23 | import edu.jhuapl.tinkerpop.AccumuloEdge; 24 | import edu.jhuapl.tinkerpop.AccumuloGraphException; 25 | import edu.jhuapl.tinkerpop.AccumuloVertex; 26 | import edu.jhuapl.tinkerpop.Constants; 27 | import edu.jhuapl.tinkerpop.GlobalInstances; 28 | 29 | /** 30 | * TODO 31 | */ 32 | public class EdgeParser extends ElementParser { 33 | 34 | public EdgeParser(GlobalInstances globals) { 35 | super(globals); 36 | } 37 | 38 | @Override 39 | public AccumuloEdge parse(String id, Iterable> entries) { 40 | AccumuloEdge edge = makeEdge(id, entries); 41 | setInMemoryProperties(edge, entries); 42 | return edge; 43 | } 44 | 45 | /** 46 | * Make and return an edge object. If the entries 47 | * contain label/endpoint information, set those too. 48 | * @param id 49 | * @param entries 50 | * @return 51 | */ 52 | private AccumuloEdge makeEdge(String id, Iterable> entries) { 53 | for (Entry entry : entries) { 54 | String cf = entry.getKey().getColumnFamily().toString(); 55 | if (Constants.LABEL.equals(cf)) { 56 | String cq = entry.getKey().getColumnQualifier().toString(); 57 | String[] parts = cq.split(Constants.ID_DELIM); 58 | String inVertexId = parts[0]; 59 | String outVertexId = parts[1]; 60 | String label = AccumuloByteSerializer.deserialize(entry.getValue().get()); 61 | return new AccumuloEdge(globals, id, 62 | new AccumuloVertex(globals, inVertexId), 63 | new AccumuloVertex(globals, outVertexId), label); 64 | } 65 | } 66 | 67 | // This should not happen. 68 | throw new AccumuloGraphException("Unable to parse edge from entries"); 69 | } 70 | } 71 | -------------------------------------------------------------------------------- /src/main/java/edu/jhuapl/tinkerpop/parser/ElementIndexParser.java: -------------------------------------------------------------------------------- 1 | /* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory 2 | * 3 | * Licensed under the Apache License, Version 2.0 (the "License"); 4 | * you may not use this file except in compliance with the License. 5 | * You may obtain a copy of the License at 6 | * 7 | * http://www.apache.org/licenses/LICENSE-2.0 8 | * 9 | * Unless required by applicable law or agreed to in writing, software 10 | * distributed under the License is distributed on an "AS IS" BASIS, 11 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | * See the License for the specific language governing permissions and 13 | * limitations under the License. 14 | */ 15 | package edu.jhuapl.tinkerpop.parser; 16 | 17 | import java.util.Iterator; 18 | import java.util.Map.Entry; 19 | 20 | import org.apache.accumulo.core.data.Key; 21 | import org.apache.accumulo.core.data.Value; 22 | 23 | import edu.jhuapl.tinkerpop.AccumuloByteSerializer; 24 | import edu.jhuapl.tinkerpop.AccumuloElement; 25 | import edu.jhuapl.tinkerpop.AccumuloGraphException; 26 | import edu.jhuapl.tinkerpop.GlobalInstances; 27 | 28 | /** 29 | * Parser for elements based on an index table. 30 | */ 31 | public abstract class ElementIndexParser 32 | implements EntryParser { 33 | 34 | protected final GlobalInstances globals; 35 | 36 | public ElementIndexParser(GlobalInstances globals) { 37 | this.globals = globals; 38 | } 39 | 40 | @Override 41 | public T parse(Iterable> entries) { 42 | Iterator> it = entries.iterator(); 43 | 44 | if (it.hasNext()) { 45 | Entry entry = it.next(); 46 | 47 | if (it.hasNext()) { 48 | throw new AccumuloGraphException("Unexpected multiple entries for index table"); 49 | } 50 | 51 | String id = entry.getKey().getColumnQualifierData().toString(); 52 | T element = instantiate(id); 53 | 54 | // While we're here, read the property key/value. 55 | String key = entry.getKey().getColumnFamily().toString(); 56 | Object value = AccumuloByteSerializer.deserialize(entry.getKey().getRow().getBytes()); 57 | element.setPropertyInMemory(key, value); 58 | 59 | return element; 60 | } 61 | else { 62 | throw new AccumuloGraphException("No index table entries found"); 63 | } 64 | } 65 | 66 | /** 67 | * Instantiate an object to be returned. 68 | * @param id 69 | * @return 70 | */ 71 | protected abstract T instantiate(String id); 72 | } 73 | -------------------------------------------------------------------------------- /src/main/java/edu/jhuapl/tinkerpop/parser/ElementParser.java: -------------------------------------------------------------------------------- 1 | /* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory 2 | * 3 | * Licensed under the Apache License, Version 2.0 (the "License"); 4 | * you may not use this file except in compliance with the License. 5 | * You may obtain a copy of the License at 6 | * 7 | * http://www.apache.org/licenses/LICENSE-2.0 8 | * 9 | * Unless required by applicable law or agreed to in writing, software 10 | * distributed under the License is distributed on an "AS IS" BASIS, 11 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | * See the License for the specific language governing permissions and 13 | * limitations under the License. 14 | */ 15 | package edu.jhuapl.tinkerpop.parser; 16 | 17 | import java.util.Map; 18 | import java.util.Map.Entry; 19 | 20 | import org.apache.accumulo.core.data.Key; 21 | import org.apache.accumulo.core.data.Value; 22 | 23 | import edu.jhuapl.tinkerpop.AccumuloElement; 24 | import edu.jhuapl.tinkerpop.GlobalInstances; 25 | 26 | /** 27 | * TODO 28 | */ 29 | public abstract class ElementParser implements EntryParser { 30 | 31 | protected GlobalInstances globals; 32 | 33 | public ElementParser(GlobalInstances globals) { 34 | this.globals = globals; 35 | } 36 | 37 | @Override 38 | public T parse(Iterable> entries) { 39 | String id = entries.iterator().next().getKey().getRow().toString(); 40 | return parse(id, entries); 41 | } 42 | 43 | /** 44 | * Given the element id and set of entries, 45 | * create the type of element. This can leverage 46 | * the property loader, etc. 47 | * @param id 48 | * @param entries 49 | * @return 50 | */ 51 | public abstract T parse(String id, Iterable> entries); 52 | 53 | /** 54 | * Parse out the property entries and set them for the given element. 55 | * @param element 56 | * @param entries 57 | */ 58 | protected void setInMemoryProperties(T element, Iterable> entries) { 59 | Map props = new PropertyParser().parse(entries); 60 | if(props == null) return; 61 | for (Entry ent : props.entrySet()) { 62 | element.setPropertyInMemory(ent.getKey(), ent.getValue()); 63 | } 64 | } 65 | } 66 | -------------------------------------------------------------------------------- /src/main/java/edu/jhuapl/tinkerpop/parser/EntryParser.java: -------------------------------------------------------------------------------- 1 | /* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory 2 | * 3 | * Licensed under the Apache License, Version 2.0 (the "License"); 4 | * you may not use this file except in compliance with the License. 5 | * You may obtain a copy of the License at 6 | * 7 | * http://www.apache.org/licenses/LICENSE-2.0 8 | * 9 | * Unless required by applicable law or agreed to in writing, software 10 | * distributed under the License is distributed on an "AS IS" BASIS, 11 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | * See the License for the specific language governing permissions and 13 | * limitations under the License. 14 | */ 15 | package edu.jhuapl.tinkerpop.parser; 16 | 17 | import java.util.Map.Entry; 18 | 19 | import org.apache.accumulo.core.data.Key; 20 | import org.apache.accumulo.core.data.Value; 21 | 22 | /** 23 | * TODO 24 | */ 25 | public interface EntryParser { 26 | 27 | public T parse(Iterable> entries); 28 | } 29 | -------------------------------------------------------------------------------- /src/main/java/edu/jhuapl/tinkerpop/parser/IndexedItem.java: -------------------------------------------------------------------------------- 1 | /* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory 2 | * 3 | * Licensed under the Apache License, Version 2.0 (the "License"); 4 | * you may not use this file except in compliance with the License. 5 | * You may obtain a copy of the License at 6 | * 7 | * http://www.apache.org/licenses/LICENSE-2.0 8 | * 9 | * Unless required by applicable law or agreed to in writing, software 10 | * distributed under the License is distributed on an "AS IS" BASIS, 11 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | * See the License for the specific language governing permissions and 13 | * limitations under the License. 14 | */ 15 | package edu.jhuapl.tinkerpop.parser; 16 | 17 | import com.tinkerpop.blueprints.Element; 18 | import com.tinkerpop.blueprints.IndexableGraph; 19 | import com.tinkerpop.blueprints.KeyIndexableGraph; 20 | 21 | /** 22 | * An indexed item. For {@link IndexableGraph}, 23 | * the key is the index name. For {@link KeyIndexableGraph} 24 | * the key is the indexed key. 25 | * @author Michael Lieberman 26 | * 27 | */ 28 | public class IndexedItem { 29 | private final String key; 30 | private final Class elementClass; 31 | 32 | public IndexedItem(String key, Class elementClass) { 33 | this.key = key; 34 | this.elementClass = elementClass; 35 | } 36 | 37 | public String getKey() { 38 | return key; 39 | } 40 | 41 | public Class getElementClass() { 42 | return elementClass; 43 | } 44 | } 45 | -------------------------------------------------------------------------------- /src/main/java/edu/jhuapl/tinkerpop/parser/IndexedItemsListParser.java: -------------------------------------------------------------------------------- 1 | /* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory 2 | * 3 | * Licensed under the Apache License, Version 2.0 (the "License"); 4 | * you may not use this file except in compliance with the License. 5 | * You may obtain a copy of the License at 6 | * 7 | * http://www.apache.org/licenses/LICENSE-2.0 8 | * 9 | * Unless required by applicable law or agreed to in writing, software 10 | * distributed under the License is distributed on an "AS IS" BASIS, 11 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | * See the License for the specific language governing permissions and 13 | * limitations under the License. 14 | */ 15 | package edu.jhuapl.tinkerpop.parser; 16 | 17 | import java.util.ArrayList; 18 | import java.util.List; 19 | import java.util.Map.Entry; 20 | 21 | import org.apache.accumulo.core.data.Key; 22 | import org.apache.accumulo.core.data.Value; 23 | 24 | import com.tinkerpop.blueprints.Edge; 25 | import com.tinkerpop.blueprints.Element; 26 | import com.tinkerpop.blueprints.IndexableGraph; 27 | import com.tinkerpop.blueprints.KeyIndexableGraph; 28 | import com.tinkerpop.blueprints.Vertex; 29 | 30 | import edu.jhuapl.tinkerpop.AccumuloGraphException; 31 | 32 | /** 33 | * Entry parser for index metadata. The format 34 | * is the same for both {@link IndexableGraph} 35 | * and {@link KeyIndexableGraph} functionality. 36 | * For the former, this parser returns names of 37 | * indexes. For the latter, the parser returns 38 | * indexed keys. 39 | */ 40 | public class IndexedItemsListParser implements EntryParser> { 41 | 42 | private final Class elementClass; 43 | 44 | /** 45 | * Constructor to return all items regardless 46 | * of element class. 47 | */ 48 | public IndexedItemsListParser() { 49 | this(Element.class); 50 | } 51 | 52 | /** 53 | * Create a parser for items of the specified element class. 54 | * This may be Vertex, Edge, or Element. 55 | * @param elementClass 56 | */ 57 | public IndexedItemsListParser(Class elementClass) { 58 | // Validate element class. 59 | if (!Vertex.class.equals(elementClass) && 60 | !Edge.class.equals(elementClass) && 61 | !Element.class.equals(elementClass)) { 62 | throw new IllegalArgumentException("elementClass must be Vertex, Edge or Element"); 63 | } 64 | this.elementClass = elementClass; 65 | } 66 | 67 | @SuppressWarnings("unchecked") 68 | @Override 69 | public List parse(Iterable> entries) { 70 | List items = new ArrayList(); 71 | 72 | for (Entry entry : entries) { 73 | Class clazz; 74 | try { 75 | clazz = (Class) Class.forName(entry.getKey() 76 | .getColumnQualifier().toString()); 77 | } catch (ClassNotFoundException e) { 78 | throw new AccumuloGraphException(e); 79 | } 80 | 81 | if (Element.class.equals(elementClass) || 82 | elementClass.equals(clazz)) { 83 | IndexedItem item = new IndexedItem(entry.getKey() 84 | .getRow().toString(), clazz); 85 | items.add(item); 86 | } 87 | } 88 | 89 | return items; 90 | } 91 | } 92 | -------------------------------------------------------------------------------- /src/main/java/edu/jhuapl/tinkerpop/parser/PropertyParser.java: -------------------------------------------------------------------------------- 1 | /* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory 2 | * 3 | * Licensed under the Apache License, Version 2.0 (the "License"); 4 | * you may not use this file except in compliance with the License. 5 | * You may obtain a copy of the License at 6 | * 7 | * http://www.apache.org/licenses/LICENSE-2.0 8 | * 9 | * Unless required by applicable law or agreed to in writing, software 10 | * distributed under the License is distributed on an "AS IS" BASIS, 11 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | * See the License for the specific language governing permissions and 13 | * limitations under the License. 14 | */ 15 | package edu.jhuapl.tinkerpop.parser; 16 | 17 | import java.util.HashMap; 18 | import java.util.Map; 19 | import java.util.Map.Entry; 20 | 21 | import org.apache.accumulo.core.data.Key; 22 | import org.apache.accumulo.core.data.Value; 23 | 24 | import edu.jhuapl.tinkerpop.AccumuloByteSerializer; 25 | import edu.jhuapl.tinkerpop.Constants; 26 | 27 | /** 28 | * TODO 29 | */ 30 | public class PropertyParser implements EntryParser> { 31 | 32 | @Override 33 | public Map parse(Iterable> entries) { 34 | Map props = null; 35 | 36 | for (Entry entry : entries) { 37 | if (props == null) { 38 | props = new HashMap(); 39 | } 40 | 41 | Key key = entry.getKey(); 42 | 43 | if (!isMetaKey(key)) { 44 | String attr = key.getColumnFamily().toString(); 45 | Object value = AccumuloByteSerializer.deserialize(entry.getValue().get()); 46 | props.put(attr, value); 47 | } 48 | } 49 | 50 | return props; 51 | } 52 | 53 | /** 54 | * Test whether the given Accumulo key represents a 55 | * metadata key (e.g. existence, edge endpoint, etc), 56 | * rather than a property. 57 | * @param key 58 | * @return 59 | */ 60 | private static boolean isMetaKey(Key key) { 61 | String cf = key.getColumnFamily().toString(); 62 | return Constants.LABEL.equals(cf) || 63 | Constants.IN_EDGE.equals(cf) || 64 | Constants.OUT_EDGE.equals(cf); 65 | } 66 | } 67 | -------------------------------------------------------------------------------- /src/main/java/edu/jhuapl/tinkerpop/parser/VertexIndexParser.java: -------------------------------------------------------------------------------- 1 | /* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory 2 | * 3 | * Licensed under the Apache License, Version 2.0 (the "License"); 4 | * you may not use this file except in compliance with the License. 5 | * You may obtain a copy of the License at 6 | * 7 | * http://www.apache.org/licenses/LICENSE-2.0 8 | * 9 | * Unless required by applicable law or agreed to in writing, software 10 | * distributed under the License is distributed on an "AS IS" BASIS, 11 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | * See the License for the specific language governing permissions and 13 | * limitations under the License. 14 | */ 15 | package edu.jhuapl.tinkerpop.parser; 16 | 17 | import edu.jhuapl.tinkerpop.AccumuloVertex; 18 | import edu.jhuapl.tinkerpop.GlobalInstances; 19 | 20 | /** 21 | * Vertex-specific index parser. 22 | */ 23 | public class VertexIndexParser extends ElementIndexParser { 24 | 25 | public VertexIndexParser(GlobalInstances globals) { 26 | super(globals); 27 | } 28 | 29 | @Override 30 | protected AccumuloVertex instantiate(String id) { 31 | return new AccumuloVertex(globals, id); 32 | } 33 | } 34 | -------------------------------------------------------------------------------- /src/main/java/edu/jhuapl/tinkerpop/parser/VertexParser.java: -------------------------------------------------------------------------------- 1 | /* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory 2 | * 3 | * Licensed under the Apache License, Version 2.0 (the "License"); 4 | * you may not use this file except in compliance with the License. 5 | * You may obtain a copy of the License at 6 | * 7 | * http://www.apache.org/licenses/LICENSE-2.0 8 | * 9 | * Unless required by applicable law or agreed to in writing, software 10 | * distributed under the License is distributed on an "AS IS" BASIS, 11 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | * See the License for the specific language governing permissions and 13 | * limitations under the License. 14 | */ 15 | package edu.jhuapl.tinkerpop.parser; 16 | 17 | import java.util.Map.Entry; 18 | 19 | import org.apache.accumulo.core.data.Key; 20 | import org.apache.accumulo.core.data.Value; 21 | 22 | import edu.jhuapl.tinkerpop.AccumuloVertex; 23 | import edu.jhuapl.tinkerpop.GlobalInstances; 24 | 25 | /** 26 | * TODO 27 | */ 28 | public class VertexParser extends ElementParser { 29 | 30 | public VertexParser(GlobalInstances globals) { 31 | super(globals); 32 | } 33 | 34 | @Override 35 | public AccumuloVertex parse(String id, Iterable> entries) { 36 | AccumuloVertex vertex = new AccumuloVertex(globals, id); 37 | setInMemoryProperties(vertex, entries); 38 | return vertex; 39 | } 40 | } 41 | -------------------------------------------------------------------------------- /src/main/java/edu/jhuapl/tinkerpop/tables/BaseTableWrapper.java: -------------------------------------------------------------------------------- 1 | /* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory 2 | * 3 | * Licensed under the Apache License, Version 2.0 (the "License"); 4 | * you may not use this file except in compliance with the License. 5 | * You may obtain a copy of the License at 6 | * 7 | * http://www.apache.org/licenses/LICENSE-2.0 8 | * 9 | * Unless required by applicable law or agreed to in writing, software 10 | * distributed under the License is distributed on an "AS IS" BASIS, 11 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | * See the License for the specific language governing permissions and 13 | * limitations under the License. 14 | */ 15 | package edu.jhuapl.tinkerpop.tables; 16 | 17 | import java.util.Collections; 18 | import java.util.Map.Entry; 19 | 20 | import org.apache.accumulo.core.client.BatchDeleter; 21 | import org.apache.accumulo.core.client.BatchScanner; 22 | import org.apache.accumulo.core.client.BatchWriter; 23 | import org.apache.accumulo.core.client.Scanner; 24 | import org.apache.accumulo.core.data.Key; 25 | import org.apache.accumulo.core.data.Range; 26 | import org.apache.accumulo.core.data.Value; 27 | 28 | import edu.jhuapl.tinkerpop.AccumuloGraphException; 29 | import edu.jhuapl.tinkerpop.GlobalInstances; 30 | 31 | /** 32 | * Table wrapper with common functionality. 33 | */ 34 | public abstract class BaseTableWrapper { 35 | 36 | protected GlobalInstances globals; 37 | private String tableName; 38 | 39 | public BaseTableWrapper(GlobalInstances globals, String tableName) { 40 | this.globals = globals; 41 | this.tableName = tableName; 42 | } 43 | 44 | protected Scanner getScanner() { 45 | try { 46 | return globals.getConfig().getConnector().createScanner(tableName, 47 | globals.getConfig().getAuthorizations()); 48 | 49 | } catch (Exception e) { 50 | throw new AccumuloGraphException(e); 51 | } 52 | } 53 | 54 | protected BatchScanner getBatchScanner() { 55 | try { 56 | BatchScanner scanner = globals.getConfig().getConnector().createBatchScanner(tableName, 57 | globals.getConfig().getAuthorizations(), globals.getConfig().getQueryThreads()); 58 | scanner.setRanges(Collections.singletonList(new Range())); 59 | return scanner; 60 | } catch (Exception e) { 61 | throw new AccumuloGraphException(e); 62 | } 63 | } 64 | 65 | protected BatchWriter getWriter() { 66 | try { 67 | return globals.getMtbw().getBatchWriter(tableName); 68 | } catch (Exception e) { 69 | throw new AccumuloGraphException(e); 70 | } 71 | } 72 | 73 | protected BatchDeleter getDeleter() { 74 | try { 75 | return globals.getConfig().getConnector().createBatchDeleter(tableName, 76 | globals.getConfig().getAuthorizations(), globals.getConfig().getMaxWriteThreads(), 77 | globals.getConfig().getBatchWriterConfig()); 78 | } catch (Exception e) { 79 | throw new AccumuloGraphException(e); 80 | } 81 | } 82 | 83 | public void dump() { 84 | System.out.println("Dump of table "+tableName+":"); 85 | Scanner s = getScanner(); 86 | for (Entry entry : s) { 87 | System.out.println(" "+entry); 88 | } 89 | } 90 | } 91 | -------------------------------------------------------------------------------- /src/main/java/edu/jhuapl/tinkerpop/tables/core/EdgeTableWrapper.java: -------------------------------------------------------------------------------- 1 | /* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory 2 | * 3 | * Licensed under the Apache License, Version 2.0 (the "License"); 4 | * you may not use this file except in compliance with the License. 5 | * You may obtain a copy of the License at 6 | * 7 | * http://www.apache.org/licenses/LICENSE-2.0 8 | * 9 | * Unless required by applicable law or agreed to in writing, software 10 | * distributed under the License is distributed on an "AS IS" BASIS, 11 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | * See the License for the specific language governing permissions and 13 | * limitations under the License. 14 | */ 15 | package edu.jhuapl.tinkerpop.tables.core; 16 | 17 | import java.util.ArrayList; 18 | import java.util.Iterator; 19 | import java.util.List; 20 | import java.util.Map.Entry; 21 | import java.util.regex.Pattern; 22 | 23 | import org.apache.accumulo.core.client.BatchScanner; 24 | import org.apache.accumulo.core.client.IteratorSetting; 25 | import org.apache.accumulo.core.client.Scanner; 26 | import org.apache.accumulo.core.data.Key; 27 | import org.apache.accumulo.core.data.Range; 28 | import org.apache.accumulo.core.data.Value; 29 | import org.apache.accumulo.core.iterators.user.RegExFilter; 30 | import org.apache.accumulo.core.util.PeekingIterator; 31 | import org.apache.hadoop.io.Text; 32 | 33 | import com.tinkerpop.blueprints.CloseableIterable; 34 | import com.tinkerpop.blueprints.Edge; 35 | 36 | import edu.jhuapl.tinkerpop.AccumuloByteSerializer; 37 | import edu.jhuapl.tinkerpop.AccumuloEdge; 38 | import edu.jhuapl.tinkerpop.AccumuloGraphException; 39 | import edu.jhuapl.tinkerpop.AccumuloGraphUtils; 40 | import edu.jhuapl.tinkerpop.AccumuloVertex; 41 | import edu.jhuapl.tinkerpop.Constants; 42 | import edu.jhuapl.tinkerpop.GlobalInstances; 43 | import edu.jhuapl.tinkerpop.ScannerIterable; 44 | import edu.jhuapl.tinkerpop.mutator.Mutators; 45 | import edu.jhuapl.tinkerpop.mutator.edge.EdgeMutator; 46 | import edu.jhuapl.tinkerpop.parser.EdgeParser; 47 | 48 | 49 | /** 50 | * Wrapper around {@link Edge} tables. 51 | */ 52 | public class EdgeTableWrapper extends ElementTableWrapper { 53 | 54 | public EdgeTableWrapper(GlobalInstances globals) { 55 | super(globals, globals.getConfig().getEdgeTableName()); 56 | } 57 | 58 | /** 59 | * Write the given edge to the edge table. Does not 60 | * currently write the edge's properties. 61 | * 62 | *

Note: This only adds the edge information. Vertex 63 | * endpoint information needs to be written to the vertex 64 | * table via {@link VertexTableWrapper}. 65 | * @param edge 66 | */ 67 | public void writeEdge(Edge edge) { 68 | Mutators.apply(getWriter(), new EdgeMutator.Add(edge)); 69 | globals.checkedFlush(); 70 | } 71 | 72 | public void deleteEdge(Edge edge) { 73 | Mutators.apply(getWriter(), new EdgeMutator.Delete(edge)); 74 | globals.checkedFlush(); 75 | } 76 | 77 | public CloseableIterable getEdges() { 78 | Scanner scan = getScanner(); 79 | scan.fetchColumnFamily(new Text(Constants.LABEL)); 80 | 81 | if (globals.getConfig().getPreloadedProperties() != null) { 82 | for (String key : globals.getConfig().getPreloadedProperties()) { 83 | scan.fetchColumnFamily(new Text(key)); 84 | } 85 | } 86 | 87 | final EdgeParser parser = new EdgeParser(globals); 88 | 89 | return new ScannerIterable(scan) { 90 | @Override 91 | public Edge next(PeekingIterator> iterator) { 92 | // TODO could also check local cache before creating a new instance? 93 | 94 | String rowId = iterator.peek().getKey().getRow().toString(); 95 | 96 | List> entries = 97 | new ArrayList>(); 98 | 99 | // MDL 05 Jan 2014: Why is this equalsIgnoreCase?? 100 | while (iterator.peek() != null && rowId.equalsIgnoreCase(iterator 101 | .peek().getKey().getRow().toString())) { 102 | entries.add(iterator.next()); 103 | } 104 | 105 | AccumuloEdge edge = parser.parse(rowId, entries); 106 | globals.getCaches().cache(edge, Edge.class); 107 | 108 | return edge; 109 | } 110 | }; 111 | } 112 | 113 | public Iterable getEdges(String key, Object value) { 114 | AccumuloGraphUtils.nullCheckProperty(key, value); 115 | if (key.equalsIgnoreCase("label")) { 116 | key = Constants.LABEL; 117 | } 118 | 119 | BatchScanner scan = getBatchScanner(); 120 | scan.fetchColumnFamily(new Text(key)); 121 | 122 | byte[] val = AccumuloByteSerializer.serialize(value); 123 | if (val[0] != AccumuloByteSerializer.SERIALIZABLE) { 124 | IteratorSetting is = new IteratorSetting(10, "filter", RegExFilter.class); 125 | RegExFilter.setRegexs(is, null, null, null, Pattern.quote(new String(val)), false); 126 | scan.addScanIterator(is); 127 | 128 | return new ScannerIterable(scan) { 129 | 130 | @Override 131 | public Edge next(PeekingIterator> iterator) { 132 | 133 | Key k = iterator.next().getKey(); 134 | 135 | if (k.getColumnFamily().toString().equals(Constants.LABEL)) { 136 | String[] vals = k.getColumnQualifier().toString().split(Constants.ID_DELIM); 137 | return new AccumuloEdge(globals, k.getRow().toString(), 138 | new AccumuloVertex(globals, vals[0]), 139 | new AccumuloVertex(globals, vals[1]), null); 140 | } 141 | return new AccumuloEdge(globals, k.getRow().toString()); 142 | } 143 | }; 144 | } else { 145 | // TODO 146 | throw new UnsupportedOperationException("Filtering on binary data not currently supported."); 147 | } 148 | } 149 | 150 | public void loadEndpointsAndLabel(AccumuloEdge edge) { 151 | Scanner s = getScanner(); 152 | 153 | try { 154 | s.setRange(new Range(edge.getId().toString())); 155 | s.fetchColumnFamily(new Text(Constants.LABEL)); 156 | Iterator> iter = s.iterator(); 157 | if (!iter.hasNext()) { 158 | dump(); 159 | throw new AccumuloGraphException("Unable to find edge row: "+edge); 160 | } 161 | 162 | Entry entry = iter.next(); 163 | 164 | String cq = entry.getKey().getColumnQualifier().toString(); 165 | String[] ids = cq.split(Constants.ID_DELIM); 166 | 167 | String label = AccumuloByteSerializer.deserialize(entry.getValue().get()); 168 | 169 | edge.setVertices(new AccumuloVertex(globals, ids[0]), 170 | new AccumuloVertex(globals, ids[1])); 171 | edge.setLabel(label); 172 | 173 | } finally { 174 | s.close(); 175 | } 176 | } 177 | } 178 | -------------------------------------------------------------------------------- /src/main/java/edu/jhuapl/tinkerpop/tables/core/ElementTableWrapper.java: -------------------------------------------------------------------------------- 1 | /* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory 2 | * 3 | * Licensed under the Apache License, Version 2.0 (the "License"); 4 | * you may not use this file except in compliance with the License. 5 | * You may obtain a copy of the License at 6 | * 7 | * http://www.apache.org/licenses/LICENSE-2.0 8 | * 9 | * Unless required by applicable law or agreed to in writing, software 10 | * distributed under the License is distributed on an "AS IS" BASIS, 11 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | * See the License for the specific language governing permissions and 13 | * limitations under the License. 14 | */ 15 | package edu.jhuapl.tinkerpop.tables.core; 16 | 17 | import java.util.HashSet; 18 | import java.util.Iterator; 19 | import java.util.Map; 20 | import java.util.Set; 21 | import java.util.Map.Entry; 22 | 23 | import org.apache.accumulo.core.client.BatchWriter; 24 | import org.apache.accumulo.core.client.IteratorSetting; 25 | import org.apache.accumulo.core.client.Scanner; 26 | import org.apache.accumulo.core.data.Key; 27 | import org.apache.accumulo.core.data.Range; 28 | import org.apache.accumulo.core.data.Value; 29 | import org.apache.accumulo.core.iterators.user.RegExFilter; 30 | import org.apache.hadoop.io.Text; 31 | 32 | import com.tinkerpop.blueprints.Element; 33 | import com.tinkerpop.blueprints.util.StringFactory; 34 | 35 | import edu.jhuapl.tinkerpop.AccumuloByteSerializer; 36 | import edu.jhuapl.tinkerpop.Constants; 37 | import edu.jhuapl.tinkerpop.GlobalInstances; 38 | import edu.jhuapl.tinkerpop.mutator.property.ClearPropertyMutator; 39 | import edu.jhuapl.tinkerpop.mutator.property.WritePropertyMutator; 40 | import edu.jhuapl.tinkerpop.mutator.Mutators; 41 | import edu.jhuapl.tinkerpop.parser.PropertyParser; 42 | import edu.jhuapl.tinkerpop.tables.BaseTableWrapper; 43 | 44 | /** 45 | * Wrapper around tables with operations 46 | * common to {@link Element}s. 47 | */ 48 | public abstract class ElementTableWrapper extends BaseTableWrapper { 49 | 50 | private BatchWriter writer; 51 | 52 | public ElementTableWrapper(GlobalInstances globals, String tableName) { 53 | super(globals, tableName); 54 | writer = super.getWriter(); 55 | } 56 | 57 | /** 58 | * Give a single instance of the writer for this table. 59 | */ 60 | @Override 61 | protected BatchWriter getWriter() { 62 | return writer; 63 | } 64 | 65 | /** 66 | * Read the given property from the backing table 67 | * for the given element id. 68 | * @param id 69 | * @param key 70 | * @return 71 | */ 72 | public V readProperty(Element element, String key) { 73 | Scanner s = getScanner(); 74 | 75 | s.setRange(new Range(element.getId().toString())); 76 | 77 | Text colf = StringFactory.LABEL.equals(key) 78 | ? new Text(Constants.LABEL) : new Text(key); 79 | s.fetchColumnFamily(colf); 80 | 81 | V value = null; 82 | 83 | Iterator> iter = s.iterator(); 84 | if (iter.hasNext()) { 85 | value = AccumuloByteSerializer.deserialize(iter.next().getValue().get()); 86 | } 87 | s.close(); 88 | 89 | return value; 90 | } 91 | 92 | /** 93 | * Read all properties for the given element 94 | * from the backing table. 95 | * If the element has no properties, return an empty Map. 96 | * If the element does not exist, return null. 97 | * @param element 98 | * @return 99 | */ 100 | public Map readAllProperties(Element element) { 101 | return readProperties(element, null); 102 | } 103 | 104 | /** 105 | * Read the given properties for the given element. 106 | * If propertyKeys is null, read all properties. 107 | * If the element has no properties, return an empty Map. 108 | * If the element does not exist, return null. 109 | * @param id 110 | * @param propertyKeys 111 | * @return 112 | */ 113 | public Map readProperties(Element element, String[] propertyKeys) { 114 | Scanner s = getScanner(); 115 | s.setRange(Range.exact((String) element.getId())); 116 | 117 | // If propertyKeys is null, we read everything. 118 | // Otherwise, limit to the given attributes. 119 | if (propertyKeys != null) { 120 | s.fetchColumnFamily(new Text(Constants.LABEL)); 121 | 122 | for (String key : propertyKeys) { 123 | s.fetchColumnFamily(new Text(key)); 124 | } 125 | } 126 | 127 | Map props = new PropertyParser().parse(s); 128 | s.close(); 129 | 130 | return props; 131 | } 132 | 133 | /** 134 | * Return true if the element with given id exists. 135 | * @param id 136 | * @return 137 | */ 138 | public boolean elementExists(String id) { 139 | Scanner scan = null; 140 | try { 141 | scan = getScanner(); 142 | scan.setRange(Range.exact(id)); 143 | scan.fetchColumnFamily(new Text(Constants.LABEL)); 144 | return new PropertyParser().parse(scan) != null; 145 | 146 | } finally { 147 | if (scan != null) { 148 | scan.close(); 149 | } 150 | } 151 | } 152 | 153 | /** 154 | * Get all property keys for the given element id. 155 | * @param id 156 | * @return 157 | */ 158 | public Set readPropertyKeys(Element element) { 159 | Scanner s = getScanner(); 160 | 161 | s.setRange(new Range(element.getId().toString())); 162 | 163 | Set keys = new HashSet(); 164 | 165 | for (Entry entry : s) { 166 | String cf = entry.getKey().getColumnFamily().toString(); 167 | keys.add(cf); 168 | } 169 | 170 | s.close(); 171 | 172 | // Remove some special keys. 173 | keys.remove(Constants.IN_EDGE); 174 | keys.remove(Constants.LABEL); 175 | keys.remove(Constants.OUT_EDGE); 176 | 177 | return keys; 178 | } 179 | 180 | /** 181 | * Delete the property entry from property table. 182 | * @param id 183 | * @param key 184 | */ 185 | public void clearProperty(Element element, String key) { 186 | Mutators.apply(getWriter(), 187 | new ClearPropertyMutator(element.getId().toString(), key)); 188 | globals.checkedFlush(); 189 | } 190 | 191 | /** 192 | * Write the given property to the property table. 193 | * @param id 194 | * @param key 195 | * @param value 196 | */ 197 | public void writeProperty(Element element, String key, Object value) { 198 | Mutators.apply(getWriter(), 199 | new WritePropertyMutator(element.getId().toString(), 200 | key, value)); 201 | globals.checkedFlush(); 202 | } 203 | 204 | /** 205 | * Add custom iterator to the given scanner so that 206 | * it will only return keys with value corresponding to an edge. 207 | * @param scan 208 | * @param labels 209 | */ 210 | protected void applyEdgeLabelValueFilter(Scanner scan, String... labels) { 211 | StringBuilder regex = new StringBuilder(); 212 | for (String lab : labels) { 213 | if (regex.length() != 0) 214 | regex.append("|"); 215 | regex.append(".*"+Constants.ID_DELIM+"\\Q").append(lab).append("\\E$"); 216 | } 217 | 218 | IteratorSetting is = new IteratorSetting(10, "edgeValueFilter", RegExFilter.class); 219 | RegExFilter.setRegexs(is, null, null, null, regex.toString(), false); 220 | scan.addScanIterator(is); 221 | } 222 | 223 | public void close() { 224 | // TODO? 225 | } 226 | } 227 | -------------------------------------------------------------------------------- /src/main/java/edu/jhuapl/tinkerpop/tables/index/BaseIndexValuesTableWrapper.java: -------------------------------------------------------------------------------- 1 | /* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory 2 | * 3 | * Licensed under the Apache License, Version 2.0 (the "License"); 4 | * you may not use this file except in compliance with the License. 5 | * You may obtain a copy of the License at 6 | * 7 | * http://www.apache.org/licenses/LICENSE-2.0 8 | * 9 | * Unless required by applicable law or agreed to in writing, software 10 | * distributed under the License is distributed on an "AS IS" BASIS, 11 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | * See the License for the specific language governing permissions and 13 | * limitations under the License. 14 | */ 15 | package edu.jhuapl.tinkerpop.tables.index; 16 | 17 | import java.util.Arrays; 18 | import java.util.Collections; 19 | import java.util.Map.Entry; 20 | import java.util.regex.Pattern; 21 | 22 | import org.apache.accumulo.core.client.BatchDeleter; 23 | import org.apache.accumulo.core.client.BatchWriter; 24 | import org.apache.accumulo.core.client.IteratorSetting; 25 | import org.apache.accumulo.core.client.Scanner; 26 | import org.apache.accumulo.core.data.Key; 27 | import org.apache.accumulo.core.data.Range; 28 | import org.apache.accumulo.core.data.Value; 29 | import org.apache.accumulo.core.iterators.user.RegExFilter; 30 | import org.apache.accumulo.core.util.PeekingIterator; 31 | import org.apache.hadoop.io.Text; 32 | 33 | import com.tinkerpop.blueprints.CloseableIterable; 34 | import com.tinkerpop.blueprints.Element; 35 | import com.tinkerpop.blueprints.IndexableGraph; 36 | import com.tinkerpop.blueprints.Vertex; 37 | 38 | import edu.jhuapl.tinkerpop.AccumuloByteSerializer; 39 | import edu.jhuapl.tinkerpop.AccumuloElement; 40 | import edu.jhuapl.tinkerpop.AccumuloGraphException; 41 | import edu.jhuapl.tinkerpop.AccumuloGraphUtils; 42 | import edu.jhuapl.tinkerpop.GlobalInstances; 43 | import edu.jhuapl.tinkerpop.ScannerIterable; 44 | import edu.jhuapl.tinkerpop.mutator.Mutators; 45 | import edu.jhuapl.tinkerpop.mutator.index.IndexValueMutator; 46 | import edu.jhuapl.tinkerpop.parser.EdgeIndexParser; 47 | import edu.jhuapl.tinkerpop.parser.ElementIndexParser; 48 | import edu.jhuapl.tinkerpop.parser.VertexIndexParser; 49 | import edu.jhuapl.tinkerpop.tables.BaseTableWrapper; 50 | 51 | /** 52 | * Wrapper around index tables containing properties 53 | * and values. 54 | */ 55 | public abstract class BaseIndexValuesTableWrapper extends BaseTableWrapper { 56 | 57 | protected final Class elementType; 58 | 59 | protected BaseIndexValuesTableWrapper(GlobalInstances globals, 60 | Class elementType, String tableName) { 61 | super(globals, tableName); 62 | this.elementType = elementType; 63 | } 64 | 65 | /** 66 | * Return class of this index. 67 | * @return 68 | */ 69 | public Class getElementType() { 70 | return elementType; 71 | } 72 | 73 | /** 74 | * Add the property to this index, if autoindexing is enabled 75 | * and/or the given key has indexing enabled. 76 | * @param element 77 | * @param key 78 | * @param value 79 | */ 80 | public void setPropertyForIndex(Element element, String key, Object value) { 81 | setPropertyForIndex(element, key, value, false); 82 | } 83 | 84 | /** 85 | * Add the property to this index. 86 | * 87 | *

Note that this requires a round-trip to Accumulo to see 88 | * if the property exists if the provided key has an index. 89 | * So for best performance, create indices after bulk ingest. 90 | *

If the force parameter is true, set the property regardless 91 | * of whether indexing is enabled for the given key. This is needed 92 | * for {@link IndexableGraph} operations. 93 | * @param element 94 | * @param key 95 | * @param value 96 | * @param force 97 | */ 98 | public void setPropertyForIndex(Element element, String key, Object value, 99 | boolean force) { 100 | AccumuloGraphUtils.validateProperty(key, value); 101 | if (force || globals.getConfig().getAutoIndex() || 102 | globals.getIndexMetadataWrapper() 103 | .getIndexedKeys(elementType).contains(key)) { 104 | BatchWriter writer = getWriter(); 105 | 106 | Object oldValue = element.getProperty(key); 107 | if (oldValue != null && !oldValue.equals(value)) { 108 | Mutators.apply(writer, new IndexValueMutator.Delete(element, key, oldValue)); 109 | } 110 | 111 | Mutators.apply(writer, new IndexValueMutator.Add(element, key, value)); 112 | globals.checkedFlush(); 113 | } 114 | } 115 | 116 | /** 117 | * Remove property from the index. 118 | * @param element 119 | * @param key 120 | * @param value 121 | */ 122 | public void removePropertyFromIndex(Element element, String key, Object value) { 123 | if (value != null) { 124 | Mutators.apply(getWriter(), new IndexValueMutator.Delete(element, key, value)); 125 | globals.checkedFlush(); 126 | } 127 | } 128 | 129 | /** 130 | * Get elements with the key/value pair. 131 | * @param key 132 | * @param value 133 | * @return 134 | */ 135 | @SuppressWarnings("unchecked") 136 | public CloseableIterable readElementsFromIndex(String key, Object value) { 137 | Scanner scan = getScanner(); 138 | byte[] id = AccumuloByteSerializer.serialize(value); 139 | scan.setRange(Range.exact(new Text(id))); 140 | scan.fetchColumnFamily(new Text(key)); 141 | 142 | final ElementIndexParser parser = 143 | Vertex.class.equals(elementType) ? new VertexIndexParser(globals) : 144 | new EdgeIndexParser(globals); 145 | 146 | return new ScannerIterable(scan) { 147 | @Override 148 | public T next(PeekingIterator> iterator) { 149 | return (T) parser.parse(Arrays.asList(iterator.next())); 150 | } 151 | }; 152 | } 153 | 154 | /** 155 | * Remove the given element's properties from the index. 156 | * @param element 157 | */ 158 | public void removeElementFromIndex(Element element) { 159 | BatchDeleter deleter = null; 160 | 161 | try { 162 | deleter = getDeleter(); 163 | deleter.setRanges(Collections.singleton(new Range())); 164 | 165 | IteratorSetting is = new IteratorSetting(10, "getEdgeFilter", RegExFilter.class); 166 | RegExFilter.setRegexs(is, null, null, 167 | "^"+Pattern.quote(element.getId().toString())+"$", null, false); 168 | deleter.addScanIterator(is); 169 | deleter.delete(); 170 | deleter.close(); 171 | } catch (Exception e) { 172 | throw new AccumuloGraphException(e); 173 | } finally { 174 | if (deleter != null) { 175 | deleter.close(); 176 | } 177 | } 178 | } 179 | } 180 | -------------------------------------------------------------------------------- /src/main/java/edu/jhuapl/tinkerpop/tables/index/BaseKeyIndexTableWrapper.java: -------------------------------------------------------------------------------- 1 | /* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory 2 | * 3 | * Licensed under the Apache License, Version 2.0 (the "License"); 4 | * you may not use this file except in compliance with the License. 5 | * You may obtain a copy of the License at 6 | * 7 | * http://www.apache.org/licenses/LICENSE-2.0 8 | * 9 | * Unless required by applicable law or agreed to in writing, software 10 | * distributed under the License is distributed on an "AS IS" BASIS, 11 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | * See the License for the specific language governing permissions and 13 | * limitations under the License. 14 | */ 15 | package edu.jhuapl.tinkerpop.tables.index; 16 | 17 | import com.tinkerpop.blueprints.CloseableIterable; 18 | import com.tinkerpop.blueprints.Edge; 19 | import com.tinkerpop.blueprints.Element; 20 | import com.tinkerpop.blueprints.Vertex; 21 | 22 | import edu.jhuapl.tinkerpop.AccumuloGraphException; 23 | import edu.jhuapl.tinkerpop.GlobalInstances; 24 | import edu.jhuapl.tinkerpop.tables.core.EdgeTableWrapper; 25 | import edu.jhuapl.tinkerpop.tables.core.ElementTableWrapper; 26 | import edu.jhuapl.tinkerpop.tables.core.VertexTableWrapper; 27 | 28 | /** 29 | * Base class for key index tables. 30 | */ 31 | public abstract class BaseKeyIndexTableWrapper extends BaseIndexValuesTableWrapper { 32 | 33 | protected BaseKeyIndexTableWrapper(GlobalInstances globals, 34 | Class elementType, String tableName) { 35 | super(globals, elementType, tableName); 36 | } 37 | 38 | /** 39 | * Rebuild this index for the given table. 40 | * @param table 41 | * @param key 42 | */ 43 | public void rebuildIndex(String key, Class elementClass) { 44 | ElementTableWrapper wrapper = globals.getElementWrapper(elementClass); 45 | if (wrapper instanceof VertexTableWrapper) { 46 | CloseableIterable iter = ((VertexTableWrapper) wrapper).getVertices(); 47 | for (Vertex v : iter) { 48 | rebuild(wrapper, v, key); 49 | } 50 | iter.close(); 51 | } 52 | else if (wrapper instanceof EdgeTableWrapper) { 53 | CloseableIterable iter = ((EdgeTableWrapper) wrapper).getEdges(); 54 | for (Edge e : iter) { 55 | rebuild(wrapper, e, key); 56 | } 57 | iter.close(); 58 | } 59 | else { 60 | throw new AccumuloGraphException("Unexpected table wrapper: "+wrapper.getClass()); 61 | } 62 | globals.checkedFlush(); 63 | } 64 | 65 | /** 66 | * Add given element to index for the given key. 67 | * @param element 68 | * @param key 69 | */ 70 | private void rebuild(ElementTableWrapper wrapper, 71 | Element element, String key) { 72 | Object value = wrapper.readProperty(element, key); 73 | if (value != null) { 74 | setPropertyForIndex(element, key, value); 75 | } 76 | } 77 | } 78 | -------------------------------------------------------------------------------- /src/main/java/edu/jhuapl/tinkerpop/tables/index/EdgeKeyIndexTableWrapper.java: -------------------------------------------------------------------------------- 1 | /* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory 2 | * 3 | * Licensed under the Apache License, Version 2.0 (the "License"); 4 | * you may not use this file except in compliance with the License. 5 | * You may obtain a copy of the License at 6 | * 7 | * http://www.apache.org/licenses/LICENSE-2.0 8 | * 9 | * Unless required by applicable law or agreed to in writing, software 10 | * distributed under the License is distributed on an "AS IS" BASIS, 11 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | * See the License for the specific language governing permissions and 13 | * limitations under the License. 14 | */ 15 | package edu.jhuapl.tinkerpop.tables.index; 16 | 17 | import java.util.Arrays; 18 | import java.util.Map.Entry; 19 | 20 | import org.apache.accumulo.core.client.Scanner; 21 | import org.apache.accumulo.core.data.Key; 22 | import org.apache.accumulo.core.data.Range; 23 | import org.apache.accumulo.core.data.Value; 24 | import org.apache.accumulo.core.util.PeekingIterator; 25 | import org.apache.hadoop.io.Text; 26 | 27 | import com.tinkerpop.blueprints.Edge; 28 | 29 | import edu.jhuapl.tinkerpop.AccumuloByteSerializer; 30 | import edu.jhuapl.tinkerpop.AccumuloEdge; 31 | import edu.jhuapl.tinkerpop.GlobalInstances; 32 | import edu.jhuapl.tinkerpop.ScannerIterable; 33 | import edu.jhuapl.tinkerpop.parser.EdgeIndexParser; 34 | 35 | /** 36 | * Wrapper around {@link Edge} index table. 37 | */ 38 | public class EdgeKeyIndexTableWrapper extends BaseKeyIndexTableWrapper { 39 | 40 | public EdgeKeyIndexTableWrapper(GlobalInstances globals) { 41 | super(globals, Edge.class, globals.getConfig() 42 | .getEdgeKeyIndexTableName()); 43 | } 44 | 45 | /** 46 | * Retrieve edges from the index table based 47 | * on the given key/value. 48 | * @param key 49 | * @param value 50 | * @return 51 | */ 52 | public Iterable getEdges(String key, Object value) { 53 | Scanner s = getScanner(); 54 | 55 | Text row = new Text(AccumuloByteSerializer.serialize(value)); 56 | s.setRange(Range.exact(row)); 57 | s.fetchColumnFamily(new Text(key)); 58 | 59 | final EdgeIndexParser parser = new EdgeIndexParser(globals); 60 | 61 | return new ScannerIterable(s) { 62 | 63 | @Override 64 | public Edge next(PeekingIterator> iterator) { 65 | Entry entry = iterator.next(); 66 | AccumuloEdge e = parser.parse(Arrays.asList(entry)); 67 | 68 | // Check if we have it cached already, in which 69 | // case use the cached version. 70 | AccumuloEdge cached = (AccumuloEdge) globals.getCaches() 71 | .retrieve(e.getId(), Edge.class); 72 | if (cached != null) { 73 | for (String key : e.getPropertyKeysInMemory()) { 74 | cached.setPropertyInMemory(key, e.getPropertyInMemory(key)); 75 | } 76 | 77 | return cached; 78 | } 79 | 80 | // We don't have it, so cache the new one and return it. 81 | globals.getCaches().cache(e, Edge.class); 82 | return e; 83 | } 84 | }; 85 | } 86 | } 87 | -------------------------------------------------------------------------------- /src/main/java/edu/jhuapl/tinkerpop/tables/index/IndexMetadataTableWrapper.java: -------------------------------------------------------------------------------- 1 | /* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory 2 | * 3 | * Licensed under the Apache License, Version 2.0 (the "License"); 4 | * you may not use this file except in compliance with the License. 5 | * You may obtain a copy of the License at 6 | * 7 | * http://www.apache.org/licenses/LICENSE-2.0 8 | * 9 | * Unless required by applicable law or agreed to in writing, software 10 | * distributed under the License is distributed on an "AS IS" BASIS, 11 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | * See the License for the specific language governing permissions and 13 | * limitations under the License. 14 | */ 15 | package edu.jhuapl.tinkerpop.tables.index; 16 | 17 | import java.util.ArrayList; 18 | import java.util.HashSet; 19 | import java.util.List; 20 | import java.util.Set; 21 | 22 | import org.apache.accumulo.core.client.Scanner; 23 | import org.apache.hadoop.io.Text; 24 | 25 | import com.tinkerpop.blueprints.Element; 26 | import com.tinkerpop.blueprints.Index; 27 | import com.tinkerpop.blueprints.IndexableGraph; 28 | import com.tinkerpop.blueprints.KeyIndexableGraph; 29 | import com.tinkerpop.blueprints.util.ExceptionFactory; 30 | 31 | import edu.jhuapl.tinkerpop.AccumuloIndex; 32 | import edu.jhuapl.tinkerpop.Constants.IndexMetadataEntryType; 33 | import edu.jhuapl.tinkerpop.GlobalInstances; 34 | import edu.jhuapl.tinkerpop.mutator.Mutators; 35 | import edu.jhuapl.tinkerpop.mutator.index.IndexMetadataMutator; 36 | import edu.jhuapl.tinkerpop.parser.IndexedItem; 37 | import edu.jhuapl.tinkerpop.parser.IndexedItemsListParser; 38 | import edu.jhuapl.tinkerpop.tables.BaseTableWrapper; 39 | 40 | /** 41 | * Stores metadata, in particular the indexed keys 42 | * for {@link KeyIndexableGraph}, and the list of 43 | * named indexes for {@link IndexableGraph}. 44 | */ 45 | public class IndexMetadataTableWrapper extends BaseTableWrapper { 46 | 47 | public IndexMetadataTableWrapper(GlobalInstances globals) { 48 | super(globals, globals.getConfig().getIndexMetadataTableName()); 49 | } 50 | 51 | 52 | //////// Methods for KeyIndexableGraph //////// 53 | 54 | public void writeKeyMetadataEntry(String key, Class clazz) { 55 | Mutators.apply(getWriter(), new IndexMetadataMutator.Add(key, clazz, 56 | IndexMetadataEntryType.__INDEX_KEY__)); 57 | } 58 | 59 | public void clearKeyMetadataEntry(String key, Class clazz) { 60 | Mutators.apply(getWriter(), new IndexMetadataMutator.Delete(key, clazz, 61 | IndexMetadataEntryType.__INDEX_KEY__)); 62 | } 63 | 64 | public Set getIndexedKeys(Class elementClass) { 65 | if (elementClass == null) { 66 | throw ExceptionFactory.classForElementCannotBeNull(); 67 | } 68 | 69 | IndexedItemsListParser parser = new IndexedItemsListParser(elementClass); 70 | 71 | Scanner scan = null; 72 | try { 73 | scan = getScanner(); 74 | scan.fetchColumnFamily(new Text(IndexMetadataEntryType.__INDEX_KEY__.name())); 75 | 76 | Set keys = new HashSet(); 77 | for (IndexedItem item : parser.parse(scan)) { 78 | keys.add(item.getKey()); 79 | } 80 | 81 | return keys; 82 | 83 | } finally { 84 | if (scan != null) { 85 | scan.close(); 86 | } 87 | } 88 | } 89 | 90 | 91 | //////// Methods for IndexableGraph //////// 92 | 93 | @SuppressWarnings({"rawtypes", "unchecked"}) 94 | public Iterable> getIndices() { 95 | List> indexes = new ArrayList>(); 96 | 97 | IndexedItemsListParser parser = new IndexedItemsListParser(); 98 | 99 | Scanner scan = null; 100 | try { 101 | scan = getScanner(); 102 | scan.fetchColumnFamily(new Text(IndexMetadataEntryType.__INDEX_NAME__.name())); 103 | 104 | for (IndexedItem item : parser.parse(scan)) { 105 | indexes.add(new AccumuloIndex(globals, 106 | item.getKey(), item.getElementClass())); 107 | } 108 | 109 | return indexes; 110 | 111 | } finally { 112 | if (scan != null) { 113 | scan.close(); 114 | } 115 | } 116 | } 117 | 118 | public Index getIndex(String indexName, 119 | Class indexClass) { 120 | IndexedItemsListParser parser = new IndexedItemsListParser(); 121 | 122 | Scanner scan = null; 123 | try { 124 | scan = getScanner(); 125 | scan.fetchColumnFamily(new Text(IndexMetadataEntryType.__INDEX_NAME__.name())); 126 | 127 | for (IndexedItem item : parser.parse(scan)) { 128 | if (item.getKey().equals(indexName)) { 129 | if (item.getElementClass().equals(indexClass)) { 130 | return new AccumuloIndex(globals, indexName, 131 | indexClass); 132 | } 133 | else { 134 | throw ExceptionFactory.indexDoesNotSupportClass(indexName, indexClass); 135 | } 136 | } 137 | } 138 | return null; 139 | 140 | } finally { 141 | if (scan != null) { 142 | scan.close(); 143 | } 144 | } 145 | } 146 | 147 | public Index createIndex(String indexName, 148 | Class indexClass) { 149 | for (Index index : getIndices()) { 150 | if (index.getIndexName().equals(indexName)) { 151 | throw ExceptionFactory.indexAlreadyExists(indexName); 152 | } 153 | } 154 | 155 | writeIndexNameEntry(indexName, indexClass); 156 | return new AccumuloIndex(globals, indexName, indexClass); 157 | } 158 | 159 | private void writeIndexNameEntry(String indexName, 160 | Class indexClass) { 161 | Mutators.apply(getWriter(), new IndexMetadataMutator.Add(indexName, 162 | indexClass, IndexMetadataEntryType.__INDEX_NAME__)); 163 | } 164 | 165 | public void clearIndexNameEntry(String indexName, 166 | Class indexClass) { 167 | Mutators.apply(getWriter(), new IndexMetadataMutator.Delete(indexName, 168 | indexClass, IndexMetadataEntryType.__INDEX_NAME__)); 169 | } 170 | } 171 | -------------------------------------------------------------------------------- /src/main/java/edu/jhuapl/tinkerpop/tables/index/NamedIndexTableWrapper.java: -------------------------------------------------------------------------------- 1 | /* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory 2 | * 3 | * Licensed under the Apache License, Version 2.0 (the "License"); 4 | * you may not use this file except in compliance with the License. 5 | * You may obtain a copy of the License at 6 | * 7 | * http://www.apache.org/licenses/LICENSE-2.0 8 | * 9 | * Unless required by applicable law or agreed to in writing, software 10 | * distributed under the License is distributed on an "AS IS" BASIS, 11 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | * See the License for the specific language governing permissions and 13 | * limitations under the License. 14 | */ 15 | package edu.jhuapl.tinkerpop.tables.index; 16 | 17 | import com.tinkerpop.blueprints.Element; 18 | import com.tinkerpop.blueprints.IndexableGraph; 19 | 20 | import edu.jhuapl.tinkerpop.GlobalInstances; 21 | 22 | /** 23 | * Wrapper around a named index table (for {@link IndexableGraph}). 24 | */ 25 | public class NamedIndexTableWrapper extends BaseIndexValuesTableWrapper { 26 | 27 | public NamedIndexTableWrapper(GlobalInstances globals, 28 | Class elementType, String indexName) { 29 | super(globals, elementType, 30 | globals.getConfig().getNamedIndexTableName(indexName)); 31 | } 32 | } 33 | -------------------------------------------------------------------------------- /src/main/java/edu/jhuapl/tinkerpop/tables/index/VertexKeyIndexTableWrapper.java: -------------------------------------------------------------------------------- 1 | /* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory 2 | * 3 | * Licensed under the Apache License, Version 2.0 (the "License"); 4 | * you may not use this file except in compliance with the License. 5 | * You may obtain a copy of the License at 6 | * 7 | * http://www.apache.org/licenses/LICENSE-2.0 8 | * 9 | * Unless required by applicable law or agreed to in writing, software 10 | * distributed under the License is distributed on an "AS IS" BASIS, 11 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | * See the License for the specific language governing permissions and 13 | * limitations under the License. 14 | */ 15 | package edu.jhuapl.tinkerpop.tables.index; 16 | 17 | import java.util.Arrays; 18 | import java.util.Map.Entry; 19 | 20 | import org.apache.accumulo.core.client.Scanner; 21 | import org.apache.accumulo.core.data.Key; 22 | import org.apache.accumulo.core.data.Range; 23 | import org.apache.accumulo.core.data.Value; 24 | import org.apache.accumulo.core.util.PeekingIterator; 25 | import org.apache.hadoop.io.Text; 26 | 27 | import com.tinkerpop.blueprints.Vertex; 28 | 29 | import edu.jhuapl.tinkerpop.AccumuloByteSerializer; 30 | import edu.jhuapl.tinkerpop.AccumuloVertex; 31 | import edu.jhuapl.tinkerpop.GlobalInstances; 32 | import edu.jhuapl.tinkerpop.ScannerIterable; 33 | import edu.jhuapl.tinkerpop.parser.VertexIndexParser; 34 | 35 | /** 36 | * Wrapper around {@link Vertex} index table. 37 | */ 38 | public class VertexKeyIndexTableWrapper extends BaseKeyIndexTableWrapper { 39 | 40 | public VertexKeyIndexTableWrapper(GlobalInstances globals) { 41 | super(globals, Vertex.class, globals.getConfig() 42 | .getVertexKeyIndexTableName()); 43 | } 44 | 45 | /** 46 | * Use the index to retrieve vertices with the 47 | * given key/value. 48 | * @param key 49 | * @param value 50 | */ 51 | public Iterable getVertices(String key, Object value) { 52 | Scanner s = getScanner(); 53 | 54 | Text row = new Text(AccumuloByteSerializer.serialize(value)); 55 | s.setRange(Range.exact(row)); 56 | s.fetchColumnFamily(new Text(key)); 57 | 58 | final VertexIndexParser parser = new VertexIndexParser(globals); 59 | 60 | return new ScannerIterable(s) { 61 | 62 | @Override 63 | public Vertex next(PeekingIterator> iterator) { 64 | Entry entry = iterator.next(); 65 | AccumuloVertex v = parser.parse(Arrays.asList(entry)); 66 | 67 | // Check if we have it cached already, in which 68 | // case use the cached version. 69 | AccumuloVertex cached = (AccumuloVertex) globals.getCaches() 70 | .retrieve(v.getId(), Vertex.class); 71 | if (cached != null) { 72 | for (String key : v.getPropertyKeysInMemory()) { 73 | cached.setPropertyInMemory(key, v.getPropertyInMemory(key)); 74 | } 75 | 76 | return cached; 77 | } 78 | 79 | // We don't have it, so cache the new one and return it. 80 | globals.getCaches().cache(v, Vertex.class); 81 | return v; 82 | } 83 | }; 84 | } 85 | } 86 | -------------------------------------------------------------------------------- /src/main/resources/log4j.properties: -------------------------------------------------------------------------------- 1 | log4j.rootLogger=INFO, stdout 2 | 3 | log4j.appender.stdout=org.apache.log4j.ConsoleAppender 4 | log4j.appender.stdout.Target=System.out 5 | log4j.appender.stdout.layout=org.apache.log4j.PatternLayout 6 | log4j.appender.stdout.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %-5p %c - %m%n 7 | -------------------------------------------------------------------------------- /src/test/java/edu/jhuapl/tinkerpop/AccumuloBulkIngesterTest.java: -------------------------------------------------------------------------------- 1 | /* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory 2 | * 3 | * Licensed under the Apache License, Version 2.0 (the "License"); 4 | * you may not use this file except in compliance with the License. 5 | * You may obtain a copy of the License at 6 | * 7 | * http://www.apache.org/licenses/LICENSE-2.0 8 | * 9 | * Unless required by applicable law or agreed to in writing, software 10 | * distributed under the License is distributed on an "AS IS" BASIS, 11 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | * See the License for the specific language governing permissions and 13 | * limitations under the License. 14 | */ 15 | package edu.jhuapl.tinkerpop; 16 | 17 | import static org.junit.Assert.assertEquals; 18 | import static org.junit.Assert.assertNotNull; 19 | import static org.junit.Assert.assertTrue; 20 | 21 | import java.util.Iterator; 22 | 23 | import org.junit.Test; 24 | 25 | import com.tinkerpop.blueprints.Direction; 26 | import com.tinkerpop.blueprints.Edge; 27 | import com.tinkerpop.blueprints.Vertex; 28 | 29 | public class AccumuloBulkIngesterTest { 30 | 31 | @Test 32 | public void testBulkIngester() throws Exception { 33 | AccumuloGraphConfiguration cfg = AccumuloGraphTestUtils.generateGraphConfig("propertyBuilder").setClear(true); 34 | 35 | AccumuloBulkIngester ingester = new AccumuloBulkIngester(cfg); 36 | 37 | for (String t : cfg.getTableNames()) { 38 | assertTrue(cfg.getConnector().tableOperations().exists(t)); 39 | } 40 | 41 | ingester.addVertex("A").finish(); 42 | ingester.addVertex("B").add("P1", "V1").add("P2", "2").finish(); 43 | ingester.addEdge("A", "B", "edge").add("P3", "V3").finish(); 44 | ingester.shutdown(true); 45 | 46 | AccumuloGraph graph = new AccumuloGraph(cfg.clone().setClear(false)); 47 | Vertex v1 = graph.getVertex("A"); 48 | assertNotNull(v1); 49 | 50 | Iterator it = v1.getEdges(Direction.OUT).iterator(); 51 | assertTrue(it.hasNext()); 52 | 53 | Edge e = it.next(); 54 | assertEquals("edge", e.getLabel()); 55 | 56 | Vertex v2 = e.getVertex(Direction.IN); 57 | assertEquals("B", v2.getId()); 58 | assertEquals("V1", v2.getProperty("P1")); 59 | assertEquals("2", v2.getProperty("P2")); 60 | 61 | graph.shutdown(); 62 | } 63 | 64 | } 65 | -------------------------------------------------------------------------------- /src/test/java/edu/jhuapl/tinkerpop/AccumuloElementTest.java: -------------------------------------------------------------------------------- 1 | /* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory 2 | * 3 | * Licensed under the Apache License, Version 2.0 (the "License"); 4 | * you may not use this file except in compliance with the License. 5 | * You may obtain a copy of the License at 6 | * 7 | * http://www.apache.org/licenses/LICENSE-2.0 8 | * 9 | * Unless required by applicable law or agreed to in writing, software 10 | * distributed under the License is distributed on an "AS IS" BASIS, 11 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | * See the License for the specific language governing permissions and 13 | * limitations under the License. 14 | */ 15 | package edu.jhuapl.tinkerpop; 16 | 17 | import static org.junit.Assert.*; 18 | 19 | import org.junit.Test; 20 | 21 | import com.tinkerpop.blueprints.Edge; 22 | import com.tinkerpop.blueprints.Graph; 23 | import com.tinkerpop.blueprints.Vertex; 24 | 25 | /** 26 | * Tests related to Accumulo elements. 27 | */ 28 | public class AccumuloElementTest { 29 | 30 | @Test 31 | public void testNonStringIds() throws Exception { 32 | Graph graph = AccumuloGraphTestUtils.makeGraph("nonStringIds"); 33 | 34 | Object[] ids = new Object[] { 35 | 10, 20, 30L, 40L, 36 | 50.0f, 60.0f, 70.0d, 80.0d, 37 | (byte) 'a', (byte) 'b', 'c', 'd', 38 | "str1", "str2", 39 | new GenericObject("str3"), new GenericObject("str4"), 40 | }; 41 | 42 | Object[] edgeIds = new Object[] { 43 | 100, 200, 300L, 400L, 44 | 500.0f, 600.0f, 700.0d, 800.0d, 45 | (byte) 'e', (byte) 'f', 'g', 'h', 46 | "str5", "str6", 47 | new GenericObject("str7"), new GenericObject("str8"), 48 | }; 49 | 50 | for (int i = 0; i < ids.length; i++) { 51 | assertNull(graph.getVertex(ids[i])); 52 | Vertex v = graph.addVertex(ids[i]); 53 | assertNotNull(v); 54 | assertNotNull(graph.getVertex(ids[i])); 55 | } 56 | assertEquals(ids.length, count(graph.getVertices())); 57 | 58 | for (int i = 1; i < edgeIds.length; i++) { 59 | assertNull(graph.getEdge(edgeIds[i-1])); 60 | Edge e = graph.addEdge(edgeIds[i-1], 61 | graph.getVertex(ids[i-1]), 62 | graph.getVertex(ids[i]), "label"); 63 | assertNotNull(e); 64 | assertNotNull(graph.getEdge(edgeIds[i-1])); 65 | } 66 | assertEquals(edgeIds.length-1, count(graph.getEdges())); 67 | 68 | graph.shutdown(); 69 | } 70 | 71 | @Test 72 | public void testGetVerticesInRange() { 73 | AccumuloGraph graph = (AccumuloGraph) AccumuloGraphTestUtils 74 | .makeGraph("testGetVerticesInRange"); 75 | 76 | assertEquals(0, count(graph.getVerticesInRange(null, null))); 77 | for (int i = 0; i < 10; i++) { 78 | graph.addVertex(id(i)); 79 | } 80 | 81 | assertEquals(10, count(graph.getVerticesInRange(null, null))); 82 | assertEquals(0, count(graph.getVerticesInRange(id(20), null))); 83 | assertEquals(10, count(graph.getVerticesInRange(null, id(20)))); 84 | assertEquals(0, count(graph.getVerticesInRange(id(20), id(20)))); 85 | 86 | for (int i = 0; i < 10; i++) { 87 | assertEquals(i+1, count(graph.getVerticesInRange(null, id(i)))); 88 | assertEquals(10-i, count(graph.getVerticesInRange(id(i), null))); 89 | assertEquals(1, count(graph.getVerticesInRange(id(i), id(i)))); 90 | } 91 | 92 | graph.shutdown(); 93 | } 94 | 95 | private static String id(int idNum) { 96 | return String.format("%08d", idNum); 97 | } 98 | 99 | private static int count(Iterable iter) { 100 | int count = 0; 101 | for (@SuppressWarnings("unused") Object obj : iter) { 102 | count++; 103 | } 104 | return count; 105 | } 106 | 107 | private static class GenericObject { 108 | private final Object id; 109 | 110 | public GenericObject(Object id) { 111 | this.id = id; 112 | } 113 | 114 | @Override 115 | public String toString() { 116 | return "GenericObject [id=" + id + "]"; 117 | } 118 | } 119 | } 120 | -------------------------------------------------------------------------------- /src/test/java/edu/jhuapl/tinkerpop/AccumuloGraphTest.java: -------------------------------------------------------------------------------- 1 | /* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory 2 | * 3 | * Licensed under the Apache License, Version 2.0 (the "License"); 4 | * you may not use this file except in compliance with the License. 5 | * You may obtain a copy of the License at 6 | * 7 | * http://www.apache.org/licenses/LICENSE-2.0 8 | * 9 | * Unless required by applicable law or agreed to in writing, software 10 | * distributed under the License is distributed on an "AS IS" BASIS, 11 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | * See the License for the specific language governing permissions and 13 | * limitations under the License. 14 | */ 15 | package edu.jhuapl.tinkerpop; 16 | 17 | import java.lang.reflect.Method; 18 | 19 | import com.tinkerpop.blueprints.EdgeTestSuite; 20 | import com.tinkerpop.blueprints.Graph; 21 | import com.tinkerpop.blueprints.GraphFactory; 22 | import com.tinkerpop.blueprints.GraphTestSuite; 23 | import com.tinkerpop.blueprints.IndexTestSuite; 24 | import com.tinkerpop.blueprints.IndexableGraphTestSuite; 25 | import com.tinkerpop.blueprints.KeyIndexableGraphTestSuite; 26 | import com.tinkerpop.blueprints.TestSuite; 27 | import com.tinkerpop.blueprints.VertexTestSuite; 28 | import com.tinkerpop.blueprints.impls.GraphTest; 29 | import com.tinkerpop.blueprints.util.io.gml.GMLReaderTestSuite; 30 | import com.tinkerpop.blueprints.util.io.graphml.GraphMLReaderTestSuite; 31 | import com.tinkerpop.blueprints.util.io.graphson.GraphSONReaderTestSuite; 32 | 33 | public class AccumuloGraphTest extends GraphTest { 34 | 35 | ThreadLocal testGraphName = new ThreadLocal(); 36 | 37 | public void testVertexTestSuite() throws Exception { 38 | this.stopWatch(); 39 | doTestSuite(new VertexTestSuite(this)); 40 | printTestPerformance("VertexTestSuite", this.stopWatch()); 41 | } 42 | 43 | public void testEdgeTestSuite() throws Exception { 44 | this.stopWatch(); 45 | doTestSuite(new EdgeTestSuite(this)); 46 | printTestPerformance("EdgeTestSuite", this.stopWatch()); 47 | } 48 | 49 | public void testGraphTestSuite() throws Exception { 50 | this.stopWatch(); 51 | doTestSuite(new GraphTestSuite(this)); 52 | printTestPerformance("GraphTestSuite", this.stopWatch()); 53 | } 54 | 55 | public void testKeyIndexableGraphTestSuite() throws Exception { 56 | this.stopWatch(); 57 | doTestSuite(new KeyIndexableGraphTestSuite(this)); 58 | printTestPerformance("KeyIndexableGraphTestSuite", this.stopWatch()); 59 | } 60 | 61 | public void testIndexableGraphTestSuite() throws Exception { 62 | this.stopWatch(); 63 | doTestSuite(new IndexableGraphTestSuite(this)); 64 | printTestPerformance("IndexableGraphTestSuite", this.stopWatch()); 65 | } 66 | 67 | public void testIndexTestSuite() throws Exception { 68 | this.stopWatch(); 69 | doTestSuite(new IndexTestSuite(this)); 70 | printTestPerformance("IndexTestSuite", this.stopWatch()); 71 | } 72 | 73 | public void testGraphMLReaderTestSuite() throws Exception { 74 | this.stopWatch(); 75 | doTestSuite(new GraphMLReaderTestSuite(this)); 76 | printTestPerformance("GraphMLReaderTestSuite", this.stopWatch()); 77 | } 78 | 79 | public void testGMLReaderTestSuite() throws Exception { 80 | this.stopWatch(); 81 | doTestSuite(new GMLReaderTestSuite(this)); 82 | printTestPerformance("GMLReaderTestSuite", this.stopWatch()); 83 | } 84 | 85 | public void testGraphSONReaderTestSuite() throws Exception { 86 | this.stopWatch(); 87 | doTestSuite(new GraphSONReaderTestSuite(this)); 88 | printTestPerformance("GraphSONReaderTestSuite", this.stopWatch()); 89 | } 90 | 91 | @Override 92 | public void doTestSuite(final TestSuite testSuite) throws Exception { 93 | String doTest = System.getProperty("testTinkerGraph"); 94 | if (doTest == null || doTest.equals("true")) { 95 | for (Method method : testSuite.getClass().getDeclaredMethods()) { 96 | if (method.getName().startsWith("test")) { 97 | System.out.println("Testing " + method.getName() + "..."); 98 | method.invoke(testSuite); 99 | dropGraph(testGraphName.get()); 100 | } 101 | } 102 | } 103 | } 104 | 105 | @Override 106 | public Graph generateGraph() { 107 | return generateGraph("AAA_test_graph"); 108 | } 109 | 110 | static { 111 | // So setting up the Accumulo Mock Cluster does not affect the test 112 | // times 113 | new AccumuloGraphTest().generateGraph(); 114 | } 115 | 116 | @Override 117 | public void dropGraph(final String graphDirectoryName) { 118 | if (graphDirectoryName != null) { 119 | AccumuloGraphConfiguration cfg = AccumuloGraphTestUtils.generateGraphConfig(graphDirectoryName); 120 | try { 121 | for (String table : cfg.getConnector().tableOperations().list()) { 122 | cfg.getConnector().tableOperations().delete(table); 123 | } 124 | } catch (Exception e) { 125 | throw new AccumuloGraphException(e); 126 | } 127 | } 128 | } 129 | 130 | @Override 131 | public Object convertId(final Object id) { 132 | return id.toString(); 133 | } 134 | 135 | @Override 136 | public Graph generateGraph(String graphDirectoryName) { 137 | AccumuloGraphConfiguration cfg = AccumuloGraphTestUtils.generateGraphConfig(graphDirectoryName); 138 | testGraphName.set(graphDirectoryName); 139 | return GraphFactory.open(cfg.getConfiguration()); 140 | } 141 | } 142 | -------------------------------------------------------------------------------- /src/test/java/edu/jhuapl/tinkerpop/AccumuloGraphTestUtils.java: -------------------------------------------------------------------------------- 1 | /* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory 2 | * 3 | * Licensed under the Apache License, Version 2.0 (the "License"); 4 | * you may not use this file except in compliance with the License. 5 | * You may obtain a copy of the License at 6 | * 7 | * http://www.apache.org/licenses/LICENSE-2.0 8 | * 9 | * Unless required by applicable law or agreed to in writing, software 10 | * distributed under the License is distributed on an "AS IS" BASIS, 11 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | * See the License for the specific language governing permissions and 13 | * limitations under the License. 14 | */ 15 | package edu.jhuapl.tinkerpop; 16 | 17 | import com.tinkerpop.blueprints.Graph; 18 | import com.tinkerpop.blueprints.GraphFactory; 19 | 20 | import edu.jhuapl.tinkerpop.AccumuloGraphConfiguration.InstanceType; 21 | 22 | public class AccumuloGraphTestUtils { 23 | 24 | public static AccumuloGraphConfiguration generateGraphConfig(String graphDirectoryName) { 25 | return new AccumuloGraphConfiguration().setInstanceType(InstanceType.Mock) 26 | .setGraphName(graphDirectoryName).setCreate(true); 27 | } 28 | 29 | public static Graph makeGraph(String name) { 30 | return GraphFactory.open(generateGraphConfig(name)); 31 | } 32 | } 33 | -------------------------------------------------------------------------------- /src/test/java/edu/jhuapl/tinkerpop/AutoIndexTest.java: -------------------------------------------------------------------------------- 1 | package edu.jhuapl.tinkerpop; 2 | 3 | import static org.junit.Assert.*; 4 | 5 | import org.junit.Test; 6 | 7 | import com.tinkerpop.blueprints.Edge; 8 | import com.tinkerpop.blueprints.Element; 9 | import com.tinkerpop.blueprints.GraphFactory; 10 | import com.tinkerpop.blueprints.Vertex; 11 | 12 | public class AutoIndexTest { 13 | 14 | @Test 15 | public void testVertexAutoIndex() throws Exception { 16 | AccumuloGraph graph = (AccumuloGraph) GraphFactory.open(AccumuloGraphTestUtils 17 | .generateGraphConfig("VertexAutoIndexTest").setAutoIndex(true).getConfiguration()); 18 | String id = "1234"; 19 | String key = "name"; 20 | String value = "bananaman"; 21 | 22 | Vertex v1 = graph.addVertex(id); 23 | v1.setProperty(key, value); 24 | 25 | Iterable elements = graph.getGlobals() 26 | .getVertexKeyIndexWrapper().readElementsFromIndex(key, value); 27 | int count = 0; 28 | for (Element element : elements) { 29 | assertTrue(element instanceof Vertex); 30 | assertEquals(id, element.getId()); 31 | assertEquals(value, element.getProperty(key)); 32 | count++; 33 | } 34 | assertEquals(1, count); 35 | 36 | graph.removeVertex(v1); 37 | elements = graph.getGlobals() 38 | .getVertexKeyIndexWrapper().readElementsFromIndex(key, value); 39 | assertEquals(0, count(elements)); 40 | } 41 | 42 | @Test 43 | public void testVertexNoAutoIndex() throws Exception { 44 | AccumuloGraph graph = (AccumuloGraph) GraphFactory.open(AccumuloGraphTestUtils 45 | .generateGraphConfig("VertexNoAutoIndexTest").getConfiguration()); 46 | String id = "1234"; 47 | String key = "name"; 48 | String value = "bananaman"; 49 | 50 | Vertex v1 = graph.addVertex(id); 51 | v1.setProperty(key, value); 52 | 53 | Iterable elements = graph.getGlobals() 54 | .getVertexKeyIndexWrapper().readElementsFromIndex(key, value); 55 | assertEquals(0, count(elements)); 56 | } 57 | 58 | @Test 59 | public void testEdgeAutoIndex() throws Exception { 60 | AccumuloGraph graph = (AccumuloGraph) GraphFactory.open(AccumuloGraphTestUtils 61 | .generateGraphConfig("EdgeAutoIndex").setAutoIndex(true).getConfiguration()); 62 | String id1 = "A"; 63 | String id2 = "B"; 64 | String eid = "X"; 65 | String key = "name"; 66 | String value = "bananaman"; 67 | 68 | Vertex v1 = graph.addVertex(id1); 69 | Vertex v2 = graph.addVertex(id2); 70 | Edge e = graph.addEdge(eid, v1, v2, "edge"); 71 | e.setProperty(key, value); 72 | 73 | Iterable elements = graph.getGlobals() 74 | .getEdgeKeyIndexWrapper().readElementsFromIndex(key, value); 75 | int count = 0; 76 | for (Element element : elements) { 77 | assertTrue(element instanceof Edge); 78 | assertEquals(eid, element.getId()); 79 | assertEquals(value, element.getProperty(key)); 80 | count++; 81 | } 82 | assertEquals(1, count); 83 | 84 | graph.removeVertex(v1); 85 | elements = graph.getGlobals() 86 | .getEdgeKeyIndexWrapper().readElementsFromIndex(key, value); 87 | assertEquals(0, count(elements)); 88 | } 89 | 90 | @Test 91 | public void testEdgeNoAutoIndex() throws Exception { 92 | AccumuloGraph graph = (AccumuloGraph) GraphFactory.open(AccumuloGraphTestUtils 93 | .generateGraphConfig("EdgeNoAutoIndexTest").getConfiguration()); 94 | String id1 = "A"; 95 | String id2 = "B"; 96 | String eid = "X"; 97 | String key = "name"; 98 | String value = "bananaman"; 99 | 100 | Vertex v1 = graph.addVertex(id1); 101 | Vertex v2 = graph.addVertex(id2); 102 | Edge e = graph.addEdge(eid, v1, v2, "edge"); 103 | e.setProperty(key, value); 104 | 105 | Iterable elements = graph.getGlobals() 106 | .getEdgeKeyIndexWrapper().readElementsFromIndex(key, value); 107 | assertEquals(0, count(elements)); 108 | } 109 | 110 | @SuppressWarnings("unused") 111 | private static int count(Iterable it) { 112 | int count = 0; 113 | for (Object obj : it) { 114 | count++; 115 | } 116 | return count; 117 | } 118 | } 119 | -------------------------------------------------------------------------------- /src/test/java/edu/jhuapl/tinkerpop/DistributedInstanceTest.java: -------------------------------------------------------------------------------- 1 | package edu.jhuapl.tinkerpop; 2 | 3 | import org.apache.commons.configuration.Configuration; 4 | import org.junit.Ignore; 5 | 6 | import com.tinkerpop.blueprints.Graph; 7 | import com.tinkerpop.blueprints.GraphFactory; 8 | 9 | import edu.jhuapl.tinkerpop.AccumuloGraphConfiguration.InstanceType; 10 | 11 | /** 12 | * Run the test suite for a Distributed instance type. 13 | *

Note: This is disabled by default since we can't 14 | * guarantee an Accumulo cluster setup in a test environment. 15 | * To use this, set the constants below and remove 16 | * the @Ignore annotation. 17 | */ 18 | @Ignore 19 | public class DistributedInstanceTest extends AccumuloGraphTest { 20 | 21 | // Connection constants. 22 | private static final String ZOOKEEPERS = "zkhost-1,zkhost-2"; 23 | private static final String INSTANCE = "accumulo-instance"; 24 | private static final String USER = "user"; 25 | private static final String PASSWORD = "password"; 26 | 27 | @Override 28 | public Graph generateGraph(String graphDirectoryName) { 29 | Configuration cfg = new AccumuloGraphConfiguration() 30 | .setInstanceType(InstanceType.Distributed) 31 | .setZooKeeperHosts(ZOOKEEPERS) 32 | .setInstanceName(INSTANCE) 33 | .setUser(USER).setPassword(PASSWORD) 34 | .setGraphName(graphDirectoryName) 35 | .setCreate(true); 36 | testGraphName.set(graphDirectoryName); 37 | return GraphFactory.open(cfg); 38 | } 39 | } 40 | -------------------------------------------------------------------------------- /src/test/java/edu/jhuapl/tinkerpop/ElementCacheTest.java: -------------------------------------------------------------------------------- 1 | /* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory 2 | * 3 | * Licensed under the Apache License, Version 2.0 (the "License"); 4 | * you may not use this file except in compliance with the License. 5 | * You may obtain a copy of the License at 6 | * 7 | * http://www.apache.org/licenses/LICENSE-2.0 8 | * 9 | * Unless required by applicable law or agreed to in writing, software 10 | * distributed under the License is distributed on an "AS IS" BASIS, 11 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | * See the License for the specific language governing permissions and 13 | * limitations under the License. 14 | */ 15 | package edu.jhuapl.tinkerpop; 16 | 17 | import org.junit.Test; 18 | 19 | import static org.junit.Assert.*; 20 | 21 | import com.tinkerpop.blueprints.Edge; 22 | import com.tinkerpop.blueprints.Element; 23 | import com.tinkerpop.blueprints.Graph; 24 | import com.tinkerpop.blueprints.GraphFactory; 25 | import com.tinkerpop.blueprints.Vertex; 26 | 27 | import edu.jhuapl.tinkerpop.cache.ElementCache; 28 | 29 | public class ElementCacheTest { 30 | 31 | @Test 32 | public void testElementCacheSize() throws Exception { 33 | AccumuloGraphConfiguration cfg = AccumuloGraphTestUtils 34 | .generateGraphConfig("elementCacheSize"); 35 | Graph graph = GraphFactory.open(cfg.getConfiguration()); 36 | 37 | Vertex[] verts = new Vertex[10]; 38 | for (int i = 0; i < verts.length; i++) { 39 | verts[i] = graph.addVertex(i); 40 | } 41 | 42 | Edge[] edges = new Edge[9]; 43 | for (int i = 0; i < edges.length; i++) { 44 | edges[i] = graph.addEdge(null, 45 | verts[0], verts[i+1], "edge"); 46 | } 47 | 48 | sizeTests(verts); 49 | sizeTests(edges); 50 | 51 | graph.shutdown(); 52 | } 53 | 54 | private void sizeTests(Element[] elts) { 55 | ElementCache cache = 56 | new ElementCache(3, 120000); 57 | for (Element e : elts) { 58 | cache.cache(e); 59 | } 60 | 61 | for (Element e : elts) { 62 | cache.cache(e); 63 | } 64 | 65 | int total = 0; 66 | for (Element e : elts) { 67 | if (cache.retrieve(e.getId()) != null) { 68 | total++; 69 | } 70 | } 71 | assertTrue(total < elts.length); 72 | 73 | cache.clear(); 74 | for (Element e : elts) { 75 | assertNull(cache.retrieve(e.getId())); 76 | } 77 | } 78 | 79 | @Test 80 | public void testElementCacheTimeout() throws Exception { 81 | AccumuloGraphConfiguration cfg = AccumuloGraphTestUtils 82 | .generateGraphConfig("elementCacheTimeout"); 83 | Graph graph = GraphFactory.open(cfg.getConfiguration()); 84 | 85 | ElementCache cache = 86 | new ElementCache(10, 1000); 87 | 88 | Vertex v1 = graph.addVertex(1); 89 | Vertex v2 = graph.addVertex(2); 90 | assertNull(cache.retrieve(1)); 91 | assertNull(cache.retrieve(2)); 92 | 93 | cache.cache(v1); 94 | assertNotNull(cache.retrieve(v1.getId())); 95 | Thread.sleep(1500); 96 | assertNull(cache.retrieve(v1.getId())); 97 | 98 | Edge e = graph.addEdge(null, v1, v2, "label"); 99 | assertNull(cache.retrieve(e.getId())); 100 | 101 | cache.cache(e); 102 | assertNotNull(cache.retrieve(e.getId())); 103 | Thread.sleep(1500); 104 | assertNull(cache.retrieve(e.getId())); 105 | 106 | graph.shutdown(); 107 | } 108 | } 109 | -------------------------------------------------------------------------------- /src/test/java/edu/jhuapl/tinkerpop/ExtendedAccumuloGraphTest.java: -------------------------------------------------------------------------------- 1 | package edu.jhuapl.tinkerpop; 2 | 3 | import com.tinkerpop.blueprints.Graph; 4 | import com.tinkerpop.blueprints.GraphFactory; 5 | 6 | public class ExtendedAccumuloGraphTest extends AccumuloGraphTest { 7 | 8 | @Override 9 | public Graph generateGraph(String graphDirectoryName) { 10 | AccumuloGraphConfiguration cfg = AccumuloGraphTestUtils.generateGraphConfig(graphDirectoryName); 11 | cfg.setEdgeCacheParams(20, 30000) 12 | .setPreloadedProperties(new String[] {"name"}) 13 | .setPreloadedEdgeLabels(new String[] {"knows"}) 14 | .setPropertyCacheTimeout("name", 100000); 15 | testGraphName.set(graphDirectoryName); 16 | return GraphFactory.open(cfg.getConfiguration()); 17 | } 18 | 19 | } 20 | -------------------------------------------------------------------------------- /src/test/java/edu/jhuapl/tinkerpop/ExtendedElementTest.java: -------------------------------------------------------------------------------- 1 | /* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory 2 | * 3 | * Licensed under the Apache License, Version 2.0 (the "License"); 4 | * you may not use this file except in compliance with the License. 5 | * You may obtain a copy of the License at 6 | * 7 | * http://www.apache.org/licenses/LICENSE-2.0 8 | * 9 | * Unless required by applicable law or agreed to in writing, software 10 | * distributed under the License is distributed on an "AS IS" BASIS, 11 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | * See the License for the specific language governing permissions and 13 | * limitations under the License. 14 | */ 15 | package edu.jhuapl.tinkerpop; 16 | 17 | import static org.junit.Assert.*; 18 | 19 | import org.junit.Test; 20 | 21 | import com.tinkerpop.blueprints.Graph; 22 | import com.tinkerpop.blueprints.GraphFactory; 23 | import com.tinkerpop.blueprints.Vertex; 24 | 25 | /** 26 | * Tests related to implementation-specific elements. 27 | */ 28 | public class ExtendedElementTest { 29 | 30 | private Graph makeGraph(AccumuloGraphConfiguration cfg) { 31 | return GraphFactory.open(cfg.getConfiguration()); 32 | } 33 | 34 | @Test 35 | public void testExistenceChecks() throws Exception { 36 | AccumuloGraphConfiguration cfg = 37 | AccumuloGraphTestUtils.generateGraphConfig("yesExistenceChecks"); 38 | Graph graph = makeGraph(cfg); 39 | 40 | String id; 41 | 42 | 43 | id = "doubleAdd"; 44 | assertNotNull(graph.addVertex(id)); 45 | try { 46 | graph.addVertex(id); 47 | fail(); 48 | } catch (Exception e) { } 49 | 50 | Vertex vAdd = graph.getVertex(id); 51 | assertNotNull(vAdd); 52 | graph.removeVertex(vAdd); 53 | assertNull(graph.getVertex(id)); 54 | 55 | 56 | id = "doubleRemove"; 57 | Vertex vRemove = graph.addVertex(id); 58 | assertNotNull(vRemove); 59 | graph.removeVertex(vRemove); 60 | try { 61 | graph.removeVertex(vRemove); 62 | fail(); 63 | } catch (Exception e) { } 64 | assertNull(graph.getVertex(id)); 65 | 66 | 67 | id = "notExist"; 68 | assertNull(graph.getVertex(id)); 69 | 70 | 71 | graph.shutdown(); 72 | } 73 | 74 | @Test 75 | public void testSkipExistenceChecks() throws Exception { 76 | AccumuloGraphConfiguration cfg = 77 | AccumuloGraphTestUtils.generateGraphConfig("skipExistenceChecks"); 78 | cfg.setSkipExistenceChecks(true); 79 | Graph graph = makeGraph(cfg); 80 | 81 | String id; 82 | 83 | id = "doubleAdd"; 84 | assertNotNull(graph.addVertex(id)); 85 | assertNotNull(graph.addVertex(id)); 86 | Vertex vAdd = graph.getVertex(id); 87 | assertNotNull(vAdd); 88 | graph.removeVertex(vAdd); 89 | assertNotNull(graph.getVertex(id)); 90 | 91 | 92 | id = "doubleRemove"; 93 | Vertex vRemove = graph.addVertex(id); 94 | assertNotNull(vRemove); 95 | graph.removeVertex(vRemove); 96 | assertNotNull(graph.getVertex(id)); 97 | // MDL 24 Dec 2014: removeVertex still does checks. 98 | //graph.removeVertex(vRemove); 99 | //assertNotNull(graph.getVertex(id)); 100 | 101 | 102 | id = "notExist"; 103 | assertNotNull(graph.getVertex(id)); 104 | 105 | graph.shutdown(); 106 | } 107 | } 108 | -------------------------------------------------------------------------------- /src/test/java/edu/jhuapl/tinkerpop/MockInstanceTest.java: -------------------------------------------------------------------------------- 1 | package edu.jhuapl.tinkerpop; 2 | 3 | import org.apache.commons.configuration.Configuration; 4 | 5 | import com.tinkerpop.blueprints.Graph; 6 | import com.tinkerpop.blueprints.GraphFactory; 7 | 8 | import edu.jhuapl.tinkerpop.AccumuloGraphConfiguration.InstanceType; 9 | 10 | /** 11 | * Run the test suite for a Mock instance type. 12 | */ 13 | public class MockInstanceTest extends AccumuloGraphTest { 14 | 15 | @Override 16 | public Graph generateGraph(String graphDirectoryName) { 17 | Configuration cfg = new AccumuloGraphConfiguration() 18 | .setInstanceType(InstanceType.Mock) 19 | .setGraphName(graphDirectoryName); 20 | testGraphName.set(graphDirectoryName); 21 | return GraphFactory.open(cfg); 22 | } 23 | } 24 | -------------------------------------------------------------------------------- /src/test/java/edu/jhuapl/tinkerpop/PropertyCacheTest.java: -------------------------------------------------------------------------------- 1 | /* Copyright 2014 The Johns Hopkins University Applied Physics Laboratory 2 | * 3 | * Licensed under the Apache License, Version 2.0 (the "License"); 4 | * you may not use this file except in compliance with the License. 5 | * You may obtain a copy of the License at 6 | * 7 | * http://www.apache.org/licenses/LICENSE-2.0 8 | * 9 | * Unless required by applicable law or agreed to in writing, software 10 | * distributed under the License is distributed on an "AS IS" BASIS, 11 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | * See the License for the specific language governing permissions and 13 | * limitations under the License. 14 | */ 15 | package edu.jhuapl.tinkerpop; 16 | 17 | import static org.junit.Assert.*; 18 | 19 | import org.junit.Test; 20 | 21 | import edu.jhuapl.tinkerpop.cache.PropertyCache; 22 | 23 | /** 24 | * Test the {@link PropertyCache} object. 25 | */ 26 | public class PropertyCacheTest { 27 | 28 | @Test 29 | public void testUncachedProperty() throws Exception { 30 | AccumuloGraphConfiguration cfg = 31 | AccumuloGraphTestUtils.generateGraphConfig("uncached"); 32 | PropertyCache cache = new PropertyCache(cfg); 33 | cache.put("K1", "V1"); 34 | cache.put("K2", "V2"); 35 | cache.put("K3", "V3"); 36 | assertNull(cache.get("K1")); 37 | assertNull(cache.get("K2")); 38 | assertNull(cache.get("K3")); 39 | } 40 | 41 | @Test 42 | public void testDefaultCachedProperty() throws Exception { 43 | AccumuloGraphConfiguration cfg = 44 | AccumuloGraphTestUtils.generateGraphConfig("defaultCached"); 45 | cfg.setPropertyCacheTimeout(null, 1000); 46 | 47 | PropertyCache cache = new PropertyCache(cfg); 48 | cache.put("K1", "V1"); 49 | assertEquals("V1", cache.get("K1")); 50 | Thread.sleep(1500); 51 | assertNull(cache.get("K1")); 52 | 53 | cache.put("K2", "V2"); 54 | assertEquals("V2", cache.get("K2")); 55 | cache.remove("K2"); 56 | assertNull(cache.get("K2")); 57 | 58 | cache.put("K3", "V3"); 59 | cache.clear(); 60 | assertNull(cache.get("K3")); 61 | } 62 | 63 | @Test 64 | public void testSpecificCachedProperty() throws Exception { 65 | AccumuloGraphConfiguration cfg = 66 | AccumuloGraphTestUtils.generateGraphConfig("specificCached"); 67 | cfg.setPropertyCacheTimeout(null, 1000); 68 | cfg.setPropertyCacheTimeout("long", 2000); 69 | cfg.setPropertyCacheTimeout("longer", 3000); 70 | 71 | PropertyCache cache = new PropertyCache(cfg); 72 | cache.put("default", "V1"); 73 | cache.put("long", "V2"); 74 | cache.put("longer", "V3"); 75 | 76 | assertEquals("V1", cache.get("default")); 77 | assertEquals("V2", cache.get("long")); 78 | assertEquals("V3", cache.get("longer")); 79 | 80 | Thread.sleep(1500); 81 | assertNull(cache.get("default")); 82 | Thread.sleep(1000); 83 | assertNull(cache.get("long")); 84 | Thread.sleep(1000); 85 | assertNull(cache.get("longer")); 86 | } 87 | } 88 | -------------------------------------------------------------------------------- /src/test/java/edu/jhuapl/tinkerpop/mapreduce/ElementOutputFormatTest.java: -------------------------------------------------------------------------------- 1 | package edu.jhuapl.tinkerpop.mapreduce; 2 | 3 | import static org.junit.Assert.assertEquals; 4 | import static org.junit.Assert.assertNull; 5 | 6 | import java.io.IOException; 7 | 8 | import org.apache.accumulo.core.util.CachedConfiguration; 9 | import org.apache.hadoop.conf.Configuration; 10 | import org.apache.hadoop.conf.Configured; 11 | import org.apache.hadoop.io.NullWritable; 12 | import org.apache.hadoop.io.Text; 13 | import org.apache.hadoop.mapreduce.Job; 14 | import org.apache.hadoop.mapreduce.Mapper; 15 | import org.apache.hadoop.util.Tool; 16 | import org.apache.hadoop.util.ToolRunner; 17 | import org.junit.Test; 18 | 19 | import com.tinkerpop.blueprints.Element; 20 | import com.tinkerpop.blueprints.Graph; 21 | import com.tinkerpop.blueprints.GraphFactory; 22 | import com.tinkerpop.blueprints.Vertex; 23 | 24 | import edu.jhuapl.tinkerpop.AccumuloGraphConfiguration; 25 | import edu.jhuapl.tinkerpop.AccumuloGraphConfiguration.InstanceType; 26 | 27 | public class ElementOutputFormatTest { 28 | private static AssertionError e1 = null; 29 | private static AssertionError e2 = null; 30 | 31 | private static class MRTester extends Configured implements Tool { 32 | 33 | private static class TestVertexMapper extends Mapper { 34 | int count = 0; 35 | 36 | @Override 37 | protected void map(Text k, Vertex v, Context context) throws IOException, InterruptedException { 38 | try { 39 | assertEquals(k.toString(), v.getId().toString()); 40 | 41 | v.setProperty("NAME", "BANANA" + v.getId()); 42 | context.write(NullWritable.get(), v); 43 | } catch (AssertionError e) { 44 | e1 = e; 45 | } 46 | count++; 47 | } 48 | 49 | @Override 50 | protected void cleanup(Context context) throws IOException, InterruptedException { 51 | try { 52 | assertEquals(100, count); 53 | } catch (AssertionError e) { 54 | e2 = e; 55 | } 56 | } 57 | } 58 | 59 | @Override 60 | public int run(String[] args) throws Exception { 61 | 62 | setConf(new Configuration()); 63 | 64 | getConf().set("fs.default.name", "local"); 65 | 66 | Job job = Job.getInstance(getConf(), this.getClass().getSimpleName() + "_" + System.currentTimeMillis()); 67 | job.setJarByClass(this.getClass()); 68 | AccumuloGraphConfiguration cfg = new AccumuloGraphConfiguration().setInstanceName("_mapreduce_instance2").setUser("root").setPassword("".getBytes()) 69 | .setGraphName("_mapreduce_table_2").setInstanceType(InstanceType.Mock).setCreate(true); 70 | job.setInputFormatClass(VertexInputFormat.class); 71 | 72 | VertexInputFormat.setAccumuloGraphConfiguration(job, cfg); 73 | 74 | ElementOutputFormat.setAccumuloGraphConfiguration(job, cfg); 75 | 76 | job.setMapperClass(TestVertexMapper.class); 77 | 78 | job.setMapOutputKeyClass(NullWritable.class); 79 | job.setMapOutputValueClass(Element.class); 80 | job.setOutputFormatClass(ElementOutputFormat.class); 81 | 82 | job.setNumReduceTasks(0); 83 | 84 | job.waitForCompletion(true); 85 | 86 | return job.isSuccessful() ? 0 : 1; 87 | } 88 | 89 | public static int main(String[] args) throws Exception { 90 | return ToolRunner.run(CachedConfiguration.getInstance(), new MRTester(), args); 91 | } 92 | } 93 | 94 | @Test 95 | public void testVertexOutputFormatMap() throws Exception { 96 | final String INSTANCE_NAME = "_mapreduce_instance2"; 97 | final String TEST_TABLE_1 = "_mapreduce_table_2"; 98 | 99 | if (!System.getProperty("os.name").startsWith("Windows")) { 100 | Graph g = GraphFactory.open(new AccumuloGraphConfiguration().setInstanceName(INSTANCE_NAME).setUser("root").setPassword("".getBytes()) 101 | .setGraphName(TEST_TABLE_1).setInstanceType(InstanceType.Mock).setCreate(true).getConfiguration()); 102 | for (int i = 0; i < 100; i++) { 103 | g.addVertex(i + ""); 104 | } 105 | assertEquals(0, MRTester.main(new String[] {})); 106 | assertNull(e1); 107 | assertNull(e2); 108 | assertEquals(g.getVertex("1").getProperty("NAME"), "BANANA1"); 109 | } 110 | } 111 | } 112 | -------------------------------------------------------------------------------- /src/test/java/edu/jhuapl/tinkerpop/mapreduce/InputFormatsTest.java: -------------------------------------------------------------------------------- 1 | package edu.jhuapl.tinkerpop.mapreduce; 2 | 3 | import static org.junit.Assert.assertEquals; 4 | import static org.junit.Assert.assertNull; 5 | 6 | import java.io.IOException; 7 | 8 | import org.apache.accumulo.core.util.CachedConfiguration; 9 | import org.apache.hadoop.conf.Configuration; 10 | import org.apache.hadoop.conf.Configured; 11 | import org.apache.hadoop.io.NullWritable; 12 | import org.apache.hadoop.io.Text; 13 | import org.apache.hadoop.mapreduce.Job; 14 | import org.apache.hadoop.mapreduce.Mapper; 15 | import org.apache.hadoop.mapreduce.lib.output.NullOutputFormat; 16 | import org.apache.hadoop.util.Tool; 17 | import org.apache.hadoop.util.ToolRunner; 18 | import org.junit.Test; 19 | 20 | import com.tinkerpop.blueprints.Direction; 21 | import com.tinkerpop.blueprints.Edge; 22 | import com.tinkerpop.blueprints.Graph; 23 | import com.tinkerpop.blueprints.GraphFactory; 24 | import com.tinkerpop.blueprints.Vertex; 25 | 26 | import edu.jhuapl.tinkerpop.AccumuloGraphConfiguration; 27 | import edu.jhuapl.tinkerpop.AccumuloGraphConfiguration.InstanceType; 28 | 29 | public class InputFormatsTest { 30 | 31 | private static AssertionError e1 = null; 32 | private static AssertionError e2 = null; 33 | 34 | private static class MRTester extends Configured implements Tool { 35 | 36 | private static class TestEdgeMapper extends Mapper { 37 | int count = 0; 38 | 39 | @Override 40 | protected void map(Text k, Edge v, Context context) throws IOException, InterruptedException { 41 | try { 42 | assertEquals(k.toString(), v.getId().toString()); 43 | MapReduceEdge e = (MapReduceEdge) v; 44 | assertEquals(e.getVertexId(Direction.OUT) + "a", e.getVertexId(Direction.IN)); 45 | } catch (AssertionError e) { 46 | e1 = e; 47 | } 48 | count++; 49 | } 50 | 51 | @Override 52 | protected void cleanup(Context context) throws IOException, InterruptedException { 53 | try { 54 | assertEquals(100, count); 55 | } catch (AssertionError e) { 56 | e2 = e; 57 | } 58 | } 59 | } 60 | 61 | private static class TestVertexMapper extends Mapper { 62 | // Key key = null; 63 | int count = 0; 64 | 65 | @Override 66 | protected void map(Text k, Vertex v, Context context) throws IOException, InterruptedException { 67 | try { 68 | assertEquals(k.toString(), v.getId().toString()); 69 | } catch (AssertionError e) { 70 | e1 = e; 71 | } 72 | count++; 73 | } 74 | 75 | @Override 76 | protected void cleanup(Context context) throws IOException, InterruptedException { 77 | try { 78 | assertEquals(100, count); 79 | } catch (AssertionError e) { 80 | e2 = e; 81 | } 82 | } 83 | } 84 | 85 | @Override 86 | public int run(String[] args) throws Exception { 87 | 88 | if (args.length != 5) { 89 | throw new IllegalArgumentException("Usage : " + MRTester.class.getName() + " "); 90 | } 91 | 92 | String user = args[0]; 93 | String pass = args[1]; 94 | String table = args[2]; 95 | 96 | String instanceName = args[3]; 97 | 98 | setConf(new Configuration()); 99 | // getConf().set("mapred.job.tracker", "local"); 100 | getConf().set("fs.default.name", "local"); 101 | 102 | Job job = Job.getInstance(getConf(), this.getClass().getSimpleName() + "_" + System.currentTimeMillis()); 103 | job.setJarByClass(this.getClass()); 104 | AccumuloGraphConfiguration cfg = new AccumuloGraphConfiguration().setInstanceName(instanceName).setUser(user).setPassword(pass.getBytes()) 105 | .setGraphName(table).setInstanceType(InstanceType.Mock).setCreate(true); 106 | if (Boolean.parseBoolean(args[4])) { 107 | 108 | job.setInputFormatClass(EdgeInputFormat.class); 109 | 110 | EdgeInputFormat.setAccumuloGraphConfiguration(job, cfg); 111 | 112 | job.setMapperClass(TestEdgeMapper.class); 113 | } else { 114 | job.setInputFormatClass(VertexInputFormat.class); 115 | 116 | VertexInputFormat.setAccumuloGraphConfiguration(job, cfg); 117 | job.setMapperClass(TestVertexMapper.class); 118 | } 119 | 120 | job.setMapOutputKeyClass(NullWritable.class); 121 | job.setMapOutputValueClass(NullWritable.class); 122 | job.setOutputFormatClass(NullOutputFormat.class); 123 | 124 | job.setNumReduceTasks(0); 125 | 126 | job.waitForCompletion(true); 127 | 128 | return job.isSuccessful() ? 0 : 1; 129 | } 130 | 131 | public static int main(String[] args) throws Exception { 132 | return ToolRunner.run(CachedConfiguration.getInstance(), new MRTester(), args); 133 | } 134 | } 135 | 136 | @Test 137 | public void testVertexInputMap() throws Exception { 138 | final String INSTANCE_NAME = "_mapreduce_instance"; 139 | final String TEST_TABLE_NAME = "_mapreduce_table_vertexInputMap"; 140 | 141 | if (!System.getProperty("os.name").startsWith("Windows")) { 142 | Graph g = GraphFactory.open(new AccumuloGraphConfiguration().setInstanceName(INSTANCE_NAME) 143 | .setUser("root").setPassword("".getBytes()) 144 | .setGraphName(TEST_TABLE_NAME).setInstanceType(InstanceType.Mock) 145 | .setCreate(true).getConfiguration()); 146 | 147 | for (int i = 0; i < 100; i++) { 148 | g.addVertex(i + ""); 149 | } 150 | 151 | assertEquals(0, MRTester.main(new String[]{"root", "", 152 | TEST_TABLE_NAME, INSTANCE_NAME, "false"})); 153 | assertNull(e1); 154 | assertNull(e2); 155 | 156 | g.shutdown(); 157 | } 158 | } 159 | 160 | @Test 161 | public void testEdgeInputMap() throws Exception { 162 | final String INSTANCE_NAME = "_mapreduce_instance"; 163 | final String TEST_TABLE_NAME = "_mapreduce_table_edgeInputMap"; 164 | 165 | if (!System.getProperty("os.name").startsWith("Windows")) { 166 | Graph g = GraphFactory.open(new AccumuloGraphConfiguration().setInstanceName(INSTANCE_NAME) 167 | .setUser("root").setPassword("".getBytes()) 168 | .setGraphName(TEST_TABLE_NAME).setInstanceType(InstanceType.Mock) 169 | .setAutoFlush(true).setCreate(true).getConfiguration()); 170 | 171 | for (int i = 0; i < 100; i++) { 172 | Vertex v1 = g.addVertex(i+""); 173 | Vertex v2 = g.addVertex(i+"a"); 174 | g.addEdge(null, v1, v2, "knows"); 175 | } 176 | 177 | assertEquals(0, MRTester.main(new String[]{"root", "", 178 | TEST_TABLE_NAME, INSTANCE_NAME, "true"})); 179 | assertNull(e1); 180 | assertNull(e2); 181 | 182 | g.shutdown(); 183 | } 184 | } 185 | 186 | } 187 | -------------------------------------------------------------------------------- /table-structure.md: -------------------------------------------------------------------------------- 1 | AccumuloGraph Table Schema 2 | ========================== 3 | 4 | AccumuloGraph uses a number of backing tables for data storage. 5 | This file documents the structure and organization of these 6 | tables. The tables and records are structured in such a way as 7 | to implement the Blueprints operations in terms of efficient Accumulo 8 | operations, i.e., prefix searches, contiguous scans, and 9 | batch operations. 10 | For our purposes, Accumulo entries consist of four fields: 11 | row (R), column family (CF), column qualifier (CQ), and value (V). 12 | For more information on Accumulo's schema and how to 13 | access records efficiently, see the 14 | [Accumulo documentation](https://accumulo.apache.org/1.5/accumulo_user_manual.html). 15 | 16 | The tables used by AccumuloGraph are prefixed by the configured graph 17 | name and can be classed as element/property tables, and 18 | index tables. Their structure is discussed below. 19 | 20 | Elements and properties 21 | ----------------------- 22 | 23 | Vertex and edge information, along with their properties, are stored 24 | in the *graphname*\_vertex and *graphname*\_edge tables respectively. 25 | 26 | First, an entry declares the existence of a vertex in the vertex table. 27 | 28 | | R | CF | CQ | V | 29 | |---|----|----|---| 30 | | *vertex_id* | `_LABEL_` | `_EXISTS_`| *[empty]* | 31 | 32 | A similar entry declares the existence of an edge in the edge table. 33 | 34 | | R | CF | CQ | V | 35 | |---|----|----|---| 36 | | *edge_id* | `_LABEL_` | *in_vertex_id*`_DELIM_`*out_vertex_id* | *edge_label* | 37 | 38 | When adding an edge, additional entries are stored in the 39 | vertex table for each endpoint of the edge. These facilitate the 40 | `Vertex.getEdge` and `Vertex.getVertex` operations. 41 | 42 | | R | CF | CQ | V | 43 | |---|----|----|---| 44 | | *in_vertex_id* | `_IN_EDGE_` | *out_vertex_id*`_DELIM_`*edge_id* | *edge_label* | 45 | | *out_vertex_id* | `_OUT_EDGE_` | *in_vertex_id*`_DELIM_`*edge_id* | *edge_label* | 46 | 47 | Finally, vertex and edge properties are stored in their respective 48 | tables. Entry formats are the same for both vertices and edges. 49 | Note that property values are serialized such that their type 50 | can be deduced when deserializing. 51 | 52 | | R | CF | CQ | V | 53 | |---|----|----|---| 54 | | *element_id* | *property_key* | *[empty]* | *property_value* | 55 | 56 | Indexes 57 | ------- 58 | 59 | Several tables store index-related information, 60 | including index value tables that store index 61 | property keys and values, and index metadata tables 62 | that store information about what indexes exist 63 | and what properties are indexed. 64 | 65 | For `KeyIndexableGraph`, index value tables 66 | include *graphname*\_vertex\_key\_index 67 | and *graphname*\_edge\_key\_index for vertex 68 | and edge properties, respectively. 69 | For `IndexableGraph`, index value tables are 70 | named *graphname*\_index\_*indexname*, 71 | where *indexname* is the index name. 72 | The entry formats in all these tables are the same: 73 | 74 | | R | CF | CQ | V | 75 | |---|----|----|---| 76 | | *property_value* | *property_key* | *element_id* | *[empty]* | 77 | 78 | Property values are serialized in the same way as above. 79 | 80 | In addition to the index value tables, an index metadata table, 81 | *graphname*\_index\_metadata, stores index information. For 82 | `KeyIndexableGraph`, records in this table enumerate the property keys 83 | that are indexed. 84 | 85 | | R | CF | CQ | V | 86 | |---|----|----|---| 87 | | *property_key* | `_INDEX_KEY_` | *element_class* | *[empty]* | 88 | 89 | For `IndexableGraph`, records enumerate the existing indexes. 90 | 91 | | R | CF | CQ | V | 92 | |---|----|----|---| 93 | | *index_name* | `_INDEX_NAME_` | *element_class* | *[empty]* | 94 | --------------------------------------------------------------------------------