├── .gitignore ├── AUTHORS ├── LICENSE ├── README.md ├── dat ├── MaxMind │ ├── world-cities-us.1.xml.gz │ ├── world-cities-us.2.xml.gz │ ├── world-cities-us.3.xml.gz │ └── world-cities-us.txt.gz ├── nasdaq-company-list.xml ├── nasdaq-google.xml ├── nasdaq-intel.xml └── us-congress-representative-list.xml ├── pom.xml ├── scripts ├── README.md ├── post-element-list.sh └── post-element.sh └── src └── main ├── java └── cleo │ └── primer │ ├── ElementDAO.java │ ├── GenericTypeaheadInstance.java │ ├── RestDAO.java │ ├── TypeaheadInstance.java │ ├── rest │ ├── model │ │ ├── ElementDTO.java │ │ ├── ElementListDTO.java │ │ └── ExceptionDTO.java │ └── resources │ │ └── ElementsResource.java │ └── util │ └── MaxMindWorldCitiesXML.java ├── resources ├── applicationContext.xml ├── config │ └── generic-typeahead │ │ ├── i001.config │ │ └── i002.config └── log4j.properties └── webapp ├── WEB-INF └── web.xml └── index.html /.gitignore: -------------------------------------------------------------------------------- 1 | .classpath 2 | .settings 3 | .project 4 | eclipse-build 5 | target 6 | docs 7 | logs 8 | -------------------------------------------------------------------------------- /AUTHORS: -------------------------------------------------------------------------------- 1 | Authors 2 | ======= 3 | 4 | * Jingwei Wu 5 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Apache License 2 | Version 2.0, January 2004 3 | http://www.apache.org/licenses/ 4 | 5 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 6 | 7 | 1. Definitions. 8 | 9 | "License" shall mean the terms and conditions for use, reproduction, 10 | and distribution as defined by Sections 1 through 9 of this document. 11 | 12 | "Licensor" shall mean the copyright owner or entity authorized by 13 | the copyright owner that is granting the License. 14 | 15 | "Legal Entity" shall mean the union of the acting entity and all 16 | other entities that control, are controlled by, or are under common 17 | control with that entity. For the purposes of this definition, 18 | "control" means (i) the power, direct or indirect, to cause the 19 | direction or management of such entity, whether by contract or 20 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 21 | outstanding shares, or (iii) beneficial ownership of such entity. 22 | 23 | "You" (or "Your") shall mean an individual or Legal Entity 24 | exercising permissions granted by this License. 25 | 26 | "Source" form shall mean the preferred form for making modifications, 27 | including but not limited to software source code, documentation 28 | source, and configuration files. 29 | 30 | "Object" form shall mean any form resulting from mechanical 31 | transformation or translation of a Source form, including but 32 | not limited to compiled object code, generated documentation, 33 | and conversions to other media types. 34 | 35 | "Work" shall mean the work of authorship, whether in Source or 36 | Object form, made available under the License, as indicated by a 37 | copyright notice that is included in or attached to the work 38 | (an example is provided in the Appendix below). 39 | 40 | "Derivative Works" shall mean any work, whether in Source or Object 41 | form, that is based on (or derived from) the Work and for which the 42 | editorial revisions, annotations, elaborations, or other modifications 43 | represent, as a whole, an original work of authorship. For the purposes 44 | of this License, Derivative Works shall not include works that remain 45 | separable from, or merely link (or bind by name) to the interfaces of, 46 | the Work and Derivative Works thereof. 47 | 48 | "Contribution" shall mean any work of authorship, including 49 | the original version of the Work and any modifications or additions 50 | to that Work or Derivative Works thereof, that is intentionally 51 | submitted to Licensor for inclusion in the Work by the copyright owner 52 | or by an individual or Legal Entity authorized to submit on behalf of 53 | the copyright owner. For the purposes of this definition, "submitted" 54 | means any form of electronic, verbal, or written communication sent 55 | to the Licensor or its representatives, including but not limited to 56 | communication on electronic mailing lists, source code control systems, 57 | and issue tracking systems that are managed by, or on behalf of, the 58 | Licensor for the purpose of discussing and improving the Work, but 59 | excluding communication that is conspicuously marked or otherwise 60 | designated in writing by the copyright owner as "Not a Contribution." 61 | 62 | "Contributor" shall mean Licensor and any individual or Legal Entity 63 | on behalf of whom a Contribution has been received by Licensor and 64 | subsequently incorporated within the Work. 65 | 66 | 2. Grant of Copyright License. Subject to the terms and conditions of 67 | this License, each Contributor hereby grants to You a perpetual, 68 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 69 | copyright license to reproduce, prepare Derivative Works of, 70 | publicly display, publicly perform, sublicense, and distribute the 71 | Work and such Derivative Works in Source or Object form. 72 | 73 | 3. Grant of Patent License. Subject to the terms and conditions of 74 | this License, each Contributor hereby grants to You a perpetual, 75 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 76 | (except as stated in this section) patent license to make, have made, 77 | use, offer to sell, sell, import, and otherwise transfer the Work, 78 | where such license applies only to those patent claims licensable 79 | by such Contributor that are necessarily infringed by their 80 | Contribution(s) alone or by combination of their Contribution(s) 81 | with the Work to which such Contribution(s) was submitted. If You 82 | institute patent litigation against any entity (including a 83 | cross-claim or counterclaim in a lawsuit) alleging that the Work 84 | or a Contribution incorporated within the Work constitutes direct 85 | or contributory patent infringement, then any patent licenses 86 | granted to You under this License for that Work shall terminate 87 | as of the date such litigation is filed. 88 | 89 | 4. Redistribution. You may reproduce and distribute copies of the 90 | Work or Derivative Works thereof in any medium, with or without 91 | modifications, and in Source or Object form, provided that You 92 | meet the following conditions: 93 | 94 | (a) You must give any other recipients of the Work or 95 | Derivative Works a copy of this License; and 96 | 97 | (b) You must cause any modified files to carry prominent notices 98 | stating that You changed the files; and 99 | 100 | (c) You must retain, in the Source form of any Derivative Works 101 | that You distribute, all copyright, patent, trademark, and 102 | attribution notices from the Source form of the Work, 103 | excluding those notices that do not pertain to any part of 104 | the Derivative Works; and 105 | 106 | (d) If the Work includes a "NOTICE" text file as part of its 107 | distribution, then any Derivative Works that You distribute must 108 | include a readable copy of the attribution notices contained 109 | within such NOTICE file, excluding those notices that do not 110 | pertain to any part of the Derivative Works, in at least one 111 | of the following places: within a NOTICE text file distributed 112 | as part of the Derivative Works; within the Source form or 113 | documentation, if provided along with the Derivative Works; or, 114 | within a display generated by the Derivative Works, if and 115 | wherever such third-party notices normally appear. The contents 116 | of the NOTICE file are for informational purposes only and 117 | do not modify the License. You may add Your own attribution 118 | notices within Derivative Works that You distribute, alongside 119 | or as an addendum to the NOTICE text from the Work, provided 120 | that such additional attribution notices cannot be construed 121 | as modifying the License. 122 | 123 | You may add Your own copyright statement to Your modifications and 124 | may provide additional or different license terms and conditions 125 | for use, reproduction, or distribution of Your modifications, or 126 | for any such Derivative Works as a whole, provided Your use, 127 | reproduction, and distribution of the Work otherwise complies with 128 | the conditions stated in this License. 129 | 130 | 5. Submission of Contributions. Unless You explicitly state otherwise, 131 | any Contribution intentionally submitted for inclusion in the Work 132 | by You to the Licensor shall be under the terms and conditions of 133 | this License, without any additional terms or conditions. 134 | Notwithstanding the above, nothing herein shall supersede or modify 135 | the terms of any separate license agreement you may have executed 136 | with Licensor regarding such Contributions. 137 | 138 | 6. Trademarks. This License does not grant permission to use the trade 139 | names, trademarks, service marks, or product names of the Licensor, 140 | except as required for reasonable and customary use in describing the 141 | origin of the Work and reproducing the content of the NOTICE file. 142 | 143 | 7. Disclaimer of Warranty. Unless required by applicable law or 144 | agreed to in writing, Licensor provides the Work (and each 145 | Contributor provides its Contributions) on an "AS IS" BASIS, 146 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 147 | implied, including, without limitation, any warranties or conditions 148 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 149 | PARTICULAR PURPOSE. You are solely responsible for determining the 150 | appropriateness of using or redistributing the Work and assume any 151 | risks associated with Your exercise of permissions under this License. 152 | 153 | 8. Limitation of Liability. In no event and under no legal theory, 154 | whether in tort (including negligence), contract, or otherwise, 155 | unless required by applicable law (such as deliberate and grossly 156 | negligent acts) or agreed to in writing, shall any Contributor be 157 | liable to You for damages, including any direct, indirect, special, 158 | incidental, or consequential damages of any character arising as a 159 | result of this License or out of the use or inability to use the 160 | Work (including but not limited to damages for loss of goodwill, 161 | work stoppage, computer failure or malfunction, or any and all 162 | other commercial damages or losses), even if such Contributor 163 | has been advised of the possibility of such damages. 164 | 165 | 9. Accepting Warranty or Additional Liability. While redistributing 166 | the Work or Derivative Works thereof, You may choose to offer, 167 | and charge a fee for, acceptance of support, warranty, indemnity, 168 | or other liability obligations and/or rights consistent with this 169 | License. However, in accepting such obligations, You may act only 170 | on Your own behalf and on Your sole responsibility, not on behalf 171 | of any other Contributor, and only if You agree to indemnify, 172 | defend, and hold each Contributor harmless for any liability 173 | incurred by, or claims asserted against, such Contributor by reason 174 | of your accepting any such warranty or additional liability. 175 | 176 | END OF TERMS AND CONDITIONS 177 | 178 | APPENDIX: How to apply the Apache License to your work. 179 | 180 | To apply the Apache License to your work, attach the following 181 | boilerplate notice, with the fields enclosed by brackets "[]" 182 | replaced with your own identifying information. (Don't include 183 | the brackets!) The text should be enclosed in the appropriate 184 | comment syntax for the file format. We also recommend that a 185 | file or class name and description of purpose be included on the 186 | same "printed page" as the copyright notice for easier 187 | identification within third-party archives. 188 | 189 | Copyright [yyyy] [name of copyright owner] 190 | 191 | Licensed under the Apache License, Version 2.0 (the "License"); 192 | you may not use this file except in compliance with the License. 193 | You may obtain a copy of the License at 194 | 195 | http://www.apache.org/licenses/LICENSE-2.0 196 | 197 | Unless required by applicable law or agreed to in writing, software 198 | distributed under the License is distributed on an "AS IS" BASIS, 199 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 200 | See the License for the specific language governing permissions and 201 | limitations under the License. 202 | 203 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # cleo-primer 2 | 3 | The cleo-primer package provides a basic RESTful implementation of partial, out-of-order and real-time typeahead services. 4 | It is built on [Cleo](http://sna-projects.com/cleo/), an open source package from LinkedIn. 5 | 6 | ### Homepage 7 | 8 | Find out more about Cleo at [http://sna-projects.com/cleo](http://sna-projects.com/cleo). 9 | 10 | ### License 11 | 12 | Apache Public License (APL) 2.0 13 | 14 | ### Artifacts 15 | 16 | cleo-primer.war 17 | 18 | ### Maven 19 | 20 | groupId: com.sna-projects.cleo 21 | 22 | artifactId: cleo-primer 23 | 24 | version: 1.0 25 | 26 | ### Build the war 27 | 28 | mvn clean package 29 | 30 | ### Launch WebApp 31 | 32 | Launch the cleo-primer web application using the next command from the 33 | main folder: 34 | 35 | MAVEN_OPTS="-Xms1g -Xmx1g" mvn jetty:run -Dcleo.instance.name=Company -Dcleo.instance.type=cleo.primer.GenericTypeaheadInstance -Dcleo.instance.conf=src/main/resources/config/generic-typeahead 36 | 37 | You can customize your web application by choosing different values for parameters 38 | cleo.instance.name, cleo.instance.type and cleo.instance.conf. Depending on the size 39 | of your data sets, you may need to specify a different JVM heap size. 40 | 41 | ### Post a list of new elements 42 | 43 | ./scripts/post-element-list.sh dat/nasdaq-company-list.xml 44 | 45 | ### Post a new element 46 | 47 | ./scripts/post-element.sh dat/nasdaq-google.xml dat/nasdaq-intel.xml 48 | 49 | ### Search 50 | 51 | Visit the URL below to try out cleo-primer. 52 | 53 | http://localhost:8080/cleo-primer 54 | 55 | http://localhost:8080/cleo-primer/rest/elements/search?query=goo 56 | 57 | ### Eclipse 58 | 59 | Set up Eclipse by executing the command below: 60 | 61 | mvn eclipse:eclipse 62 | 63 | Inside Eclipse, select Preferences > Java > Build Path > Classpath Variables. Define a new classpath variable M2_REPO and assign maven repository. 64 | 65 | For more information, check out http://maven.apache.org/guides/mini/guide-ide-eclipse.html 66 | 67 | ### Contribute 68 | 69 | For help please see the [discussion group](http://groups.google.com/group/cleo-typeahead). Bugs and feature requests can be filed [here](https://github.com/linkedin/cleo/issues). 70 | -------------------------------------------------------------------------------- /dat/MaxMind/world-cities-us.1.xml.gz: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jingwei/cleo-primer/503b24a979f1d5484f873886931ba7a027bc9174/dat/MaxMind/world-cities-us.1.xml.gz -------------------------------------------------------------------------------- /dat/MaxMind/world-cities-us.2.xml.gz: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jingwei/cleo-primer/503b24a979f1d5484f873886931ba7a027bc9174/dat/MaxMind/world-cities-us.2.xml.gz -------------------------------------------------------------------------------- /dat/MaxMind/world-cities-us.3.xml.gz: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jingwei/cleo-primer/503b24a979f1d5484f873886931ba7a027bc9174/dat/MaxMind/world-cities-us.3.xml.gz -------------------------------------------------------------------------------- /dat/MaxMind/world-cities-us.txt.gz: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jingwei/cleo-primer/503b24a979f1d5484f873886931ba7a027bc9174/dat/MaxMind/world-cities-us.txt.gz -------------------------------------------------------------------------------- /dat/nasdaq-google.xml: -------------------------------------------------------------------------------- 1 | 2 | 3 | 1063 4 | Google Inc. (GOOG) 5 | Technology - Computer Software: Programming Data Processing 6 | http://www.nasdaq.com/symbol/goog 7 | 159350 8 | google 9 | inc. 10 | goog 11 | 12 | -------------------------------------------------------------------------------- /dat/nasdaq-intel.xml: -------------------------------------------------------------------------------- 1 | 2 | 3 | 1253 4 | Intel Corporation (INTC) 5 | Technology - Semiconductors 6 | http://www.nasdaq.com/symbol/intc 7 | 129668 8 | intel 9 | corporation 10 | intc 11 | 12 | -------------------------------------------------------------------------------- /pom.xml: -------------------------------------------------------------------------------- 1 | 2 | 5 | 4.0.0 6 | 7 | com.sna-projects.cleo 8 | cleo-primer 9 | war 10 | 1.0 11 | cleo-primer 12 | https://github.com/jingwei/cleo-primer 13 | 14 | 15 | 16 | 17 | org.apache.maven.plugins 18 | maven-compiler-plugin 19 | 2.3.2 20 | 21 | ${jdkLevel} 22 | ${jdkLevel} 23 | UTF-8 24 | 25 | 26 | 27 | 28 | org.mortbay.jetty 29 | maven-jetty-plugin 30 | ${jetty.version} 31 | 32 | manual 33 | 34 | 35 | 8080 36 | 37 | 38 | 39 | 40 | org.apache.commons.logging.Log 41 | org.apache.commons.logging.impl.Log4JLogger 42 | 43 | 44 | slf4j 45 | false 46 | 47 | 48 | log4j.configuration 49 | file:./src/main/resources/log4j.properties 50 | 51 | 52 | 53 | 54 | 55 | 56 | 57 | 58 | 59 | com.sun.jersey 60 | jersey-json 61 | ${jersey.version} 62 | 63 | 64 | com.sun.jersey 65 | jersey-client 66 | ${jersey.version} 67 | 68 | 69 | com.sun.jersey 70 | jersey-server 71 | ${jersey.version} 72 | 73 | 74 | com.sun.jersey.contribs 75 | jersey-spring 76 | ${jersey.version} 77 | 78 | 79 | com.sna-projects.krati 80 | krati 81 | ${krati.version} 82 | 83 | 84 | com.sna-projects.cleo 85 | cleo 86 | ${cleo.version} 87 | 88 | 89 | 90 | 91 | 92 | maven2-repository.dev.java.net 93 | Java.net Repository for Maven 94 | http://download.java.net/maven/2 95 | 96 | 97 | 98 | 99 | 1.6 100 | 1.12 101 | 6.1.25 102 | 0.4.9 103 | 1.2.6-1 104 | 105 | 106 | 107 | 108 | -------------------------------------------------------------------------------- /scripts/README.md: -------------------------------------------------------------------------------- 1 | # How To Rest 2 | 3 | ### GET - Search 4 | 5 | curl -v -X GET -H "Accept: application/json" http://localhost:8080/cleo-primer/rest/elements/search?query=goo 6 | 7 | curl -v -X GET -H "Accept: application/xml" http://localhost:8080/cleo-primer/rest/elements/search?query=goo 8 | 9 | ### GET - Retrieve an element 10 | 11 | curl -v -X GET -H "Accept: application/json" http://localhost:8080/cleo-primer/rest/elements/1063 12 | 13 | curl -v -X GET -H "Accept: application/xml" http://localhost:8080/cleo-primer/rest/elements/1063 14 | 15 | ### GET - Retrieve all elements 16 | 17 | curl -v -X GET -H "Accept: application/json" http://localhost:8080/cleo-primer/rest/elements 18 | 19 | curl -v -X GET -H "Accept: application/xml" http://localhost:8080/cleo-primer/rest/elements 20 | 21 | ### GET - Retrieve a range of elements 22 | 23 | curl -v -X GET -H "Accept: application/json" http://localhost:8080/cleo-primer/rest/elements/1..100 24 | 25 | curl -v -X GET -H "Accept: application/xml" http://localhost:8080/cleo-primer/rest/elements/1..100 26 | 27 | ### DELETE - Remove an element 28 | 29 | curl -v -X DELETE http://localhost:8080/cleo-primer/rest/elements/1 30 | 31 | ### PUT - Update an element 32 | 33 | curl -v -X PUT -H "Content-type: application/xml" -H "Accept: application/xml" http://localhost:8080/cleo-primer/rest/elements/1063 -d ' 34 | 35 | 1063 36 | Google Inc. (GOOG) 37 | Technology - Computer Software: Programming Data Processing 38 | http://www.nasdaq.com/symbol/goog 39 | 159350 40 | google 41 | inc. 42 | goog 43 | 44 | ' 45 | 46 | ### POST - Add a new element 47 | 48 | curl -v -X POST -H "Content-type: application/xml" -H "Accept: application/xml" http://localhost:8080/cleo-primer/rest/elements/_ -d ' 49 | 50 | 1063 51 | Google Inc. (GOOG) 52 | Technology - Computer Software: Programming Data Processing 53 | http://www.nasdaq.com/symbol/goog 54 | 159350 55 | google 56 | inc. 57 | goog 58 | 59 | ' 60 | 61 | ### POST - Add a list of new elements 62 | 63 | curl -v -X POST -H "Content-type: application/xml" -H "Accept: application/xml" http://localhost:8080/cleo-primer/rest/elements -d ' 64 | 65 | 66 | 1063 67 | Google Inc. (GOOG) 68 | Technology - Computer Software: Programming Data Processing 69 | http://www.nasdaq.com/symbol/goog 70 | 159350 71 | google 72 | inc. 73 | goog 74 | 75 | 76 | 1253 77 | Intel Corporation (INTC) 78 | Technology - Semiconductors 79 | http://www.nasdaq.com/symbol/intc 80 | 129668 81 | intel 82 | corporation 83 | intc 84 | 85 | 86 | ' 87 | 88 | ### POST - Flush elements 89 | 90 | curl -v -X POST http://localhost:8080/cleo-primer/rest/elements/flush 91 | -------------------------------------------------------------------------------- /scripts/post-element-list.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | 3 | if [ $# -eq 0 ]; then 4 | echo "Usage:" 5 | echo " $0 [xml file] ..." 6 | echo "Example:" 7 | echo " $0 dat/nasdaq-company-list.xml" 8 | exit 1; 9 | fi 10 | 11 | for f in $@; do 12 | curl -v -X POST -H "Content-type: application/xml" -H "Accept: application/xml" http://localhost:8080/cleo-primer/rest/elements/ --data-binary @$f 13 | done 14 | 15 | -------------------------------------------------------------------------------- /scripts/post-element.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | 3 | if [ $# -eq 0 ]; then 4 | echo "Usage:" 5 | echo " $0 [xml file] ..." 6 | echo "Example:" 7 | echo " $0 dat/nasdaq-google.xml" 8 | exit 1; 9 | fi 10 | 11 | for f in $@; do 12 | curl -v -X POST -H "Content-type: application/xml" -H "Accept: application/xml" http://localhost:8080/cleo-primer/rest/elements/_ --data-binary @$f 13 | done 14 | 15 | -------------------------------------------------------------------------------- /src/main/java/cleo/primer/ElementDAO.java: -------------------------------------------------------------------------------- 1 | /* 2 | * Copyright (c) 2011-2012 3 | * 4 | * Licensed under the Apache License, Version 2.0 (the "License"); you may not 5 | * use this file except in compliance with the License. You may obtain a copy of 6 | * the License at 7 | * 8 | * http://www.apache.org/licenses/LICENSE-2.0 9 | * 10 | * Unless required by applicable law or agreed to in writing, software 11 | * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT 12 | * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the 13 | * License for the specific language governing permissions and limitations under 14 | * the License. 15 | */ 16 | 17 | package cleo.primer; 18 | 19 | import java.io.File; 20 | import java.util.ArrayList; 21 | import java.util.List; 22 | 23 | import cleo.primer.rest.model.ElementDTO; 24 | import cleo.search.Indexer; 25 | import cleo.search.store.ArrayStoreElement; 26 | import cleo.search.typeahead.Typeahead; 27 | 28 | /** 29 | * ElementDAO - A singleton Data Access Object (DAO) for elements of type {@link ElementDTO}. 30 | * 31 | * @author jwu 32 | * @since 12/22, 2012 33 | */ 34 | public enum ElementDAO implements RestDAO { 35 | INSTANCE; 36 | 37 | private TypeaheadInstance loader; 38 | 39 | private ElementDAO () { 40 | try { 41 | String name = System.getProperty("cleo.instance.name"); 42 | String type = System.getProperty("cleo.instance.type"); 43 | String conf = System.getProperty("cleo.instance.conf"); 44 | File confPath = new File(conf); 45 | 46 | @SuppressWarnings("unchecked") 47 | Class> instanceClass = (Class>)Class.forName(type); 48 | loader = instanceClass.getConstructor(String.class, File.class).newInstance(name, confPath); 49 | } catch (Exception e) { 50 | e.printStackTrace(); 51 | } 52 | } 53 | 54 | public final Indexer getIndexer() { 55 | return loader.getIndexer(); 56 | } 57 | 58 | public final Typeahead getSearcher() { 59 | return loader.getSearcher(); 60 | } 61 | 62 | public final ArrayStoreElement getElementStore() { 63 | return loader.getElementStore(); 64 | } 65 | 66 | public List getElements() { 67 | ArrayStoreElement store = getElementStore(); 68 | List list = new ArrayList(store.capacity()); 69 | 70 | int start = store.getIndexStart(); 71 | int end = start + store.length(); 72 | for(int i = start; i < end; i++) { 73 | ElementDTO element = getElementStore().getElement(i); 74 | if(element != null && element.isSearchable()) { 75 | list.add(element); 76 | } 77 | } 78 | 79 | return list; 80 | } 81 | 82 | @Override // HTTP-GET 83 | public ElementDTO getElement(int elementId) { 84 | ElementDTO element = getElementStore().getElement(elementId); 85 | if(element != null && element.isSearchable()) { 86 | return element; 87 | } else { 88 | return null; 89 | } 90 | } 91 | 92 | @Override // HTTP-DELETE 93 | public ElementDTO deleteElement(int elementId) throws Exception { 94 | ElementDTO oldElement = getElementStore().getElement(elementId); 95 | 96 | if(oldElement != null) { 97 | ElementDTO newElement = (ElementDTO)oldElement.clone(); 98 | newElement.setTerms(new String[0]); 99 | getIndexer().index(newElement); 100 | } 101 | 102 | if(oldElement != null && oldElement.isSearchable()) { 103 | return oldElement; 104 | } else { 105 | return null; 106 | } 107 | } 108 | 109 | @Override // HTTP-PUT 110 | public ElementDTO updateElement(ElementDTO element) throws Exception { 111 | ElementDTO oldElement = getElementStore().getElement(element.getElementId()); 112 | getIndexer().index(element); 113 | 114 | if(oldElement != null && oldElement.isSearchable()) { 115 | return oldElement; 116 | } else { 117 | return null; 118 | } 119 | } 120 | 121 | @Override // HTTP-POST 122 | public boolean insertElement(ElementDTO element) throws Exception { 123 | int elementId = element.getElementId(); 124 | if(getElementStore().hasIndex(elementId)) { 125 | return getIndexer().index(element); 126 | } 127 | 128 | return false; 129 | } 130 | } 131 | -------------------------------------------------------------------------------- /src/main/java/cleo/primer/GenericTypeaheadInstance.java: -------------------------------------------------------------------------------- 1 | /* 2 | * Copyright (c) 2011-2012 3 | * 4 | * Licensed under the Apache License, Version 2.0 (the "License"); you may not 5 | * use this file except in compliance with the License. You may obtain a copy of 6 | * the License at 7 | * 8 | * http://www.apache.org/licenses/LICENSE-2.0 9 | * 10 | * Unless required by applicable law or agreed to in writing, software 11 | * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT 12 | * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the 13 | * License for the specific language governing permissions and limitations under 14 | * the License. 15 | */ 16 | 17 | package cleo.primer; 18 | 19 | import java.io.File; 20 | import java.io.FileFilter; 21 | import java.io.IOException; 22 | import java.util.ArrayList; 23 | import java.util.List; 24 | 25 | import cleo.search.Element; 26 | import cleo.search.Indexer; 27 | import cleo.search.MultiIndexer; 28 | import cleo.search.selector.ScoredElementSelectorFactory; 29 | import cleo.search.store.ArrayStoreElement; 30 | import cleo.search.store.MultiArrayStoreElement; 31 | import cleo.search.tool.GenericTypeaheadInitializer; 32 | import cleo.search.typeahead.GenericTypeahead; 33 | import cleo.search.typeahead.GenericTypeaheadConfig; 34 | import cleo.search.typeahead.MultiTypeahead; 35 | import cleo.search.typeahead.Typeahead; 36 | import cleo.search.typeahead.TypeaheadConfigFactory; 37 | 38 | /** 39 | * GenericTypeaheadInstance 40 | * 41 | * @author jwu 42 | * @since 12/22, 2011 43 | */ 44 | public class GenericTypeaheadInstance implements TypeaheadInstance { 45 | final Indexer indexer; 46 | final Typeahead searcher; 47 | final ArrayStoreElement elementStore; 48 | 49 | public GenericTypeaheadInstance(String name, File configPath) throws Exception { 50 | File[] configFiles = configPath.listFiles(new FileFilter() { 51 | public boolean accept(File path) { 52 | return path.getName().endsWith(".config"); 53 | } 54 | }); 55 | 56 | List> indexerList = new ArrayList>(); 57 | List> searcherList = new ArrayList>(); 58 | List> storeList = new ArrayList>(); 59 | 60 | for(File configFile : configFiles) { 61 | System.out.println(configFile.getPath()); 62 | GenericTypeahead gta = createTypeahead(configFile); 63 | indexerList.add(gta); 64 | searcherList.add(gta); 65 | storeList.add(gta.getElementStore()); 66 | } 67 | 68 | // Create indexer, searcher and elementStore 69 | indexer = new MultiIndexer(name, indexerList); 70 | searcher = new MultiTypeahead(name, searcherList); 71 | elementStore = new MultiArrayStoreElement(storeList); 72 | 73 | // Flush indexes upon shutdown 74 | addShutdownHook(); 75 | } 76 | 77 | protected void addShutdownHook() { 78 | Runtime.getRuntime().addShutdownHook(new Thread() { 79 | @Override 80 | public void run() { 81 | try { 82 | indexer.flush(); 83 | } catch (IOException e) { 84 | e.printStackTrace(); 85 | } 86 | } 87 | }); 88 | } 89 | 90 | protected GenericTypeahead createTypeahead(File configFile) throws Exception { 91 | // Create typeahead config 92 | GenericTypeaheadConfig config = 93 | TypeaheadConfigFactory.createGenericTypeaheadConfig(configFile); 94 | config.setSelectorFactory(new ScoredElementSelectorFactory()); 95 | 96 | // Create typeahead initializer 97 | GenericTypeaheadInitializer initializer = 98 | new GenericTypeaheadInitializer(config); 99 | 100 | return (GenericTypeahead)initializer.getTypeahead(); 101 | } 102 | 103 | public final Indexer getIndexer() { 104 | return indexer; 105 | } 106 | 107 | public final Typeahead getSearcher() { 108 | return searcher; 109 | } 110 | 111 | public final ArrayStoreElement getElementStore() { 112 | return elementStore; 113 | } 114 | } 115 | -------------------------------------------------------------------------------- /src/main/java/cleo/primer/RestDAO.java: -------------------------------------------------------------------------------- 1 | /* 2 | * Copyright (c) 2011-2012 3 | * 4 | * Licensed under the Apache License, Version 2.0 (the "License"); you may not 5 | * use this file except in compliance with the License. You may obtain a copy of 6 | * the License at 7 | * 8 | * http://www.apache.org/licenses/LICENSE-2.0 9 | * 10 | * Unless required by applicable law or agreed to in writing, software 11 | * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT 12 | * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the 13 | * License for the specific language governing permissions and limitations under 14 | * the License. 15 | */ 16 | 17 | package cleo.primer; 18 | 19 | /** 20 | * RestDAO 21 | * 22 | * @author jwu 23 | * @since 01/05, 2012 24 | */ 25 | public interface RestDAO { 26 | 27 | // HTTP-GET 28 | public T getElement(int elementId); 29 | 30 | // HTTP-DELETE 31 | public T deleteElement(int elementId) throws Exception; 32 | 33 | // HTTP-PUT 34 | public T updateElement(T element) throws Exception; 35 | 36 | // HTTP-POST 37 | public boolean insertElement(T element) throws Exception; 38 | } 39 | -------------------------------------------------------------------------------- /src/main/java/cleo/primer/TypeaheadInstance.java: -------------------------------------------------------------------------------- 1 | /* 2 | * Copyright (c) 2011-2012 3 | * 4 | * Licensed under the Apache License, Version 2.0 (the "License"); you may not 5 | * use this file except in compliance with the License. You may obtain a copy of 6 | * the License at 7 | * 8 | * http://www.apache.org/licenses/LICENSE-2.0 9 | * 10 | * Unless required by applicable law or agreed to in writing, software 11 | * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT 12 | * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the 13 | * License for the specific language governing permissions and limitations under 14 | * the License. 15 | */ 16 | 17 | package cleo.primer; 18 | 19 | import cleo.search.Element; 20 | import cleo.search.Indexer; 21 | import cleo.search.store.ArrayStoreElement; 22 | import cleo.search.typeahead.Typeahead; 23 | 24 | /** 25 | * TypeaheadInstance 26 | * 27 | * @author jwu 28 | * @since 12/22, 2011 29 | */ 30 | public interface TypeaheadInstance { 31 | 32 | /** 33 | * @return the typeahead indexer. 34 | */ 35 | public Indexer getIndexer(); 36 | 37 | /** 38 | * @return the typeahead searcher. 39 | */ 40 | public Typeahead getSearcher(); 41 | 42 | /** 43 | * @return the underlying element store. 44 | */ 45 | public ArrayStoreElement getElementStore(); 46 | } 47 | -------------------------------------------------------------------------------- /src/main/java/cleo/primer/rest/model/ElementDTO.java: -------------------------------------------------------------------------------- 1 | /* 2 | * Copyright (c) 2011-2012 3 | * 4 | * Licensed under the Apache License, Version 2.0 (the "License"); you may not 5 | * use this file except in compliance with the License. You may obtain a copy of 6 | * the License at 7 | * 8 | * http://www.apache.org/licenses/LICENSE-2.0 9 | * 10 | * Unless required by applicable law or agreed to in writing, software 11 | * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT 12 | * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the 13 | * License for the specific language governing permissions and limitations under 14 | * the License. 15 | */ 16 | 17 | package cleo.primer.rest.model; 18 | 19 | import java.util.Arrays; 20 | 21 | import javax.xml.bind.annotation.XmlElement; 22 | import javax.xml.bind.annotation.XmlRootElement; 23 | import javax.xml.bind.annotation.XmlTransient; 24 | 25 | import cleo.search.Element; 26 | 27 | /** 28 | * ElementDTO 29 | * 30 | * @author jwu 31 | * @since 12/22, 2011 32 | */ 33 | @XmlRootElement(name = "element") 34 | public class ElementDTO implements Element, Cloneable { 35 | private static final long serialVersionUID = 1L; 36 | private static final String[] EMPTY_TERMS = new String[0]; 37 | 38 | /** 39 | * Element ID. 40 | */ 41 | private int id; 42 | 43 | /** 44 | * Element name. 45 | */ 46 | private String name; 47 | 48 | /** 49 | * Element title 50 | */ 51 | private String title; 52 | 53 | /** 54 | * Element media (e.g. picture URL) 55 | */ 56 | private String media; 57 | 58 | /** 59 | * Element web URL 60 | */ 61 | private String url; 62 | 63 | /** 64 | * Element score. 65 | */ 66 | private float score; 67 | 68 | /** 69 | * The terms are standardized words that an element can be searched. 70 | */ 71 | @XmlElement(name = "term") 72 | private String[] terms = EMPTY_TERMS; 73 | 74 | /** 75 | * Element creation timestamp. 76 | */ 77 | private long timestamp = System.currentTimeMillis(); 78 | 79 | public void setId(int id) { 80 | this.id = id; 81 | } 82 | 83 | public int getId() { 84 | return id; 85 | } 86 | 87 | @Override @XmlTransient() 88 | public void setTerms(String... terms) { 89 | this.terms = (terms == null) ? EMPTY_TERMS : terms; 90 | } 91 | 92 | @Override 93 | public String[] getTerms() { 94 | return terms; 95 | } 96 | 97 | public void setName(String name) { 98 | this.name = name; 99 | } 100 | 101 | public String getName() { 102 | return name; 103 | } 104 | 105 | public void setTitle(String title) { 106 | this.title = title; 107 | } 108 | 109 | public String getTitle() { 110 | return title; 111 | } 112 | 113 | public void setMedia(String media) { 114 | this.media = media; 115 | } 116 | 117 | public String getMedia() { 118 | return media; 119 | } 120 | 121 | public String getUrl() { 122 | return url; 123 | } 124 | 125 | public void setUrl(String url) { 126 | this.url = url; 127 | } 128 | 129 | @Override 130 | public void setScore(float score) { 131 | this.score = score; 132 | } 133 | 134 | @Override 135 | public float getScore() { 136 | return score; 137 | } 138 | 139 | @Override 140 | public void setTimestamp(long timestamp) { 141 | this.timestamp = timestamp; 142 | } 143 | 144 | @Override 145 | public long getTimestamp() { 146 | return timestamp; 147 | } 148 | 149 | @Override @XmlTransient 150 | public int getElementId() { 151 | return id; 152 | } 153 | 154 | @Override 155 | public void setElementId(int id) { 156 | this.id = id; 157 | } 158 | 159 | @Override 160 | public Object clone() { 161 | ElementDTO obj = new ElementDTO(); 162 | 163 | obj.id = id; 164 | obj.name = name; 165 | obj.title = title; 166 | obj.media = media; 167 | obj.score = score; 168 | obj.terms = terms; 169 | obj.timestamp = timestamp; 170 | 171 | return obj; 172 | } 173 | 174 | @Override 175 | public int hashCode() { 176 | int hashCode = id; 177 | hashCode += timestamp / 23; 178 | 179 | if(terms != null) { 180 | for(String t : terms) { 181 | hashCode += t.hashCode(); 182 | } 183 | } 184 | 185 | return hashCode; 186 | } 187 | 188 | @Override 189 | public boolean equals(Object o) { 190 | if(o == null) return false; 191 | if(o.getClass() == getClass()) { 192 | ElementDTO e = (ElementDTO)o; 193 | return id == e.id && 194 | timestamp == e.timestamp && 195 | Arrays.equals(terms, e.terms) && 196 | score == e.score; 197 | } else { 198 | return false; 199 | } 200 | } 201 | 202 | @Override 203 | public int compareTo(Element e) { 204 | return score < e.getScore() ? -1 : (score == e.getScore() ? (getElementId() - e.getElementId()) : 1); 205 | } 206 | 207 | public boolean isSearchable() { 208 | return terms != null && terms.length > 0; 209 | } 210 | } 211 | -------------------------------------------------------------------------------- /src/main/java/cleo/primer/rest/model/ElementListDTO.java: -------------------------------------------------------------------------------- 1 | /* 2 | * Copyright (c) 2011-2012 3 | * 4 | * Licensed under the Apache License, Version 2.0 (the "License"); you may not 5 | * use this file except in compliance with the License. You may obtain a copy of 6 | * the License at 7 | * 8 | * http://www.apache.org/licenses/LICENSE-2.0 9 | * 10 | * Unless required by applicable law or agreed to in writing, software 11 | * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT 12 | * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the 13 | * License for the specific language governing permissions and limitations under 14 | * the License. 15 | */ 16 | 17 | package cleo.primer.rest.model; 18 | 19 | import java.util.ArrayList; 20 | import java.util.List; 21 | 22 | import javax.xml.bind.annotation.XmlElement; 23 | import javax.xml.bind.annotation.XmlRootElement; 24 | 25 | @XmlRootElement(name = "element-list") 26 | public class ElementListDTO { 27 | 28 | @XmlElement(name = "element") 29 | public List elements; 30 | 31 | public ElementListDTO() { 32 | this.elements = new ArrayList(); 33 | } 34 | 35 | public ElementListDTO(List elements) { 36 | this.elements = elements; 37 | } 38 | } 39 | -------------------------------------------------------------------------------- /src/main/java/cleo/primer/rest/model/ExceptionDTO.java: -------------------------------------------------------------------------------- 1 | /* 2 | * Copyright (c) 2011-2012 3 | * 4 | * Licensed under the Apache License, Version 2.0 (the "License"); you may not 5 | * use this file except in compliance with the License. You may obtain a copy of 6 | * the License at 7 | * 8 | * http://www.apache.org/licenses/LICENSE-2.0 9 | * 10 | * Unless required by applicable law or agreed to in writing, software 11 | * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT 12 | * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the 13 | * License for the specific language governing permissions and limitations under 14 | * the License. 15 | */ 16 | 17 | package cleo.primer.rest.model; 18 | 19 | import java.io.PrintWriter; 20 | import java.io.StringWriter; 21 | 22 | import javax.xml.bind.annotation.XmlElement; 23 | import javax.xml.bind.annotation.XmlRootElement; 24 | 25 | @XmlRootElement(name = "exception") 26 | public class ExceptionDTO { 27 | 28 | @XmlElement 29 | public String message; 30 | 31 | @XmlElement(name = "class") 32 | public String className; 33 | 34 | @XmlElement(name = "stack-trace") 35 | public String stackTrace; 36 | 37 | public ExceptionDTO() {} 38 | 39 | public ExceptionDTO(Exception exception) { 40 | this.message = exception.getMessage(); 41 | this.className = exception.getClass().getName(); 42 | 43 | StringWriter stringWriter = new StringWriter(); 44 | exception.printStackTrace(new PrintWriter(stringWriter)); 45 | this.stackTrace = stringWriter.toString(); 46 | } 47 | } 48 | -------------------------------------------------------------------------------- /src/main/java/cleo/primer/rest/resources/ElementsResource.java: -------------------------------------------------------------------------------- 1 | /* 2 | * Copyright (c) 2011-2012 3 | * 4 | * Licensed under the Apache License, Version 2.0 (the "License"); you may not 5 | * use this file except in compliance with the License. You may obtain a copy of 6 | * the License at 7 | * 8 | * http://www.apache.org/licenses/LICENSE-2.0 9 | * 10 | * Unless required by applicable law or agreed to in writing, software 11 | * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT 12 | * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the 13 | * License for the specific language governing permissions and limitations under 14 | * the License. 15 | */ 16 | 17 | package cleo.primer.rest.resources; 18 | 19 | import java.io.IOException; 20 | import java.util.ArrayList; 21 | import java.util.List; 22 | 23 | import javax.ws.rs.Consumes; 24 | import javax.ws.rs.DELETE; 25 | import javax.ws.rs.QueryParam; 26 | import javax.ws.rs.GET; 27 | import javax.ws.rs.POST; 28 | import javax.ws.rs.PUT; 29 | import javax.ws.rs.Path; 30 | import javax.ws.rs.PathParam; 31 | import javax.ws.rs.Produces; 32 | import javax.ws.rs.core.Context; 33 | import javax.ws.rs.core.MediaType; 34 | import javax.ws.rs.core.Request; 35 | import javax.ws.rs.core.Response; 36 | import javax.ws.rs.core.UriInfo; 37 | import javax.ws.rs.core.Response.Status; 38 | 39 | import org.springframework.stereotype.Component; 40 | 41 | import cleo.primer.ElementDAO; 42 | import cleo.primer.rest.model.ElementDTO; 43 | import cleo.primer.rest.model.ElementListDTO; 44 | import cleo.primer.rest.model.ExceptionDTO; 45 | import cleo.search.collector.Collector; 46 | import cleo.search.collector.SortedCollector; 47 | 48 | @Component 49 | @Path("/elements") 50 | public class ElementsResource { 51 | // Allows to insert contextual objects such as 52 | // ServletContext, Request, Response and UriInfo 53 | @Context 54 | UriInfo uriInfo; 55 | 56 | @Context 57 | Request request; 58 | 59 | @GET 60 | @Produces({MediaType.APPLICATION_XML, MediaType.APPLICATION_JSON}) 61 | public ElementListDTO getElements() { 62 | List list = ElementDAO.INSTANCE.getElements(); 63 | return new ElementListDTO(list); 64 | } 65 | 66 | @GET 67 | @Path("/{start}..{end}") 68 | @Produces({MediaType.APPLICATION_XML, MediaType.APPLICATION_JSON}) 69 | public ElementListDTO getElements(@PathParam("start")int start, @PathParam("end")int end) { 70 | List list = new ArrayList(); 71 | 72 | for(int index = start; index <= end; index++) { 73 | ElementDTO dto = ElementDAO.INSTANCE.getElement(index); 74 | if(dto != null) { 75 | list.add(dto); 76 | } 77 | } 78 | 79 | return new ElementListDTO(list); 80 | } 81 | 82 | @POST 83 | @Consumes({MediaType.APPLICATION_XML, MediaType.APPLICATION_JSON}) 84 | public Response addElements(ElementListDTO elementListDTO) { 85 | for(ElementDTO elementDTO : elementListDTO.elements) { 86 | try { 87 | ElementDAO.INSTANCE.insertElement(elementDTO); 88 | } catch (Exception e) { 89 | e.printStackTrace(); 90 | } 91 | } 92 | 93 | return Response.status(Status.OK).build(); 94 | } 95 | 96 | @POST 97 | @Path("/_") 98 | @Consumes({MediaType.APPLICATION_XML, MediaType.APPLICATION_JSON}) 99 | public Response addElement(ElementDTO elementDTO) { 100 | try { 101 | ElementDAO.INSTANCE.insertElement(elementDTO); 102 | return Response.status(Status.OK).build(); 103 | } catch (Exception e) { 104 | return Response.status(Status.SEE_OTHER).entity(new ExceptionDTO(e)).build(); 105 | } 106 | } 107 | 108 | @GET 109 | @Path("/{index}") 110 | @Produces({MediaType.APPLICATION_XML, MediaType.APPLICATION_JSON}) 111 | public Response getElement(@PathParam("index")int index) { 112 | try { 113 | ElementDTO dto = ElementDAO.INSTANCE.getElement(index); 114 | return Response.status(Status.OK).entity(dto).build(); 115 | } catch(Exception e) { 116 | return Response.status(Status.NOT_FOUND).entity(new ExceptionDTO(e)).build(); 117 | } 118 | } 119 | 120 | @PUT 121 | @Path("/{index}") 122 | @Consumes({MediaType.APPLICATION_XML, MediaType.APPLICATION_JSON}) 123 | @Produces({MediaType.APPLICATION_XML, MediaType.APPLICATION_JSON}) 124 | public Response putElement(@PathParam("index")int index, ElementDTO elementDTO) { 125 | ElementDTO old; 126 | try { 127 | old = ElementDAO.INSTANCE.updateElement(elementDTO); 128 | return Response.status(Status.OK).entity(old).build(); 129 | } catch (Exception e) { 130 | return Response.status(Status.NOT_FOUND).entity(new ExceptionDTO(e)).build(); 131 | } 132 | } 133 | 134 | @DELETE 135 | @Path("/{index}") 136 | @Produces({MediaType.APPLICATION_XML, MediaType.APPLICATION_JSON}) 137 | public Response deleteElement(@PathParam("index")int index) { 138 | try { 139 | ElementDTO old = ElementDAO.INSTANCE.deleteElement(index); 140 | return Response.status(Status.OK).entity(old).build(); 141 | } catch(Exception e) { 142 | return Response.status(Status.NOT_FOUND).entity(new ExceptionDTO(e)).build(); 143 | } 144 | } 145 | 146 | @GET 147 | @Path("/search") 148 | @Produces({MediaType.APPLICATION_XML, MediaType.APPLICATION_JSON}) 149 | public ElementListDTO search(@QueryParam("uid")int uid, 150 | @QueryParam("query")String query) { 151 | if(query == null) { 152 | return new ElementListDTO(); 153 | } 154 | 155 | String[] terms = query.replaceAll("\\W+", " ").toLowerCase().split(" "); 156 | Collector collector = new SortedCollector(10, 100); 157 | collector = ElementDAO.INSTANCE.getSearcher().search(uid, terms, collector); 158 | 159 | return new ElementListDTO(collector.elements()); 160 | } 161 | 162 | @POST 163 | @Path("/flush") 164 | @Produces(MediaType.TEXT_HTML) 165 | public Response flush() { 166 | try { 167 | ElementDAO.INSTANCE.getIndexer().flush(); 168 | return Response.status(Status.OK).entity("OK").build(); 169 | } catch (IOException ioe) { 170 | return Response.status(Status.SERVICE_UNAVAILABLE).entity(new ExceptionDTO(ioe)).build(); 171 | } 172 | } 173 | } 174 | -------------------------------------------------------------------------------- /src/main/java/cleo/primer/util/MaxMindWorldCitiesXML.java: -------------------------------------------------------------------------------- 1 | /* 2 | * Copyright (c) 2011-2012 3 | * 4 | * Licensed under the Apache License, Version 2.0 (the "License"); you may not 5 | * use this file except in compliance with the License. You may obtain a copy of 6 | * the License at 7 | * 8 | * http://www.apache.org/licenses/LICENSE-2.0 9 | * 10 | * Unless required by applicable law or agreed to in writing, software 11 | * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT 12 | * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the 13 | * License for the specific language governing permissions and limitations under 14 | * the License. 15 | */ 16 | 17 | package cleo.primer.util; 18 | 19 | import java.io.BufferedReader; 20 | import java.io.File; 21 | import java.io.FileReader; 22 | import java.io.IOException; 23 | import java.io.PrintWriter; 24 | 25 | /** 26 | * MaxMindWorldCitiesXML converts MaxMind free world cities data to XML. 27 | * 28 | * 
 29 |  * Country,City,AccentCity,Region,Population,Latitude,Longitude
 30 |  * ad,aixirivali,Aixirivali,06,,42.4666667,1.5
 31 |  * us,winchester,Winchester,WY,,43.8600000,-108.1600000
 32 |  *
33 | * 34 | * @author jwu 35 | * @since 01/15, 2012 36 | */ 37 | public class MaxMindWorldCitiesXML { 38 | public static String XML_CHARSET_NAME = "UTF-8"; 39 | public static int XML_NUM_OF_ELEMENTS = 100000; 40 | 41 | /** 42 | * XMLize a String. 43 | */ 44 | public static String xmlize(String s) { 45 | return s.replaceAll("\"|\\a|\\e|\\f|\\v", "") 46 | .replaceAll("&", "&") 47 | .replaceAll("'", "'") 48 | .replaceAll("<", "<") 49 | .replaceAll(">", ">") 50 | .replaceAll("\\s+", " ") 51 | .trim(); 52 | } 53 | 54 | /** 55 | * Generate MaxMind world cities in XML files. 56 | * 57 | * 
 58 |      * java MaxMindWorldCitiesXML <WorldCitiesTextFile> <NumOfElementsPerXML>
 59 |      * java MaxMindWorldCitiesXML world-cities.txt 100000
 60 |      *
61 | */ 62 | public static void main(String[] args) throws IOException { 63 | File txtFile = new File(args[0]); 64 | String xmlPath = txtFile.getCanonicalPath().replaceAll(".txt$", ""); 65 | 66 | int numElementsPerXML = XML_NUM_OF_ELEMENTS; 67 | try { 68 | numElementsPerXML = Integer.parseInt(args[1]); 69 | } catch(Exception e) { 70 | numElementsPerXML = XML_NUM_OF_ELEMENTS; 71 | } 72 | 73 | BufferedReader r = new BufferedReader(new FileReader(txtFile)); 74 | 75 | PrintWriter w = null; 76 | 77 | int id = 0; 78 | String line = r.readLine(); 79 | while((line = r.readLine()) != null) { 80 | if(id % numElementsPerXML == 0) { 81 | if (w != null) { 82 | w.println(""); 83 | w.flush(); 84 | w.close(); 85 | } 86 | 87 | int xmlId = id / numElementsPerXML + 1; 88 | File xmlFile = new File(xmlPath + "." + xmlId + ".xml"); 89 | w = new PrintWriter(xmlFile, XML_CHARSET_NAME); 90 | w.println(""); 91 | w.println(""); 92 | } 93 | 94 | // Split CSV 95 | String[] parts = line.split(","); 96 | 97 | String country = parts[0]; 98 | String[] terms = parts[1].replaceAll("\"", "").split(",|-|\\s+"); 99 | String name = parts[2]; 100 | String region = parts[3]; 101 | float score = parts[4].length() == 0 ? 0 : (Long.parseLong(parts[4]) / 1000000f); 102 | 103 | w.println(" "); 104 | w.printf (" %d%n", ++id); 105 | w.printf (" %s (%s)%n", xmlize(name), country.toUpperCase()); 106 | w.printf (" %s%n", country); 107 | w.printf (" %s%n", region); 108 | w.printf (" %s%n", parts[5]); 109 | w.printf (" %s%n", parts[6]); 110 | w.printf (" %f%n", score); 111 | for(String t : terms) { 112 | if(t.length() > 0) { 113 | w.printf(" %s%n", xmlize(t)); 114 | } 115 | } 116 | w.printf (" %s%n", country); 117 | w.println(" "); 118 | } 119 | 120 | if (w != null) { 121 | w.println(""); 122 | w.flush(); 123 | w.close(); 124 | } 125 | 126 | r.close(); 127 | } 128 | } 129 | -------------------------------------------------------------------------------- /src/main/resources/applicationContext.xml: -------------------------------------------------------------------------------- 1 | 5 | 6 | 7 | -------------------------------------------------------------------------------- /src/main/resources/config/generic-typeahead/i001.config: -------------------------------------------------------------------------------- 1 | cleo.search.generic.typeahead.config.name=i001 2 | cleo.search.generic.typeahead.config.partition.start=0 3 | cleo.search.generic.typeahead.config.partition.count=1000000 4 | cleo.search.generic.typeahead.config.homeDir=generic-typeahead/i001 5 | 6 | cleo.search.generic.typeahead.config.elementSerializer.class=cleo.search.ElementJavaSerializer 7 | 8 | cleo.search.generic.typeahead.config.elementStoreDir=${cleo.search.generic.typeahead.config.homeDir}/element-store 9 | cleo.search.generic.typeahead.config.elementStoreIndexStart=${cleo.search.generic.typeahead.config.partition.start} 10 | cleo.search.generic.typeahead.config.elementStoreCapacity=${cleo.search.generic.typeahead.config.partition.count} 11 | cleo.search.generic.typeahead.config.elementStoreSegmentMB=32 12 | cleo.search.generic.typeahead.config.elementStoreCached=true 13 | 14 | cleo.search.generic.typeahead.config.connectionsStoreDir=${cleo.search.generic.typeahead.config.homeDir}/connections-store 15 | cleo.search.generic.typeahead.config.connectionsStoreCapacity=1000000 16 | cleo.search.generic.typeahead.config.connectionsStoreSegmentMB=32 17 | cleo.search.generic.typeahead.config.connectionsStoreIndexSegmentMB=8 18 | 19 | cleo.search.generic.typeahead.config.filterPrefixLength=2 20 | cleo.search.generic.typeahead.config.maxKeyLength=5 21 | -------------------------------------------------------------------------------- /src/main/resources/config/generic-typeahead/i002.config: -------------------------------------------------------------------------------- 1 | cleo.search.generic.typeahead.config.name=i002 2 | cleo.search.generic.typeahead.config.partition.start=1000000 3 | cleo.search.generic.typeahead.config.partition.count=1000000 4 | cleo.search.generic.typeahead.config.homeDir=generic-typeahead/i002 5 | 6 | cleo.search.generic.typeahead.config.elementSerializer.class=cleo.search.ElementJavaSerializer 7 | 8 | cleo.search.generic.typeahead.config.elementStoreDir=${cleo.search.generic.typeahead.config.homeDir}/element-store 9 | cleo.search.generic.typeahead.config.elementStoreIndexStart=${cleo.search.generic.typeahead.config.partition.start} 10 | cleo.search.generic.typeahead.config.elementStoreCapacity=${cleo.search.generic.typeahead.config.partition.count} 11 | cleo.search.generic.typeahead.config.elementStoreSegmentMB=32 12 | cleo.search.generic.typeahead.config.elementStoreCached=true 13 | 14 | cleo.search.generic.typeahead.config.connectionsStoreDir=${cleo.search.generic.typeahead.config.homeDir}/connections-store 15 | cleo.search.generic.typeahead.config.connectionsStoreCapacity=1000000 16 | cleo.search.generic.typeahead.config.connectionsStoreSegmentMB=32 17 | cleo.search.generic.typeahead.config.connectionsStoreIndexSegmentMB=8 18 | 19 | cleo.search.generic.typeahead.config.filterPrefixLength=2 20 | cleo.search.generic.typeahead.config.maxKeyLength=5 21 | -------------------------------------------------------------------------------- /src/main/resources/log4j.properties: -------------------------------------------------------------------------------- 1 | log4j.rootLogger=INFO,cleoLogFile 2 | 3 | log4j.appender.cleoLogFile=org.apache.log4j.FileAppender 4 | log4j.appender.cleoLogFile.layout=org.apache.log4j.PatternLayout 5 | log4j.appender.cleoLogFile.layout.ConversionPattern=%d{yyyy/MM/dd HH:mm:ss.SSS} %p [%c] %m%n 6 | log4j.appender.cleoLogFile.file=target/logs/cleo.log 7 | log4j.appender.cleoLogFile.encoding=UTF-8 8 | 9 | log4j.appender.kratiLogFile=org.apache.log4j.FileAppender 10 | log4j.appender.kratiLogFile.layout=org.apache.log4j.PatternLayout 11 | log4j.appender.kratiLogFile.layout.ConversionPattern=%d{yyyy/MM/dd HH:mm:ss.SSS} %p [%c] %m%n 12 | log4j.appender.kratiLogFile.file=target/logs/krati.log 13 | log4j.appender.kratiLogFile.encoding=UTF-8 14 | 15 | # Turn on all our debugging info 16 | # log4j.logger=DEBUG 17 | log4j.logger.cleo=cleoLogFile 18 | log4j.logger.krati=kratiLogFile 19 | -------------------------------------------------------------------------------- /src/main/webapp/WEB-INF/web.xml: -------------------------------------------------------------------------------- 1 | 2 | 5 | 6 | 7 | contextConfigLocation 8 | classpath:applicationContext.xml 9 | 10 | 11 | 12 | org.springframework.web.context.ContextLoaderListener 13 | 14 | 15 | 16 | org.springframework.web.context.request.RequestContextListener 17 | 18 | 19 | 20 | Jersey Spring Web Application 21 | com.sun.jersey.spi.spring.container.servlet.SpringServlet 22 | 23 | com.sun.jersey.config.property.packages 24 | cleo.primer.rest.resources 25 | 26 | 1 27 | 28 | 29 | 30 | Jersey Spring Web Application 31 | /rest/* 32 | 33 | 34 | -------------------------------------------------------------------------------- /src/main/webapp/index.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 142 | 143 | 144 | 145 |
146 |

Cleo!

147 | 148 |
149 |
150 | 151 |
152 |
153 |
154 |
155 | 156 | 212 | 213 | 214 | --------------------------------------------------------------------------------