├── .gitignore ├── LICENSE ├── README.md ├── pom.xml └── src ├── main └── java │ └── org │ └── udger │ └── parser │ ├── LRUCache.java │ ├── UdgerIpResult.java │ ├── UdgerParser.java │ ├── UdgerSqlQuery.java │ ├── UdgerUaResult.java │ └── WordDetector.java └── test ├── java └── org │ └── udger │ └── parser │ ├── LRUCacheTest.java │ ├── UdgerIpTest.java │ ├── UdgerParserChangeDBTest.java │ ├── UdgerParserTest.java │ ├── UdgerPerformanceTest.java │ ├── UdgerUaTest.java │ ├── test_ip.json │ └── test_ua.json └── resources ├── udgerdb_test_v3.dat └── udgerdb_test_v3_switch.dat /.gitignore: -------------------------------------------------------------------------------- 1 | #Eclipse 2 | **/.settings 3 | **/.project 4 | **/.classpath 5 | **/target 6 | 7 | #IntelliJ 8 | *.idea 9 | *.iml -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2013 by Jure Ham and contributors. 4 | Copyright (c) 2015 by The Udger.com Team. 5 | 6 | Permission is hereby granted, free of charge, to any person obtaining a copy 7 | of this software and associated documentation files (the "Software"), to deal 8 | in the Software without restriction, including without limitation the rights 9 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 10 | copies of the Software, and to permit persons to whom the Software is 11 | furnished to do so, subject to the following conditions: 12 | 13 | The above copyright notice and this permission notice shall be included in all 14 | copies or substantial portions of the Software. 15 | 16 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 17 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 18 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 19 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 20 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 21 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 22 | SOFTWARE. -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Udger client for Java (data ver. 3) 2 | Local parser is very fast and accurate useragent string detection solution. Enables developers to locally install and integrate a highly-scalable product. 3 | We provide the detection of the devices (personal computer, tablet, Smart TV, Game console etc.), operating system and client SW type (browser, e-mail client etc.). 4 | It also provides information about IP addresses (Public proxies, VPN services, Tor exit nodes, Fake crawlers, Web scrapers .. etc.) 5 | 6 | 7 | - Tested with more the 50.000 unique user agents. 8 | - Up to date data provided by https://udger.com/ 9 | - Support for >=Java6 10 | 11 | ### Performance 12 | Udger java parser uses LRU cache for last N requests. The size of cache can be defined in constructor, default size is 10000 requests. Parser's performance is tuned continuously, currently it reaches following rates: 13 | 14 | - >100.000 requests per second if LRU is hitted 15 | - 2.000 requests per second without caching 16 | 17 | Using the in memory DB option will even make it faster. 18 | 19 | ### Compile from git repo 20 | 21 | ```sh 22 | $ git clone https://github.com/udger/udger-java 23 | $ cd udger-java/ 24 | $ maven package 25 | ``` 26 | 27 | ### Requirements 28 | Udger data is stored in SQLite database file. Udger-java connects to SqLite using JDBC driver. SQLiteJDBC jdbc driver is recommended. If you are using Maven2, add the following XML fragments into your pom.xml file: 29 | 30 | ```xml 31 | 32 | org.xerial 33 | sqlite-jdbc 34 | 3.8.11.2 35 | 36 | ``` 37 | 38 | ### Usage 39 | 40 | #### How to Specify Udger Database 41 | 42 | Example how to create UdgerParser from udger db file `C:\work\udgerdb_v3.dat` (in Windows) 43 | 44 | ```java 45 | UdgerParser.ParserDbData parserDbData = new UdgerParser.ParserDbData("C:/work/udgerdb_v3.dat"); 46 | UdgerParser up = = new UdgerParser(parserDbData); 47 | ... 48 | up.close(); 49 | ``` 50 | 51 | and from a UNIX (Linux, Mac OS X, etc) udger db file `/home/john/work/udgerdb_v3.dat` 52 | 53 | ```java 54 | UdgerParser.ParserDbData parserDbData = new UdgerParser.ParserDbData("/home/john/work/udgerdb_v3.dat"); 55 | UdgerParser up = = new UdgerParser(parserDbData); 56 | ... 57 | up.close(); 58 | ``` 59 | 60 | UdgerParser implements Closeable interface, therefore it must be either opened in `try (...)` statement or explicitly closed. 61 | Since the SQLite connection creating is time consuming task, it is recommended to keep the UdgerParser's instances in 62 | an instance pool. UdgerParser is not thread safe object, therefore it can't be used from multiple thread simultaneously. 63 | 64 | Intention of class `UdgerParser.ParserDbData` is to keep precalculated DB-specific data and then improve instantiation 65 | of `UdgerParser`. Using `UdgerParser.ParserDbData` the Udger database can be switched in runtime. 66 | 67 | 68 | #### How to make use of In Memory feature 69 | 70 | The Udger client supports the SQLite DB transactions with the database being in memory. Enabling this feature will make the parser even faster to parse the user agents. Internally the client will re-create the Udger SQLite database from the file into the systems main memory and perform all transactions to it. Since this will require additional memory for operation, it needs to be used carefully with object pools. During pooling with multiple parsers in the pool, this feature will create a separate in memory DB for each new parser and have a single connection to it. This will further allow more concurrency since all connections (from all pooled parsers) now have their own copy of the database. 71 | To enable in memory feature simply use the below constructor and pass inMemoryEnabled as `true`. The internal LRU cache can be used by setting a size > 0 or disabled by passing 0 for the third argument. 72 | 73 | Example: 74 | 75 | ```java 76 | UdgerParser.ParserDbData parserDbData = new UdgerParser.ParserDbData("/home/john/work/udgerdb_v3.dat"); 77 | UdgerParser up = new UdgerParser(parserDbData, true, 10000); 78 | ... 79 | uo.close(); 80 | ``` 81 | 82 | ### Usage with maven 83 | 84 | ```xml 85 | 86 | org.udger.parser 87 | udger-parser 88 | 1.1.1 89 | 90 | ``` 91 | 92 | #### Sample.java 93 | 94 | ```java 95 | public class Sample { 96 | public static void main(String[] args) { 97 | UdgerParser.ParserDbData parserDbData = new UdgerParser.ParserDbData("/home/john/work/udgerdb_v3.dat"); 98 | try (UdgerParser up = new UdgerParser(parserDbData)) { 99 | UdgerUaResult uaRet = up.parseUa("Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_2) AppleWebKit/601.3.9 (KHTML, like Gecko) Version/9.0.2 Safari/601.3.9"); 100 | UdgerIpResult ipRet = up.parseIp("108.61.199.93"); 101 | } catch (SQLException e) { 102 | e.printStackTrace(); 103 | } catch (UnknownHostException e) { 104 | e.printStackTrace(); 105 | } 106 | } 107 | } 108 | ``` 109 | 110 | ### Automatic updates download 111 | - for auto-update data use Udger data updater (https://udger.com/support/documentation/?doc=62) 112 | 113 | ### Documentation for programmers 114 | - https://udger.com/pub/documentation/parser/JAVA/html/ 115 | 116 | ### Author 117 | The Udger.com Team (info@udger.com) 118 | 119 | ### old v1 format 120 | If you still use the previous format of the db (v1), you can use https://github.com/adhar1985/DIUASparser 121 | -------------------------------------------------------------------------------- /pom.xml: -------------------------------------------------------------------------------- 1 | 3 | 4 | 4.0.0 5 | org.udger.parser 6 | udger-parser 7 | 1.2.4 8 | Udger user agent and IP parser 9 | Parse user agent string or IP using udger sqlite DB 10 | http://udger.com 11 | 12 | 13 | 14 | JAVA Udger client LICENSE 15 | https://udger.com/legal/java_parser 16 | 17 | 18 | 19 | 20 | 21 | Udger Team 22 | support@udger.com 23 | udger 24 | http://www.udger.com 25 | 26 | 27 | 28 | 29 | scm:git:git@github.com:udger/udger-java.git 30 | scm:git:ssh://github.com:udger/udger-java.git 31 | https://github.com/udger/udger-java 32 | 33 | 34 | 35 | 36 | 37 | junit 38 | junit 39 | 4.11 40 | test 41 | 42 | 43 | 44 | org.xerial 45 | sqlite-jdbc 46 | 3.36.0.3 47 | 48 | 49 | 50 | javax.json 51 | javax.json-api 52 | 1.0 53 | test 54 | 55 | 56 | 57 | org.glassfish 58 | javax.json 59 | 1.0.4 60 | test 61 | 62 | 63 | 64 | org.apache.commons 65 | commons-lang3 66 | 3.4 67 | test 68 | 69 | 70 | 71 | org.mockito 72 | mockito-all 73 | 1.10.19 74 | test 75 | 76 | 77 | 78 | 79 | 80 | 81 | ossrh 82 | https://oss.sonatype.org/content/repositories/snapshots 83 | 84 | 85 | ossrh 86 | https://oss.sonatype.org/service/local/staging/deploy/maven2/ 87 | 88 | 89 | 90 | 91 | 92 | 93 | ${project.basedir}/src/test/resources 94 | 95 | 96 | 97 | 98 | org.apache.maven.plugins 99 | maven-compiler-plugin 100 | 3.1 101 | 102 | 1.7 103 | 1.7 104 | 105 | 106 | 107 | org.apache.maven.plugins 108 | maven-jar-plugin 109 | 2.4 110 | 111 | 112 | 113 | 114 | 115 | 116 | org.sonatype.plugins 117 | nexus-staging-maven-plugin 118 | 1.6.7 119 | true 120 | 121 | ossrh 122 | https://oss.sonatype.org/ 123 | true 124 | 125 | 126 | 127 | 128 | org.apache.maven.plugins 129 | maven-source-plugin 130 | 2.2.1 131 | 132 | 133 | attach-sources 134 | 135 | jar-no-fork 136 | 137 | 138 | 139 | 140 | 141 | org.apache.maven.plugins 142 | maven-javadoc-plugin 143 | 2.9.1 144 | 145 | 146 | attach-javadocs 147 | 148 | jar 149 | 150 | 151 | 152 | 153 | 154 | org.apache.maven.plugins 155 | maven-gpg-plugin 156 | 1.5 157 | 158 | 159 | sign-artifacts 160 | verify 161 | 162 | sign 163 | 164 | 165 | 166 | 167 | 168 | org.sonatype.plugins 169 | nexus-staging-maven-plugin 170 | 1.6.7 171 | true 172 | 173 | ossrh 174 | https://oss.sonatype.org/ 175 | true 176 | 177 | 178 | 179 | 180 | 181 | 182 | -------------------------------------------------------------------------------- /src/main/java/org/udger/parser/LRUCache.java: -------------------------------------------------------------------------------- 1 | package org.udger.parser; 2 | 3 | import java.io.Serializable; 4 | import java.util.HashMap; 5 | import java.util.Map; 6 | 7 | /** 8 | * The Class LRUCache. Simple LRU cache for UA Parser 9 | */ 10 | public class LRUCache implements Serializable { 11 | 12 | private static final long serialVersionUID = 275929298283639982L; 13 | 14 | private static class Node implements Serializable { 15 | private static final long serialVersionUID = -2815264316130381309L; 16 | private Node prev; 17 | private Node next; 18 | private K key; 19 | private V value; 20 | } 21 | 22 | private Node head; 23 | private Node tail; 24 | private int capacity; 25 | 26 | private final Map> map = new HashMap<>(); 27 | 28 | public LRUCache(int capacity) { 29 | this.capacity = capacity; 30 | } 31 | 32 | public int getCapacity() { 33 | return capacity; 34 | } 35 | 36 | public void setCapacity(int capacity) { 37 | if (this.capacity > capacity) { 38 | while (map.size() > capacity) { 39 | assert (tail != null); 40 | map.remove(tail.key); 41 | tail = tail.prev; 42 | tail.next = null; 43 | } 44 | } 45 | this.capacity = capacity; 46 | } 47 | 48 | public void clear() { 49 | this.map.clear(); 50 | } 51 | 52 | public V get(K uaString) { 53 | Node node = map.get(uaString); 54 | if (node != null) { 55 | if (head != node) { 56 | if (node.next != null) { 57 | node.next.prev = node.prev; 58 | } else { 59 | tail = node.prev; 60 | } 61 | node.prev.next = node.next; 62 | head.prev = node; 63 | node.next = head; 64 | node.prev = null; 65 | head = node; 66 | } 67 | return node.value; 68 | } 69 | return null; 70 | } 71 | 72 | public void put(K key, V value) { 73 | Node node = map.get(key); 74 | if (node == null) { 75 | node = new Node<>(); 76 | node.value = value; 77 | node.key = key; 78 | node.next = head; 79 | node.prev = null; 80 | if (head != null) { 81 | head.prev = node; 82 | } 83 | if (tail == null) { 84 | tail = head; 85 | } 86 | head = node; 87 | map.put(key, node); 88 | if (map.size() > capacity) { 89 | assert (tail != null); 90 | map.remove(tail.key); 91 | tail = tail.prev; 92 | tail.next = null; 93 | } 94 | } 95 | node.value = value; 96 | } 97 | 98 | } 99 | 100 | -------------------------------------------------------------------------------- /src/main/java/org/udger/parser/UdgerIpResult.java: -------------------------------------------------------------------------------- 1 | /* 2 | Udger-update - Data updater for udger local and cloud parser 3 | 4 | author The Udger.com Team (info@udger.com) 5 | copyright Copyright (c) Udger s.r.o. 6 | license GNU Lesser General Public License 7 | link https://udger.com/products 8 | */ 9 | package org.udger.parser; 10 | 11 | import java.io.Serializable; 12 | 13 | public class UdgerIpResult implements Serializable { 14 | 15 | private static final long serialVersionUID = 1L; 16 | 17 | // IP 18 | private final String ip; 19 | private int ipVer = 0; 20 | private String ipClassification = ""; 21 | private String ipClassificationCode = ""; 22 | private String ipLastSeen = ""; 23 | private String ipHostname = ""; 24 | private String ipCountry = ""; 25 | private String ipCountryCode = ""; 26 | private String ipCity = ""; 27 | private String crawlerName = ""; 28 | private String crawlerVer = ""; 29 | private String crawlerVerMajor = ""; 30 | private String crawlerFamily = ""; 31 | private String crawlerFamilyCode = ""; 32 | private String crawlerFamilyHomepage = ""; 33 | private String crawlerFamilyVendor = ""; 34 | private String crawlerFamilyVendorCode = ""; 35 | private String crawlerFamilyVendorHomepage = ""; 36 | private String crawlerFamilyIcon = ""; 37 | private String crawlerFamilyInfoUrl = ""; 38 | private String crawlerLastSeen = ""; 39 | private String crawlerCategory = ""; 40 | private String crawlerCategoryCode = ""; 41 | private String crawlerRespectRobotstxt = ""; 42 | 43 | // DATACENTER 44 | private String dataCenterName = ""; 45 | private String dataCenterNameCode = ""; 46 | private String dataCenterHomePage = ""; 47 | 48 | public UdgerIpResult(String ipString) { 49 | this.ip = ipString; 50 | } 51 | 52 | public String getIp() { 53 | return ip; 54 | } 55 | 56 | public int getIpVer() { 57 | return ipVer; 58 | } 59 | public void setIpVer(int ipVer) { 60 | this.ipVer = ipVer; 61 | } 62 | public String getIpClassification() { 63 | return ipClassification; 64 | } 65 | public void setIpClassification(String ipClassification) { 66 | this.ipClassification = ipClassification; 67 | } 68 | public String getIpClassificationCode() { 69 | return ipClassificationCode; 70 | } 71 | public void setIpClassificationCode(String ipClassificationCode) { 72 | this.ipClassificationCode = ipClassificationCode; 73 | } 74 | public String getIpLastSeen() { 75 | return ipLastSeen; 76 | } 77 | public void setIpLastSeen(String ipLastSeen) { 78 | this.ipLastSeen = ipLastSeen; 79 | } 80 | public String getIpHostname() { 81 | return ipHostname; 82 | } 83 | public void setIpHostname(String ipHostname) { 84 | this.ipHostname = ipHostname; 85 | } 86 | public String getIpCountry() { 87 | return ipCountry; 88 | } 89 | public void setIpCountry(String ipCountry) { 90 | this.ipCountry = ipCountry; 91 | } 92 | public String getIpCountryCode() { 93 | return ipCountryCode; 94 | } 95 | public void setIpCountryCode(String ipCountryCode) { 96 | this.ipCountryCode = ipCountryCode; 97 | } 98 | public String getIpCity() { 99 | return ipCity; 100 | } 101 | public void setIpCity(String ipCity) { 102 | this.ipCity = ipCity; 103 | } 104 | public String getCrawlerName() { 105 | return crawlerName; 106 | } 107 | public void setCrawlerName(String crawlerName) { 108 | this.crawlerName = crawlerName; 109 | } 110 | public String getCrawlerVer() { 111 | return crawlerVer; 112 | } 113 | public void setCrawlerVer(String crawlerVer) { 114 | this.crawlerVer = crawlerVer; 115 | } 116 | public String getCrawlerVerMajor() { 117 | return crawlerVerMajor; 118 | } 119 | public void setCrawlerVerMajor(String crawlerVerMajor) { 120 | this.crawlerVerMajor = crawlerVerMajor; 121 | } 122 | public String getCrawlerFamily() { 123 | return crawlerFamily; 124 | } 125 | public void setCrawlerFamily(String crawlerFamily) { 126 | this.crawlerFamily = crawlerFamily; 127 | } 128 | public String getCrawlerFamilyCode() { 129 | return crawlerFamilyCode; 130 | } 131 | public void setCrawlerFamilyCode(String crawlerFamilyCode) { 132 | this.crawlerFamilyCode = crawlerFamilyCode; 133 | } 134 | public String getCrawlerFamilyHomepage() { 135 | return crawlerFamilyHomepage; 136 | } 137 | public void setCrawlerFamilyHomepage(String crawlerFamilyHomepage) { 138 | this.crawlerFamilyHomepage = crawlerFamilyHomepage; 139 | } 140 | public String getCrawlerFamilyVendor() { 141 | return crawlerFamilyVendor; 142 | } 143 | public void setCrawlerFamilyVendor(String crawlerFamilyVendor) { 144 | this.crawlerFamilyVendor = crawlerFamilyVendor; 145 | } 146 | public String getCrawlerFamilyVendorCode() { 147 | return crawlerFamilyVendorCode; 148 | } 149 | public void setCrawlerFamilyVendorCode(String crawlerFamilyVendorCode) { 150 | this.crawlerFamilyVendorCode = crawlerFamilyVendorCode; 151 | } 152 | public String getCrawlerFamilyVendorHomepage() { 153 | return crawlerFamilyVendorHomepage; 154 | } 155 | public void setCrawlerFamilyVendorHomepage(String crawlerFamilyVendorHomepage) { 156 | this.crawlerFamilyVendorHomepage = crawlerFamilyVendorHomepage; 157 | } 158 | public String getCrawlerFamilyIcon() { 159 | return crawlerFamilyIcon; 160 | } 161 | public void setCrawlerFamilyIcon(String crawlerFamilyIcon) { 162 | this.crawlerFamilyIcon = crawlerFamilyIcon; 163 | } 164 | public String getCrawlerFamilyInfoUrl() { 165 | return crawlerFamilyInfoUrl; 166 | } 167 | public void setCrawlerFamilyInfoUrl(String crawlerFamilyInfoUrl) { 168 | this.crawlerFamilyInfoUrl = crawlerFamilyInfoUrl; 169 | } 170 | public String getCrawlerLastSeen() { 171 | return crawlerLastSeen; 172 | } 173 | public void setCrawlerLastSeen(String crawlerLastSeen) { 174 | this.crawlerLastSeen = crawlerLastSeen; 175 | } 176 | public String getCrawlerCategory() { 177 | return crawlerCategory; 178 | } 179 | public void setCrawlerCategory(String crawlerCategory) { 180 | this.crawlerCategory = crawlerCategory; 181 | } 182 | public String getCrawlerCategoryCode() { 183 | return crawlerCategoryCode; 184 | } 185 | public void setCrawlerCategoryCode(String crawlerCategoryCode) { 186 | this.crawlerCategoryCode = crawlerCategoryCode; 187 | } 188 | public String getCrawlerRespectRobotstxt() { 189 | return crawlerRespectRobotstxt; 190 | } 191 | public void setCrawlerRespectRobotstxt(String crawlerRespectRobotstxt) { 192 | this.crawlerRespectRobotstxt = crawlerRespectRobotstxt; 193 | } 194 | public String getDataCenterName() { 195 | return dataCenterName; 196 | } 197 | public void setDataCenterName(String dataCenterName) { 198 | this.dataCenterName = dataCenterName; 199 | } 200 | public String getDataCenterNameCode() { 201 | return dataCenterNameCode; 202 | } 203 | public void setDataCenterNameCode(String dataCenterNameCode) { 204 | this.dataCenterNameCode = dataCenterNameCode; 205 | } 206 | public String getDataCenterHomePage() { 207 | return dataCenterHomePage; 208 | } 209 | public void setDataCenterHomePage(String dataCenterHomePage) { 210 | this.dataCenterHomePage = dataCenterHomePage; 211 | } 212 | 213 | @Override 214 | public int hashCode() { 215 | final int prime = 31; 216 | int result = 1; 217 | result = prime * result + ((crawlerCategory == null) ? 0 : crawlerCategory.hashCode()); 218 | result = prime * result + ((crawlerCategoryCode == null) ? 0 : crawlerCategoryCode.hashCode()); 219 | result = prime * result + ((crawlerFamily == null) ? 0 : crawlerFamily.hashCode()); 220 | result = prime * result + ((crawlerFamilyCode == null) ? 0 : crawlerFamilyCode.hashCode()); 221 | result = prime * result + ((crawlerFamilyHomepage == null) ? 0 : crawlerFamilyHomepage.hashCode()); 222 | result = prime * result + ((crawlerFamilyIcon == null) ? 0 : crawlerFamilyIcon.hashCode()); 223 | result = prime * result + ((crawlerFamilyInfoUrl == null) ? 0 : crawlerFamilyInfoUrl.hashCode()); 224 | result = prime * result + ((crawlerFamilyVendor == null) ? 0 : crawlerFamilyVendor.hashCode()); 225 | result = prime * result + ((crawlerFamilyVendorCode == null) ? 0 : crawlerFamilyVendorCode.hashCode()); 226 | result = prime * result + ((crawlerFamilyVendorHomepage == null) ? 0 : crawlerFamilyVendorHomepage.hashCode()); 227 | result = prime * result + ((crawlerLastSeen == null) ? 0 : crawlerLastSeen.hashCode()); 228 | result = prime * result + ((crawlerName == null) ? 0 : crawlerName.hashCode()); 229 | result = prime * result + ((crawlerRespectRobotstxt == null) ? 0 : crawlerRespectRobotstxt.hashCode()); 230 | result = prime * result + ((crawlerVer == null) ? 0 : crawlerVer.hashCode()); 231 | result = prime * result + ((crawlerVerMajor == null) ? 0 : crawlerVerMajor.hashCode()); 232 | result = prime * result + ((dataCenterHomePage == null) ? 0 : dataCenterHomePage.hashCode()); 233 | result = prime * result + ((dataCenterName == null) ? 0 : dataCenterName.hashCode()); 234 | result = prime * result + ((dataCenterNameCode == null) ? 0 : dataCenterNameCode.hashCode()); 235 | result = prime * result + ((ip == null) ? 0 : ip.hashCode()); 236 | result = prime * result + ((ipCity == null) ? 0 : ipCity.hashCode()); 237 | result = prime * result + ((ipClassification == null) ? 0 : ipClassification.hashCode()); 238 | result = prime * result + ((ipClassificationCode == null) ? 0 : ipClassificationCode.hashCode()); 239 | result = prime * result + ((ipCountry == null) ? 0 : ipCountry.hashCode()); 240 | result = prime * result + ((ipCountryCode == null) ? 0 : ipCountryCode.hashCode()); 241 | result = prime * result + ((ipHostname == null) ? 0 : ipHostname.hashCode()); 242 | result = prime * result + ((ipLastSeen == null) ? 0 : ipLastSeen.hashCode()); 243 | result = prime * result + ipVer; 244 | return result; 245 | } 246 | 247 | @Override 248 | public boolean equals(Object obj) { 249 | if (this == obj) 250 | return true; 251 | if (obj == null) 252 | return false; 253 | if (getClass() != obj.getClass()) 254 | return false; 255 | UdgerIpResult other = (UdgerIpResult) obj; 256 | if (crawlerCategory == null) { 257 | if (other.crawlerCategory != null) 258 | return false; 259 | } else if (!crawlerCategory.equals(other.crawlerCategory)) 260 | return false; 261 | if (crawlerCategoryCode == null) { 262 | if (other.crawlerCategoryCode != null) 263 | return false; 264 | } else if (!crawlerCategoryCode.equals(other.crawlerCategoryCode)) 265 | return false; 266 | if (crawlerFamily == null) { 267 | if (other.crawlerFamily != null) 268 | return false; 269 | } else if (!crawlerFamily.equals(other.crawlerFamily)) 270 | return false; 271 | if (crawlerFamilyCode == null) { 272 | if (other.crawlerFamilyCode != null) 273 | return false; 274 | } else if (!crawlerFamilyCode.equals(other.crawlerFamilyCode)) 275 | return false; 276 | if (crawlerFamilyHomepage == null) { 277 | if (other.crawlerFamilyHomepage != null) 278 | return false; 279 | } else if (!crawlerFamilyHomepage.equals(other.crawlerFamilyHomepage)) 280 | return false; 281 | if (crawlerFamilyIcon == null) { 282 | if (other.crawlerFamilyIcon != null) 283 | return false; 284 | } else if (!crawlerFamilyIcon.equals(other.crawlerFamilyIcon)) 285 | return false; 286 | if (crawlerFamilyInfoUrl == null) { 287 | if (other.crawlerFamilyInfoUrl != null) 288 | return false; 289 | } else if (!crawlerFamilyInfoUrl.equals(other.crawlerFamilyInfoUrl)) 290 | return false; 291 | if (crawlerFamilyVendor == null) { 292 | if (other.crawlerFamilyVendor != null) 293 | return false; 294 | } else if (!crawlerFamilyVendor.equals(other.crawlerFamilyVendor)) 295 | return false; 296 | if (crawlerFamilyVendorCode == null) { 297 | if (other.crawlerFamilyVendorCode != null) 298 | return false; 299 | } else if (!crawlerFamilyVendorCode.equals(other.crawlerFamilyVendorCode)) 300 | return false; 301 | if (crawlerFamilyVendorHomepage == null) { 302 | if (other.crawlerFamilyVendorHomepage != null) 303 | return false; 304 | } else if (!crawlerFamilyVendorHomepage.equals(other.crawlerFamilyVendorHomepage)) 305 | return false; 306 | if (crawlerLastSeen == null) { 307 | if (other.crawlerLastSeen != null) 308 | return false; 309 | } else if (!crawlerLastSeen.equals(other.crawlerLastSeen)) 310 | return false; 311 | if (crawlerName == null) { 312 | if (other.crawlerName != null) 313 | return false; 314 | } else if (!crawlerName.equals(other.crawlerName)) 315 | return false; 316 | if (crawlerRespectRobotstxt == null) { 317 | if (other.crawlerRespectRobotstxt != null) 318 | return false; 319 | } else if (!crawlerRespectRobotstxt.equals(other.crawlerRespectRobotstxt)) 320 | return false; 321 | if (crawlerVer == null) { 322 | if (other.crawlerVer != null) 323 | return false; 324 | } else if (!crawlerVer.equals(other.crawlerVer)) 325 | return false; 326 | if (crawlerVerMajor == null) { 327 | if (other.crawlerVerMajor != null) 328 | return false; 329 | } else if (!crawlerVerMajor.equals(other.crawlerVerMajor)) 330 | return false; 331 | if (dataCenterHomePage == null) { 332 | if (other.dataCenterHomePage != null) 333 | return false; 334 | } else if (!dataCenterHomePage.equals(other.dataCenterHomePage)) 335 | return false; 336 | if (dataCenterName == null) { 337 | if (other.dataCenterName != null) 338 | return false; 339 | } else if (!dataCenterName.equals(other.dataCenterName)) 340 | return false; 341 | if (dataCenterNameCode == null) { 342 | if (other.dataCenterNameCode != null) 343 | return false; 344 | } else if (!dataCenterNameCode.equals(other.dataCenterNameCode)) 345 | return false; 346 | if (ip == null) { 347 | if (other.ip != null) 348 | return false; 349 | } else if (!ip.equals(other.ip)) 350 | return false; 351 | if (ipCity == null) { 352 | if (other.ipCity != null) 353 | return false; 354 | } else if (!ipCity.equals(other.ipCity)) 355 | return false; 356 | if (ipClassification == null) { 357 | if (other.ipClassification != null) 358 | return false; 359 | } else if (!ipClassification.equals(other.ipClassification)) 360 | return false; 361 | if (ipClassificationCode == null) { 362 | if (other.ipClassificationCode != null) 363 | return false; 364 | } else if (!ipClassificationCode.equals(other.ipClassificationCode)) 365 | return false; 366 | if (ipCountry == null) { 367 | if (other.ipCountry != null) 368 | return false; 369 | } else if (!ipCountry.equals(other.ipCountry)) 370 | return false; 371 | if (ipCountryCode == null) { 372 | if (other.ipCountryCode != null) 373 | return false; 374 | } else if (!ipCountryCode.equals(other.ipCountryCode)) 375 | return false; 376 | if (ipHostname == null) { 377 | if (other.ipHostname != null) 378 | return false; 379 | } else if (!ipHostname.equals(other.ipHostname)) 380 | return false; 381 | if (ipLastSeen == null) { 382 | if (other.ipLastSeen != null) 383 | return false; 384 | } else if (!ipLastSeen.equals(other.ipLastSeen)) 385 | return false; 386 | if (ipVer != other.ipVer) 387 | return false; 388 | return true; 389 | } 390 | 391 | @Override 392 | public String toString() { 393 | return "UdgerIpResult [" + 394 | "ip=" + ip + 395 | ", ipVer=" + ipVer + 396 | ", ipClassification=" + ipClassification + 397 | ", ipClassificationCode=" + ipClassificationCode + 398 | ", ipLastSeen=" + ipLastSeen + 399 | ", ipHostname=" + ipHostname + 400 | ", ipCountry=" + ipCountry + 401 | ", ipCountryCode=" + ipCountryCode + 402 | ", ipCity=" + ipCity + 403 | ", crawlerName=" + crawlerName + 404 | ", crawlerVer=" + crawlerVer + 405 | ", crawlerVerMajor=" + crawlerVerMajor + 406 | ", crawlerFamily=" + crawlerFamily + 407 | ", crawlerFamilyCode=" + crawlerFamilyCode + 408 | ", crawlerFamilyHomepage=" + crawlerFamilyHomepage + 409 | ", crawlerFamilyVendor=" + crawlerFamilyVendor + 410 | ", crawlerFamilyVendorCode=" + crawlerFamilyVendorCode + 411 | ", crawlerFamilyVendorHomepage=" + crawlerFamilyVendorHomepage + 412 | ", crawlerFamilyIcon=" + crawlerFamilyIcon + 413 | ", crawlerFamilyInfoUrl=" + crawlerFamilyInfoUrl + 414 | ", crawlerLastSeen=" + crawlerLastSeen + 415 | ", crawlerCategory=" + crawlerCategory + 416 | ", crawlerCategoryCode=" + crawlerCategoryCode + 417 | ", crawlerRespectRobotstxt=" + crawlerRespectRobotstxt + 418 | ", dataCenterName=" + dataCenterName + 419 | ", dataCenterNameCode=" + dataCenterNameCode + 420 | ", dataCenterHomePage=" + dataCenterHomePage + 421 | "]"; 422 | } 423 | 424 | } 425 | -------------------------------------------------------------------------------- /src/main/java/org/udger/parser/UdgerParser.java: -------------------------------------------------------------------------------- 1 | /* 2 | UdgerParser - Java agent string parser based on Udger https://udger.com/products/local_parser 3 | 4 | author The Udger.com Team (info@udger.com) 5 | copyright Copyright (c) Udger s.r.o. 6 | license GNU Lesser General Public License 7 | link https://udger.com/products 8 | */ 9 | package org.udger.parser; 10 | 11 | import org.sqlite.SQLiteConfig; 12 | 13 | import java.io.Closeable; 14 | import java.io.File; 15 | import java.io.IOException; 16 | import java.lang.ref.SoftReference; 17 | import java.net.Inet4Address; 18 | import java.net.Inet6Address; 19 | import java.net.InetAddress; 20 | import java.net.UnknownHostException; 21 | import java.sql.*; 22 | import java.util.*; 23 | import java.util.logging.Logger; 24 | import java.util.regex.Matcher; 25 | import java.util.regex.Pattern; 26 | 27 | /** 28 | * Main parser's class handles parser requests for user agent or IP. 29 | */ 30 | public class UdgerParser implements Closeable { 31 | 32 | private static final Logger LOG = Logger.getLogger(UdgerParser.class.getName()); 33 | 34 | private static final String DB_FILENAME = "udgerdb_v3.dat"; 35 | private static final String UDGER_UA_DEV_BRAND_LIST_URL = "https://udger.com/resources/ua-list/devices-brand-detail?brand="; 36 | private static final String ID_CRAWLER = "crawler"; 37 | private static final Pattern PAT_UNPERLIZE = Pattern.compile("^/?(.*?)/si$"); 38 | 39 | /** 40 | * Holds precalculated data for single DB. Intention is to have single ParserDbData associated with multiple UdgerParser(s) 41 | */ 42 | public static class ParserDbData { 43 | 44 | private WordDetector clientWordDetector; 45 | private WordDetector deviceWordDetector; 46 | private WordDetector osWordDetector; 47 | 48 | private List clientRegstringList; 49 | private List osRegstringList; 50 | private List deviceRegstringList; 51 | 52 | private volatile boolean prepared = false; 53 | 54 | private final String dbFileName; 55 | 56 | public ParserDbData(String dbFileName) { 57 | this.dbFileName = dbFileName; 58 | } 59 | 60 | protected void prepare(Connection connection) throws SQLException { 61 | if (!prepared) { 62 | synchronized (this) { 63 | if (!prepared) { 64 | clientRegstringList = prepareRegexpStruct(connection, "udger_client_regex"); 65 | osRegstringList = prepareRegexpStruct(connection, "udger_os_regex"); 66 | deviceRegstringList = prepareRegexpStruct(connection, "udger_deviceclass_regex"); 67 | 68 | clientWordDetector = createWordDetector(connection, "udger_client_regex", "udger_client_regex_words"); 69 | deviceWordDetector = createWordDetector(connection, "udger_deviceclass_regex", "udger_deviceclass_regex_words"); 70 | osWordDetector = createWordDetector(connection, "udger_os_regex", "udger_os_regex_words"); 71 | prepared = true; 72 | } 73 | } 74 | } 75 | } 76 | 77 | } 78 | 79 | private static class ClientInfo { 80 | private Integer clientId; 81 | private Integer classId; 82 | } 83 | 84 | private static class IdRegString { 85 | int id; 86 | int wordId1; 87 | int wordId2; 88 | Pattern pattern; 89 | } 90 | 91 | private static class MatcherWithIdRegString { 92 | private final Matcher matcher; 93 | private final IdRegString irs; 94 | 95 | private MatcherWithIdRegString(Matcher matcher, IdRegString irs) { 96 | this.matcher = matcher; 97 | this.irs = irs; 98 | } 99 | } 100 | 101 | private ParserDbData parserDbData; 102 | 103 | private Connection connection; 104 | 105 | private final Map> regexCache = new HashMap<>(); 106 | 107 | private Map preparedStmtMap = new HashMap<>(); 108 | 109 | private LRUCache cache; 110 | 111 | private boolean osParserEnabled = true; 112 | private boolean deviceParserEnabled = true; 113 | private boolean deviceBrandParserEnabled = true; 114 | private boolean inMemoryEnabled = false; 115 | 116 | /** 117 | * Instantiates a new udger parser with LRU cache with capacity of 10.000 items 118 | * 119 | * @param parserDbData the parser data associated with single DB 120 | */ 121 | public UdgerParser(ParserDbData parserDbData) { 122 | this(parserDbData, 10000); 123 | } 124 | 125 | /** 126 | * Instantiates a new udger parser. 127 | * 128 | * @param parserDbData the parser data associated with single DB 129 | * @param cacheCapacity the LRU cache capacity 130 | */ 131 | public UdgerParser(ParserDbData parserDbData, int cacheCapacity) { 132 | this.parserDbData = parserDbData; 133 | if (cacheCapacity > 0) { 134 | cache = new LRUCache<>(cacheCapacity); 135 | } 136 | } 137 | 138 | /** 139 | * Instantiates a new udger parser with LRU cache with capacity of 10.000 items 140 | * 141 | * @param parserDbData the parser data associated with single DB 142 | * @param inMemoryEnabled the true for in memory mode 143 | * @param cacheCapacity the LRU cache capacity 144 | */ 145 | public UdgerParser(ParserDbData parserDbData, boolean inMemoryEnabled, int cacheCapacity) { 146 | this(parserDbData, cacheCapacity); 147 | this.inMemoryEnabled = inMemoryEnabled; 148 | } 149 | 150 | @Override 151 | public void close() throws IOException { 152 | try { 153 | for (PreparedStatement preparedStmt : preparedStmtMap.values()) { 154 | preparedStmt.close(); 155 | } 156 | preparedStmtMap.clear(); 157 | if (connection != null && !connection.isClosed()) { 158 | connection.close(); 159 | connection = null; 160 | } 161 | if (cache != null) { 162 | cache.clear(); 163 | } 164 | regexCache.clear(); 165 | } catch (SQLException e) { 166 | throw new IOException(e.getMessage()); 167 | } 168 | } 169 | 170 | /** 171 | * Returns true if the sqlite DB connection has not been closed and is still valid. 172 | * 173 | * @param timeoutMillis the timeout millis 174 | * @return true, if is valid 175 | * @throws IOException Signals that an I/O exception has occurred. 176 | */ 177 | public boolean isValid(int timeoutMillis) throws IOException { 178 | try { 179 | return connection == null || connection.isValid(timeoutMillis); 180 | } catch (SQLException e) { 181 | throw new IOException("Failed to validate connection within " + timeoutMillis + " millis.", e); 182 | } 183 | } 184 | 185 | /** 186 | * Parses the user agent string and stores results of parsing in UdgerUaResult. 187 | * If the parser was initialized to use an in memory DB, then the DB is not set to read only. 188 | * This does not matter since the connection is internal to this client, as such there are 189 | * no chance of external modifications. 190 | * 191 | * @param uaString the user agent string 192 | * @return the intance of UdgerUaResult storing results of parsing 193 | * @throws SQLException the SQL exception 194 | */ 195 | public UdgerUaResult parseUa(String uaString) throws SQLException { 196 | 197 | UdgerUaResult ret; 198 | 199 | if (cache != null) { 200 | ret = cache.get(uaString); 201 | if (ret != null) { 202 | return ret; 203 | } 204 | } 205 | 206 | ret = new UdgerUaResult(uaString); 207 | 208 | prepare(); 209 | 210 | ClientInfo clientInfo = clientDetector(uaString, ret); 211 | 212 | if (!"Crawler".equals(ret.getUaClass())) { 213 | if (osParserEnabled) { 214 | osDetector(uaString, ret, clientInfo); 215 | } 216 | 217 | if (deviceParserEnabled) { 218 | deviceDetector(uaString, ret, clientInfo); 219 | } 220 | 221 | if (deviceBrandParserEnabled) { 222 | if (ret.getOsFamilyCode() != null && !ret.getOsFamilyCode().isEmpty()) { 223 | fetchDeviceBrand(uaString, ret); 224 | } 225 | } 226 | } 227 | 228 | if (cache != null) { 229 | cache.put(uaString, ret); 230 | } 231 | 232 | return ret; 233 | } 234 | 235 | /** 236 | * Parses the IP string and stores results of parsing in UdgerIpResult. 237 | * 238 | * @param ipString the IP string 239 | * @return the instance of UdgerIpResult storing results of parsing 240 | * @throws SQLException the SQL exception 241 | * @throws UnknownHostException the unknown host exception 242 | */ 243 | public UdgerIpResult parseIp(String ipString) throws SQLException, UnknownHostException { 244 | 245 | UdgerIpResult ret = new UdgerIpResult(ipString); 246 | 247 | InetAddress addr = InetAddress.getByName(ipString); 248 | Long ipv4int = null; 249 | String normalizedIp = null; 250 | 251 | if (addr instanceof Inet4Address) { 252 | ipv4int = 0L; 253 | for (byte b : addr.getAddress()) { 254 | ipv4int = ipv4int << 8 | (b & 0xFF); 255 | } 256 | normalizedIp = addr.getHostAddress(); 257 | } else if (addr instanceof Inet6Address) { 258 | normalizedIp = addr.getHostAddress().replaceAll("((?:(?:^|:)0+\\b){2,}):?(?!\\S*\\b\\1:0+\\b)(\\S*)", "::$2"); 259 | } 260 | 261 | ret.setIpClassification("Unrecognized"); 262 | ret.setIpClassificationCode("unrecognized"); 263 | 264 | if (normalizedIp != null) { 265 | 266 | prepare(); 267 | 268 | try (ResultSet ipRs = getFirstRow(UdgerSqlQuery.SQL_IP, normalizedIp)) { 269 | if (ipRs != null && ipRs.next()) { 270 | fetchUdgerIp(ipRs, ret); 271 | if (!ID_CRAWLER.equals(ret.getIpClassificationCode())) { 272 | ret.setCrawlerFamilyInfoUrl(""); 273 | } 274 | } 275 | } 276 | 277 | if (ipv4int != null) { 278 | ret.setIpVer(4); 279 | ResultSet dataCenterRs = getFirstRow(UdgerSqlQuery.SQL_DATACENTER, ipv4int, ipv4int); 280 | fetchDataCenterAndCloseRs(dataCenterRs, ret); 281 | } else { 282 | ret.setIpVer(6); 283 | int[] ipArray = ip6ToArray((Inet6Address) addr); 284 | ResultSet dataCenterRs = getFirstRow(UdgerSqlQuery.SQL_DATACENTER_RANGE6, 285 | ipArray[0], ipArray[0], 286 | ipArray[1], ipArray[1], 287 | ipArray[2], ipArray[2], 288 | ipArray[3], ipArray[3], 289 | ipArray[4], ipArray[4], 290 | ipArray[5], ipArray[5], 291 | ipArray[6], ipArray[6], 292 | ipArray[7], ipArray[7] 293 | ); 294 | fetchDataCenterAndCloseRs(dataCenterRs, ret); 295 | } 296 | } 297 | 298 | return ret; 299 | } 300 | 301 | private void fetchDataCenterAndCloseRs(ResultSet dataCenterRs, UdgerIpResult ret) throws SQLException { 302 | if (dataCenterRs != null) { 303 | try { 304 | if (dataCenterRs.next()) { 305 | fetchDataCenter(dataCenterRs, ret); 306 | } 307 | } finally { 308 | dataCenterRs.close(); 309 | } 310 | } 311 | } 312 | 313 | /** 314 | * Checks if is OS parser enabled. OS parser is enabled by default 315 | * 316 | * @return true, if is OS parser enabled 317 | */ 318 | public boolean isOsParserEnabled() { 319 | return osParserEnabled; 320 | } 321 | 322 | /** 323 | * Enable/disable the OS parser. OS parser is enabled by default. If enabled following fields 324 | * of UdgerUaResult are processed by the OS parser: 325 | *
    326 | *
  • osFamily, osFamilyCode, OS, osCode, osHomePage, osIcon, osIconBig
    • 327 | *
  • osFamilyVendor, osFamilyVendorCode, osFamilyVedorHomepage, osInfoUrl
    • 328 | *
329 | *

330 | * If the OSs fields are not necessary then disabling this feature can increase 331 | * the parser's performance. 332 | * 333 | * @param osParserEnabled the true if os parser is to be enabled 334 | */ 335 | public void setOsParserEnabled(boolean osParserEnabled) { 336 | this.osParserEnabled = osParserEnabled; 337 | } 338 | 339 | /** 340 | * Checks if is device parser enabled. Device parser is enabled by default 341 | * 342 | * @return true, if device parser is enabled 343 | */ 344 | public boolean isDeviceParserEnabled() { 345 | return deviceParserEnabled; 346 | } 347 | 348 | /** 349 | * Enable/disable the device parser. Device parser is enabled by default. If enabled following fields 350 | * of UdgerUaResult are filled by the device parser: 351 | *

    352 | *
  • deviceClass, deviceClassCode, deviceClassIcon
    • 353 | *
  • deviceClassIconBig, deviceClassInfoUrl
    • 354 | *
355 | *

356 | * If the DEVICEs fields are not necessary then disabling this feature can increase 357 | * the parser's performance. 358 | * 359 | * @param deviceParserEnabled the true if device parser is to be enabled 360 | */ 361 | public void setDeviceParserEnabled(boolean deviceParserEnabled) { 362 | this.deviceParserEnabled = deviceParserEnabled; 363 | } 364 | 365 | /** 366 | * Checks if is device brand parser enabled. Device brand parser is enabled by default. 367 | * 368 | * @return true, if device brand parser is enabled 369 | */ 370 | public boolean isDeviceBrandParserEnabled() { 371 | return deviceBrandParserEnabled; 372 | } 373 | 374 | /** 375 | * Enable/disable the device brand parser. Device brand parser is enabled by default. If enabled following fields 376 | * of UdgerUaResult are filled by the device brand parser: 377 | *

    378 | *
  • deviceMarketname, deviceBrand, deviceBrandCode, deviceBrandHomepage
    • 379 | *
  • deviceBrandIcon, deviceBrandIconBig, deviceBrandInfoUrl
    • 380 | *
381 | *

382 | * If the BRANDs fields are not necessary then disabling this feature can increase 383 | * the parser's performance. 384 | * 385 | * @param deviceBrandParserEnabled the true if device brand parser is to be enabled 386 | */ 387 | public void setDeviceBrandParserEnabled(boolean deviceBrandParserEnabled) { 388 | this.deviceBrandParserEnabled = deviceBrandParserEnabled; 389 | } 390 | 391 | private static WordDetector createWordDetector(Connection connection, String regexTableName, String wordTableName) throws SQLException { 392 | 393 | Set usedWords = new HashSet<>(); 394 | 395 | addUsedWords(usedWords, connection, regexTableName, "word_id"); 396 | addUsedWords(usedWords, connection, regexTableName, "word2_id"); 397 | 398 | WordDetector result = new WordDetector(); 399 | 400 | try (final Statement statement = connection.createStatement(); 401 | final ResultSet rs = statement.executeQuery("SELECT * FROM " + wordTableName)) { 402 | if (rs != null) { 403 | while (rs.next()) { 404 | int id = rs.getInt("id"); 405 | if (usedWords.contains(id)) { 406 | String word = rs.getString("word").toLowerCase(); 407 | result.addWord(id, word); 408 | } 409 | } 410 | } 411 | } 412 | return result; 413 | } 414 | 415 | private static void addUsedWords(Set usedWords, Connection connection, String regexTableName, String wordIdColumn) throws SQLException { 416 | try (Statement statement = connection.createStatement(); 417 | ResultSet rs = statement.executeQuery("SELECT " + wordIdColumn + " FROM " + regexTableName)) { 418 | if (rs != null) { 419 | while (rs.next()) { 420 | usedWords.add(rs.getInt(wordIdColumn)); 421 | } 422 | } 423 | } 424 | } 425 | 426 | private MatcherWithIdRegString findMatcherIdRegString(String uaString, Set foundClientWords, List list) { 427 | for (IdRegString irs : list) { 428 | if ((irs.wordId1 == 0 || foundClientWords.contains(irs.wordId1)) && 429 | (irs.wordId2 == 0 || foundClientWords.contains(irs.wordId2))) { 430 | Matcher matcher = irs.pattern.matcher(uaString); 431 | if (matcher.find()) 432 | return new MatcherWithIdRegString(matcher, irs); 433 | } 434 | } 435 | return null; 436 | } 437 | 438 | private static List prepareRegexpStruct(Connection connection, String regexpTableName) throws SQLException { 439 | List ret = new ArrayList<>(); 440 | try (Statement statement = connection.createStatement(); 441 | ResultSet rs = statement.executeQuery("SELECT rowid, regstring, word_id, word2_id FROM " + regexpTableName + " ORDER BY sequence")) { 442 | if (rs != null) { 443 | while (rs.next()) { 444 | IdRegString irs = new IdRegString(); 445 | irs.id = rs.getInt("rowid"); 446 | irs.wordId1 = rs.getInt("word_id"); 447 | irs.wordId2 = rs.getInt("word2_id"); 448 | String regex = rs.getString("regstring"); 449 | Matcher m = PAT_UNPERLIZE.matcher(regex); 450 | if (m.matches()) { 451 | regex = m.group(1); 452 | } 453 | irs.pattern = Pattern.compile(regex, Pattern.CASE_INSENSITIVE | Pattern.DOTALL); 454 | ret.add(irs); 455 | } 456 | } 457 | } 458 | return ret; 459 | } 460 | 461 | private ClientInfo clientDetector(String uaString, UdgerUaResult ret) throws SQLException { 462 | ClientInfo clientInfo = new ClientInfo(); 463 | try (ResultSet userAgentRs1 = getFirstRow(UdgerSqlQuery.SQL_CRAWLER, uaString)) { 464 | if (userAgentRs1 != null && userAgentRs1.next()) { 465 | fetchUserAgent(userAgentRs1, ret); 466 | clientInfo.classId = 99; 467 | clientInfo.clientId = -1; 468 | } else { 469 | MatcherWithIdRegString mwirs = findMatcherIdRegString(uaString, parserDbData.clientWordDetector.findWords(uaString), parserDbData.clientRegstringList); 470 | if (mwirs != null) { 471 | try (ResultSet userAgentRs2 = getFirstRow(UdgerSqlQuery.SQL_CLIENT, mwirs.irs.id)) { 472 | if (userAgentRs2 != null && userAgentRs2.next()) { 473 | fetchUserAgent(userAgentRs2, ret); 474 | clientInfo.classId = ret.getClassId(); 475 | clientInfo.clientId = ret.getClientId(); 476 | patchVersions(mwirs.matcher, ret); 477 | } 478 | } 479 | } else { 480 | ret.setUaClass("Unrecognized"); 481 | ret.setUaClassCode("unrecognized"); 482 | } 483 | } 484 | } 485 | return clientInfo; 486 | } 487 | 488 | private void osDetector(String uaString, UdgerUaResult ret, ClientInfo clientInfo) throws SQLException { 489 | MatcherWithIdRegString mwirs = findMatcherIdRegString(uaString, parserDbData.osWordDetector.findWords(uaString), parserDbData.osRegstringList); 490 | if (mwirs != null) { 491 | try (ResultSet opSysRs = getFirstRow(UdgerSqlQuery.SQL_OS, mwirs.irs.id)) { 492 | if (opSysRs != null && opSysRs.next()) { 493 | fetchOperatingSystem(opSysRs, ret); 494 | } 495 | } 496 | } else { 497 | if (clientInfo.clientId != null && clientInfo.clientId != 0) { 498 | try (ResultSet opSysRs = getFirstRow(UdgerSqlQuery.SQL_CLIENT_OS, clientInfo.clientId.toString())) { 499 | if (opSysRs != null && opSysRs.next()) { 500 | fetchOperatingSystem(opSysRs, ret); 501 | } 502 | } 503 | } 504 | } 505 | } 506 | 507 | private void deviceDetector(String uaString, UdgerUaResult ret, ClientInfo clientInfo) throws SQLException { 508 | MatcherWithIdRegString mwirs = findMatcherIdRegString(uaString, parserDbData.deviceWordDetector.findWords(uaString), parserDbData.deviceRegstringList); 509 | if (mwirs != null) { 510 | try (ResultSet devRs = getFirstRow(UdgerSqlQuery.SQL_DEVICE, mwirs.irs.id)) { 511 | if (devRs != null && devRs.next()) { 512 | fetchDevice(devRs, ret); 513 | } 514 | } 515 | } else { 516 | if (clientInfo.classId != null && clientInfo.classId != -1) { 517 | try (ResultSet devRs = getFirstRow(UdgerSqlQuery.SQL_CLIENT_CLASS, clientInfo.classId.toString())) { 518 | if (devRs != null && devRs.next()) { 519 | fetchDevice(devRs, ret); 520 | } 521 | } 522 | } 523 | } 524 | } 525 | 526 | private void fetchDeviceBrand(String uaString, UdgerUaResult ret) throws SQLException { 527 | PreparedStatement preparedStatement = preparedStmtMap.get(UdgerSqlQuery.SQL_DEVICE_REGEX); 528 | if (preparedStatement == null) { 529 | preparedStatement = connection.prepareStatement(UdgerSqlQuery.SQL_DEVICE_REGEX); 530 | preparedStmtMap.put(UdgerSqlQuery.SQL_DEVICE_REGEX, preparedStatement); 531 | } 532 | preparedStatement.setObject(1, ret.getOsFamilyCode()); 533 | preparedStatement.setObject(2, ret.getOsCode()); 534 | try (ResultSet devRegexRs = preparedStatement.executeQuery()) { 535 | if (devRegexRs != null) { 536 | while (devRegexRs.next()) { 537 | String devId = devRegexRs.getString("id"); 538 | String regex = devRegexRs.getString("regstring"); 539 | if (devId != null && regex != null) { 540 | Pattern patRegex = getRegexFromCache(regex); 541 | Matcher matcher = patRegex.matcher(uaString); 542 | if (matcher.find()) { 543 | try (ResultSet devNameListRs = getFirstRow(UdgerSqlQuery.SQL_DEVICE_NAME_LIST, devId, matcher.group(1))) { 544 | if (devNameListRs != null && devNameListRs.next()) { 545 | ret.setDeviceMarketname(devNameListRs.getString("marketname")); 546 | ret.setDeviceBrand(devNameListRs.getString("brand")); 547 | ret.setDeviceBrandCode(devNameListRs.getString("brand_code")); 548 | ret.setDeviceBrandHomepage(devNameListRs.getString("brand_url")); 549 | ret.setDeviceBrandIcon(devNameListRs.getString("icon")); 550 | ret.setDeviceBrandIconBig(devNameListRs.getString("icon_big")); 551 | ret.setDeviceBrandInfoUrl(UDGER_UA_DEV_BRAND_LIST_URL + devNameListRs.getString("brand_code")); 552 | break; 553 | } 554 | } 555 | } 556 | } 557 | } 558 | } 559 | } 560 | } 561 | 562 | 563 | private int[] ip6ToArray(Inet6Address addr) { 564 | int ret[] = new int[8]; 565 | byte[] bytes = addr.getAddress(); 566 | for (int i = 0; i < 8; i++) { 567 | ret[i] = ((bytes[i * 2] << 8) & 0xff00) | (bytes[i * 2 + 1] & 0xff); 568 | } 569 | return ret; 570 | } 571 | 572 | private void prepare() throws SQLException { 573 | connect(); 574 | parserDbData.prepare(connection); 575 | } 576 | 577 | private void connect() throws SQLException { 578 | if (connection == null) { 579 | SQLiteConfig config = new SQLiteConfig(); 580 | config.setReadOnly(true); 581 | if (inMemoryEnabled) { 582 | // we cannot use read only for in memory DB since we need to populate this DB from the file. 583 | connection = DriverManager.getConnection("jdbc:sqlite::memory:"); 584 | File dbfile = new File(parserDbData.dbFileName); 585 | try (Statement statement = connection.createStatement()) { 586 | statement.executeUpdate("restore from " + dbfile.getPath()); 587 | } catch (Exception e) { 588 | LOG.warning("Error re-constructing in memory data base from Db file " + dbfile); 589 | } 590 | } else { 591 | connection = DriverManager.getConnection("jdbc:sqlite:" + parserDbData.dbFileName, config.toProperties()); 592 | } 593 | } 594 | } 595 | 596 | private Pattern getRegexFromCache(String regex) { 597 | SoftReference patRegex = regexCache.get(regex); 598 | if (patRegex == null || patRegex.get() == null) { 599 | Matcher m = PAT_UNPERLIZE.matcher(regex); 600 | if (m.matches()) { 601 | regex = m.group(1); 602 | } 603 | patRegex = new SoftReference<>(Pattern.compile(regex, Pattern.CASE_INSENSITIVE | Pattern.DOTALL)); 604 | regexCache.put(regex, patRegex); 605 | } 606 | return patRegex.get(); 607 | } 608 | 609 | private ResultSet getFirstRow(String query, Object... params) throws SQLException { 610 | PreparedStatement preparedStatement = preparedStmtMap.get(query); 611 | if (preparedStatement == null) { 612 | preparedStatement = connection.prepareStatement(query); 613 | preparedStmtMap.put(query, preparedStatement); 614 | } 615 | for (int i = 0; i < params.length; i++) { 616 | preparedStatement.setObject(i + 1, params[i]); 617 | } 618 | preparedStatement.setMaxRows(1); 619 | return preparedStatement.executeQuery(); 620 | } 621 | 622 | 623 | private void fetchUserAgent(ResultSet rs, UdgerUaResult ret) throws SQLException { 624 | ret.setClassId(rs.getInt("class_id")); 625 | ret.setClientId(rs.getInt("client_id")); 626 | ret.setCrawlerCategory(nvl(rs.getString("crawler_category"))); 627 | ret.setCrawlerCategoryCode(nvl(rs.getString("crawler_category_code"))); 628 | ret.setCrawlerLastSeen(nvl(rs.getString("crawler_last_seen"))); 629 | ret.setCrawlerRespectRobotstxt(nvl(rs.getString("crawler_respect_robotstxt"))); 630 | ret.setUa(nvl(rs.getString("ua"))); 631 | ret.setUaClass(nvl(rs.getString("ua_class"))); 632 | ret.setUaClassCode(nvl(rs.getString("ua_class_code"))); 633 | ret.setUaEngine(nvl(rs.getString("ua_engine"))); 634 | ret.setUaFamily(nvl(rs.getString("ua_family"))); 635 | ret.setUaFamilyCode(nvl(rs.getString("ua_family_code"))); 636 | ret.setUaFamilyHomepage(nvl(rs.getString("ua_family_homepage"))); 637 | ret.setUaFamilyIcon(nvl(rs.getString("ua_family_icon"))); 638 | ret.setUaFamilyIconBig(nvl(rs.getString("ua_family_icon_big"))); 639 | ret.setUaFamilyInfoUrl(nvl(rs.getString("ua_family_info_url"))); 640 | ret.setUaFamilyVendor(nvl(rs.getString("ua_family_vendor"))); 641 | ret.setUaFamilyVendorCode(nvl(rs.getString("ua_family_vendor_code"))); 642 | ret.setUaFamilyVendorHomepage(nvl(rs.getString("ua_family_vendor_homepage"))); 643 | ret.setUaUptodateCurrentVersion(nvl(rs.getString("ua_uptodate_current_version"))); 644 | ret.setUaVersion(nvl(rs.getString("ua_version"))); 645 | ret.setUaVersionMajor(nvl(rs.getString("ua_version_major"))); 646 | } 647 | 648 | private void fetchOperatingSystem(ResultSet rs, UdgerUaResult ret) throws SQLException { 649 | ret.setOsFamily(nvl(rs.getString("os_family"))); 650 | ret.setOs(nvl(rs.getString("os"))); 651 | ret.setOsCode(nvl(rs.getString("os_code"))); 652 | ret.setOsFamilyCode(nvl(rs.getString("os_family_code"))); 653 | ret.setOsFamilyVendorHomepage(nvl(rs.getString("os_family_vendor_homepage"))); 654 | ret.setOsFamilyVendor(nvl(rs.getString("os_family_vendor"))); 655 | ret.setOsFamilyVendorCode(nvl(rs.getString("os_family_vendor_code"))); 656 | ret.setOsHomePage(nvl(rs.getString("os_home_page"))); 657 | ret.setOsIcon(nvl(rs.getString("os_icon"))); 658 | ret.setOsIconBig(nvl(rs.getString("os_icon_big"))); 659 | ret.setOsInfoUrl(nvl(rs.getString("os_info_url"))); 660 | } 661 | 662 | private void fetchDevice(ResultSet rs, UdgerUaResult ret) throws SQLException { 663 | ret.setDeviceClass(nvl(rs.getString("device_class"))); 664 | ret.setDeviceClassCode(nvl(rs.getString("device_class_code"))); 665 | ret.setDeviceClassIcon(nvl(rs.getString("device_class_icon"))); 666 | ret.setDeviceClassIconBig(nvl(rs.getString("device_class_icon_big"))); 667 | ret.setDeviceClassInfoUrl(nvl(rs.getString("device_class_info_url"))); 668 | } 669 | 670 | private void patchVersions(Matcher lastPatternMatcher, UdgerUaResult ret) { 671 | if (lastPatternMatcher != null) { 672 | String version = ""; 673 | if (lastPatternMatcher.groupCount() >= 1) { 674 | version = lastPatternMatcher.group(1); 675 | if (version == null) { 676 | version = ""; 677 | } 678 | } 679 | ret.setUaVersion(version); 680 | String versionSegments[] = version.split("\\."); 681 | if (versionSegments.length > 0) { 682 | ret.setUaVersionMajor(version.split("\\.")[0]); 683 | } else { 684 | ret.setUaVersionMajor(""); 685 | } 686 | ret.setUa((ret.getUa() != null ? ret.getUa() : "") + " " + version); 687 | } else { 688 | ret.setUaVersion(""); 689 | ret.setUaVersionMajor(""); 690 | } 691 | } 692 | 693 | private void fetchUdgerIp(ResultSet rs, UdgerIpResult ret) throws SQLException { 694 | ret.setCrawlerCategory(nvl(rs.getString("crawler_category"))); 695 | ret.setCrawlerCategoryCode(nvl(rs.getString("crawler_category_code"))); 696 | ret.setCrawlerFamily(nvl(rs.getString("crawler_family"))); 697 | ret.setCrawlerFamilyCode(nvl(rs.getString("crawler_family_code"))); 698 | ret.setCrawlerFamilyHomepage(nvl(rs.getString("crawler_family_homepage"))); 699 | ret.setCrawlerFamilyIcon(nvl(rs.getString("crawler_family_icon"))); 700 | ret.setCrawlerFamilyInfoUrl(nvl(rs.getString("crawler_family_info_url"))); 701 | ret.setCrawlerFamilyVendor(nvl(rs.getString("crawler_family_vendor"))); 702 | ret.setCrawlerFamilyVendorCode(nvl(rs.getString("crawler_family_vendor_code"))); 703 | ret.setCrawlerFamilyVendorHomepage(nvl(rs.getString("crawler_family_vendor_homepage"))); 704 | ret.setCrawlerLastSeen(nvl(rs.getString("crawler_last_seen"))); 705 | ret.setCrawlerName(nvl(rs.getString("crawler_name"))); 706 | ret.setCrawlerRespectRobotstxt(nvl(rs.getString("crawler_respect_robotstxt"))); 707 | ret.setCrawlerVer(nvl(rs.getString("crawler_ver"))); 708 | ret.setCrawlerVerMajor(nvl(rs.getString("crawler_ver_major"))); 709 | ret.setIpCity(nvl(rs.getString("ip_city"))); 710 | ret.setIpClassification(nvl(rs.getString("ip_classification"))); 711 | ret.setIpClassificationCode(nvl(rs.getString("ip_classification_code"))); 712 | ret.setIpCountry(nvl(rs.getString("ip_country"))); 713 | ret.setIpCountryCode(nvl(rs.getString("ip_country_code"))); 714 | ret.setIpHostname(nvl(rs.getString("ip_hostname"))); 715 | ret.setIpLastSeen(nvl(rs.getString("ip_last_seen"))); 716 | } 717 | 718 | private String nvl(String v) { 719 | return v != null ? v : ""; 720 | } 721 | 722 | private void fetchDataCenter(ResultSet rs, UdgerIpResult ret) throws SQLException { 723 | ret.setDataCenterHomePage(nvl(rs.getString("datacenter_homepage"))); 724 | ret.setDataCenterName(nvl(rs.getString("datacenter_name"))); 725 | ret.setDataCenterNameCode(nvl(rs.getString("datacenter_name_code"))); 726 | } 727 | 728 | } 729 | -------------------------------------------------------------------------------- /src/main/java/org/udger/parser/UdgerSqlQuery.java: -------------------------------------------------------------------------------- 1 | /* 2 | Udger-update - Data updater for udger local and cloud parser 3 | 4 | author The Udger.com Team (info@udger.com) 5 | copyright Copyright (c) Udger s.r.o. 6 | license GNU Lesser General Public License 7 | link https://udger.com/products 8 | */ 9 | package org.udger.parser; 10 | 11 | public class UdgerSqlQuery { 12 | 13 | public static final String SQL_CRAWLER = 14 | "SELECT " + 15 | "NULL AS client_id, " + 16 | "NULL AS class_id, " + 17 | "'Crawler' AS ua_class, " + 18 | "'crawler' AS ua_class_code, " + 19 | "name AS ua, " + 20 | "NULL AS ua_engine, " + 21 | "ver AS ua_version, " + 22 | "ver_major AS ua_version_major, " + 23 | "last_seen AS crawler_last_seen, " + 24 | "respect_robotstxt AS crawler_respect_robotstxt, " + 25 | "crawler_classification AS crawler_category, " + 26 | "crawler_classification_code AS crawler_category_code, " + 27 | "NULL AS ua_uptodate_current_version, " + 28 | "family AS ua_family, " + 29 | "family_code AS ua_family_code, " + 30 | "family_homepage AS ua_family_homepage, " + 31 | "family_icon AS ua_family_icon, " + 32 | "NULL AS ua_family_icon_big, " + 33 | "vendor AS ua_family_vendor, " + 34 | "vendor_code AS ua_family_vendor_code, " + 35 | "vendor_homepage AS ua_family_vendor_homepage, " + 36 | "'https://udger.com/resources/ua-list/bot-detail?bot=' || REPLACE(family, ' ', '%20') || '#id' || udger_crawler_list.id AS ua_family_info_url " + 37 | "FROM " + 38 | "udger_crawler_list " + 39 | "LEFT JOIN " + 40 | "udger_crawler_class ON udger_crawler_class.id = udger_crawler_list.class_id " + 41 | "WHERE " + 42 | "ua_string = ?"; 43 | 44 | public static final String SQL_CLIENT = 45 | "SELECT " + 46 | "ur.rowid, " + 47 | "client_id AS client_id, " + 48 | "class_id AS class_id, " + 49 | "client_classification AS ua_class, " + 50 | "client_classification_code AS ua_class_code, " + 51 | "name AS ua, " + 52 | "engine AS ua_engine, " + 53 | "NULL AS ua_version, " + 54 | "NULL AS ua_version_major, " + 55 | "NULL AS crawler_last_seen, " + 56 | "NULL AS crawler_respect_robotstxt, " + 57 | "NULL AS crawler_category, " + 58 | "NULL AS crawler_category_code, " + 59 | "uptodate_current_version AS ua_uptodate_current_version, " + 60 | "name AS ua_family, " + 61 | "name_code AS ua_family_code, " + 62 | "homepage AS ua_family_homepage, " + 63 | "icon AS ua_family_icon, " + 64 | "icon_big AS ua_family_icon_big, " + 65 | "vendor AS ua_family_vendor, " + 66 | "vendor_code AS ua_family_vendor_code, " + 67 | "vendor_homepage AS ua_family_vendor_homepage, " + 68 | "'https://udger.com/resources/ua-list/browser-detail?browser=' || REPLACE(name, ' ', '%20') AS ua_family_info_url " + 69 | "FROM " + 70 | "udger_client_regex ur " + 71 | "JOIN " + 72 | "udger_client_list ON udger_client_list.id = ur.client_id " + 73 | "JOIN " + 74 | "udger_client_class ON udger_client_class.id = udger_client_list.class_id " + 75 | "WHERE " + 76 | "ur.rowid=?"; 77 | 78 | private static final String OS_COLUMNS = 79 | "family AS os_family, " + 80 | "family_code AS os_family_code, " + 81 | "name AS os, " + 82 | "name_code AS os_code, " + 83 | "homepage AS os_home_page, " + 84 | "icon AS os_icon, " + 85 | "icon_big AS os_icon_big, " + 86 | "vendor AS os_family_vendor, " + 87 | "vendor_code AS os_family_vendor_code, " + 88 | "vendor_homepage AS os_family_vendor_homepage, " + 89 | "'https://udger.com/resources/ua-list/os-detail?os=' || REPLACE(name, ' ', '%20') AS os_info_url "; 90 | 91 | public static final String SQL_OS = 92 | "SELECT " + 93 | "ur.rowid, " + 94 | OS_COLUMNS + 95 | "FROM " + 96 | "udger_os_regex ur " + 97 | "JOIN " + 98 | "udger_os_list ON udger_os_list.id = ur.os_id " + 99 | "WHERE " + 100 | "ur.rowid=?"; 101 | 102 | public static final String SQL_CLIENT_OS = 103 | "SELECT " + 104 | OS_COLUMNS + 105 | "FROM " + 106 | "udger_client_os_relation " + 107 | "JOIN " + 108 | "udger_os_list ON udger_os_list.id = udger_client_os_relation.os_id " + 109 | "WHERE " + 110 | "client_id = ?"; 111 | 112 | private static final String DEVICE_COLUMNS = 113 | "name AS device_class, " + 114 | "name_code AS device_class_code, " + 115 | "icon AS device_class_icon, " + 116 | "icon_big AS device_class_icon_big, " + 117 | "'https://udger.com/resources/ua-list/device-detail?device=' || REPLACE(name, ' ', '%20') AS device_class_info_url "; 118 | 119 | public static final String SQL_DEVICE = 120 | "SELECT " + 121 | "ur.rowid, " + 122 | DEVICE_COLUMNS + 123 | "FROM " + 124 | "udger_deviceclass_regex ur " + 125 | "JOIN " + 126 | "udger_deviceclass_list ON udger_deviceclass_list.id = ur.deviceclass_id " + 127 | "WHERE " + 128 | "ur.rowid=?" 129 | ; 130 | 131 | public static final String SQL_CLIENT_CLASS = 132 | "SELECT " + 133 | DEVICE_COLUMNS + 134 | "FROM " + 135 | "udger_deviceclass_list " + 136 | "JOIN " + 137 | "udger_client_class ON udger_client_class.deviceclass_id = udger_deviceclass_list.id " + 138 | "WHERE " + 139 | "udger_client_class.id = ?"; 140 | 141 | private static final String IP_COLUMNS = 142 | "ip_classification AS ip_classification, " + 143 | "ip_classification_code AS ip_classification_code, " + 144 | "ip_last_seen AS ip_last_seen, " + 145 | "ip_hostname AS ip_hostname, " + 146 | "ip_country AS ip_country, " + 147 | "ip_country_code AS ip_country_code, " + 148 | "ip_city AS ip_city, " + 149 | "name AS crawler_name, " + 150 | "ver AS crawler_ver, " + 151 | "ver_major AS crawler_ver_major, " + 152 | "family AS crawler_family, " + 153 | "family_code AS crawler_family_code, " + 154 | "family_homepage AS crawler_family_homepage, " + 155 | "vendor AS crawler_family_vendor, " + 156 | "vendor_code AS crawler_family_vendor_code, " + 157 | "vendor_homepage AS crawler_family_vendor_homepage, " + 158 | "family_icon AS crawler_family_icon, " + 159 | "'https://udger.com/resources/ua-list/bot-detail?bot=' || REPLACE(family, ' ', '%20') || '#id' || udger_crawler_list.id AS crawler_family_info_url, " + 160 | "last_seen AS crawler_last_seen, " + 161 | "crawler_classification AS crawler_category, " + 162 | "crawler_classification_code AS crawler_category_code, " + 163 | "respect_robotstxt AS crawler_respect_robotstxt "; 164 | 165 | public static final String SQL_IP = 166 | "SELECT " + 167 | IP_COLUMNS + 168 | "FROM " + 169 | "udger_ip_list " + 170 | "JOIN " + 171 | "udger_ip_class ON udger_ip_class.id=udger_ip_list.class_id " + 172 | "LEFT JOIN " + 173 | "udger_crawler_list ON udger_crawler_list.id=udger_ip_list.crawler_id " + 174 | "LEFT JOIN " + 175 | "udger_crawler_class ON udger_crawler_class.id=udger_crawler_list.class_id " + 176 | "WHERE " + 177 | "ip = ? " + 178 | "ORDER BY " + 179 | "sequence"; 180 | 181 | private static final String DATACENTER_COLUMNS = 182 | "name AS datacenter_name, " + 183 | "name_code AS datacenter_name_code, " + 184 | "homepage AS datacenter_homepage "; 185 | 186 | public static final String SQL_DATACENTER = 187 | "SELECT " + 188 | DATACENTER_COLUMNS + 189 | "FROM " + 190 | "udger_datacenter_range " + 191 | "JOIN " + 192 | "udger_datacenter_list ON udger_datacenter_range.datacenter_id = udger_datacenter_list.id " + 193 | "WHERE " + 194 | "iplong_from <= ? AND iplong_to >= ?"; 195 | 196 | public static final String SQL_DATACENTER_RANGE6 = 197 | "SELECT " + 198 | DATACENTER_COLUMNS + 199 | "FROM " + 200 | "udger_datacenter_range6 " + 201 | "JOIN " + 202 | "udger_datacenter_list ON udger_datacenter_range6.datacenter_id=udger_datacenter_list.id " + 203 | "WHERE " + 204 | "iplong_from0 <= ? AND iplong_to0 >= ? AND " + 205 | "iplong_from1 <= ? AND iplong_to1 >= ? AND " + 206 | "iplong_from2 <= ? AND iplong_to2 >= ? AND " + 207 | "iplong_from3 <= ? AND iplong_to3 >= ? AND " + 208 | "iplong_from4 <= ? AND iplong_to4 >= ? AND " + 209 | "iplong_from5 <= ? AND iplong_to5 >= ? AND " + 210 | "iplong_from6 <= ? AND iplong_to6 >= ? AND " + 211 | "iplong_from7 <= ? AND iplong_to7 >=?"; 212 | 213 | public static final String SQL_DEVICE_REGEX = 214 | "SELECT " + 215 | "id," + 216 | "regstring " + 217 | "FROM " + 218 | "udger_devicename_regex " + 219 | "WHERE " + 220 | "os_family_code=? AND (os_code='-all-' OR os_code=?) " + 221 | "ORDER BY sequence"; 222 | 223 | public static final String SQL_DEVICE_NAME_LIST = 224 | "SELECT " + 225 | "marketname," + 226 | "brand_code," + 227 | "brand," + 228 | "brand_url," + 229 | "icon," + 230 | "icon_big " + 231 | "FROM " + 232 | "udger_devicename_list " + 233 | "JOIN " + 234 | "udger_devicename_brand ON udger_devicename_brand.id=udger_devicename_list.brand_id " + 235 | "WHERE " + 236 | "regex_id = ? AND code = ?"; 237 | 238 | } 239 | -------------------------------------------------------------------------------- /src/main/java/org/udger/parser/UdgerUaResult.java: -------------------------------------------------------------------------------- 1 | /* 2 | Udger-update - Data updater for udger local and cloud parser 3 | 4 | author The Udger.com Team (info@udger.com) 5 | copyright Copyright (c) Udger s.r.o. 6 | license GNU Lesser General Public License 7 | link https://udger.com/products 8 | */ 9 | package org.udger.parser; 10 | 11 | import java.io.Serializable; 12 | 13 | public class UdgerUaResult implements Serializable { 14 | 15 | private static final long serialVersionUID = 1L; 16 | 17 | // UA 18 | private final String uaString; 19 | private Integer clientId; 20 | private Integer classId; 21 | private String uaClass = ""; 22 | private String uaClassCode = ""; 23 | private String ua = ""; 24 | private String uaEngine = ""; 25 | private String uaVersion = ""; 26 | private String uaVersionMajor = ""; 27 | private String crawlerLastSeen = ""; 28 | private String crawlerRespectRobotstxt = ""; 29 | private String crawlerCategory = ""; 30 | private String crawlerCategoryCode = ""; 31 | private String uaUptodateCurrentVersion = ""; 32 | private String uaFamily = ""; 33 | private String uaFamilyCode = ""; 34 | private String uaFamilyHomepage = ""; 35 | private String uaFamilyIcon = ""; 36 | private String uaFamilyIconBig = ""; 37 | private String uaFamilyVendor = ""; 38 | private String uaFamilyVendorCode = ""; 39 | private String uaFamilyVendorHomepage = ""; 40 | private String uaFamilyInfoUrl = ""; 41 | 42 | // OS 43 | private String osFamily = ""; 44 | private String osFamilyCode = ""; 45 | private String os = ""; 46 | private String osCode = ""; 47 | private String osHomePage = ""; 48 | private String osIcon = ""; 49 | private String osIconBig = ""; 50 | private String osFamilyVendor = ""; 51 | private String osFamilyVendorCode = ""; 52 | private String osFamilyVendorHomepage = ""; 53 | private String osInfoUrl = ""; 54 | 55 | // DEVICE 56 | private String deviceClass = ""; 57 | private String deviceClassCode = ""; 58 | private String deviceClassIcon = ""; 59 | private String deviceClassIconBig = ""; 60 | private String deviceClassInfoUrl = ""; 61 | 62 | private String deviceMarketname = ""; 63 | private String deviceBrand = ""; 64 | private String deviceBrandCode = ""; 65 | private String deviceBrandHomepage = ""; 66 | private String deviceBrandIcon = ""; 67 | private String deviceBrandIconBig = ""; 68 | private String deviceBrandInfoUrl = ""; 69 | 70 | public UdgerUaResult(String uaString) { 71 | this.uaString = uaString; 72 | } 73 | 74 | public String getUaString() { 75 | return uaString; 76 | } 77 | 78 | public Integer getClientId() { 79 | return clientId; 80 | } 81 | public void setClientId(Integer clientId) { 82 | this.clientId = clientId; 83 | } 84 | public Integer getClassId() { 85 | return classId; 86 | } 87 | public void setClassId(Integer classId) { 88 | this.classId = classId; 89 | } 90 | public String getUaClass() { 91 | return uaClass; 92 | } 93 | public void setUaClass(String uaClass) { 94 | this.uaClass = uaClass; 95 | } 96 | public String getUaClassCode() { 97 | return uaClassCode; 98 | } 99 | public void setUaClassCode(String uaClassCode) { 100 | this.uaClassCode = uaClassCode; 101 | } 102 | public String getUa() { 103 | return ua; 104 | } 105 | public void setUa(String ua) { 106 | this.ua = ua; 107 | } 108 | public String getUaEngine() { 109 | return uaEngine; 110 | } 111 | public void setUaEngine(String uaEngine) { 112 | this.uaEngine = uaEngine; 113 | } 114 | public String getUaVersion() { 115 | return uaVersion; 116 | } 117 | public void setUaVersion(String uaVersion) { 118 | this.uaVersion = uaVersion; 119 | } 120 | public String getUaVersionMajor() { 121 | return uaVersionMajor; 122 | } 123 | public void setUaVersionMajor(String uaVersionMajor) { 124 | this.uaVersionMajor = uaVersionMajor; 125 | } 126 | public String getCrawlerLastSeen() { 127 | return crawlerLastSeen; 128 | } 129 | public void setCrawlerLastSeen(String crawlerLastSeen) { 130 | this.crawlerLastSeen = crawlerLastSeen; 131 | } 132 | public String getCrawlerRespectRobotstxt() { 133 | return crawlerRespectRobotstxt; 134 | } 135 | public void setCrawlerRespectRobotstxt(String crawlerRespectRobotstxt) { 136 | this.crawlerRespectRobotstxt = crawlerRespectRobotstxt; 137 | } 138 | public String getCrawlerCategory() { 139 | return crawlerCategory; 140 | } 141 | public void setCrawlerCategory(String crawlerCategory) { 142 | this.crawlerCategory = crawlerCategory; 143 | } 144 | public String getCrawlerCategoryCode() { 145 | return crawlerCategoryCode; 146 | } 147 | public void setCrawlerCategoryCode(String crawlerCategoryCode) { 148 | this.crawlerCategoryCode = crawlerCategoryCode; 149 | } 150 | public String getUaUptodateCurrentVersion() { 151 | return uaUptodateCurrentVersion; 152 | } 153 | public void setUaUptodateCurrentVersion(String uaUptodateCurrentVersion) { 154 | this.uaUptodateCurrentVersion = uaUptodateCurrentVersion; 155 | } 156 | public String getUaFamily() { 157 | return uaFamily; 158 | } 159 | public void setUaFamily(String uaFamily) { 160 | this.uaFamily = uaFamily; 161 | } 162 | public String getUaFamilyCode() { 163 | return uaFamilyCode; 164 | } 165 | public void setUaFamilyCode(String uaFamilyCode) { 166 | this.uaFamilyCode = uaFamilyCode; 167 | } 168 | public String getUaFamilyHomepage() { 169 | return uaFamilyHomepage; 170 | } 171 | public void setUaFamilyHomepage(String uaFamilyHomepage) { 172 | this.uaFamilyHomepage = uaFamilyHomepage; 173 | } 174 | public String getUaFamilyIcon() { 175 | return uaFamilyIcon; 176 | } 177 | public void setUaFamilyIcon(String uaFamilyIcon) { 178 | this.uaFamilyIcon = uaFamilyIcon; 179 | } 180 | public String getUaFamilyIconBig() { 181 | return uaFamilyIconBig; 182 | } 183 | public void setUaFamilyIconBig(String uaFamilyIconBig) { 184 | this.uaFamilyIconBig = uaFamilyIconBig; 185 | } 186 | public String getUaFamilyVendor() { 187 | return uaFamilyVendor; 188 | } 189 | public void setUaFamilyVendor(String uaFamilyVendor) { 190 | this.uaFamilyVendor = uaFamilyVendor; 191 | } 192 | public String getUaFamilyVendorCode() { 193 | return uaFamilyVendorCode; 194 | } 195 | public void setUaFamilyVendorCode(String uaFamilyVendorCode) { 196 | this.uaFamilyVendorCode = uaFamilyVendorCode; 197 | } 198 | public String getUaFamilyVendorHomepage() { 199 | return uaFamilyVendorHomepage; 200 | } 201 | public void setUaFamilyVendorHomepage(String uaFamilyVendorHomepage) { 202 | this.uaFamilyVendorHomepage = uaFamilyVendorHomepage; 203 | } 204 | public String getUaFamilyInfoUrl() { 205 | return uaFamilyInfoUrl; 206 | } 207 | public void setUaFamilyInfoUrl(String uaFamilyInfoUrl) { 208 | this.uaFamilyInfoUrl = uaFamilyInfoUrl; 209 | } 210 | 211 | public String getOsFamily() { 212 | return osFamily; 213 | } 214 | public void setOsFamily(String osFamily) { 215 | this.osFamily = osFamily; 216 | } 217 | public String getOsFamilyCode() { 218 | return osFamilyCode; 219 | } 220 | public void setOsFamilyCode(String osFamilyCode) { 221 | this.osFamilyCode = osFamilyCode; 222 | } 223 | public String getOs() { 224 | return os; 225 | } 226 | public void setOs(String os) { 227 | this.os = os; 228 | } 229 | public String getOsCode() { 230 | return osCode; 231 | } 232 | public void setOsCode(String osCode) { 233 | this.osCode = osCode; 234 | } 235 | public String getOsHomePage() { 236 | return osHomePage; 237 | } 238 | public void setOsHomePage(String osHomePage) { 239 | this.osHomePage = osHomePage; 240 | } 241 | public String getOsIcon() { 242 | return osIcon; 243 | } 244 | public void setOsIcon(String osIcon) { 245 | this.osIcon = osIcon; 246 | } 247 | public String getOsIconBig() { 248 | return osIconBig; 249 | } 250 | public void setOsIconBig(String osIconBig) { 251 | this.osIconBig = osIconBig; 252 | } 253 | public String getOsFamilyVendor() { 254 | return osFamilyVendor; 255 | } 256 | public void setOsFamilyVendor(String osFamilyVendor) { 257 | this.osFamilyVendor = osFamilyVendor; 258 | } 259 | public String getOsFamilyVendorCode() { 260 | return osFamilyVendorCode; 261 | } 262 | public void setOsFamilyVendorCode(String osFamilyVendorCode) { 263 | this.osFamilyVendorCode = osFamilyVendorCode; 264 | } 265 | public String getOsFamilyVendorHomepage() { 266 | return osFamilyVendorHomepage; 267 | } 268 | public void setOsFamilyVendorHomepage(String osFamilyVendorHomepage) { 269 | this.osFamilyVendorHomepage = osFamilyVendorHomepage; 270 | } 271 | public String getOsInfoUrl() { 272 | return osInfoUrl; 273 | } 274 | public void setOsInfoUrl(String osInfoUrl) { 275 | this.osInfoUrl = osInfoUrl; 276 | } 277 | 278 | public String getDeviceClass() { 279 | return deviceClass; 280 | } 281 | public void setDeviceClass(String deviceClass) { 282 | this.deviceClass = deviceClass; 283 | } 284 | public String getDeviceClassCode() { 285 | return deviceClassCode; 286 | } 287 | public void setDeviceClassCode(String deviceClassCode) { 288 | this.deviceClassCode = deviceClassCode; 289 | } 290 | public String getDeviceClassIcon() { 291 | return deviceClassIcon; 292 | } 293 | public void setDeviceClassIcon(String deviceClassIcon) { 294 | this.deviceClassIcon = deviceClassIcon; 295 | } 296 | public String getDeviceClassIconBig() { 297 | return deviceClassIconBig; 298 | } 299 | public void setDeviceClassIconBig(String deviceClassIconBig) { 300 | this.deviceClassIconBig = deviceClassIconBig; 301 | } 302 | public String getDeviceClassInfoUrl() { 303 | return deviceClassInfoUrl; 304 | } 305 | public void setDeviceClassInfoUrl(String deviceClassInfoUrl) { 306 | this.deviceClassInfoUrl = deviceClassInfoUrl; 307 | } 308 | 309 | public String getDeviceMarketname() { 310 | return deviceMarketname; 311 | } 312 | 313 | public void setDeviceMarketname(String deviceMarketname) { 314 | this.deviceMarketname = deviceMarketname; 315 | } 316 | 317 | public String getDeviceBrand() { 318 | return deviceBrand; 319 | } 320 | 321 | public void setDeviceBrand(String deviceBrand) { 322 | this.deviceBrand = deviceBrand; 323 | } 324 | 325 | public String getDeviceBrandCode() { 326 | return deviceBrandCode; 327 | } 328 | 329 | public void setDeviceBrandCode(String deviceBrandCode) { 330 | this.deviceBrandCode = deviceBrandCode; 331 | } 332 | 333 | public String getDeviceBrandHomepage() { 334 | return deviceBrandHomepage; 335 | } 336 | 337 | public void setDeviceBrandHomepage(String deviceBrandHomepage) { 338 | this.deviceBrandHomepage = deviceBrandHomepage; 339 | } 340 | 341 | public String getDeviceBrandIcon() { 342 | return deviceBrandIcon; 343 | } 344 | 345 | public void setDeviceBrandIcon(String deviceBrandIcon) { 346 | this.deviceBrandIcon = deviceBrandIcon; 347 | } 348 | 349 | public String getDeviceBrandIconBig() { 350 | return deviceBrandIconBig; 351 | } 352 | 353 | public void setDeviceBrandIconBig(String deviceBrandIconBig) { 354 | this.deviceBrandIconBig = deviceBrandIconBig; 355 | } 356 | 357 | public String getDeviceBrandInfoUrl() { 358 | return deviceBrandInfoUrl; 359 | } 360 | 361 | public void setDeviceBrandInfoUrl(String deviceBrandInfoUrl) { 362 | this.deviceBrandInfoUrl = deviceBrandInfoUrl; 363 | } 364 | 365 | @Override 366 | public int hashCode() { 367 | final int prime = 31; 368 | int result = 1; 369 | result = prime * result + ((classId == null) ? 0 : classId.hashCode()); 370 | result = prime * result + ((clientId == null) ? 0 : clientId.hashCode()); 371 | result = prime * result + ((crawlerCategory == null) ? 0 : crawlerCategory.hashCode()); 372 | result = prime * result + ((crawlerCategoryCode == null) ? 0 : crawlerCategoryCode.hashCode()); 373 | result = prime * result + ((crawlerLastSeen == null) ? 0 : crawlerLastSeen.hashCode()); 374 | result = prime * result + ((crawlerRespectRobotstxt == null) ? 0 : crawlerRespectRobotstxt.hashCode()); 375 | result = prime * result + ((deviceBrand == null) ? 0 : deviceBrand.hashCode()); 376 | result = prime * result + ((deviceBrandCode == null) ? 0 : deviceBrandCode.hashCode()); 377 | result = prime * result + ((deviceBrandHomepage == null) ? 0 : deviceBrandHomepage.hashCode()); 378 | result = prime * result + ((deviceBrandIcon == null) ? 0 : deviceBrandIcon.hashCode()); 379 | result = prime * result + ((deviceBrandIconBig == null) ? 0 : deviceBrandIconBig.hashCode()); 380 | result = prime * result + ((deviceBrandInfoUrl == null) ? 0 : deviceBrandInfoUrl.hashCode()); 381 | result = prime * result + ((deviceClass == null) ? 0 : deviceClass.hashCode()); 382 | result = prime * result + ((deviceClassCode == null) ? 0 : deviceClassCode.hashCode()); 383 | result = prime * result + ((deviceClassIcon == null) ? 0 : deviceClassIcon.hashCode()); 384 | result = prime * result + ((deviceClassIconBig == null) ? 0 : deviceClassIconBig.hashCode()); 385 | result = prime * result + ((deviceClassInfoUrl == null) ? 0 : deviceClassInfoUrl.hashCode()); 386 | result = prime * result + ((deviceMarketname == null) ? 0 : deviceMarketname.hashCode()); 387 | result = prime * result + ((os == null) ? 0 : os.hashCode()); 388 | result = prime * result + ((osCode == null) ? 0 : osCode.hashCode()); 389 | result = prime * result + ((osFamily == null) ? 0 : osFamily.hashCode()); 390 | result = prime * result + ((osFamilyCode == null) ? 0 : osFamilyCode.hashCode()); 391 | result = prime * result + ((osFamilyVendorHomepage == null) ? 0 : osFamilyVendorHomepage.hashCode()); 392 | result = prime * result + ((osFamilyVendor == null) ? 0 : osFamilyVendor.hashCode()); 393 | result = prime * result + ((osFamilyVendorCode == null) ? 0 : osFamilyVendorCode.hashCode()); 394 | result = prime * result + ((osHomePage == null) ? 0 : osHomePage.hashCode()); 395 | result = prime * result + ((osIcon == null) ? 0 : osIcon.hashCode()); 396 | result = prime * result + ((osIconBig == null) ? 0 : osIconBig.hashCode()); 397 | result = prime * result + ((osInfoUrl == null) ? 0 : osInfoUrl.hashCode()); 398 | result = prime * result + ((ua == null) ? 0 : ua.hashCode()); 399 | result = prime * result + ((uaClass == null) ? 0 : uaClass.hashCode()); 400 | result = prime * result + ((uaClassCode == null) ? 0 : uaClassCode.hashCode()); 401 | result = prime * result + ((uaEngine == null) ? 0 : uaEngine.hashCode()); 402 | result = prime * result + ((uaFamily == null) ? 0 : uaFamily.hashCode()); 403 | result = prime * result + ((uaFamilyCode == null) ? 0 : uaFamilyCode.hashCode()); 404 | result = prime * result + ((uaFamilyHomepage == null) ? 0 : uaFamilyHomepage.hashCode()); 405 | result = prime * result + ((uaFamilyIcon == null) ? 0 : uaFamilyIcon.hashCode()); 406 | result = prime * result + ((uaFamilyIconBig == null) ? 0 : uaFamilyIconBig.hashCode()); 407 | result = prime * result + ((uaFamilyInfoUrl == null) ? 0 : uaFamilyInfoUrl.hashCode()); 408 | result = prime * result + ((uaFamilyVendor == null) ? 0 : uaFamilyVendor.hashCode()); 409 | result = prime * result + ((uaFamilyVendorCode == null) ? 0 : uaFamilyVendorCode.hashCode()); 410 | result = prime * result + ((uaFamilyVendorHomepage == null) ? 0 : uaFamilyVendorHomepage.hashCode()); 411 | result = prime * result + ((uaString == null) ? 0 : uaString.hashCode()); 412 | result = prime * result + ((uaUptodateCurrentVersion == null) ? 0 : uaUptodateCurrentVersion.hashCode()); 413 | result = prime * result + ((uaVersion == null) ? 0 : uaVersion.hashCode()); 414 | result = prime * result + ((uaVersionMajor == null) ? 0 : uaVersionMajor.hashCode()); 415 | return result; 416 | } 417 | 418 | @Override 419 | public boolean equals(Object obj) { 420 | if (this == obj) 421 | return true; 422 | if (obj == null) 423 | return false; 424 | if (getClass() != obj.getClass()) 425 | return false; 426 | UdgerUaResult other = (UdgerUaResult) obj; 427 | if (classId == null) { 428 | if (other.classId != null) 429 | return false; 430 | } else if (!classId.equals(other.classId)) 431 | return false; 432 | if (clientId == null) { 433 | if (other.clientId != null) 434 | return false; 435 | } else if (!clientId.equals(other.clientId)) 436 | return false; 437 | if (crawlerCategory == null) { 438 | if (other.crawlerCategory != null) 439 | return false; 440 | } else if (!crawlerCategory.equals(other.crawlerCategory)) 441 | return false; 442 | if (crawlerCategoryCode == null) { 443 | if (other.crawlerCategoryCode != null) 444 | return false; 445 | } else if (!crawlerCategoryCode.equals(other.crawlerCategoryCode)) 446 | return false; 447 | if (crawlerLastSeen == null) { 448 | if (other.crawlerLastSeen != null) 449 | return false; 450 | } else if (!crawlerLastSeen.equals(other.crawlerLastSeen)) 451 | return false; 452 | if (crawlerRespectRobotstxt == null) { 453 | if (other.crawlerRespectRobotstxt != null) 454 | return false; 455 | } else if (!crawlerRespectRobotstxt.equals(other.crawlerRespectRobotstxt)) 456 | return false; 457 | if (deviceBrand == null) { 458 | if (other.deviceBrand != null) 459 | return false; 460 | } else if (!deviceBrand.equals(other.deviceBrand)) 461 | return false; 462 | if (deviceBrandCode == null) { 463 | if (other.deviceBrandCode != null) 464 | return false; 465 | } else if (!deviceBrandCode.equals(other.deviceBrandCode)) 466 | return false; 467 | if (deviceBrandHomepage == null) { 468 | if (other.deviceBrandHomepage != null) 469 | return false; 470 | } else if (!deviceBrandHomepage.equals(other.deviceBrandHomepage)) 471 | return false; 472 | if (deviceBrandIcon == null) { 473 | if (other.deviceBrandIcon != null) 474 | return false; 475 | } else if (!deviceBrandIcon.equals(other.deviceBrandIcon)) 476 | return false; 477 | if (deviceBrandIconBig == null) { 478 | if (other.deviceBrandIconBig != null) 479 | return false; 480 | } else if (!deviceBrandIconBig.equals(other.deviceBrandIconBig)) 481 | return false; 482 | if (deviceBrandInfoUrl == null) { 483 | if (other.deviceBrandInfoUrl != null) 484 | return false; 485 | } else if (!deviceBrandInfoUrl.equals(other.deviceBrandInfoUrl)) 486 | return false; 487 | if (deviceClass == null) { 488 | if (other.deviceClass != null) 489 | return false; 490 | } else if (!deviceClass.equals(other.deviceClass)) 491 | return false; 492 | if (deviceClassCode == null) { 493 | if (other.deviceClassCode != null) 494 | return false; 495 | } else if (!deviceClassCode.equals(other.deviceClassCode)) 496 | return false; 497 | if (deviceClassIcon == null) { 498 | if (other.deviceClassIcon != null) 499 | return false; 500 | } else if (!deviceClassIcon.equals(other.deviceClassIcon)) 501 | return false; 502 | if (deviceClassIconBig == null) { 503 | if (other.deviceClassIconBig != null) 504 | return false; 505 | } else if (!deviceClassIconBig.equals(other.deviceClassIconBig)) 506 | return false; 507 | if (deviceClassInfoUrl == null) { 508 | if (other.deviceClassInfoUrl != null) 509 | return false; 510 | } else if (!deviceClassInfoUrl.equals(other.deviceClassInfoUrl)) 511 | return false; 512 | if (deviceMarketname == null) { 513 | if (other.deviceMarketname != null) 514 | return false; 515 | } else if (!deviceMarketname.equals(other.deviceMarketname)) 516 | return false; 517 | if (os == null) { 518 | if (other.os != null) 519 | return false; 520 | } else if (!os.equals(other.os)) 521 | return false; 522 | if (osCode == null) { 523 | if (other.osCode != null) 524 | return false; 525 | } else if (!osCode.equals(other.osCode)) 526 | return false; 527 | if (osFamily == null) { 528 | if (other.osFamily != null) 529 | return false; 530 | } else if (!osFamily.equals(other.osFamily)) 531 | return false; 532 | if (osFamilyCode == null) { 533 | if (other.osFamilyCode != null) 534 | return false; 535 | } else if (!osFamilyCode.equals(other.osFamilyCode)) 536 | return false; 537 | if (osFamilyVendorHomepage == null) { 538 | if (other.osFamilyVendorHomepage != null) 539 | return false; 540 | } else if (!osFamilyVendorHomepage.equals(other.osFamilyVendorHomepage)) 541 | return false; 542 | if (osFamilyVendor == null) { 543 | if (other.osFamilyVendor != null) 544 | return false; 545 | } else if (!osFamilyVendor.equals(other.osFamilyVendor)) 546 | return false; 547 | if (osFamilyVendorCode == null) { 548 | if (other.osFamilyVendorCode != null) 549 | return false; 550 | } else if (!osFamilyVendorCode.equals(other.osFamilyVendorCode)) 551 | return false; 552 | if (osHomePage == null) { 553 | if (other.osHomePage != null) 554 | return false; 555 | } else if (!osHomePage.equals(other.osHomePage)) 556 | return false; 557 | if (osIcon == null) { 558 | if (other.osIcon != null) 559 | return false; 560 | } else if (!osIcon.equals(other.osIcon)) 561 | return false; 562 | if (osIconBig == null) { 563 | if (other.osIconBig != null) 564 | return false; 565 | } else if (!osIconBig.equals(other.osIconBig)) 566 | return false; 567 | if (osInfoUrl == null) { 568 | if (other.osInfoUrl != null) 569 | return false; 570 | } else if (!osInfoUrl.equals(other.osInfoUrl)) 571 | return false; 572 | if (ua == null) { 573 | if (other.ua != null) 574 | return false; 575 | } else if (!ua.equals(other.ua)) 576 | return false; 577 | if (uaClass == null) { 578 | if (other.uaClass != null) 579 | return false; 580 | } else if (!uaClass.equals(other.uaClass)) 581 | return false; 582 | if (uaClassCode == null) { 583 | if (other.uaClassCode != null) 584 | return false; 585 | } else if (!uaClassCode.equals(other.uaClassCode)) 586 | return false; 587 | if (uaEngine == null) { 588 | if (other.uaEngine != null) 589 | return false; 590 | } else if (!uaEngine.equals(other.uaEngine)) 591 | return false; 592 | if (uaFamily == null) { 593 | if (other.uaFamily != null) 594 | return false; 595 | } else if (!uaFamily.equals(other.uaFamily)) 596 | return false; 597 | if (uaFamilyCode == null) { 598 | if (other.uaFamilyCode != null) 599 | return false; 600 | } else if (!uaFamilyCode.equals(other.uaFamilyCode)) 601 | return false; 602 | if (uaFamilyHomepage == null) { 603 | if (other.uaFamilyHomepage != null) 604 | return false; 605 | } else if (!uaFamilyHomepage.equals(other.uaFamilyHomepage)) 606 | return false; 607 | if (uaFamilyIcon == null) { 608 | if (other.uaFamilyIcon != null) 609 | return false; 610 | } else if (!uaFamilyIcon.equals(other.uaFamilyIcon)) 611 | return false; 612 | if (uaFamilyIconBig == null) { 613 | if (other.uaFamilyIconBig != null) 614 | return false; 615 | } else if (!uaFamilyIconBig.equals(other.uaFamilyIconBig)) 616 | return false; 617 | if (uaFamilyInfoUrl == null) { 618 | if (other.uaFamilyInfoUrl != null) 619 | return false; 620 | } else if (!uaFamilyInfoUrl.equals(other.uaFamilyInfoUrl)) 621 | return false; 622 | if (uaFamilyVendor == null) { 623 | if (other.uaFamilyVendor != null) 624 | return false; 625 | } else if (!uaFamilyVendor.equals(other.uaFamilyVendor)) 626 | return false; 627 | if (uaFamilyVendorCode == null) { 628 | if (other.uaFamilyVendorCode != null) 629 | return false; 630 | } else if (!uaFamilyVendorCode.equals(other.uaFamilyVendorCode)) 631 | return false; 632 | if (uaFamilyVendorHomepage == null) { 633 | if (other.uaFamilyVendorHomepage != null) 634 | return false; 635 | } else if (!uaFamilyVendorHomepage.equals(other.uaFamilyVendorHomepage)) 636 | return false; 637 | if (uaString == null) { 638 | if (other.uaString != null) 639 | return false; 640 | } else if (!uaString.equals(other.uaString)) 641 | return false; 642 | if (uaUptodateCurrentVersion == null) { 643 | if (other.uaUptodateCurrentVersion != null) 644 | return false; 645 | } else if (!uaUptodateCurrentVersion.equals(other.uaUptodateCurrentVersion)) 646 | return false; 647 | if (uaVersion == null) { 648 | if (other.uaVersion != null) 649 | return false; 650 | } else if (!uaVersion.equals(other.uaVersion)) 651 | return false; 652 | if (uaVersionMajor == null) { 653 | if (other.uaVersionMajor != null) 654 | return false; 655 | } else if (!uaVersionMajor.equals(other.uaVersionMajor)) 656 | return false; 657 | return true; 658 | } 659 | 660 | @Override 661 | public String toString() { 662 | return "UdgerUaResult [" + 663 | "uaString=" + uaString + 664 | ", clientId=" + clientId + 665 | ", classId=" + classId + 666 | ", uaClass=" + uaClass + 667 | ", uaClassCode=" + uaClassCode + 668 | ", ua=" + ua + 669 | ", uaEngine=" + uaEngine + 670 | ", uaVersion=" + uaVersion + 671 | ", uaVersionMajor=" + uaVersionMajor + 672 | ", crawlerLastSeen=" + crawlerLastSeen + 673 | ", crawlerRespectRobotstxt=" + crawlerRespectRobotstxt + 674 | ", crawlerCategory=" + crawlerCategory + 675 | ", crawlerCategoryCode=" + crawlerCategoryCode + 676 | ", uaUptodateCurrentVersion=" + uaUptodateCurrentVersion + 677 | ", uaFamily=" + uaFamily + 678 | ", uaFamilyCode=" + uaFamilyCode + 679 | ", uaFamilyHomepage=" + uaFamilyHomepage + 680 | ", uaFamilyIcon=" + uaFamilyIcon + 681 | ", uaFamilyIconBig=" + uaFamilyIconBig + 682 | ", uaFamilyVendor=" + uaFamilyVendor + 683 | ", uaFamilyVendorCode=" + uaFamilyVendorCode + 684 | ", uaFamilyVendorHomepage=" + uaFamilyVendorHomepage + 685 | ", uaFamilyInfoUrl=" + uaFamilyInfoUrl + 686 | ", osFamily=" + osFamily + 687 | ", osFamilyCode=" + osFamilyCode + 688 | ", os=" + os + 689 | ", osCode=" + osCode + 690 | ", osHomePage=" + osHomePage + 691 | ", osIcon=" + osIcon + 692 | ", osIconBig=" + osIconBig + 693 | ", osFamilyVendor=" + osFamilyVendor + 694 | ", osFamilyVendorCode=" + osFamilyVendorCode + 695 | ", osFamilyVendorHomepage=" + osFamilyVendorHomepage + 696 | ", osInfoUrl=" + osInfoUrl + 697 | ", deviceClass=" + deviceClass + 698 | ", deviceClassCode=" + deviceClassCode + 699 | ", deviceClassIcon=" + deviceClassIcon + 700 | ", deviceClassIconBig=" + deviceClassIconBig + 701 | ", deviceClassInfoUrl=" + deviceClassInfoUrl + 702 | ", deviceMarketname=" + deviceMarketname + 703 | ", deviceBrand=" + deviceBrand + 704 | ", deviceBrandCode=" + deviceBrandCode + 705 | ", deviceBrandHomepage=" + deviceBrandHomepage + 706 | ", deviceBrandIcon=" + deviceBrandIcon + 707 | ", deviceBrandIconBig=" + deviceBrandIconBig + 708 | ", deviceBrandInfoUrl=" + deviceBrandInfoUrl + 709 | "]"; 710 | } 711 | 712 | } 713 | -------------------------------------------------------------------------------- /src/main/java/org/udger/parser/WordDetector.java: -------------------------------------------------------------------------------- 1 | package org.udger.parser; 2 | 3 | import java.io.Serializable; 4 | import java.util.ArrayList; 5 | import java.util.HashSet; 6 | import java.util.List; 7 | import java.util.Set; 8 | import java.util.logging.Logger; 9 | 10 | public class WordDetector implements Serializable { 11 | 12 | private static final long serialVersionUID = -2123898245391386812L; 13 | 14 | private static final Logger LOG = Logger.getLogger(WordDetector.class.getName()); 15 | 16 | private static class WordInfo { 17 | int id; 18 | String word; 19 | 20 | public WordInfo(int id, String word) { 21 | this.id = id; 22 | this.word = word; 23 | } 24 | } 25 | 26 | private static final int ARRAY_DIMENSION = 'z' - 'a'; 27 | private static final int ARRAY_SIZE = (ARRAY_DIMENSION + 1) * (ARRAY_DIMENSION + 1); 28 | 29 | private List wordArray[]; 30 | private int minWordSize = Integer.MAX_VALUE; 31 | 32 | public WordDetector() { 33 | wordArray = new List[ARRAY_SIZE]; 34 | } 35 | 36 | public void addWord(int id, String word) { 37 | 38 | if (word.length() < minWordSize) { 39 | minWordSize = word.length(); 40 | } 41 | 42 | String s = word.toLowerCase(); 43 | int index = (s.charAt(0) - 'a') * ARRAY_DIMENSION + s.charAt(1) - 'a'; 44 | if (index >= 0 && index < ARRAY_SIZE) { 45 | List wList = wordArray[index]; 46 | if (wList == null) { 47 | wList = new ArrayList<>(); 48 | wordArray[index] = wList; 49 | } 50 | wList.add(new WordInfo(id, s)); 51 | } else { 52 | LOG.warning("Index out of hashmap" + id + " : "+ s); 53 | } 54 | } 55 | 56 | public Set findWords(String text) { 57 | 58 | Set ret = new HashSet<>(); 59 | 60 | final String s = text.toLowerCase(); 61 | final int dimension = 'z' - 'a'; 62 | for(int i=0; i < s.length() - (minWordSize - 1); i++) { 63 | final char c1 = s.charAt(i); 64 | final char c2 = s.charAt(i + 1); 65 | if (c1 >= 'a' && c1 <= 'z' && c2 >= 'a' && c2 <= 'z') { 66 | final int index = (c1 - 'a') * dimension + c2 - 'a'; 67 | List l = wordArray[index]; 68 | if (l != null) { 69 | for (WordInfo wi : l) { 70 | if (s.startsWith(wi.word, i)) { 71 | ret.add(wi.id); 72 | } 73 | } 74 | } 75 | } 76 | } 77 | return ret; 78 | } 79 | 80 | } 81 | -------------------------------------------------------------------------------- /src/test/java/org/udger/parser/LRUCacheTest.java: -------------------------------------------------------------------------------- 1 | package org.udger.parser; 2 | 3 | import org.junit.Test; 4 | 5 | import static org.junit.Assert.*; 6 | 7 | public class LRUCacheTest { 8 | 9 | private LRUCache cache = new LRUCache<>(3); 10 | 11 | @Test 12 | public void testLRUCacheRespectCapacity() throws Exception { 13 | for(int i = 0; i < cache.getCapacity() + 1; i++){ 14 | cache.put(i,i); 15 | } 16 | assertNull("Expected the first value put into cache to be absent after adding one more than capacity allows.", cache.get(0)); 17 | } 18 | 19 | @Test 20 | public void testLRUCacheUpdatesLRUOrder() throws Exception { 21 | for(int i = 0; i < cache.getCapacity(); i++){ 22 | cache.put(i,i); 23 | } 24 | assertNotNull("Expected to first put value to be in the cache.", cache.get(0)); 25 | cache.put(cache.getCapacity()+1, cache.getCapacity()+1); 26 | assertNull("Expected the first value to have gone to the tail and second value to be removed for the next.", cache.get(1)); 27 | } 28 | } -------------------------------------------------------------------------------- /src/test/java/org/udger/parser/UdgerIpTest.java: -------------------------------------------------------------------------------- 1 | package org.udger.parser; 2 | 3 | import java.io.IOException; 4 | import java.io.InputStream; 5 | import java.net.UnknownHostException; 6 | import java.sql.SQLException; 7 | 8 | import javax.json.JsonArray; 9 | import javax.json.JsonObject; 10 | import javax.json.JsonReader; 11 | 12 | public class UdgerIpTest { 13 | 14 | public static void main(String args[]) { 15 | InputStream is = UdgerUaTest.class.getResourceAsStream("test_ip.json"); 16 | JsonReader jr = javax.json.Json.createReader(is); 17 | JsonArray ja = jr.readArray(); 18 | UdgerParser up = null; 19 | try { 20 | UdgerParser.ParserDbData parserDbData = new UdgerParser.ParserDbData("udgerdb_v3.dat"); 21 | up = new UdgerParser(parserDbData); 22 | for (int i=0; i < ja.size(); i++) { 23 | JsonObject jar = ja.getJsonObject(i); 24 | JsonObject jor = jar.getJsonObject("ret"); 25 | String query = jar.getJsonObject("test").getString("teststring"); 26 | try { 27 | UdgerIpResult ret = up.parseIp(query); 28 | System.out.print("### Test : " + (i+1) + " - "); 29 | if (checkResult(ret, jor)) { 30 | System.out.println("SUCCEEDED"); 31 | } else { 32 | System.out.println("FAILED!"); 33 | } 34 | System.out.println("Query: " + query); 35 | // System.out.println("Result: " + ReflectionToStringBuilder.toString(ret, ToStringStyle.MULTI_LINE_STYLE)); 36 | 37 | } catch (SQLException e) { 38 | e.printStackTrace(); 39 | } catch (UnknownHostException e) { 40 | // TODO Auto-generated catch block 41 | e.printStackTrace(); 42 | } 43 | } 44 | } finally { 45 | if (up != null) { 46 | try { 47 | up.close(); 48 | } catch (IOException e) { 49 | } 50 | } 51 | } 52 | } 53 | 54 | private static boolean checkResult(UdgerIpResult ret, JsonObject jor) { 55 | boolean result = true; 56 | result = testEqual(jor, "ip_classification_code", ret.getIpClassificationCode()) && result; 57 | result = testEqual(jor, "datacenter_homepage", ret.getDataCenterHomePage()) && result; 58 | result = testEqual(jor, "crawler_respect_robotstxt", ret.getCrawlerRespectRobotstxt()) && result; 59 | result = testEqual(jor, "crawler_family_vendor_code", ret.getCrawlerFamilyVendorCode()) && result; 60 | result = testEqual(jor, "crawler_category", ret.getCrawlerCategory()) && result; 61 | result = testEqual(jor, "crawler_ver_major", ret.getCrawlerVerMajor()) && result; 62 | result = testEqual(jor, "ip_country", ret.getIpCountry()) && result; 63 | result = testEqual(jor, "crawler_ver", ret.getCrawlerVer()) && result; 64 | result = testEqual(jor, "crawler_name", ret.getCrawlerName()) && result; 65 | result = testEqual(jor, "ip_city", ret.getIpCity()) && result; 66 | result = testEqual(jor, "datacenter_name_code", ret.getDataCenterNameCode()) && result; 67 | result = testEqual(jor, "crawler_family_info_url", ret.getCrawlerFamilyInfoUrl()) && result; 68 | result = testEqual(jor, "ip_country_code", ret.getIpCountryCode()) && result; 69 | result = testEqual(jor, "ip_ver", String.valueOf(ret.getIpVer())) && result; 70 | result = testEqual(jor, "ip_classification", ret.getIpClassification()) && result; 71 | result = testEqual(jor, "crawler_family_code", ret.getCrawlerFamilyCode()) && result; 72 | result = testEqual(jor, "crawler_family_icon", ret.getCrawlerFamilyIcon()) && result; 73 | result = testEqual(jor, "crawler_family_homepage", ret.getCrawlerFamilyHomepage()) && result; 74 | result = testEqual(jor, "crawler_family_vendor_homepage", ret.getCrawlerFamilyVendorHomepage()) && result; 75 | result = testEqual(jor, "datacenter_name", ret.getDataCenterName()) && result; 76 | result = testEqual(jor, "ip", ret.getIp()) && result; 77 | result = testEqual(jor, "crawler_family", ret.getCrawlerFamily()) && result; 78 | result = testEqual(jor, "ip_hostname", ret.getIpHostname()) && result; 79 | result = testEqual(jor, "crawler_category_code", ret.getCrawlerCategoryCode()) && result; 80 | result = testEqual(jor, "crawler_family_vendor", ret.getCrawlerFamilyVendor()) && result; 81 | return result; 82 | } 83 | 84 | private static boolean testEqual(JsonObject jor, String test, String ret) { 85 | String expected = jor.getString(test); 86 | if (!expected.equals(ret)) { 87 | System.out.println("Failed \"" + test + "\" : value=" + ret + " expected:" + expected); 88 | return false; 89 | } 90 | return true; 91 | } 92 | } 93 | -------------------------------------------------------------------------------- /src/test/java/org/udger/parser/UdgerParserChangeDBTest.java: -------------------------------------------------------------------------------- 1 | package org.udger.parser; 2 | 3 | import static org.junit.Assert.assertEquals; 4 | 5 | import java.io.IOException; 6 | import java.net.URL; 7 | import java.sql.SQLException; 8 | 9 | import org.junit.Test; 10 | 11 | public class UdgerParserChangeDBTest { 12 | 13 | @Test 14 | public void testUaString1() throws SQLException, IOException { 15 | String uaQuery = "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:40.0) Gecko/20100101 Firefox/40.0"; 16 | URL resource = this.getClass().getClassLoader().getResource("udgerdb_test_v3.dat"); 17 | UdgerParser.ParserDbData parserDbData = new UdgerParser.ParserDbData(resource.getFile()); 18 | try (UdgerParser parser = new UdgerParser(parserDbData)) { 19 | 20 | UdgerUaResult qr = parser.parseUa(uaQuery); 21 | assertEquals(qr.getUaUptodateCurrentVersion(), "50"); 22 | } 23 | 24 | URL resource2 = this.getClass().getClassLoader().getResource("udgerdb_test_v3_switch.dat"); 25 | UdgerParser.ParserDbData parserDbData2 = new UdgerParser.ParserDbData(resource2.getFile()); 26 | try (UdgerParser parser2 = new UdgerParser(parserDbData2)) { 27 | UdgerUaResult qr2 = parser2.parseUa(uaQuery); 28 | assertEquals(qr2.getUaUptodateCurrentVersion(), "50X"); 29 | } 30 | } 31 | } 32 | -------------------------------------------------------------------------------- /src/test/java/org/udger/parser/UdgerParserTest.java: -------------------------------------------------------------------------------- 1 | package org.udger.parser; 2 | 3 | import static org.junit.Assert.assertEquals; 4 | import static org.junit.Assert.fail; 5 | 6 | import java.io.IOException; 7 | import java.net.URL; 8 | import java.net.UnknownHostException; 9 | import java.sql.SQLException; 10 | import java.util.concurrent.ConcurrentLinkedQueue; 11 | import java.util.concurrent.CyclicBarrier; 12 | 13 | import org.junit.After; 14 | import org.junit.Before; 15 | import org.junit.Test; 16 | 17 | public class UdgerParserTest { 18 | 19 | private UdgerParser parser; 20 | private UdgerParser inMemoryParser; 21 | private UdgerParser.ParserDbData parserDbData; 22 | 23 | @Before 24 | public void initialize() throws SQLException { 25 | URL resource = this.getClass().getClassLoader().getResource("udgerdb_test_v3.dat"); 26 | parserDbData = new UdgerParser.ParserDbData(resource.getFile()); 27 | parser = new UdgerParser(parserDbData); 28 | inMemoryParser = new UdgerParser(parserDbData, true, 0); // no cache 29 | } 30 | 31 | @After 32 | public void close() throws IOException { 33 | parser.close(); 34 | } 35 | 36 | @Test 37 | public void testUaString1() throws SQLException { 38 | String uaQuery = "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:40.0) Gecko/20100101 Firefox/40.0"; 39 | UdgerUaResult qr = parser.parseUa(uaQuery); 40 | assertEquals(qr.getUa(), "Firefox 40.0"); 41 | assertEquals(qr.getOs(), "Windows 10"); 42 | assertEquals(qr.getUaFamily(), "Firefox"); 43 | } 44 | 45 | @Test 46 | public void testIp() throws SQLException, UnknownHostException { 47 | String ipQuery = "108.61.199.93"; 48 | UdgerIpResult qr = parser.parseIp(ipQuery); 49 | assertEquals(qr.getIpClassificationCode(), "crawler"); 50 | } 51 | 52 | @Test 53 | public void testUaStringInMemoryParser() throws SQLException { 54 | String uaQuery = "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:40.0) Gecko/20100101 Firefox/40.0"; 55 | UdgerUaResult qr = inMemoryParser.parseUa(uaQuery); 56 | assertEquals(qr.getUa(), "Firefox 40.0"); 57 | assertEquals(qr.getOs(), "Windows 10"); 58 | assertEquals(qr.getUaFamily(), "Firefox"); 59 | } 60 | 61 | @Test 62 | public void testIpInMemoryParser() throws SQLException, UnknownHostException { 63 | String ipQuery = "108.61.199.93"; 64 | UdgerIpResult qr = inMemoryParser.parseIp(ipQuery); 65 | assertEquals(qr.getIpClassificationCode(), "crawler"); 66 | } 67 | 68 | @Test 69 | public void testParserDbDataThreadSafety() throws Throwable { 70 | final int numThreads = 500; 71 | final String uaQuery = "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:40.0) Gecko/20100101 Firefox/40.0"; 72 | 73 | final CyclicBarrier gate = new CyclicBarrier(numThreads); 74 | final ConcurrentLinkedQueue failures = new ConcurrentLinkedQueue<>(); 75 | 76 | Thread[] threads = new Thread[numThreads]; 77 | for (int i = 0; i < numThreads; i++) { 78 | threads[i] = new Thread(new Runnable() { 79 | @Override 80 | public void run() { 81 | UdgerParser threadParser = new UdgerParser(parserDbData); 82 | try { 83 | gate.await(); 84 | for (int j = 0; j < 100; j++) { 85 | UdgerUaResult qr = threadParser.parseUa(uaQuery); 86 | assertEquals(qr.getUa(), "Firefox 40.0"); 87 | assertEquals(qr.getOs(), "Windows 10"); 88 | assertEquals(qr.getUaFamily(), "Firefox"); 89 | } 90 | } catch (Throwable t) { 91 | failures.add(t); 92 | } 93 | } 94 | }); 95 | threads[i].start(); 96 | } 97 | 98 | for (int i = 0; i < numThreads; i++) { 99 | threads[i].join(); 100 | } 101 | 102 | if (!failures.isEmpty()) { 103 | for (Throwable throwable : failures) { 104 | throwable.printStackTrace(); 105 | } 106 | 107 | fail("Parsing threads failed, see printed exceptions"); 108 | } 109 | } 110 | } 111 | -------------------------------------------------------------------------------- /src/test/java/org/udger/parser/UdgerPerformanceTest.java: -------------------------------------------------------------------------------- 1 | package org.udger.parser; 2 | 3 | import java.io.IOException; 4 | import java.io.InputStream; 5 | import java.sql.SQLException; 6 | 7 | import javax.json.JsonArray; 8 | import javax.json.JsonObject; 9 | import javax.json.JsonReader; 10 | 11 | import org.udger.parser.UdgerParser.ParserDbData; 12 | 13 | public class UdgerPerformanceTest { 14 | 15 | private static JsonArray jsonArray; 16 | private static UdgerParser[] POOL; 17 | 18 | private static void createPool() { 19 | POOL = new UdgerParser[10]; 20 | for (int i=0; i<=9; i++) { 21 | POOL[i] = new UdgerParser(new UdgerParser.ParserDbData("udgerdb_v3_" + i + ".dat")); 22 | } 23 | } 24 | 25 | private static void closePool() throws IOException { 26 | for (int i=0; i<=9; i++) { 27 | if (POOL[i] != null) { 28 | POOL[i].close(); 29 | } 30 | } 31 | } 32 | 33 | public static void main(String args[]) { 34 | InputStream is = UdgerUaTest.class.getResourceAsStream("test_ua.json"); 35 | JsonReader jsonReader = javax.json.Json.createReader(is); 36 | jsonArray = jsonReader.readArray(); 37 | UdgerParser.ParserDbData parserDbData = new UdgerParser.ParserDbData("udgerdb_v3.dat"); 38 | for (int i=0; i<10; i++) { 39 | System.out.println("### Test : " + (i+1)); 40 | testSerial(parserDbData); 41 | } 42 | } 43 | 44 | private static void testSerial(ParserDbData parserDbData) { 45 | UdgerParser up = null; 46 | try { 47 | up = new UdgerParser(parserDbData); 48 | long tm = 0; 49 | for (int j=0; j<100; j++) { 50 | for (int i=0; i < jsonArray.size(); i++) { 51 | JsonObject jar = jsonArray.getJsonObject(i); 52 | String query = jar.getJsonObject("test").getString("teststring"); 53 | try { 54 | long prev = System.nanoTime(); 55 | UdgerUaResult ret = up.parseUa(query); 56 | tm += System.nanoTime() - prev; 57 | } catch (SQLException e) { 58 | e.printStackTrace(); 59 | } 60 | } 61 | } 62 | long numQueries = 100 * jsonArray.size(); 63 | System.out.println("TOTAL Queries: " + numQueries + " time : " + tm / 1000000 + "ms AVG : " + 1000000000 * numQueries / (float) tm + "/s"); 64 | } finally { 65 | if (up != null) { 66 | try { 67 | up.close(); 68 | } catch (IOException e) { 69 | } 70 | } 71 | } 72 | } 73 | 74 | private static void testParallel() { 75 | try { 76 | createPool(); 77 | long tm = 0; 78 | System.out.println("TOTAL Queries: " + 100 * jsonArray.size() + " time : " + tm + " AVG : " + 1000 * 100 * jsonArray.size() / (float) tm + "/s"); 79 | } finally { 80 | try { 81 | closePool(); 82 | } catch (IOException e) { 83 | } 84 | } 85 | } 86 | 87 | } 88 | -------------------------------------------------------------------------------- /src/test/java/org/udger/parser/UdgerUaTest.java: -------------------------------------------------------------------------------- 1 | package org.udger.parser; 2 | 3 | import java.io.IOException; 4 | import java.io.InputStream; 5 | import java.sql.SQLException; 6 | 7 | import javax.json.JsonArray; 8 | import javax.json.JsonObject; 9 | import javax.json.JsonReader; 10 | 11 | public class UdgerUaTest { 12 | 13 | public static void main(String args[]) throws SQLException { 14 | InputStream is = UdgerUaTest.class.getResourceAsStream("test_ua.json"); 15 | JsonReader jr = javax.json.Json.createReader(is); 16 | JsonArray ja = jr.readArray(); 17 | UdgerParser up = null; 18 | try { 19 | UdgerParser.ParserDbData parserDbData = new UdgerParser.ParserDbData("udgerdb_v3.dat"); 20 | up = new UdgerParser(parserDbData); 21 | for (int i=0; i < ja.size(); i++) { 22 | JsonObject jar = ja.getJsonObject(i); 23 | JsonObject jor = jar.getJsonObject("ret"); 24 | String query = jar.getJsonObject("test").getString("teststring"); 25 | try { 26 | UdgerUaResult ret = up.parseUa(query); 27 | System.out.print("### Test : " + (i+1) + " - "); 28 | if (checkResult(ret, jor)) { 29 | System.out.println("SUCCEEDED"); 30 | } else { 31 | System.out.println("FAILED!"); 32 | } 33 | System.out.println("Query: " + query); 34 | // System.out.println("Result: " + ReflectionToStringBuilder.toString(ret, ToStringStyle.MULTI_LINE_STYLE)); 35 | } catch (SQLException e) { 36 | e.printStackTrace(); 37 | } 38 | } 39 | } finally { 40 | if (up != null) { 41 | try { 42 | up.close(); 43 | } catch (IOException e) { 44 | } 45 | } 46 | } 47 | } 48 | 49 | private static boolean checkResult(UdgerUaResult ret, JsonObject jor) { 50 | boolean result = true; 51 | result = testEqual(jor, "ua_engine", ret.getUaEngine()) && result; 52 | result = testEqual(jor, "ua_version", ret.getUaVersion()) && result; 53 | result = testEqual(jor, "ua_family_code", ret.getUaFamilyCode()) && result; 54 | result = testEqual(jor, "ua_family_icon_big", ret.getUaFamilyIconBig()) && result; 55 | result = testEqual(jor, "crawler_category", ret.getCrawlerCategory()) && result; 56 | result = testEqual(jor, "ua_family_icon", ret.getUaFamilyIcon()) && result; 57 | result = testEqual(jor, "ua_family_vendor", ret.getUaFamilyVendor()) && result; 58 | result = testEqual(jor, "ua_family_vendor_code", ret.getUaFamilyVendorCode()) && result; 59 | result = testEqual(jor, "ua_uptodate_current_version", ret.getUaUptodateCurrentVersion()) && result; 60 | result = testEqual(jor, "ua_class_code", ret.getUaClassCode()) && result; 61 | result = testEqual(jor, "ua", ret.getUa()) && result; 62 | result = testEqual(jor, "ua_family", ret.getUaFamily()) && result; 63 | result = testEqual(jor, "ua_family_homepage", ret.getUaFamilyHomepage()) && result; 64 | result = testEqual(jor, "ua_version_major", ret.getUaVersionMajor()) && result; 65 | result = testEqual(jor, "ua_family_info_url", ret.getUaFamilyInfoUrl()) && result; 66 | result = testEqual(jor, "crawler_respect_robotstxt", ret.getCrawlerRespectRobotstxt()) && result; 67 | result = testEqual(jor, "ua_class", ret.getUaClass()) && result; 68 | result = testEqual(jor, "ua_family_vendor_homepage", ret.getUaFamilyVendorHomepage()) && result; 69 | result = testEqual(jor, "crawler_category_code", ret.getCrawlerCategoryCode()) && result; 70 | // result = testEqual(jor, "ua_string", ret.getUserAgent() != null ? ret.getUserAgent().get) && result; 71 | 72 | result = testEqual(jor, "os_family_vendor_homepage", ret.getOsFamilyVendorHomepage()) && result; 73 | result = testEqual(jor, "os_icon_big", ret.getOsIconBig()) && result; 74 | result = testEqual(jor, "os_homepage", ret.getOsHomePage()) && result; 75 | result = testEqual(jor, "os_icon", ret.getOsIcon()) && result; 76 | result = testEqual(jor, "os", ret.getOs()) && result; 77 | result = testEqual(jor, "os_family_code", ret.getOsFamilyCode()) && result; 78 | result = testEqual(jor, "os_family_vendor", ret.getOsFamilyVendor()) && result; 79 | result = testEqual(jor, "os_family_vendor_code", ret.getOsFamilyVendorCode()) && result; 80 | result = testEqual(jor, "os_code", ret.getOsCode()) && result; 81 | result = testEqual(jor, "os_family", ret.getOsFamily()) && result; 82 | result = testEqual(jor, "os_info_url", ret.getOsInfoUrl()) && result; 83 | 84 | result = testEqual(jor, "device_class", ret.getDeviceClass()) && result; 85 | result = testEqual(jor, "device_class_icon_big", ret.getDeviceClassIconBig()) && result; 86 | result = testEqual(jor, "device_class_icon", ret.getDeviceClassIcon()) && result; 87 | result = testEqual(jor, "device_class_info_url", ret.getDeviceClassInfoUrl()) && result; 88 | result = testEqual(jor, "device_class_code", ret.getDeviceClassCode()) && result; 89 | 90 | result = testEqual(jor, "device_marketname", ret.getDeviceMarketname()) && result; 91 | result = testEqual(jor, "device_brand", ret.getDeviceBrand()) && result; 92 | result = testEqual(jor, "device_brand_code", ret.getDeviceBrandCode()) && result; 93 | result = testEqual(jor, "device_brand_homepage", ret.getDeviceBrandHomepage()) && result; 94 | result = testEqual(jor, "device_brand_icon", ret.getDeviceBrandIcon()) && result; 95 | result = testEqual(jor, "device_brand_icon_big", ret.getDeviceBrandIconBig()) && result; 96 | result = testEqual(jor, "device_brand_info_url", ret.getDeviceBrandInfoUrl()) && result; 97 | 98 | return result; 99 | } 100 | 101 | private static boolean testEqual(JsonObject jor, String test, String ret) { 102 | String expected = jor.getString(test); 103 | if (!expected.equals(ret) && expected.startsWith("https://")) { 104 | expected = expected.replaceAll(" ", "%20"); 105 | } 106 | if (!expected.equals(ret)) { 107 | System.out.println("Failed \"" + test + "\" : value=" + ret + " expected:" + expected); 108 | return false; 109 | } 110 | return true; 111 | } 112 | 113 | } 114 | -------------------------------------------------------------------------------- /src/test/java/org/udger/parser/test_ip.json: -------------------------------------------------------------------------------- 1 | [ 2 | { 3 | "test": { 4 | "teststring": "108.61.199.93" 5 | }, 6 | "ret": { 7 | "ip": "108.61.199.93", 8 | "ip_ver": "4", 9 | "ip_classification": "Crawler", 10 | "ip_classification_code": "crawler", 11 | "ip_hostname": "108.61.199.93.vultr.com", 12 | "ip_country": "Netherlands", 13 | "ip_country_code": "NL", 14 | "ip_city": "Amsterdam", 15 | "crawler_name": "PINGOMETER", 16 | "crawler_ver": "", 17 | "crawler_ver_major": "", 18 | "crawler_family": "PINGOMETER", 19 | "crawler_family_code": "pingometer", 20 | "crawler_family_homepage": "", 21 | "crawler_family_vendor": "Pingometer, LLC", 22 | "crawler_family_vendor_code": "pingometer_llc", 23 | "crawler_family_vendor_homepage": "http:\/\/pingometer.com\/", 24 | "crawler_family_icon": "bot_pingometer.png", 25 | "crawler_family_info_url": "https:\/\/udger.com\/resources\/ua-list\/bot-detail?bot=PINGOMETER#id20112", 26 | "crawler_category": "Site monitor", 27 | "crawler_category_code": "site_monitor", 28 | "crawler_respect_robotstxt": "no", 29 | "datacenter_name": "Choopa, LLC.", 30 | "datacenter_name_code": "choopa", 31 | "datacenter_homepage": "https:\/\/www.choopa.com\/" 32 | } 33 | }, 34 | { 35 | "test": { 36 | "teststring": "2a02:598:111::9" 37 | }, 38 | "ret": { 39 | "ip": "2a02:598:111::9", 40 | "ip_ver": "6", 41 | "ip_classification": "Crawler", 42 | "ip_classification_code": "crawler", 43 | "ip_hostname": "fulltextrobot-dev-2a02-598-111--9.seznam.cz", 44 | "ip_country": "Czech Republic", 45 | "ip_country_code": "CZ", 46 | "ip_city": "Prague", 47 | "crawler_name": "SeznamBot\/3.2-test1", 48 | "crawler_ver": "3.2-test1", 49 | "crawler_ver_major": "3", 50 | "crawler_family": "SeznamBot", 51 | "crawler_family_code": "seznambot", 52 | "crawler_family_homepage": "http:\/\/napoveda.seznam.cz\/en\/seznambot-intro\/", 53 | "crawler_family_vendor": "Seznam.cz, a.s.", 54 | "crawler_family_vendor_code": "seznam-cz_as", 55 | "crawler_family_vendor_homepage": "http:\/\/onas.seznam.cz\/", 56 | "crawler_family_icon": "seznam.png", 57 | "crawler_family_info_url": "https:\/\/udger.com\/resources\/ua-list\/bot-detail?bot=SeznamBot#id28914", 58 | "crawler_category": "Search engine bot", 59 | "crawler_category_code": "search_engine_bot", 60 | "crawler_respect_robotstxt": "unknown", 61 | "datacenter_name": "Seznam.cz", 62 | "datacenter_name_code": "seznam_cz", 63 | "datacenter_homepage": "http:\/\/onas.seznam.cz\/" 64 | } 65 | }, 66 | { 67 | "test": { 68 | "teststring": "2001:41d0:8:d54c::1" 69 | }, 70 | "ret": { 71 | "ip": "2001:41d0:8:d54c::1", 72 | "ip_ver": "6", 73 | "ip_classification": "Cgi proxy", 74 | "ip_classification_code": "cgi_proxy", 75 | "ip_hostname": "", 76 | "ip_country": "France", 77 | "ip_country_code": "FR", 78 | "ip_city": "Cachan", 79 | "crawler_name": "", 80 | "crawler_ver": "", 81 | "crawler_ver_major": "", 82 | "crawler_family": "", 83 | "crawler_family_code": "", 84 | "crawler_family_homepage": "", 85 | "crawler_family_vendor": "", 86 | "crawler_family_vendor_code": "", 87 | "crawler_family_vendor_homepage": "", 88 | "crawler_family_icon": "", 89 | "crawler_family_info_url": "", 90 | "crawler_category": "", 91 | "crawler_category_code": "", 92 | "crawler_respect_robotstxt": "", 93 | "datacenter_name": "OVH", 94 | "datacenter_name_code": "ovh", 95 | "datacenter_homepage": "http:\/\/www.ovh.com\/" 96 | } 97 | }, 98 | { 99 | "test": { 100 | "teststring": "66.249.64.73" 101 | }, 102 | "ret": { 103 | "ip": "66.249.64.73", 104 | "ip_ver": "4", 105 | "ip_classification": "Crawler", 106 | "ip_classification_code": "crawler", 107 | "ip_hostname": "crawl-66-249-64-73.googlebot.com", 108 | "ip_country": "United States", 109 | "ip_country_code": "US", 110 | "ip_city": "Mountain View", 111 | "crawler_name": "Googlebot\/2.1", 112 | "crawler_ver": "2.1", 113 | "crawler_ver_major": "2", 114 | "crawler_family": "Googlebot", 115 | "crawler_family_code": "googlebot", 116 | "crawler_family_homepage": "http:\/\/www.google.com\/bot.html", 117 | "crawler_family_vendor": "Google Inc.", 118 | "crawler_family_vendor_code": "google_inc", 119 | "crawler_family_vendor_homepage": "https:\/\/www.google.com\/about\/company\/", 120 | "crawler_family_icon": "bot_googlebot.png", 121 | "crawler_family_info_url": "https:\/\/udger.com\/resources\/ua-list\/bot-detail?bot=Googlebot#id31", 122 | "crawler_category": "Search engine bot", 123 | "crawler_category_code": "search_engine_bot", 124 | "crawler_respect_robotstxt": "yes", 125 | "datacenter_name": "Google sites", 126 | "datacenter_name_code": "googgle_sites", 127 | "datacenter_homepage": "http:\/\/sites.google.com\/" 128 | } 129 | }, 130 | { 131 | "test": { 132 | "teststring": "90.177.52.111" 133 | }, 134 | "ret": { 135 | "ip": "90.177.52.111", 136 | "ip_ver": "4", 137 | "ip_classification": "Unrecognized", 138 | "ip_classification_code": "unrecognized", 139 | "ip_hostname": "", 140 | "ip_country": "", 141 | "ip_country_code": "", 142 | "ip_city": "", 143 | "crawler_name": "", 144 | "crawler_ver": "", 145 | "crawler_ver_major": "", 146 | "crawler_family": "", 147 | "crawler_family_code": "", 148 | "crawler_family_homepage": "", 149 | "crawler_family_vendor": "", 150 | "crawler_family_vendor_code": "", 151 | "crawler_family_vendor_homepage": "", 152 | "crawler_family_icon": "", 153 | "crawler_family_info_url": "", 154 | "crawler_category": "", 155 | "crawler_category_code": "", 156 | "crawler_respect_robotstxt": "", 157 | "datacenter_name": "", 158 | "datacenter_name_code": "", 159 | "datacenter_homepage": "" 160 | } 161 | } 162 | ] -------------------------------------------------------------------------------- /src/test/java/org/udger/parser/test_ua.json: -------------------------------------------------------------------------------- 1 | [ 2 | { 3 | "test": { 4 | "teststring": "Mozilla\/5.0 (Windows NT 10.0; WOW64; rv:40.0) Gecko\/20100101 Firefox\/40.0" 5 | }, 6 | "ret": { 7 | "ua_string": "Mozilla\/5.0 (Windows NT 10.0; WOW64; rv:40.0) Gecko\/20100101 Firefox\/40.0", 8 | "ua_class": "Browser", 9 | "ua_class_code": "browser", 10 | "ua": "Firefox 40.0", 11 | "ua_version": "40.0", 12 | "ua_version_major": "40", 13 | "ua_uptodate_current_version": "50", 14 | "ua_family": "Firefox", 15 | "ua_family_code": "firefox", 16 | "ua_family_homepage": "http:\/\/www.firefox.com\/", 17 | "ua_family_vendor": "Mozilla Foundation", 18 | "ua_family_vendor_code": "mozilla_foundation", 19 | "ua_family_vendor_homepage": "http:\/\/www.mozilla.org\/", 20 | "ua_family_icon": "firefox.png", 21 | "ua_family_icon_big": "firefox_big.png", 22 | "ua_family_info_url": "https:\/\/udger.com\/resources\/ua-list\/browser-detail?browser=Firefox", 23 | "ua_engine": "Gecko", 24 | "os": "Windows 10", 25 | "os_code": "windows_10", 26 | "os_homepage": "https:\/\/en.wikipedia.org\/wiki\/Windows_10", 27 | "os_icon": "windows10.png", 28 | "os_icon_big": "windows10_big.png", 29 | "os_info_url": "https:\/\/udger.com\/resources\/ua-list\/os-detail?os=Windows 10", 30 | "os_family": "Windows", 31 | "os_family_code": "windows", 32 | "os_family_vendor": "Microsoft Corporation.", 33 | "os_family_vendor_code": "microsoft_corporation", 34 | "os_family_vendor_homepage": "https:\/\/www.microsoft.com\/about\/", 35 | "device_class": "Desktop", 36 | "device_class_code": "desktop", 37 | "device_class_icon": "desktop.png", 38 | "device_class_icon_big": "desktop_big.png", 39 | "device_class_info_url": "https:\/\/udger.com\/resources\/ua-list\/device-detail?device=Desktop", 40 | "device_marketname": "", 41 | "device_brand": "", 42 | "device_brand_code": "", 43 | "device_brand_homepage": "", 44 | "device_brand_icon": "", 45 | "device_brand_icon_big": "", 46 | "device_brand_info_url": "", 47 | "crawler_category": "", 48 | "crawler_category_code": "", 49 | "crawler_respect_robotstxt": "" 50 | } 51 | }, 52 | { 53 | "test": { 54 | "teststring": "Mozilla\/5.0 (compatible; SeznamBot\/3.2; +http:\/\/fulltext.sblog.cz\/)" 55 | }, 56 | "ret": { 57 | "ua_string": "Mozilla\/5.0 (compatible; SeznamBot\/3.2; +http:\/\/fulltext.sblog.cz\/)", 58 | "ua_class": "Crawler", 59 | "ua_class_code": "crawler", 60 | "ua": "SeznamBot\/3.2", 61 | "ua_version": "3.2", 62 | "ua_version_major": "3", 63 | "ua_uptodate_current_version": "", 64 | "ua_family": "SeznamBot", 65 | "ua_family_code": "seznambot", 66 | "ua_family_homepage": "http:\/\/napoveda.seznam.cz\/en\/seznambot-intro\/", 67 | "ua_family_vendor": "Seznam.cz, a.s.", 68 | "ua_family_vendor_code": "seznam-cz_as", 69 | "ua_family_vendor_homepage": "http:\/\/onas.seznam.cz\/", 70 | "ua_family_icon": "seznam.png", 71 | "ua_family_icon_big": "", 72 | "ua_family_info_url": "https:\/\/udger.com\/resources\/ua-list\/bot-detail?bot=SeznamBot#id12571", 73 | "ua_engine": "", 74 | "os": "", 75 | "os_code": "", 76 | "os_homepage": "", 77 | "os_icon": "", 78 | "os_icon_big": "", 79 | "os_info_url": "", 80 | "os_family": "", 81 | "os_family_code": "", 82 | "os_family_vendor": "", 83 | "os_family_vendor_code": "", 84 | "os_family_vendor_homepage": "", 85 | "device_class": "Unrecognized", 86 | "device_class_code": "unrecognized", 87 | "device_class_icon": "other.png", 88 | "device_class_icon_big": "other_big.png", 89 | "device_class_info_url": "https:\/\/udger.com\/resources\/ua-list\/device-detail?device=Unrecognized", 90 | "device_marketname": "", 91 | "device_brand": "", 92 | "device_brand_code": "", 93 | "device_brand_homepage": "", 94 | "device_brand_icon": "", 95 | "device_brand_icon_big": "", 96 | "device_brand_info_url": "", 97 | "crawler_category": "Search engine bot", 98 | "crawler_category_code": "search_engine_bot", 99 | "crawler_respect_robotstxt": "yes" 100 | } 101 | }, 102 | { 103 | "test": { 104 | "teststring": "Mozilla\/5.0 (iPad; CPU OS 7_0 like Mac OS X) AppleWebKit\/537.51.1 (KHTML, like Gecko) Version\/7.0 Mobile\/11A465 Safari\/9537.53" 105 | }, 106 | "ret": { 107 | "ua_string": "Mozilla\/5.0 (iPad; CPU OS 7_0 like Mac OS X) AppleWebKit\/537.51.1 (KHTML, like Gecko) Version\/7.0 Mobile\/11A465 Safari\/9537.53", 108 | "ua_class": "Mobile browser", 109 | "ua_class_code": "mobile_browser", 110 | "ua": "Safari mobile 7.0", 111 | "ua_version": "7.0", 112 | "ua_version_major": "7", 113 | "ua_uptodate_current_version": "", 114 | "ua_family": "Safari mobile", 115 | "ua_family_code": "safari_mobile", 116 | "ua_family_homepage": "https:\/\/en.wikipedia.org\/wiki\/Safari_%28web_browser%29", 117 | "ua_family_vendor": "Apple Inc.", 118 | "ua_family_vendor_code": "apple_inc", 119 | "ua_family_vendor_homepage": "http:\/\/www.apple.com\/", 120 | "ua_family_icon": "safari.png", 121 | "ua_family_icon_big": "safari_big.png", 122 | "ua_family_info_url": "https:\/\/udger.com\/resources\/ua-list\/browser-detail?browser=Safari mobile", 123 | "ua_engine": "WebKit", 124 | "os": "iOS 7", 125 | "os_code": "ios_7", 126 | "os_homepage": "https:\/\/en.wikipedia.org\/wiki\/IOS_7", 127 | "os_icon": "iphone.png", 128 | "os_icon_big": "iphone_big.png", 129 | "os_info_url": "https:\/\/udger.com\/resources\/ua-list\/os-detail?os=iOS 7", 130 | "os_family": "iOS", 131 | "os_family_code": "ios", 132 | "os_family_vendor": "Apple Inc.", 133 | "os_family_vendor_code": "apple_inc", 134 | "os_family_vendor_homepage": "http:\/\/www.apple.com\/", 135 | "device_class": "Tablet", 136 | "device_class_code": "tablet", 137 | "device_class_icon": "tablet.png", 138 | "device_class_icon_big": "tablet_big.png", 139 | "device_class_info_url": "https:\/\/udger.com\/resources\/ua-list\/device-detail?device=Tablet", 140 | "device_marketname": "iPad", 141 | "device_brand": "Apple", 142 | "device_brand_code": "apple", 143 | "device_brand_homepage": "http:\/\/www.apple.com\/", 144 | "device_brand_icon": "apple.png", 145 | "device_brand_icon_big": "apple_big.png", 146 | "device_brand_info_url": "https:\/\/udger.com\/resources\/ua-list\/devices-brand-detail?brand=apple", 147 | "crawler_category": "", 148 | "crawler_category_code": "", 149 | "crawler_respect_robotstxt": "" 150 | } 151 | }, 152 | { 153 | "test": { 154 | "teststring": "Mozilla\/5.0 (Linux; U; Android 4.0.4; sk-sk; Luna TAB474 Build\/LunaTAB474) AppleWebKit\/534.30 (KHTML, like Gecko) Version\/4.0 Safari\/534.30" 155 | }, 156 | "ret": { 157 | "ua_string": "Mozilla\/5.0 (Linux; U; Android 4.0.4; sk-sk; Luna TAB474 Build\/LunaTAB474) AppleWebKit\/534.30 (KHTML, like Gecko) Version\/4.0 Safari\/534.30", 158 | "ua_class": "Mobile browser", 159 | "ua_class_code": "mobile_browser", 160 | "ua": "Android browser 4.0", 161 | "ua_version": "4.0", 162 | "ua_version_major": "4", 163 | "ua_uptodate_current_version": "", 164 | "ua_family": "Android browser", 165 | "ua_family_code": "android_browser", 166 | "ua_family_homepage": "http:\/\/developer.android.com\/reference\/android\/webkit\/package-summary.html", 167 | "ua_family_vendor": "Google Inc.", 168 | "ua_family_vendor_code": "google_inc", 169 | "ua_family_vendor_homepage": "https:\/\/www.google.com\/about\/company\/", 170 | "ua_family_icon": "androidWebkit.png", 171 | "ua_family_icon_big": "androidWebkit_big.png", 172 | "ua_family_info_url": "https:\/\/udger.com\/resources\/ua-list\/browser-detail?browser=Android browser", 173 | "ua_engine": "WebKit", 174 | "os": "Android 4.0.x Ice Cream Sandwich", 175 | "os_code": "android_4", 176 | "os_homepage": "https:\/\/en.wikipedia.org\/wiki\/Android_%28operating_system%29", 177 | "os_icon": "android.png", 178 | "os_icon_big": "android_big.png", 179 | "os_info_url": "https:\/\/udger.com\/resources\/ua-list\/os-detail?os=Android 4.0.x Ice Cream Sandwich", 180 | "os_family": "Android", 181 | "os_family_code": "android", 182 | "os_family_vendor": "Google, Inc.", 183 | "os_family_vendor_code": "google_inc", 184 | "os_family_vendor_homepage": "https:\/\/www.google.com\/about\/company\/", 185 | "device_class": "Tablet", 186 | "device_class_code": "tablet", 187 | "device_class_icon": "tablet.png", 188 | "device_class_icon_big": "tablet_big.png", 189 | "device_class_info_url": "https:\/\/udger.com\/resources\/ua-list\/device-detail?device=Tablet", 190 | "device_marketname": "Luna 10", 191 | "device_brand": "Yarvik", 192 | "device_brand_code": "yarvik", 193 | "device_brand_homepage": "http:\/\/yarvik.com\/", 194 | "device_brand_icon": "yarvik.png", 195 | "device_brand_icon_big": "", 196 | "device_brand_info_url": "https:\/\/udger.com\/resources\/ua-list\/devices-brand-detail?brand=yarvik", 197 | "crawler_category": "", 198 | "crawler_category_code": "", 199 | "crawler_respect_robotstxt": "" 200 | } 201 | }, 202 | { 203 | "test": { 204 | "teststring": "Mozilla\/5.0 (Playstation Vita 1.61) AppleWebKit\/531.22.8 (KHTML, like Gecko) Silk\/3.2" 205 | }, 206 | "ret": { 207 | "ua_string": "Mozilla\/5.0 (Playstation Vita 1.61) AppleWebKit\/531.22.8 (KHTML, like Gecko) Silk\/3.2", 208 | "ua_class": "Mobile browser", 209 | "ua_class_code": "mobile_browser", 210 | "ua": "PS Vita browser 3.2", 211 | "ua_version": "3.2", 212 | "ua_version_major": "3", 213 | "ua_uptodate_current_version": "", 214 | "ua_family": "PS Vita browser", 215 | "ua_family_code": "ps_vita_browser", 216 | "ua_family_homepage": "", 217 | "ua_family_vendor": "Sony Computer Entertainment", 218 | "ua_family_vendor_code": "sony_computer_entertainment", 219 | "ua_family_vendor_homepage": "http:\/\/www.scei.co.jp\/", 220 | "ua_family_icon": "ps-vita-browser.png", 221 | "ua_family_icon_big": "", 222 | "ua_family_info_url": "https:\/\/udger.com\/resources\/ua-list\/browser-detail?browser=PS Vita browser", 223 | "ua_engine": "WebKit", 224 | "os": "LiveArea", 225 | "os_code": "live_area", 226 | "os_homepage": "https:\/\/en.wikipedia.org\/wiki\/LiveArea", 227 | "os_icon": "ps-vitaLiveArea.png", 228 | "os_icon_big": "", 229 | "os_info_url": "https:\/\/udger.com\/resources\/ua-list\/os-detail?os=LiveArea", 230 | "os_family": "LiveArea", 231 | "os_family_code": "live_area", 232 | "os_family_vendor": "Sony Computer Entertainment", 233 | "os_family_vendor_code": "sony", 234 | "os_family_vendor_homepage": "http:\/\/www.scei.co.jp\/", 235 | "device_class": "Game console", 236 | "device_class_code": "game_console", 237 | "device_class_icon": "console.png", 238 | "device_class_icon_big": "console_big.png", 239 | "device_class_info_url": "https:\/\/udger.com\/resources\/ua-list\/device-detail?device=Game console", 240 | "device_marketname": "Playstation Vita", 241 | "device_brand": "Sony", 242 | "device_brand_code": "sony", 243 | "device_brand_homepage": "http:\/\/www.sony.com\/", 244 | "device_brand_icon": "sony.png", 245 | "device_brand_icon_big": "sony_big.png", 246 | "device_brand_info_url": "https:\/\/udger.com\/resources\/ua-list\/devices-brand-detail?brand=sony", 247 | "crawler_category": "", 248 | "crawler_category_code": "", 249 | "crawler_respect_robotstxt": "" 250 | } 251 | }, 252 | { 253 | "test": { 254 | "teststring": "Mozilla\/5.0 (CrKey armv7l 1.4.15250) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/31.0.1650.0 Safari\/537.36" 255 | }, 256 | "ret": { 257 | "ua_string": "Mozilla\/5.0 (CrKey armv7l 1.4.15250) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/31.0.1650.0 Safari\/537.36", 258 | "ua_class": "Browser", 259 | "ua_class_code": "browser", 260 | "ua": "Chrome 31.0.1650.0", 261 | "ua_version": "31.0.1650.0", 262 | "ua_version_major": "31", 263 | "ua_uptodate_current_version": "55", 264 | "ua_family": "Chrome", 265 | "ua_family_code": "chrome", 266 | "ua_family_homepage": "http:\/\/www.google.com\/chrome\/", 267 | "ua_family_vendor": "Google Inc.", 268 | "ua_family_vendor_code": "google_inc", 269 | "ua_family_vendor_homepage": "https:\/\/www.google.com\/about\/company\/", 270 | "ua_family_icon": "chrome.png", 271 | "ua_family_icon_big": "chrome_big.png", 272 | "ua_family_info_url": "https:\/\/udger.com\/resources\/ua-list\/browser-detail?browser=Chrome", 273 | "ua_engine": "WebKit\/Blink", 274 | "os": "Android", 275 | "os_code": "android", 276 | "os_homepage": "https:\/\/en.wikipedia.org\/wiki\/Android_%28operating_system%29", 277 | "os_icon": "android.png", 278 | "os_icon_big": "android_big.png", 279 | "os_info_url": "https:\/\/udger.com\/resources\/ua-list\/os-detail?os=Android", 280 | "os_family": "Android", 281 | "os_family_code": "android", 282 | "os_family_vendor": "Google, Inc.", 283 | "os_family_vendor_code": "google_inc", 284 | "os_family_vendor_homepage": "https:\/\/www.google.com\/about\/company\/", 285 | "device_class": "Smart TV", 286 | "device_class_code": "smart_tv", 287 | "device_class_icon": "smarttv.png", 288 | "device_class_icon_big": "smarttv_big.png", 289 | "device_class_info_url": "https:\/\/udger.com\/resources\/ua-list\/device-detail?device=Smart TV", 290 | "device_marketname": "Chromecast", 291 | "device_brand": "Google", 292 | "device_brand_code": "google", 293 | "device_brand_homepage": "https:\/\/www.google.com\/", 294 | "device_brand_icon": "google.png", 295 | "device_brand_icon_big": "google_big.png", 296 | "device_brand_info_url": "https:\/\/udger.com\/resources\/ua-list\/devices-brand-detail?brand=google", 297 | "crawler_category": "", 298 | "crawler_category_code": "", 299 | "crawler_respect_robotstxt": "" 300 | } 301 | }, 302 | { 303 | "test": { 304 | "teststring": "Mozilla\/5.0 (compatible; MSIE 10.0; Windows Phone 8.0; Trident\/6.0; IEMobile\/10.0; ARM; Touch; NOKIA; Lumia 820)" 305 | }, 306 | "ret": { 307 | "ua_string": "Mozilla\/5.0 (compatible; MSIE 10.0; Windows Phone 8.0; Trident\/6.0; IEMobile\/10.0; ARM; Touch; NOKIA; Lumia 820)", 308 | "ua_class": "Mobile browser", 309 | "ua_class_code": "mobile_browser", 310 | "ua": "IE Mobile 10.0", 311 | "ua_version": "10.0", 312 | "ua_version_major": "10", 313 | "ua_uptodate_current_version": "", 314 | "ua_family": "IE Mobile", 315 | "ua_family_code": "ie_mobile", 316 | "ua_family_homepage": "https:\/\/en.wikipedia.org\/wiki\/Internet_Explorer_Mobile", 317 | "ua_family_vendor": "Microsoft Corporation.", 318 | "ua_family_vendor_code": "microsoft_corporation", 319 | "ua_family_vendor_homepage": "https:\/\/www.microsoft.com\/about\/", 320 | "ua_family_icon": "iemobile.png", 321 | "ua_family_icon_big": "", 322 | "ua_family_info_url": "https:\/\/udger.com\/resources\/ua-list\/browser-detail?browser=IE Mobile", 323 | "ua_engine": "Trident", 324 | "os": "Windows Phone 8", 325 | "os_code": "windows_phone_8", 326 | "os_homepage": "https:\/\/en.wikipedia.org\/wiki\/Windows_Phone_8", 327 | "os_icon": "windowsPhone8.png", 328 | "os_icon_big": "", 329 | "os_info_url": "https:\/\/udger.com\/resources\/ua-list\/os-detail?os=Windows Phone 8", 330 | "os_family": "Windows", 331 | "os_family_code": "windows", 332 | "os_family_vendor": "Microsoft Corporation.", 333 | "os_family_vendor_code": "microsoft_corporation", 334 | "os_family_vendor_homepage": "https:\/\/www.microsoft.com\/about\/", 335 | "device_class": "Smartphone", 336 | "device_class_code": "smartphone", 337 | "device_class_icon": "phone.png", 338 | "device_class_icon_big": "phone_big.png", 339 | "device_class_info_url": "https:\/\/udger.com\/resources\/ua-list\/device-detail?device=Smartphone", 340 | "device_marketname": "Lumia 820", 341 | "device_brand": "Nokia", 342 | "device_brand_code": "nokia", 343 | "device_brand_homepage": "http:\/\/www.nokia.com\/", 344 | "device_brand_icon": "nokia.png", 345 | "device_brand_icon_big": "nokia_big.png", 346 | "device_brand_info_url": "https:\/\/udger.com\/resources\/ua-list\/devices-brand-detail?brand=nokia", 347 | "crawler_category": "", 348 | "crawler_category_code": "", 349 | "crawler_respect_robotstxt": "" 350 | } 351 | }, 352 | { 353 | "test": { 354 | "teststring": "Mozilla\/5.0 (Linux; U; Android 4.0.4; en-us; Glass 1 Build\/IMM76L; XE9) AppleWebKit\/534.30 (KHTML, like Gecko) Version\/4.0 Mobile Safari\/534.30" 355 | }, 356 | "ret": { 357 | "ua_string": "Mozilla\/5.0 (Linux; U; Android 4.0.4; en-us; Glass 1 Build\/IMM76L; XE9) AppleWebKit\/534.30 (KHTML, like Gecko) Version\/4.0 Mobile Safari\/534.30", 358 | "ua_class": "Mobile browser", 359 | "ua_class_code": "mobile_browser", 360 | "ua": "Android browser 4.0", 361 | "ua_version": "4.0", 362 | "ua_version_major": "4", 363 | "ua_uptodate_current_version": "", 364 | "ua_family": "Android browser", 365 | "ua_family_code": "android_browser", 366 | "ua_family_homepage": "http:\/\/developer.android.com\/reference\/android\/webkit\/package-summary.html", 367 | "ua_family_vendor": "Google Inc.", 368 | "ua_family_vendor_code": "google_inc", 369 | "ua_family_vendor_homepage": "https:\/\/www.google.com\/about\/company\/", 370 | "ua_family_icon": "androidWebkit.png", 371 | "ua_family_icon_big": "androidWebkit_big.png", 372 | "ua_family_info_url": "https:\/\/udger.com\/resources\/ua-list\/browser-detail?browser=Android browser", 373 | "ua_engine": "WebKit", 374 | "os": "Android 4.0.x Ice Cream Sandwich", 375 | "os_code": "android_4", 376 | "os_homepage": "https:\/\/en.wikipedia.org\/wiki\/Android_%28operating_system%29", 377 | "os_icon": "android.png", 378 | "os_icon_big": "android_big.png", 379 | "os_info_url": "https:\/\/udger.com\/resources\/ua-list\/os-detail?os=Android 4.0.x Ice Cream Sandwich", 380 | "os_family": "Android", 381 | "os_family_code": "android", 382 | "os_family_vendor": "Google, Inc.", 383 | "os_family_vendor_code": "google_inc", 384 | "os_family_vendor_homepage": "https:\/\/www.google.com\/about\/company\/", 385 | "device_class": "Wearable computer", 386 | "device_class_code": "wearable_computer", 387 | "device_class_icon": "wearable.png", 388 | "device_class_icon_big": "wearable_big.png", 389 | "device_class_info_url": "https:\/\/udger.com\/resources\/ua-list\/device-detail?device=Wearable computer", 390 | "device_marketname": "Google Glass", 391 | "device_brand": "Google", 392 | "device_brand_code": "google", 393 | "device_brand_homepage": "https:\/\/www.google.com\/", 394 | "device_brand_icon": "google.png", 395 | "device_brand_icon_big": "google_big.png", 396 | "device_brand_info_url": "https:\/\/udger.com\/resources\/ua-list\/devices-brand-detail?brand=google", 397 | "crawler_category": "", 398 | "crawler_category_code": "", 399 | "crawler_respect_robotstxt": "" 400 | } 401 | }, 402 | { 403 | "test": { 404 | "teststring": "Microsoft Office\/16.0 (Windows NT 10.0; Microsoft Outlook 16.0.6326; Pro)" 405 | }, 406 | "ret": { 407 | "ua_string": "Microsoft Office\/16.0 (Windows NT 10.0; Microsoft Outlook 16.0.6326; Pro)", 408 | "ua_class": "E-mail client", 409 | "ua_class_code": "email_client", 410 | "ua": "Outlook 2016 ", 411 | "ua_version": "", 412 | "ua_version_major": "", 413 | "ua_uptodate_current_version": "", 414 | "ua_family": "Outlook 2016", 415 | "ua_family_code": "outlook_2016", 416 | "ua_family_homepage": "https:\/\/en.wikipedia.org\/wiki\/Microsoft_Outlook", 417 | "ua_family_vendor": "Microsoft Corporation.", 418 | "ua_family_vendor_code": "microsoft_corporation", 419 | "ua_family_vendor_homepage": "https:\/\/www.microsoft.com\/about\/", 420 | "ua_family_icon": "outlook2016.png", 421 | "ua_family_icon_big": "", 422 | "ua_family_info_url": "https:\/\/udger.com\/resources\/ua-list\/browser-detail?browser=Outlook 2016", 423 | "ua_engine": "", 424 | "os": "Windows 10", 425 | "os_code": "windows_10", 426 | "os_homepage": "https:\/\/en.wikipedia.org\/wiki\/Windows_10", 427 | "os_icon": "windows10.png", 428 | "os_icon_big": "windows10_big.png", 429 | "os_info_url": "https:\/\/udger.com\/resources\/ua-list\/os-detail?os=Windows 10", 430 | "os_family": "Windows", 431 | "os_family_code": "windows", 432 | "os_family_vendor": "Microsoft Corporation.", 433 | "os_family_vendor_code": "microsoft_corporation", 434 | "os_family_vendor_homepage": "https:\/\/www.microsoft.com\/about\/", 435 | "device_class": "Desktop", 436 | "device_class_code": "desktop", 437 | "device_class_icon": "desktop.png", 438 | "device_class_icon_big": "desktop_big.png", 439 | "device_class_info_url": "https:\/\/udger.com\/resources\/ua-list\/device-detail?device=Desktop", 440 | "device_marketname": "", 441 | "device_brand": "", 442 | "device_brand_code": "", 443 | "device_brand_homepage": "", 444 | "device_brand_icon": "", 445 | "device_brand_icon_big": "", 446 | "device_brand_info_url": "", 447 | "crawler_category": "", 448 | "crawler_category_code": "", 449 | "crawler_respect_robotstxt": "" 450 | } 451 | }, 452 | { 453 | "test": { 454 | "teststring": "Mozilla\/5.0 (compatible; Googlebot\/2.1; +http:\/\/www.google.com\/bot.html)" 455 | }, 456 | "ret": { 457 | "ua_string": "Mozilla\/5.0 (compatible; Googlebot\/2.1; +http:\/\/www.google.com\/bot.html)", 458 | "ua_class": "Crawler", 459 | "ua_class_code": "crawler", 460 | "ua": "Googlebot\/2.1", 461 | "ua_version": "2.1", 462 | "ua_version_major": "2", 463 | "ua_uptodate_current_version": "", 464 | "ua_family": "Googlebot", 465 | "ua_family_code": "googlebot", 466 | "ua_family_homepage": "http:\/\/www.google.com\/bot.html", 467 | "ua_family_vendor": "Google Inc.", 468 | "ua_family_vendor_code": "google_inc", 469 | "ua_family_vendor_homepage": "https:\/\/www.google.com\/about\/company\/", 470 | "ua_family_icon": "bot_googlebot.png", 471 | "ua_family_icon_big": "", 472 | "ua_family_info_url": "https:\/\/udger.com\/resources\/ua-list\/bot-detail?bot=Googlebot#id31", 473 | "ua_engine": "", 474 | "os": "", 475 | "os_code": "", 476 | "os_homepage": "", 477 | "os_icon": "", 478 | "os_icon_big": "", 479 | "os_info_url": "", 480 | "os_family": "", 481 | "os_family_code": "", 482 | "os_family_vendor": "", 483 | "os_family_vendor_code": "", 484 | "os_family_vendor_homepage": "", 485 | "device_class": "Unrecognized", 486 | "device_class_code": "unrecognized", 487 | "device_class_icon": "other.png", 488 | "device_class_icon_big": "other_big.png", 489 | "device_class_info_url": "https:\/\/udger.com\/resources\/ua-list\/device-detail?device=Unrecognized", 490 | "device_marketname": "", 491 | "device_brand": "", 492 | "device_brand_code": "", 493 | "device_brand_homepage": "", 494 | "device_brand_icon": "", 495 | "device_brand_icon_big": "", 496 | "device_brand_info_url": "", 497 | "crawler_category": "Search engine bot", 498 | "crawler_category_code": "search_engine_bot", 499 | "crawler_respect_robotstxt": "yes" 500 | } 501 | }, 502 | { 503 | "test": { 504 | "teststring": "PINGOMETER_BOT_(HTTPS:\/\/PINGOMETER.COM)" 505 | }, 506 | "ret": { 507 | "ua_string": "PINGOMETER_BOT_(HTTPS:\/\/PINGOMETER.COM)", 508 | "ua_class": "Crawler", 509 | "ua_class_code": "crawler", 510 | "ua": "PINGOMETER", 511 | "ua_version": "", 512 | "ua_version_major": "", 513 | "ua_uptodate_current_version": "", 514 | "ua_family": "PINGOMETER", 515 | "ua_family_code": "pingometer", 516 | "ua_family_homepage": "", 517 | "ua_family_vendor": "Pingometer, LLC", 518 | "ua_family_vendor_code": "pingometer_llc", 519 | "ua_family_vendor_homepage": "http:\/\/pingometer.com\/", 520 | "ua_family_icon": "bot_pingometer.png", 521 | "ua_family_icon_big": "", 522 | "ua_family_info_url": "https:\/\/udger.com\/resources\/ua-list\/bot-detail?bot=PINGOMETER#id20112", 523 | "ua_engine": "", 524 | "os": "", 525 | "os_code": "", 526 | "os_homepage": "", 527 | "os_icon": "", 528 | "os_icon_big": "", 529 | "os_info_url": "", 530 | "os_family": "", 531 | "os_family_code": "", 532 | "os_family_vendor": "", 533 | "os_family_vendor_code": "", 534 | "os_family_vendor_homepage": "", 535 | "device_class": "Unrecognized", 536 | "device_class_code": "unrecognized", 537 | "device_class_icon": "other.png", 538 | "device_class_icon_big": "other_big.png", 539 | "device_class_info_url": "https:\/\/udger.com\/resources\/ua-list\/device-detail?device=Unrecognized", 540 | "device_marketname": "", 541 | "device_brand": "", 542 | "device_brand_code": "", 543 | "device_brand_homepage": "", 544 | "device_brand_icon": "", 545 | "device_brand_icon_big": "", 546 | "device_brand_info_url": "", 547 | "crawler_category": "Site monitor", 548 | "crawler_category_code": "site_monitor", 549 | "crawler_respect_robotstxt": "no" 550 | } 551 | }, 552 | { 553 | "test": { 554 | "teststring": "Monogram\/1.3 CFNetwork\/758.2.8 Darwin\/15.0.0" 555 | }, 556 | "ret": { 557 | "ua_string": "Monogram\/1.3 CFNetwork\/758.2.8 Darwin\/15.0.0", 558 | "ua_class": "Unrecognized", 559 | "ua_class_code": "unrecognized", 560 | "ua": "", 561 | "ua_version": "", 562 | "ua_version_major": "", 563 | "ua_uptodate_current_version": "", 564 | "ua_family": "", 565 | "ua_family_code": "", 566 | "ua_family_homepage": "", 567 | "ua_family_vendor": "", 568 | "ua_family_vendor_code": "", 569 | "ua_family_vendor_homepage": "", 570 | "ua_family_icon": "", 571 | "ua_family_icon_big": "", 572 | "ua_family_info_url": "", 573 | "ua_engine": "", 574 | "os": "iOS 9", 575 | "os_code": "ios_9", 576 | "os_homepage": "https:\/\/en.wikipedia.org\/wiki\/IOS_9", 577 | "os_icon": "iphone.png", 578 | "os_icon_big": "iphone_big.png", 579 | "os_info_url": "https:\/\/udger.com\/resources\/ua-list\/os-detail?os=iOS 9", 580 | "os_family": "iOS", 581 | "os_family_code": "ios", 582 | "os_family_vendor": "Apple Inc.", 583 | "os_family_vendor_code": "apple_inc", 584 | "os_family_vendor_homepage": "http:\/\/www.apple.com\/", 585 | "device_class": "", 586 | "device_class_code": "", 587 | "device_class_icon": "", 588 | "device_class_icon_big": "", 589 | "device_class_info_url": "", 590 | "device_marketname": "", 591 | "device_brand": "", 592 | "device_brand_code": "", 593 | "device_brand_homepage": "", 594 | "device_brand_icon": "", 595 | "device_brand_icon_big": "", 596 | "device_brand_info_url": "", 597 | "crawler_category": "", 598 | "crawler_category_code": "", 599 | "crawler_respect_robotstxt": "" 600 | } 601 | }, 602 | { 603 | "test": { 604 | "teststring": "lorem ipsum dolor sit amet" 605 | }, 606 | "ret": { 607 | "ua_string": "lorem ipsum dolor sit amet", 608 | "ua_class": "Unrecognized", 609 | "ua_class_code": "unrecognized", 610 | "ua": "", 611 | "ua_version": "", 612 | "ua_version_major": "", 613 | "ua_uptodate_current_version": "", 614 | "ua_family": "", 615 | "ua_family_code": "", 616 | "ua_family_homepage": "", 617 | "ua_family_vendor": "", 618 | "ua_family_vendor_code": "", 619 | "ua_family_vendor_homepage": "", 620 | "ua_family_icon": "", 621 | "ua_family_icon_big": "", 622 | "ua_family_info_url": "", 623 | "ua_engine": "", 624 | "os": "", 625 | "os_code": "", 626 | "os_homepage": "", 627 | "os_icon": "", 628 | "os_icon_big": "", 629 | "os_info_url": "", 630 | "os_family": "", 631 | "os_family_code": "", 632 | "os_family_vendor": "", 633 | "os_family_vendor_code": "", 634 | "os_family_vendor_homepage": "", 635 | "device_class": "", 636 | "device_class_code": "", 637 | "device_class_icon": "", 638 | "device_class_icon_big": "", 639 | "device_class_info_url": "", 640 | "device_marketname": "", 641 | "device_brand": "", 642 | "device_brand_code": "", 643 | "device_brand_homepage": "", 644 | "device_brand_icon": "", 645 | "device_brand_icon_big": "", 646 | "device_brand_info_url": "", 647 | "crawler_category": "", 648 | "crawler_category_code": "", 649 | "crawler_respect_robotstxt": "" 650 | } 651 | } 652 | ] -------------------------------------------------------------------------------- /src/test/resources/udgerdb_test_v3.dat: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udger/udger-java/c055a6d7f23de6e4e1a478386b0b413e2683b99e/src/test/resources/udgerdb_test_v3.dat -------------------------------------------------------------------------------- /src/test/resources/udgerdb_test_v3_switch.dat: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/udger/udger-java/c055a6d7f23de6e4e1a478386b0b413e2683b99e/src/test/resources/udgerdb_test_v3_switch.dat --------------------------------------------------------------------------------