End Point:
Default:
Send a '
Example:
24 |The server responds with a JSON object containing two fields.
28 |On success, field '
On error, field '
After starting the query process, monitor its status using the
Example:
48 |
7 |
8 | An open source [Translation Memory](https://en.wikipedia.org/wiki/Translation_memory) Engine written in Java.
9 |
10 | TMEngine is based on the translation memory library used by [Swordfish III](https://www.maxprograms.com/products/swordfish.html), [Fluenta](https://www.maxprograms.com/products/fluenta.html) and [RemoteTM](https://www.maxprograms.com/products/remotetm.html).
11 |
12 | TMEngine can be used either as an embedded library that manages translation memories in a Java application or as a standalone TM server via its REST API.
13 |
14 | ## Requirements
15 |
16 | - JDK 11 or newer is required for compiling and building.
17 | - Apache Ant 1.10.6 or newer
18 |
19 | ## Building
20 |
21 | - Checkout this repository.
22 | - Point your JAVA_HOME variable to JDK 11
23 | - Run `ant compile`.
24 |
25 | ## Downloads
26 |
27 | Ready to use distributions are available at [https://www.maxprograms.com/products/tmengine.html](https://www.maxprograms.com/products/tmengine.html).
28 |
29 | ## Related Links
30 |
31 | - [TMEngine Manual (PDF)](https://www.maxprograms.com/support/tmengine.pdf)
32 | - [TMEgine Manual (Web Help)](https://www.maxprograms.com/support/tmengine.html)
33 |
34 | ## Standalone Server
35 |
36 | Running `.\tmserver.bat` or `./tmserver.sh` without parameters displays help for starting TMEngine as a standalone server.
37 | ```
38 | Usage:
39 |
40 | tmserver.sh [-help] [-version] [-port portNumber]
41 |
42 | Where:
43 |
44 | -help: (optional) Display this help information and exit
45 | -version: (optional) Display version & build information and exit
46 | -port: (optional) Port for running HTTP server. Default is 8000
47 | ```
48 |
49 | Visit http://localhost:8000/TMServer/stop to stop the server. Adjust the port number if required.
50 |
51 | ## Java Library
52 |
53 | TMEngine can be embedded in Java applications that need to deal with Translation Memory data.
54 |
55 | Add all .jar files from `/lib` folder to the classpath of your application and use instances of `ITmEngine` interface.
56 |
57 | Two classes implement interface `ITmEngine`:
58 |
59 | - `MapDbEngine`: a translation memory engine built using [MapDB](http://mapdb.org)
60 | - `SQLEngine`: an implementation designed to be used with [MariaDB](https://mariadb.org/) or [MySQL](https://www.mysql.com/)
61 |
62 | See more details on the available Java methods in the [documentation](https://www.maxprograms.com/support/tmengine/TMEngine.html).
63 |
--------------------------------------------------------------------------------
/docs/concepts/rest/ProcessStatus.dita:
--------------------------------------------------------------------------------
1 |
2 |
3 | End Point:
Default:
Send a
Example:
25 |The server responds with a JSON object.
29 |On successful status check, field '
Example:
33 | 34 | 35 |Field '
-
38 |
- Pending: processing is still going on.
39 |
{ 40 | "result": "Pending", 41 | "status": "OK" 42 | }
43 | - Completed: processing has finished. If the process produces any data, it is
44 | placed in the '
data ' field. 45 |{ 46 | "result": "Completed", 47 | "data": { 48 | "imported": "57678" 49 | }, 50 | "status": "OK" 51 | }
52 | - Failed: processing failed. Failure reason is provided in
53 | '
reason ' 54 | field.{ 55 | "result": "Failed", 56 | "reason": "/Volumes/Data/something.tmx (No such file or directory)", 57 | "status": "failed" 58 | } 59 |
60 |
If process status cannot be checked, the server omits the '
End Point:
Default:
Send a '
Example:
48 |The server responds with a JSON object containing two fields.
65 |On success, field '
On error, field '
After starting the export process, monitor its status using the
End Point:
Default:
Send a '
Example:
48 |The server responds with a JSON object containing two fields.
55 |On success, field '
On error, field '
After starting the import process, monitor its status using the
Example:
74 | 75 |End Point:
Default:
Send a '
ID of the memory where the search should be performed
22 |Example:
53 |On success, field '
On error, field '
After starting the search process, monitor its status using the
On successful completion, the result will contain an array of <tu> elements that
79 | contain the searched text in the
Example:
81 |End Point:
Default:
Send a '
ID of the memory to create. The value of '
Default value is current server time represented as the number of 25 | milliseconds since January 1, 1970, 00:00:00 GMT
26 |Text string used to identify the owner of the memory.
38 |Default value is the login name of the user running the server.
39 |Type of engine to use. Possible values are:
-
46 |
- '
MapDbEngine ' (default)
47 | - '
SQLEngine '
48 |
Name or IP of the server running MySQL or MariaDB.
57 |Required for
Port in which MySQL or MariaDB listens for requests.
66 |Required for
ID of of the MySQL or MariaDB user creating the database.
75 |Required for
Password of the MySQL or MariaDB user creating the database.
83 |Required for
Example:
89 |The server responds with a JSON object containing two fields.
103 |On success, field '
Example:
107 |On error, field '
Example:
116 |End Point:
Default:
Send a '
ID of the memory where the search should be performed
22 |Example:
53 |The server responds with a JSON object containing two fields.
62 |On success, field '
On error, field '
After starting the search process, monitor its status using the
On successful completion, the result will contain an array of similar segments in the
80 |
Example:
82 | 83 |59 | if (type.equals("sub") || type.equals("hi")) { 60 | Element e = (Element) o; 61 | text.append(pureText(e)); 62 | } 63 | } 64 | } 65 | return text.toString(); 66 | } 67 | 68 | public static String normalizeLang(String langCode) throws IOException { 69 | if (registry == null) { 70 | registry = new RegistryParser(); 71 | } 72 | if (langCode == null) { 73 | return null; 74 | } 75 | if (!registry.getTagDescription(langCode).isEmpty()) { 76 | return langCode; 77 | } 78 | String lang = langCode.replaceAll("_", "-"); 79 | String[] parts = lang.split("-"); 80 | 81 | if (parts.length == 2) { 82 | if (parts[1].length() == 2) { 83 | // has country code 84 | String code = lang.substring(0, 2).toLowerCase() + "-" + lang.substring(3).toUpperCase(); 85 | if (!registry.getTagDescription(code).isEmpty()) { 86 | return code; 87 | } 88 | return null; 89 | } 90 | if (isRegion(parts[1])) { 91 | // try lowercasing language code while keeping region number 92 | String code = lang.substring(0, 2).toLowerCase() + "-" + parts[1]; 93 | if (!registry.getTagDescription(code).isEmpty()) { 94 | return code; 95 | } 96 | return null; 97 | } 98 | // may have a script 99 | String code = lang.substring(0, 2).toLowerCase() + "-" + lang.substring(3, 4).toUpperCase() 100 | + lang.substring(4).toLowerCase(); 101 | if (!registry.getTagDescription(code).isEmpty()) { 102 | return code; 103 | } 104 | return null; 105 | } 106 | // check if its a valid thing with more than 2 parts 107 | if (!registry.getTagDescription(lang).isEmpty()) { 108 | return lang; 109 | } 110 | return null; 111 | } 112 | 113 | private static boolean isRegion(String string) { 114 | if (string.length() != 3) { 115 | return false; 116 | } 117 | return isNumber(string.charAt(0)) && isNumber(string.charAt(1)) && isNumber(string.charAt(2)); 118 | } 119 | 120 | private static boolean isNumber(char c) { 121 | return c >= '0' && c <= '9'; 122 | } 123 | 124 | public static String createId() throws InterruptedException { 125 | Date now = new Date(); 126 | long lng = now.getTime(); 127 | // wait until we are in the next millisecond 128 | // before leaving to ensure uniqueness 129 | Thread.sleep(1); 130 | return "" + lng; 131 | } 132 | 133 | public static String tmxDate() { 134 | Calendar calendar = Calendar.getInstance(TimeZone.getTimeZone("GMT")); 135 | String sec = (calendar.get(Calendar.SECOND) < 10 ? "0" : "") + calendar.get(Calendar.SECOND); 136 | String min = (calendar.get(Calendar.MINUTE) < 10 ? "0" : "") + calendar.get(Calendar.MINUTE); 137 | String hour = (calendar.get(Calendar.HOUR_OF_DAY) < 10 ? "0" : "") + calendar.get(Calendar.HOUR_OF_DAY); 138 | String mday = (calendar.get(Calendar.DATE) < 10 ? "0" : "") + calendar.get(Calendar.DATE); 139 | String mon = (calendar.get(Calendar.MONTH) < 9 ? "0" : "") + (calendar.get(Calendar.MONTH) + 1); 140 | String longyear = "" + calendar.get(Calendar.YEAR); 141 | 142 | return longyear + mon + mday + "T" + hour + min + sec + "Z"; 143 | } 144 | 145 | public static long getGMTtime(String tmxDate) { 146 | Calendar calendar = Calendar.getInstance(TimeZone.getTimeZone("GMT")); 147 | try { 148 | int second = Integer.parseInt(tmxDate.substring(13, 15)); 149 | int minute = Integer.parseInt(tmxDate.substring(11, 13)); 150 | int hour = Integer.parseInt(tmxDate.substring(9, 11)); 151 | int date = Integer.parseInt(tmxDate.substring(6, 8)); 152 | int month = Integer.parseInt(tmxDate.substring(4, 6)) - 1; 153 | int year = Integer.parseInt(tmxDate.substring(0, 4)); 154 | calendar.set(year, month, date, hour, minute, second); 155 | return calendar.getTimeInMillis(); 156 | } catch (NumberFormatException e) { 157 | logger.log(Level.WARNING, "Unsupported TMX date: " + tmxDate); 158 | return 0l; 159 | } 160 | } 161 | 162 | public static Element buildTuv(String lang, String seg) 163 | throws SAXException, IOException, ParserConfigurationException { 164 | Element tuv = new Element("tuv"); 165 | tuv.setAttribute("xml:lang", lang); 166 | SAXBuilder builder = new SAXBuilder(); 167 | Element e = builder.build(new ByteArrayInputStream(("" + seg + " ").getBytes(StandardCharsets.UTF_8)))
168 | .getRootElement();
169 | tuv.addContent(e);
170 | return tuv;
171 | }
172 |
173 | public static String creationDate() {
174 | Calendar calendar = Calendar.getInstance(Locale.US);
175 | String sec = (calendar.get(Calendar.SECOND) < 10 ? "0" : "") + calendar.get(Calendar.SECOND);
176 | String min = (calendar.get(Calendar.MINUTE) < 10 ? "0" : "") + calendar.get(Calendar.MINUTE);
177 | String hour = (calendar.get(Calendar.HOUR_OF_DAY) < 10 ? "0" : "") + calendar.get(Calendar.HOUR_OF_DAY);
178 | String mday = (calendar.get(Calendar.DATE) < 10 ? "0" : "") + calendar.get(Calendar.DATE);
179 | String mon = (calendar.get(Calendar.MONTH) < 9 ? "0" : "") + (calendar.get(Calendar.MONTH) + 1);
180 | String longyear = "" + calendar.get(Calendar.YEAR);
181 |
182 | return longyear + mon + mday + "T" + hour + min + sec + "Z";
183 | }
184 |
185 | public static String extractText(Element seg) {
186 | List l = seg.getContent();
187 | Iterator i = l.iterator();
188 | StringBuilder text = new StringBuilder();
189 | while (i.hasNext()) {
190 | XMLNode o = i.next();
191 | if (o.getNodeType() == XMLNode.TEXT_NODE) {
192 | text.append(((TextNode) o).getText());
193 | } else if (o.getNodeType() == XMLNode.ELEMENT_NODE) {
194 | Element e = (Element) o;
195 | String type = e.getName();
196 | // discard all inline elements
197 | // except and
198 | if (type.equals("sub") || type.equals("hi")) { 199 | text.append(extractText(e)); 200 | } 201 | } 202 | } 203 | return text.toString(); 204 | } 205 | 206 | public static void deleteFolder(String folder) throws IOException { 207 | File f = new File(folder); 208 | if (f.isDirectory()) { 209 | String[] list = f.list(); 210 | for (int i = 0; i < list.length; i++) { 211 | deleteFolder(new File(f, list[i]).getAbsolutePath()); 212 | } 213 | } 214 | Files.delete(f.toPath()); 215 | } 216 | 217 | 218 | } 219 | -------------------------------------------------------------------------------- /docs/concepts/methods.dita: -------------------------------------------------------------------------------- 1 | 2 | 3 |
4 | Interface Methods
5 |
7 |
135 |
136 |
--------------------------------------------------------------------------------
/src/com/maxprograms/tmx/tmx11.dtd:
--------------------------------------------------------------------------------
1 |
13 |
14 |
78 |
79 |
80 |
81 |
82 |
83 |
84 |
85 |
86 |
87 |
88 |
90 |
91 |
92 |
93 |
94 |
96 |
97 |
98 |
99 |
112 |
113 |
114 |
115 |
116 |
117 |
118 |
119 |
122 |
123 |
124 |
125 |
126 |
129 |
131 |
132 |
133 |
134 |
139 |
140 |
141 |
142 |
146 |
147 |
148 |
149 |
164 |
165 |
166 |
167 |
180 |
181 |
182 |
183 |
184 |
185 |
186 |
187 |
188 |
192 |
193 |
194 |
196 |
197 |
198 |
199 |
200 |
204 |
205 |
206 |
210 |
211 |
212 |
214 |
215 |
216 |
217 |
--------------------------------------------------------------------------------
/src/com/maxprograms/tmx/tmx12.dtd:
--------------------------------------------------------------------------------
1 |
13 |
14 |
33 |
34 |
98 |
99 |
100 |
101 |
102 |
103 |
104 |
105 |
106 |
107 |
108 |
109 |
111 |
112 |
113 |
114 |
115 |
117 |
118 |
119 |
120 |
133 |
134 |
135 |
136 |
137 |
138 |
139 |
140 |
143 |
144 |
145 |
146 |
147 |
150 |
152 |
153 |
154 |
155 |
160 |
161 |
162 |
163 |
167 |
168 |
169 |
170 |
185 |
186 |
187 |
188 |
201 |
202 |
203 |
204 |
205 |
206 |
207 |
208 |
209 |
213 |
214 |
215 |
217 |
218 |
219 |
222 |
223 |
224 |
228 |
229 |
230 |
234 |
235 |
236 |
238 |
239 |
240 |
243 |
244 |
245 |
246 |
247 |
--------------------------------------------------------------------------------
/src/com/maxprograms/tmx/tmx13.dtd:
--------------------------------------------------------------------------------
1 |
13 |
14 |
27 |
28 |
47 |
48 |
112 |
113 |
114 |
115 |
116 |
117 |
118 |
119 |
120 |
121 |
122 |
123 |
125 |
126 |
127 |
128 |
129 |
131 |
132 |
133 |
134 |
147 |
148 |
149 |
150 |
151 |
152 |
153 |
154 |
158 |
159 |
160 |
161 |
162 |
165 |
167 |
168 |
169 |
170 |
175 |
176 |
177 |
178 |
182 |
183 |
184 |
185 |
200 |
201 |
202 |
203 |
217 |
218 |
219 |
220 |
221 |
222 |
223 |
224 |
225 |
226 |
230 |
231 |
232 |
234 |
235 |
236 |
239 |
240 |
241 |
245 |
246 |
247 |
251 |
252 |
253 |
255 |
256 |
257 |
260 |
261 |
262 |
263 |
264 |
265 |
266 |
267 |
268 |
--------------------------------------------------------------------------------
/src/com/maxprograms/tmx/tmx14.dtd:
--------------------------------------------------------------------------------
1 |
13 |
14 |
25 |
26 |
39 |
40 |
59 |
60 |
124 |
125 |
126 |
127 |
128 |
129 |
130 |
131 |
132 |
133 |
134 |
135 |
137 |
138 |
139 |
140 |
141 |
143 |
144 |
145 |
146 |
159 |
160 |
161 |
162 |
163 |
164 |
165 |
166 |
170 |
171 |
172 |
173 |
174 |
177 |
179 |
180 |
181 |
182 |
187 |
188 |
189 |
190 |
195 |
196 |
197 |
198 |
199 |
200 |
215 |
216 |
217 |
218 |
232 |
233 |
234 |
235 |
236 |
237 |
238 |
239 |
240 |
241 |
245 |
246 |
247 |
249 |
250 |
251 |
254 |
255 |
256 |
260 |
261 |
262 |
266 |
267 |
268 |
271 |
272 |
273 |
274 |
275 |
276 |
278 |
279 |
280 |
281 |
282 |
283 |
--------------------------------------------------------------------------------
/LICENSE:
--------------------------------------------------------------------------------
1 | Eclipse Public License - v 1.0
2 |
3 | THE ACCOMPANYING PROGRAM IS PROVIDED UNDER THE TERMS OF THIS ECLIPSE
4 | PUBLIC LICENSE ("AGREEMENT"). ANY USE, REPRODUCTION OR DISTRIBUTION OF
5 | THE PROGRAM CONSTITUTES RECIPIENT'S ACCEPTANCE OF THIS AGREEMENT.
6 |
7 | 1. DEFINITIONS
8 |
9 | "Contribution" means:
10 |
11 | a) in the case of the initial Contributor, the initial code and
12 | documentation distributed under this Agreement, and
13 |
14 | b) in the case of each subsequent Contributor:
15 |
16 | i) changes to the Program, and
17 |
18 | ii) additions to the Program;
19 |
20 | where such changes and/or additions to the Program originate from
21 | and are distributed by that particular Contributor. A Contribution
22 | 'originates' from a Contributor if it was added to the Program by
23 | such Contributor itself or anyone acting on such Contributor's
24 | behalf. Contributions do not include additions to the Program
25 | which: (i) are separate modules of software distributed in
26 | conjunction with the Program under their own license agreement,
27 | and (ii) are not derivative works of the Program.
28 |
29 | "Contributor" means any person or entity that distributes the Program.
30 |
31 | "Licensed Patents" mean patent claims licensable by a Contributor
32 | which are necessarily infringed by the use or sale of its Contribution
33 | alone or when combined with the Program.
34 |
35 | "Program" means the Contributions distributed in accordance with this
36 | Agreement.
37 |
38 | "Recipient" means anyone who receives the Program under this
39 | Agreement, including all Contributors.
40 |
41 | 2. GRANT OF RIGHTS
42 |
43 | a) Subject to the terms of this Agreement, each Contributor hereby
44 | grants Recipient a non-exclusive, worldwide, royalty-free
45 | copyright license to reproduce, prepare derivative works of,
46 | publicly display, publicly perform, distribute and sublicense the
47 | Contribution of such Contributor, if any, and such derivative
48 | works, in source code and object code form.
49 |
50 | b) Subject to the terms of this Agreement, each Contributor hereby
51 | grants Recipient a non-exclusive, worldwide, royalty-free patent
52 | license under Licensed Patents to make, use, sell, offer to sell,
53 | import and otherwise transfer the Contribution of such
54 | Contributor, if any, in source code and object code form. This
55 | patent license shall apply to the combination of the Contribution
56 | and the Program if, at the time the Contribution is added by the
57 | Contributor, such addition of the Contribution causes such
58 | combination to be covered by the Licensed Patents. The patent
59 | license shall not apply to any other combinations which include
60 | the Contribution. No hardware per se is licensed hereunder.
61 |
62 | c) Recipient understands that although each Contributor grants the
63 | licenses to its Contributions set forth herein, no assurances are
64 | provided by any Contributor that the Program does not infringe the
65 | patent or other intellectual property rights of any other
66 | entity. Each Contributor disclaims any liability to Recipient for
67 | claims brought by any other entity based on infringement of
68 | intellectual property rights or otherwise. As a condition to
69 | exercising the rights and licenses granted hereunder, each
70 | Recipient hereby assumes sole responsibility to secure any other
71 | intellectual property rights needed, if any. For example, if a
72 | third party patent license is required to allow Recipient to
73 | distribute the Program, it is Recipient's responsibility to
74 | acquire that license before distributing the Program.
75 |
76 | d) Each Contributor represents that to its knowledge it has
77 | sufficient copyright rights in its Contribution, if any, to grant
78 | the copyright license set forth in this Agreement.
79 |
80 | 3. REQUIREMENTS
81 |
82 | A Contributor may choose to distribute the Program in object code form
83 | under its own license agreement, provided that:
84 |
85 | a) it complies with the terms and conditions of this Agreement;
86 | and
87 |
88 | b) its license agreement:
89 |
90 | i) effectively disclaims on behalf of all Contributors all
91 | warranties and conditions, express and implied, including
92 | warranties or conditions of title and non-infringement, and
93 | implied warranties or conditions of merchantability and fitness
94 | for a particular purpose;
95 |
96 | ii) effectively excludes on behalf of all Contributors all
97 | liability for damages, including direct, indirect, special,
98 | incidental and consequential damages, such as lost profits;
99 |
100 | iii) states that any provisions which differ from this Agreement
101 | are offered by that Contributor alone and not by any other party;
102 | and
103 |
104 | iv) states that source code for the Program is available from such
105 | Contributor, and informs licensees how to obtain it in a
106 | reasonable manner on or through a medium customarily used for
107 | software exchange.
108 |
109 | When the Program is made available in source code form:
110 |
111 | a) it must be made available under this Agreement; and
112 |
113 | b) a copy of this Agreement must be included with each copy of the
114 | Program.
115 |
116 | Contributors may not remove or alter any copyright notices contained
117 | within the Program.
118 |
119 | Each Contributor must identify itself as the originator of its
120 | Contribution, if any, in a manner that reasonably allows subsequent
121 | Recipients to identify the originator of the Contribution.
122 |
123 | 4. COMMERCIAL DISTRIBUTION
124 |
125 | Commercial distributors of software may accept certain
126 | responsibilities with respect to end users, business partners and the
127 | like. While this license is intended to facilitate the commercial use
128 | of the Program, the Contributor who includes the Program in a
129 | commercial product offering should do so in a manner which does not
130 | create potential liability for other Contributors. Therefore, if a
131 | Contributor includes the Program in a commercial product offering,
132 | such Contributor ("Commercial Contributor") hereby agrees to defend
133 | and indemnify every other Contributor ("Indemnified Contributor")
134 | against any losses, damages and costs (collectively "Losses") arising
135 | from claims, lawsuits and other legal actions brought by a third party
136 | against the Indemnified Contributor to the extent caused by the acts
137 | or omissions of such Commercial Contributor in connection with its
138 | distribution of the Program in a commercial product offering. The
139 | obligations in this section do not apply to any claims or Losses
140 | relating to any actual or alleged intellectual property
141 | infringement. In order to qualify, an Indemnified Contributor must: a)
142 | promptly notify the Commercial Contributor in writing of such claim,
143 | and b) allow the Commercial Contributor to control, and cooperate with
144 | the Commercial Contributor in, the defense and any related settlement
145 | negotiations. The Indemnified Contributor may participate in any such
146 | claim at its own expense.
147 |
148 | For example, a Contributor might include the Program in a commercial
149 | product offering, Product X. That Contributor is then a Commercial
150 | Contributor. If that Commercial Contributor then makes performance
151 | claims, or offers warranties related to Product X, those performance
152 | claims and warranties are such Commercial Contributor's responsibility
153 | alone. Under this section, the Commercial Contributor would have to
154 | defend claims against the other Contributors related to those
155 | performance claims and warranties, and if a court requires any other
156 | Contributor to pay any damages as a result, the Commercial Contributor
157 | must pay those damages.
158 |
159 | 5. NO WARRANTY
160 |
161 | EXCEPT AS EXPRESSLY SET FORTH IN THIS AGREEMENT, THE PROGRAM IS
162 | PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
163 | KIND, EITHER EXPRESS OR IMPLIED INCLUDING, WITHOUT LIMITATION, ANY
164 | WARRANTIES OR CONDITIONS OF TITLE, NON-INFRINGEMENT, MERCHANTABILITY
165 | OR FITNESS FOR A PARTICULAR PURPOSE. Each Recipient is solely
166 | responsible for determining the appropriateness of using and
167 | distributing the Program and assumes all risks associated with its
168 | exercise of rights under this Agreement , including but not limited to
169 | the risks and costs of program errors, compliance with applicable
170 | laws, damage to or loss of data, programs or equipment, and
171 | unavailability or interruption of operations.
172 |
173 | 6. DISCLAIMER OF LIABILITY
174 |
175 | EXCEPT AS EXPRESSLY SET FORTH IN THIS AGREEMENT, NEITHER RECIPIENT NOR
176 | ANY CONTRIBUTORS SHALL HAVE ANY LIABILITY FOR ANY DIRECT, INDIRECT,
177 | INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING
178 | WITHOUT LIMITATION LOST PROFITS), HOWEVER CAUSED AND ON ANY THEORY OF
179 | LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
180 | NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OR
181 | DISTRIBUTION OF THE PROGRAM OR THE EXERCISE OF ANY RIGHTS GRANTED
182 | HEREUNDER, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.
183 |
184 | 7. GENERAL
185 |
186 | If any provision of this Agreement is invalid or unenforceable under
187 | applicable law, it shall not affect the validity or enforceability of
188 | the remainder of the terms of this Agreement, and without further
189 | action by the parties hereto, such provision shall be reformed to the
190 | minimum extent necessary to make such provision valid and enforceable.
191 |
192 | If Recipient institutes patent litigation against any entity
193 | (including a cross-claim or counterclaim in a lawsuit) alleging that
194 | the Program itself (excluding combinations of the Program with other
195 | software or hardware) infringes such Recipient's patent(s), then such
196 | Recipient's rights granted under Section 2(b) shall terminate as of
197 | the date such litigation is filed.
198 |
199 | All Recipient's rights under this Agreement shall terminate if it
200 | fails to comply with any of the material terms or conditions of this
201 | Agreement and does not cure such failure in a reasonable period of
202 | time after becoming aware of such noncompliance. If all Recipient's
203 | rights under this Agreement terminate, Recipient agrees to cease use
204 | and distribution of the Program as soon as reasonably
205 | practicable. However, Recipient's obligations under this Agreement and
206 | any licenses granted by Recipient relating to the Program shall
207 | continue and survive.
208 |
209 | Everyone is permitted to copy and distribute copies of this Agreement,
210 | but in order to avoid inconsistency the Agreement is copyrighted and
211 | may only be modified in the following manner. The Agreement Steward
212 | reserves the right to publish new versions (including revisions) of
213 | this Agreement from time to time. No one other than the Agreement
214 | Steward has the right to modify this Agreement. The Eclipse Foundation
215 | is the initial Agreement Steward. The Eclipse Foundation may assign
216 | the responsibility to serve as the Agreement Steward to a suitable
217 | separate entity. Each new version of the Agreement will be given a
218 | distinguishing version number. The Program (including Contributions)
219 | may always be distributed subject to the version of the Agreement
220 | under which it was received. In addition, after a new version of the
221 | Agreement is published, Contributor may elect to distribute the
222 | Program (including its Contributions) under the new version. Except as
223 | expressly stated in Sections 2(a) and 2(b) above, Recipient receives
224 | no rights or licenses to the intellectual property of any Contributor
225 | under this Agreement, whether expressly, by implication, estoppel or
226 | otherwise. All rights in the Program not expressly granted under this
227 | Agreement are reserved.
228 |
229 | This Agreement is governed by the laws of the State of New York and
230 | the intellectual property laws of the United States of America. No
231 | party to this Agreement will bring a legal action under this Agreement
232 | more than one year after the cause of action arose. Each party waives
233 | its rights to a jury trial in any resulting litigation.
--------------------------------------------------------------------------------
/src/com/maxprograms/tmengine/MapDbEngine.java:
--------------------------------------------------------------------------------
1 | /*******************************************************************************
2 | * Copyright (c) 2003-2021 Maxprograms.
3 | *
4 | * This program and the accompanying materials
5 | * are made available under the terms of the Eclipse Public License 1.0
6 | * which accompanies this distribution, and is available at
7 | * https://www.eclipse.org/org/documents/epl-v10.html
8 | *
9 | * Contributors:
10 | * Maxprograms - initial API and implementation
11 | *******************************************************************************/
12 | package com.maxprograms.tmengine;
13 |
14 | import java.io.File;
15 | import java.io.FileOutputStream;
16 | import java.io.IOException;
17 | import java.lang.System.Logger;
18 | import java.lang.System.Logger.Level;
19 | import java.nio.charset.StandardCharsets;
20 | import java.text.MessageFormat;
21 | import java.util.Calendar;
22 | import java.util.Collections;
23 | import java.util.Hashtable;
24 | import java.util.Iterator;
25 | import java.util.List;
26 | import java.util.Map;
27 | import java.util.NavigableSet;
28 | import java.util.Set;
29 | import java.util.TreeSet;
30 | import java.util.Vector;
31 | import java.util.regex.Pattern;
32 | import java.util.regex.PatternSyntaxException;
33 |
34 | import javax.xml.parsers.ParserConfigurationException;
35 |
36 | import org.mapdb.Fun;
37 | import org.mapdb.Fun.Tuple2;
38 | import org.xml.sax.SAXException;
39 |
40 | import com.maxprograms.tmx.TMXReader;
41 | import com.maxprograms.tmutils.TMUtils;
42 | import com.maxprograms.xml.Attribute;
43 | import com.maxprograms.xml.Element;
44 | import com.maxprograms.xml.Indenter;
45 |
46 | public class MapDbEngine implements ITmEngine, AutoCloseable {
47 |
48 | private static final Logger LOGGER = System.getLogger(MapDbEngine.class.getName());
49 |
50 | private String dbname;
51 | private File database;
52 | private TuDatabase tuDb;
53 | private TuvDatabase tuvDb;
54 | private FuzzyIndex fuzzyIndex;
55 |
56 | private String currProject;
57 | private String currSubject;
58 | private String currCustomer;
59 | private String creationDate;
60 |
61 | private Set tuAttributes;
62 |
63 | private long next;
64 |
65 | public MapDbEngine(String dbname, String workFolder) throws IOException {
66 | this.dbname = dbname;
67 | tuAttributes = Collections.synchronizedSortedSet(new TreeSet<>());
68 | String[] array = new String[] { "tuid", "o-encoding", "datatype", "usagecount", "lastusagedate", "creationtool",
69 | "creationtoolversion", "creationdate", "creationid", "changedate", "segtype", "changeid", "o-tmf",
70 | "srclang" };
71 | for (int i = 0; i < array.length; i++) {
72 | tuAttributes.add(array[i]);
73 | }
74 | File wfolder = new File(workFolder);
75 | database = new File(wfolder, dbname);
76 | if (!database.exists()) {
77 | database.mkdirs();
78 | }
79 | try {
80 | tuDb = new TuDatabase(database);
81 | } catch (Exception e) {
82 | LOGGER.log(Level.ERROR, e.getMessage(), e);
83 | MessageFormat mf = new MessageFormat("TU storage of database {0} is damaged.");
84 | throw new IOException(mf.format(new String[] { dbname }));
85 | }
86 | try {
87 | tuvDb = new TuvDatabase(database);
88 | } catch (Exception e) {
89 | LOGGER.log(Level.ERROR, e.getMessage(), e);
90 | MessageFormat mf = new MessageFormat("TUV storage of database {0} is damaged.");
91 | throw new IOException(mf.format(new String[] { dbname }));
92 | }
93 | try {
94 | fuzzyIndex = new FuzzyIndex(database);
95 | } catch (Exception e) {
96 | LOGGER.log(Level.ERROR, e.getMessage(), e);
97 | MessageFormat mf = new MessageFormat("Fuzzy index of database {0} is damaged.");
98 | throw new IOException(mf.format(new String[] { dbname }));
99 | }
100 | }
101 |
102 | @Override
103 | public void deleteDatabase() throws IOException {
104 | close();
105 | TMUtils.deleteFolder(database.getAbsolutePath());
106 | }
107 |
108 | @Override
109 | public String getType() {
110 | return MapDbEngine.class.getName();
111 | }
112 |
113 | @Override
114 | public synchronized void close() throws IOException {
115 | fuzzyIndex.close();
116 | tuDb.close();
117 | tuvDb.close();
118 | }
119 |
120 | @Override
121 | public String getName() {
122 | return dbname;
123 | }
124 |
125 | @Override
126 | public int storeTMX(String tmxFile, String project, String customer, String subject)
127 | throws SAXException, IOException, ParserConfigurationException {
128 | next = 0l;
129 | currProject = project != null ? project : "";
130 | currSubject = subject != null ? subject : "";
131 | currCustomer = customer != null ? customer : "";
132 | creationDate = TMUtils.creationDate();
133 |
134 | TMXReader reader = new TMXReader(this);
135 | reader.parse(new File(tmxFile).toURI().toURL());
136 | commit();
137 |
138 | return reader.getCount();
139 | }
140 |
141 | @Override
142 | public void exportMemory(String tmxfile, Set langs, String srcLang, Map props)
143 | throws IOException, SAXException, ParserConfigurationException {
144 | Map properties = props != null ? props : new Hashtable<>();
145 | try (FileOutputStream output = new FileOutputStream(new File(tmxfile))) {
146 | writeHeader(output, srcLang, properties);
147 | writeString(output, " \n");
148 |
149 | Set set = tuDb.getKeys();
150 | Iterator it = set.iterator();
151 | while (it.hasNext()) {
152 | int hash = it.next();
153 | Map tuProps = tuDb.getTu(hash);
154 | Element tu = buildElement(tuProps);
155 | if (langs != null) {
156 | List tuvs = tu.getChildren("tuv");
157 | Iterator et = tuvs.iterator();
158 | while (et.hasNext()) {
159 | Element tuv = et.next();
160 | if (!langs.contains(tuv.getAttributeValue("xml:lang"))) {
161 | tu.removeChild(tuv);
162 | }
163 | }
164 | }
165 | Indenter.indent(tu, 3, 2);
166 | writeString(output, " " + tu.toString() + "\n");
167 | }
168 | writeString(output, " \n");
169 | writeString(output, "");
170 | }
171 | }
172 |
173 | @Override
174 | public void flag(String tuid) {
175 | Map properties = tuDb.getTu(tuid.hashCode());
176 | if (properties != null) {
177 | properties.put("x-flag", "SW-Flag");
178 | tuDb.store(tuid, properties);
179 | }
180 | }
181 |
182 | @Override
183 | public Set getAllClients() {
184 | return tuDb.getCustomers();
185 | }
186 |
187 | @Override
188 | public Set getAllLanguages() {
189 | return tuDb.getLanguages();
190 | }
191 |
192 | @Override
193 | public Set getAllProjects() {
194 | return tuDb.getProjects();
195 | }
196 |
197 | @Override
198 | public Set getAllSubjects() {
199 | return tuDb.getSubjects();
200 | }
201 |
202 | @Override
203 | public List searchTranslation(String searchStr, String srcLang, String tgtLang, int similarity,
204 | boolean caseSensitive) throws IOException, SAXException, ParserConfigurationException {
205 |
206 | List result = new Vector<>();
207 |
208 | if (similarity == 100) {
209 | // check for perfect matches
210 | Set perfect = tuvDb.getPerfectMatches(srcLang, searchStr);
211 | if (!perfect.isEmpty()) {
212 | Iterator it = perfect.iterator();
213 | while (it.hasNext()) {
214 | String tuid = it.next();
215 | String puretext = tuvDb.getPureText(srcLang, tuid.hashCode());
216 | boolean isMatch = true;
217 | if (caseSensitive) {
218 | isMatch = searchStr.equals(puretext);
219 | }
220 | if (isMatch) {
221 | String targetSeg = tuvDb.getSegText(tgtLang, tuid);
222 | if (targetSeg != null) {
223 | String sourceSeg = tuvDb.getSegText(srcLang, tuid);
224 | Element source = TMUtils.buildTuv(srcLang, sourceSeg);
225 | Element target = TMUtils.buildTuv(tgtLang, targetSeg);
226 | Map properties = tuDb.getTu(tuid.hashCode());
227 | Match match = new Match(source, target, 100, dbname, properties);
228 | result.add(match);
229 | }
230 | }
231 | }
232 | }
233 | }
234 | if (similarity < 100) {
235 | // Check for fuzzy matches
236 | int[] ngrams = NGrams.getNGrams(searchStr);
237 | int size = ngrams.length;
238 | if (size == 0) {
239 | return result;
240 | }
241 | int min = size * similarity / 100;
242 | int max = size * (200 - similarity) / 100;
243 |
244 | Map candidates = new Hashtable<>();
245 | String lowerSearch = searchStr.toLowerCase();
246 |
247 | NavigableSet> index = fuzzyIndex.getIndex(srcLang);
248 | for (int i = 0; i < ngrams.length; i++) {
249 | Iterable keys = Fun.filter(index, ngrams[i]);
250 | Iterator it = keys.iterator();
251 | while (it.hasNext()) {
252 | String tuid = it.next();
253 | if (candidates.containsKey(tuid)) {
254 | int count = candidates.get(tuid);
255 | candidates.put(tuid, count + 1);
256 | } else {
257 | candidates.put(tuid, 1);
258 | }
259 | }
260 | }
261 |
262 | Set tuids = candidates.keySet();
263 | Iterator it = tuids.iterator();
264 | while (it.hasNext()) {
265 | String tuid = it.next();
266 | int count = candidates.get(tuid);
267 | if (count >= min && count <= max) {
268 | int distance;
269 | String puretext = tuvDb.getPureText(srcLang, tuid.hashCode());
270 | if (caseSensitive) {
271 | distance = MatchQuality.similarity(searchStr, puretext);
272 | } else {
273 | distance = MatchQuality.similarity(lowerSearch, puretext.toLowerCase());
274 | }
275 | if (distance >= similarity) {
276 | String targetSeg = tuvDb.getSegText(tgtLang, tuid);
277 | if (targetSeg != null) {
278 | String sourceSeg = tuvDb.getSegText(srcLang, tuid);
279 | Element source = TMUtils.buildTuv(srcLang, sourceSeg);
280 | Element target = TMUtils.buildTuv(tgtLang, targetSeg);
281 | Map properties = tuDb.getTu(tuid.hashCode());
282 | Match match = new Match(source, target, distance, dbname, properties);
283 | result.add(match);
284 | }
285 | }
286 | }
287 | }
288 | }
289 | return result;
290 | }
291 |
292 | private Element buildElement(Map properties)
293 | throws IOException, SAXException, ParserConfigurationException {
294 | Element tu = new Element("tu");
295 | Set keys = properties.keySet();
296 | Iterator it = keys.iterator();
297 | while (it.hasNext()) {
298 | String key = it.next();
299 | if (tuAttributes.contains(key)) {
300 | tu.setAttribute(key, properties.get(key));
301 | } else {
302 | Element prop = new Element("prop");
303 | prop.setAttribute("type", key);
304 | prop.setText(properties.get(key));
305 | tu.addContent(prop);
306 | }
307 | }
308 | String tuid = tu.getAttributeValue("tuid");
309 | Set langs = tuDb.getLanguages();
310 | it = langs.iterator();
311 | while (it.hasNext()) {
312 | String lang = it.next();
313 | String seg = tuvDb.getSegText(lang, tuid);
314 | if (seg != null) {
315 | Element tuv = TMUtils.buildTuv(lang, seg);
316 | tu.addContent(tuv);
317 | }
318 | }
319 | return tu;
320 | }
321 |
322 | @Override
323 | public List concordanceSearch(String searchStr, String srcLang, int limit, boolean isRegexp,
324 | boolean caseSensitive) throws IOException, SAXException, ParserConfigurationException {
325 | List result = new Vector<>();
326 | Pattern pattern = null;
327 | if (isRegexp) {
328 | try {
329 | pattern = Pattern.compile(searchStr);
330 | } catch (PatternSyntaxException pse) {
331 | throw new IOException(pse.getMessage());
332 | }
333 | }
334 | String lowerStr = searchStr.toLowerCase();
335 | NavigableSet keySet = tuvDb.getKeySet(srcLang);
336 | Iterator it = keySet.iterator();
337 | while (it.hasNext()) {
338 | int hash = it.next();
339 | String pureText = tuvDb.getPureText(srcLang, hash);
340 | if (isRegexp) {
341 | if (pattern != null && pattern.matcher(pureText).matches()) {
342 | result.add(buildElement(tuDb.getTu(hash)));
343 | if (result.size() == limit) {
344 | return result;
345 | }
346 | }
347 | } else {
348 | if (caseSensitive) {
349 | if (pureText.indexOf(searchStr) != -1) {
350 | result.add(buildElement(tuDb.getTu(hash)));
351 | if (result.size() == limit) {
352 | return result;
353 | }
354 | }
355 | } else {
356 | if (pureText.toLowerCase().indexOf(lowerStr) != -1) {
357 | result.add(buildElement(tuDb.getTu(hash)));
358 | if (result.size() == limit) {
359 | return result;
360 | }
361 | }
362 | }
363 | }
364 |
365 | }
366 | return result;
367 | }
368 |
369 | @Override
370 | public void storeTu(Element tu) throws IOException {
371 | String tuid = tu.getAttributeValue("tuid");
372 | if (tuid.isEmpty()) {
373 | tuid = nextId();
374 | tu.setAttribute("tuid", tuid);
375 | }
376 | if (tu.getAttributeValue("creationdate").isEmpty()) {
377 | tu.setAttribute("creationdate", creationDate);
378 | }
379 | if (tu.getAttributeValue("creationid").isEmpty()) {
380 | tu.setAttribute("creationid", System.getProperty("user.name"));
381 | }
382 | Map tuProperties = new Hashtable<>();
383 |
384 | List atts = tu.getAttributes();
385 | Iterator at = atts.iterator();
386 | while (at.hasNext()) {
387 | Attribute a = at.next();
388 | tuProperties.put(a.getName(), a.getValue());
389 | }
390 | List properties = tu.getChildren("prop");
391 | Iterator kt = properties.iterator();
392 | while (kt.hasNext()) {
393 | Element prop = kt.next();
394 | tuProperties.put(prop.getAttributeValue("type"), prop.getText());
395 | }
396 | if (currSubject != null && !currSubject.isEmpty()) {
397 | tuProperties.put("subject", currSubject);
398 | }
399 | if (currCustomer != null && !currCustomer.isEmpty()) {
400 | tuProperties.put("customer", currCustomer);
401 | }
402 | if (currProject != null && !currProject.isEmpty()) {
403 | tuProperties.put("project", currProject);
404 | }
405 | List tuvs = tu.getChildren("tuv");
406 | Set tuLangs = Collections.synchronizedSortedSet(new TreeSet<>());
407 |
408 | Iterator it = tuvs.iterator();
409 | while (it.hasNext()) {
410 | Element tuv = it.next();
411 | String lang = TMUtils.normalizeLang(tuv.getAttributeValue("xml:lang"));
412 | if (lang == null) {
413 | // Invalid language code, ignore this tuv
414 | continue;
415 | }
416 | tuDb.storeLanguage(lang);
417 | tuvDb.remove(lang, tuid);
418 | if (!tuLangs.contains(lang)) {
419 | Element seg = tuv.getChild("seg");
420 | String puretext = TMUtils.extractText(seg);
421 | if (!puretext.isBlank()) {
422 | String segText = seg.toString();
423 | segText = segText.substring("".length());
424 | segText = segText.substring(0, segText.length() - " ".length());
425 | tuvDb.store(lang, tuid, puretext, segText);
426 |
427 | int[] ngrams = NGrams.getNGrams(puretext);
428 | NavigableSet> index = fuzzyIndex.getIndex(lang);
429 | for (int i = 0; i < ngrams.length; i++) {
430 | Tuple2 entry = Fun.t2(ngrams[i], tuid);
431 | if (!index.contains(entry)) {
432 | index.add(entry);
433 | }
434 | }
435 | tuLangs.add(lang);
436 | }
437 | }
438 | }
439 | tuDb.store(tuid, tuProperties);
440 | }
441 |
442 | @Override
443 | public synchronized void commit() {
444 | fuzzyIndex.commit();
445 | tuDb.commit();
446 | tuvDb.commit();
447 | }
448 |
449 | @Override
450 | public Element getTu(String tuid) throws IOException, SAXException, ParserConfigurationException {
451 | Map properties = tuDb.getTu(tuid.hashCode());
452 | return buildElement(properties);
453 | }
454 |
455 | private static void writeHeader(FileOutputStream output, String srcLang, Map properties)
456 | throws IOException {
457 | writeString(output, "\n");
458 | writeString(output,
459 | "\n");
460 | writeString(output, "\n");
461 | writeString(output,
462 | " keys = properties.keySet();
471 | Iterator it = keys.iterator();
472 | while (it.hasNext()) {
473 | String key = it.next();
474 | writeString(output, " " + properties.get(key) + " \n");
475 | }
476 | writeString(output, " \n");
477 | }
478 | }
479 |
480 | private static void writeString(FileOutputStream output, String string) throws IOException {
481 | output.write(string.getBytes(StandardCharsets.UTF_8));
482 | }
483 |
484 | private String nextId() {
485 | if (next == 0l) {
486 | next = Calendar.getInstance().getTimeInMillis();
487 | }
488 | return "" + next++;
489 | }
490 |
491 | @Override
492 | public void removeTu(String tuid) throws IOException, SAXException, ParserConfigurationException {
493 | Element tu = getTu(tuid);
494 | tuDb.remove(tuid);
495 |
496 | List tuvs = tu.getChildren("tuv");
497 | Iterator it = tuvs.iterator();
498 | while (it.hasNext()) {
499 | Element tuv = it.next();
500 | String lang = tuv.getAttributeValue("xml:lang");
501 | tuvDb.remove(lang, tuid);
502 |
503 | Element seg = tuv.getChild("seg");
504 | String puretext = TMUtils.extractText(seg);
505 |
506 | int[] ngrams = NGrams.getNGrams(puretext);
507 | NavigableSet> index = fuzzyIndex.getIndex(lang);
508 | for (int i = 0; i < ngrams.length; i++) {
509 | Tuple2 entry = Fun.t2(ngrams[i], tuid);
510 | if (index.contains(entry)) {
511 | index.remove(entry);
512 | }
513 | }
514 | }
515 | }
516 |
517 | }
518 |
--------------------------------------------------------------------------------
/src/com/maxprograms/tmengine/SQLEngine.java:
--------------------------------------------------------------------------------
1 | /*******************************************************************************
2 | * Copyright (c) 2003, 2018 Maxprograms.
3 | *
4 | * This program and the accompanying materials
5 | * are made available under the terms of the Eclipse Public License 1.0
6 | * which accompanies this distribution, and is available at
7 | * https://www.eclipse.org/org/documents/epl-v10.html
8 | *
9 | * Contributors:
10 | * Maxprograms - initial API and implementation
11 | *******************************************************************************/
12 | package com.maxprograms.tmengine;
13 |
14 | import java.io.File;
15 | import java.io.FileOutputStream;
16 | import java.io.IOException;
17 | import java.lang.System.Logger;
18 | import java.lang.System.Logger.Level;
19 | import java.nio.charset.StandardCharsets;
20 | import java.sql.Connection;
21 | import java.sql.DriverManager;
22 | import java.sql.PreparedStatement;
23 | import java.sql.ResultSet;
24 | import java.sql.SQLException;
25 | import java.sql.Statement;
26 | import java.util.Calendar;
27 | import java.util.Collections;
28 | import java.util.Hashtable;
29 | import java.util.Iterator;
30 | import java.util.List;
31 | import java.util.Map;
32 | import java.util.Properties;
33 | import java.util.Set;
34 | import java.util.TreeSet;
35 | import java.util.Vector;
36 |
37 | import javax.xml.parsers.ParserConfigurationException;
38 |
39 | import com.maxprograms.tmutils.TMUtils;
40 | import com.maxprograms.tmx.TMXReader;
41 | import com.maxprograms.xml.Attribute;
42 | import com.maxprograms.xml.Element;
43 | import com.maxprograms.xml.Indenter;
44 |
45 | import org.xml.sax.SAXException;
46 |
47 | public class SQLEngine implements ITmEngine {
48 |
49 | private static final Logger LOGGER = System.getLogger(SQLEngine.class.getName());
50 |
51 | private Connection conn;
52 | private String dbName;
53 | private String serverName;
54 | private int port;
55 | private String userName;
56 | private String password;
57 |
58 | private String currProject;
59 | private String currSubject;
60 | private String currCustomer;
61 | private String creationDate;
62 |
63 | private long next;
64 |
65 | private Set languages;
66 |
67 | private PreparedStatement insertProperties;
68 | private PreparedStatement removeProperties;
69 | private PreparedStatement insertTuv;
70 | private PreparedStatement removeTuv;
71 | private PreparedStatement checkTu;
72 | private PreparedStatement selectProperties;
73 | private PreparedStatement selectSeg;
74 | private PreparedStatement selectPureText;
75 |
76 | private Hashtable insertNgram;
77 | private Hashtable removeNgram;
78 | private Hashtable selectNgram;
79 |
80 | private Set tuAttributes;
81 |
82 | public SQLEngine(String dbName, String serverName, int port, String userName, String password) throws SQLException {
83 | this.dbName = dbName;
84 | this.serverName = serverName;
85 | this.port = port;
86 | this.userName = userName;
87 | this.password = password;
88 | StringBuilder connBuilder = new StringBuilder();
89 | connBuilder.append("jdbc:mariadb://");
90 | connBuilder.append(serverName);
91 | connBuilder.append(':');
92 | connBuilder.append(port);
93 | connBuilder.append('/');
94 | connBuilder.append(dbName);
95 | connBuilder.append("?user=");
96 | connBuilder.append(userName);
97 | connBuilder.append("&password=");
98 | connBuilder.append(password);
99 | try {
100 | conn = DriverManager.getConnection(connBuilder.toString()); // "jdbc:mariadb://localhost:3306/DB?user=root&password=myPassword"
101 | conn.setAutoCommit(false);
102 | } catch (SQLException e) {
103 | createDatabase();
104 | conn = DriverManager.getConnection(connBuilder.toString());
105 | conn.setAutoCommit(false);
106 | LOGGER.log(Level.INFO, "Database " + dbName + " created.");
107 | }
108 | insertNgram = new Hashtable<>();
109 | removeNgram = new Hashtable<>();
110 | selectNgram = new Hashtable<>();
111 | }
112 |
113 | @Override
114 | public String getType() {
115 | return SQLEngine.class.getName();
116 | }
117 |
118 | private void createDatabase() throws SQLException {
119 | StringBuilder serverBuilder = new StringBuilder();
120 | serverBuilder.append("jdbc:mariadb://");
121 | serverBuilder.append(serverName);
122 | serverBuilder.append(':');
123 | serverBuilder.append(port);
124 | serverBuilder.append('/');
125 | Properties prop = new Properties();
126 | prop.setProperty("user", userName);
127 | prop.setProperty("password", password);
128 | prop.setProperty("useUnicode", "true");
129 | prop.setProperty("characterEncoding", StandardCharsets.UTF_8.name());
130 | try (Connection connection = DriverManager.getConnection(serverBuilder.toString(), prop)) {
131 | try (Statement stmt = connection.createStatement()) {
132 | stmt.execute("CREATE DATABASE `" + dbName + "` CHARACTER SET utf8");
133 | stmt.execute("CREATE TABLE `" + dbName + "`.tuv ( tuid VARCHAR(30) NOT NULL,"
134 | + " lang VARCHAR(15) NOT NULL, seg TEXT NOT NULL, pureText TEXT,"
135 | + " PRIMARY KEY (tuid,lang));");
136 | stmt.execute("CREATE TABLE `" + dbName + "`.tuprop ( tuid VARCHAR(30) NOT NULL,"
137 | + " propType VARCHAR(30) NOT NULL, content TEXT, PRIMARY KEY (tuid, propType)" + ");");
138 | stmt.execute("CREATE TABLE `" + dbName + "`.langs ( lang VARCHAR(15) NOT NULL);");
139 | }
140 | }
141 | }
142 |
143 | @Override
144 | public void deleteDatabase() throws SQLException {
145 | close();
146 | StringBuilder serverBuilder = new StringBuilder();
147 | serverBuilder.append("jdbc:mariadb://");
148 | serverBuilder.append(serverName);
149 | serverBuilder.append(':');
150 | serverBuilder.append(port);
151 | serverBuilder.append('/');
152 | Properties prop = new Properties();
153 | prop.setProperty("user", userName);
154 | prop.setProperty("password", password);
155 | prop.setProperty("useUnicode", "true");
156 | prop.setProperty("characterEncoding", StandardCharsets.UTF_8.name());
157 | try (Connection connection = DriverManager.getConnection(serverBuilder.toString(), prop)) {
158 | try (Statement stmt = connection.createStatement()) {
159 | stmt.execute("DROP DATABASE `" + dbName + "`");
160 | }
161 | }
162 | }
163 |
164 | @Override
165 | public void close() throws SQLException {
166 | conn.commit();
167 | if (insertProperties != null) {
168 | insertProperties.close();
169 | }
170 | if (removeProperties != null) {
171 | removeProperties.close();
172 | }
173 | if (selectProperties != null) {
174 | selectProperties.close();
175 | }
176 | if (insertTuv != null) {
177 | insertTuv.close();
178 | }
179 | if (removeTuv != null) {
180 | removeTuv.close();
181 | }
182 | if (checkTu != null) {
183 | checkTu.close();
184 | }
185 | if (selectSeg != null) {
186 | selectSeg.close();
187 | }
188 | if (selectPureText != null) {
189 | selectPureText.close();
190 | }
191 | Set keys = insertNgram.keySet();
192 | Iterator it = keys.iterator();
193 | while (it.hasNext()) {
194 | insertNgram.get(it.next()).close();
195 | }
196 | keys = removeNgram.keySet();
197 | it = keys.iterator();
198 | while (it.hasNext()) {
199 | removeNgram.get(it.next()).close();
200 | }
201 | keys = selectNgram.keySet();
202 | it = keys.iterator();
203 | while (it.hasNext()) {
204 | selectNgram.get(it.next()).close();
205 | }
206 | conn.close();
207 | }
208 |
209 | @Override
210 | public String getName() {
211 | return dbName;
212 | }
213 |
214 | @Override
215 | public int storeTMX(String tmxFile, String project, String customer, String subject)
216 | throws SAXException, IOException, ParserConfigurationException, SQLException {
217 | next = 0l;
218 | if (customer == null) {
219 | customer = "";
220 | }
221 | if (subject == null) {
222 | subject = "";
223 | }
224 | if (project == null) {
225 | project = "";
226 | }
227 |
228 | currProject = project;
229 | currSubject = subject;
230 | currCustomer = customer;
231 | creationDate = TMUtils.creationDate();
232 |
233 | TMXReader reader = new TMXReader(this);
234 | reader.parse(new File(tmxFile).toURI().toURL());
235 | commit();
236 |
237 | return reader.getCount();
238 | }
239 |
240 | @Override
241 | public void exportMemory(String tmxfile, Set langs, String srcLang, Map properties)
242 | throws IOException, SAXException, ParserConfigurationException, SQLException {
243 | if (languages == null) {
244 | getAllLanguages();
245 | }
246 | if (properties == null) {
247 | properties = new Hashtable<>();
248 | }
249 | try (FileOutputStream output = new FileOutputStream(new File(tmxfile))) {
250 | writeHeader(output, srcLang, properties);
251 | writeString(output, " \n");
252 | try (Statement stmt = conn.createStatement()) {
253 | try (ResultSet rs = stmt.executeQuery("SELECT DISTINCT tuid FROM `" + dbName + "`.tuprop")) {
254 | while (rs.next()) {
255 | String tuid = rs.getString(1);
256 | Element tu = getTu(tuid, langs);
257 | Indenter.indent(tu, 3, 2);
258 | writeString(output, " " + tu.toString() + "\n");
259 | }
260 | }
261 | }
262 | writeString(output, " \n");
263 | writeString(output, " ");
264 | }
265 | }
266 |
267 | @Override
268 | public void flag(String tuid) throws SQLException {
269 | if (checkPreviousTu(tuid)) {
270 | Hashtable properties = getTuProperies(tuid);
271 | if (!properties.containsKey("x-flag")) {
272 | if (insertProperties == null) {
273 | insertProperties = conn.prepareStatement(
274 | "INSERT INTO `" + dbName + "`.tuprop (tuid, propType, content) VALUES (?,?,?)");
275 | }
276 | insertProperties.setString(1, tuid);
277 | insertProperties.setString(2, "x-flag");
278 | insertProperties.setString(3, "SW-Flag");
279 | insertProperties.execute();
280 | }
281 | }
282 | }
283 |
284 | @Override
285 | public Set getAllClients() throws SQLException {
286 | Set result = Collections.synchronizedSortedSet(new TreeSet<>());
287 | try (Statement stmt = conn.createStatement()) {
288 | try (ResultSet rs = stmt
289 | .executeQuery("SELECT DISTINCT content FROM `" + dbName + "`.tuprop WHERE propType='customer'")) {
290 | while (rs.next()) {
291 | result.add(rs.getNString(1));
292 | }
293 | }
294 | }
295 | return result;
296 | }
297 |
298 | @Override
299 | public Set getAllLanguages() throws SQLException {
300 | if (languages == null) {
301 | languages = Collections.synchronizedSortedSet(new TreeSet<>());
302 | try (Statement stmt = conn.createStatement()) {
303 | try (ResultSet rs = stmt.executeQuery("SELECT lang FROM `" + dbName + "`.langs")) {
304 | while (rs.next()) {
305 | languages.add(rs.getString(1));
306 | }
307 | }
308 | }
309 | }
310 | return languages;
311 | }
312 |
313 | @Override
314 | public Set getAllProjects() throws SQLException {
315 | Set result = Collections.synchronizedSortedSet(new TreeSet<>());
316 | try (Statement stmt = conn.createStatement()) {
317 | try (ResultSet rs = stmt
318 | .executeQuery("SELECT DISTINCT content FROM `" + dbName + "`.tuprop WHERE propType='project'")) {
319 | while (rs.next()) {
320 | result.add(rs.getNString(1));
321 | }
322 | }
323 | }
324 | return result;
325 | }
326 |
327 | @Override
328 | public Set getAllSubjects() throws SQLException {
329 | Set result = Collections.synchronizedSortedSet(new TreeSet<>());
330 | try (Statement stmt = conn.createStatement()) {
331 | try (ResultSet rs = stmt
332 | .executeQuery("SELECT DISTINCT content FROM `" + dbName + "`.tuprop WHERE propType='subject'")) {
333 | while (rs.next()) {
334 | result.add(rs.getNString(1));
335 | }
336 | }
337 | }
338 | return result;
339 | }
340 |
341 | @Override
342 | public List searchTranslation(String searchStr, String srcLang, String tgtLang, int similarity,
343 | boolean caseSensitive) throws IOException, SAXException, ParserConfigurationException, SQLException {
344 | List result = new Vector<>();
345 |
346 | int[] ngrams = NGrams.getNGrams(searchStr);
347 | int size = ngrams.length;
348 | if (size == 0) {
349 | return result;
350 | }
351 |
352 | int minLength = searchStr.length() * similarity / 100;
353 | int maxLength = searchStr.length() * (200 - similarity) / 100;
354 |
355 | StringBuilder set = new StringBuilder();
356 | set.append("" + ngrams[0]);
357 | for (int i = 1; i < size; i++) {
358 | set.append("," + ngrams[i]);
359 | }
360 |
361 | Set candidates = Collections.synchronizedSortedSet(new TreeSet<>());
362 | String lowerSearch = searchStr.toLowerCase();
363 |
364 | PreparedStatement stmt = selectNgram.get(srcLang);
365 | if (stmt == null) {
366 | stmt = conn.prepareStatement("SELECT tuid from `" + dbName + "`.matrix_"
367 | + srcLang.replace('-', '_').toLowerCase() + " WHERE ngram in (?) and segSize>=? AND segSize<=?");
368 | selectNgram.put(srcLang, stmt);
369 | }
370 | stmt.setString(1, set.toString());
371 | stmt.setInt(2, minLength);
372 | stmt.setInt(3, maxLength);
373 |
374 | try (ResultSet rs = stmt.executeQuery()) {
375 | while (rs.next()) {
376 | String tuid = rs.getString(1);
377 | candidates.add(tuid);
378 | }
379 | }
380 |
381 | Iterator it = candidates.iterator();
382 | while (it.hasNext()) {
383 | String tuid = it.next();
384 | int distance;
385 | String puretext = getPureText(srcLang, tuid);
386 | if (caseSensitive) {
387 | distance = MatchQuality.similarity(searchStr, puretext);
388 | } else {
389 | distance = MatchQuality.similarity(lowerSearch, puretext.toLowerCase());
390 | }
391 | if (distance >= similarity) {
392 | String targetSeg = getSegText(tgtLang, tuid);
393 | if (targetSeg != null) {
394 | String sourceSeg = getSegText(srcLang, tuid);
395 | Element source = TMUtils.buildTuv(srcLang, sourceSeg);
396 | Element target = TMUtils.buildTuv(tgtLang, targetSeg);
397 | Hashtable properties = getTuProperies(tuid);
398 | Match match = new Match(source, target, distance, dbName, properties);
399 | result.add(match);
400 | }
401 | }
402 | }
403 | return result;
404 | }
405 |
406 | private String getPureText(String lang, String tuid) throws SQLException {
407 | if (selectPureText == null) {
408 | selectPureText = conn.prepareStatement("SELECT pureText FROM `" + dbName + "`.tuv WHERE tuid=? AND lang=?");
409 | }
410 | String pureText = "";
411 | selectPureText.setString(1, tuid);
412 | selectPureText.setString(2, lang);
413 | try (ResultSet rs = selectPureText.executeQuery()) {
414 | while (rs.next()) {
415 | pureText = rs.getNString(1);
416 | }
417 | }
418 | return pureText;
419 | }
420 |
421 | @Override
422 | public List concordanceSearch(String searchStr, String srcLang, int limit, boolean isRegexp,
423 | boolean caseSensitive) throws IOException, SAXException, ParserConfigurationException, SQLException {
424 | Set candidates = Collections.synchronizedSortedSet(new TreeSet<>());
425 | if (isRegexp) {
426 | try (PreparedStatement stmt = conn.prepareStatement(
427 | "SELECT tuid, pureText FROM `" + dbName + "`.tuv WHERE lang=? AND pureText REGEXP ? LIMIT ?")) {
428 | stmt.setString(1, srcLang);
429 | stmt.setString(2, searchStr);
430 | stmt.setInt(3, limit);
431 | try (ResultSet rs = stmt.executeQuery()) {
432 | while (rs.next()) {
433 | candidates.add(rs.getString(1));
434 | }
435 | }
436 | }
437 | } else {
438 | if (!caseSensitive) {
439 | try (PreparedStatement stmt = conn.prepareStatement("SELECT tuid, pureText FROM `" + dbName
440 | + "`.tuv WHERE lang=? AND LOWER(pureText) LIKE ? LIMIT ?")) {
441 | stmt.setString(1, srcLang);
442 | stmt.setString(2, "%" + searchStr.toLowerCase() + "%");
443 | stmt.setInt(3, limit);
444 | try (ResultSet rs = stmt.executeQuery()) {
445 | while (rs.next()) {
446 | candidates.add(rs.getString(1));
447 | }
448 | }
449 | }
450 | } else {
451 | try (PreparedStatement stmt = conn.prepareStatement(
452 | "SELECT tuid, pureText FROM `" + dbName + "`.tuv WHERE lang=? AND pureText LIKE ? LIMIT ?")) {
453 | stmt.setString(1, srcLang);
454 | stmt.setString(2, "%" + searchStr + "%");
455 | stmt.setInt(3, limit);
456 | try (ResultSet rs = stmt.executeQuery()) {
457 | while (rs.next()) {
458 | candidates.add(rs.getString(1));
459 | }
460 | }
461 | }
462 | }
463 | }
464 | List result = new Vector<>();
465 | Iterator it = candidates.iterator();
466 | while (it.hasNext()) {
467 | Element tu = getTu(it.next());
468 | result.add(tu);
469 | }
470 | return result;
471 | }
472 |
473 | @Override
474 | public void storeTu(Element tu) throws IOException, SQLException {
475 | String tuid = tu.getAttributeValue("tuid");
476 | boolean isNew = false;
477 | if (tuid.isEmpty()) {
478 | tuid = nextId();
479 | tu.setAttribute("tuid", tuid);
480 | isNew = true;
481 | }
482 | if (!isNew) {
483 | isNew = checkPreviousTu(tuid);
484 | }
485 | if (tu.getAttributeValue("creationdate").isEmpty()) {
486 | tu.setAttribute("creationdate", creationDate);
487 | }
488 | if (tu.getAttributeValue("creationid").isEmpty()) {
489 | tu.setAttribute("creationid", System.getProperty("user.name"));
490 | }
491 | Hashtable tuProperties = new Hashtable<>();
492 |
493 | List atts = tu.getAttributes();
494 | Iterator at = atts.iterator();
495 | while (at.hasNext()) {
496 | Attribute a = at.next();
497 | tuProperties.put(a.getName(), a.getValue());
498 | }
499 | List properties = tu.getChildren("prop");
500 | Iterator kt = properties.iterator();
501 | while (kt.hasNext()) {
502 | Element prop = kt.next();
503 | tuProperties.put(prop.getAttributeValue("type"), prop.getText());
504 | }
505 | if (currSubject != null && !currSubject.isEmpty()) {
506 | tuProperties.put("subject", currSubject);
507 | }
508 | if (currCustomer != null && !currCustomer.isEmpty()) {
509 | tuProperties.put("customer", currCustomer);
510 | }
511 | if (currProject != null && !currProject.isEmpty()) {
512 | tuProperties.put("project", currProject);
513 | }
514 | List tuvs = tu.getChildren("tuv");
515 | Set tuLangs = Collections.synchronizedSortedSet(new TreeSet<>());
516 |
517 | Iterator it = tuvs.iterator();
518 | while (it.hasNext()) {
519 | Element tuv = it.next();
520 | String lang = TMUtils.normalizeLang(tuv.getAttributeValue("xml:lang"));
521 | if (lang == null) {
522 | // Invalid language code, ignore this tuv
523 | continue;
524 | }
525 | storeLanguage(lang);
526 | if (!isNew) {
527 | removeTuv(lang, tuid);
528 | }
529 | if (!tuLangs.contains(lang)) {
530 | Element seg = tuv.getChild("seg");
531 | String puretext = TMUtils.extractText(seg);
532 | if (puretext.isBlank()) {
533 | continue;
534 | }
535 |
536 | String segText = seg.toString();
537 | segText = segText.substring("".length());
538 | segText = segText.substring(0, segText.length() - " ".length());
539 | storeTuv(lang, tuid, puretext, segText);
540 |
541 | int[] ngrams = NGrams.getNGrams(puretext);
542 | storeNgrams(lang, tuid, ngrams, puretext.length());
543 |
544 | tuLangs.add(lang);
545 | }
546 | }
547 | if (!isNew) {
548 | removeTuProperties(tuid);
549 | }
550 | storeTuProperties(tuid, tuProperties);
551 | }
552 |
553 | private boolean checkPreviousTu(String tuid) throws SQLException {
554 | if (checkTu == null) {
555 | checkTu = conn.prepareStatement("SELECT COUNT(*) FROM `" + dbName + "`.tuprop WHERE tuid=?");
556 | }
557 | checkTu.setString(1, tuid);
558 | int count = 0;
559 | try (ResultSet rs = checkTu.executeQuery()) {
560 | while (rs.next()) {
561 | count = rs.getInt(1);
562 | }
563 | }
564 | return count != 0;
565 | }
566 |
567 | private void storeNgrams(String lang, String tuid, int[] ngrams, int segSize) throws SQLException {
568 | PreparedStatement stmt = insertNgram.get(lang);
569 | if (stmt == null) {
570 | stmt = conn.prepareStatement("INSERT INTO `" + dbName + "`.matrix_" + lang.replace('-', '_').toLowerCase()
571 | + " (tuid, ngram, segSize) VALUES (?,?,?)");
572 | insertNgram.put(lang, stmt);
573 | }
574 | stmt.setString(1, tuid);
575 | stmt.setInt(3, segSize);
576 | for (int i = 0; i < ngrams.length; i++) {
577 | stmt.setInt(2, ngrams[i]);
578 | stmt.execute();
579 | }
580 | }
581 |
582 | private void storeTuv(String lang, String tuid, String puretext, String segText) throws SQLException {
583 | if (insertTuv == null) {
584 | insertTuv = conn
585 | .prepareStatement("INSERT INTO `" + dbName + "`.tuv (lang, tuid, seg, pureText ) VALUES (?,?,?,?)");
586 | }
587 | insertTuv.setString(1, lang);
588 | insertTuv.setString(2, tuid);
589 | insertTuv.setNString(3, segText);
590 | insertTuv.setNString(4, puretext);
591 | insertTuv.execute();
592 | }
593 |
594 | private void removeTuProperties(String tuid) throws SQLException {
595 | if (removeProperties == null) {
596 | removeProperties = conn.prepareStatement("DELETE FROM `" + dbName + "`.tuprop WHERE tuid=?");
597 | }
598 | removeProperties.setString(1, tuid);
599 | removeProperties.execute();
600 | }
601 |
602 | private void storeTuProperties(String tuid, Hashtable properties) throws SQLException {
603 | if (insertProperties == null) {
604 | insertProperties = conn
605 | .prepareStatement("INSERT INTO `" + dbName + "`.tuprop (tuid, propType, content) VALUES (?,?,?)");
606 | }
607 | insertProperties.setString(1, tuid);
608 | Set keys = properties.keySet();
609 | Iterator it = keys.iterator();
610 | while (it.hasNext()) {
611 | String prop = it.next();
612 | insertProperties.setString(2, prop);
613 | insertProperties.setNString(3, properties.get(prop));
614 | insertProperties.execute();
615 | }
616 | }
617 |
618 | private void removeTuv(String lang, String tuid) throws SQLException {
619 | if (removeTuv == null) {
620 | removeTuv = conn.prepareStatement("DELETE FROM `" + dbName + "`.tuv WHERE tuid=? AND lang=?");
621 | }
622 | removeTuv.setString(1, tuid);
623 | removeTuv.setString(2, lang);
624 | removeTuv.execute();
625 | PreparedStatement stmt = removeNgram.get(lang);
626 | if (stmt == null) {
627 | stmt = conn.prepareStatement(
628 | "DELETE FROM `" + dbName + "`.matrix_" + lang.replace('-', '_').toLowerCase() + " WHERE tuid=?");
629 | removeNgram.put(lang, stmt);
630 | }
631 | stmt.setString(1, tuid);
632 | stmt.execute();
633 | }
634 |
635 | private void storeLanguage(String lang) throws SQLException {
636 | if (languages == null) {
637 | getAllLanguages();
638 | }
639 | if (!languages.contains(lang)) {
640 | try (PreparedStatement stmt = conn
641 | .prepareStatement("INSERT INTO `" + dbName + "`.langs (lang) VALUES (?)")) {
642 | stmt.setString(1, lang);
643 | stmt.execute();
644 | }
645 | try (Statement stmt = conn.createStatement()) {
646 | stmt.execute("CREATE TABLE `" + dbName + "`.matrix_" + lang.replace('-', '_').toLowerCase() + " ("
647 | + " tuid VARCHAR(30) NOT NULL, ngram INTEGER NOT NULL, segSize INTEGER,"
648 | + " INDEX `ngrams` (`ngram` ASC) VISIBLE, PRIMARY KEY (tuid,ngram));");
649 | }
650 | conn.commit();
651 | languages.add(lang);
652 | }
653 | }
654 |
655 | @Override
656 | public void commit() throws SQLException {
657 | conn.commit();
658 | }
659 |
660 | private Element getTu(String tuid, Set langs)
661 | throws SQLException, SAXException, IOException, ParserConfigurationException {
662 | if (tuAttributes == null) {
663 | tuAttributes = Collections.synchronizedSortedSet(new TreeSet<>());
664 | String[] array = new String[] { "tuid", "o-encoding", "datatype", "usagecount", "lastusagedate",
665 | "creationtool", "creationtoolversion", "creationdate", "creationid", "changedate", "segtype",
666 | "changeid", "o-tmf", "srclang" };
667 | for (int i = 0; i < array.length; i++) {
668 | tuAttributes.add(array[i]);
669 | }
670 | }
671 | Hashtable properties = getTuProperies(tuid);
672 | Element tu = new Element("tu");
673 | Set keys = properties.keySet();
674 | Iterator it = keys.iterator();
675 | while (it.hasNext()) {
676 | String key = it.next();
677 | if (tuAttributes.contains(key)) {
678 | tu.setAttribute(key, properties.get(key));
679 | } else {
680 | Element prop = new Element("prop");
681 | prop.setAttribute("type", key);
682 | prop.setText(properties.get(key));
683 | tu.addContent(prop);
684 | }
685 | }
686 | if (langs.isEmpty()) {
687 | langs = getAllLanguages();
688 | }
689 | it = langs.iterator();
690 | while (it.hasNext()) {
691 | String lang = it.next();
692 | String seg = getSegText(lang, tuid);
693 | if (!seg.isEmpty()) {
694 | Element tuv = TMUtils.buildTuv(lang, seg);
695 | tu.addContent(tuv);
696 | }
697 | }
698 | return tu;
699 | }
700 |
701 | @Override
702 | public Element getTu(String tuid) throws IOException, SAXException, ParserConfigurationException, SQLException {
703 | return getTu(tuid, Collections.synchronizedSortedSet(new TreeSet<>()));
704 | }
705 |
706 | private String getSegText(String lang, String tuid) throws SQLException {
707 | if (selectSeg == null) {
708 | selectSeg = conn.prepareStatement("SELECT seg FROM `" + dbName + "`.tuv WHERE tuid=? AND lang=?");
709 | }
710 | String seg = "";
711 | selectSeg.setString(1, tuid);
712 | selectSeg.setString(2, lang);
713 | try (ResultSet rs = selectSeg.executeQuery()) {
714 | while (rs.next()) {
715 | seg = rs.getNString(1);
716 | }
717 | }
718 | return seg;
719 | }
720 |
721 | private Hashtable getTuProperies(String tuid) throws SQLException {
722 | if (selectProperties == null) {
723 | selectProperties = conn
724 | .prepareStatement("SELECT propType, content FROM `" + dbName + "`.tuprop WHERE tuid=?");
725 | }
726 | selectProperties.setString(1, tuid);
727 | Hashtable properties = new Hashtable<>();
728 | try (ResultSet rs = selectProperties.executeQuery()) {
729 | while (rs.next()) {
730 | properties.put(rs.getString(1), rs.getNString(2));
731 | }
732 | }
733 | return properties;
734 | }
735 |
736 | private String nextId() {
737 | if (next == 0l) {
738 | next = Calendar.getInstance().getTimeInMillis();
739 | }
740 | return "" + next++;
741 | }
742 |
743 | private static void writeHeader(FileOutputStream output, String srcLang, Map properties)
744 | throws IOException {
745 | writeString(output, "\n");
746 | writeString(output,
747 | "\n");
748 | writeString(output, "\n");
749 | writeString(output,
750 | " keys = properties.keySet();
759 | Iterator it = keys.iterator();
760 | while (it.hasNext()) {
761 | String key = it.next();
762 | writeString(output, " " + properties.get(key) + " \n");
763 | }
764 | writeString(output, " \n");
765 | }
766 | }
767 |
768 | private static void writeString(FileOutputStream output, String string) throws IOException {
769 | output.write(string.getBytes(StandardCharsets.UTF_8));
770 | }
771 |
772 | @Override
773 | public void removeTu(String tuid) throws IOException, SAXException, ParserConfigurationException, SQLException {
774 | Element tu = getTu(tuid);
775 | removeTuProperties(tuid);
776 | List tuvs = tu.getChildren("tuv");
777 |
778 | Iterator it = tuvs.iterator();
779 | while (it.hasNext()) {
780 | Element tuv = it.next();
781 | String lang = tuv.getAttributeValue("xml:lang");
782 | removeTuv(lang, tuid);
783 | }
784 | }
785 | }
786 |
--------------------------------------------------------------------------------
The methods exposed by
-
9 |
public abstract String getName(); 10 |Returns the name of the engine instance.
11 |
12 | public abstract String getType(); 13 |Returns the
15 |ITmEngine type used for storing data. Possible 14 | values are:-
16 |
- MapDbEngine 17 |
- SQLEngine 18 |
20 |
21 | public abstract void close() throws IOException, SQLException; 22 |Closes the engine instance.
23 |
24 |
25 | public abstract int storeTMX(String tmxFile, String project, String customer, 26 | String subject) throws SAXException, IOException, 27 | ParserConfigurationException, SQLException; 28 |Imports a TMX document located at "
31 |tmxFile " and associates its 29 | data with "project ", "customer " and 30 | "subject " properties
32 |
33 | public abstract void exportMemory(String tmxfile, Set<String> langs, String 34 | srcLang, Map<String, String> properties) throws IOException, 35 | SAXException, ParserConfigurationException, SQLException; 36 |Exports engine data to a TMX document located at "
37 |tmxfile ".The "
40 |langs " argument may contain a set of language codes. If 38 | "langs " is not null, only the entries with the given 39 | language codes are exported.The "
44 |srcLang " argument indicates the source language assigned to 41 | the TMX file. It can be one of the languages from the data set (see "getAllLanguages() ") or the string 43 | "*all* ".The "
46 |properties " argument may contain a set of property-value 45 | pairs to be set in the exported TMX file.
47 |
48 | public abstract void flag(String tuid) throws SQLException; 49 |Adds the property "
52 |x-flag " and sets its value to 50 | "SW-Flag " to the translation unit identified by the 51 | "tuid " argument.
53 |
54 | public abstract Set<String> getAllLanguages() throws SQLException; 55 |Returns a collection containing all language codes used in the engine's data.
56 |
57 |
58 | public abstract Set<String> getAllClients() throws SQLException; 59 |Returns a collection containing all values assigned to the 60 | "
61 |client " property.
62 |
63 | public abstract Set<String> getAllProjects() throws SQLException; 64 |Returns a collection containing all values assigned to the 65 | "
66 |project " property.
67 |
68 | public abstract Set<String> getAllSubjects() throws SQLException; 69 |Returns a collection containing all values assigned to the 70 | "
71 |subject " property.
72 |
73 | public abstract List<Match> searchTranslation(String searchStr, String 74 | srcLang, String tgtLang, int similarity, boolean 75 | caseSensitive) throws IOException, SAXException, 76 | ParserConfigurationException, SQLException; 77 |Returns a list of possible translations of the "
79 |searchStr " 78 | argument.The search result is restricted to entries with the source language indicated by 80 | "
83 |srcLang " and target language indicated by 81 | "tgtLang " whose similarity to the given text is greater or 82 | equal to the value of the "similarity " argument.The "
85 |caseSensitive " argument indicates whether the search engine 84 | should consider letter case differences or not.
86 |
87 | public abstract List<Element> concordanceSearch(String searchStr, 88 | String srcLang, int limit, boolean isRegexp, boolean 89 | caseSensitive) throws IOException, SAXException, 90 | ParserConfigurationException, SQLException; 91 |Returns a list of all translation units (<tu> elements) that contain the text 92 | indicated in "
93 |searchStr " argument.Searches are performed against the translation unit variant (<tuv> element) 94 | with language set to "
95 |srcLang ".Search result contains at most "
97 |limit " entries. Returned data 96 | is in no particular order.Argument "
100 |isRegexp " indicates whether the 98 | "searchStr " parameter should be considered a regular 99 | expression that matches the whole segment.The "
102 |caseSensitive " argument indicates whether the search engine 101 | should consider letter case differences or not.
103 |
104 | public abstract void storeTu(Element tu) throws IOException, SQLException; 105 |Stores translation unit "
107 |tu " into the database, overwriting any 106 | existing <tu> element with the same "id" attribute.
108 |
109 | public abstract void commit() throws SQLException; 110 |Flushes to disk any data held in memory and not written yet.
111 |
112 |
113 | public abstract Element getTu(String tuid) throws IOException, 114 | SAXException, ParserConfigurationException, SQLException; 115 |Returns the translation unit (<tu> element) that has the "id" attribute set to 116 | the "
117 |tuid " argument.
118 |
119 | public abstract void removeTu(String tuid) throws IOException, SAXException, 120 | ParserConfigurationException, SQLException; 121 |Removes from the database the <tu> element that has the "id" attribute set to 122 | the "
123 |tuid " argument.
124 |
125 | public abstract void deleteDatabase() throws IOException, SQLException; 126 |-
127 |
- When used with "
MapDbEngine " instances, closes the engine 128 | and removes all associated files from disk;
129 | - When used with "
SQLEngine " instances, closes the engine and 130 | drops the associated database from the SQL server.
131 |
133 | - When used with "