├── LICENSE
├── README.md
├── _attachments
├── images
│ ├── csv2Couchdb.png
│ ├── favicon.png
│ ├── html5-badge-h-performance.png
│ ├── localCSVs.png
│ ├── recline-logo.png
│ ├── spinner.gif
│ └── square-logo - 23 x 23.png
├── index.html
├── license.html
├── scripts
│ ├── app.js
│ ├── bulkLoad.js
│ ├── jquery-1.6.1.min.js
│ ├── jquery.couch.app.js
│ ├── jquery.couch.app.util.js
│ ├── jquery.couch.js
│ └── jquery.csvIn.min.js
└── style
│ └── main.css
├── _id
├── couchapp.json
└── language
/LICENSE:
--------------------------------------------------------------------------------
1 | csv2couchdb is released under the MIT License:
2 |
3 | ----------------------------------------------------------
4 | Copyright (c) 2011 Mango Information Systems SPRL, http://www.mango-is.com
5 |
6 | Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
7 |
8 | The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
9 |
10 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
11 | ----------------------------------------------------------
12 |
13 | Source code is available on github: https://github.com/Mango-information-systems/csv2couchdb
14 |
15 | csv2couchdb uses the following components:
16 |
17 | couchdb - Apache License version 2: http://www.apache.org/licenses/LICENSE-2.0
18 | jQuery javascript library - MIT License: http://github.com/jquery/jquery/blob/master/MIT-LICENSE.txt
19 | csvIn jQuery plugin - MIT License: https://github.com/Mango-information-systems/jquery.csvIn/blob/master/LICENSE
20 |
21 | Some of the visuals of this application have been derived from the following sources:
22 |
23 | couchdb logo (c) Damien Katz - Creative Commons Attribution Unported 3: http://creativecommons.org/licenses/by/3.0/deed.en
24 | HTML5 logo (c) W3C - Creative Commons Attribution Unported 3: http://creativecommons.org/licenses/by/3.0/deed.en
25 | public domain cliparts from http://www.openclipart.org/
26 | public domain spinner from http://mentalized.net/activity-indicators/
27 | logo of Google refine - new BSD License: http://code.google.com/p/google-refine/source/browse/trunk/LICENSE.txt
28 |
29 |
--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
1 | ##csv2couchdb - populate couchdb from delimited files
2 |
3 | csv2couchdb is a couchapp allowing to populate couchdb using data from CSV files. It takes advantage of the HTML5 file API to process files at client side before the upload to couchdb.
4 |
5 | ### Features:
6 |
7 | * Read local files
8 | * Select multiple files at a time
9 | * Automatic detection of CSV format
10 | * Preview loaded files to adjust settings
11 | * Customize document to generate
12 | * Filter rows and columns
13 | * Customize header labels
14 | * Generate either one doc per file or one doc per row
15 | * Generate ids based on file name or use couchdb random ids
16 | * Bulk load to couchdb
17 | * Select target database
18 | * Overwrite existing documents in case of conflict (optional)
19 |
20 | ### Usage examples:
21 |
22 | Consider the following CSV file that would be stored in your computer:
23 |
24 | color;popularity
25 | blue;5
26 | green;4.5
27 | red;3
28 | orange;4
29 |
30 | 1. Generate one document per file
31 |
32 | You would insert the following document to couchdb:
33 |
34 | ````javascript
35 | {
36 | "headers": ["color", "popularity"],
37 | "rows":[
38 | ["blue","5"],
39 | ["green","4.5"],
40 | ["red","3"],
41 | ["orange","4"]
42 | ]
43 | }
44 | ````
45 |
46 | 2. Generate one document per row
47 |
48 | You would insert the following documents to couchdb:
49 |
50 | ````javascript
51 | {
52 | "color" : "blue",
53 | "popularity" : "5"
54 | },
55 | {
56 | "color" : "green",
57 | "popularity": "4.5"
58 | },
59 | {
60 | "color" : "red",
61 | "popularity" : "3"
62 | },
63 | {
64 | "color" : "orange",
65 | "popularity" : "4"
66 | }
67 | ````
68 |
69 | 3. Filter rows
70 |
71 | Setting option `Get lines from 1 to 2` would generate a document containing only the first two lines (`blue` and `green` data)
72 |
73 | 4. Filter columns
74 |
75 | Untick the column header `popularity` to insert only colors.
76 |
77 | 5. Customize properties names:
78 |
79 | Click on `popularity` in the preview and replace it by another label you want, eg `rating`
80 |
81 | ### Demo
82 |
83 | ~~A demo is available at this location: http://mango-reports.cloudant.com/mango-apps/_design/csv2couchdb/index.html
84 | **Warning**: the demo couch is read-only, so you will not have access to the whole application. We recommend replicating to your own couch to get all features (see next section).~~
85 |
86 | ###Installation
87 |
88 | ~~Simply replicate the sample couchapp to your couchdb instance:
89 |
90 | curl -X POST http://user:pass@YOURCOUCH/_replicate -d '{"source":"http://mango-reports.cloudant.com/mango-apps/","target":"YOURDB", "doc_ids":["_design/csv2couchdb"]}' -H "Content-type: application/json"~~
91 |
92 | Clone this repository into the right folder, and you should be good to go.
93 |
94 | After you install it, the app is available from this url:
95 |
96 | http://yourcouch/yourdb/_design/csv2couchdb/index.html
97 |
98 |
99 | ### Status
100 |
101 | csv2couchdb is a new software and certainly contains bugs. It has been tested in Firefox 5 and Google chrome 12. Large files could be inserted using Firefox, whereas their parsing caused Google Chrome tab to crash.
102 |
103 | Consider this app as an alpha software.
104 |
105 | Thanks for reporting any issue that you would find.
106 |
107 | ### Roadmap
108 |
109 | This app is not maintained anymore. Compatibility with newer versions of couchdb has not been checked.
110 |
111 | contact: either via github message, twitter ([@mango_info](http://twitter.com/mango_info)) or via contact form in http://www.mango-is.com
112 |
113 | ### License and credits
114 |
115 | Please refer to the LICENSE file located in the same folder as the current file
116 |
--------------------------------------------------------------------------------
/_attachments/images/csv2Couchdb.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/Mango-information-systems/csv2couchdb/b78bb525df8242bb8998c2d6aeb45260147bc8bf/_attachments/images/csv2Couchdb.png
--------------------------------------------------------------------------------
/_attachments/images/favicon.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/Mango-information-systems/csv2couchdb/b78bb525df8242bb8998c2d6aeb45260147bc8bf/_attachments/images/favicon.png
--------------------------------------------------------------------------------
/_attachments/images/html5-badge-h-performance.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/Mango-information-systems/csv2couchdb/b78bb525df8242bb8998c2d6aeb45260147bc8bf/_attachments/images/html5-badge-h-performance.png
--------------------------------------------------------------------------------
/_attachments/images/localCSVs.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/Mango-information-systems/csv2couchdb/b78bb525df8242bb8998c2d6aeb45260147bc8bf/_attachments/images/localCSVs.png
--------------------------------------------------------------------------------
/_attachments/images/recline-logo.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/Mango-information-systems/csv2couchdb/b78bb525df8242bb8998c2d6aeb45260147bc8bf/_attachments/images/recline-logo.png
--------------------------------------------------------------------------------
/_attachments/images/spinner.gif:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/Mango-information-systems/csv2couchdb/b78bb525df8242bb8998c2d6aeb45260147bc8bf/_attachments/images/spinner.gif
--------------------------------------------------------------------------------
/_attachments/images/square-logo - 23 x 23.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/Mango-information-systems/csv2couchdb/b78bb525df8242bb8998c2d6aeb45260147bc8bf/_attachments/images/square-logo - 23 x 23.png
--------------------------------------------------------------------------------
/_attachments/index.html:
--------------------------------------------------------------------------------
1 |
2 |
3 |
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
13 |
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
14 |
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
37 |
38 |
--------------------------------------------------------------------------------
/_attachments/scripts/app.js:
--------------------------------------------------------------------------------
1 | /*
2 | csv2couchdb couchapp released under MIT License
3 | (c) Mango Information Systems SPRL - 2011-2012
4 | version 0.3 - February, 27th 2012
5 |
6 | todo:
7 | handle file not conform error (when file is not a csv)
8 | handle file load error: the given file should be excluded (ideally retry option) so that other files still can be loaded
9 | */
10 |
11 | /* Extract data from csv files at client side then push them into couchdb
12 | Consists in 5 steps:
13 | 1) select files (HTML5 file API)
14 | 2) define settings for each file
15 | 3) set documents and couchdb settings
16 | 4) get feedback about bulk insert and select files to overwrite in case of conflict and retry in case of error
17 | 5) (if applicable) get feedback about overwrites / retries
18 | */
19 |
20 | // fileData contains data from read files
21 | var filesData = [];
22 |
23 | // currentStepkeeps track of steps
24 | var currentStep = 1;
25 |
26 | var $db = '';
27 |
28 | function errorHandler(evt) {
29 | // handle errors occuring when reading file
30 | switch(evt.target.error.code) {
31 | case evt.target.error.NOT_FOUND_ERR:
32 | alert('File Not Found!');
33 | break;
34 | case evt.target.error.NOT_READABLE_ERR:
35 | console.log(evt.target.error);
36 | alert('File is not readable');
37 | break;
38 | case evt.target.error.ABORT_ERR:
39 | break; // noop
40 | default:
41 | alert('An error occurred reading this file.');
42 | console.log(evt.target.error);
43 | };
44 | }
45 |
46 | function updateProgress(evt, fileIndex) {
47 | // update the file reading progress status display
48 | // evt is an ProgressEvent.
49 | progress = $('#bar'+fileIndex);
50 |
51 | if (evt.lengthComputable) {
52 | var percentLoaded = Math.round((evt.loaded / evt.total) * 100);
53 | // increase the progress bar length.
54 | if (percentLoaded < 100) {
55 | progress.find('.percent').width(percentLoaded+'%');
56 | progress.find('.percent').text( percentLoaded+'%');
57 | }
58 | }
59 | }
60 |
61 | function handleFileSelect(evt) {
62 | var files = evt.target.files; // FileList object
63 |
64 | // files is a FileList of File objects. List some properties.
65 | var output = [];
66 |
67 | var counter=0;
68 |
69 | for (var i = 0, f; f = files[i]; i++) {
70 | // 1. show input and processing options for each file
71 |
72 | var $fileDiv = $('#step2 .template')
73 | .clone()
74 | .removeClass('template')
75 | .attr('id','file'+i)
76 |
77 | // set ids for the elements of the div (necessary to have labels clickable)
78 | $fileDiv.find('input').each(function() {
79 | $(this).attr('id', $(this).attr('id')+i);
80 | $(this).prop('checked', false);
81 | });
82 | $fileDiv.find('label').each(function() {
83 | $(this).attr('for', $(this).attr('for')+i);
84 | });
85 |
86 | $fileDiv.find('.progress_bar').attr('id','bar'+i);
87 |
88 | $fileDiv
89 | .appendTo('#filesContainer')
90 | .fadeIn();
91 |
92 | var reader = new FileReader();
93 | // Closure to capture the file information.
94 |
95 | progress = $('#bar'+counter);
96 |
97 | // Initialize progress indicator on new file selection.
98 | progress.find('.percent').width('0%');
99 | progress.find('.percent').text('0%');
100 |
101 | reader.onerror = errorHandler;
102 |
103 | reader.onprogress = function(e) {
104 | // handle progress of the file read
105 | updateProgress(e,counter);
106 | }
107 |
108 | // onload event necessary so that processing starts only once the file is read
109 | reader.onload = (function(theFile) {
110 | return function(e) {
111 | // Send file data to csv processor.
112 |
113 | var $input = $('#file'+counter);
114 |
115 | // display a spinner during the file update
116 | $input.find('div.filePreview > div:nth-child(2)').html(' Parsing CSV data...');
117 |
118 | // update file preview div title
119 | $input.find('div.filePreview > div:nth-child(1)').text('Preview of 5 first rows');
120 |
121 | // add data to the global variable storing all files
122 | filesData.push({"name": theFile.name, "index":counter, "data": e.target.result});
123 |
124 | // process the current file
125 | processCsv($input, {});
126 |
127 | counter++;
128 | };
129 | })(f);
130 |
131 | // Read in the image file as a data URL.
132 | reader.readAsText(f);
133 |
134 | }
135 |
136 | currentStep++;
137 |
138 | // 2. show output options
139 | //displayOutputSettings();
140 |
141 | // 3. hide step 1
142 | $('#step1').hide();
143 |
144 | // 4. show step 2
145 | $('#step2')
146 | .show();
147 |
148 | }
149 |
150 | function getFileOptions ($fileDiv) {
151 | // return values of the csv file settings chosen by the user
152 | // based on the page's form
153 | // return an object with following hashes: delimitors, delim, headerCheck, quote, quoteMark, startLine, endLine, excludedColumns
154 | // delim and quote are symbols whereas delimitors and quoteMark are text values
155 | var result = {};
156 |
157 | result.delimitors = [];
158 | $fileDiv.find('input[name="delimitor"]').each(function() {
159 | // get selected delimitors (preset delimitors)
160 | // .each used instead of for loop to save three selections of same item (item, then prop, then id)
161 | if($(this).prop('checked') && this.value !="custom") {
162 | result.delimitors.push(this.value);
163 | }
164 | });
165 | if ($fileDiv.find('.customdelimitors').val() != "") {
166 | // get custom delimitor if exists
167 | result.delimitors.push($fileDiv.find('.customdelimitors').val());
168 | }
169 |
170 | delim = encodeDelimitors(result.delimitors);
171 | result.delim = delim.regular.join('') + delim.custom;
172 |
173 | // get headerCheck value
174 | result.headerCheck = $fileDiv.find('input[name="header"]').prop('checked');
175 |
176 | result.quoteMark= "";
177 | // get quote marker
178 | $fileDiv.find('input[name="quoteMark"]').each(function() {
179 | // get selected quoteMark
180 | // .each used instead of for loop to save three selections of same item (item, then prop, then id)
181 | if($(this).prop('checked')) {
182 | result.quoteMark = this.value;
183 | }
184 | });
185 |
186 | result.startLine = $fileDiv.find('[type="text"][name="startLine"]').val();
187 |
188 | result.endLine = $fileDiv.find('[type="text"][name="endLine"]').val();
189 |
190 | result.excludedColumns = [];
191 |
192 | $('span.excluded', $fileDiv).each(function() {
193 | // addding each excluded column to the list
194 | result.excludedColumns.push($(this).parent()[0].cellIndex);
195 | });
196 |
197 | return result;
198 | }
199 |
200 | function processCsv($fileDiv, fileOpts) {
201 | // step 2: show options and preview for csv file
202 |
203 | if (!fileOpts)
204 | fileOpts = {};
205 | // retrieving index of the file
206 | var fileIndex = $fileDiv.attr('id').substr(4,1);
207 | var file = filesData[fileIndex];
208 |
209 | // detect column delimitor from a standard list at first run
210 | if (!fileOpts.delimitors) {
211 | fileOpts.delimitors = [];
212 |
213 | // detect delimitor
214 | fileOpts.delim = $.csvIn.detectDelimitor(file.data);
215 | // store string form of the delimitor
216 | fileOpts.delimitors.push(decodeDelimitor(fileOpts.delim));
217 | }
218 |
219 | // initialize quote marker at first run
220 | if(!fileOpts.quoteMark) {
221 | fileOpts.quoteMark = 'doubleQuote';
222 | fileOpts.quote= encodeQuoteMarkers(fileOpts.quoteMark);
223 | }
224 |
225 | // ignore excluded columns for the preview generation
226 | fileOpts.excludedColumns = [];
227 |
228 | // override startLine for the preview generation (header must appear)
229 | fileOpts.startLine = 0;
230 |
231 | // endLine not overriden for the preview generation because full parsing is needed to display total length
232 | // fileOpts.endLine = 6;
233 | // convert the file content into a javascript array
234 |
235 | var currentCsvArray = $.csvIn.toArray(file.data, fileOpts);
236 |
237 | // console.log($.csvIn.toJSON(file.data, fileOpts));
238 |
239 | if(typeof fileOpts.headerCheck == 'undefined')
240 | // detect presence of header in case of first run
241 | fileOpts.headerCheck = $.csvIn.isHeader(currentCsvArray[0]);
242 |
243 | // setting file name
244 | $fileDiv.find('.divTitle').eq(0).text(file.name);
245 |
246 | for (i in fileOpts.delimitors) {
247 | // check checkbox for selected delimitors
248 | if ($.inArray(fileOpts.delimitors[i], ["space", "colon", "comma", "semicolon", "hyphen", "tab"]) !=-1) {
249 | // regular delimitor, check appropriate checkbox
250 | $fileDiv.find('#'+fileOpts.delimitors[i]+file.index).prop('checked', true);
251 | }
252 | else {
253 | // delimitor is custom one, check custom checkbox and fill value inside text input
254 | if (typeof fileOpts.delimitors[i]!='undefined' && fileOpts.delimitors[i] !="") {
255 | // check custom only if there is a defined delimitor
256 | $fileDiv.find('#custom'+file.index).prop('checked', true);
257 | $fileDiv.find('#customdelimitors'+file.index).append(fileOpts.delimitors[i]);
258 | }
259 | }
260 | }
261 |
262 | // check radio button for the selected quote marker
263 | $fileDiv.find('#'+fileOpts.quoteMark+file.index).prop('checked', true);
264 |
265 | if (fileOpts.headerCheck)
266 | // check header checkbox in case a header has been detected
267 | $fileDiv.find('[type="checkbox"][name="header"]').prop('checked',true);
268 |
269 | // initialize row filters at first run
270 | if(!fileOpts.startLine) {
271 | fileOpts.startLine = 1;
272 | if (fileOpts.headerCheck)
273 | // increment start line to skip header if applicable
274 | fileOpts.startLine++;
275 | }
276 |
277 | // initialize row filters at first run
278 | if(!fileOpts.endLine) {
279 | fileOpts.endLine = currentCsvArray.length;
280 | }
281 |
282 | $fileDiv.find('[type="text"][name="startLine"]').val(fileOpts.startLine);
283 |
284 | $fileDiv.find('[type="text"][name="endLine"]').val(fileOpts.endLine);
285 |
286 | $fileDiv.find('span.totalRows').text('Total file length: '+currentCsvArray.length+' lines');
287 |
288 | // show file preview
289 | var result = previewCsv(currentCsvArray, fileOpts.headerCheck);
290 | $fileDiv.find('div.filePreview > div:nth-child(2)').html(result);
291 |
292 | }
293 |
294 | function previewCsv(data, headerCheck) {
295 | // show a preview of the csv file
296 |
297 | var result = "
";
298 | if(headerCheck) {
299 | result += "
";
300 |
301 | for (i in data[0]) {
302 | result += "
"+data[0][i]+"
";
303 | }
304 | result += "
";
305 | }
306 | else {
307 | result += "
";
308 |
309 | for (i in data[0]) {
310 | result += "
Column "+ (parseInt(i)+1) +"
";
311 | }
312 | result += "
";
313 | }
314 |
315 | // set length of the preview according to headerCheck and length of the file
316 | if (headerCheck) {
317 | var startLine = 1;
318 | var endLine = data.length>6?5:data.length-1;
319 | }
320 | else {
321 | var startLine = 0;
322 | var endLine = data.length>5?4:data.length-1;
323 | }
324 |
325 | result += "";
326 |
327 | for (i =startLine;i<=endLine;i++) {
328 | result += "
";
329 | for (j in data[i]) {
330 | result += "
"+data[i][j]+"
";
331 | }
332 | result += "
";
333 | }
334 | result +="
";
335 |
336 | return result;
337 | }
338 |
339 | function columnSelection($targetTable, columnIds) {
340 | // add or remove excluded class to the given columns
341 |
342 | for (i in columnIds) {
343 | // toggle class for all cells belonging to the column
344 | $targetTable.find('tr >td:nth-child('+columnIds[i]+')').toggleClass('excluded');
345 | $targetTable.find('tr >th:nth-child('+columnIds[i]+') >span').toggleClass('excluded');
346 | }
347 |
348 | }
349 |
350 | function decodeDelimitor(delim) {
351 | // return name corresponding to delimitor code
352 | // necessary because signs cannot be used in DOM element ids
353 | switch(delim) {
354 | case " ":
355 | return "space";
356 | break;
357 | case ",":
358 | return "comma";
359 | break;
360 | case ":":
361 | return "colon";
362 | break;
363 | case ";":
364 | return "semicolon";
365 | break;
366 | case "-":
367 | return "hyphen";
368 | break;
369 | case "\t":
370 | return "tab";
371 | break;
372 | }
373 | }
374 |
375 | function encodeDelimitors(delimitors) {
376 | // encode delimitors
377 |
378 | var result = {};
379 | result.regular = [];
380 | result.custom = "";
381 |
382 | for (i in delimitors) {
383 | // encode delimitors
384 | switch (delimitors[i]) {
385 | case "space":
386 | result.regular.push(" ");
387 | break;
388 | case "tab":
389 | result.regular.push("\t");
390 | break;
391 | case "colon":
392 | result.regular.push(":");
393 | break;
394 | case "semicolon":
395 | result.regular.push(";");
396 | break;
397 | case "comma":
398 | result.regular.push(",");
399 | break;
400 | case "hyphen":
401 | result.regular.push("-");
402 | break;
403 | default:
404 | if (typeof delimitors[i] != 'undefined' && delimitors[i] !="") {
405 | result.regular.push(delimitors[i]);
406 | result.custom = delimitors[i];
407 | }
408 | break;
409 | }
410 | }
411 | return result;
412 | }
413 |
414 | function encodeQuoteMarkers(mark) {
415 | // encode quote markers
416 | switch(mark) {
417 | // encode quote marker
418 | case "doubleQuote":
419 | result = "\"";
420 | break;
421 | case "simpleQuote":
422 | result = "'";
423 | break;
424 | case "none":
425 | result = 0;
426 | break;
427 | }
428 | return result;
429 | }
430 |
431 | function generateDocuments () {
432 | // generate JSON couchdb document
433 | // fileDiv is a jquery selector containing the files information, whereas opts are the options applicable to all files
434 | var result = [];
435 |
436 | // get output format value
437 | var outputFormat = $('input[name="outputFormat"]:checked').val();
438 |
439 | // get ids option value
440 | var ids = $('#ids').val();
441 |
442 | var $filesDiv = $('div.container', '#filesContainer');
443 |
444 | $filesDiv.each(function() {
445 | //$('#step'+currentStep+' >div').each(function() {
446 | // process individual file
447 |
448 | $input = $(this);
449 |
450 | var fileIndex = $input.attr('id').substr(4,1);
451 |
452 | // retrieving file options
453 | var fileOptions = getFileOptions ($input);
454 |
455 | // update start and end lines (0-indexed)
456 | fileOptions.startLine--;
457 | fileOptions.endLine--;
458 |
459 | fileOptions.customHeaders = [];
460 |
461 | $(this).find('th > span').each(function() {
462 | // retrieve custom headers labels
463 | fileOptions.customHeaders.push($(this).text().trim());
464 | });
465 |
466 | // generate JSON document
467 | if (outputFormat == "byRow") {
468 | // generate one document per data row
469 |
470 | // convert the file content into a javascript array of JSON objects
471 | JSONdoc = $.csvIn.toJSON(filesData[fileIndex].data, fileOptions);
472 |
473 | if (ids == 'custom') {
474 | // process with generation of custom _id : file name and row number
475 | for (i in JSONdoc) {
476 | // go through each row, generate id and add document
477 | JSONdoc[i]._id = filesData[fileIndex].name+ (parseInt(i)+1);
478 | result.push(JSONdoc[i]);
479 | }
480 | }
481 | else {
482 | // add document without generating id
483 | for (i in JSONdoc) {
484 | // go through each row to add document
485 | result.push(JSONdoc[i]);
486 | }
487 | }
488 |
489 | }
490 | else {
491 | // generate one document per file
492 |
493 | JSONdoc = {};
494 |
495 | // convert the file content into a javascript array
496 | JSONdoc.rows = $.csvIn.toArray(filesData[fileIndex].data, fileOptions);
497 |
498 | if (ids == 'custom')
499 | // generate custom _id : file name
500 | JSONdoc._id = filesData[fileIndex].name;
501 |
502 | JSONdoc.headers = fileOptions.customHeaders;
503 |
504 | // add array as a document
505 | result.push(JSONdoc);
506 | }
507 | });
508 | return result;
509 |
510 | }
511 |
512 | function pushToDB(docs) {
513 | // insert documents into couchdb
514 |
515 | // get output settings
516 | var dbName = $('#dbName').val();
517 |
518 | var bulkLoader = new Worker('scripts/bulkLoad.js');
519 |
520 | // bulk upload using workers - logic inspired by maxodgen's recline
521 | // https://github.com/maxogden/recline/blob/master/attachments/script/costco.js#L111
522 |
523 | var url = window.location.protocol + "//" + window.location.host + "/" + dbName + "/_bulk_docs";
524 |
525 | bulkLoader.postMessage({"docs":docs, "url":url });
526 |
527 | bulkLoader.onmessage = function (event) {
528 | // react according to result of processing
529 | var result = event.data;
530 |
531 | result = $.parseJSON(event.data);
532 |
533 | if(result.success) {
534 | // confirm that insertion was succcessfully done
535 |
536 | $('#bulkLoadSpinner').hide();
537 |
538 | $('#successDocs, #postProcessing').fadeIn();
539 |
540 | $db = $.couch.db(dbName);
541 |
542 | // check presence of recline design document in the database
543 | $db.allDocs({
544 | keys: ['_design/recline'],
545 | success: function(data) {
546 | if(!data.rows[0].error) {
547 | // insert link to recline in case recline design document exists in the target DB
548 | $('
Continue to recline
Process the documents in recline
').insertAfter('#moreFiles');
549 | // bind click event listener
550 | $('#recline').live('click',function() {
551 | // go to recline
552 | reclineUrl = '../../../' + $('#dbName').val() + '/_design/recline/_rewrite';
553 | window.location.href = reclineUrl;
554 | });
555 |
556 |
557 | }
558 | }
559 | });
560 |
561 | var successCount = 0;
562 | for (i in result.response) {
563 | if (!result.response[i].error) {
564 | // show confirmation for successful document insert
565 | // show list of inserted files by Id
566 | successCount++;
567 | }
568 | else if (result.response[i].error == 'conflict') {
569 | // show conflict documents and provide overwrite option
570 |
571 | // display the failed documents div in case it is not already visible
572 | $('#conflictDocs').not(':visible').fadeIn();
573 |
574 | var $conflictDoc = $('.template', '#conflictDocs')
575 | .clone()
576 | .removeClass('template')
577 | .attr('id','doc'+i)
578 |
579 | $conflictDoc.find('span')
580 | .text(result.response[i].id);
581 |
582 | $conflictDoc.find('input')
583 | .attr('fileName', docs[i]._id)
584 | .data('doc',docs[i])
585 | .data('formId','doc'+i);
586 |
587 | $conflictDoc
588 | .appendTo('#conflictDocs')
589 | .fadeIn();
590 |
591 | // get revision id of existing document
592 | $db.allDocs({
593 | keys: [result.response[i].id],
594 | context: $conflictDoc.find('input'),
595 | success: function(data) {
596 | this.context
597 | .data('rev',data.rows[0].value.rev);
598 | },
599 | error: function(jqXHR, textStatus, errorThrown) {
600 | // throw appropriate error message
601 | console.log(jqXHR);
602 | console.log(textStatus);
603 | console.log(errorThrown);
604 | }
605 | });
606 | }
607 | else {
608 | // show failed document inserts and provide retry option
609 |
610 | // display the failed documents div in case it is not already visible
611 | $('#failDocs').not(':visible').fadeIn();
612 |
613 | var $failDoc = $('.template', '#failDocs')
614 | .clone()
615 | .removeClass('template')
616 | .attr('id','doc'+i)
617 |
618 | $failDoc.find('span').eq(1)
619 | .text(result.response[i].id);
620 |
621 | $failDoc.find('span').eq(2)
622 | .text(result.response[i].error + ': ' + result.response[i].reason);
623 |
624 | $failDoc.find('input')
625 | .attr('fileName', docs[i]._id)
626 | .data('doc',docs[i])
627 | .data('formId','doc'+i);
628 |
629 | $failDoc
630 | .appendTo('#failDocs')
631 | .fadeIn();
632 |
633 | // get revision id of existing document
634 | $db.allDocs({
635 | keys: [result.response[i].id],
636 | context: $failDoc.find('input'),
637 | success: function(data) {
638 | this.context
639 | .data('rev',data.rows[0].value.rev);
640 | },
641 | error: function(jqXHR, textStatus, errorThrown) {
642 | // throw appropriate error message
643 | console.log(jqXHR);
644 | console.log(textStatus);
645 | console.log(errorThrown);
646 | }
647 | });
648 | }
649 | }
650 | if (successCount > 1)
651 | $('#successDocs').find('p').html(successCount + ' new documents were successfully inserted ');
652 | else if (successCount == 1)
653 | $('#successDocs').find('p').html('1 new document was successfully inserted');
654 | else
655 | // successCount == 0
656 | $('#successDocs').find('p').html('No new document was inserted');
657 |
658 | }
659 | else {
660 | // throw appropriate error message
661 | $('#step4').html(result.response.textStatus + ' ' + result.response.errorThrown);
662 | console.log(result.response.jqXHR);
663 | console.log(result.response.textStatus);
664 | console.log(result.response.errorThrown);
665 | }
666 | }
667 | }
668 |
669 | $(document).ready(function() {
670 |
671 | // Check whether user is already connected
672 | $.couch.session({
673 | success: function(data) {
674 | if (data.userCtx.name != null)
675 | // show logged in user name and logout button
676 | $('#session').html('logged in as '+data.userCtx.name+ ' ');
677 | else
678 | // show logon form
679 | $('#session').html('');
680 |
681 | },
682 | error: function(data) {
683 | // show logon form
684 | $('#session').html('');
685 | }
686 | });
687 |
688 | // Check for the various File API support.
689 | if (window.File && window.FileReader && window.FileList && window.Blob) {
690 | // Great success! All the File APIs are supported.
691 | }
692 | else {
693 | $('#techDisclaimer').css('border','3px solid red');
694 | $('#html5Status').html('The HTML5 File APIs are not fully supported in this browser. Please try another browser (Firefox, Google Chrome, Opera)');
695 | }
696 |
697 | // todo: check whether this UI is suitable (jqueryUI buttons)
698 | // $( ".fileSettings > fieldset" ).eq(0).buttonset();
699 |
700 |
701 | // trigger proper function at file selection
702 | // document.getElementById('files').addEventListener('change', handleFileSelect, false);
703 | $('#files').live('change', function(evt) {
704 | handleFileSelect(evt);
705 | });
706 |
707 | $('[type="submit"][name="next"]').live('click', function(e) {
708 | // go to next step
709 | // console.log($('#step'+currentStep+' >div'))
710 | e.preventDefault();
711 |
712 | // 1. hide current step
713 | $('#step'+currentStep)
714 | .hide()
715 |
716 | // 2. increment step
717 | currentStep++;
718 |
719 | // 3. show next step
720 | $('#step'+currentStep)
721 | .fadeIn()
722 |
723 | // 4. perform specific processing if applicable
724 | if (currentStep == 3) {
725 | // step 3, load databases list inside select
726 | $.couch.allDbs({
727 | success: function(data) {
728 | $dbsSelect = $('#dbName');
729 | $dbsSelect.empty();
730 | data.sort();
731 | for (i in data) {
732 | if (data[i][0] !='_') {
733 | $dbsSelect
734 | .append($("")
735 | .attr("value",data[i])
736 | .text(data[i]));
737 | }
738 | }
739 | },
740 | error: function(jqXHR, textStatus, errorThrown) {
741 | // replace dbs list with text input in case the database list is not accessible to the loged-in user
742 | $('#dbName').replaceWith('');
743 | }
744 | });
745 | }
746 | else if (currentStep == 4) {
747 | // step 4, load documents into couchdb
748 | // 4.2 generate couchdb documents
749 | var docs = generateDocuments ();
750 | // temporary preview of the documents:
751 | /* var myText = '
'+JSON.stringify(docs, null, '\t')+'
';
752 | $('#step'+currentStep)
753 | .html(myText)
754 | */
755 | // 4.3 push documents to couchdb
756 | pushToDB(docs);
757 | }
758 |
759 | });
760 |
761 | $('[type="submit"][name="back"]').live('click', function(e) {
762 | // go to previous step
763 |
764 | e.preventDefault();
765 |
766 | // 1. hide current step
767 | $('#step'+currentStep)
768 | .hide()
769 |
770 | // 2. reset appropriate elements
771 | if (currentStep == 2) {
772 | $('#step'+currentStep).find('div.container').not('div.template').remove();
773 | filesData = [];
774 | $('#files').replaceWith('');
775 | }
776 |
777 | // 3. decrement step
778 | currentStep--;
779 |
780 | // 4. show previous step
781 | $('#step'+currentStep)
782 | .show();
783 | });
784 |
785 | $('[type="checkbox"][name="columnSelector"]').live('click', function() {
786 | // toggle selected column display
787 | var $input = $(this);
788 | var fileIndex = $input.parents().filter('div')[2].id.substr(4,1);
789 | var columnIndex = $input.parent()[0].cellIndex+1;
790 |
791 | columnSelection($input.parents('table'), [columnIndex]);
792 |
793 | });
794 |
795 | $('[type="checkbox"][name="delimitor"], [type="checkbox"][name="header"], [type="radio"][name="quoteMark"]').live('click', function() {
796 | // re-process the file when user clicks on a delimitor checkbox
797 |
798 | // cache file div selector
799 | var $input = $(this).parents().filter('div').eq(1);
800 |
801 | // store file index
802 | var fileIndex = $input.attr('id').substr(4,1);
803 |
804 | if (this.value=="custom" && $(this).prop("checked") == false) {
805 | // empty custom delimitor text input in case user unticks custom checkbox
806 | $input.find('.customdelimitors').val("");
807 | }
808 |
809 | if (this.name=="header") {
810 | if ($(this).prop("checked") == false && $input.find('[name="startLine"]').val() == 2)
811 | // set start row to 1 in case user unticks header box and first row is 2
812 | $input.find('[name="startLine"]').val(1);
813 | else if ($(this).prop("checked") == true && $input.find('[name="startLine"]').val() == 1)
814 | // set start row to 2 in case user ticks header box and first row is 1
815 | $input.find('[name="startLine"]').val(2);
816 | }
817 |
818 | //retrieve selected options for the file
819 | var opts = getFileOptions($input);
820 |
821 | // refresh file display
822 | processCsv($input, opts);
823 |
824 | });
825 |
826 | $('.customdelimitors').live('blur',function() {
827 | // check/uncheck custom box and trigger preview refresh when custom delimitor has been typed
828 |
829 | // cache file div selector
830 | var $input = $(this).parents().filter('div').eq(1);
831 |
832 | var opts = getFileOptions($input);
833 |
834 | // store file index
835 | // var fileIndex = $input.attr('id').substr(4,1);
836 |
837 | processCsv($input, opts);
838 |
839 | $(this).parent().find('input[value="custom"]')
840 | .prop("checked",this.value==""?false:true);
841 | });
842 |
843 | $('[type="radio"][name="outputFormat"]').live('change', function() {
844 | // update custom id generation value when user changes the output format option
845 | if (this.value == "byRow") {
846 | // custom id for one document by row
847 | $(this).parent().find('#ids option[value="custom"]').html('file name and row number');
848 | }
849 | else {
850 | // custom id for one document by file
851 | $(this).parent().find('#ids option[value="custom"]').html('file name');
852 | }
853 | });
854 |
855 | $('[type="submit"][name="logIn"]').live('click', function(e) {
856 | // sign user in
857 | e.preventDefault();
858 |
859 | var username = $('input[name="username"]', '#session').val();
860 | var password = $('input[name="password"]', '#session').val();
861 |
862 | // login the user
863 | $.couch.login({
864 | 'name': username,
865 | 'password': password,
866 | success: function(data) {
867 | /*
868 | // replace login form by user name
869 | $('#session').html('logged in as '+data.name+ ' ');
870 | */
871 | // workaround to avoid some bug in couchdb1.0.2: username is not always returned in the callback function.
872 | $.couch.session({
873 | success: function(data) {
874 | if (data.userCtx.name != null)
875 | // replace login form by user name
876 | $('#session').html('logged in as '+data.userCtx.name+ ' ');
877 | }
878 | });
879 | }
880 | });
881 |
882 | });
883 |
884 | $('[type="submit"][name="logOut"]').live('click', function(e) {
885 | // sign user in
886 | e.preventDefault();
887 |
888 | // log the user out
889 | $.couch.logout();
890 |
891 | // restore login form
892 | $('#session').html('');
893 | });
894 |
895 | $('[type="submit"][name="overwrite"]').live('click', function(e) {
896 | // overwrite selected document
897 | e.preventDefault();
898 |
899 | // cache the overwrite button selector
900 | $input = $(this);
901 |
902 | // disable overwrite button
903 | $input.prop('disabled',true);
904 |
905 | // display a spinner during the file update
906 | $('').insertAfter($input);
907 |
908 | doc = $input.data('doc');
909 | doc._rev = $input.data('rev');
910 | formId = $input.data('formId');
911 |
912 | $db.saveDoc(
913 | doc,{
914 | success: function(data) {
915 | // confirm that insertion was succcessfully done
916 | $('#'+formId).replaceWith('