├── .gitignore ├── .travis.yml ├── README.md ├── ladon.js ├── package.json └── test.js /.gitignore: -------------------------------------------------------------------------------- 1 | npm-debug.log 2 | node_modules 3 | -------------------------------------------------------------------------------- /.travis.yml: -------------------------------------------------------------------------------- 1 | language: node_js 2 | node_js: 3 | - '0.10' 4 | - '0.11' 5 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | Ladon 2 | ===== 3 | 4 | [![Dependency Status](http://img.shields.io/david/danielgtaylor/ladon.svg?style=flat)](https://david-dm.org/danielgtaylor/ladon) [![Build Status](http://img.shields.io/travis/danielgtaylor/ladon.svg?style=flat)](https://travis-ci.org/danielgtaylor/ladon) [![NPM version](http://img.shields.io/npm/v/ladon.svg?style=flat)](https://www.npmjs.org/package/ladon) [![NPM license](http://img.shields.io/npm/l/ladon.svg?style=flat)](https://www.npmjs.org/package/ladon) 5 | 6 | A small, simple utility to process many files in parallel. It is meant for people comfortable with using a terminal but strives to be as easy to use as humanly possible. 7 | 8 | Ladon is named after the multiheaded serpent dragon from Greek mythology, slain by Heracles and thrust into the sky as the constellation Draco. His many heads allow you to efficiently work on many files at once. 9 | 10 | Features 11 | -------- 12 | * Supports Windows, Mac OS X, Linux, etc 13 | * Select files via a simple [glob](https://www.npmjs.org/package/glob) 14 | * Autodetects system CPU count 15 | * Configurable parallel process count 16 | * Simple command template syntax 17 | * Built-in templated `mkdir -p` support 18 | 19 | Installation 20 | ------------ 21 | The only dependency is [Node.js](http://nodejs.org/). Standard global installation via NPM applies: 22 | 23 | ```bash 24 | sudo npm install -g ladon 25 | ``` 26 | 27 | Examples 28 | -------- 29 | The following are some examples of what is possible: 30 | 31 | ```bash 32 | # Print all text file names relative to the current directory 33 | ladon "**/*.txt" -- echo RELPATH 34 | 35 | # Calculate SHA1 sums of all your PDFs and save them in a file 36 | ladon "~/Documents/**/*.pdf" -- shasum FULLPATH >hashes.txt 37 | 38 | # Generate thumbnails with ImageMagick and keep directory structure 39 | ladon -m thumbs/RELDIR "**/*.jpg" -- convert FULLPATH -thumbnail 100x100^ -gravity center -extent 100x100 thumbs/RELPATH 40 | ``` 41 | 42 | You can also replace common `bash`-isms with true parallel processing: 43 | 44 | ```bash 45 | # Typical bash for loop 46 | for f in ~/Music/*.wav; do lame -V 2 $f ${f%.*}.mp3; done 47 | 48 | # Parallelized via ladon to use all CPUs 49 | ladon "~/Music/*.wav" -- lame -V 2 FULLPATH DIRNAME/BASENAME.mp3 50 | ``` 51 | 52 | Tutorial 53 | -------- 54 | The following is a brief walkthrough of how to use `ladon`. 55 | 56 | ### Command Structure 57 | The basic command structure consists of two parts that are split by `--`. The first part consists of the `ladon` command and its options, while the second consists of the command you want to run in parallel. The only required `ladon` option is a file selector glob. 58 | 59 | ```bash 60 | ladon [options] glob -- command 61 | ``` 62 | 63 | Ladon works by selecting files via a glob, which supports wildcards like `*` and `**` to match any file or any directory recursively. For example, `*.txt` would select all the text files in the current directory, while `**/*.txt` would select all the text files in the current directory __and__ any child directories. [Read the full glob syntax](https://www.npmjs.org/package/glob). 64 | 65 | The second half of the command structure is the command you wish to run in parallel over the selected files. It can be anything you want and can use special variables (documented below). 66 | 67 | ### Using Variables 68 | A full variable reference can be found below. The most common use case is to get the full path to a file that you wish to process, and this can be done via the `FULLPATH` variable. For example, to print out the full path to each file that is selected by your glob: 69 | 70 | ```bash 71 | ladon "*" -- echo FULLPATH 72 | ``` 73 | 74 | You can also safely mix variables with normal text to construct new paths: 75 | 76 | ```bash 77 | ladon "*" -- echo RELDIR/BASENAME.zip 78 | ``` 79 | 80 | ### Making Directories 81 | Many commands will generate a new file. Sometimes you want to overwrite existing files, but other times you'd rather create a copy. Ladon has built-in support for a templated `mkdir -p` feature which will recursively ensure directories exist before running your command on a selected file. The `RELDIR` variable is very useful here. This is perhaps best illustrated via example: 82 | 83 | ```bash 84 | # Recursively copy all files and keep directory structure 85 | ladon -m foo/RELDIR "myfiles/**/*" -- cp FULLPATH foo/RELPATH 86 | ``` 87 | 88 | In the example above every file and directory in the `myfiles` directory will be copied over to the new directory `foo`. If there is a `myfiles/docs/test.txt` then there will be a `foo/docs/test.txt` file created. 89 | 90 | Variable Reference 91 | ------------------ 92 | The following variables can be used in both the command and the directory name when using the `--makedirs` option. The examples below assume that the _current working directory_ is `/home/dan/`. 93 | 94 | | __Variable__ | __Description__ | __Example__ | 95 | | ------------ | --------------------------------------------- | ------------------------- | 96 | | FULLPATH | Full path, equivalent to DIRNAME/BASENAME.EXT | `/home/dan/books/foo.txt` | 97 | | DIRNAME | Directory name | `/home/dan/books` | 98 | | BASENAME | File name without extension | `foo` | 99 | | EXT | File name extension | `txt` | 100 | | RELDIR | Relative directory name | `books` | 101 | | RELPATH | Relative file path | `books/foo.txt` | 102 | 103 | Use as a Library 104 | ---------------- 105 | You can also use ladon as a basic library in Node.js. 106 | 107 | ```bash 108 | npm install ladon 109 | ``` 110 | 111 | Then, just `require` and use it: 112 | 113 | ```javascript 114 | var ladon = require('ladon'); 115 | 116 | // The command parser (based on yargs) 117 | var args = ladon.parser.parse(['ladon', '**/*.txt', '--', 'echo', 'FULLPATH']); 118 | 119 | // Run the command 120 | ladon.run(args, function (err) { 121 | if (err) console.error(err.toString()); 122 | }); 123 | ``` 124 | 125 | Alternatives 126 | ------------ 127 | 128 | * [xargs](http://offbytwo.com/2011/06/26/things-you-didnt-know-about-xargs.html) 129 | * [GNU Parallel](http://www.gnu.org/software/parallel/) 130 | 131 | License 132 | ------- 133 | See http://dgt.mit-license.org/ 134 | -------------------------------------------------------------------------------- /ladon.js: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env node 2 | var argv, async, cp, glob, _help, mkdirp, os, parser, path, quote, render, run; 3 | 4 | async = require('async'); 5 | cp = require('child_process'); 6 | glob = require('glob'); 7 | mkdirp = require('mkdirp'); 8 | os = require('os'); 9 | path = require('path'); 10 | 11 | // Surround a string with quotes 12 | quote = exports.quote = function (str) { 13 | return '"' + str + '"'; 14 | }; 15 | 16 | // Render a path template with the given filename 17 | render = exports.render = function (filename, relStart, doQuote, template) { 18 | var ext = filename.split('.').pop(); 19 | var q = doQuote ? quote : function (s) { return s; }; 20 | 21 | return template 22 | .replace(new RegExp('FULLPATH', 'g'), q(filename)) 23 | .replace(new RegExp('DIRNAME', 'g'), q(path.dirname(filename))) 24 | .replace(new RegExp('BASENAME', 'g'), q(path.basename(filename, '.' + ext))) 25 | .replace(new RegExp('EXT', 'g'), q(ext)) 26 | .replace(new RegExp('RELDIR', 'g'), q(path.dirname(filename).substr(relStart))) 27 | .replace(new RegExp('RELPATH', 'g'), q(filename.substr(relStart))); 28 | }; 29 | 30 | // Setup argument parser and options 31 | parser = exports.parser = require('yargs') 32 | .usage('$0 ' + require('./package.json').version + 33 | ' via nodejs-' + process.versions.node + '\n' + 34 | 'Usage: $0 [options] glob -- command [args]') 35 | .example('$0 "**/*.txt" -- echo RELPATH', 'List all text files') 36 | .example('$0 "**/*.txt" -- cat FULLPATH >combined.txt', 'Combine all text files') 37 | .example('', '') 38 | .example('https://github.com/danielgtaylor/ladon#readme', 'More examples') 39 | .options('f', { 40 | alias: 'fail', 41 | describe: 'Fail on first error', 42 | boolean: true 43 | }) 44 | .options('m', { 45 | alias: 'makedirs', 46 | describe: 'Make directories (supports variables)', 47 | string: true 48 | }) 49 | .options('p', { 50 | alias: 'processes', 51 | describe: 'Maximum number of processes', 52 | 'default': os.cpus().length 53 | }) 54 | .options('v', { 55 | alias: 'verbose', 56 | describe: 'Verbose output to sdterr', 57 | boolean: true, 58 | 'default': false 59 | }); 60 | 61 | // Print extra help information 62 | _help = parser.help; 63 | parser.help = function () { 64 | var helpStr = _help(); 65 | var variables = [ 66 | ['FULLPATH', 'Full path, equivalent to DIRNAME' + path.sep + 'BASENAME.EXT'], 67 | ['DIRNAME', 'Directory name'], 68 | ['BASENAME', 'File name without extension'], 69 | ['EXT', 'File name extension'], 70 | ['RELDIR', 'Relative directory name'], 71 | ['RELPATH', 'Relative file path'] 72 | ]; 73 | 74 | helpStr += '\n\nVariables:\n'; 75 | helpStr += variables.map(function (x) { 76 | var str = ' ' + x[0]; 77 | 78 | for (var i = x[0].length; i < 10; i++) { 79 | str += ' '; 80 | } 81 | 82 | return str + x[1]; 83 | }).join('\n'); 84 | 85 | return helpStr; 86 | }; 87 | 88 | run = exports.run = function (argv, done) { 89 | if (argv instanceof Function) { 90 | done = argv; 91 | argv = undefined; 92 | } 93 | 94 | if (argv === undefined) argv = parser.argv; 95 | if (done === undefined) done = function () {}; 96 | 97 | if (argv._.length < 2) { 98 | parser.showHelp(); 99 | return done(new Error('Must pass at least a glob and command!')); 100 | } 101 | 102 | // Handle basic home directory expansion 103 | if (argv._[0][0] == '~') { 104 | argv._[0] = (process.env.HOME || process.env.USERPROFILE) + argv._[0].substr(1); 105 | } 106 | 107 | // Resolve full path to glob 108 | argv._[0] = path.resolve(argv._[0]); 109 | 110 | // Get relative start position for paths for RELDIR and RELNAME 111 | var relativeStart = argv._[0].indexOf('**'); 112 | if (relativeStart == -1) { 113 | relativeStart = process.cwd().length + 1; 114 | } 115 | 116 | // Find and process files 117 | new glob.Glob(argv._[0], { 118 | nocase: true 119 | }, function(err, filenames) { 120 | var _process; 121 | 122 | if (err) return done(err); 123 | 124 | if (argv.verbose) 125 | console.error('Processing ' + filenames.length + ' files...'); 126 | 127 | _process = function(filename, processDone) { 128 | var cmd = render(filename, relativeStart, true, argv._.slice(1).join(' ')); 129 | 130 | if (argv.makedirs) { 131 | // Ensure directories exist before running commands! 132 | mkdirp.sync(render(filename, relativeStart, false, argv.makedirs)); 133 | } 134 | 135 | if (argv.verbose) 136 | console.error("Processing " + filename + '\n' + cmd); 137 | 138 | // Run the command and dump the output 139 | cp.exec(cmd, function(err, stdout, stderr) { 140 | if (err) { 141 | if (argv.fail) { 142 | return processDone(err); 143 | } else { 144 | console.error(err); 145 | } 146 | } 147 | 148 | if (stdout) process.stdout.write(stdout); 149 | if (stderr) process.stderr.write(stderr); 150 | 151 | processDone(); 152 | }); 153 | }; 154 | 155 | async.eachLimit(filenames, argv.processes, _process, function(err) { 156 | if (err) return done(err); 157 | done(); 158 | }); 159 | }); 160 | }; 161 | 162 | if (require.main === module) { 163 | run(function (err) { 164 | if (err) console.error(err.toString()); 165 | }); 166 | } 167 | -------------------------------------------------------------------------------- /package.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "ladon", 3 | "version": "1.0.6", 4 | "description": "Run commands in parallel on many files, like GNU parallel", 5 | "main": "ladon.js", 6 | "bin": { 7 | "ladon": "ladon.js" 8 | }, 9 | "scripts": { 10 | "test": "node_modules/mocha/bin/_mocha -R spec test.js", 11 | "prepublish": "node_modules/mocha/bin/_mocha -R spec test.js" 12 | }, 13 | "author": "Daniel G. Taylor", 14 | "license": "MIT", 15 | "dependencies": { 16 | "async": "~0.9.0", 17 | "glob": "~4.0.5", 18 | "mkdirp": "^0.5.0", 19 | "yargs": "^1.2.1" 20 | }, 21 | "devDependencies": { 22 | "mocha": "^1.18.2" 23 | }, 24 | "repository": { 25 | "type": "git", 26 | "url": "https://github.com/danielgtaylor/ladon.git" 27 | }, 28 | "keywords": [ 29 | "parallel", 30 | "process" 31 | ], 32 | "bugs": { 33 | "url": "https://github.com/danielgtaylor/ladon/issues" 34 | }, 35 | "homepage": "https://github.com/danielgtaylor/ladon" 36 | } 37 | -------------------------------------------------------------------------------- /test.js: -------------------------------------------------------------------------------- 1 | var assert, ladon; 2 | 3 | assert = require('assert'); 4 | ladon = require('./ladon'); 5 | 6 | describe ('ladon', function () { 7 | it ('Should provide a parser', function () { 8 | assert(ladon.parser); 9 | }); 10 | 11 | it ('Should quote a string', function () { 12 | assert.equal('"test"', ladon.quote('test')); 13 | }); 14 | 15 | it ('Should render a template with variables', function () { 16 | var filename, rendered, template; 17 | 18 | filename = '/home/dan/books/test.txt'; 19 | template = 'FULLPATH'; 20 | rendered = ladon.render(filename, 10, false, template); 21 | 22 | assert.equal(filename, rendered); 23 | 24 | template = 'RELDIR/BASENAME.EXT'; 25 | rendered = ladon.render(filename, 10, false, template); 26 | 27 | assert.equal('books/test.txt', rendered); 28 | 29 | template = 'RELPATH RELPATH'; 30 | rendered = ladon.render(filename, 10, true, template); 31 | 32 | assert.equal('"books/test.txt" "books/test.txt"', rendered); 33 | }); 34 | 35 | it ('Should return a help string', function () { 36 | assert(ladon.parser.help()); 37 | }); 38 | 39 | it ('Should require at least a glob and command', function (done) { 40 | var showHelp = ladon.parser.showHelp; 41 | ladon.parser.showHelp = function () {}; 42 | ladon.run({'_':[]}, function (err) { 43 | ladon.parser.showHelp = showHelp; 44 | 45 | assert(err); 46 | 47 | done(); 48 | }); 49 | }); 50 | 51 | it ('Should run commands', function (done) { 52 | var argv, child_process, glob, tmp = {}; 53 | 54 | argv = { 55 | processes: 2, 56 | _: ['*.js', 'echo', 'FULLPATH'] 57 | }; 58 | 59 | // Mock globbing 60 | glob = require('glob'); 61 | tmp.Glob = glob.Glob; 62 | glob.Glob = function (globString, options, globDone) { 63 | tmp.GlobCalled = true; 64 | globDone(null, ['test.js', 'test2.js']); 65 | return {}; 66 | }; 67 | 68 | // Mock exec 69 | child_process = require('child_process'); 70 | tmp.exec = child_process.exec; 71 | child_process.exec = function (cmd, execDone) { 72 | tmp.execCalled = true; 73 | execDone(null, '', ''); 74 | }; 75 | 76 | ladon.run(argv, function (err) { 77 | glob.Glob = tmp.Glob; 78 | child_process.exec = tmp.exec; 79 | 80 | if (err) return done(err); 81 | 82 | assert(tmp.GlobCalled); 83 | assert(tmp.execCalled); 84 | 85 | done(); 86 | }); 87 | }); 88 | }); 89 | --------------------------------------------------------------------------------