├── .gitignore ├── COPYING ├── Makefile ├── NEWS.md ├── README.md ├── _nq ├── nq.1 ├── nq.c ├── nq.sh ├── nqtail.1 ├── nqtail.c ├── nqtail.sh ├── nqterm ├── nqterm.1 └── tests /.gitignore: -------------------------------------------------------------------------------- 1 | *~ 2 | nq 3 | nqtail 4 | ,*.* 5 | -------------------------------------------------------------------------------- /COPYING: -------------------------------------------------------------------------------- 1 | nq is in the public domain. 2 | 3 | To the extent possible under law, Leah Neukirchen 4 | has waived all copyright and related or neighboring rights to this work. 5 | 6 | http://creativecommons.org/publicdomain/zero/1.0/ 7 | -------------------------------------------------------------------------------- /Makefile: -------------------------------------------------------------------------------- 1 | ALL=nq nqtail nqterm 2 | 3 | CFLAGS=-g -Wall -O2 4 | 5 | DESTDIR= 6 | PREFIX=/usr/local 7 | BINDIR=$(PREFIX)/bin 8 | MANDIR=$(PREFIX)/share/man 9 | 10 | INSTALL=install 11 | 12 | all: $(ALL) 13 | 14 | clean: FRC 15 | rm -f nq nqtail 16 | 17 | check: FRC all 18 | prove -v ./tests 19 | 20 | install: FRC all 21 | mkdir -p $(DESTDIR)$(BINDIR) $(DESTDIR)$(MANDIR)/man1 22 | $(INSTALL) -m0755 $(ALL) $(DESTDIR)$(BINDIR) 23 | $(INSTALL) -m0644 $(ALL:=.1) $(DESTDIR)$(MANDIR)/man1 24 | 25 | FRC: 26 | -------------------------------------------------------------------------------- /NEWS.md: -------------------------------------------------------------------------------- 1 | ## 1.0 (2024-07-03) 2 | 3 | * **Incompatible change:** The fq utility has been renamed to nqtail. 4 | * **Incompatible change:** The tq utility has been renamed to nqterm. 5 | * nq: add support for a $NQFAILDIR 6 | 7 | ## 0.5 (2022-03-26) 8 | 9 | * **Notable change:** nq now creates files with permissions 0666 and 10 | subject to your umask (like most programs that create new files). 11 | If your queue needs to remain secret, prohibit access to the whole 12 | directory. 13 | * Support for nq in a multi-user environment: having read permission 14 | for queued tasks in the directory is enough to wait for them. 15 | * Makefile: support INSTALL variable. 16 | * Bugfix: create $NQDONEDIR properly 17 | 18 | ## 0.4 (2021-03-15) 19 | 20 | * nq: now scales a lot better 21 | * nq: set $NQDONEDIR to move finished jobs there 22 | * fq: add kevent/kqueue support 23 | * Bugfixes 24 | 25 | ## 0.3.1 (2018-03-07) 26 | 27 | * Fix build on FreeBSD, OpenBSD and macOS. 28 | 29 | ## 0.3 (2018-03-06) 30 | 31 | * nq: add `-c` to clean job file when the process succeeded. 32 | * nq: avoid unnecessary quoting for the exec line. 33 | * Bugfix when `-q` was used with empty command lines. 34 | 35 | ## 0.2.2 (2017-12-21) 36 | 37 | * fq: fix when `$NQDIR` is set and inotify is used. (Thanks to Sebastian Reuße) 38 | * Support for NetBSD 7. 39 | 40 | ## 0.2.1 (2017-04-27) 41 | 42 | * fq: `-q` erroneously was on by default. 43 | 44 | ## 0.2 (2017-04-26) 45 | 46 | * fq: add `-n` to not wait 47 | * Support for platforms without O_DIRECTORY. 48 | * Support for SmartOS. 49 | 50 | ## 0.1 (2015-08-28) 51 | 52 | * Initial release 53 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | ## nq: queue utilities 2 | 3 | These small utilities allow creating very lightweight job queue 4 | systems which require no setup, maintenance, supervision, or any 5 | long-running processes. 6 | 7 | `nq` should run on any POSIX.1-2008 compliant system which also 8 | provides a working flock(2). Tested on Linux 2.6.37, Linux 4.1, 9 | OpenBSD 5.7, FreeBSD 10.1, NetBSD 7.0.2, Mac OS X 10.3 and 10 | SmartOS joyent_20160304T005100Z. 11 | 12 | The intended purpose is ad-hoc queuing of command lines (e.g., for 13 | building several targets of a Makefile, downloading multiple files one 14 | at a time, running benchmarks in several configurations, or simply as 15 | a glorified `nohup`). But as any good Unix tool, it can be abused for 16 | whatever you like. 17 | 18 | Job order is enforced by a timestamp `nq` gets immediately when 19 | started. Synchronization happens on file-system level. Timer 20 | resolution is milliseconds. No sub-second file system time stamps are 21 | required. Polling is not used. Exclusive execution is maintained 22 | strictly. 23 | 24 | Enforcing job order works like this: 25 | - every job has a flock(2)ed output file, ala `,TIMESTAMP.PID` 26 | - every job starts only after all earlier flock(2)ed files are unlocked 27 | - Why flock(2)? Because it locks the file handle, which is shared 28 | across exec(2) with the child process (the actual job), and it will 29 | unlock when the file is closed (usually when the job terminates). 30 | 31 | You enqueue (get it?) new jobs using `nq CMDLINE...`. The job ID is 32 | output (unless suppressed using `-q`) and `nq` detaches immediately, 33 | running the job in the background. STDOUT and STDERR are redirected 34 | into the log file. 35 | 36 | `nq` tries hard (but does not guarantee) to ensure the log file of the 37 | currently running job has `+x` bit set. Thus you can use `ls -F` to get 38 | a quick overview of the state of your queue. 39 | 40 | The "file extension" of the log file is actually the PID, so you can 41 | kill jobs easily. Before the job is started, it is the PID of `nq`, 42 | so you can cancel a queued job by killing it as well. 43 | 44 | Due to the initial `exec` line in the log files, you can resubmit a 45 | job by executing it as a shell command file (i.e. running `sh $jobid`). 46 | 47 | You can wait for jobs to finish using `nq -w`, possibly listing job 48 | IDs you want to wait for; the default is all of them. Likewise, you 49 | can test if there are jobs which need to be waited upon using `-t`. 50 | 51 | By default, job IDs are per-directory, but you can set `$NQDIR` to put 52 | them elsewhere. Creating `nq` wrappers setting `$NQDIR` to provide 53 | different queues for different purposes is encouraged. 54 | 55 | All these operations take worst-case quadratic time in the amount of 56 | lock files produced, so you should clean them regularly. 57 | 58 | ## Examples 59 | 60 | Build targets `clean`, `depends`, `all`, without occupying the terminal: 61 | 62 | % nq make clean 63 | % nq make depends 64 | % nq make all 65 | % nqtail 66 | ... look at output, can interrupt with C-c any time 67 | without stopping the build ... 68 | 69 | Simple download queue, accessible from multiple terminals: 70 | 71 | % mkdir -p /tmp/downloads 72 | % alias qget='NQDIR=/tmp/downloads nq wget' 73 | % alias qwait='NQDIR=/tmp/downloads nqtail -q' 74 | window1% qget http://mymirror/big1.iso 75 | window2% qget http://mymirror/big2.iso 76 | window3% qget http://mymirror/big3.iso 77 | % qwait 78 | ... wait for all downloads to finish ... 79 | 80 | As `nohup` replacement (The benchmark will run in background, every run 81 | gets a different output file, and the command line you ran is logged, 82 | too!): 83 | 84 | % ssh remote 85 | remote% nq ./run-benchmark 86 | ,14f6f3034f8.17035 87 | remote% ^D 88 | % ssh remote 89 | remote% nqtail 90 | ... see output, nqtail exits when job finished ... 91 | 92 | ## Assumptions 93 | 94 | `nq` will only work correctly when: 95 | - `$NQDIR` (respectively `.`) is writable. 96 | - `flock(2)` works in `$NQDIR` (respectively `.`). 97 | - `gettimeofday` behaves monotonic (using `CLOCK_MONOTONIC` would 98 | create confusing file names). Else job order can be confused and 99 | multiple tasks can run at once due to race conditions. 100 | - No other programs put files matching `,*` into `$NQDIR` (respectively `.`). 101 | 102 | ## nq helpers 103 | 104 | Two helper programs are provided: 105 | 106 | **`nqtail`** outputs the log of the currently running jobs, exiting 107 | when the jobs are done. If no job is running, the output of the last 108 | job is shown. `nqtail -a` shows the output of all jobs, `nqtail -q` 109 | only shows one line per job. `nqtail` uses `inotify` on Linux and 110 | falls back to polling for size change else. (`nqtail.sh` is a similar 111 | tool, not quite as robust, implemented as shell-script calling 112 | `tail`.) 113 | 114 | **`nqterm`** wraps `nq` and displays the `nqtail` output in a new 115 | `tmux` or screen window. 116 | 117 | (A pure shell implementation of `nq` is provided as `nq.sh`. It needs 118 | `flock` from util-linux, and only has a timer resolution of 1s. 119 | Lock files from `nq` and `nq.sh` should not be mixed.) 120 | 121 | ## Installation 122 | 123 | Use `make all` to build, `make install` to install relative to `PREFIX` 124 | (`/usr/local` by default). The `DESTDIR` convention is respected. 125 | You can also just copy the binaries into your `PATH`. 126 | 127 | You can use `make check` to run a simple test suite, if you have 128 | Perl's `prove` installed. 129 | 130 | ## Comparison to `at`, `batch`, and `task-spooler` 131 | 132 | * `at` runs jobs at a given time. 133 | `batch` runs jobs "when system load levels permit". 134 | `nq` and [`task-spooler`](https://vicerveza.homeunix.net/~viric/soft/ts/) 135 | run jobs in sequence with no regard to the system's load average. 136 | 137 | * `at` and `batch` have 52 built-in queues: a-z and A-Z. 138 | Any directory can be a queue for `nq`. 139 | `task-spooler` can have different queues for different terminals. 140 | 141 | * You can follow the output of an `nq` queue tail-style with `nqtail`. 142 | 143 | * The syntax is different: `at` and `batch` take whole scripts from 144 | the standard input or a file; `nq` takes a single command as its 145 | command line arguments. 146 | 147 | * `nq` doesn't rely on a daemon, and uses a directory to manage the queue. 148 | `task-spooler` automatically launches a daemon to manage a queue. 149 | 150 | * `task-spooler` can set a maximum number of simultaneous jobs. 151 | 152 | ## Copyright 153 | 154 | `nq` is in the public domain. 155 | 156 | To the extent possible under law, 157 | Leah Neukirchen 158 | has waived all copyright and related or 159 | neighboring rights to this work. 160 | 161 | http://creativecommons.org/publicdomain/zero/1.0/ 162 | -------------------------------------------------------------------------------- /_nq: -------------------------------------------------------------------------------- 1 | #compdef nq nqtail nqterm 2 | 3 | _nq_job() { 4 | compadd "$@" -- ${NQDIR:-.}/,*.*(:t) 5 | } 6 | 7 | _nq() { 8 | case "$service" in 9 | nqtail) _arguments -s -A : \ 10 | '-q[show only one line per job]' \ 11 | '-a[output for all jobs]' \ 12 | '*::job:_nq_job' 13 | ;; 14 | nq) _arguments -A : \ 15 | '-w[wait for jobs]:*:job:_nq_job' \ 16 | '-t[check jobs]:*:job:_nq_job' \ 17 | '(-):command name: _command_names -e' \ 18 | '*::arguments:_normal' 19 | ;; 20 | nqterm) _arguments : \ 21 | '(-):command name: _command_names -e' \ 22 | '*::arguments:_normal' 23 | ;; 24 | esac 25 | } 26 | 27 | _nq "$@" 28 | -------------------------------------------------------------------------------- /nq.1: -------------------------------------------------------------------------------- 1 | .Dd July 3, 2024 2 | .Dt NQ 1 3 | .Os 4 | .Sh NAME 5 | .Nm nq 6 | .Nd job queue utility 7 | .Sh SYNOPSIS 8 | .Nm 9 | .Op Fl c 10 | .Op Fl q 11 | .Ar command\ line ... 12 | .Nm 13 | .Fl t 14 | .Ar job\ id ... 15 | .Nm 16 | .Fl w 17 | .Ar job\ id ... 18 | .Sh DESCRIPTION 19 | The 20 | .Nm 21 | utility provides a very lightweight queuing system without 22 | requiring setup, 23 | maintenance, 24 | supervision 25 | or any long-running processes. 26 | .Pp 27 | Job order is enforced by a timestamp 28 | .Nm 29 | gets immediately when started. 30 | Synchronization happens on file-system level. 31 | Timer resolution is milliseconds. 32 | No sub-second file system time stamps are required. 33 | Polling is not used. 34 | Exclusive execution is maintained strictly. 35 | .Pp 36 | You enqueue(!) new jobs into the queue by running 37 | .Pp 38 | .Dl nq Ar command line ... 39 | .Pp 40 | The job id (a file name relative to 41 | .Ev NQDIR , 42 | which defaults to the current directory) is 43 | output (unless suppressed using 44 | .Fl q ) 45 | and 46 | .Nm 47 | detaches from the terminal immediately, 48 | running the job in the background. 49 | Standard output and standard error are redirected into the job id file. 50 | .Xr nqtail 1 51 | can be used to conveniently watch the log files. 52 | .Pp 53 | The options are as follows: 54 | .Bl -tag -width Ds 55 | .It Fl c 56 | Clean up job id file when process exited with status 0. 57 | .It Fl q 58 | Suppress output of the job id after spawning new job. 59 | .It Fl t 60 | Enter 61 | .Em test mode : 62 | exit with status 0 when 63 | .Em all 64 | of the listed job ids are already done, else with status 1. 65 | .It Fl w 66 | Enter 67 | .Em waiting mode : 68 | wait in the foreground until 69 | .Em all 70 | listed job ids are done. 71 | .El 72 | .Sh ENVIRONMENT 73 | .Bl -hang -width "NQDONEDIR" 74 | .It Ev NQDIR 75 | Directory where lock files/job output resides. 76 | Each 77 | .Ev NQDIR 78 | can be considered a separate queue. 79 | The current working directory is used when 80 | .Ev NQDIR 81 | is unset. 82 | .Ev NQDIR 83 | is created if needed. 84 | .It Ev NQDONEDIR 85 | When set, specifies a directory 86 | .Po 87 | must be on the same file system as 88 | .Ev NQDIR 89 | .Pc 90 | where lock files/job output is moved 91 | to after successful execution of the job. 92 | .Pp 93 | Ignored when 94 | .Fl c 95 | is used. 96 | .It Ev NQFAILDIR 97 | When set, specifies a directory 98 | .Po 99 | must be on the same file system as 100 | .Ev NQDIR 101 | .Pc 102 | where lock files/job output is moved 103 | to after unsuccessful execution of the job. 104 | .It Ev NQJOBID 105 | The job id of the currently running job, 106 | exposed to the job itself. 107 | .El 108 | .Sh FILES 109 | .Nm 110 | expects to control all files in 111 | .Ev NQDIR 112 | (respectively 113 | .Pa \&. ) 114 | which start with 115 | .Dq Li \&, 116 | or 117 | .Dq Li ., . 118 | These files are created according to the following scheme: 119 | .Pp 120 | .Dl ,hexadecimal-time-stamp.pid 121 | .Sh EXIT STATUS 122 | The 123 | .Nm 124 | utility exits 0 on success, and >0 if an error occurs; 125 | unless 126 | .Em test mode 127 | is used, in which case exit status 1 means there is a job running. 128 | .Pp 129 | On fatal errors, exit codes 111 and 222 are used. 130 | .Sh EXAMPLES 131 | Build 132 | .Xr make 1 133 | targets 134 | .Ic clean , 135 | .Ic depends , 136 | .Ic all , 137 | without occupying the terminal: 138 | .Bd -literal -offset indent 139 | % nq make clean 140 | % nq make depends 141 | % nq make all 142 | % nqtail 143 | \&... look at output, can interrupt with C-c any time 144 | without stopping the build ... 145 | .Ed 146 | .Pp 147 | Simple download queue, accessible from multiple terminals: 148 | .Bd -literal -offset indent 149 | % alias qget='NQDIR=/tmp/downloads nq wget' 150 | % alias qwait='NQDIR=/tmp/downloads nqtail -q' 151 | window1% qget http://mymirror/big1.iso 152 | window2% qget http://mymirror/big2.iso 153 | window3% qget http://mymirror/big3.iso 154 | % qwait 155 | \&... wait for all downloads to finish ... 156 | .Ed 157 | .Pp 158 | As 159 | .Xr nohup 1 160 | replacement 161 | (The benchmark will run in background, 162 | every run gets a different output file, 163 | and the command line you ran is logged too.): 164 | .Bd -literal -offset indent 165 | % ssh remote 166 | remote% nq ./run-benchmark 167 | ,14f6f3034f8.17035 168 | remote% ^D 169 | % ssh remote 170 | remote% nqtail 171 | \&... see output, nqtail exits when job finished ... 172 | .Ed 173 | .Sh TRICKS 174 | The "file extension" of the log file is actually the PID of the job. 175 | .Nm 176 | runs all jobs in a separate process group, 177 | so you can kill an entire job process tree at once using 178 | .Xr kill 1 179 | with a negative PID. 180 | Before the job is started, it is the PID of 181 | .Nm , 182 | so you can cancel a queued job by killing it as well. 183 | .Pp 184 | Thanks to the initial 185 | .Li exec 186 | line in the log files, you can resubmit a 187 | job by executing it as a shell command file, 188 | i.e. running 189 | .Pp 190 | .Dl sh Em job\ id 191 | .Pp 192 | Creating 193 | .Nm 194 | wrappers setting 195 | .Ev NQDIR 196 | to provide different queues for different purposes is encouraged. 197 | .Sh INTERNALS 198 | Enforcing job order works like this: 199 | .Bl -dash -compact 200 | .It 201 | every job has an 202 | output file locked using 203 | .Xr flock 2 204 | and named according to 205 | .Sx FILES . 206 | .It 207 | every job starts only after all earlier 208 | flocked files are unlocked. 209 | .It 210 | the lock is released by the kernel after the job terminates. 211 | .El 212 | .Sh ASSUMPTIONS 213 | .Nm 214 | will only work correctly when: 215 | .Bl -dash 216 | .It 217 | .Ev NQDIR 218 | (respectively 219 | .Pa \&. ) 220 | is writable. 221 | .It 222 | .Xr flock 2 223 | works correctly in 224 | .Ev NQDIR 225 | (respectively 226 | .Pa \&. ) . 227 | .It 228 | .Xr gettimeofday 2 229 | behaves monotonic (using 230 | .Dv CLOCK_MONOTONIC 231 | would create confusing file names after reboot). 232 | .It 233 | No other programs put files matching 234 | .Li ,* 235 | into 236 | .Ev NQDIR 237 | (respectively 238 | .Pa \&. ) . 239 | .El 240 | .Sh SEE ALSO 241 | .Xr nqtail 1 , 242 | .Xr nqterm 1 . 243 | .Pp 244 | Alternatives to the 245 | .Nm 246 | system include 247 | .Xr batch 1 , 248 | .Xr qsub 1 , 249 | .Xr schedule 1 , 250 | .Xr srun 1 , 251 | and 252 | .Xr ts 1 . 253 | .\" .Sh STANDARDS 254 | .\" .Sh HISTORY 255 | .Sh AUTHORS 256 | .An Leah Neukirchen Aq Mt leah@vuxu.org 257 | .Sh CAVEATS 258 | All reliable queue status information is in main memory only, 259 | which makes restarting a job queue after a reboot difficult. 260 | .Sh LICENSE 261 | .Nm 262 | is in the public domain. 263 | .Pp 264 | To the extent possible under law, 265 | the creator of this work 266 | has waived all copyright and related or 267 | neighboring rights to this work. 268 | .Pp 269 | .Lk http://creativecommons.org/publicdomain/zero/1.0/ 270 | .\" .Sh BUGS 271 | -------------------------------------------------------------------------------- /nq.c: -------------------------------------------------------------------------------- 1 | /* 2 | * nq CMD... - run CMD... in background and in order, saving output 3 | * -w ... wait for all jobs/listed jobs queued so far to finish 4 | * -t ... exit 0 if no (listed) job needs waiting 5 | * -q quiet, do not output job id 6 | * -c clean, don't keep output if job exited with status 0 7 | * 8 | * - requires POSIX.1-2008 and having flock(2) 9 | * - enforcing order works like this: 10 | * - every job has a flock(2)ed output file ala ",TIMESTAMP.PID" 11 | * - every job starts only after all earlier flock(2)ed files finished 12 | * - the lock is released when job terminates 13 | * - no sub-second file system time stamps are required, jobs are started 14 | * with millisecond precision 15 | * - we try hard to make the currently running ,* file have +x bit 16 | * - you can re-queue jobs using "sh ,jobid" 17 | * 18 | * To the extent possible under law, Leah Neukirchen 19 | * has waived all copyright and related or neighboring rights to this work. 20 | * http://creativecommons.org/publicdomain/zero/1.0/ 21 | */ 22 | 23 | /* for FreeBSD. */ 24 | #define _WITH_DPRINTF 25 | 26 | #if defined(__sun) && defined(__SVR4) && !defined(HAVE_DPRINTF) 27 | #define NEED_DPRINTF 28 | #endif 29 | 30 | #include 31 | #include 32 | #include 33 | #include 34 | #include 35 | 36 | #include 37 | #include 38 | #include 39 | #include 40 | #include 41 | #include 42 | #include 43 | #include 44 | #include 45 | #include 46 | 47 | #ifndef O_DIRECTORY 48 | #define O_DIRECTORY 0 49 | #endif 50 | 51 | #ifdef NEED_DPRINTF 52 | #include 53 | static int 54 | dprintf(int fd, const char *fmt, ...) 55 | { 56 | char buf[128]; // good enough for usage in nq 57 | va_list ap; 58 | int r; 59 | 60 | va_start(ap, fmt); 61 | r = vsnprintf(buf, sizeof buf, fmt, ap); 62 | va_end(ap); 63 | if (r >= 0 && r < sizeof buf) 64 | return write(fd, buf, r); 65 | return -1; 66 | } 67 | #endif 68 | 69 | static void 70 | swrite(int fd, char *str) 71 | { 72 | size_t l = strlen(str); 73 | 74 | if (write(fd, str, l) != l) { 75 | perror("write"); 76 | exit(222); 77 | } 78 | } 79 | 80 | static void 81 | write_execline(int fd, int argc, char *argv[]) 82 | { 83 | int i; 84 | char *s; 85 | 86 | swrite(fd, "exec"); 87 | 88 | for (i = 0; i < argc; i++) { 89 | if (!strpbrk(argv[i], 90 | "\001\002\003\004\005\006\007\010" 91 | "\011\012\013\014\015\016\017\020" 92 | "\021\022\023\024\025\026\027\030" 93 | "\031\032\033\034\035\036\037\040" 94 | "`^#*[]=|\\?${}()'\"<>&;\177")) { 95 | swrite(fd, " "); 96 | swrite(fd, argv[i]); 97 | } else { 98 | swrite(fd, " '"); 99 | for (s = argv[i]; *s; s++) { 100 | if (*s == '\'') 101 | swrite(fd, "'\\''"); 102 | else 103 | write(fd, s, 1); 104 | } 105 | swrite(fd, "'"); 106 | } 107 | } 108 | } 109 | 110 | static void 111 | setx(int fd, int executable) 112 | { 113 | struct stat st; 114 | fstat(fd, &st); 115 | if (executable) 116 | st.st_mode |= 0100; 117 | else 118 | st.st_mode &= ~0100; 119 | fchmod(fd, st.st_mode); 120 | } 121 | 122 | int 123 | main(int argc, char *argv[]) 124 | { 125 | int64_t ms; 126 | int dirfd = -1, donedirfd = -1, faildirfd = -1, lockfd = -1; 127 | int opt = 0, cflag = 0, qflag = 0, tflag = 0, wflag = 0; 128 | int pipefd[2]; 129 | char lockfile[64], newestlocked[64]; 130 | pid_t child; 131 | struct timeval started; 132 | struct dirent *ent; 133 | DIR *dir; 134 | 135 | /* timestamp is milliseconds since epoch. */ 136 | gettimeofday(&started, NULL); 137 | ms = (int64_t)started.tv_sec*1000 + started.tv_usec/1000; 138 | 139 | while ((opt = getopt(argc, argv, "+chqtw")) != -1) { 140 | switch (opt) { 141 | case 'c': 142 | cflag = 1; 143 | break; 144 | case 'w': 145 | wflag = 1; 146 | break; 147 | case 't': 148 | tflag = 1; 149 | break; 150 | case 'q': 151 | qflag = 1; 152 | break; 153 | case 'h': 154 | default: 155 | goto usage; 156 | } 157 | } 158 | 159 | if (!tflag && !wflag && argc <= optind) { 160 | usage: 161 | swrite(2, "usage: nq [-c] [-q] [-w ... | -t ... | CMD...]\n"); 162 | exit(1); 163 | } 164 | 165 | char *path = getenv("NQDIR"); 166 | if (!path) 167 | path = "."; 168 | 169 | if (mkdir(path, 0777) < 0) { 170 | if (errno != EEXIST) { 171 | perror("mkdir $NQDIR"); 172 | exit(111); 173 | } 174 | } 175 | 176 | dirfd = open(path, O_RDONLY | O_DIRECTORY); 177 | if (dirfd < 0) { 178 | perror("dir open"); 179 | exit(111); 180 | } 181 | 182 | char *donepath = getenv("NQDONEDIR"); 183 | if (donepath) { 184 | if (mkdir(donepath, 0777) < 0) { 185 | if (errno != EEXIST) { 186 | perror("mkdir $NQDONEDIR"); 187 | exit(111); 188 | } 189 | } 190 | 191 | donedirfd = open(donepath, O_RDONLY | O_DIRECTORY); 192 | if (donedirfd < 0) { 193 | perror("dir open"); 194 | exit(111); 195 | } 196 | } 197 | 198 | char *failpath = getenv("NQFAILDIR"); 199 | if (failpath) { 200 | if (mkdir(failpath, 0777) < 0) { 201 | if (errno != EEXIST) { 202 | perror("mkdir $NQFAILDIR"); 203 | exit(111); 204 | } 205 | } 206 | 207 | faildirfd = open(failpath, O_RDONLY | O_DIRECTORY); 208 | if (faildirfd < 0) { 209 | perror("dir open"); 210 | exit(111); 211 | } 212 | } 213 | 214 | if (tflag || wflag) { 215 | snprintf(lockfile, sizeof lockfile, 216 | ".,%011" PRIx64 ".%d", ms, getpid()); 217 | goto wait; 218 | } 219 | 220 | if (pipe(pipefd) < 0) { 221 | perror("pipe"); 222 | exit(111); 223 | }; 224 | 225 | /* first fork, parent exits to run in background. */ 226 | child = fork(); 227 | if (child == -1) { 228 | perror("fork"); 229 | exit(111); 230 | } 231 | else if (child > 0) { 232 | char c; 233 | 234 | /* wait until child has backgrounded. */ 235 | close(pipefd[1]); 236 | read(pipefd[0], &c, 1); 237 | 238 | exit(0); 239 | } 240 | 241 | close(pipefd[0]); 242 | 243 | /* second fork, child later execs the job, parent collects status. */ 244 | child = fork(); 245 | if (child == -1) { 246 | perror("fork"); 247 | exit(111); 248 | } 249 | else if (child > 0) { 250 | int status; 251 | 252 | /* output expected lockfile name. */ 253 | snprintf(lockfile, sizeof lockfile, 254 | ",%011" PRIx64 ".%d", ms, child); 255 | if (!qflag) 256 | dprintf(1, "%s\n", lockfile); 257 | close(0); 258 | close(1); 259 | close(2); 260 | 261 | /* signal parent to exit. */ 262 | close(pipefd[1]); 263 | 264 | wait(&status); 265 | 266 | lockfd = openat(dirfd, lockfile, O_RDWR | O_APPEND); 267 | if (lockfd < 0) { 268 | perror("open"); 269 | exit(222); 270 | } 271 | 272 | setx(lockfd, 0); 273 | if (WIFEXITED(status)) { 274 | dprintf(lockfd, "\n[exited with status %d.]\n", 275 | WEXITSTATUS(status)); 276 | if (WEXITSTATUS(status) == 0) { 277 | if (cflag) 278 | unlinkat(dirfd, lockfile, 0); 279 | else if (donepath) 280 | renameat(dirfd, lockfile, 281 | donedirfd, lockfile); 282 | } 283 | if (WEXITSTATUS(status) != 0) { 284 | if (failpath) 285 | renameat(dirfd, lockfile, faildirfd, lockfile); 286 | /* errors above are ignored */ 287 | } 288 | } else { 289 | dprintf(lockfd, "\n[killed by signal %d.]\n", 290 | WTERMSIG(status)); 291 | } 292 | 293 | exit(0); 294 | } 295 | 296 | close(pipefd[1]); 297 | 298 | /* create and lock lockfile. since this cannot be done in one step, 299 | use a different filename first. */ 300 | snprintf(lockfile, sizeof lockfile, 301 | ".,%011" PRIx64 ".%d", ms, getpid()); 302 | lockfd = openat(dirfd, lockfile, 303 | O_CREAT | O_EXCL | O_RDWR | O_APPEND, 0666); 304 | if (lockfd < 0) { 305 | perror("open"); 306 | exit(222); 307 | } 308 | if (flock(lockfd, LOCK_EX) < 0) { 309 | perror("flock"); 310 | exit(222); 311 | } 312 | 313 | /* drop leading '.' */ 314 | renameat(dirfd, lockfile, dirfd, lockfile+1); 315 | 316 | /* block until rename is committed */ 317 | fsync(dirfd); 318 | 319 | write_execline(lockfd, argc, argv); 320 | 321 | if (dup2(lockfd, 2) < 0 || 322 | dup2(lockfd, 1) < 0) { 323 | perror("dup2"); 324 | exit(222); 325 | } 326 | 327 | wait: 328 | if ((tflag || wflag) && argc - optind > 0) { 329 | /* wait for files passed as command line arguments. */ 330 | 331 | int i; 332 | for (i = optind; i < argc; i++) { 333 | int fd; 334 | 335 | if (strchr(argv[i], '/')) 336 | fd = open(argv[i], O_RDONLY); 337 | else 338 | fd = openat(dirfd, argv[i], O_RDONLY); 339 | if (fd < 0) 340 | continue; 341 | 342 | if (flock(fd, LOCK_SH | LOCK_NB) == -1 && 343 | errno == EWOULDBLOCK) { 344 | if (tflag) 345 | exit(1); 346 | flock(fd, LOCK_SH); /* sit it out. */ 347 | } 348 | 349 | setx(fd, 0); 350 | close(fd); 351 | } 352 | } else { 353 | dir = fdopendir(dirfd); 354 | if (!dir) { 355 | perror("fdopendir"); 356 | exit(111); 357 | } 358 | 359 | again: 360 | *newestlocked = 0; 361 | 362 | while ((ent = readdir(dir))) { 363 | /* wait for all older ,* files than ours. */ 364 | 365 | if (!(ent->d_name[0] == ',' && 366 | strcmp(ent->d_name, lockfile+1) < 0 && 367 | strlen(ent->d_name) < sizeof(newestlocked))) 368 | continue; 369 | 370 | int fd = openat(dirfd, ent->d_name, O_RDONLY); 371 | if (fd < 0) 372 | continue; 373 | 374 | if (flock(fd, LOCK_SH | LOCK_NB) == -1 && 375 | errno == EWOULDBLOCK) { 376 | if (tflag) 377 | exit(1); 378 | if (strcmp(ent->d_name, newestlocked) > 0) 379 | strcpy(newestlocked, ent->d_name); 380 | } else { 381 | setx(fd, 0); 382 | } 383 | 384 | close(fd); 385 | } 386 | 387 | if (*newestlocked) { 388 | int fd = openat(dirfd, newestlocked, O_RDONLY); 389 | if (fd >= 0) { 390 | flock(fd, LOCK_SH); /* sit it out. */ 391 | close(fd); 392 | } 393 | rewinddir(dir); 394 | goto again; 395 | } 396 | 397 | closedir(dir); /* closes dirfd too. */ 398 | } 399 | 400 | if (tflag || wflag) 401 | exit(0); 402 | 403 | /* ready to run. */ 404 | 405 | swrite(lockfd, "\n\n"); 406 | setx(lockfd, 1); 407 | 408 | close(lockfd); 409 | 410 | setenv("NQJOBID", lockfile+1, 1); 411 | setsid(); 412 | execvp(argv[optind], argv+optind); 413 | 414 | perror("execvp"); 415 | return 222; 416 | } 417 | -------------------------------------------------------------------------------- /nq.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | # nq CMD... - run CMD... in background and in order, saving output to ,* files 3 | # 4 | # - needs POSIX sh + util-linux flock(1) (see nq.c for portable version) 5 | # - when run from tmux, display output in a new window (needs 6 | # GNU tail, C-c to abort the job.) 7 | # - we try hard to make the currently running ,* file have +x bit 8 | # - enforcing order works like this: 9 | # - every job has a flock(2)ed file 10 | # - every job starts only after all earlier flock(2)ed files finished 11 | # - the lock is released when job terminates 12 | # 13 | # To the extent possible under law, Leah Neukirchen 14 | # has waived all copyright and related or neighboring rights to this work. 15 | # http://creativecommons.org/publicdomain/zero/1.0/ 16 | 17 | if [ -z "$NQ" ]; then 18 | export NQ=$(date +%s) 19 | "$0" "$@" & c=$! 20 | ( 21 | # wait for job to finish 22 | flock -x .,$NQ.$c -c true 23 | flock -x ,$NQ.$c -c true 24 | chmod -x ,$NQ.$c 25 | ) & 26 | exit 27 | fi 28 | 29 | us=",$NQ.$$" 30 | 31 | exec 9>>.$us 32 | # first flock(2) the file, then make it known under the real name 33 | flock -x 9 34 | mv .$us $us 35 | 36 | printf "## nq $*" 1>&9 37 | 38 | if [ -n "$TMUX" ]; then 39 | tmux new-window -a -d -n '<' -c '#{pane_current_path}' \ 40 | "trap true INT QUIT TERM EXIT; 41 | tail -F --pid=$$ $us || kill $$; 42 | printf '\n[%d exited, ^D to exit.]\n' $$; 43 | cat >/dev/null" 44 | fi 45 | 46 | waiting=true 47 | while $waiting; do 48 | waiting=false 49 | # this must traverse in lexical (= numerical) order: 50 | # check all older locks are released 51 | for f in ,*; do 52 | # reached the current lock, good to go 53 | [ $f = $us ] && break 54 | 55 | if ! flock -x -n $f -c "chmod -x $f"; then 56 | # force retrying all locks again; 57 | # an earlier lock could just now have really appeared 58 | waiting=true 59 | flock -x $f -c true 60 | fi 61 | done 62 | done 63 | 64 | printf '\n' 1>&9 65 | 66 | chmod +x $us 67 | exec "$@" 2>&1 1>&9 68 | -------------------------------------------------------------------------------- /nqtail.1: -------------------------------------------------------------------------------- 1 | .Dd July 3, 2024 2 | .Dt NQTAIL 1 3 | .Os 4 | .Sh NAME 5 | .Nm nqtail 6 | .Nd job queue log viewer 7 | .Sh SYNOPSIS 8 | .Nm 9 | .Op Fl a 10 | .Op Fl n 11 | .Op Fl q 12 | .Op Ar job\ id ... 13 | .Sh DESCRIPTION 14 | .Nm 15 | is a simple utility for 16 | .Dq following 17 | the output of 18 | .Xr nq 1 19 | jobs. 20 | .Pp 21 | Without arguments, the output of the currently running and queued 22 | as-of-now jobs is emitted; else the presented job ids are used. 23 | .Pp 24 | .Nm 25 | automatically terminates after the corresponding jobs are done. 26 | .Pp 27 | The options are as follows: 28 | .Bl -tag -width Ds 29 | .It Fl a 30 | Output all log files, even of already finished jobs. 31 | .It Fl n 32 | Don't wait for new output. 33 | Can be used to look at enqueued commands. 34 | .It Fl q 35 | Only print the first line of each job output 36 | (i.e. the 37 | .Li exec 38 | line). 39 | .El 40 | .Sh ENVIRONMENT 41 | .Bl -hang -width Ds 42 | .It Ev NQDIR 43 | Directory where lock files/job output resides, see 44 | .Xr nq 1 . 45 | .El 46 | .Sh EXIT STATUS 47 | .Ex -std 48 | .Sh INTERNALS 49 | On Linux, 50 | .Xr inotify 7 51 | is used to monitor job output. 52 | On FreeBSD and macOS, 53 | .Xr kqueue 2 54 | is used. 55 | On other operating systems, polling is used. 56 | .Sh SEE ALSO 57 | .Xr nq 1 , 58 | .Xr nqterm 1 59 | .Sh AUTHORS 60 | .An Leah Neukirchen Aq Mt leah@vuxu.org 61 | .Sh LICENSE 62 | .Nm 63 | is in the public domain. 64 | .Pp 65 | To the extent possible under law, 66 | the creator of this work 67 | has waived all copyright and related or 68 | neighboring rights to this work. 69 | .Pp 70 | .Lk http://creativecommons.org/publicdomain/zero/1.0/ 71 | .\" .Sh BUGS 72 | -------------------------------------------------------------------------------- /nqtail.c: -------------------------------------------------------------------------------- 1 | /* 2 | * nqtail [FILES...] - follow output of nq jobs, quitting when they are done 3 | * 4 | * To the extent possible under law, Leah Neukirchen 5 | * has waived all copyright and related or neighboring rights to this work. 6 | * http://creativecommons.org/publicdomain/zero/1.0/ 7 | */ 8 | 9 | #include 10 | #include 11 | #include 12 | 13 | #include 14 | #include 15 | #include 16 | #include 17 | #include 18 | #include 19 | #include 20 | #include 21 | 22 | #define DELAY 250000 23 | 24 | #ifdef __linux__ 25 | #define USE_INOTIFY 26 | #endif 27 | 28 | #if defined(__FreeBSD__) || defined(__APPLE__) 29 | #define USE_KEVENT 30 | #endif 31 | 32 | #ifdef USE_INOTIFY 33 | #include 34 | char ibuf[8192]; 35 | #endif 36 | 37 | #ifdef USE_KEVENT 38 | #include 39 | #endif 40 | 41 | char buf[8192]; 42 | 43 | static int 44 | islocked(int fd) 45 | { 46 | if (flock(fd, LOCK_SH | LOCK_NB) == -1) { 47 | return (errno == EWOULDBLOCK); 48 | } else { 49 | flock(fd, LOCK_UN); 50 | return 0; 51 | } 52 | } 53 | 54 | static int 55 | alphabetic(const void *a, const void *b) 56 | { 57 | return strcmp(*(char **)a, *(char **)b); 58 | } 59 | 60 | int 61 | main(int argc, char *argv[]) 62 | { 63 | int i, fd, dirfd; 64 | off_t off, loff; 65 | ssize_t rd; 66 | int didsth = 0, seen_nl = 0; 67 | int opt = 0, aflag = 0, nflag = 0, qflag = 0; 68 | char *path; 69 | 70 | #ifdef USE_INOTIFY 71 | int ifd, wd; 72 | #endif 73 | #ifdef USE_KEVENT 74 | int kq, note; 75 | struct kevent kev; 76 | #endif 77 | 78 | close(0); 79 | 80 | while ((opt = getopt(argc, argv, "+anq")) != -1) { 81 | switch (opt) { 82 | case 'a': 83 | aflag = 1; 84 | break; 85 | case 'n': 86 | nflag = 1; 87 | break; 88 | case 'q': 89 | qflag = 1; 90 | break; 91 | default: 92 | fputs("usage: nqtail [-anq] [JOBID...]\n", stderr); 93 | exit(1); 94 | } 95 | } 96 | 97 | path = getenv("NQDIR"); 98 | if (!path) 99 | path = "."; 100 | 101 | #ifdef O_DIRECTORY 102 | dirfd = open(path, O_RDONLY | O_DIRECTORY); 103 | #else 104 | dirfd = open(path, O_RDONLY); 105 | #endif 106 | if (dirfd < 0) { 107 | perror("open dir"); 108 | exit(111); 109 | } 110 | 111 | if (optind == argc) { /* behave as if $NQDIR/,* was passed. */ 112 | DIR *dir; 113 | struct dirent *d; 114 | int len = 0; 115 | 116 | argc = 0; 117 | argv = 0; 118 | optind = 0; 119 | 120 | dir = fdopendir(dirfd); 121 | if (!dir) { 122 | perror("fdopendir"); 123 | exit(111); 124 | } 125 | 126 | while ((d = readdir(dir))) { 127 | if (d->d_name[0] != ',') 128 | continue; 129 | if (argc >= len) { 130 | len = 2*len + 1; 131 | argv = realloc(argv, len * sizeof (char *)); 132 | if (!argv) 133 | exit(222); 134 | } 135 | argv[argc] = strdup(d->d_name); 136 | if (!argv[argc]) 137 | exit(222); 138 | argc++; 139 | } 140 | 141 | qsort(argv, argc, sizeof (char *), alphabetic); 142 | } 143 | 144 | #ifdef USE_INOTIFY 145 | ifd = inotify_init(); 146 | if (ifd < 0) 147 | exit(111); 148 | #endif 149 | #ifdef USE_KEVENT 150 | kq = kqueue(); 151 | if (kq < 0) 152 | exit(111); 153 | #endif 154 | 155 | for (i = optind; i < argc; i++) { 156 | loff = 0; 157 | seen_nl = 0; 158 | 159 | fd = openat(dirfd, argv[i], O_RDONLY); 160 | if (fd < 0) 161 | continue; 162 | 163 | /* skip not running jobs, unless -a was passed, or we did not 164 | * output anything yet and are at the last argument. */ 165 | if (!aflag && !islocked(fd) && (didsth || i != argc - 1)) { 166 | close(fd); 167 | continue; 168 | } 169 | 170 | write(1, "==> ", 4); 171 | write(1, argv[i], strlen(argv[i])); 172 | write(1, qflag ? " " : "\n", 1); 173 | 174 | didsth = 1; 175 | 176 | #ifdef USE_INOTIFY 177 | char fullpath[PATH_MAX]; 178 | snprintf(fullpath, sizeof fullpath, "%s/%s", path, argv[i]); 179 | wd = inotify_add_watch(ifd, fullpath, IN_MODIFY | IN_CLOSE_WRITE); 180 | if (wd == -1) { 181 | perror("inotify_add_watch"); 182 | exit(111); 183 | } 184 | #endif 185 | #ifdef USE_KEVENT 186 | note = NOTE_WRITE; 187 | #ifdef __APPLE__ 188 | note |= NOTE_FUNLOCK; 189 | #endif 190 | #ifdef __FreeBSD__ 191 | note |= NOTE_CLOSE_WRITE; 192 | #endif 193 | EV_SET(&kev, fd, EVFILT_VNODE, EV_ADD | EV_CLEAR, note, 0, NULL); 194 | if (kevent(kq, &kev, 1, NULL, 0, NULL) < 0) { 195 | perror("kevent"); 196 | exit(111); 197 | } 198 | #endif 199 | 200 | while (1) { 201 | off = lseek(fd, 0, SEEK_END); 202 | 203 | if (off < loff) 204 | loff = off; /* file truncated */ 205 | 206 | if (off == loff) { 207 | if (nflag && islocked(fd)) 208 | break; 209 | 210 | if (flock(fd, LOCK_SH | LOCK_NB) == -1 && 211 | errno == EWOULDBLOCK) { 212 | #if defined(USE_INOTIFY) 213 | /* any inotify event is good */ 214 | read(ifd, ibuf, sizeof ibuf); 215 | #elif defined(USE_KEVENT) 216 | kevent(kq, NULL, 0, &kev, 1, NULL); 217 | #else 218 | /* poll for size change */ 219 | while (off == lseek(fd, 0, SEEK_END)) 220 | usleep(DELAY); 221 | #endif 222 | continue; 223 | } else { 224 | flock(fd, LOCK_UN); 225 | break; 226 | } 227 | } 228 | 229 | if (off - loff > sizeof buf) 230 | off = loff + sizeof buf; 231 | 232 | rd = pread(fd, &buf, off - loff, loff); 233 | if (qflag) { 234 | if (!seen_nl) { 235 | char *s; 236 | if ((s = memchr(buf, '\n', rd))) { 237 | write(1, buf, s+1-buf); 238 | seen_nl = 1; 239 | } else { 240 | write(1, buf, rd); 241 | } 242 | } 243 | } else { 244 | write(1, buf, rd); 245 | } 246 | 247 | loff += rd; 248 | } 249 | 250 | if (qflag && !seen_nl) 251 | write(1, "\n", 1); 252 | 253 | #ifdef USE_INOTIFY 254 | inotify_rm_watch(ifd, wd); 255 | #endif 256 | #ifdef USE_KEVENT 257 | EV_SET(&kev, fd, EVFILT_VNODE, EV_DELETE, 0, 0, NULL); 258 | kevent(kq, &kev, 1, NULL, 0, NULL); 259 | #endif 260 | close(fd); 261 | } 262 | 263 | #ifdef USE_INOTIFY 264 | close(ifd); 265 | #endif 266 | #ifdef USE_KEVENT 267 | close(kq); 268 | #endif 269 | return 0; 270 | } 271 | -------------------------------------------------------------------------------- /nqtail.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | # nqtail - tail -F the queue outputs, quitting when the job finishes 3 | 4 | tailed=false 5 | for f in ${NQDIR:-.}/,*; do 6 | if ! nq -t $f; then 7 | tailed=true 8 | printf '==> %s\n' "$f" 9 | tail -F $f & p=$! 10 | nq -w $f 11 | kill $p 12 | fi 13 | done 14 | 15 | if ! $tailed; then 16 | cat $f 17 | fi 18 | -------------------------------------------------------------------------------- /nqterm: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | # nqterm CMD... - tmux/screen wrapper for nq to display output in new window 3 | 4 | set -e 5 | 6 | s=$(nq "$@") 7 | p=${s##*.} 8 | 9 | printf '%s\n' "$s" 10 | 11 | if [ -n "$p" ]; then 12 | if [ -n "$TMUX" ]; then 13 | tmux new-window -a -d -n '<' -c '#{pane_current_path}' \ 14 | "trap true INT QUIT TERM EXIT; 15 | nqtail $s || kill $p; 16 | printf '[%d exited, ^D to exit.]\n' $p; 17 | cat >/dev/null" 18 | elif [ -n "$STY" ]; then 19 | screen -t '<' sh -c "trap true INT QUIT TERM EXIT; 20 | nqtail $s || kill $p 21 | printf '[%d exited, ^D to exit.]\n' $p; 22 | cat >/dev/null" 23 | screen -X other 24 | fi 25 | fi 26 | -------------------------------------------------------------------------------- /nqterm.1: -------------------------------------------------------------------------------- 1 | .Dd July 3, 2024 2 | .Dt NQTERM 1 3 | .Os 4 | .Sh NAME 5 | .Nm nqterm 6 | .Nd job queue wrapper for tmux/screen 7 | .Sh SYNOPSIS 8 | .Nm 9 | .Ar command\ line ... 10 | .Sh DESCRIPTION 11 | .Nm 12 | is a tiny wrapper around the 13 | .Xr nq 1 14 | job queue which automatically spawns a corresponding 15 | .Xr nqtail 1 16 | watching process in a new 17 | .Xr tmux 1 18 | or 19 | .Xr screen 1 20 | window. 21 | .Pp 22 | You can cancel the 23 | .Xr nq 1 24 | job by pressing 25 | .Ic C-c 26 | in the job output window. 27 | .Pp 28 | After the job has finished, the window will 29 | close on 30 | .Ic C-d . 31 | .Sh ENVIRONMENT 32 | .Bl -hang -width Ds 33 | .It Ev NQDIR 34 | Directory where lock files/job output resides, see 35 | .Xr nq 1 . 36 | .El 37 | .Sh EXIT STATUS 38 | .Ex -std 39 | .Sh SEE ALSO 40 | .Xr nq 1 , 41 | .Xr nqtail 1 , 42 | .Xr screen 1 , 43 | .Xr tmux 1 44 | .Sh AUTHORS 45 | .An Leah Neukirchen Aq Mt leah@vuxu.org 46 | .Sh LICENSE 47 | .Nm 48 | is in the public domain. 49 | .Pp 50 | To the extent possible under law, 51 | the creator of this work 52 | has waived all copyright and related or 53 | neighboring rights to this work. 54 | .Pp 55 | .Lk http://creativecommons.org/publicdomain/zero/1.0/ 56 | .\" .Sh BUGS 57 | -------------------------------------------------------------------------------- /tests: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | 3 | : ${NQ:=../nq} 4 | : ${NQTAIL:=../nqtail} 5 | 6 | set -e 7 | 8 | check() { 9 | msg=$1 10 | shift 11 | if eval "$@" 2>/dev/null 1>&2; then 12 | printf 'ok - %s\n' "$msg" 13 | else 14 | printf 'not ok - %s\n' "$msg" 15 | false 16 | fi 17 | true 18 | } 19 | 20 | printf '1..36\n' 21 | 22 | rm -rf test.dir 23 | mkdir test.dir 24 | ( 25 | cd test.dir 26 | 27 | printf '# nq tests\n' 28 | check 'fails with no arguments' ! $NQ 29 | check 'succeeds enqueuing true' 'f=$($NQ true)' 30 | sleep 1 31 | check 'generated a lockfile' test -f $f 32 | check 'lockfile contains exec line' grep -q exec.*nq.*true $f 33 | check 'lockfile contains status line' grep -q exited.*status.*0 $f 34 | check 'lockfile is not executable' ! test -x $f 35 | ) 36 | 37 | rm -rf test.dir 38 | mkdir test.dir 39 | ( 40 | cd test.dir 41 | 42 | printf '# queue tests\n' 43 | check 'enqueing true' f1=$($NQ true) 44 | check 'enqueing sleep 500' f2=$($NQ sleep 500) 45 | check 'first job is done already' $NQ -t $f1 46 | check 'not all jobs are done already' ! $NQ -t 47 | check 'running job is executable' test -x $f2 48 | check 'running job not done already' ! $NQ -t $f 49 | check 'can kill running job' kill ${f2##*.} 50 | sleep 1 51 | check 'killed job is not executable anymore' ! test -x $f2 52 | check 'killed job contains status line' grep -q killed.*signal.*15 $f2 53 | ) 54 | 55 | rm -rf test.dir 56 | mkdir test.dir 57 | ( 58 | cd test.dir 59 | 60 | printf '# env tests\n' 61 | check 'enqueing env' f1=$($NQ env) 62 | $NQ -w 63 | check 'NQJOBID is set' grep -q NQJOBID=$f1 $f1 64 | ) 65 | 66 | rm -rf test.dir 67 | mkdir test.dir 68 | ( 69 | cd test.dir 70 | 71 | printf '# killing tests\n' 72 | check 'spawning four jobs' 'f1=$($NQ sleep 100)' 73 | check 'spawning four jobs' 'f2=$($NQ sleep 1)' 74 | check 'spawning four jobs' 'f3=$($NQ sleep 100)' 75 | check 'spawning four jobs' 'f4=$($NQ sleep 1)' 76 | check 'killing first job' kill ${f1##*.} 77 | check 'killing third job' kill ${f3##*.} 78 | check 'second job is running' ! $NQ -t $f2 79 | $NQ -w $f2 80 | check 'fourth job is running' ! $NQ -t $f4 81 | check 'all jobs are done' $NQ -w 82 | ) 83 | 84 | rm -rf test.dir 85 | mkdir test.dir 86 | ( 87 | cd test.dir 88 | 89 | printf '# nqtail tests\n' 90 | check 'spawning four jobs' 'f1=$($NQ sleep 100)' 91 | check 'spawning four jobs' 'f2=$($NQ echo two)' 92 | check 'spawning four jobs' 'f3=$($NQ sleep 300)' 93 | check 'spawning four jobs' 'f4=$($NQ sleep 400)' 94 | check 'nqtail tracks first job' '($NQTAIL ,* & p=$!; sleep 1; kill $p) | sed 3q | grep -q sleep.*100' 95 | check 'killing first job' kill ${f1##*.} 96 | check 'killing fourth job' kill ${f4##*.} 97 | sleep 1 98 | check 'nqtail tracks third job' '($NQTAIL ,* & p=$!; sleep 1; kill $p) | sed 3q | grep -q sleep.*300' 99 | check 'killing third job' kill ${f3##*.} 100 | sleep 1 101 | check 'nqtail outputs last job when no job running' '$NQTAIL ,* | sed 3q | grep -q sleep.*400' 102 | ) 103 | 104 | rm -rf test.dir 105 | --------------------------------------------------------------------------------