├── Dockerfile ├── src ├── header ├── fp ├── pipelock ├── init ├── multi ├── wait ├── remote ├── parallel ├── fn ├── atom ├── rc ├── list ├── semaphore ├── heap ├── future ├── ref └── gc ├── README.md └── bash-lambda /Dockerfile: -------------------------------------------------------------------------------- 1 | FROM ubuntu:16.04 2 | ADD bash-lambda /root/ 3 | RUN echo source /root/bash-lambda >> /root/.bashrc 4 | CMD /bin/bash 5 | -------------------------------------------------------------------------------- /src/header: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # Bash-lambda 3 | 4 | # Source this file to create a function allocation space and enable first-class 5 | # functions in the shell. The heap will be deleted automatically when the shell 6 | # exits. See https://github.com/spencertipping/bash-lambda for documentation. 7 | 8 | -------------------------------------------------------------------------------- /src/fp: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # Bash-lambda functional programming constructs 3 | 4 | # $(comp $f $g $h) x = f $(g $(h x)) 5 | # $(partial $f x) y = f x y 6 | bash_lambda_comp() { 7 | declare i body 8 | for (( i = $#; i >= 1; i -= 1 )); do 9 | if (( $i == $# )); then body="\$(${!i} \"\$@\")"; 10 | else body="\$(${!i} $body)"; fi 11 | done 12 | bash_lambda_fn "echo $body"; } 13 | 14 | bash_lambda_partial() { 15 | bash_lambda_fn "$* \"\$@\""; } 16 | -------------------------------------------------------------------------------- /src/pipelock: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # Bash-lambda pipe locks 3 | 4 | # A pipe-lock is a way to block one process until another one lets it through. 5 | # This provides a way to block without polling, and is used internally by 6 | # futures. 7 | 8 | # WARNING 9 | # If you hand a pipelock to someone, you MUST block on it. If you don't, then 10 | # whoever attempts to unlock the pipelock will block, and this will result in 11 | # all kinds of problems and strange bugs. 12 | 13 | bash_lambda_pipelock() { 14 | declare file=$(bash_lambda_sym pipelock) 15 | rm $file && mkfifo $file 16 | echo $file; } 17 | 18 | bash_lambda_pipelock_grab() { cat $1 >& /dev/null; } 19 | bash_lambda_pipelock_release() { echo > $1; } 20 | -------------------------------------------------------------------------------- /src/init: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # Bash-lambda function exporting and GC hooks 3 | 4 | # Export the bash_lambda library into the current heap 5 | bash_lambda_init() { 6 | declare -f | grep '^bash_lambda' | sed 's/ .*//' | 7 | (declare fn; while read fn; do 8 | bash_lambda_extern $fn > /dev/null 9 | if [[ -z "$BASH_LAMBDA_NO_ALIASES" ]]; then 10 | bash_lambda_def ${fn##bash_lambda_} $fn; fi; done 11 | 12 | bash_lambda_reload_rc 13 | bash_lambda_message 'λ') & } 14 | 15 | # Initialize the heap only if we own it. 16 | [[ "$BASH_LAMBDA_OWN_HEAP" == yes ]] && (bash_lambda_init) 17 | 18 | # Run a GC, if necessary, after each command 19 | export PROMPT_COMMAND="${PROMPT_COMMAND:-:}; bash_lambda_auto_gc" 20 | -------------------------------------------------------------------------------- /src/multi: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # Bash-lambda multimethods 3 | 4 | # These work only on fully-named parameters, not on stuff coming from stdin. If 5 | # we can observe the name, then we can extract the type from the beginning of 6 | # the filename. For example: 7 | # 8 | # $ future $f 9 | # /tmp/blheap-xxxx-xxxx/future_xxxxxxxx 10 | # $ ref_type $(future $f) 11 | # future 12 | # $ 13 | # 14 | # We then prepend this to the multimethod name to get the specific function 15 | # name: 16 | # 17 | # $ defmulti get 18 | # $ get $(future $f) -> future_get $(future $f) 19 | 20 | bash_lambda_defmulti() { 21 | declare multi_name=$1 22 | bash_lambda_defn $1 '$(bash_lambda_ref_type $1)_'$multi_name' "$@"'; } 23 | 24 | # Multimethod definitions for bash-lambda functions 25 | # state finished block notify src/future 26 | # get src/future src/atom 27 | # unsafe_get src/atom 28 | # count src/list src/semaphore 29 | # grab release wrap src/semaphore src/atom src/pipelock 30 | 31 | (declare method 32 | for method in state finished block notify \ 33 | get \ 34 | unsafe_get \ 35 | count \ 36 | grab release wrap; do 37 | bash_lambda_defmulti $method; done) > /dev/null 38 | -------------------------------------------------------------------------------- /src/wait: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # Bash-lambda wait functions 3 | 4 | # Various ways to wait for things. Usually you would use this with semaphores, 5 | # but you could do it with any function. 6 | 7 | bash_lambda_poll_constantly() { echo $1; } 8 | bash_lambda_poll_linearly() { bash_lambda_fn "echo \$((\$1 + ${1:-1}))"; } 9 | bash_lambda_poll_exponentially() { bash_lambda_fn "echo \$((\$1 * ${1:-2}))"; } 10 | 11 | bash_lambda_waiting_fn() { 12 | # Returns a thunk that waits for the given function to return true, polling 13 | # constantly by default (you can change this by passing a polling interval 14 | # function as the third argument). Echoes all output from the function. The 15 | # second argument specifies the initial polling duration. 16 | declare f=$1 delay=${2:-1} adjust=${3:-bash_lambda_poll_constantly} 17 | bash_lambda_fn "declare delay=$delay 18 | until $f \"\$@\"; do sleep \$delay 19 | delay=\$($adjust \$delay); done"; } 20 | 21 | bash_lambda_wait_until() { $(bash_lambda_waiting_fn "$@"); } 22 | 23 | bash_lambda_spinning_fn() { 24 | # Returns a thunk that spins until the given function returns true. This 25 | # should be used only for very quick waits. 26 | declare f=$1 27 | bash_lambda_fn "until $f \"\$@\"; do :; done"; } 28 | 29 | bash_lambda_spin_until() { $(bash_lambda_spinning_fn "$@"); } 30 | 31 | bash_lambda_wait_wrap() { $(wait_until $(bash_lambda_partial wrap "$@")); } 32 | bash_lambda_spin_wrap() { $(spin_until $(bash_lambda_partial wrap "$@")); } 33 | -------------------------------------------------------------------------------- /src/remote: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # Bash-lambda remote functions 3 | 4 | # Remote functions allow you to transparently migrate the execution of a 5 | # function to another machine. Any relevant heap state will be transferred 6 | # automatically. Remote functions use passwordless SSH and their stdin/stdout 7 | # and exit codes are transparently proxied. 8 | 9 | # Note that remote functions are run from the current $PWD, except that any 10 | # changes to $HOME are taken into account. For instance, if I'm spencertipping 11 | # on one computer and spencer on another, and I run this from ~/bin, then: 12 | # 13 | # $ remote the-spencertipping-machine $(fn 'echo $PWD') 14 | # /home/spencertipping/bin 15 | # $ remote the-spencer-machine $(fn 'echo $PWD') 16 | # /home/spencer/bin 17 | # $ 18 | 19 | # If the remote command produces a nonzero exit code, then the heap snapshot is 20 | # preserved to help you find out what went wrong. You can get rid of old 21 | # snapshots on remote systems using remote_clean. 22 | 23 | bash_lambda_remote() { 24 | declare host=$1 f=$2 cd_into=${PWD#$HOME/} 25 | cd_into=${cd_into#$HOME} # in case no trailing slash 26 | declare snapshot=$(bash_lambda_ref_snapshot $f) 27 | 28 | declare path="export PATH=\"\$PATH:$BASH_LAMBDA_HEAP\"" 29 | declare pre="cd $cd_into && tar xzP" 30 | declare clean="rm -rf \"$BASH_LAMBDA_HEAP\"" 31 | 32 | ssh $host "$pre && $path && $f && $clean" < $snapshot; } 33 | 34 | bash_lambda_remote_clean() { 35 | # Removes this instance's heap snapshot from the remote instance. 36 | ssh $host "rm -rf \"$BASH_LAMBDA_HEAP\""; } 37 | -------------------------------------------------------------------------------- /src/parallel: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # Bash-lambda parallel execution 3 | 4 | # Parallel tasks are defined as an interface that transposes parallelism over 5 | # lists. They generally return futures. You should set the 6 | # BASH_LAMBDA_PARALLELISM value to something more specific to your own system, 7 | # depending on available CPUs and memory. 8 | # 9 | # There is a considerable degree of overhead (on the order of seconds) 10 | # associated with using these parallel methods. 11 | 12 | export BASH_LAMBDA_PARALLELISM=4 13 | 14 | bash_lambda_parallel_map() { 15 | # Maps a function over a list in parallel. Results retain their original 16 | # order. An optional third parameter allows you to override the default level 17 | # of parallelism. 18 | 19 | # This function returns a list of futures, each of which represents an 20 | # intermediate outcome. The futures are generated lazily, so in theory you 21 | # could have an infinite source list and, so long as you requested only a 22 | # finite number of elements, you would be ok. (In practice, I think this is a 23 | # dangerous strategy.) To get the completed results for a segment of the 24 | # list: 25 | # 26 | # $ get $(transpose $(list $(parallel_map $f $xs | take 5))) 27 | 28 | # Note: If your function returns a nonzero exit code, it will be run again on 29 | # the same data. 30 | 31 | declare f=$1 xs=$2 n=${3:-$BASH_LAMBDA_PARALLELISM} 32 | declare s=$(bash_lambda_semaphore $n) 33 | 34 | declare blocking_f=$(bash_lambda_waiting_fn \ 35 | $(bash_lambda_semaphore_wrap $s $f)) 36 | 37 | bash_lambda_map $(bash_lambda_partial bash_lambda_future $blocking_f) $xs; } 38 | -------------------------------------------------------------------------------- /src/fn: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # Bash-lambda function and closure allocation 3 | 4 | bash_lambda_fn_body() { 5 | echo '#!/bin/bash' 6 | declare i 7 | for (( i = 1; i <= $#; i += 1 )); do 8 | if (( $i < $# )); then echo "declare -r ${!i}=\$$i" 9 | else echo "${!i}"; fi 10 | done; } 11 | 12 | bash_lambda_fn() { 13 | # Yup, we're allocating closures by writing to files and returning their 14 | # names to the callers. This gives you controllable persistence, heap 15 | # allocation, and the ability to reference the same closures across multiple 16 | # processes. 17 | bash_lambda_fn_body "$@" | bash_lambda_cons fn; } 18 | 19 | bash_lambda_cons_fn() { 20 | # Same as bash_lambda_fn, but body is specified from stdin. Useful for 21 | # multiline functions when used with heredocs. 22 | (bash_lambda_fn_body "$@" ''; cat -) | bash_lambda_cons fn; } 23 | 24 | bash_lambda_defn() { declare name=$1; shift 25 | bash_lambda_gc_pin \ 26 | $(bash_lambda_fn_body "$@" | bash_lambda_cons -n $name); } 27 | 28 | # Exports functions into named files in the heap. This allows them to reference 29 | # each other from inside heap-allocated closures. Any exported functions are 30 | # pinned so that they will never be garbage-collected. 31 | bash_lambda_extern() { 32 | bash_lambda_gc_pin $( (echo "#!/bin/bash" 33 | declare -f "$1" 34 | echo "$1 \"\$@\"") | bash_lambda_cons -n $1); } 35 | 36 | bash_lambda_def() { rm -f $BASH_LAMBDA_HEAP/$1 37 | ln -s $2 $(bash_lambda_gc_pin $BASH_LAMBDA_HEAP/$1); } 38 | 39 | bash_lambda_defalias() { 40 | declare name=$1; shift 41 | bash_lambda_def "$name" $(bash_lambda_cons_fn <<<"$*"); } 42 | -------------------------------------------------------------------------------- /src/atom: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # Bash-lambda atomic values 3 | 4 | # An atomic value supports thread-safe assignment and access. It does this by 5 | # guarding every read and write with a semaphore. You can also request a 6 | # transaction against the current value of an atomic value, so that during that 7 | # transaction any requests to change the value will block or fail. 8 | 9 | bash_lambda_atom() { 10 | printf "%s\n%s" "$(bash_lambda_mutex)" \ 11 | "$(echo "$1" | bash_lambda_cons)" | bash_lambda_cons atom; } 12 | 13 | bash_lambda_atom_grab() { 14 | bash_lambda_mutex_grab "$(bash_lambda_nth 0 "$1")"; } 15 | 16 | bash_lambda_atom_release() { bash_lambda_mutex_release "$1"; } 17 | bash_lambda_atom_get() { $(bash_lambda_atom_wrap "$1" cat); } 18 | 19 | # Unsafe-get is considerably faster than get, but does not respect the 20 | # atomicity of running transactions. Use this only when you have a single state 21 | # transition acting on the atom. (See src/future for an example) 22 | bash_lambda_atom_unsafe_get() { cat "$(bash_lambda_nth 1 "$1")"; } 23 | 24 | bash_lambda_atom_wrap() { 25 | # A generalized transaction/set function. $(wrap $atom $f) does one of two 26 | # things. If it acquires the atom's lock, then it invokes $f on a file 27 | # containing the atom's current value. $f is free to modify the contents of 28 | # this file. 29 | 30 | # You will probably want to use one of the functions in src/wait instead of 31 | # calling wrap directly. 32 | 33 | declare atom=$1 f=$2 34 | bash_lambda_fn "declare lock 35 | if lock=\$(bash_lambda_atom_grab $atom); then 36 | $f \"\$(bash_lambda_nth 1 $atom)\" 37 | declare status=\$? 38 | bash_lambda_atom_release \$lock 39 | exit \$status; fi"; } 40 | -------------------------------------------------------------------------------- /src/rc: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # Bash-lambda rc file functions 3 | 4 | export BASH_LAMBDA_RC=${BASH_LAMBDA_RC:-$HOME/.bash-lambda} 5 | export BASH_LAMBDA_EDITOR=${BASH_LAMBDA_EDITOR:-${EDITOR:-$VISUAL}} 6 | 7 | export COLUMNS # so that bash_lambda_message can be called from subshells 8 | 9 | bash_lambda_message() { 10 | declare m="$*" 11 | (( $COLUMNS )) && echo -en "\033[s\033[$((COLUMNS - ${#m}))G$m\033[u" 1>&2; } 12 | 13 | bash_lambda_setup_rc() { 14 | [[ -e "$BASH_LAMBDA_RC" ]] || sed 's/^ //' > "$BASH_LAMBDA_RC" <<'EOF' 15 | #!/bin/bash 16 | # You can put function defs here. Variables you define here aren't visible, 17 | # since this file is always evaluated (usually asynchronously) from inside a 18 | # subshell. 19 | # 20 | # This file is sourced asynchronously when you start your shell, so adding 21 | # definitions won't increase the amount of time required to open a new 22 | # terminal. 23 | # 24 | # See https://github.com/spencertipping/bash-lambda for details about 25 | # defining functions. 26 | # 27 | # For example: 28 | 29 | # File tests (wrappers for -d, -x, etc) 30 | def bash_tests $(ref $(list a b c d e f g h k p r s t u w x G L N O S z n)) 31 | defn deffiletest x 'defn is-$x f "[[ -$x \$f ]]"' 32 | map deffiletest $(bash_tests) 33 | 34 | defn newerthan file 'bash_lambda_fn f "[[ \$f -nt $file ]]"' 35 | defn olderthan file 'bash_lambda_fn f "[[ \$f -ot $file ]]"' 36 | defn eq file 'bash_lambda_fn f "[[ \$f -ef $file ]]"' 37 | 38 | # Content tests 39 | defn contains pattern 'egrep -o $pattern' 40 | defn without pattern 'sed "s/${pattern/\//\\\/}//g"' 41 | EOF 42 | } 43 | 44 | bash_lambda_reload_rc() { 45 | [[ -e "$BASH_LAMBDA_RC" ]] && (. "$BASH_LAMBDA_RC" > /dev/null); } 46 | 47 | bash_lambda_defs() { 48 | bash_lambda_setup_rc 49 | $BASH_LAMBDA_EDITOR "$BASH_LAMBDA_RC" 50 | (bash_lambda_reload_rc &); } 51 | -------------------------------------------------------------------------------- /src/list: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # Bash-lambda list programming constructs 3 | 4 | bash_lambda_list() { declare x 5 | for x; do echo "$x"; done | bash_lambda_cons list; } 6 | 7 | bash_lambda_nth() { cat ${2:--} | head -n$(($1 + 1)) | tail -n1; } 8 | 9 | bash_lambda_take() { cat ${2:--} | head -n$1; } 10 | bash_lambda_drop() { cat ${2:--} | tail -n+$(($1 + 1)); } 11 | 12 | # Here, 'map' works as both map and mapcat since cons and append are the same 13 | # operation. This arises due to the associativity of cons. 14 | bash_lambda_map() { 15 | cat ${2:--} | (declare x; while read x; do $1 "$x"; done); } 16 | 17 | bash_lambda_reduce() { 18 | declare f=$1 x=$2 19 | cat ${3:--} | (declare y; while read y; do x="$($f "$x" "$y")"; done 20 | echo "$x"); } 21 | 22 | bash_lambda_reductions() { 23 | declare f=$1 x=$2 24 | cat ${3:--} | (declare y; while read y; do x="$($f "$x" "$y")"; echo "$x" 25 | done); } 26 | 27 | bash_lambda_filter() { 28 | cat ${2:--} | (declare x; while read x; do 29 | $1 "$x" > /dev/null && echo "$x"; done); } 30 | 31 | bash_lambda_partition() { 32 | cat ${2:--} | split -l $1 -u --filter='bash_lambda_cons list'; } 33 | 34 | # This is a multimethod that is normally invoked as just 'count'; e.g: 35 | # $ count $(list 1 2 3) 36 | # If you want to count lines from a stream, you should use wc -l instead. 37 | bash_lambda_list_count() { wc -l < "$1"; } 38 | 39 | # List generators 40 | bash_lambda_iterate() { 41 | declare x=$2 42 | echo "$x"; while x="$($1 "$x")"; do echo "$x"; done; } 43 | 44 | bash_lambda_repeatedly() { 45 | declare i f=$1 n=$2 46 | for (( i = 0; i != ${n:--1}; i += 1 )); do $f || return $?; done; } 47 | 48 | # Tests over lists 49 | bash_lambda_some() { 50 | cat ${2:--} | (declare x; while read x; do 51 | if $1 "$x" > /dev/null; then 52 | echo "$x"; return 0; fi; done; return 1); } 53 | 54 | bash_lambda_every() { 55 | cat ${2:--} | (declare x; while read x; do 56 | if ! $1 "$x" > /dev/null; then 57 | echo "$x"; return 1; fi; done; return 0); } 58 | -------------------------------------------------------------------------------- /src/semaphore: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # Bash-lambda semaphores and mutexes 3 | 4 | # Semaphores are directories that contain empty numbered subdirectories along 5 | # with an immutable file containing the capacity of the semaphore. Locking is 6 | # done by attempting to create one of these empty subdirectories, and ls -d | 7 | # wc -l is used to get the number of used entries. 8 | 9 | bash_lambda_semaphore() { 10 | declare capacity=$1 semaphore=$(bash_lambda_dir semaphore) 11 | echo $capacity > "$semaphore/capacity" 12 | echo "$semaphore"; } 13 | 14 | bash_lambda_semaphore_grab() { 15 | # Take an item from the semaphore; returns the item's identifier (a path, but 16 | # you don't need to know this) and exit code 0 if successful, returns 1 and 17 | # outputs nothing if unsuccessful. 18 | declare semaphore=$1 i capacity=$(<"$1/capacity") 19 | for (( i = 0; i < $capacity; i += 1 )); do 20 | if mkdir "$semaphore/semaphore_$i" >& /dev/null; then 21 | echo "$semaphore/semaphore_$i"; return 0; fi; done 22 | return 1; } 23 | 24 | bash_lambda_semaphore_release() { 25 | # Releases an item from the semaphore. You can just pass the item here; no 26 | # need to pass the semaphore also. 27 | rmdir "$1"; } 28 | 29 | bash_lambda_semaphore_count() { 30 | # Counts the number of items available in the semaphore. 31 | declare semaphore=$1 32 | echo $(( $(<"$semaphore/capacity") + 1 - $(ls -d "$semaphore/"* | wc -l) )); } 33 | 34 | bash_lambda_semaphore_wrap() { 35 | # Wraps a function's execution with a grab/release of a semaphore item. 36 | declare s=$1 f=$2 37 | bash_lambda_fn "declare lock 38 | if lock=\$(bash_lambda_semaphore_grab $s); then 39 | $f \"\$@\" 40 | declare status=\$? 41 | bash_lambda_semaphore_release \$lock 42 | exit \$status; fi"; } 43 | 44 | # Mutex: a more efficient way to do a single-element semaphore. 45 | bash_lambda_mutex() { bash_lambda_dir mutex; } 46 | bash_lambda_mutex_grab() { mkdir "$1/lock" >& /dev/null && echo "$1/lock"; } 47 | bash_lambda_mutex_release() { rmdir "$1"; } 48 | bash_lambda_mutex_count() { [[ -d "$1/lock" ]]; echo $?; } 49 | 50 | bash_lambda_mutex_wrap() { 51 | declare m=$1 f=$2 52 | bash_lambda_fn "declare lock 53 | if lock=\$(bash_lambda_mutex_grab $m); then 54 | $f \"\$@\" 55 | declare status=\$? 56 | bash_lambda_mutex_release \$lock 57 | exit \$status; fi"; } 58 | -------------------------------------------------------------------------------- /src/heap: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # Bash-lambda disk-based heap 3 | 4 | # We need 128 bits of entropy for the heap directory name. This gives us 5 | # guarantees about the soundness of conservative garbage collection ... for the 6 | # rest of the story, see src/gc. 7 | 8 | # 128 bits total / 15 bits per random number = 9 numbers 9 | bash_lambda_genkey() { declare i 10 | for (( i = 0; i < 9; i += 1 )); do 11 | printf '%04x' $RANDOM; done; } 12 | 13 | # Is the heap directory set already? If so, then we don't own it, so we won't 14 | # nuke it when the process ends. 15 | [[ -z "$BASH_LAMBDA_HEAP" ]] && declare -r BASH_LAMBDA_OWN_HEAP=yes 16 | 17 | # Allow the user to override the heap location if they want to. However, this 18 | # may compromise the effectiveness of conservative GC. 19 | export BASH_LAMBDA_KEY=$(bash_lambda_genkey) 20 | export BASH_LAMBDA_HEAP=${BASH_LAMBDA_HEAP:-${TMPDIR:-/tmp}/blheap-$$-$BASH_LAMBDA_KEY} 21 | 22 | if [[ "$BASH_LAMBDA_OWN_HEAP" == yes ]]; then 23 | mkdir -p "$BASH_LAMBDA_HEAP" || return 1 24 | ln -s "$BASH_LAMBDA_HEAP" "$BASH_LAMBDA_HEAP/.weak-references" 25 | date +%s > "$BASH_LAMBDA_HEAP/.last-gc" 26 | fi 27 | 28 | bash_lambda_nuke_heap() { 29 | [[ -e "$BASH_LAMBDA_HEAP/.gc-pid" ]] && kill $(<"$BASH_LAMBDA_HEAP/.gc-pid") 30 | [[ "${BASH_LAMBDA_HEAP:0:5}" == "/tmp/" ]] && rm -rf "$BASH_LAMBDA_HEAP"; } 31 | 32 | [[ "$BASH_LAMBDA_OWN_HEAP" == yes ]] && trap bash_lambda_nuke_heap EXIT 33 | export PATH="$PATH:$BASH_LAMBDA_HEAP" 34 | 35 | # Heap allocation 36 | bash_lambda_gensym() { mktemp "$BASH_LAMBDA_HEAP/${1:-gensym}_XXXXXXXXXXXXXX"; } 37 | bash_lambda_gendir() { mktemp -d "$BASH_LAMBDA_HEAP/${1:-gensym}_XXXXXXXXXXX"; } 38 | 39 | bash_lambda_sym() { bash_lambda_gc_guard "$(bash_lambda_gensym "$@")"; } 40 | bash_lambda_dir() { bash_lambda_gc_guard "$(bash_lambda_gendir "$@")"; } 41 | 42 | bash_lambda_gc_guard() { 43 | declare file=$1 44 | 45 | # If garbage collection is in-progress, mark the object to prevent it from 46 | # being collected this time around. We don't mark it as having been visited, 47 | # however; doing that would cause anything it makes live to be collected 48 | # erroneously. 49 | touch "$BASH_LAMBDA_HEAP/.gc-marked-set/${file##*/}" >& /dev/null 50 | echo "$file"; } 51 | 52 | bash_lambda_cons() { 53 | if [[ $1 == '-n' ]]; then declare file=$BASH_LAMBDA_HEAP/$2 54 | else declare file="$(bash_lambda_gensym $1)"; fi 55 | 56 | cat > $(bash_lambda_gc_guard "$file") && # Take everything from stdin 57 | chmod u+x "$file" && # All conses are executable 58 | echo "$file"; } 59 | -------------------------------------------------------------------------------- /src/future: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # Bash-lambda futures (asynchronous processes) 3 | 4 | # Future locking 5 | # It is surprisingly difficult to reliably wait for a future. We can't use the 6 | # 'wait' command because the future could belong to another subshell. We can't 7 | # use the process table or /proc because PIDs get reused. This leaves us with 8 | # two options: we can use polling against some lock file, or we can devise some 9 | # scheme with pipelocks. 10 | # 11 | # It turns out that we can use a semaphore to do everything we need. 12 | 13 | bash_lambda_future() { 14 | # Construct a future around the given function application. To do this, we 15 | # run the function in the background and pipe its output to a temporary file. 16 | # We also record its exit code. 17 | 18 | declare output=$(bash_lambda_cons future_output < /dev/null) 19 | declare status=$(bash_lambda_cons future_status < /dev/null) 20 | declare state=$(bash_lambda_atom running) 21 | declare result=$(printf $"%s\n%s\n%s\n" $output $status $state | \ 22 | bash_lambda_cons future) 23 | 24 | ("$@" > $output; echo $? > $status; notify $result) > /dev/null & 25 | echo $result; } 26 | 27 | bash_lambda_future_finished() { 28 | # Exits with 0 if the future is finished, 1 if still running. If 0, a 29 | # future_get call will block only on IO, but not on job completion. This 30 | # operation can use unsafe_get because futures never un-finish. 31 | [[ "$(bash_lambda_atom_unsafe_get "$(bash_lambda_nth 2 $1)")" == done ]]; } 32 | 33 | bash_lambda_future_state() { 34 | bash_lambda_spin_wrap $(bash_lambda_nth 2 $1) cat; } 35 | 36 | bash_lambda_future_block() { 37 | # Block on completion of the future. Spin-locks against the future's state. 38 | declare pipelock=$(bash_lambda_spin_wrap $(bash_lambda_nth 2 $1) \ 39 | $(fn x "[[ \"\$(<\$x)\" == done ]] || bash_lambda_pipelock | tee -a \$x")) 40 | [[ -z "$pipelock" ]] || bash_lambda_pipelock_grab $pipelock; } 41 | 42 | bash_lambda_future_notify() { 43 | # Notify all listeners that this future is done. This amounts to unblocking 44 | # all of the pipelocks that have been appended to the state. 45 | bash_lambda_spin_wrap $(bash_lambda_nth 2 $1) \ 46 | $(fn x '[[ "$(<$x)" == done ]] || drop 1 $x | bash_lambda_map release 47 | echo done > $x'); } 48 | 49 | bash_lambda_future_get() { 50 | # This function blocks on the future's process if it is still running, and 51 | # its stdout and exit code are proxied. 52 | bash_lambda_future_finished "$1" || bash_lambda_future_block "$1" 53 | cat "$(bash_lambda_nth 0 "$1")" 54 | return "$(< "$(bash_lambda_nth 1 "$1")")"; } 55 | 56 | bash_lambda_future_map() { 57 | # Returns a future of a function applied to this future's value. 58 | bash_lambda_future $(fn "$2 \$(future_get $1)"); } 59 | 60 | bash_lambda_future_unsafe_get() { 61 | # Gets whatever stdout has been produced so far. The process may not have 62 | # exited, so this function returns 0. 63 | cat "$(bash_lambda_nth 0 "$1")"; } 64 | 65 | bash_lambda_future_transpose() { 66 | bash_lambda_future $(bash_lambda_partial \ 67 | bash_lambda_map bash_lambda_future_get $1); } 68 | -------------------------------------------------------------------------------- /src/ref: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # Bash-lambda reference functions 3 | 4 | bash_lambda_ref() { 5 | # Returns a reference to something in the heap. A reference is a function 6 | # that, when run, echoes the thing it refers to. You need to create refs to 7 | # point to heap objects if you can't expose variables (from ~/.bash-lambda, 8 | # for example). 9 | bash_lambda_fn "echo $1"; } 10 | 11 | # A weak reference is a path that resolves the same way as a regular path, but 12 | # that isn't followed by the garbage collector. src/heap sets up a symlink for 13 | # this purpose. 14 | bash_lambda_weak_ref() { 15 | echo "$BASH_LAMBDA_HEAP/.weak-references/${1##$BASH_LAMBDA_HEAP/}"; } 16 | 17 | bash_lambda_ref_resolve() { 18 | if [[ -e "$BASH_LAMBDA_HEAP/$1" ]]; then echo "$BASH_LAMBDA_HEAP/$1" 19 | else echo "$1"; fi; } 20 | 21 | bash_lambda_ref_snapshot() { 22 | tar -czP $(bash_lambda_ref_closure "$1") | bash_lambda_cons snapshot; } 23 | 24 | bash_lambda_ref_intern() { cat ${1:--} | tar -xzP; } 25 | 26 | bash_lambda_ref_closure() { 27 | declare visited=$(bash_lambda_dir) object=$(bash_lambda_ref_resolve "$1") 28 | bash_lambda_ref_visit "$visited" "$object" 29 | ls -d "$visited"/* | (declare x; while read x; do 30 | echo "$BASH_LAMBDA_HEAP/${x##$visited/}"; done); } 31 | 32 | bash_lambda_ref_type() { declare base=${1##*/}; echo "${base%%_*}"; } 33 | 34 | bash_lambda_ref_visit() { 35 | # The ref in question must be an object that exists in the heap. We expect it 36 | # to be a full pathname, though the object itself should be a direct child of 37 | # the heap directory. 38 | declare directory=$1 ref=$2 39 | declare ref_name="${ref#$BASH_LAMBDA_HEAP/}" 40 | 41 | # Is this something that belongs to the GC? 42 | [[ "$ref_name" == ".gc-marked-set" || 43 | "$ref_name" == ".gc-visited-set" ]] && return 0 44 | 45 | # No need to mark an object in a subdirectory. It isn't a direct child of the 46 | # heap, so its storage is already being managed by the directory it belongs 47 | # to. 48 | if [[ ! ("$ref_name" =~ /) ]]; then 49 | # Have we already visited this object? If so, no need to revisit it. 50 | [[ -e "$directory/$ref_name" ]] && return 0 51 | 52 | # Otherwise, mark the object and everything it points to. 53 | touch "$directory/$ref_name"; fi 54 | 55 | bash_lambda_ref_read "$ref" | bash_lambda_ref_children | (declare x 56 | while read x; do bash_lambda_ref_visit "$directory" "$x"; done); } 57 | 58 | bash_lambda_ref_read() { 59 | # Read the contents of the given object. The result will be an output stream 60 | # suitable for consumption by bash_lambda_ref_children. 61 | if [[ -d "$1" ]]; then [[ -e "$1"/* ]] && ls -d "$1"/* 62 | elif [[ -L "$1" ]]; then readlink "$1" 63 | elif [[ -p "$1" ]]; then return 0 64 | elif [[ -e "$1" ]]; then cat "$1"; fi; } 65 | 66 | bash_lambda_ref_children() { 67 | # Locate occurrences of the heap directory. This name contains 128 bits of 68 | # pseudorandom entropy, so we are unlikely to see it spuriously referenced. 69 | # If we observe a path that exists, then we consider that to be a reference 70 | # for GC purposes. 71 | 72 | egrep -o "$BASH_LAMBDA_HEAP/[^ [:space:]/\)\}\"']+" | (declare ref 73 | while read ref; do [[ -e "$ref" ]] && echo "$ref"; done); } 74 | -------------------------------------------------------------------------------- /src/gc: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # Bash-lambda concurrent mark-sweep garbage collector 3 | 4 | # BASH_LAMBDA_GC_SECONDS=0 will disable automatic GC 5 | # BASH_LAMBDA_GC_CONCURRENT=0 will cause GC to be synchronous 6 | export BASH_LAMBDA_GC_SECONDS=${BASH_LAMBDA_GC_SECONDS:-30} 7 | export BASH_LAMBDA_GC_CONCURRENT=${BASH_LAMBDA_GC_CONCURRENT:-1} 8 | 9 | mkdir -p "$BASH_LAMBDA_HEAP"/.gc-permanent 10 | 11 | # Can we see this unexported variable from the garbage collector? If not, then 12 | # we don't have access to the full root set. If you unset this, GC will refuse 13 | # to run. We need this to be a weak ref so the exported variable doesn't show 14 | # up in the root set by itself. These two variables should be identical modulo 15 | # path. 16 | BASH_LAMBDA_GC_CANARY=$(bash_lambda_cons < /dev/null) 17 | export BASH_LAMBDA_GC_KEY=$(bash_lambda_weak_ref $BASH_LAMBDA_GC_CANARY) 18 | 19 | # This function can be run asynchronously: bash_lambda_gc &. See 20 | # bash_lambda_auto_gc for an example. It knows which heap to run on, and it 21 | # echoes information about the number of objects and bytes reclaimed. 22 | bash_lambda_gc() { 23 | # Fix a bug wherein the asynchronous GC can run with a PWD on another 24 | # filesystem, preventing that filesystem from being unmounted while the GC is 25 | # happening (despite the fact that the GC itself doesn't use the PWD). 26 | cd 27 | 28 | # Avert disaster: never GC a heap unless it is in /tmp. 29 | if [[ "${BASH_LAMBDA_HEAP:0:5}" != "/tmp/" ]]; then 30 | echo 'bash_lambda_gc refuses to run because $BASH_LAMBDA_HEAP is set to' 31 | echo 'a directory outside of /tmp/. If you really want to do this, you' 32 | echo 'should do something like /tmp/../wherever.' 33 | return 2; fi 34 | 35 | # Try to acquire the GC lock. If we can't, another GC is probably running; so 36 | # we exit immediately since GC is (ideally) idempotent. 37 | mkdir "$BASH_LAMBDA_HEAP"/.gc-visited-set || return 1 38 | mkdir -p "$BASH_LAMBDA_HEAP"/.gc-marked-set 39 | 40 | echo $$ > "$BASH_LAMBDA_HEAP"/.gc-pid 41 | 42 | # We have the lock. At this point the heap allocator will mark any new 43 | # objects that are created, so we can use the root set that exists right now 44 | # and start marking objects referenced by that root set. 45 | bash_lambda_gc_roots | bash_lambda_ref_children | (declare x; while read x; do 46 | bash_lambda_ref_visit "$BASH_LAMBDA_HEAP/.gc-visited-set" "$x"; done) 47 | 48 | # We should mark the canary. If we haven't done this, then something is wrong 49 | # with the root-set computation. 50 | if [[ ! -e "$BASH_LAMBDA_HEAP/.gc-visited-set/${BASH_LAMBDA_GC_KEY##*/}" ]] 51 | then 52 | echo 'bash_lambda_gc cannot see the full root set. You should make sure' 53 | echo 'it is being run as a function, not as an indirect script; try using' 54 | echo 'bash_lambda_gc instead of just gc.' 55 | rm -r "$BASH_LAMBDA_HEAP/".gc-{visited,marked}-set 56 | return 1; fi 57 | 58 | # Now nuke anything that isn't either marked or permanent. 59 | ls -d "$BASH_LAMBDA_HEAP"/* | ( 60 | declare ref reclaimed=() reclaimed_size=0 61 | while read ref; do 62 | if [[ ! -e "$BASH_LAMBDA_HEAP/.gc-marked-set/${ref##*/}" && 63 | ! -e "$BASH_LAMBDA_HEAP/.gc-visited-set/${ref##*/}" && 64 | ! -e "$BASH_LAMBDA_HEAP/.gc-permanent/${ref##*/}" ]]; then 65 | reclaimed+=("$ref") 66 | reclaimed_size=$((reclaimed_size + $(du -sb "$ref" | cut -f 1))) 67 | rm -rf "$ref"; fi; done 68 | echo "${#reclaimed[@]} $reclaimed_size") 69 | 70 | # We are now done. Remove the marked-set directory and echo some stats about 71 | # the stuff we collected. 72 | rm -rf "$BASH_LAMBDA_HEAP/".gc-{marked,visited}-set; } 73 | 74 | bash_lambda_auto_gc() { 75 | # Trigger a concurrent GC if it's been more than some number of seconds since 76 | # the last one. This turns out to be one of the few constant-time ways we can 77 | # do this safely. 78 | if [[ ! -e "$BASH_LAMBDA_HEAP/.last-gc" ]] || \ 79 | (( $BASH_LAMBDA_GC_SECONDS && 80 | $(date +%s) - $(<"$BASH_LAMBDA_HEAP/.last-gc") > 81 | $BASH_LAMBDA_GC_SECONDS )); then 82 | date +%s > "$BASH_LAMBDA_HEAP/.last-gc" 83 | 84 | # NOTE: Running GC concurrently is experimental and is known to have some 85 | # problems unless done carefully. See comments in bash_lambda_gc for 86 | # details. 87 | if (( $BASH_LAMBDA_GC_CONCURRENT )); then 88 | (bash_lambda_gc --concurrent >> "$BASH_LAMBDA_HEAP/.gc-log" &) 89 | else 90 | bash_lambda_gc >> "$BASH_LAMBDA_HEAP/.gc-log" 91 | fi; fi; } 92 | 93 | bash_lambda_heap_stats() { 94 | printf '%-20s %s\n%-20s %d\n%-20s %d\n' \ 95 | 'heap size:' $(du -sh "$BASH_LAMBDA_HEAP" | cut -f 1) \ 96 | 'objects:' $(ls "$BASH_LAMBDA_HEAP" | wc -l) \ 97 | 'permanent:' $(ls "$BASH_LAMBDA_HEAP/.gc-permanent" | wc -l); } 98 | 99 | bash_lambda_heap_ls() { cd "$BASH_LAMBDA_HEAP"; ls "$@"; } 100 | 101 | bash_lambda_gc_pin() { touch "$BASH_LAMBDA_HEAP/.gc-permanent/${1##*/}"; 102 | echo "$1"; } 103 | 104 | bash_lambda_gc_unpin() { rm -f "$BASH_LAMBDA_HEAP/.gc-permanent/${1##*/}"; 105 | echo "$1"; } 106 | 107 | bash_lambda_gc_roots() { 108 | declare; ps ax; ls -d "$BASH_LAMBDA_HEAP/.gc-permanent"/* | (declare x 109 | while read x; do echo "$BASH_LAMBDA_HEAP/${x##*/}"; done); } 110 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Bash lambda 2 | 3 | **Note:** Since [switching to zsh for performance 4 | reasons](http://spencertipping.com/posts/2013.0814.bash-is-irrecoverably-broken.html) 5 | I'm no longer actively maintaining this project. I'll probably write something 6 | similar for zsh, but hopefully much faster since zsh has real hash support. 7 | 8 | Real lambda support for bash (a functionally complete hack). Includes a set of 9 | functions for functional programming, list allocation and traversal, futures, 10 | complete closure serialization, remote closure execution, multimethods, and 11 | concurrent mark/sweep garbage collection with weak reference support. 12 | 13 | MIT license as usual. 14 | 15 | ## Try it with Docker 16 | So you don't garbage-collect any heapfiles on your real filesystem. 17 | 18 | ```sh 19 | $ docker run --rm -it spencertipping/bash-lambda 20 | ``` 21 | 22 | ## Getting started 23 | 24 | Load the library into your shell (the `source` line can go into .bashrc): 25 | 26 | ``` 27 | $ git clone git://github.com/spencertipping/bash-lambda 28 | $ source bash-lambda/bash-lambda 29 | ``` 30 | 31 | *Before you do this, be aware that bash-lambda garbage-collects a part of your 32 | filesystem. Think about what might theoretically go wrong with this.* I source 33 | it in my .bashrc without problems, but as always, your mileage may vary. 34 | 35 | Sourcing `bash-lambda` will do a few things: 36 | 37 | 1. Create a heap directory in `$TMPDIR` (usually `/tmp`) 38 | 2. Add this heap directory to your `$PATH` 39 | 3. Add the asynchronous GC hook to your `$PROMPT_COMMAND` 40 | 4. Add lots of variables and functions to your shell process 41 | 5. Add an `EXIT` trap to nuke the heap directory 42 | 43 | Loading the library takes very little time even though a new heap is created 44 | for every process. This is because the heap setup and sourcing of your 45 | `.bash-lambda` file (run `defs` to edit it) is all done asynchronously. When 46 | the background initialization is complete, a small `λ` will appear in the 47 | rightmost column of your window. 48 | 49 | You can write arbitrary text to the right-hand side of the terminal by using 50 | the `message` function: 51 | 52 | ``` 53 | $ message hi 54 | $ hi 55 | ``` 56 | 57 | Before getting started, go ahead and run `defs` once. This will initialize your 58 | `~/.bash-lambda` file with some useful functions for common file operations. If 59 | you already have a `~/.bash-lambda` file from an earlier version, you can stash 60 | it somewhere, run `defs`, and then merge in previous definitions. 61 | 62 | ## Defining functions 63 | 64 | The functions provided by bash-lambda take their nomenclature from Clojure, and 65 | they are as analogous as I could make them while remaining somewhat useful in a 66 | bash context. They are, by example: 67 | 68 | ### Anonymous functions 69 | 70 | These are created with the `fn` form and work like this: 71 | 72 | ``` 73 | $ $(fn name 'echo "hello there, $name"') spencer 74 | hello there, spencer 75 | $ greet=$(fn name 'echo "hello there, $name"') 76 | $ $greet spencer 77 | hello there, spencer 78 | $ 79 | ``` 80 | 81 | The spec is `fn formals... 'body'`. You can use an alternative form, `cons_fn`, 82 | which takes the body from stdin: `cons_fn formals... <<'end'\n body \nend`. 83 | 84 | ### Named functions 85 | 86 | These are created using `defn`: 87 | 88 | ``` 89 | $ defn greet name 'echo "hi there, $name"' 90 | /tmp/blheap-xxxx-xxxxxxxxxxxxxxxxxxx/greet 91 | $ greet spencer 92 | hi there, spencer 93 | $ 94 | ``` 95 | 96 | Notice that we didn't need to dereference `greet` this time, since it's now a 97 | named function instead of a variable. 98 | 99 | You can use `def` to name any value: 100 | 101 | ``` 102 | $ def increment $(fn x 'echo $((x + 1))') 103 | $ increment 10 104 | 11 105 | $ 106 | ``` 107 | 108 | ### Partials and composition 109 | 110 | Like Clojure, bash-lambda gives you functions to create partial (curried) 111 | functions and compositions. These are called `partial` and `comp`, respectively. 112 | 113 | ``` 114 | $ add=$(fn x y 'echo $((x + y))') 115 | $ $(partial $add 5) 6 116 | 11 117 | $ $(comp $(partial $add 1) $(partial $add 2)) 5 118 | 8 119 | $ 120 | ``` 121 | 122 | ## Lists 123 | 124 | Bash-lambda gives you the usual suspects for list manipulation. However, there 125 | are a few differences from the way they are normally implemented. 126 | 127 | ### Mapping over lists 128 | 129 | ``` 130 | $ map $(partial $add 5) $(list 1 2 3 4) 131 | 6 132 | 7 133 | 8 134 | 9 135 | $ seq 1 4 | map $(partial $add 1) 136 | 2 137 | 3 138 | 4 139 | 5 140 | $ 141 | ``` 142 | 143 | The `list` function boxes up a series of values into a single file for later 144 | use. It's worth noting that this won't work: 145 | 146 | ``` 147 | $ map $(partial $add 5) $(map $(partial $add 1) $(list 1 2 3)) 148 | cat: 2: No such file or directory 149 | $ 150 | ``` 151 | 152 | You need to wrap the inner `map` into a list if you want to use applicative 153 | notation: 154 | 155 | ``` 156 | $ map $(partial $add 5) $(list $(map $(partial $add 1) $(list 1 2 3))) 157 | 7 158 | 8 159 | 9 160 | $ 161 | ``` 162 | 163 | Alternatively, just use pipes. This allows you to process lists lazily. 164 | 165 | ``` 166 | $ map $(partial $add 1) $(list 1 2 3) | map $(partial $add 5) 167 | 7 168 | 8 169 | 9 170 | $ 171 | ``` 172 | 173 | ### Reducing and filtering 174 | 175 | Two functions `reduce` and `filter` do what you would expect. (`reduce` isn't 176 | named `fold` like the Clojure function because `fold` is a useful shell utility 177 | already) 178 | 179 | ``` 180 | $ reduce $add 0 $(list 1 2 3) 181 | 6 182 | $ even=$(fn x '((x % 2 == 0))') 183 | $ seq 1 10 | filter $even 184 | 2 185 | 4 186 | 6 187 | 8 188 | 10 189 | $ 190 | ``` 191 | 192 | Higher-order functions work like you would expect: 193 | 194 | ``` 195 | $ sum_list=$(partial reduce $add 0) 196 | $ $sum_list $(list 1 2 3) 197 | 6 198 | $ rand_int=$(fn 'echo $RANDOM') 199 | $ repeatedly $rand_int 100 | $sum_list 200 | 1566864 201 | 202 | $ our_numbers=$(list $(repeatedly $rand_int 100)) 203 | ``` 204 | 205 | Another function, `reductions`, returns the intermediate results of reducing a 206 | list: 207 | 208 | ``` 209 | $ seq 1 4 | reductions $add 0 210 | 1 211 | 3 212 | 6 213 | 10 214 | $ 215 | ``` 216 | 217 | Filtering is useful when working with files, especially in conjunction with 218 | some builtins that come with the standard `~/.bash-lambda` file. For example: 219 | 220 | ``` 221 | $ ls | filter is-d # same as ls | filter $(fn '[[ -d $1 ]]') 222 | src 223 | $ ls | filter $(newerthan build) 224 | ``` 225 | 226 | ### Flatmap (mapcat in Clojure) 227 | 228 | It turns out that `map` already does what you need to write `mapcat`. The lists 229 | in bash-lambda behave more like Perl lists than like Lisp lists -- that is, 230 | consing is associative, so things are flattened out unless you box them up into 231 | files. Therefore, `mapcat` is just a matter of writing multiple values from a 232 | function: 233 | 234 | ``` 235 | $ self_and_one_more=$(fn x 'echo $x; echo $((x + 1))') 236 | $ map $self_and_one_more $(list 1 2 3) 237 | 1 238 | 2 239 | 2 240 | 3 241 | 3 242 | 4 243 | $ 244 | ``` 245 | 246 | ### Infinite lists 247 | 248 | UNIX has a long tradition of using pipes to form lazy computations, and 249 | bash-lambda is designed to do the same thing. You can generate infinite lists 250 | using `iterate` and `repeatedly`, each of which is roughly equivalent to the 251 | Clojure function of the same name: 252 | 253 | ``` 254 | $ repeatedly $rand_int 100 # notice count is after, not before, function 255 | <100 numbers> 256 | $ iterate $(partial $add 1) 0 257 | 0 258 | 1 259 | 2 260 | 3 261 | ... 262 | $ 263 | ``` 264 | 265 | Note that both `iterate` and `repeatedly` will continue forever, even if you 266 | use `take` (a wrapper for `head`) to select only a few lines. I'm not sure how 267 | to fix this at the moment, but I'd be very happy to accept a fix if anyone has 268 | one. (See `src/list` for these functions) 269 | 270 | ### Searching lists 271 | 272 | Bash-lambda provides `some` and `every` to find list values. These behave like 273 | Clojure's `some` and `every`, but each one returns the element it used to 274 | terminate the loop, if any. 275 | 276 | ``` 277 | $ ls /bin/* | some $(fn '[[ -x $1 ]]') 278 | /bin/bash 279 | $ echo $? 280 | 0 281 | $ ls /etc/* | every $(fn '[[ -d $1 ]]') 282 | /etc/adduser.conf 283 | $ echo $? 284 | 1 285 | $ 286 | ``` 287 | 288 | If `some` or `every` reaches the end of the list, then it outputs nothing and 289 | its only result is its status code. (For `some`, it means nothing was found, so 290 | it returns 1; for `every`, it means they all satisfied the predicate, so it 291 | returns 0.) 292 | 293 | It also gives you the `nth` function, which does exactly what you would expect: 294 | 295 | ``` 296 | $ nth 0 $(list 1 2 3) 297 | 1 298 | $ list 1 2 3 | nth 2 299 | 3 300 | $ 301 | ``` 302 | 303 | ## Closures 304 | 305 | There are two ways you can allocate closures, one of which is true to the usual 306 | Lisp way but is horrendously ugly: 307 | 308 | ``` 309 | $ average=$(fn xs "echo \$((\$($sum_list \$xs) / \$(wc -l < \$xs)))") 310 | $ $average $our_numbers 311 | 14927 312 | ``` 313 | 314 | Here we're closing over the current value of `$sum_list` and emulating 315 | Lisp-style quasiquoting by deliberately escaping everything else. (Well, with a 316 | good bit of bash mixed in.) 317 | 318 | The easier way is to make the 'sum_list' function visible within closures by 319 | giving it a name. While we're at it, let's do the same for `average`. 320 | 321 | ``` 322 | $ def sum-list $sum_list 323 | $ defn average xs 'echo $(($(sum-list $xs) / $(wc -l < $xs)))' 324 | ``` 325 | 326 | Named functions don't need to be dereferenced, since they aren't variables: 327 | 328 | ``` 329 | $ average $our_numbers 330 | 14927 331 | ``` 332 | 333 | Values are just files, so you can save one for later: 334 | 335 | ``` 336 | $ cp $our_numbers the-list 337 | ``` 338 | 339 | ## Atoms and locking 340 | 341 | Atoms are atomic values that use spin or wait locks to provide transactions. 342 | They are used extensively under the hood for things like futures, but in 343 | general you will probably not find much use for them. 344 | 345 | However, if you're curious, you should check out `src/atom` and 346 | `src/semaphore`. 347 | 348 | ### Pipelocks 349 | 350 | This provides an instant lock/release lock that doesn't incur the overhead of a 351 | full spin or delay: 352 | 353 | ``` 354 | $ lock=$(pipelock) 355 | $ ln -s $lock my-pipelock 356 | $ pipelock_grab $lock # blocks until... 357 | 358 | other-terminal$ pipelock_release my-pipelock 359 | other-terminal$ 360 | 361 | # ... we free it using pipelock_release from the other terminal 362 | $ 363 | ``` 364 | 365 | See `src/pipelock` for implementation details. 366 | 367 | ## Futures and remote execution 368 | 369 | Futures are asynchronous processes that you can later force to get their 370 | values. Bash-lambda implements the `future` function, which asynchronously 371 | executes another function and allows you to monitor its status: 372 | 373 | ``` 374 | $ f=$(future $(fn 'sleep 10; echo hi')) 375 | $ future_finished $f || echo waiting 376 | waiting 377 | $ time future_get $f 378 | hi 379 | 380 | real 0m7.707s 381 | user 0m0.016s 382 | sys 0m0.052s 383 | $ 384 | ``` 385 | 386 | ### Futures as memoized results 387 | 388 | When you create a future, standard out is piped into a file that is then 389 | replayed. This file is preserved as long as you have live references to the 390 | future object, so you can replay the command's output at very little cost. For 391 | example: 392 | 393 | ``` 394 | $ f=$(future $(fn 'sleep 10; echo hi')) 395 | $ time get $f # long runtime 396 | hi 397 | 398 | real 0m8.436s 399 | user 0m0.128s 400 | sys 0m0.208s 401 | $ time get $f # short runtime; result is cached 402 | hi 403 | 404 | real 0m0.116s 405 | user 0m0.044s 406 | sys 0m0.060s 407 | $ 408 | ``` 409 | 410 | The exit code is also memoized; however, standard error and other side-effects 411 | are not. There is no way to clear a future after it has finished executing. 412 | 413 | ### Mapping 414 | 415 | You can use the `future_map` function to transform the output of a future when 416 | it completes. You can also use this to trigger alerts. For example: 417 | 418 | ``` 419 | $ f=$(future $(fn 'sleep 10')) 420 | $ future_map $f $(fn 'message done') 421 | $ 422 | ``` 423 | 424 | If you do this, the word 'done' will show up on the right side of the terminal 425 | in a few seconds. 426 | 427 | Command output is usually captured by the future. The only reason `message` 428 | works is that it prints to standard error, which is let through synchronously. 429 | This way you can immediately observe failures in future processes. 430 | 431 | Note that `future_map` is not a multimethod. The reason is that the regular 432 | `map` function needs to be able to process stdin, which has no `ref_type`. As a 433 | result, `map` is not a multimethod, so by extension, `future_map` is not an 434 | implementation of `map`. 435 | 436 | ### Partial results 437 | 438 | You can use `unsafe_get` to retrieve whatever stdout has been produced without 439 | blocking on the process itself. The function isn't unsafe in that it will blow 440 | up or anything, but it may not fully reflect the state of the future (e.g. the 441 | future may have completed by the time `unsafe_get` returns): 442 | 443 | ``` 444 | $ f=$(future $(fn 'sleep 10; echo hi')) 445 | $ unsafe_get $f # returns quickly 446 | $ get $f # takes a while 447 | hi 448 | $ 449 | ``` 450 | 451 | ### Futures and lists 452 | 453 | You can transpose a list of futures into a future of a list using 454 | `future_transpose`: 455 | 456 | ``` 457 | $ f=$(fn 'sleep 10; echo $RANDOM') 458 | $ futures=$(list $(repeatedly $(partial future $f) 10)) 459 | $ single=$(future_transpose $futures) 460 | $ future_finished $single || echo waiting 461 | waiting 462 | $ future_get $single # takes a moment 463 | 21297 464 | 26453 465 | 28753 466 | 23369 467 | 21573 468 | 19249 469 | 25975 470 | 12058 471 | 21774 472 | 469 473 | $ 474 | ``` 475 | 476 | The resulting list is order-preserving. 477 | 478 | Both `future_finished` and `future_get` are implemented as multimethods, so you 479 | can simply use `finished` and `get` instead. 480 | 481 | ### Parallel mapping 482 | 483 | You can apply bounded parallelism to the elements of a list by using 484 | `parallel_map`. This function returns a lazily computed list of futures; if you 485 | need all of the results at once, you can use `future_transpose`. 486 | 487 | The advantage of `parallel_map $f` over `map $(comp future $f)` is that you can 488 | limit the maximum number of running jobs. This lets you have functionality 489 | analogous to `make -jN`, but for general-purpose list mapping. Like `map`, 490 | `parallel_map` is order-preserving. For example: 491 | 492 | ``` 493 | $ compile=$(fn cfile 'gcc $cfile -o ${cfile%.c}') 494 | $ futures=$(list $(ls | parallel_map $compile)) 495 | $ map get $futures # force each one 496 | ... 497 | $ 498 | ``` 499 | 500 | ### Remote execution 501 | 502 | The `remote` function sends a function, along with any heap dependencies it 503 | has, to another machine via SSH and runs it there, piping back the standard 504 | output to the local machine. For example: 505 | 506 | ``` 507 | $ check_disk_space=$(fn 'df -h') 508 | $ remote some-machine $check_disk_space 509 | ... 510 | Filesystem Size Used Avail Use% Mounted on 511 | /dev/sda1 250G 125G 125G 50% / 512 | ... 513 | $ 514 | ``` 515 | 516 | This can be useful in conjunction with futures. 517 | 518 | ## Multimethods 519 | 520 | These aren't quite as cool as the ones in Clojure (and they're a lot slower), 521 | but bash-lambda implements multimethods that allow you to have OO-style 522 | polymorphism. This is based around the idea of a `ref_type`, which is a 523 | filename prefix that gets added to various kinds of objects: 524 | 525 | ``` 526 | $ map ref_type $(list $(list) $(fn 'true') $(pipelock) $(semaphore 5)) 527 | list 528 | fn 529 | pipelock 530 | semaphore 531 | $ 532 | ``` 533 | 534 | Whenever you have a function called `$(ref_type $x)_name`, for instance 535 | `future_get`, you can omit the `future_` part if you're passing an object with 536 | that `ref_type` as the first argument (which you often do). So: 537 | 538 | ``` 539 | $ get $(future $(fn 'echo hi')) 540 | hi 541 | $ future_get $(future $(fn 'echo hi')) 542 | hi 543 | $ get $(atom 100) 544 | 100 545 | $ atom_get $(atom 100) 546 | 100 547 | $ 548 | ``` 549 | 550 | You use `defmulti` to define a new multimethod. Examples of this are in 551 | `src/multi`. 552 | 553 | ## References and garbage collection 554 | 555 | Bash-lambda implements a conservative concurrent mark-sweep garbage collector 556 | that runs automatically if an allocation is made more than 30 seconds since the 557 | last GC run. This prevents the disk from accumulating tons of unused files from 558 | anonymous functions, partials, compositions, etc. 559 | 560 | You can also trigger a synchronous GC by running the `bash_lambda_gc` function: 561 | 562 | ``` 563 | $ bash_lambda_gc 564 | 0 0 565 | $ 566 | ``` 567 | 568 | The two numbers are the number and size of reclaimed objects. If it says `0 0`, 569 | then no garbage was found. 570 | 571 | ### Saving complete references 572 | 573 | You can serialize any anonymous function, composition, partial application, 574 | list, or completed future (you can't serialize a running future for obvious 575 | reasons). To do that, use the `ref_snapshot` function, which returns the 576 | filename of a heap-allocated tar.gz file: 577 | 578 | ``` 579 | $ f=$(fn x 'echo $((x + 1))') 580 | $ g=$(comp $f $f) 581 | $ h=$(comp $g $f) 582 | $ ref_snapshot $h 583 | /tmp/blheap-12328-29f272454ea973c4561b2d1238957b7d0b2c/snapshot_u2C3u3QXO3cRpQ 584 | $ tar -tvf $(ref_snapshot $h) 585 | -rwx------ spencertipping/spencertipping 44 2012-10-11 23:01 /tmp/blheap-12328-29f272454ea973c4561b2d1238957b7d0b2c/fn_mm7fTv75AQZ4lf 586 | -rwx------ spencertipping/spencertipping 174 2012-10-11 23:01 /tmp/blheap-12328-29f272454ea973c4561b2d1238957b7d0b2c/fn_oiKxIgZQwPnmTD 587 | -rwx------ spencertipping/spencertipping 174 2012-10-11 23:01 /tmp/blheap-12328-29f272454ea973c4561b2d1238957b7d0b2c/fn_Uq2O7rtISqbfw3 588 | $ 589 | ``` 590 | 591 | This tar.gz file will be garbage-collected just like any other object. You can 592 | extract a heap snapshot on the same or a different machine by using `tar 593 | -xzPf`, or by using `ref_intern`: 594 | 595 | ``` 596 | $ def f $(comp ...) 597 | $ scp $(ref_snapshot $BASH_LAMBDA_HEAP/f) other-machine:snapshot 598 | $ ssh other-machine 599 | other-machine$ ref_intern snapshot 600 | other-machine$ original-heap-name/f 601 | ``` 602 | 603 | Notice that you'll need to communicate not only the function's data but also 604 | its name; `ref_snapshot` and `ref_intern` are low-level functions that aren't 605 | designed to be used directly for remoting (though they probably do most of the 606 | work). The `remote` function uses `ref_snapshot` under the hood and takes care 607 | of most of the details of running stuff remotely. 608 | 609 | ### Inspecting heap state 610 | 611 | You can see how much memory the heap is using by running `heap_stats`: 612 | 613 | ``` 614 | $ heap_stats 615 | heap size: 120K 616 | objects: 54 617 | permanent: 54 618 | $ 619 | ``` 620 | 621 | You can also inspect the root set and find the immediate references for any 622 | object: 623 | 624 | ``` 625 | $ gc_roots | ref_children 626 | /tmp/blheap-xxxx-xxxxxxxxxxx/gc_roots 627 | /tmp/blheap-xxxx-xxxxxxxxxxx/gc_roots 628 | /tmp/blheap-xxxx-xxxxxxxxxxx/gc_roots 629 | $ add=$(fn x y 'echo $((x + y))') 630 | $ echo $add 631 | /tmp/blheap-xxxx-xxxxxxxxxxx/fn_xxxx_xxxxxxxxx 632 | $ cat $(partial $add 5) | gc_refs 633 | /tmp/blheap-xxxx-xxxxxxxxxxx/fn_xxxx_xxxxxxxxx 634 | $ 635 | ``` 636 | 637 | The output of `ref_children` includes weak references. You can detect weak or 638 | fictitious references by looking for slashes in whatever follows 639 | `$BASH_LAMBDA_HEAP/`: 640 | 641 | ``` 642 | $ is_real_ref=$(fn x '[[ ! ${x##$BASH_LAMBDA_HEAP/} =~ / ]]') 643 | $ $is_real_ref $is_real_ref || echo not real 644 | $ $is_real_ref $(weak_ref $is_real_ref) || echo not real 645 | not real 646 | $ 647 | ``` 648 | 649 | If you need the full transitive closure, you can use `ref_closure`. This 650 | function encapsulates the algorithm used by the GC to find live references. 651 | 652 | ### Pinning objects 653 | 654 | The root set is built from all variables you have declared in your shell and 655 | all running processes. This includes any functions you've written, etc. 656 | However, there may be cases where you need to pin a reference so that it will 657 | never be collected. You can do this using `bash_lambda_gc_pin`, or just 658 | `gc_pin` for short: 659 | 660 | ``` 661 | $ pinned_function=$(gc_pin $(fn x 'echo $x')) 662 | ``` 663 | 664 | You can use an analogous function, unpin, to remove something's pinned status: 665 | 666 | ``` 667 | $ f=$(gc_unpin $pinned_function) 668 | ``` 669 | 670 | (`gc_pin` and `gc_unpin` return the value you give them for convenience, but 671 | side-effectfully change its status within the garbage collector) 672 | 673 | ### Weak references 674 | 675 | You can construct a weak reference to anything in the heap using `weak_ref`: 676 | 677 | ``` 678 | $ f=$(fn x 'echo hi') 679 | $ g=$(weak_ref $f) 680 | $ $f 681 | hi 682 | $ $g 683 | hi 684 | $ unset f 685 | $ $g 686 | hi 687 | $ bash_lambda_gc 688 | 1 36 689 | $ $g 690 | no such file or directory 691 | $ 692 | ``` 693 | 694 | Weak references (and all references, for that matter) can be checked using 695 | bash's `-e` test: 696 | 697 | ``` 698 | $ f=$(fn x 'echo hi') 699 | $ exists=$(fn name '[[ -e $name ]] && echo yes || echo no') 700 | $ $exists $f 701 | yes 702 | $ g=$(weak_ref $f) 703 | $ $exists $g 704 | yes 705 | $ unset f 706 | $ bash_lambda_gc 707 | 1 36 708 | $ $exists $g 709 | no 710 | $ 711 | ``` 712 | 713 | ### Limits of concurrent GC in bash 714 | 715 | Bash-lambda doesn't own its heap and memory space the same way that the JVM 716 | does. As a result, there are a few cases where GC will be inaccurate, causing 717 | objects to be collected when they shouldn't. So far these cases are: 718 | 719 | 1. The window of time between parameter substitution and command invocation. 720 | Allocations made by those parameter substitutions will be live but may be 721 | collected anyway since they are not visible in the process table. 722 | 2. ~~By extension, any commands that have delayed parts: 723 | `sleep 10; map $(fn ...) $xs`. We can't read the memory of the bash 724 | process, so we won't be able to know whether the `$(fn)` is still in the 725 | live set.~~ 726 | This is incorrect. Based on some tests I ran, `$()` expressions inside 727 | delayed commands are evaluated only once the delayed commands are. 728 | Therefore, the only cause of pathological delays would be something like 729 | this: `cat $(fn 'echo hi'; sleep 10)`, which would delay the visibility of 730 | the `$(fn)` form. 731 | 732 | Bash-lambda does its best to work around these problems, but there may still be 733 | edge cases. See `src/gc` for a full discussion of these issues, and please let 734 | me know if you run into bugs in the garbage collector. 735 | -------------------------------------------------------------------------------- /bash-lambda: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # Bash-lambda 3 | 4 | # Source this file to create a function allocation space and enable first-class 5 | # functions in the shell. The heap will be deleted automatically when the shell 6 | # exits. See https://github.com/spencertipping/bash-lambda for documentation. 7 | 8 | #!/bin/bash 9 | # Bash-lambda disk-based heap 10 | 11 | # We need 128 bits of entropy for the heap directory name. This gives us 12 | # guarantees about the soundness of conservative garbage collection ... for the 13 | # rest of the story, see src/gc. 14 | 15 | # 128 bits total / 15 bits per random number = 9 numbers 16 | bash_lambda_genkey() { declare i 17 | for (( i = 0; i < 9; i += 1 )); do 18 | printf '%04x' $RANDOM; done; } 19 | 20 | # Is the heap directory set already? If so, then we don't own it, so we won't 21 | # nuke it when the process ends. 22 | [[ -z "$BASH_LAMBDA_HEAP" ]] && declare -r BASH_LAMBDA_OWN_HEAP=yes 23 | 24 | # Allow the user to override the heap location if they want to. However, this 25 | # may compromise the effectiveness of conservative GC. 26 | export BASH_LAMBDA_KEY=$(bash_lambda_genkey) 27 | export BASH_LAMBDA_HEAP=${BASH_LAMBDA_HEAP:-${TMPDIR:-/tmp}/blheap-$$-$BASH_LAMBDA_KEY} 28 | 29 | if [[ "$BASH_LAMBDA_OWN_HEAP" == yes ]]; then 30 | mkdir -p "$BASH_LAMBDA_HEAP" || return 1 31 | ln -s "$BASH_LAMBDA_HEAP" "$BASH_LAMBDA_HEAP/.weak-references" 32 | date +%s > "$BASH_LAMBDA_HEAP/.last-gc" 33 | fi 34 | 35 | bash_lambda_nuke_heap() { 36 | [[ -e "$BASH_LAMBDA_HEAP/.gc-pid" ]] && kill $(<"$BASH_LAMBDA_HEAP/.gc-pid") 37 | [[ "${BASH_LAMBDA_HEAP:0:5}" == "/tmp/" ]] && rm -rf "$BASH_LAMBDA_HEAP"; } 38 | 39 | [[ "$BASH_LAMBDA_OWN_HEAP" == yes ]] && trap bash_lambda_nuke_heap EXIT 40 | export PATH="$PATH:$BASH_LAMBDA_HEAP" 41 | 42 | # Heap allocation 43 | bash_lambda_gensym() { mktemp "$BASH_LAMBDA_HEAP/${1:-gensym}_XXXXXXXXXXXXXX"; } 44 | bash_lambda_gendir() { mktemp -d "$BASH_LAMBDA_HEAP/${1:-gensym}_XXXXXXXXXXX"; } 45 | 46 | bash_lambda_sym() { bash_lambda_gc_guard "$(bash_lambda_gensym "$@")"; } 47 | bash_lambda_dir() { bash_lambda_gc_guard "$(bash_lambda_gendir "$@")"; } 48 | 49 | bash_lambda_gc_guard() { 50 | declare file=$1 51 | 52 | # If garbage collection is in-progress, mark the object to prevent it from 53 | # being collected this time around. We don't mark it as having been visited, 54 | # however; doing that would cause anything it makes live to be collected 55 | # erroneously. 56 | touch "$BASH_LAMBDA_HEAP/.gc-marked-set/${file##*/}" >& /dev/null 57 | echo "$file"; } 58 | 59 | bash_lambda_cons() { 60 | if [[ $1 == '-n' ]]; then declare file=$BASH_LAMBDA_HEAP/$2 61 | else declare file="$(bash_lambda_gensym $1)"; fi 62 | 63 | cat > $(bash_lambda_gc_guard "$file") && # Take everything from stdin 64 | chmod u+x "$file" && # All conses are executable 65 | echo "$file"; } 66 | #!/bin/bash 67 | # Bash-lambda reference functions 68 | 69 | bash_lambda_ref() { 70 | # Returns a reference to something in the heap. A reference is a function 71 | # that, when run, echoes the thing it refers to. You need to create refs to 72 | # point to heap objects if you can't expose variables (from ~/.bash-lambda, 73 | # for example). 74 | bash_lambda_fn "echo $1"; } 75 | 76 | # A weak reference is a path that resolves the same way as a regular path, but 77 | # that isn't followed by the garbage collector. src/heap sets up a symlink for 78 | # this purpose. 79 | bash_lambda_weak_ref() { 80 | echo "$BASH_LAMBDA_HEAP/.weak-references/${1##$BASH_LAMBDA_HEAP/}"; } 81 | 82 | bash_lambda_ref_resolve() { 83 | if [[ -e "$BASH_LAMBDA_HEAP/$1" ]]; then echo "$BASH_LAMBDA_HEAP/$1" 84 | else echo "$1"; fi; } 85 | 86 | bash_lambda_ref_snapshot() { 87 | tar -czP $(bash_lambda_ref_closure "$1") | bash_lambda_cons snapshot; } 88 | 89 | bash_lambda_ref_intern() { cat ${1:--} | tar -xzP; } 90 | 91 | bash_lambda_ref_closure() { 92 | declare visited=$(bash_lambda_dir) object=$(bash_lambda_ref_resolve "$1") 93 | bash_lambda_ref_visit "$visited" "$object" 94 | ls -d "$visited"/* | (declare x; while read x; do 95 | echo "$BASH_LAMBDA_HEAP/${x##$visited/}"; done); } 96 | 97 | bash_lambda_ref_type() { declare base=${1##*/}; echo "${base%%_*}"; } 98 | 99 | bash_lambda_ref_visit() { 100 | # The ref in question must be an object that exists in the heap. We expect it 101 | # to be a full pathname, though the object itself should be a direct child of 102 | # the heap directory. 103 | declare directory=$1 ref=$2 104 | declare ref_name="${ref#$BASH_LAMBDA_HEAP/}" 105 | 106 | # Is this something that belongs to the GC? 107 | [[ "$ref_name" == ".gc-marked-set" || 108 | "$ref_name" == ".gc-visited-set" ]] && return 0 109 | 110 | # No need to mark an object in a subdirectory. It isn't a direct child of the 111 | # heap, so its storage is already being managed by the directory it belongs 112 | # to. 113 | if [[ ! ("$ref_name" =~ /) ]]; then 114 | # Have we already visited this object? If so, no need to revisit it. 115 | [[ -e "$directory/$ref_name" ]] && return 0 116 | 117 | # Otherwise, mark the object and everything it points to. 118 | touch "$directory/$ref_name"; fi 119 | 120 | bash_lambda_ref_read "$ref" | bash_lambda_ref_children | (declare x 121 | while read x; do bash_lambda_ref_visit "$directory" "$x"; done); } 122 | 123 | bash_lambda_ref_read() { 124 | # Read the contents of the given object. The result will be an output stream 125 | # suitable for consumption by bash_lambda_ref_children. 126 | if [[ -d "$1" ]]; then [[ -e "$1"/* ]] && ls -d "$1"/* 127 | elif [[ -L "$1" ]]; then readlink "$1" 128 | elif [[ -p "$1" ]]; then return 0 129 | elif [[ -e "$1" ]]; then cat "$1"; fi; } 130 | 131 | bash_lambda_ref_children() { 132 | # Locate occurrences of the heap directory. This name contains 128 bits of 133 | # pseudorandom entropy, so we are unlikely to see it spuriously referenced. 134 | # If we observe a path that exists, then we consider that to be a reference 135 | # for GC purposes. 136 | 137 | egrep -o "$BASH_LAMBDA_HEAP/[^ [:space:]/\)\}\"']+" | (declare ref 138 | while read ref; do [[ -e "$ref" ]] && echo "$ref"; done); } 139 | #!/bin/bash 140 | # Bash-lambda concurrent mark-sweep garbage collector 141 | 142 | # BASH_LAMBDA_GC_SECONDS=0 will disable automatic GC 143 | # BASH_LAMBDA_GC_CONCURRENT=0 will cause GC to be synchronous 144 | export BASH_LAMBDA_GC_SECONDS=${BASH_LAMBDA_GC_SECONDS:-30} 145 | export BASH_LAMBDA_GC_CONCURRENT=${BASH_LAMBDA_GC_CONCURRENT:-1} 146 | 147 | mkdir -p "$BASH_LAMBDA_HEAP"/.gc-permanent 148 | 149 | # Can we see this unexported variable from the garbage collector? If not, then 150 | # we don't have access to the full root set. If you unset this, GC will refuse 151 | # to run. We need this to be a weak ref so the exported variable doesn't show 152 | # up in the root set by itself. These two variables should be identical modulo 153 | # path. 154 | BASH_LAMBDA_GC_CANARY=$(bash_lambda_cons < /dev/null) 155 | export BASH_LAMBDA_GC_KEY=$(bash_lambda_weak_ref $BASH_LAMBDA_GC_CANARY) 156 | 157 | # This function can be run asynchronously: bash_lambda_gc &. See 158 | # bash_lambda_auto_gc for an example. It knows which heap to run on, and it 159 | # echoes information about the number of objects and bytes reclaimed. 160 | bash_lambda_gc() { 161 | # Fix a bug wherein the asynchronous GC can run with a PWD on another 162 | # filesystem, preventing that filesystem from being unmounted while the GC is 163 | # happening (despite the fact that the GC itself doesn't use the PWD). 164 | cd 165 | 166 | # Avert disaster: never GC a heap unless it is in /tmp. 167 | if [[ "${BASH_LAMBDA_HEAP:0:5}" != "/tmp/" ]]; then 168 | echo 'bash_lambda_gc refuses to run because $BASH_LAMBDA_HEAP is set to' 169 | echo 'a directory outside of /tmp/. If you really want to do this, you' 170 | echo 'should do something like /tmp/../wherever.' 171 | return 2; fi 172 | 173 | # Try to acquire the GC lock. If we can't, another GC is probably running; so 174 | # we exit immediately since GC is (ideally) idempotent. 175 | mkdir "$BASH_LAMBDA_HEAP"/.gc-visited-set || return 1 176 | mkdir -p "$BASH_LAMBDA_HEAP"/.gc-marked-set 177 | 178 | echo $$ > "$BASH_LAMBDA_HEAP"/.gc-pid 179 | 180 | # We have the lock. At this point the heap allocator will mark any new 181 | # objects that are created, so we can use the root set that exists right now 182 | # and start marking objects referenced by that root set. 183 | bash_lambda_gc_roots | bash_lambda_ref_children | (declare x; while read x; do 184 | bash_lambda_ref_visit "$BASH_LAMBDA_HEAP/.gc-visited-set" "$x"; done) 185 | 186 | # We should mark the canary. If we haven't done this, then something is wrong 187 | # with the root-set computation. 188 | if [[ ! -e "$BASH_LAMBDA_HEAP/.gc-visited-set/${BASH_LAMBDA_GC_KEY##*/}" ]] 189 | then 190 | echo 'bash_lambda_gc cannot see the full root set. You should make sure' 191 | echo 'it is being run as a function, not as an indirect script; try using' 192 | echo 'bash_lambda_gc instead of just gc.' 193 | rm -r "$BASH_LAMBDA_HEAP/".gc-{visited,marked}-set 194 | return 1; fi 195 | 196 | # Now nuke anything that isn't either marked or permanent. 197 | ls -d "$BASH_LAMBDA_HEAP"/* | ( 198 | declare ref reclaimed=() reclaimed_size=0 199 | while read ref; do 200 | if [[ ! -e "$BASH_LAMBDA_HEAP/.gc-marked-set/${ref##*/}" && 201 | ! -e "$BASH_LAMBDA_HEAP/.gc-visited-set/${ref##*/}" && 202 | ! -e "$BASH_LAMBDA_HEAP/.gc-permanent/${ref##*/}" ]]; then 203 | reclaimed+=("$ref") 204 | reclaimed_size=$((reclaimed_size + $(du -sb "$ref" | cut -f 1))) 205 | rm -rf "$ref"; fi; done 206 | echo "${#reclaimed[@]} $reclaimed_size") 207 | 208 | # We are now done. Remove the marked-set directory and echo some stats about 209 | # the stuff we collected. 210 | rm -rf "$BASH_LAMBDA_HEAP/".gc-{marked,visited}-set; } 211 | 212 | bash_lambda_auto_gc() { 213 | # Trigger a concurrent GC if it's been more than some number of seconds since 214 | # the last one. This turns out to be one of the few constant-time ways we can 215 | # do this safely. 216 | if [[ ! -e "$BASH_LAMBDA_HEAP/.last-gc" ]] || \ 217 | (( $BASH_LAMBDA_GC_SECONDS && 218 | $(date +%s) - $(<"$BASH_LAMBDA_HEAP/.last-gc") > 219 | $BASH_LAMBDA_GC_SECONDS )); then 220 | date +%s > "$BASH_LAMBDA_HEAP/.last-gc" 221 | 222 | # NOTE: Running GC concurrently is experimental and is known to have some 223 | # problems unless done carefully. See comments in bash_lambda_gc for 224 | # details. 225 | if (( $BASH_LAMBDA_GC_CONCURRENT )); then 226 | (bash_lambda_gc --concurrent >> "$BASH_LAMBDA_HEAP/.gc-log" &) 227 | else 228 | bash_lambda_gc >> "$BASH_LAMBDA_HEAP/.gc-log" 229 | fi; fi; } 230 | 231 | bash_lambda_heap_stats() { 232 | printf '%-20s %s\n%-20s %d\n%-20s %d\n' \ 233 | 'heap size:' $(du -sh "$BASH_LAMBDA_HEAP" | cut -f 1) \ 234 | 'objects:' $(ls "$BASH_LAMBDA_HEAP" | wc -l) \ 235 | 'permanent:' $(ls "$BASH_LAMBDA_HEAP/.gc-permanent" | wc -l); } 236 | 237 | bash_lambda_heap_ls() { cd "$BASH_LAMBDA_HEAP"; ls "$@"; } 238 | 239 | bash_lambda_gc_pin() { touch "$BASH_LAMBDA_HEAP/.gc-permanent/${1##*/}"; 240 | echo "$1"; } 241 | 242 | bash_lambda_gc_unpin() { rm -f "$BASH_LAMBDA_HEAP/.gc-permanent/${1##*/}"; 243 | echo "$1"; } 244 | 245 | bash_lambda_gc_roots() { 246 | declare; ps ax; ls -d "$BASH_LAMBDA_HEAP/.gc-permanent"/* | (declare x 247 | while read x; do echo "$BASH_LAMBDA_HEAP/${x##*/}"; done); } 248 | #!/bin/bash 249 | # Bash-lambda function and closure allocation 250 | 251 | bash_lambda_fn_body() { 252 | echo '#!/bin/bash' 253 | declare i 254 | for (( i = 1; i <= $#; i += 1 )); do 255 | if (( $i < $# )); then echo "declare -r ${!i}=\$$i" 256 | else echo "${!i}"; fi 257 | done; } 258 | 259 | bash_lambda_fn() { 260 | # Yup, we're allocating closures by writing to files and returning their 261 | # names to the callers. This gives you controllable persistence, heap 262 | # allocation, and the ability to reference the same closures across multiple 263 | # processes. 264 | bash_lambda_fn_body "$@" | bash_lambda_cons fn; } 265 | 266 | bash_lambda_cons_fn() { 267 | # Same as bash_lambda_fn, but body is specified from stdin. Useful for 268 | # multiline functions when used with heredocs. 269 | (bash_lambda_fn_body "$@" ''; cat -) | bash_lambda_cons fn; } 270 | 271 | bash_lambda_defn() { declare name=$1; shift 272 | bash_lambda_gc_pin \ 273 | $(bash_lambda_fn_body "$@" | bash_lambda_cons -n $name); } 274 | 275 | # Exports functions into named files in the heap. This allows them to reference 276 | # each other from inside heap-allocated closures. Any exported functions are 277 | # pinned so that they will never be garbage-collected. 278 | bash_lambda_extern() { 279 | bash_lambda_gc_pin $( (echo "#!/bin/bash" 280 | declare -f "$1" 281 | echo "$1 \"\$@\"") | bash_lambda_cons -n $1); } 282 | 283 | bash_lambda_def() { rm -f $BASH_LAMBDA_HEAP/$1 284 | ln -s $2 $(bash_lambda_gc_pin $BASH_LAMBDA_HEAP/$1); } 285 | 286 | bash_lambda_defalias() { 287 | declare name=$1; shift 288 | bash_lambda_def "$name" $(bash_lambda_cons_fn <<<"$*"); } 289 | #!/bin/bash 290 | # Bash-lambda functional programming constructs 291 | 292 | # $(comp $f $g $h) x = f $(g $(h x)) 293 | # $(partial $f x) y = f x y 294 | bash_lambda_comp() { 295 | declare i body 296 | for (( i = $#; i >= 1; i -= 1 )); do 297 | if (( $i == $# )); then body="\$(${!i} \"\$@\")"; 298 | else body="\$(${!i} $body)"; fi 299 | done 300 | bash_lambda_fn "echo $body"; } 301 | 302 | bash_lambda_partial() { 303 | bash_lambda_fn "$* \"\$@\""; } 304 | #!/bin/bash 305 | # Bash-lambda multimethods 306 | 307 | # These work only on fully-named parameters, not on stuff coming from stdin. If 308 | # we can observe the name, then we can extract the type from the beginning of 309 | # the filename. For example: 310 | # 311 | # $ future $f 312 | # /tmp/blheap-xxxx-xxxx/future_xxxxxxxx 313 | # $ ref_type $(future $f) 314 | # future 315 | # $ 316 | # 317 | # We then prepend this to the multimethod name to get the specific function 318 | # name: 319 | # 320 | # $ defmulti get 321 | # $ get $(future $f) -> future_get $(future $f) 322 | 323 | bash_lambda_defmulti() { 324 | declare multi_name=$1 325 | bash_lambda_defn $1 '$(bash_lambda_ref_type $1)_'$multi_name' "$@"'; } 326 | 327 | # Multimethod definitions for bash-lambda functions 328 | # state finished block notify src/future 329 | # get src/future src/atom 330 | # unsafe_get src/atom 331 | # count src/list src/semaphore 332 | # grab release wrap src/semaphore src/atom src/pipelock 333 | 334 | (declare method 335 | for method in state finished block notify \ 336 | get \ 337 | unsafe_get \ 338 | count \ 339 | grab release wrap; do 340 | bash_lambda_defmulti $method; done) > /dev/null 341 | #!/bin/bash 342 | # Bash-lambda pipe locks 343 | 344 | # A pipe-lock is a way to block one process until another one lets it through. 345 | # This provides a way to block without polling, and is used internally by 346 | # futures. 347 | 348 | # WARNING 349 | # If you hand a pipelock to someone, you MUST block on it. If you don't, then 350 | # whoever attempts to unlock the pipelock will block, and this will result in 351 | # all kinds of problems and strange bugs. 352 | 353 | bash_lambda_pipelock() { 354 | declare file=$(bash_lambda_sym pipelock) 355 | rm $file && mkfifo $file 356 | echo $file; } 357 | 358 | bash_lambda_pipelock_grab() { cat $1 >& /dev/null; } 359 | bash_lambda_pipelock_release() { echo > $1; } 360 | #!/bin/bash 361 | # Bash-lambda semaphores and mutexes 362 | 363 | # Semaphores are directories that contain empty numbered subdirectories along 364 | # with an immutable file containing the capacity of the semaphore. Locking is 365 | # done by attempting to create one of these empty subdirectories, and ls -d | 366 | # wc -l is used to get the number of used entries. 367 | 368 | bash_lambda_semaphore() { 369 | declare capacity=$1 semaphore=$(bash_lambda_dir semaphore) 370 | echo $capacity > "$semaphore/capacity" 371 | echo "$semaphore"; } 372 | 373 | bash_lambda_semaphore_grab() { 374 | # Take an item from the semaphore; returns the item's identifier (a path, but 375 | # you don't need to know this) and exit code 0 if successful, returns 1 and 376 | # outputs nothing if unsuccessful. 377 | declare semaphore=$1 i capacity=$(<"$1/capacity") 378 | for (( i = 0; i < $capacity; i += 1 )); do 379 | if mkdir "$semaphore/semaphore_$i" >& /dev/null; then 380 | echo "$semaphore/semaphore_$i"; return 0; fi; done 381 | return 1; } 382 | 383 | bash_lambda_semaphore_release() { 384 | # Releases an item from the semaphore. You can just pass the item here; no 385 | # need to pass the semaphore also. 386 | rmdir "$1"; } 387 | 388 | bash_lambda_semaphore_count() { 389 | # Counts the number of items available in the semaphore. 390 | declare semaphore=$1 391 | echo $(( $(<"$semaphore/capacity") + 1 - $(ls -d "$semaphore/"* | wc -l) )); } 392 | 393 | bash_lambda_semaphore_wrap() { 394 | # Wraps a function's execution with a grab/release of a semaphore item. 395 | declare s=$1 f=$2 396 | bash_lambda_fn "declare lock 397 | if lock=\$(bash_lambda_semaphore_grab $s); then 398 | $f \"\$@\" 399 | declare status=\$? 400 | bash_lambda_semaphore_release \$lock 401 | exit \$status; fi"; } 402 | 403 | # Mutex: a more efficient way to do a single-element semaphore. 404 | bash_lambda_mutex() { bash_lambda_dir mutex; } 405 | bash_lambda_mutex_grab() { mkdir "$1/lock" >& /dev/null && echo "$1/lock"; } 406 | bash_lambda_mutex_release() { rmdir "$1"; } 407 | bash_lambda_mutex_count() { [[ -d "$1/lock" ]]; echo $?; } 408 | 409 | bash_lambda_mutex_wrap() { 410 | declare m=$1 f=$2 411 | bash_lambda_fn "declare lock 412 | if lock=\$(bash_lambda_mutex_grab $m); then 413 | $f \"\$@\" 414 | declare status=\$? 415 | bash_lambda_mutex_release \$lock 416 | exit \$status; fi"; } 417 | #!/bin/bash 418 | # Bash-lambda atomic values 419 | 420 | # An atomic value supports thread-safe assignment and access. It does this by 421 | # guarding every read and write with a semaphore. You can also request a 422 | # transaction against the current value of an atomic value, so that during that 423 | # transaction any requests to change the value will block or fail. 424 | 425 | bash_lambda_atom() { 426 | printf "%s\n%s" "$(bash_lambda_mutex)" \ 427 | "$(echo "$1" | bash_lambda_cons)" | bash_lambda_cons atom; } 428 | 429 | bash_lambda_atom_grab() { 430 | bash_lambda_mutex_grab "$(bash_lambda_nth 0 "$1")"; } 431 | 432 | bash_lambda_atom_release() { bash_lambda_mutex_release "$1"; } 433 | bash_lambda_atom_get() { $(bash_lambda_atom_wrap "$1" cat); } 434 | 435 | # Unsafe-get is considerably faster than get, but does not respect the 436 | # atomicity of running transactions. Use this only when you have a single state 437 | # transition acting on the atom. (See src/future for an example) 438 | bash_lambda_atom_unsafe_get() { cat "$(bash_lambda_nth 1 "$1")"; } 439 | 440 | bash_lambda_atom_wrap() { 441 | # A generalized transaction/set function. $(wrap $atom $f) does one of two 442 | # things. If it acquires the atom's lock, then it invokes $f on a file 443 | # containing the atom's current value. $f is free to modify the contents of 444 | # this file. 445 | 446 | # You will probably want to use one of the functions in src/wait instead of 447 | # calling wrap directly. 448 | 449 | declare atom=$1 f=$2 450 | bash_lambda_fn "declare lock 451 | if lock=\$(bash_lambda_atom_grab $atom); then 452 | $f \"\$(bash_lambda_nth 1 $atom)\" 453 | declare status=\$? 454 | bash_lambda_atom_release \$lock 455 | exit \$status; fi"; } 456 | #!/bin/bash 457 | # Bash-lambda list programming constructs 458 | 459 | bash_lambda_list() { declare x 460 | for x; do echo "$x"; done | bash_lambda_cons list; } 461 | 462 | bash_lambda_nth() { cat ${2:--} | head -n$(($1 + 1)) | tail -n1; } 463 | 464 | bash_lambda_take() { cat ${2:--} | head -n$1; } 465 | bash_lambda_drop() { cat ${2:--} | tail -n+$(($1 + 1)); } 466 | 467 | # Here, 'map' works as both map and mapcat since cons and append are the same 468 | # operation. This arises due to the associativity of cons. 469 | bash_lambda_map() { 470 | cat ${2:--} | (declare x; while read x; do $1 "$x"; done); } 471 | 472 | bash_lambda_reduce() { 473 | declare f=$1 x=$2 474 | cat ${3:--} | (declare y; while read y; do x="$($f "$x" "$y")"; done 475 | echo "$x"); } 476 | 477 | bash_lambda_reductions() { 478 | declare f=$1 x=$2 479 | cat ${3:--} | (declare y; while read y; do x="$($f "$x" "$y")"; echo "$x" 480 | done); } 481 | 482 | bash_lambda_filter() { 483 | cat ${2:--} | (declare x; while read x; do 484 | $1 "$x" > /dev/null && echo "$x"; done); } 485 | 486 | bash_lambda_partition() { 487 | cat ${2:--} | split -l $1 -u --filter='bash_lambda_cons list'; } 488 | 489 | # This is a multimethod that is normally invoked as just 'count'; e.g: 490 | # $ count $(list 1 2 3) 491 | # If you want to count lines from a stream, you should use wc -l instead. 492 | bash_lambda_list_count() { wc -l < "$1"; } 493 | 494 | # List generators 495 | bash_lambda_iterate() { 496 | declare x=$2 497 | echo "$x"; while x="$($1 "$x")"; do echo "$x"; done; } 498 | 499 | bash_lambda_repeatedly() { 500 | declare i f=$1 n=$2 501 | for (( i = 0; i != ${n:--1}; i += 1 )); do $f || return $?; done; } 502 | 503 | # Tests over lists 504 | bash_lambda_some() { 505 | cat ${2:--} | (declare x; while read x; do 506 | if $1 "$x" > /dev/null; then 507 | echo "$x"; return 0; fi; done; return 1); } 508 | 509 | bash_lambda_every() { 510 | cat ${2:--} | (declare x; while read x; do 511 | if ! $1 "$x" > /dev/null; then 512 | echo "$x"; return 1; fi; done; return 0); } 513 | #!/bin/bash 514 | # Bash-lambda futures (asynchronous processes) 515 | 516 | # Future locking 517 | # It is surprisingly difficult to reliably wait for a future. We can't use the 518 | # 'wait' command because the future could belong to another subshell. We can't 519 | # use the process table or /proc because PIDs get reused. This leaves us with 520 | # two options: we can use polling against some lock file, or we can devise some 521 | # scheme with pipelocks. 522 | # 523 | # It turns out that we can use a semaphore to do everything we need. 524 | 525 | bash_lambda_future() { 526 | # Construct a future around the given function application. To do this, we 527 | # run the function in the background and pipe its output to a temporary file. 528 | # We also record its exit code. 529 | 530 | declare output=$(bash_lambda_cons future_output < /dev/null) 531 | declare status=$(bash_lambda_cons future_status < /dev/null) 532 | declare state=$(bash_lambda_atom running) 533 | declare result=$(printf $"%s\n%s\n%s\n" $output $status $state | \ 534 | bash_lambda_cons future) 535 | 536 | ("$@" > $output; echo $? > $status; notify $result) > /dev/null & 537 | echo $result; } 538 | 539 | bash_lambda_future_finished() { 540 | # Exits with 0 if the future is finished, 1 if still running. If 0, a 541 | # future_get call will block only on IO, but not on job completion. This 542 | # operation can use unsafe_get because futures never un-finish. 543 | [[ "$(bash_lambda_atom_unsafe_get "$(bash_lambda_nth 2 $1)")" == done ]]; } 544 | 545 | bash_lambda_future_state() { 546 | bash_lambda_spin_wrap $(bash_lambda_nth 2 $1) cat; } 547 | 548 | bash_lambda_future_block() { 549 | # Block on completion of the future. Spin-locks against the future's state. 550 | declare pipelock=$(bash_lambda_spin_wrap $(bash_lambda_nth 2 $1) \ 551 | $(fn x "[[ \"\$(<\$x)\" == done ]] || bash_lambda_pipelock | tee -a \$x")) 552 | [[ -z "$pipelock" ]] || bash_lambda_pipelock_grab $pipelock; } 553 | 554 | bash_lambda_future_notify() { 555 | # Notify all listeners that this future is done. This amounts to unblocking 556 | # all of the pipelocks that have been appended to the state. 557 | bash_lambda_spin_wrap $(bash_lambda_nth 2 $1) \ 558 | $(fn x '[[ "$(<$x)" == done ]] || drop 1 $x | bash_lambda_map release 559 | echo done > $x'); } 560 | 561 | bash_lambda_future_get() { 562 | # This function blocks on the future's process if it is still running, and 563 | # its stdout and exit code are proxied. 564 | bash_lambda_future_finished "$1" || bash_lambda_future_block "$1" 565 | cat "$(bash_lambda_nth 0 "$1")" 566 | return "$(< "$(bash_lambda_nth 1 "$1")")"; } 567 | 568 | bash_lambda_future_map() { 569 | # Returns a future of a function applied to this future's value. 570 | bash_lambda_future $(fn "$2 \$(future_get $1)"); } 571 | 572 | bash_lambda_future_unsafe_get() { 573 | # Gets whatever stdout has been produced so far. The process may not have 574 | # exited, so this function returns 0. 575 | cat "$(bash_lambda_nth 0 "$1")"; } 576 | 577 | bash_lambda_future_transpose() { 578 | bash_lambda_future $(bash_lambda_partial \ 579 | bash_lambda_map bash_lambda_future_get $1); } 580 | #!/bin/bash 581 | # Bash-lambda remote functions 582 | 583 | # Remote functions allow you to transparently migrate the execution of a 584 | # function to another machine. Any relevant heap state will be transferred 585 | # automatically. Remote functions use passwordless SSH and their stdin/stdout 586 | # and exit codes are transparently proxied. 587 | 588 | # Note that remote functions are run from the current $PWD, except that any 589 | # changes to $HOME are taken into account. For instance, if I'm spencertipping 590 | # on one computer and spencer on another, and I run this from ~/bin, then: 591 | # 592 | # $ remote the-spencertipping-machine $(fn 'echo $PWD') 593 | # /home/spencertipping/bin 594 | # $ remote the-spencer-machine $(fn 'echo $PWD') 595 | # /home/spencer/bin 596 | # $ 597 | 598 | # If the remote command produces a nonzero exit code, then the heap snapshot is 599 | # preserved to help you find out what went wrong. You can get rid of old 600 | # snapshots on remote systems using remote_clean. 601 | 602 | bash_lambda_remote() { 603 | declare host=$1 f=$2 cd_into=${PWD#$HOME/} 604 | cd_into=${cd_into#$HOME} # in case no trailing slash 605 | declare snapshot=$(bash_lambda_ref_snapshot $f) 606 | 607 | declare path="export PATH=\"\$PATH:$BASH_LAMBDA_HEAP\"" 608 | declare pre="cd $cd_into && tar xzP" 609 | declare clean="rm -rf \"$BASH_LAMBDA_HEAP\"" 610 | 611 | ssh $host "$pre && $path && $f && $clean" < $snapshot; } 612 | 613 | bash_lambda_remote_clean() { 614 | # Removes this instance's heap snapshot from the remote instance. 615 | ssh $host "rm -rf \"$BASH_LAMBDA_HEAP\""; } 616 | #!/bin/bash 617 | # Bash-lambda wait functions 618 | 619 | # Various ways to wait for things. Usually you would use this with semaphores, 620 | # but you could do it with any function. 621 | 622 | bash_lambda_poll_constantly() { echo $1; } 623 | bash_lambda_poll_linearly() { bash_lambda_fn "echo \$((\$1 + ${1:-1}))"; } 624 | bash_lambda_poll_exponentially() { bash_lambda_fn "echo \$((\$1 * ${1:-2}))"; } 625 | 626 | bash_lambda_waiting_fn() { 627 | # Returns a thunk that waits for the given function to return true, polling 628 | # constantly by default (you can change this by passing a polling interval 629 | # function as the third argument). Echoes all output from the function. The 630 | # second argument specifies the initial polling duration. 631 | declare f=$1 delay=${2:-1} adjust=${3:-bash_lambda_poll_constantly} 632 | bash_lambda_fn "declare delay=$delay 633 | until $f \"\$@\"; do sleep \$delay 634 | delay=\$($adjust \$delay); done"; } 635 | 636 | bash_lambda_wait_until() { $(bash_lambda_waiting_fn "$@"); } 637 | 638 | bash_lambda_spinning_fn() { 639 | # Returns a thunk that spins until the given function returns true. This 640 | # should be used only for very quick waits. 641 | declare f=$1 642 | bash_lambda_fn "until $f \"\$@\"; do :; done"; } 643 | 644 | bash_lambda_spin_until() { $(bash_lambda_spinning_fn "$@"); } 645 | 646 | bash_lambda_wait_wrap() { $(wait_until $(bash_lambda_partial wrap "$@")); } 647 | bash_lambda_spin_wrap() { $(spin_until $(bash_lambda_partial wrap "$@")); } 648 | #!/bin/bash 649 | # Bash-lambda parallel execution 650 | 651 | # Parallel tasks are defined as an interface that transposes parallelism over 652 | # lists. They generally return futures. You should set the 653 | # BASH_LAMBDA_PARALLELISM value to something more specific to your own system, 654 | # depending on available CPUs and memory. 655 | # 656 | # There is a considerable degree of overhead (on the order of seconds) 657 | # associated with using these parallel methods. 658 | 659 | export BASH_LAMBDA_PARALLELISM=4 660 | 661 | bash_lambda_parallel_map() { 662 | # Maps a function over a list in parallel. Results retain their original 663 | # order. An optional third parameter allows you to override the default level 664 | # of parallelism. 665 | 666 | # This function returns a list of futures, each of which represents an 667 | # intermediate outcome. The futures are generated lazily, so in theory you 668 | # could have an infinite source list and, so long as you requested only a 669 | # finite number of elements, you would be ok. (In practice, I think this is a 670 | # dangerous strategy.) To get the completed results for a segment of the 671 | # list: 672 | # 673 | # $ get $(transpose $(list $(parallel_map $f $xs | take 5))) 674 | 675 | # Note: If your function returns a nonzero exit code, it will be run again on 676 | # the same data. 677 | 678 | declare f=$1 xs=$2 n=${3:-$BASH_LAMBDA_PARALLELISM} 679 | declare s=$(bash_lambda_semaphore $n) 680 | 681 | declare blocking_f=$(bash_lambda_waiting_fn \ 682 | $(bash_lambda_semaphore_wrap $s $f)) 683 | 684 | bash_lambda_map $(bash_lambda_partial bash_lambda_future $blocking_f) $xs; } 685 | #!/bin/bash 686 | # Bash-lambda rc file functions 687 | 688 | export BASH_LAMBDA_RC=${BASH_LAMBDA_RC:-$HOME/.bash-lambda} 689 | export BASH_LAMBDA_EDITOR=${BASH_LAMBDA_EDITOR:-${EDITOR:-$VISUAL}} 690 | 691 | export COLUMNS # so that bash_lambda_message can be called from subshells 692 | 693 | bash_lambda_message() { 694 | declare m="$*" 695 | (( $COLUMNS )) && echo -en "\033[s\033[$((COLUMNS - ${#m}))G$m\033[u" 1>&2; } 696 | 697 | bash_lambda_setup_rc() { 698 | [[ -e "$BASH_LAMBDA_RC" ]] || sed 's/^ //' > "$BASH_LAMBDA_RC" <<'EOF' 699 | #!/bin/bash 700 | # You can put function defs here. Variables you define here aren't visible, 701 | # since this file is always evaluated (usually asynchronously) from inside a 702 | # subshell. 703 | # 704 | # This file is sourced asynchronously when you start your shell, so adding 705 | # definitions won't increase the amount of time required to open a new 706 | # terminal. 707 | # 708 | # See https://github.com/spencertipping/bash-lambda for details about 709 | # defining functions. 710 | # 711 | # For example: 712 | 713 | # File tests (wrappers for -d, -x, etc) 714 | def bash_tests $(ref $(list a b c d e f g h k p r s t u w x G L N O S z n)) 715 | defn deffiletest x 'defn is-$x f "[[ -$x \$f ]]"' 716 | map deffiletest $(bash_tests) 717 | 718 | defn newerthan file 'bash_lambda_fn f "[[ \$f -nt $file ]]"' 719 | defn olderthan file 'bash_lambda_fn f "[[ \$f -ot $file ]]"' 720 | defn eq file 'bash_lambda_fn f "[[ \$f -ef $file ]]"' 721 | 722 | # Content tests 723 | defn contains pattern 'egrep -o $pattern' 724 | defn without pattern 'sed "s/${pattern/\//\\\/}//g"' 725 | EOF 726 | } 727 | 728 | bash_lambda_reload_rc() { 729 | [[ -e "$BASH_LAMBDA_RC" ]] && (. "$BASH_LAMBDA_RC" > /dev/null); } 730 | 731 | bash_lambda_defs() { 732 | bash_lambda_setup_rc 733 | $BASH_LAMBDA_EDITOR "$BASH_LAMBDA_RC" 734 | (bash_lambda_reload_rc &); } 735 | #!/bin/bash 736 | # Bash-lambda function exporting and GC hooks 737 | 738 | # Export the bash_lambda library into the current heap 739 | bash_lambda_init() { 740 | declare -f | grep '^bash_lambda' | sed 's/ .*//' | 741 | (declare fn; while read fn; do 742 | bash_lambda_extern $fn > /dev/null 743 | if [[ -z "$BASH_LAMBDA_NO_ALIASES" ]]; then 744 | bash_lambda_def ${fn##bash_lambda_} $fn; fi; done 745 | 746 | bash_lambda_reload_rc 747 | bash_lambda_message 'λ') & } 748 | 749 | # Initialize the heap only if we own it. 750 | [[ "$BASH_LAMBDA_OWN_HEAP" == yes ]] && (bash_lambda_init) 751 | 752 | # Run a GC, if necessary, after each command 753 | export PROMPT_COMMAND="${PROMPT_COMMAND:-:}; bash_lambda_auto_gc" 754 | --------------------------------------------------------------------------------