├── .gitignore ├── LICENCE ├── Makefile ├── README.md ├── doc └── overview.edoc ├── examples ├── fib.erl ├── ifl.erl ├── montecarlo.erl ├── queens.erl ├── reduce.erl └── sumEuler.erl ├── include └── skel.hrl ├── priv └── default.args ├── rebar ├── rebar.config ├── src ├── gen_sk.erl ├── sk_assembler.erl ├── sk_assembler_sequential.erl ├── sk_cluster.erl ├── sk_cluster_decomp.erl ├── sk_cluster_recomp.erl ├── sk_data.erl ├── sk_farm.erl ├── sk_farm_collector.erl ├── sk_farm_emitter.erl ├── sk_feedback.erl ├── sk_feedback_bicounter.erl ├── sk_feedback_filter.erl ├── sk_feedback_receiver.erl ├── sk_map.erl ├── sk_map_combiner.erl ├── sk_map_partitioner.erl ├── sk_ord.erl ├── sk_ord_reorderer.erl ├── sk_ord_tagger.erl ├── sk_pipe.erl ├── sk_profile.erl ├── sk_reduce.erl ├── sk_reduce_decomp.erl ├── sk_reduce_reducer.erl ├── sk_seq.erl ├── sk_sink.erl ├── sk_source.erl ├── sk_tracer.erl ├── sk_utils.erl ├── skel.app.src └── skel.erl └── tutorial ├── bin └── style.css └── src ├── cluster.png ├── diagrams.graffle ├── farm.graffle ├── farm.png ├── feedback.graffle ├── feedback.png ├── img_merge.png ├── index.md ├── map.png ├── pipe.png ├── speedup.png ├── style.css └── tutorial.md /.gitignore: -------------------------------------------------------------------------------- 1 | erl_crash.dump 2 | *.beam 3 | ebin 4 | priv/*.args 5 | results 6 | *.log 7 | .otp.plt 8 | .skel.plt -------------------------------------------------------------------------------- /LICENCE: -------------------------------------------------------------------------------- 1 | Copyright (c) 2012, University of St Andrews 2 | All rights reserved. 3 | 4 | Redistribution and use in source and binary forms, with or without 5 | modification, are permitted provided that the following conditions are met: 6 | * Redistributions of source code must retain the above copyright 7 | notice, this list of conditions and the following disclaimer. 8 | * Redistributions in binary form must reproduce the above copyright 9 | notice, this list of conditions and the following disclaimer in the 10 | documentation and/or other materials provided with the distribution. 11 | * Neither the name of the University of St Andrews nor the 12 | names of its contributors may be used to endorse or promote products 13 | derived from this software without specific prior written permission. 14 | 15 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND 16 | ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED 17 | WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 18 | DISCLAIMED. IN NO EVENT SHALL THE UNIVERSITY OF ST ANDREWS BE LIABLE FOR ANY 19 | DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES 20 | (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; 21 | LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND 22 | ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 23 | (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS 24 | SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. -------------------------------------------------------------------------------- /Makefile: -------------------------------------------------------------------------------- 1 | .PHONY : compile console typecheck typer clean tutorial 2 | 3 | all: compile 4 | 5 | clean: 6 | @rebar3 clean 7 | @rm -f doc/skel.aux doc/skel.bbl doc/skel.blg doc/skel.fdb_latexmk doc/skel.fls 8 | @rm -f doc/skel.out doc/skel.synctex.gz doc/skel.pdf doc/skel.log 9 | @rm -f ./.skel.plt ./.otp.plt skel.tar.gz 10 | @rm -f ifl.data.10.log ifl.verbose.10.log 11 | @rm -f doc/*.html doc/*.css doc/edoc-info doc/erlang.png 12 | @rm -f tutorial/bin/*.html tutorial/bin/*.png 13 | 14 | compile: src/*.erl 15 | @rebar3 compile 16 | 17 | console: compile 18 | @exec erl -args_file ./priv/default.args 19 | 20 | examples: compile 21 | @echo "==> skel (examples)" 22 | @erlc +debug_info -I include -o _build/default/lib/skel/ebin examples/*.erl 23 | 24 | typecheck: compile .skel.plt 25 | @echo "==> skel (typecheck)" 26 | @dialyzer --no_check_plt --plt ./.skel.plt -c _build/default/lib/skel/ebin -Wrace_conditions 27 | 28 | typer: compile .skel.plt 29 | @echo "==> skel (typer)" 30 | @typer --plt ./.skel.plt --show -I include -pa _build/default/lib/skel/ebin -r src 31 | 32 | pdf: doc/skel.pdf 33 | @echo "==> skel (pdf)" 34 | 35 | package: skel.tar.gz 36 | @echo "==> skel (package)" 37 | 38 | todo: 39 | @echo "==> skel (todo)" 40 | @grep -ir "TODO" --exclude Makefile --exclude README.md \ 41 | --exclude-dir .git -- . 42 | 43 | skel.tar.gz: compile 44 | @tar -czf skel.tar.gz -C .. --exclude "skel.tar.gz" skel 45 | 46 | .otp.plt: 47 | @echo "==> otp (plt) # This takes a while, go grab a coffee" 48 | @dialyzer --build_plt --output_plt ./.otp.plt --apps erts kernel stdlib debugger et tools 49 | 50 | .skel.plt: .otp.plt compile 51 | @echo "==> skel (plt)" 52 | @dialyzer --add_to_plt --plt ./.otp.plt --output_plt ./.skel.plt -c _build/default/lib/skel/ebin 53 | 54 | docs: compile 55 | @rm -f doc/*.html doc/*.css doc/edoc-info doc/erlang.png 56 | @./rebar doc 57 | 58 | doc/skel.pdf: doc/skel.tex 59 | @pdflatex -interaction=batchmode -output-directory=./doc skel.tex 60 | @pdflatex -interaction=batchmode -output-directory=./doc skel.tex 61 | 62 | md: docs 63 | @echo "==> Compiling Tutorial" 64 | @multimarkdown -b -f ./tutorial/src/*.md 65 | @mv ./tutorial/src/*.html ./tutorial/bin 66 | @cp ./tutorial/src/*.png ./tutorial/bin 67 | @echo "Documentation may be found in tutorial/bin" 68 | 69 | cleanmd: 70 | @rm -f tutorial/bin/*.html 71 | 72 | # Linux variant? (xdg-open) 73 | tutorial: md 74 | @open -a /Applications/Safari.app tutorial/bin/index.html 75 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | skel 2 | ==== 3 | 4 | A Streaming Process-based Skeleton Library for Erlang 5 | 6 | Usage 7 | ------ 8 | 9 | `make` to compile the library source 10 | 11 | `make examples` to compile the library and the examples 12 | 13 | `make console` to compile and get a console (`make examples console` will give you a console with the examples recompiled too.) 14 | 15 | `make typecheck` to typecheck 16 | 17 | `make typer` to get inferred types (only of use during development for methods without type specs) 18 | 19 | `make clean` to clear up the repo. 20 | 21 | `make todo` to see some todos 22 | 23 | Tutorial 24 | -------- 25 | 26 | A tutorial and API documentation may be found at: [Link](http://chrisb.host.cs.st-andrews.ac.uk/skel.html) 27 | 28 | 29 | Copyright 30 | --------- 31 | 32 | BSD 3-clause licence in LICENCE 33 | -------------------------------------------------------------------------------- /doc/overview.edoc: -------------------------------------------------------------------------------- 1 | ** this is the overview file for the Skel Library ** 2 | 3 | @author Sam Elliot 4 | @copyright 2012 University of St Andrews (See LICENCE) 5 | @version 0.42 6 | @title Welcome to the Skel Library 7 | 8 | -------------------------------------------------------------------------------- /examples/fib.erl: -------------------------------------------------------------------------------- 1 | -module(fib). 2 | 3 | -compile(export_all). 4 | 5 | p(Input) -> 6 | _ = fib(22) + fib(20), 7 | Input. 8 | 9 | f(Input) -> 10 | _ = fib(24), 11 | Input. 12 | 13 | f_prime(Input) -> 14 | _ = fib(22), 15 | Input. 16 | 17 | decomp(Input) -> 18 | lists:duplicate(4, Input). 19 | 20 | recomp([Input|_]) -> 21 | Input. 22 | 23 | % Computational Payload. 24 | % All that matters is that it has predictable runtime and 100% CPU utilisation 25 | fib(X) when X =< 1 -> 26 | X; 27 | fib(X) -> 28 | fib(X-1) + fib(X-2) + fib(X-3) + fib(X-4). 29 | 30 | run_all_examples() -> 31 | [run_examples(X) || X <- [8,6,4,2,1]]. 32 | 33 | run_examples(X) -> 34 | erlang:system_flag(schedulers_online, X), 35 | Ntimes = 10, 36 | Input = lists:duplicate(1024, xx), 37 | io:format("------~nRunning Examples on ~p cores. Please be Patient. ~n", [X]), 38 | io:format("Example 4: ~p~n", [sk_profile:benchmark(fun ?MODULE:example4/1, [Input], Ntimes)]), 39 | io:format("Example 3: ~p~n", [sk_profile:benchmark(fun ?MODULE:example3/1, [Input], Ntimes)]), 40 | io:format("Example 2: ~p~n", [sk_profile:benchmark(fun ?MODULE:example2/1, [Input], Ntimes)]), 41 | io:format("Example 1: ~p~n", [sk_profile:benchmark(fun ?MODULE:example1/1, [Input], Ntimes)]), 42 | io:format("Done with examples on ~p cores.~n------~n", [X]). 43 | 44 | example1(Images) -> 45 | [f(p(Im)) || Im <- Images]. 46 | 47 | example2(Images) -> 48 | skel:run([{seq, fun ?MODULE:p/1}, {seq, fun ?MODULE:f/1}], Images). 49 | 50 | example3(Images) -> 51 | skel:do([{seq, fun ?MODULE:p/1}, {cluster, [{farm, [{seq, fun ?MODULE:f_prime/1}], 4}], fun ?MODULE:decomp/1, fun ?MODULE:recomp/1}], Images). 52 | 53 | example4(Images) -> 54 | skel:run([{farm, [{seq, fun ?MODULE:p/1}], 4}, {cluster, [{farm, [{seq, fun ?MODULE:f_prime/1}], 4}], fun ?MODULE:decomp/1, fun ?MODULE:recomp/1}], Images). 55 | -------------------------------------------------------------------------------- /examples/ifl.erl: -------------------------------------------------------------------------------- 1 | -module(ifl). 2 | 3 | -compile(export_all). 4 | 5 | -define(seq, {seq, fun ?MODULE:unary/1}). 6 | -define(runs, 1). 7 | 8 | payload() -> 9 | fib(27). 10 | 11 | inputs() -> 12 | lists:seq(1, 100000). 13 | 14 | fib(X) when X =< 1 -> 15 | X; 16 | fib(X) -> 17 | fib(X-1) + fib(X-2). 18 | 19 | decomp_for(Stages) -> 20 | fun(X) -> 21 | lists:duplicate(Stages, X) 22 | end. 23 | 24 | recomp([X|_]) -> 25 | X. 26 | 27 | unary(X) -> 28 | payload(), 29 | X. 30 | 31 | binary(X, _Y) -> 32 | payload(), 33 | X. 34 | 35 | test_pipe(Stages) -> 36 | lists:duplicate(Stages, ?seq). 37 | 38 | test_farm(Stages) -> 39 | [{farm, [?seq], Stages}]. 40 | 41 | test_map(Stages) -> 42 | [{cluster, [{farm, [?seq], Stages}], decomp_for(Stages), fun ?MODULE:recomp/1}]. 43 | 44 | test_reduce(Stages) -> 45 | [{reduce, fun ?MODULE:binary/2, decomp_for(Stages)}]. 46 | 47 | 48 | benchmark() -> 49 | {ok, VerboseLog} = file:open("ifl.verbose.10.log", [write]), 50 | {ok, DataLog} = file:open("ifl.data.10.log", [write]), 51 | io:format("Starting tests!~n"), 52 | PayLoadResults = sk_profile:benchmark(fun ?MODULE:payload/0, [], 100), 53 | io:format(VerboseLog, "Payload: ~n\t~w~n", [PayLoadResults]), 54 | io:format(DataLog, "payload.dat: ~.5f~n", [proplists:get_value(mean, PayLoadResults)]), 55 | [benchmark(PipeLine, Stages, DataLog, VerboseLog) || 56 | PipeLine <- [test_pipe, test_farm, test_map, test_reduce], 57 | Stages <- [1,2,4,8,16]], 58 | io:format("Done Tests!~n"). 59 | 60 | benchmark(PipeLine, Stages, DataLog, VerboseLog) -> 61 | Results = [bm_both(PipeLine, Stages, Schedulers) || Schedulers <- [1,2,4,8]], 62 | data_log(PipeLine, Stages, Results, standard_io), 63 | data_log(PipeLine, Stages, Results, DataLog), 64 | verbose_log(PipeLine, Stages, Results, VerboseLog). 65 | 66 | bm_both(PipeLine, Stages, Schedulers) -> 67 | erlang:system_flag(schedulers_online, Schedulers), 68 | PipeLineX = ?MODULE:PipeLine(Stages), 69 | {Schedulers, bm_sequential(PipeLineX), bm_parallel(PipeLineX)}. 70 | 71 | bm_parallel(PipeLine) -> 72 | Inputs = inputs(), 73 | sk_profile:benchmark(fun() -> 74 | sk_assembler:run(PipeLine, Inputs), 75 | receive 76 | {sink_results, Results} -> Results 77 | end 78 | end, [], ?runs). 79 | 80 | bm_sequential(PipeLine) -> 81 | Inputs = inputs(), 82 | sk_profile:benchmark(fun() -> 83 | sk_assembler_sequential:run(PipeLine, Inputs), 84 | receive 85 | {sink_results, Results} -> Results 86 | end 87 | end, [], ?runs). 88 | 89 | data_log(PipeLine, Stages, Results, DataLog) -> 90 | io:format(DataLog, "~w.~w.dat: ", [PipeLine, Stages]), 91 | [io:format(DataLog, "~B ~.5f ", [Schedulers, speedup(SeqResults, ParResults)]) || 92 | {Schedulers, SeqResults, ParResults} <- Results], 93 | io:format(DataLog, "~n", []). 94 | 95 | verbose_log(PipeLine, Stages, Results, VerboseLog) -> 96 | [io:format(VerboseLog, 97 | "PL: ~w; Stages: ~w; SEQUENTIAL; Schedulers: ~w~n\t~w~n" ++ 98 | "PL: ~w; Stages: ~w; PARALLEL ; Schedulers: ~w~n\t~w~n", 99 | [PipeLine, Stages, Schedulers, SeqResults] ++ 100 | [PipeLine, Stages, Schedulers, ParResults]) || 101 | {Schedulers, SeqResults, ParResults} <- Results]. 102 | 103 | speedup(SeqResults, ParResults) -> 104 | proplists:get_value(mean, SeqResults) / proplists:get_value(mean, ParResults). 105 | 106 | 107 | -------------------------------------------------------------------------------- /examples/montecarlo.erl: -------------------------------------------------------------------------------- 1 | -module(montecarlo). 2 | %-compile(export_all). 3 | -export([run/4]). 4 | 5 | -ifdef(debug). 6 | -define(TRACE(Format, Data), io:format(string:concat("TRACE ~p:~p ", Format), [?MODULE, ?LINE] ++ Data)). 7 | -else. 8 | -define(TRACE(Format, Data), void). 9 | -endif. 10 | 11 | run(SamplesTotal, ProcTotal, BoundLower, BoundUpper) -> 12 | TimeStart = time_seconds(), 13 | Processes = [ 14 | spawn(fun() -> montecarlo(SamplesTotal, ProcNum, ProcTotal, BoundLower, BoundUpper) end) 15 | || 16 | ProcNum <- lists:seq(0, ProcTotal - 1) 17 | ], 18 | [Process ! {calc, self()} || Process <- Processes], 19 | ResultSum = lists:sum(finalize(Processes)), 20 | Result = montecarlo_finalize(SamplesTotal, BoundLower, BoundUpper, ResultSum), 21 | TimeEnd = time_seconds(), 22 | TimeElapsed = TimeEnd - TimeStart, 23 | io:format("result: ~.6f time: ~.6f~n", [Result, TimeElapsed]). 24 | 25 | montecarlo(SamplesTotal, ProcNum, ProcTotal, BoundLower, BoundUpper) -> 26 | receive 27 | {calc, Parent} -> 28 | random_seed(ProcNum), 29 | Result = samples_process(SamplesTotal, ProcNum, ProcTotal, BoundLower, BoundUpper), 30 | ?TRACE( 31 | "CALC(~p): SamplesTotal: ~p ProcNum: ~p ProcTotal: ~p BoundLower: ~p BoundUpper: ~p Result: ~p~n", 32 | [self(), SamplesTotal, ProcNum, ProcTotal, BoundLower, BoundUpper, Result] 33 | ), 34 | Parent ! {done, Result, self()} 35 | end. 36 | 37 | finalize(Processes) -> 38 | finalize(Processes, []). 39 | 40 | finalize(Processes, Results) -> 41 | receive 42 | {done, Result, From} -> 43 | IsValid = lists:member(From, Processes), 44 | if 45 | IsValid and ((length(Results) + 1) =:= length(Processes)) -> 46 | ?TRACE("DONE (final): from: ~p result: ~p results: ~p~n", [From, Result, Results]), 47 | [Result | Results]; 48 | IsValid -> 49 | ?TRACE("DONE: from: ~p result: ~p results: ~p~n", [From, Result, Results]), 50 | finalize(Processes, [Result | Results]); 51 | true -> 52 | finalize(Processes, Results) 53 | end 54 | end. 55 | 56 | samples_calc(SamplesTotal, ProcNum, ProcTotal) when ProcNum < ProcTotal -> 57 | SamplesBase = SamplesTotal div ProcTotal, 58 | SamplesRemainder = SamplesTotal rem ProcTotal, 59 | if 60 | ProcNum < SamplesRemainder -> 61 | SamplesBase + 1; 62 | true -> 63 | SamplesBase 64 | end. 65 | 66 | samples_process(SamplesTotal, ProcNum, ProcTotal, BoundLower, BoundUpper) -> 67 | Samples = samples_calc(SamplesTotal, ProcNum, ProcTotal), 68 | lists:sum([fun_calc(random_value(BoundLower, BoundUpper)) || _ <- lists:seq(1, Samples)]). 69 | 70 | montecarlo_finalize(SamplesTotal, BoundLower, BoundUpper, Results) -> 71 | Results * ((BoundUpper - BoundLower) / SamplesTotal). 72 | 73 | fun_calc(X) -> 74 | (1.0 / (math:sqrt(2.0 * math:pi()))) * math:exp(-1.0 * (math:pow(X, 2.0) / 2.0)). 75 | 76 | random_value(BoundLower, BoundUpper) when BoundLower < BoundUpper -> 77 | (random:uniform() * (BoundUpper - BoundLower)) + BoundLower. 78 | 79 | random_seed(X) -> 80 | {S1, S2, S3} = now(), 81 | random:seed(S1 + X, S2 + X, S3 + X). 82 | 83 | time_seconds() -> 84 | {MS, S, US} = now(), 85 | (MS * 1.0e+6) + S + (US * 1.0e-6). 86 | -------------------------------------------------------------------------------- /examples/queens.erl: -------------------------------------------------------------------------------- 1 | -module(queens). 2 | 3 | -compile(export_all). 4 | 5 | 6 | run(N) -> rainhas(N). 7 | 8 | rainhas(N) -> lists:map(fun(X) -> search(N, X) end, lists:seq(1, N)). 9 | 10 | search(Numero, N) -> lists:takewhile( fun (A) -> head(A) == N end, prainhas(Numero, N)). 11 | 12 | head([H|T]) -> H. 13 | 14 | prainhas(Numero, Linha) 15 | -> rainhas2(Numero, Linha, Numero). 16 | 17 | check({C,L}, {I,J}) -> (L == J) or (C+L == I+J) or (C-L == I-J). 18 | 19 | safe(P,N) -> 20 | M = (length(P)) + 1, 21 | List = [ not(check({I,J}, {M,N})) || {I,J} <- lists:zip(lists:seq(1, length(P)), P) ], 22 | lists:foldr(fun(X,Y) -> X and Y end, true, List). 23 | 24 | rainhas2(0, Linha, Numero) -> [[]]; 25 | rainhas2(M, Linha, Numero) -> [lists:append(P, [N]) || P <- rainhas2(M-1, Linha, Numero), N <- lists:append(lists:seq(Linha, Numero), lists:seq(1, Linha-1)), safe(P, N)]. 26 | 27 | parRun(N) -> skel:run([{farm, [{seq, fun(X) -> ?MODULE:search(N, X) end}], 12}], lists:seq(1,N)), 28 | receive 29 | {sink_results, Results} -> Results 30 | end. 31 | 32 | run_examples(X) -> 33 | erlang:system_flag(schedulers_online, X), 34 | io:format("Example 4: ~p~n", [sk_profile:benchmark(fun ?MODULE:parRun/1, [12], 1)]), 35 | io:format("Done with examples on ~p cores.~n------~n", [X]). 36 | 37 | 38 | 39 | 40 | -------------------------------------------------------------------------------- /examples/reduce.erl: -------------------------------------------------------------------------------- 1 | -module(reduce). 2 | 3 | -compile(export_all). 4 | 5 | % Computational Payload. 6 | % All that matters is that it has predictable runtime and 100% CPU utilisation 7 | fib(X) when X =< 1 -> 8 | X; 9 | fib(X) -> 10 | fib(X-1) + fib(X-2) + fib(X-3) + fib(X-4). 11 | 12 | id(X) -> 13 | X. 14 | 15 | reduce(X, Y) -> 16 | fib(21), 17 | X + Y. 18 | 19 | benchmark() -> 20 | io:format("Starting tests!~n", []), 21 | io:format("ReduceFn: ~w~n", [sk_profile:benchmark(fun ?MODULE:reduce/2, [0,0], 100)]), 22 | NT = 5, 23 | [bm(NS, FN, NI, PPI, NT) || NS <- [16,8], NI <- [1, 100], PPI <- [32, 64, 128], FN <- [parallel, sequential]]. 24 | 25 | bm(NSchedulers, Fn, NInputs, PartsPerInput, NTimes) -> 26 | erlang:system_flag(schedulers_online, NSchedulers), 27 | Inputs = lists:duplicate(NInputs, lists:seq(1, PartsPerInput)), 28 | Results = sk_profile:benchmark(fun ?MODULE:Fn/1, [Inputs], NTimes), 29 | io:format("~p: (~p Schedulers; ~p Inputs; ~p Parts): ~n\t~w~n", [Fn, NSchedulers, NInputs, PartsPerInput, Results]), 30 | Results. 31 | 32 | sequential(Inputs) -> 33 | [sk_reduce:fold1(fun ?MODULE:reduce/2, Input) || Input <- Inputs ]. 34 | 35 | parallel(Inputs) -> 36 | skel:do([{reduce, fun ?MODULE:reduce/2, fun ?MODULE:id/1}], Inputs). 37 | -------------------------------------------------------------------------------- /examples/sumEuler.erl: -------------------------------------------------------------------------------- 1 | -module(sumEuler). 2 | 3 | -compile(export_all). 4 | 5 | gcd(A, 0) -> A; 6 | gcd(A, B) -> gcd(B, A rem B). 7 | 8 | 9 | relPrime(X,Y) -> gcd(X, Y) == 1. 10 | 11 | mkList(N) -> lists:seq(1,N). 12 | 13 | 14 | euler(N) -> length( lists:filter(fun(X) -> ?MODULE:relPrime(N,X) end, mkList(N))). 15 | 16 | 17 | sumEuler(N) -> lists:sum(lists:map(fun ?MODULE:euler/1,mkList(N))). 18 | 19 | mapEuler(List) -> lists:map(fun ?MODULE:sumEuler/1, List). 20 | 21 | parSumEuler(N) -> skel:do([{farm, [{seq, fun ?MODULE:euler/1}], 10}], mkList(N)). 22 | 23 | run_examples(X, Y) -> 24 | erlang:system_flag(schedulers_online, X), 25 | io:format("Example 4: ~p~n", [sk_profile:benchmark(fun ?MODULE:parSumEuler/1, [Y], 1)]), 26 | io:format("Done with examples on ~p cores.~n------~n", [X]). 27 | 28 | 29 | chunk(List, X, X, ChunkSize) -> []; 30 | chunk(List, Start, Len, ChunkSize) -> 31 | lists:append([lists:sublist(List, Start, ChunkSize)], [chunk(List, Start+ChunkSize, Len, ChunkSize)]). 32 | -------------------------------------------------------------------------------- /include/skel.hrl: -------------------------------------------------------------------------------- 1 | %%%---------------------------------------------------------------------------- 2 | %%% author Sam Elliott 3 | %%% copyright 2012 University of St Andrews (See LICENCE) 4 | %%%---------------------------------------------------------------------------- 5 | %% Defines the workflow concept. A workflow is a specification that defines how work should be undertaken, it is a list of one or more skeletons. 6 | -type workflow() :: [wf_item(),...]. 7 | 8 | -type wf_item() :: {seq, worker_fun()} 9 | | {pipe, workflow()} 10 | | {ord, workflow()} 11 | | {farm, workflow(), pos_integer()} 12 | | {hyb_farm, workflow(), workflow(), pos_integer(), pos_integer()} 13 | | {cluster, workflow(), decomp_fun(), recomp_fun()} 14 | | {map, workflow()} 15 | | {map, workflow(), pos_integer()} 16 | | {hyb_map, workflow(), workflow(), pos_integer(), pos_integer()} 17 | | {reduce, reduce_fun(), decomp_fun()} 18 | | {feedback, workflow(), filter_fun()}. 19 | % Workflow items (skeletons) and their content. 20 | 21 | -type worker_fun() :: fun((any()) -> any()). % Any function that is performed by a worker unit. 22 | 23 | -type decomp_fun() :: fun((any()) -> [any(),...]). % Any function that decomposes its input into several parts, that may be used by different workers. 24 | 25 | -type recomp_fun() :: fun(([any(),...]) -> any()). % Any function that recomposes decomposed parts. 26 | 27 | -type reduce_fun() :: fun((any(), any()) -> any()). % Any function that reduces two inputs to a single output. 28 | 29 | -type filter_fun() :: fun((any()) -> boolean()). % Any function that determines whether an input matches one or more given criteria. 30 | 31 | -type maker_fun() :: fun((pid()) -> pid()). % Any function that spawns a process. 32 | 33 | 34 | 35 | 36 | -type input() :: list() | module(). % Definition for any possible worker function input. 37 | 38 | 39 | 40 | 41 | -type system_message() :: {system, sm_item()}. 42 | %% These are all the system-level messages. EOS, coordination etc. 43 | 44 | -type data_message() :: {data, any(), [dm_identifier()]}. 45 | %% The data being passed, and a stack of information as to its identity and 46 | %% ordering (used in ordered skeletons and maps). 47 | 48 | -type dm_identifier() :: feedback 49 | | {ord, pos_integer()} 50 | | {decomp, reference(), pos_integer(), pos_integer()} 51 | | {reduce, reference(), number()}. 52 | %% Used to identify different types of data message. 53 | 54 | -type sm_item() :: eos 55 | | {reduce_unit, reference(), number()} 56 | | {feedback_setup, pid(), reference()} 57 | | {feedback_reply, pid(), reference()} 58 | | {counter, term(), pid()}. 59 | %% Used to identify different types of system message. 60 | 61 | 62 | 63 | 64 | -type data_fun() :: fun((data_message()) -> data_message()). 65 | %% Any function that takes a data message as input, and similarly returns a 66 | %% data message. 67 | 68 | -type data_decomp_fun() :: fun((data_message()) -> [data_message(),...]). 69 | %% Any function that decomposes, or splits, a single data message. Producing a 70 | %% list of data messages, each containing a fragment of the input's message. 71 | 72 | -type data_recomp_fun() :: fun(([data_message(),...]) -> data_message()). 73 | %% Any function that recomposes, or joins, a list of data messages. Produces a 74 | %% single data message containing all information in the input. 75 | 76 | -type data_reduce_fun() :: fun((data_message(), data_message()) -> data_message()). 77 | %% Any function that reduces two data messages to a single data message. 78 | 79 | -type data_filter_fun() :: fun((data_message()) -> boolean()). 80 | %% Any function that determines whether a data message satisfies some given 81 | %% constraint. 82 | -------------------------------------------------------------------------------- /priv/default.args: -------------------------------------------------------------------------------- 1 | -pa _build/default/lib/skel/ebin 2 | -sname skel-test 3 | -setcookie skel-test 4 | -------------------------------------------------------------------------------- /rebar: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ParaPhrase/skel/c8c91f5c4adf1d299012bc2ecc798cc2b5ae9e63/rebar -------------------------------------------------------------------------------- /rebar.config: -------------------------------------------------------------------------------- 1 | 2 | % Include Erlang Debug Info 3 | {edoc_opts,[{private, false}]}. 4 | {erl_opts, [debug_info, warn_missing_spec]}. -------------------------------------------------------------------------------- /src/gen_sk.erl: -------------------------------------------------------------------------------- 1 | %%%---------------------------------------------------------------------------- 2 | %%% @hidden 3 | %%% @author Sam Elliott 4 | %%% @copyright 2012 University of St Andrews (See LICENCE) 5 | %%% @doc This is the start of a generic way to specify sk processes using 6 | %%% callbacks. 7 | %%% 8 | %%% It's completely unfinished. 9 | %%% 10 | %%% @end 11 | %%%---------------------------------------------------------------------------- 12 | -module(gen_sk). 13 | 14 | -include("skel.hrl"). 15 | 16 | -ifdef(TEST). 17 | -compile(export_all). 18 | -endif. 19 | 20 | -callback init(Args :: term()) -> 21 | {ok, State :: term()}. 22 | 23 | -callback handle_data(Message :: skel:data_message(), State :: term()) -> 24 | {forward, Message1 :: [skel:data_message(),...], NewState :: term()} | 25 | {eos, NewState :: term()}. 26 | 27 | -callback handle_system(Message :: skel:data_message(), State :: term()) -> 28 | {forward, NewState :: term()} | 29 | {drop, NewState :: term()} | 30 | {eos, NewState :: term()}. 31 | 32 | -callback handle_info(Info :: term(), State :: term()) -> 33 | {ok, NewState :: term()} | 34 | {eos, NewState :: term()}. 35 | 36 | -callback terminate(State :: term()) -> 37 | ok. 38 | 39 | -------------------------------------------------------------------------------- /src/sk_assembler.erl: -------------------------------------------------------------------------------- 1 | %%%---------------------------------------------------------------------------- 2 | %%% @author Sam Elliott 3 | %%% @copyright 2012 University of St Andrews (See LICENCE) 4 | %%% @headerfile "skel.hrl" 5 | %%% 6 | %%% @doc This module takes a workflow specification, and converts it in into a 7 | %%% set of (concurrent) running processes. 8 | %%% 9 | %%% 10 | %%% @end 11 | %%%---------------------------------------------------------------------------- 12 | -module(sk_assembler). 13 | 14 | -export([ 15 | make/2 16 | ,make_hyb/4 17 | ,run/2 18 | ]). 19 | 20 | -include("skel.hrl"). 21 | 22 | -ifdef(TEST). 23 | -compile(export_all). 24 | -endif. 25 | 26 | -spec make(workflow(), pid() | module()) -> pid() . 27 | %% @doc Function to produce a set of processes according to the given workflow 28 | %% specification. 29 | make(WorkFlow, EndModule) when is_atom(EndModule) -> 30 | DrainPid = (sk_sink:make(EndModule))(self()), 31 | make(WorkFlow, DrainPid); 32 | make(WorkFlow, EndPid) when is_pid(EndPid) -> 33 | MakeFns = [parse(Section) || Section <- WorkFlow], 34 | lists:foldr(fun(MakeFn, Pid) -> MakeFn(Pid) end, EndPid, MakeFns). 35 | 36 | -spec make_hyb(workflow(), pid(), pos_integer(), pos_integer()) -> pid(). 37 | make_hyb(WorkFlow, EndPid, NCPUWorkers, NGPUWorkers) when is_pid(EndPid) -> 38 | MakeFns = [parse_hyb(Section, NCPUWorkers, NGPUWorkers) || Section <- WorkFlow], 39 | lists:foldr(fun(MakeFn, Pid) -> MakeFn(Pid) end, EndPid, MakeFns). 40 | 41 | 42 | -spec run(pid() | workflow(), input()) -> pid(). 43 | %% @doc Function to produce and start a set of processes according to the 44 | %% given workflow specification and input. 45 | run(WorkFlow, Input) when is_pid(WorkFlow) -> 46 | Feeder = sk_source:make(Input), 47 | Feeder(WorkFlow); 48 | run(WorkFlow, Input) when is_list(WorkFlow) -> 49 | DrainPid = (sk_sink:make())(self()), 50 | AssembledWF = make(WorkFlow, DrainPid), 51 | run(AssembledWF, Input). 52 | 53 | parse_hyb(Section, NCPUWorkers, NGPUWorkers) -> 54 | case Section of 55 | {hyb_map, WorkFlowCPU, WorkFlowGPU} -> 56 | parse({hyb_map, WorkFlowCPU, WorkFlowGPU, NCPUWorkers, NGPUWorkers}); 57 | Other -> parse(Other) 58 | end. 59 | 60 | 61 | -spec parse(wf_item()) -> maker_fun(). 62 | %% @doc Determines the course of action to be taken according to the type of 63 | %% workflow specified. Constructs and starts specific skeleton instances. 64 | parse(Fun) when is_function(Fun, 1) -> 65 | parse({seq, Fun}); 66 | parse({seq, Fun}) when is_function(Fun, 1) -> 67 | sk_seq:make(Fun); 68 | parse({pipe, WorkFlow}) -> 69 | sk_pipe:make(WorkFlow); 70 | parse({ord, WorkFlow}) -> 71 | sk_ord:make(WorkFlow); 72 | parse({farm, WorkFlow, NWorkers}) -> 73 | sk_farm:make(NWorkers, WorkFlow); 74 | parse({hyb_farm, WorkFlowCPU, WorkFlowGPU, NCPUWorkers, NGPUWorkers}) -> 75 | sk_farm:make_hyb(NCPUWorkers, NGPUWorkers, WorkFlowCPU, WorkFlowGPU); 76 | parse({map, WorkFlow}) -> 77 | sk_map:make(WorkFlow); 78 | parse({map, WorkFlow, NWorkers}) -> 79 | sk_map:make(WorkFlow, NWorkers); 80 | parse({hyb_map, WorkFlowCPU, WorkFlowGPU}) -> 81 | sk_map:make_hyb(WorkFlowCPU, WorkFlowGPU); 82 | parse({hyb_map, WorkFlowCPU, WorkFlowGPU, NCPUWorkers, NGPUWorkers}) -> 83 | sk_map:make_hyb(WorkFlowCPU, WorkFlowGPU, NCPUWorkers, NGPUWorkers); 84 | parse({cluster, WorkFlow, Decomp, Recomp}) when is_function(Decomp, 1), 85 | is_function(Recomp, 1) -> 86 | sk_cluster:make(WorkFlow, Decomp, Recomp); 87 | parse({hyb_cluster, WorkFlow, Decomp, Recomp, NCPUWorkers, NGPUWorkers}) when 88 | is_function(Decomp, 1), is_function(Recomp, 1) -> 89 | sk_cluster:make_hyb(WorkFlow, Decomp, Recomp, NCPUWorkers, NGPUWorkers); 90 | parse({hyb_cluster, WorkFlow, TimeRatio, NCPUWorkers, NGPUWorkers}) -> 91 | sk_cluster:make_hyb(WorkFlow, TimeRatio, NCPUWorkers, NGPUWorkers); 92 | parse({hyb_cluster, WorkFlow, TimeRatio, StructSizeFun, MakeChunkFun, RecompFun, NCPUWorkers, NGPUWorkers}) -> 93 | sk_cluster:make_hyb(WorkFlow, TimeRatio, StructSizeFun, MakeChunkFun, RecompFun, NCPUWorkers, NGPUWorkers); 94 | 95 | % parse({decomp, WorkFlow, Decomp, Recomp}) when is_function(Decomp, 1), 96 | % is_function(Recomp, 1) -> 97 | % sk_decomp:make(WorkFlow, Decomp, Recomp); 98 | % parse({map, WorkFlow, Decomp, Recomp}) when is_function(Decomp, 1), 99 | % is_function(Recomp, 1) -> 100 | % sk_map:make(WorkFlow, Decomp, Recomp); 101 | parse({reduce, Reduce, Decomp}) when is_function(Reduce, 2), 102 | is_function(Decomp, 1) -> 103 | sk_reduce:make(Decomp, Reduce); 104 | parse({feedback, WorkFlow, Filter}) when is_function(Filter, 1) -> 105 | sk_feedback:make(WorkFlow, Filter). 106 | 107 | 108 | -------------------------------------------------------------------------------- /src/sk_assembler_sequential.erl: -------------------------------------------------------------------------------- 1 | %%%---------------------------------------------------------------------------- 2 | %%% @author Sam Elliott 3 | %%% @copyright 2012 University of St Andrews (See LICENCE) 4 | %%% @doc This takes a workflow specification, parses it and converts it into a 5 | %%% function that runs a given skeleton entirely sequentially. 6 | %%% 7 | %%% This is primarily used so we can use the same workflow specification when 8 | %%% we need sequential benchmarks. 9 | %%% 10 | %%% @headerfile "skel.hrl" 11 | %%% 12 | %%% @end 13 | %%%---------------------------------------------------------------------------- 14 | -module(sk_assembler_sequential). 15 | 16 | -export([ 17 | run/2 18 | ,compose/2 19 | ,id/1 20 | ]). 21 | 22 | -include("skel.hrl"). 23 | 24 | -ifdef(TEST). 25 | -compile(export_all). 26 | -endif. 27 | 28 | -spec run(workflow(), list()) -> pid(). 29 | run(WorkFlow, Inputs) -> 30 | Fun = make(WorkFlow), 31 | self() ! {sink_results, [Fun(Input) || Input <- Inputs]}, 32 | self(). 33 | 34 | -spec make(workflow()) -> worker_fun(). 35 | make(WorkFlow) -> 36 | Funs = [parse(Section) || Section <- WorkFlow], 37 | lists:foldl(fun ?MODULE:compose/2, fun ?MODULE:id/1, Funs). 38 | 39 | -spec compose(fun((B) -> C), fun((A) -> B)) -> fun((A) -> C) 40 | when A :: term(), B :: term(), C :: term(). 41 | compose(BC, AB) -> fun(X) -> BC(AB(X)) end. 42 | -spec id(A) -> A 43 | when A :: term(). 44 | id(X) -> X. 45 | 46 | -spec parse(wf_item()) -> fun(). 47 | parse(Fun) when is_function(Fun, 1) -> 48 | parse({seq, Fun}); 49 | parse({seq, Fun}) when is_function(Fun, 1) -> 50 | Fun; 51 | parse({pipe, WorkFlow}) -> 52 | make(WorkFlow); 53 | parse({ord, WorkFlow}) -> 54 | make(WorkFlow); 55 | parse({farm, WorkFlow, _}) -> 56 | make(WorkFlow); 57 | parse({decomp, WorkFlow, Decomp, Recomp}) when is_function(Decomp, 1), 58 | is_function(Recomp, 1) -> 59 | decomp_map_recomp(make(WorkFlow), Decomp, Recomp); 60 | parse({map, WorkFlow, Decomp, Recomp}) when is_function(Decomp, 1), 61 | is_function(Recomp, 1) -> 62 | decomp_map_recomp(make(WorkFlow), Decomp, Recomp); 63 | parse({reduce, Reduce, Decomp}) when is_function(Reduce, 2), 64 | is_function(Decomp, 1) -> 65 | fun(Input) -> 66 | Parts = Decomp(Input), 67 | sk_reduce:fold1(Reduce, Parts) 68 | end; 69 | parse({feedback, WorkFlow, FilterFun}) when is_function(FilterFun, 1) -> 70 | WorkFlowFun = make(WorkFlow), 71 | fun(Input) -> 72 | feedback(WorkFlowFun, FilterFun, Input) 73 | end. 74 | 75 | -spec decomp_map_recomp(worker_fun(), decomp_fun(), recomp_fun()) -> worker_fun(). 76 | decomp_map_recomp(WorkFlowFun, Decomp, Recomp) -> 77 | fun(Input) -> 78 | Recomp([WorkFlowFun(Part) || Part <- Decomp(Input)]) 79 | end. 80 | 81 | -spec feedback(worker_fun(), filter_fun(), any()) -> any(). 82 | feedback(WorkFlowFun, FilterFun, Input) -> 83 | Input1 = WorkFlowFun(Input), 84 | case FilterFun(Input1) of 85 | true -> feedback(WorkFlowFun, FilterFun, Input1); 86 | false -> Input1 87 | end. 88 | -------------------------------------------------------------------------------- /src/sk_cluster.erl: -------------------------------------------------------------------------------- 1 | %%%---------------------------------------------------------------------------- 2 | %%% @author Sam Elliott 3 | %%% @copyright 2012 University of St Andrews (See LICENCE) 4 | %%% @headerfile "skel.hrl" 5 | %%% 6 | %%% @doc This module contains the intitialisation logic of a Cluster wrapper. 7 | %%% 8 | %%% The cluster wrapper acts in a similar manner to the Map skeleton, but 9 | %%% allows the developer to customise the decomposition and recomposition 10 | %%% functions used. Inputs are decomposed according to the developer-defined 11 | %%% decomposition function, and then passed to the worker process running the 12 | %%% inner-workflow. 13 | %%% 14 | %%% Additionally, the cluster wrapper allows the developer to determine the 15 | %%% level of clustering of inputs. Hence, the identifying atom `cluster'. 16 | %%% 17 | %%% 18 | %%% === Example === 19 | %%% 20 | %%% ```skel:do([{cluster, [{farm, [{seq, fun ?MODULE:f/1}], 10}], fun ?MODULE:decomp/1, fun ?MODULE:recomp/1}], Input).''' 21 | %%% 22 | %%% We are able to replicate the second Map example using the above 23 | %%% cluster workflow item. Where `decomp/1' and `recomp/1' are developer- 24 | %%% defined, and `f/1' remains the function to be mapped to each element 25 | %%% in every input. Here we use an inner task farm with the ten workers to 26 | %%% repeatedly apply `f/1' to each partite element, but a different 27 | %%% arrangement of nested skeletons may also be used. 28 | %%% 29 | %%% @end 30 | %%%---------------------------------------------------------------------------- 31 | 32 | 33 | -module(sk_cluster). 34 | 35 | -export([ 36 | make/3, 37 | make_hyb/4, 38 | make_hyb/5, 39 | make_hyb/7 40 | ]). 41 | 42 | -include("skel.hrl"). 43 | 44 | -ifdef(TEST). 45 | -compile(export_all). 46 | -endif. 47 | 48 | -spec make(workflow(), decomp_fun(), recomp_fun()) -> fun((pid()) -> pid()). 49 | %% @doc Initialises the Cluster wrapper using both the developer-defined 50 | %% functions under `Decomp' and `Recomp' as decomposition and recomposition 51 | %% functions respectively. 52 | %% 53 | %% Inputs are decomposed, sent through the specified (inner) workflow, and 54 | %% then recomposed to be delivered as output. 55 | make(WorkFlow, Decomp, Recomp) -> 56 | fun(NextPid) -> 57 | RecompPid = spawn(sk_cluster_recomp, start, [Recomp, NextPid]), 58 | WorkerPid = sk_utils:start_worker(WorkFlow, RecompPid), 59 | spawn(sk_cluster_decomp, start, [Decomp, WorkerPid]) 60 | end. 61 | 62 | ceiling(X) -> 63 | T = erlang:trunc(X), 64 | case (X - T) of 65 | Neg when Neg < 0 -> T; 66 | Pos when Pos > 0 -> T + 1; 67 | _ -> T 68 | end. 69 | 70 | 71 | mark_tasks([], _NCPUWorkers, _NGPUWorkers) -> 72 | []; 73 | mark_tasks(_Tasks, 0, 0) -> 74 | []; 75 | mark_tasks([Task|Tasks], 0, NGPUWorkers) -> 76 | [{gpu, Task} | mark_tasks(Tasks, 0, NGPUWorkers-1)]; 77 | mark_tasks([Task|Tasks], NCPUWorkers, NGPUWorkers) -> 78 | [{cpu, Task} | mark_tasks(Tasks, NCPUWorkers-1, NGPUWorkers)]. 79 | 80 | hyb_cluster_decomp(Decomp, NCPUWorkers, NGPUWorkers, Input) -> 81 | Tasks = Decomp(Input), 82 | mark_tasks(Tasks, NCPUWorkers, NGPUWorkers). 83 | 84 | calculate_ratio(TimeRatio, NTasks, NCPUW, NGPUW) -> 85 | TasksCPU = lists:seq(0, NTasks), 86 | Time = fun(CPUTasks, GPUTasks) -> 87 | max (ceiling(CPUTasks/NCPUW)*TimeRatio, ceiling(GPUTasks/NGPUW)) 88 | end, 89 | Ratio = lists:foldl(fun(Elem,Acc) -> FooBar = Time(Elem, NTasks-Elem), 90 | if 91 | (FooBar < element(1,Acc)) or (element(1,Acc) == -1) 92 | -> {FooBar, Elem}; 93 | true -> Acc 94 | end end, 95 | {-1,0}, TasksCPU), 96 | {element(2,Ratio), NTasks-element(2,Ratio)}. 97 | 98 | calculate_chunk_sizes(NrItems, NrWorkers) -> 99 | ChunkSize = NrItems div NrWorkers, 100 | Remainder = NrItems rem NrWorkers, 101 | ChunkSizes = lists:duplicate(Remainder, {ChunkSize+1}) ++ lists:duplicate(NrWorkers-Remainder, {ChunkSize}), 102 | ChunkSizes. 103 | 104 | create_task_list([], [], _MakeChunkFun, _Input) -> 105 | []; 106 | create_task_list([CPUChunk|CPUChunks], GPUChunks, MakeChunkFun, Input) -> 107 | CPUChunkSize = element(1,CPUChunk), 108 | {Work, Rest} = MakeChunkFun(Input, CPUChunkSize), 109 | [ {cpu, Work} | create_task_list(CPUChunks, GPUChunks, MakeChunkFun, Rest) ]; 110 | create_task_list([], [GPUChunk|GPUChunks], MakeChunkFun, Input) -> 111 | GPUChunkSize = element(1,GPUChunk), 112 | {Work, Rest} = MakeChunkFun(Input, GPUChunkSize), 113 | [ {gpu, Work} | create_task_list([], GPUChunks, MakeChunkFun, Rest) ]. 114 | 115 | hyb_cluster_decomp_default(TimeRatio, StructSizeFun, MakeChunkFun, NCPUWorkers, NGPUWorkers, Input) -> 116 | NItems = StructSizeFun(Input), 117 | {CPUItems, GPUItems} = if 118 | (NCPUWorkers>0) and (NGPUWorkers>0) -> calculate_ratio(TimeRatio, NItems, NCPUWorkers, NGPUWorkers); 119 | NGPUWorkers == 0 -> {NItems,0}; 120 | NCPUWorkers == 0 -> {0, NItems} 121 | end, 122 | CPUChunkSizes = calculate_chunk_sizes(CPUItems, NCPUWorkers), 123 | GPUChunkSizes = calculate_chunk_sizes(GPUItems, NGPUWorkers), 124 | [create_task_list(CPUChunkSizes, GPUChunkSizes, MakeChunkFun, Input)]. 125 | 126 | -spec make_hyb(workflow(), decomp_fun(), recomp_fun(), pos_integer(), pos_integer()) -> fun((pid()) -> pid()). 127 | make_hyb(Workflow, Decomp, Recomp, NCPUWorkers, NGPUWorkers) -> 128 | fun(NextPid) -> 129 | RecompPid = spawn(sk_cluster_recomp, start, [Recomp, NextPid]), 130 | WorkerPid = sk_utils:start_worker_hyb(Workflow, RecompPid, NCPUWorkers, NGPUWorkers), 131 | spawn(sk_cluster_decomp, start, [fun (Input) -> hyb_cluster_decomp(Decomp, NCPUWorkers, NGPUWorkers, Input) end, 132 | WorkerPid]) 133 | end. 134 | 135 | -spec make_hyb(workflow(), float(), fun((any()) -> pos_integer()), fun((any(),pos_integer()) -> pos_integer()), 136 | fun((any())->any()), 137 | pos_integer(), pos_integer()) -> fun((pid()) -> pid()). 138 | make_hyb(Workflow, TimeRatio, StructSizeFun, MakeChunkFun, RecompFun, NCPUWorkers, NGPUWorkers) -> 139 | fun(NextPid) -> 140 | RecompPid = spawn(sk_cluster_recomp, start, [RecompFun, NextPid]), 141 | WorkerPid = sk_utils:start_worker_hyb(Workflow, RecompPid, NCPUWorkers, NGPUWorkers), 142 | spawn(sk_cluster_decomp, start, [fun (Input) -> hyb_cluster_decomp_default(TimeRatio, StructSizeFun, MakeChunkFun, NCPUWorkers, NGPUWorkers, Input) end, 143 | WorkerPid]) 144 | end. 145 | 146 | -spec make_hyb(workflow(), float(), pos_integer(), pos_integer()) -> fun((pid())->pid()). 147 | make_hyb(Workflow, TimeRatio, NCPUWorkers, NGPUWorkers) -> 148 | fun(NextPid) -> 149 | RecompPid = spawn(sk_cluster_recomp, start, [fun lists:flatten/1, NextPid]), 150 | WorkerPid = sk_utils:start_worker_hyb(Workflow, RecompPid, NCPUWorkers, NGPUWorkers), 151 | spawn(sk_cluster_decomp, start, [fun (Input) -> hyb_cluster_decomp_default(TimeRatio, fun length/1,fun (Data,Pos) -> lists:split(Pos,Data) end, NCPUWorkers, NGPUWorkers, Input) end, 152 | WorkerPid]) 153 | end. 154 | 155 | -------------------------------------------------------------------------------- /src/sk_cluster_decomp.erl: -------------------------------------------------------------------------------- 1 | %%%---------------------------------------------------------------------------- 2 | %%% @author Sam Elliott 3 | %%% @copyright 2012 University of St Andrews (See LICENCE) 4 | %%% @headerfile "skel.hrl" 5 | %%% 6 | %%% @doc This module contains what happens in the decomposition process of a 7 | %%% Cluster wrapper. 8 | %%% 9 | %%% The cluster wrapper acts in a similar manner to the Map skeleton, but 10 | %%% allows the developer to customise the decomposition and recomposition 11 | %%% functions used. Inputs are decomposed according to the developer-defined 12 | %%% decomposition function, and then passed to the worker process running the 13 | %%% inner-workflow. 14 | %%% 15 | %%% The decomposition process divides each input according to a developer- 16 | %%% defined function. This function should compliment the similarly developer- 17 | %%% defined recomposition function. 18 | %%% 19 | %%% @end 20 | %%%---------------------------------------------------------------------------- 21 | -module(sk_cluster_decomp). 22 | 23 | -export([ 24 | start/2 25 | ]). 26 | 27 | -include("skel.hrl"). 28 | 29 | -ifdef(TEST). 30 | -compile(export_all). 31 | -endif. 32 | 33 | -spec start(decomp_fun(), pid()) -> 'eos'. 34 | %% @doc Initialises the decomposition process. 35 | %% 36 | %% The decomposition process listens for input, dividing them into partite 37 | %% elements upon receipt. This decomposition is powered by the decomposition 38 | %% function given by `Decomp'. 39 | start(Decomp, NextPid) -> 40 | sk_tracer:t(75, self(), {?MODULE, start}, [{next_pid, NextPid}]), 41 | DataDecompFun = sk_data:decomp_by(Decomp), 42 | loop(DataDecompFun, NextPid). 43 | 44 | -spec loop(data_decomp_fun(), pid()) -> 'eos'. 45 | %% @doc Worker function for {@link start/2}; recursively receives input, which is then decomposed using the function under `DataDecompFun'. These decomposed elements are then dispatched to the worker processes. 46 | loop(DataDecompFun, NextPid) -> 47 | receive 48 | {data, _, _} = DataMessage -> 49 | PartitionMessages = DataDecompFun(DataMessage), 50 | Ref = make_ref(), 51 | sk_tracer:t(60, self(), {?MODULE, data}, [{ref, Ref}, {input, DataMessage}, {partitions, PartitionMessages}]), 52 | dispatch(Ref, length(PartitionMessages), PartitionMessages, NextPid), 53 | loop(DataDecompFun, NextPid); 54 | {system, eos} -> 55 | sk_tracer:t(75, self(), NextPid, {?MODULE, system}, [{msg, eos}]), 56 | NextPid ! {system, eos}, 57 | eos 58 | end. 59 | 60 | -spec dispatch(reference(), pos_integer(), [data_message(),...], pid()) -> 'ok'. 61 | %% @doc Partite elements listed in `PartitionMessages' are formatted so that 62 | %% they may be recomposed later. Each are then sent to the worker process at 63 | %% `NextPid'. 64 | dispatch(Ref, NPartitions, PartitionMessages, NextPid) -> 65 | dispatch(Ref, NPartitions, 1, PartitionMessages, NextPid). 66 | 67 | -spec dispatch(reference(), pos_integer(), pos_integer(), [data_message(),...], pid()) -> 'ok'. 68 | %% @doc Worker function for {@link dispatch/4}; recursively dispatches each data message to `NextPid'. 69 | dispatch(_Ref,_NPartitions, _Idx, [], _NextPid) -> 70 | ok; 71 | dispatch(Ref, NPartitions, Idx, [PartitionMessage|PartitionMessages], NextPid) -> 72 | PartitionMessage1 = sk_data:push({decomp, Ref, Idx, NPartitions}, PartitionMessage), 73 | sk_tracer:t(50, self(), NextPid, {?MODULE, data}, [{partition, PartitionMessage1}]), 74 | NextPid ! PartitionMessage1, 75 | dispatch(Ref, NPartitions, Idx+1, PartitionMessages, NextPid). 76 | -------------------------------------------------------------------------------- /src/sk_cluster_recomp.erl: -------------------------------------------------------------------------------- 1 | %%%---------------------------------------------------------------------------- 2 | %%% @author Sam Elliott 3 | %%% @copyright 2012 University of St Andrews (See LICENCE) 4 | %%% @headerfile "skel.hrl" 5 | %%% 6 | %%% @doc This module contains what happens in the recomposition process of a 7 | %%% Cluster wrapper. 8 | %%% 9 | %%% The cluster wrapper acts in a similar manner to the Map skeleton, but 10 | %%% allows the developer to customise the decomposition and recomposition 11 | %%% functions used. Inputs are decomposed according to the developer-defined 12 | %%% decomposition function, and then passed to the worker process running the 13 | %%% inner-workflow. 14 | %%% 15 | %%% The recomposition process recombines partite elements of each input 16 | %%% according to a developer-defined function. This function should compliment 17 | %%% the similarly developer-defined decomposition function. 18 | %%% 19 | %%% @end 20 | %%%---------------------------------------------------------------------------- 21 | -module(sk_cluster_recomp). 22 | 23 | -export([ 24 | start/2 25 | ]). 26 | 27 | -include("skel.hrl"). 28 | 29 | -ifdef(TEST). 30 | -compile(export_all). 31 | -endif. 32 | 33 | -spec start(recomp_fun(), pid()) -> 'eos'. 34 | %% @doc Initialises the recomposition process. 35 | %% 36 | %% The recomposition process listens for data messages, combining all messages 37 | %% that are a partite element of the same original input for all inputs. this 38 | %% recomposition is powered by the recomposition function given by `Recomp'. 39 | start(Recomp, NextPid) -> 40 | sk_tracer:t(75, self(), {?MODULE, start}, [{next_pid, NextPid}]), 41 | DataRecompFun = sk_data:recomp_with(Recomp), 42 | loop(dict:new(), DataRecompFun, NextPid). 43 | 44 | -spec loop(dict:dict(), data_recomp_fun(), pid()) -> 'eos'. 45 | %% @doc Worker function for {@link start/2}; recursively receives and combines 46 | %% messages using the recomposition function under `DataRecompFun'. Once all 47 | %% partite elements for each original input have been received and merged, the 48 | %% recomposed message is forwarded to `NextPid'. 49 | loop(Dict, DataRecompFun, NextPid) -> 50 | receive 51 | {data, _, _} = PartitionMessage -> 52 | {{decomp, Ref, Idx, NPartitions}, PartitionMessage1} = sk_data:pop(PartitionMessage), 53 | Dict1 = store(Ref, Idx, NPartitions, PartitionMessage1, Dict), 54 | Dict2 = combine_and_forward(Ref, Dict1, DataRecompFun, NextPid), 55 | loop(Dict2, DataRecompFun, NextPid); 56 | {system, eos} -> 57 | sk_tracer:t(75, self(), NextPid, {?MODULE, system}, [{msg, eos}]), 58 | NextPid ! {system, eos}, 59 | eos 60 | end. 61 | 62 | -spec store(reference(), pos_integer(), pos_integer(), data_message(), dict:dict()) -> dict:dict(). 63 | %% @doc Facilitates the storing of all partite and semi-combined messages, for each input. 64 | %% 65 | %% The total number of partite elements expected, the partite element data 66 | %% messages themselves, and the number of these received, are stored for each 67 | %% of the original inputs -- as grouped by `Ref' -- are stored. The updated 68 | %% dictionary is returned. 69 | store(Ref, Idx, NPartitions, PartitionMessage, Dict) -> 70 | Dict1 = dict:store({Ref, expecting}, NPartitions, Dict), 71 | Dict2 = dict:store({Ref, Idx}, PartitionMessage, Dict1), 72 | dict:update_counter({Ref, received}, 1, Dict2). 73 | 74 | -spec combine_and_forward(reference(), dict:dict(), data_recomp_fun(), pid()) -> dict:dict(). 75 | %% @doc Attempts to find the grouping reference under `Ref' in the given 76 | %% dictionary. If that reference is found, all message parts are combined 77 | %% using the recomposition function given under `DataCombinerFun'. 78 | combine_and_forward(Ref, Dict, DataCombinerFun, NextPid) -> 79 | case dict:find({Ref, expecting}, Dict) of 80 | error -> Dict; 81 | {ok, NPartitions} -> combine_and_forward(Ref, Dict, NPartitions, DataCombinerFun, NextPid) 82 | end. 83 | 84 | -spec combine_and_forward(reference(), dict:dict(), pos_integer(), data_recomp_fun(), pid()) -> dict:dict(). 85 | %% @doc Worker function for {@link combine_and_forward/4}; attempts to 86 | %% recompose a partite elements for a given original input, as indicated by 87 | %% `Ref'. Should all partite elements be stored in the dictionary, they are 88 | %% retrieved and recomposed. The result of which is sent to `NextPid', and 89 | %% those partite elements removed from the dictionary. The updated dictionary 90 | %% is returned. 91 | combine_and_forward(Ref, Dict, NPartitions, DataCombinerFun, NextPid) -> 92 | RcvdPartitions = dict:fetch({Ref, received}, Dict), 93 | if 94 | RcvdPartitions == NPartitions -> 95 | PartitionMessages = fetch_partitions(Ref, NPartitions, Dict, []), 96 | DataMessage = apply(DataCombinerFun, [PartitionMessages]), 97 | sk_tracer:t(50, self(), NextPid, {?MODULE, data}, [{ref, Ref}, {output, DataMessage}, {partitions, PartitionMessages}]), 98 | NextPid ! DataMessage, 99 | purge_partitions(Ref, NPartitions, Dict); 100 | true -> 101 | Dict 102 | end. 103 | 104 | -spec fetch_partitions(reference(), non_neg_integer(), dict:dict(), [any()]) -> [any()]. 105 | %% @doc Retrieves and returns a list of all entries with the same reference in 106 | %% the specified dictionary. 107 | fetch_partitions(_Ref, 0, _Dict, Acc) -> 108 | Acc; 109 | fetch_partitions(Ref, NPartitions, Dict, Acc) -> 110 | {ok, Piece} = dict:find({Ref, NPartitions}, Dict), 111 | fetch_partitions(Ref, NPartitions-1, Dict, [Piece|Acc]). 112 | 113 | -spec purge_partitions(reference(), non_neg_integer(), dict:dict()) -> dict:dict(). 114 | %% @doc Recursively removes all entries with the same reference in a given 115 | %% dictionary. 116 | purge_partitions(Ref, 0, Dict) -> 117 | Dict1 = dict:erase({Ref, expecting}, Dict), 118 | Dict2 = dict:erase({Ref, received}, Dict1), 119 | Dict2; 120 | purge_partitions(Ref, NPartitions, Dict) -> 121 | Dict1 = dict:erase({Ref, NPartitions}, Dict), 122 | purge_partitions(Ref, NPartitions-1, Dict1). 123 | -------------------------------------------------------------------------------- /src/sk_data.erl: -------------------------------------------------------------------------------- 1 | %%%---------------------------------------------------------------------------- 2 | %%% @author Sam Elliott 3 | %%% @copyright 2012 University of St Andrews (See LICENCE) 4 | %%% @headerfile "skel.hrl" 5 | %%% 6 | %%% @doc This module encapsulates all functions relating to data messages that 7 | %%% are sent around the system. 8 | %%% 9 | %%% Data messages are an instance of an applicative functor, allowing various 10 | %%% operations to happen in a very logical way. 11 | %%% 12 | %%% @end 13 | %%%---------------------------------------------------------------------------- 14 | -module(sk_data). 15 | 16 | -export([ 17 | fmap/1 18 | ,pure/1 19 | ,apply/1 20 | ,value/1 21 | ,push/2 22 | ,pop/1 23 | ,peek/1 24 | ,decomp_by/1 25 | ,recomp_with/1 26 | ,reduce_with/1 27 | ,filter_with/1 28 | ]). 29 | 30 | -include("skel.hrl"). 31 | 32 | -spec fmap(fun((any()) -> any())) -> data_fun(). 33 | %% @doc Allows a given function to be applied to a value in the form of a Skel 34 | %% data message. Preserves message identifiers. 35 | fmap(Fun) -> 36 | fun ({data, Value, Ids}) -> 37 | {data, Fun(Value), Ids} 38 | end. 39 | 40 | -spec pure(any()) -> data_message(). 41 | %% @doc Formats a piece of data under `D' of any data type as a data message. 42 | pure(D) -> 43 | {data, D, []}. 44 | 45 | -spec apply(data_message()) -> data_fun(). 46 | %% @doc Allows a given function, formatted as a data message, to be applied to 47 | %% a value in the form of a Skel data message. Preserves message identifiers. 48 | apply({data, Fun, _}) -> 49 | fmap(Fun). 50 | 51 | -spec value(data_message()) -> any(). 52 | %% @doc Extracts and returns a data message's value. 53 | value({data, Value, _Identifiers}) -> 54 | Value. 55 | 56 | -spec push(dm_identifier(), data_message()) -> data_message(). 57 | %% @doc Adds an identifier to a given data message. 58 | push(Identifier, {data, Value, Identifiers}) -> 59 | {data, Value, [Identifier|Identifiers]}. 60 | 61 | -spec pop(data_message()) -> {dm_identifier(), data_message()}. 62 | %% @doc Retrieves and removes the topmost identifier of a given message. 63 | %% Returns a tuple containing retrieved identifier, and the now-updated 64 | %% message. 65 | pop({data, Value, [Identifier|Identifiers]}) -> 66 | DM = {data, Value, Identifiers}, 67 | {Identifier, DM}. 68 | 69 | -spec peek(data_message()) -> {ok, dm_identifier()} | empty. 70 | %% @doc Retrieves, but does not remove, the topmost identifier of a given message. Returns only the identifier, or empty when no identifiers exist. 71 | peek({data,_Value, []}) -> 72 | empty; 73 | peek({data,_Value, [Identifier|_]}) -> 74 | {ok, Identifier}. 75 | 76 | -spec decomp_by(decomp_fun()) -> data_decomp_fun(). 77 | %% @doc Standardises the format of the developer-defined decomposition function under `DecompFun' for use in Skel. 78 | decomp_by(DecompFun) -> 79 | fun({data, Value, Ids}) -> 80 | Partitions = DecompFun(Value), 81 | [{data, X, Ids} || X <- Partitions] 82 | end. 83 | 84 | -spec recomp_with(recomp_fun()) -> data_recomp_fun(). 85 | %% @doc Standardises the format of the developer-defined recomposition function under `RecompFun' for use in Skel. 86 | recomp_with(RecompFun) -> 87 | fun([{data, _, Ids}|_] = DataMessages) -> 88 | Values = [value(X) || X <- DataMessages], 89 | {data, RecompFun(Values), Ids} 90 | end. 91 | 92 | -spec reduce_with(reduce_fun()) -> data_reduce_fun(). 93 | %% @doc Standardises the format of the developer-defined reduction function under `Reduce' for use in Skel. 94 | reduce_with(Reduce) -> 95 | fun({data, _, Ids} = DataMessage1, DataMessage2) -> 96 | {data, Reduce(value(DataMessage1), value(DataMessage2)), Ids} 97 | end. 98 | 99 | -spec filter_with(filter_fun()) -> data_filter_fun(). 100 | %% @doc Standardises the format of teh developer-defined constraint-checking function under `Filter' for use in Skel. 101 | filter_with(Filter) -> 102 | fun({data, _, _} = DataMessage1) -> 103 | Filter(value(DataMessage1)) 104 | end. 105 | -------------------------------------------------------------------------------- /src/sk_farm.erl: -------------------------------------------------------------------------------- 1 | %%%---------------------------------------------------------------------------- 2 | %%% @author Sam Elliott 3 | %%% @copyright 2012 University of St Andrews (See LICENCE) 4 | %%% @headerfile "skel.hrl" 5 | %%% 6 | %%% @doc This module contains the initialization logic of a Farm skeleton. 7 | %%% 8 | %%% A task farm has the most basic kind of stream parallelism - inputs are 9 | %%% sent to one of `n' replicas of the inner skeleton for processing. 10 | %%% 11 | %%% === Example === 12 | %%% 13 | %%% ```skel:run([{farm, [{seq, fun ?MODULE:p1/1}], 10}], Input)''' 14 | %%% 15 | %%% In this simple example, we produce a farm with ten workers to run the 16 | %%% sequential, developer-defined function `p/1' using the list of inputs 17 | %%% `Input'. 18 | %%% 19 | %%% @end 20 | %%%---------------------------------------------------------------------------- 21 | -module(sk_farm). 22 | 23 | -export([ 24 | make/2, 25 | make_hyb/4 26 | ]). 27 | 28 | -include("skel.hrl"). 29 | 30 | -spec make(pos_integer(), workflow()) -> maker_fun(). 31 | %% @doc Initialises a Farm skeleton given the number of workers and their 32 | %% inner-workflows, respectively. 33 | make(NWorkers, WorkFlow) -> 34 | fun(NextPid) -> 35 | CollectorPid = spawn(sk_farm_collector, start, [NWorkers, NextPid]), 36 | WorkerPids = sk_utils:start_workers(NWorkers, WorkFlow, CollectorPid), 37 | spawn(sk_farm_emitter, start, [WorkerPids]) 38 | end. 39 | 40 | -spec make_hyb(pos_integer(), pos_integer(), workflow(), workflow()) -> maker_fun(). 41 | make_hyb(NCPUWorkers, NGPUWorkers, WorkFlowCPU, WorkFlowGPU) -> 42 | fun(NextPid) -> 43 | CollectorPid = spawn(sk_farm_collector, start, [NCPUWorkers+NGPUWorkers, NextPid]), 44 | WorkerPids = sk_utils:start_workers_hyb(NCPUWorkers, NGPUWorkers, WorkFlowCPU, WorkFlowGPU, 45 | CollectorPid), 46 | spawn(sk_farm_emitter, start, [WorkerPids]) 47 | end. 48 | -------------------------------------------------------------------------------- /src/sk_farm_collector.erl: -------------------------------------------------------------------------------- 1 | %%%---------------------------------------------------------------------------- 2 | %%% @author Sam Elliott 3 | %%% @copyright 2012 University of St Andrews (See LICENCE) 4 | %%% @headerfile "skel.hrl" 5 | %%% 6 | %%% @doc This module contains the collector logic of a Farm skeleton. 7 | %%% 8 | %%% A task farm has the most basic kind of stream parallelism - inputs are 9 | %%% sent to one of `n' replicas of the inner skeleton for processing. 10 | %%% 11 | %%% The collector takes inputs off the inner-skeletons' output streams, sending 12 | %%% them out on the farm skeleton's output stream. It does not preserve 13 | %%% ordering. 14 | %%% 15 | %%% @end 16 | %%%---------------------------------------------------------------------------- 17 | -module(sk_farm_collector). 18 | 19 | -export([ 20 | start/2 21 | ]). 22 | 23 | -include("skel.hrl"). 24 | 25 | -spec start(pos_integer(), pid()) -> 'eos'. 26 | %% @doc Initialises the collector; forwards any and all output from the inner- 27 | %% workflow to the sink process at `NextPid'. 28 | start(NWorkers, NextPid) -> 29 | sk_tracer:t(75, self(), {?MODULE, start}, [{num_workers, NWorkers}, {next_pid, NextPid}]), 30 | loop(NWorkers, NextPid). 31 | 32 | -spec loop(pos_integer(), pid()) -> 'eos'. 33 | %% @doc Worker-function for {@link start/2}. Recursively receives, and 34 | %% forwards, any output messages from the inner-workflow. Halts when the `eos' 35 | %% system message is received, and only one active worker process remains. 36 | loop(NWorkers, NextPid) -> 37 | receive 38 | {data, _, _} = DataMessage -> 39 | sk_tracer:t(50, self(), NextPid, {?MODULE, data}, [{input, DataMessage}]), 40 | NextPid ! DataMessage, 41 | loop(NWorkers, NextPid); 42 | {system, eos} when NWorkers =< 1 -> 43 | sk_tracer:t(75, self(), NextPid, {?MODULE, system}, [{msg, eos}, {remaining, 0}]), 44 | NextPid ! {system, eos}, 45 | eos; 46 | {system, eos} -> 47 | sk_tracer:t(85, self(), {?MODULE, system}, [{msg, eos}, {remaining, NWorkers-1}]), 48 | loop(NWorkers-1, NextPid) 49 | end. 50 | 51 | -------------------------------------------------------------------------------- /src/sk_farm_emitter.erl: -------------------------------------------------------------------------------- 1 | %%%---------------------------------------------------------------------------- 2 | %%% @author Sam Elliott 3 | %%% @copyright 2012 University of St Andrews (See LICENCE) 4 | %%% @headerfile "skel.hrl" 5 | %%% 6 | %%% @doc This module contains the emitter logic of a Farm skeleton. 7 | %%% 8 | %%% A task farm has the most basic kind of stream parallelism - inputs are 9 | %%% sent to one of `n' replicas of the inner skeleton for processing. 10 | %%% 11 | %%% The emitter takes inputs off the skeleton's input stream and assigns each 12 | %%% one to one of the input streams of the 'n' inner skeletons. 13 | %%% 14 | %%% @end 15 | %%%---------------------------------------------------------------------------- 16 | -module(sk_farm_emitter). 17 | 18 | -export([ 19 | start/1 20 | ]). 21 | 22 | -include("skel.hrl"). 23 | 24 | -spec start([pid(),...]) -> 'eos'. 25 | %% @doc Initialises the emitter. Sends input as messages to the list of 26 | %% processes given by `Workers'. 27 | start(Workers) -> 28 | sk_tracer:t(75, self(), {?MODULE, start}, [{workers, Workers}]), 29 | loop(Workers). 30 | 31 | -spec loop([pid(),...]) -> 'eos'. 32 | %% @doc Inner-function to {@link start/1}; recursively captures input, sending %% that input onto the next worker in the list. Ceases only when the halt 33 | %% command is received. 34 | loop([Worker|Rest] = Workers) -> 35 | receive 36 | {data, _, _} = DataMessage -> 37 | sk_tracer:t(50, self(), Worker, {?MODULE, data}, [{input, DataMessage}]), 38 | Worker ! DataMessage, 39 | loop(Rest ++ [Worker]); 40 | {system, eos} -> 41 | sk_utils:stop_workers(?MODULE, Workers) 42 | end. 43 | -------------------------------------------------------------------------------- /src/sk_feedback.erl: -------------------------------------------------------------------------------- 1 | %%%---------------------------------------------------------------------------- 2 | %%% @author Sam Elliott 3 | %%% @copyright 2012 University of St Andrews (See LICENCE) 4 | %%% @headerfile "skel.hrl" 5 | %%% 6 | %%% @doc This module contains 'feedback' skeleton initialization logic. 7 | %%% 8 | %%% The Feedback skeleton repeatedly passes output from its inner-workflow 9 | %%% back into said workflow until a given constraint-checking function fails. 10 | %%% 11 | %%% === Example === 12 | %%% 13 | %%% ```skel:do([{feedback, [{seq, fun ?MODULE:succ/1}], fun ?MODULE:less_than/1}], Input).''' 14 | %%% 15 | %%% Here we use the feedback skeleton to ensure all elements of the list 16 | %%% Input are above a certain value. If any one element is equal to or 17 | %%% greater than that value, those elements will be recursively brought up 18 | %%% to that value. In the above example, the method `succ/1' adds 19 | %%% one to its input, where `less_than/1' acts as a check. 20 | %%% 21 | %%% @end 22 | %%%---------------------------------------------------------------------------- 23 | -module(sk_feedback). 24 | 25 | -export([ 26 | make/2 27 | ]). 28 | 29 | -include("skel.hrl"). 30 | 31 | -spec make(workflow(), filter_fun()) -> maker_fun(). 32 | %% @doc Initialises a Feedback skeleton. 33 | %% 34 | %% Constraint-checking filter processes are produced using the developer- 35 | %% defined function `FilterFun', and are used to check inputs according to 36 | %% said function. 37 | make(WorkFlow, FilterFun) -> 38 | fun(NextPid) -> 39 | Ref = make_ref(), 40 | CounterPid = spawn(sk_feedback_bicounter, start, []), 41 | FilterPid = spawn(sk_feedback_filter, start, [FilterFun, Ref, CounterPid, NextPid]), 42 | WorkerPid = sk_utils:start_worker(WorkFlow, FilterPid), 43 | spawn(sk_feedback_receiver, start, [Ref, CounterPid, FilterPid, WorkerPid]) 44 | end. 45 | -------------------------------------------------------------------------------- /src/sk_feedback_bicounter.erl: -------------------------------------------------------------------------------- 1 | %%%---------------------------------------------------------------------------- 2 | %%% @author Sam Elliott 3 | %%% @copyright 2012 University of St Andrews (See LICENCE) 4 | %%% @headerfile "skel.hrl" 5 | %%% 6 | %%% @doc This module contains 'feedback' skeleton counter logic. 7 | %%% 8 | %%% The Feedback skeleton repeatedly passes output from its inner-workflow 9 | %%% back into said workflow until a given constraint-checking function fails. 10 | %%% 11 | %%% It turns out in the feedback skeleton we have to maintain counts of numbers 12 | %%% inputs going through the inner skeleton and going through the feedback loop 13 | %%% in order to avoid race conditions upon end-of-stream. This is the dual- 14 | %%% semaphore that we use to maintain these counts. 15 | %%% 16 | %%% 17 | %%% @end 18 | %%%---------------------------------------------------------------------------- 19 | -module(sk_feedback_bicounter). 20 | 21 | -export([ 22 | start/0 23 | ,subscribe/1 24 | ,cast/2 25 | ]). 26 | 27 | -export_type([ 28 | message/0 29 | ]). 30 | 31 | -include("skel.hrl"). 32 | 33 | 34 | -type counters() :: {number(), number()}. 35 | -type message() :: {system, {counter, command(), pid()}}. 36 | -type command() :: {bi_command(), bi_command()} 37 | | subscribe. 38 | -type bi_command() :: id 39 | | incr 40 | | decr. 41 | 42 | -spec start() -> ok. 43 | %% @doc Initialises a counter. 44 | start() -> 45 | sk_tracer:t(90, self(), {?MODULE, start}, [{a, 0}, {b, 0}, {subscribers, []}]), 46 | loop({0, 0}, []). 47 | 48 | -spec subscribe(pid()) -> ok. 49 | %% @doc Subscribes the calling process to the counter at `CounterPid'. 50 | %% Subscribed processes will receive messages from the counter when it updates. 51 | subscribe(CounterPid) -> 52 | sk_tracer:t(85, self(), CounterPid, {?MODULE, system}, [{msg, counter}, {command, subscribe}]), 53 | CounterPid ! {system, {counter, subscribe, self()}}, 54 | ok. 55 | 56 | -spec cast(pid(), {bi_command(), bi_command()}) -> ok. 57 | %% @doc The commands given by a process under `Commands' are sent to be 58 | %% executed by the counter at `CounterPid'. 59 | cast(CounterPid, Commands) -> 60 | sk_tracer:t(85, self(), CounterPid, {?MODULE, system}, [{msg, counter}, {command, Commands}]), 61 | CounterPid ! {system, {counter, Commands, self()}}, 62 | ok. 63 | 64 | -spec loop(counters(), [pid()]) -> ok. 65 | %% @doc Recursively receives commands to update the counter tuple under 66 | %% `Counters'. 67 | %% 68 | %% Adds a subsriber to the list under `Subscribers', executes a given command, 69 | %% and halts the counter when both the `eos' system message is received and 70 | %% the counters are both zero. 71 | loop(Counters, Subscribers) -> 72 | receive 73 | {system, {counter, subscribe, Pid}} -> 74 | loop(Counters, [Pid|Subscribers]); 75 | {system, {counter, Command, _}} -> 76 | Counters1 = command(Counters, Command, Subscribers), 77 | loop(Counters1, Subscribers); 78 | {system, eos} when Counters =:= {0,0} -> 79 | ok; 80 | {system, _} -> 81 | loop(Counters, Subscribers) 82 | end. 83 | 84 | -spec command(counters(), {bi_command(), bi_command()}, [pid()]) -> counters(). 85 | %% @doc Executes a given command for the specified counters, notifying 86 | %% subscribers of the change. 87 | command({CA,CB}, {CommandA, CommandB}, Subscribers) -> 88 | Counters = {execute(CA, CommandA), execute(CB, CommandB)}, 89 | sk_tracer:t(85, self(), {?MODULE, execute}, [{command, {CommandA, CommandB}}, {old, {CA,CB}}, {new, Counters}]), 90 | notify_subscribers(Counters, Subscribers), 91 | Counters; 92 | command(Counters, _, _) -> 93 | Counters. 94 | 95 | -spec execute(number(), bi_command()) -> number(). 96 | %% @doc Evaluates an individual command given the counter part `C'. 97 | execute(C, id) -> C; 98 | execute(C, incr) -> C+1; 99 | execute(C, decr) -> C-1; 100 | execute(C, _) -> C. 101 | 102 | -spec notify_subscribers(counters(), [pid()]) -> ok. 103 | %% @doc Recursively sends the updated counter under `Counters' to each 104 | %% subscriber in the attached list of subscribers. 105 | notify_subscribers(_, []) -> 106 | ok; 107 | notify_subscribers(Counters, [Subscriber|Subscribers]) -> 108 | sk_tracer:t(90, self(), Subscriber, {?MODULE, system}, [{msg, counter}, {counters, Counters}]), 109 | Subscriber ! {system, {counter, Counters, self()}}, 110 | notify_subscribers(Counters, Subscribers). 111 | -------------------------------------------------------------------------------- /src/sk_feedback_filter.erl: -------------------------------------------------------------------------------- 1 | %%%---------------------------------------------------------------------------- 2 | %%% @author Sam Elliott 3 | %%% @copyright 2012 University of St Andrews (See LICENCE) 4 | %%% @headerfile "skel.hrl" 5 | %%% 6 | %%% @doc This module contains 'feedback' skeleton filter logic. 7 | %%% 8 | %%% The Feedback skeleton repeatedly passes output from its inner-workflow 9 | %%% back into said workflow until a given constraint-checking function fails. 10 | %%% 11 | %%% The filter sends the inputs back to the start of the inner skeleton should 12 | %%% they pass a given condition; forwarding them onwards otherwise. 13 | %%% 14 | %%% @end 15 | %%%---------------------------------------------------------------------------- 16 | -module(sk_feedback_filter). 17 | 18 | -export([ 19 | start/4 20 | ]). 21 | 22 | -include("skel.hrl"). 23 | 24 | -define(counter, sk_feedback_bicounter). 25 | 26 | -spec start(filter_fun(), reference(), pid(), pid()) -> 'eos'. 27 | %% @doc Initialises the constraint-checking filter process. 28 | %% 29 | %% The developer-defined function represented by `FilterFun' serves to check 30 | %% the desired constraint. A reference under `Ref' serves to group inputs, 31 | %% workers and administrative processes to avoid conflicts with multiple 32 | %% feedback skeletons. The counter process, keeping track of how many inputs 33 | %% are currently passing through both the inner-workflow and the filter 34 | %% process, and sink Pids are provided under `CounterPid' and `NextPid' 35 | %% respectively. 36 | start(FilterFun, Ref, CounterPid, NextPid) -> 37 | sk_tracer:t(75, self(), {?MODULE, start}, [{reference, Ref}, 38 | {counter_pid, CounterPid}, 39 | {next_pid, NextPid}]), 40 | ReceiverPid = setup(self(), Ref), 41 | DataFilterFun = sk_data:filter_with(FilterFun), 42 | loop(DataFilterFun, CounterPid, ReceiverPid, NextPid). 43 | 44 | -spec loop(data_filter_fun(), pid(), pid(), pid()) -> 'eos'. 45 | %% @doc Recursively receives messages from the worker process applying the 46 | %% inner-workflow. Messages are checked by the constraint function under 47 | %% `DataFilterFun', then passed back into the the inner-workflow or forwarded 48 | %% onwards to the process given by `NextPid'. 49 | loop(DataFilterFun, CounterPid, ReceiverPid, NextPid) -> 50 | receive 51 | {data,_,_} = DataMessage -> 52 | case DataFilterFun(DataMessage) of 53 | true -> feedback(DataMessage, CounterPid, ReceiverPid); 54 | false -> forward(DataMessage, CounterPid, NextPid) 55 | end, 56 | loop(DataFilterFun, CounterPid, ReceiverPid, NextPid); 57 | {system, eos} -> 58 | sk_tracer:t(75, self(), NextPid, {?MODULE, system}, [{msg, eos}]), 59 | CounterPid ! {system, eos}, 60 | NextPid ! {system, eos}, 61 | eos 62 | end. 63 | 64 | -spec setup(pid(), reference()) -> pid(). 65 | %% @doc Acknowledges the registration request of a receiver process, as described in {@link sk_feedback_receiver}. 66 | setup(FilterPid, Ref) -> 67 | receive 68 | {system, {feedback_setup, ReceiverPid, Ref}} -> 69 | sk_tracer:t(75, FilterPid, ReceiverPid, {?MODULE, system}, [{msg, feedback_reply}, {filter_pid, FilterPid}, {reference, Ref}]), 70 | ReceiverPid ! {system, {feedback_reply, FilterPid, Ref}}, 71 | ReceiverPid 72 | end. 73 | 74 | % counter: {no. of inputs in inner skeleton, no. of inputs in feedback loop} 75 | -spec feedback(data_message(), pid(), pid()) -> ok. 76 | %% @doc Inputs passing the constraint-checking function are passed here for processing. 77 | %% 78 | %% This function updates the counter at `CounterPid', and reformats the input 79 | %% message given by `DataMessage' for the inner-workflow. The now-reformatted 80 | %% message is sent to the receiver process at `Pid'. 81 | feedback(DataMessage, CounterPid, Pid) -> 82 | ?counter:cast(CounterPid, {decr, incr}), 83 | DataMessage1 = sk_data:push(feedback, DataMessage), 84 | sk_tracer:t(50, self(), Pid, {?MODULE, data}, [{output, DataMessage1}]), 85 | Pid ! DataMessage1, 86 | ok. 87 | 88 | -spec forward(data_message(), pid(), pid()) -> ok. 89 | %% @doc New inputs are passed here for processing. 90 | %% 91 | %% This function simply passes the message under `DataMessage' onto the sink 92 | %% process at `Pid' as output. The counter located at `CounterPid' is updated 93 | %% to reflect this. 94 | forward(DataMessage, CounterPid, Pid) -> 95 | ?counter:cast(CounterPid, {decr, id}), 96 | sk_tracer:t(50, self(), Pid, {?MODULE, data}, [{output, DataMessage}]), 97 | Pid ! DataMessage, 98 | ok. 99 | -------------------------------------------------------------------------------- /src/sk_feedback_receiver.erl: -------------------------------------------------------------------------------- 1 | %%%---------------------------------------------------------------------------- 2 | %%% @author Sam Elliott 3 | %%% @copyright 2012 University of St Andrews (See LICENCE) 4 | %%% @headerfile "skel.hrl" 5 | %%% 6 | %%% @doc This module contains Feedback skeleton filter logic. 7 | %%% 8 | %%% The Feedback skeleton repeatedly passes output from its inner-workflow 9 | %%% back into said workflow until a given constraint-checking function fails. 10 | %%% 11 | %%% The receiver receives inputs from the previous skeleton or the feedback 12 | %%% filter and sends them through the inner-workflow. 13 | %%% 14 | %%% @end 15 | %%%---------------------------------------------------------------------------- 16 | -module(sk_feedback_receiver). 17 | 18 | -export([ 19 | start/4 20 | ]). 21 | 22 | -include("skel.hrl"). 23 | 24 | -define(counter, sk_feedback_bicounter). 25 | 26 | -spec start(reference(), pid(), pid(), pid()) -> 'eos'. 27 | %% @doc Begins the receiver, taking recycled and non-recycled input and 28 | %% passing them to a worker for application to the inner-workflow. 29 | start(Ref, CounterPid, FilterPid, WorkerPid) -> 30 | sk_tracer:t(75, self(), {?MODULE, start}, [{reference, Ref}, 31 | {counter_pid, CounterPid}, 32 | {filter_pid, FilterPid}, 33 | {worker_pid, WorkerPid}]), 34 | setup(Ref, self(), FilterPid), 35 | loop(false, CounterPid, WorkerPid). 36 | 37 | 38 | % Handles messages from the previous skeleton(?) or the feedback filter. 39 | % case 1: std. data message. Looks at first identifier in message. If not empty: decrease the counter; otherwise do nothing. Then passes the message on to the worker. 40 | % case 2: end of stream. Receiver subscribes to the counter. So that it can find out next time it receives one if all inputs have left. 41 | % case 3: counter. Continues looping until counter = {0,0} at which point you halt. 42 | % case 4: system message. Pass it along. 43 | -spec loop(boolean(), pid(), pid()) -> 'eos'. 44 | %% @doc Inner-function for {@link start/4}; recursively receives input as data %% messages, dealing with each accordingly. 45 | %% 46 | %% A regular data message handling input is managed as to whether it has 47 | %% already passed through the inner-workflow, or is a new input. A counter 48 | %% data message may also be received, assisting with termination. 49 | loop(EosRecvd, CounterPid, WorkerPid) -> 50 | receive 51 | {data,_,_} = DataMessage -> 52 | case sk_data:peek(DataMessage) of 53 | {ok, feedback} -> from_feedback(DataMessage, CounterPid, WorkerPid); 54 | _ -> from_regular(DataMessage, CounterPid, WorkerPid) 55 | end, 56 | loop(EosRecvd, CounterPid, WorkerPid); 57 | {system, eos} -> 58 | ?counter:subscribe(CounterPid), 59 | loop(true, CounterPid, WorkerPid); 60 | {system, {counter, Counters, _}} -> 61 | case Counters of 62 | {0,0} -> 63 | WorkerPid ! {system, eos}, 64 | eos; 65 | _ -> 66 | loop(EosRecvd, CounterPid, WorkerPid) 67 | end; 68 | {system, _} = SysMsg -> 69 | WorkerPid ! SysMsg, 70 | loop(EosRecvd, CounterPid, WorkerPid) 71 | end. 72 | 73 | -spec setup(reference(), pid(), pid()) -> 'ok'. 74 | %% @doc Associates the current receiver process with the constraint-checking 75 | %% filter process given by `FilterPid'. 76 | setup(Ref, ReceiverPid, FilterPid) -> 77 | sk_tracer:t(75, self(), FilterPid, {?MODULE, system}, [{msg, feedback_setup}, {receiver_pid, ReceiverPid}, {reference, Ref}]), 78 | FilterPid ! {system, {feedback_setup, ReceiverPid, Ref}}, 79 | receive 80 | {system, {feedback_reply, FilterPid, Ref}} -> 81 | ok 82 | end. 83 | 84 | -spec from_feedback(data_message(), pid(), pid()) -> ok. 85 | %% @doc Re-formats a former-output message under `DataMessage', updates the 86 | %% counter, and passes the message to the worker process at `WorkerPid'. 87 | from_feedback(DataMessage, CounterPid, WorkerPid) -> 88 | ?counter:cast(CounterPid, {incr, decr}), 89 | {feedback, DataMessage1} = sk_data:pop(DataMessage), 90 | sk_tracer:t(50, self(), WorkerPid, {?MODULE, data}, [{output, DataMessage1}]), 91 | WorkerPid ! DataMessage1, 92 | ok. 93 | 94 | -spec from_regular(data_message(), pid(), pid()) -> ok. 95 | %% @doc Forwards a new input message under `DataMessage', updates the counter, 96 | %% and passes the message onwards to the worker process at `WorkerPid'. 97 | from_regular(DataMessage, CounterPid, WorkerPid) -> 98 | ?counter:cast(CounterPid, {incr, id}), 99 | sk_tracer:t(50, self(), WorkerPid, {?MODULE, data}, [{output, DataMessage}]), 100 | WorkerPid ! DataMessage, 101 | ok. 102 | 103 | -------------------------------------------------------------------------------- /src/sk_map.erl: -------------------------------------------------------------------------------- 1 | %%%---------------------------------------------------------------------------- 2 | %%% @author Sam Elliott 3 | %%% @copyright 2012 University of St Andrews (See LICENCE) 4 | %%% @headerfile "skel.hrl" 5 | %%% 6 | %%% @doc This module contains the Map skeleton initialisation logic. 7 | %%% 8 | %%% The Map skeleton is a parallel map. The skeleton applies a given function 9 | %%% to the elements within one or more lists. 10 | %%% 11 | %%% This implementation assumes a list of lists as input, where the 12 | %%% decomposition of said input may be expressed as the identity function. 13 | %%% Whilst this implementation of Map usually determines the number of worker 14 | %%% processes it needs automatically, the developer may explicitly set this, 15 | %%% as in {@link sk_farm}. 16 | %%% 17 | %%% 18 | %%% === Example === 19 | %%% 20 | %%% ```skel:do([{map, [{seq, fun ?MODULE:f/1}]}], Input).''' 21 | %%% 22 | %%% Here we use a Map skeleton to perform a function `f/1' over all 23 | %%% elements for all lists represented by `Input'. Returned, we receive a 24 | %%% list of lists the same as `Input' itself, bar that the elements of 25 | %%% each are the result of their application to `f/1'. 26 | %%% 27 | %%% In this example we note that the number of worker processes the Map 28 | %%% skeleton uses is determined by the length of the longest list in 29 | %%% `Input'. To constrain, or otherwise set this value, we might add an 30 | %%% extra term to the Map tuple. 31 | %%% 32 | %%% ```skel:do([{map, [{seq, fun ?MODULE:f/1}], 10}], Input).''' 33 | %%% 34 | %%% Using the same example, we now note that the number of worker 35 | %%% processes used is set to ten. Performance comparisons between these 36 | %%% two depends heavily on the chosen `Input', and the machine on which it 37 | %%% runs. 38 | %%% 39 | %%% @end 40 | %%%---------------------------------------------------------------------------- 41 | 42 | -module(sk_map). 43 | 44 | -export([make/1, make/2, make_hyb/4]). 45 | 46 | -include("skel.hrl"). 47 | 48 | -spec make(workflow()) -> maker_fun(). 49 | %% @doc Initialises an instance of the Map skeleton ready to receive inputs, 50 | %% where the number of worker processes is automatic. The function or 51 | %% functions to be applied to said inputs are given under `WorkFlow'. 52 | %% 53 | %% A combiner, or recomposition, process is created and acts as a sink for the 54 | %% workers. These workers are generated and managed from within a 55 | %% {@link sk_map_partitioner} process. 56 | make(WorkFlow) -> 57 | fun(NextPid) -> 58 | CombinerPid = spawn(sk_map_combiner, start, [NextPid]), 59 | spawn(sk_map_partitioner, start, [auto, WorkFlow, CombinerPid]) 60 | end. 61 | 62 | 63 | -spec make(workflow(), pos_integer()) -> maker_fun(). 64 | %% @doc Initialises an instance of the Map skeleton ready to receive inputs, 65 | %% using a given number of worker processes. This number is specified under 66 | %% `NWorkers', and the function or functions to be applied to any and all 67 | %% inputs are given by `WorkFlow'. 68 | %% 69 | %% A combiner, or recomposition, process is created, and acts as a sink for 70 | %% the workers. These workers are initialised with the specified workflow, and 71 | %% their Pids passed to a {@link sk_map_partitioner} process. 72 | make(WorkFlow, NWorkers) -> 73 | fun(NextPid) -> 74 | CombinerPid = spawn(sk_map_combiner, start, [NextPid, NWorkers]), 75 | WorkerPids = sk_utils:start_workers(NWorkers, WorkFlow, CombinerPid), 76 | spawn(sk_map_partitioner, start, [man, WorkerPids, CombinerPid]) 77 | end. 78 | 79 | -spec make_hyb(workflow(), workflow(), pos_integer(), pos_integer()) -> maker_fun(). 80 | %% @doc Initialises an instance of the Hybrid Map skeleton ready to receive inputs, 81 | %% using a given number of CPU and GPU worker processes. These numbers are specified under 82 | %% `NCPUWorkers' and `NGPUWorkers', and the CPU and GPU versions of the function 83 | %% to be applied to inputs are given by `WorkFlowCPU' and `WorkFlowGPU'. 84 | %% 85 | %% A combiner, or recomposition, process is created, and acts as a sink for 86 | %% the workers. These workers are initialised with the specified workflow, and 87 | %% their Pids passed to a {@link sk_map_partitioner} process. 88 | make_hyb(WorkFlowCPU, WorkFlowGPU, NCPUWorkers, NGPUWorkers) -> 89 | fun(NextPid) -> 90 | CombinerPid = spawn(sk_map_combiner, start, [NextPid, NCPUWorkers+NGPUWorkers]), 91 | {CPUWorkerPids, GPUWorkerPids} = sk_utils:start_workers_hyb(NCPUWorkers, NGPUWorkers, WorkFlowCPU, WorkFlowGPU, CombinerPid), 92 | spawn(sk_map_partitioner, start_hyb, [man, CPUWorkerPids, GPUWorkerPids, CombinerPid]) 93 | end. 94 | 95 | %make_hyb(WorkFlowCPU, WorkFlowGPU) -> 96 | % fun(NextPid) -> 97 | % CombinerPid = spawn(sk_map_combiner, start, [NextPid]), 98 | % spawn(sk_map_partitioner, start, [auto, [{seq, fun(X) -> sk_utils:hyb_worker(WorkFlowCPU, WorkFlowGPU, X) end}], 99 | % CombinerPid]) 100 | % end. 101 | 102 | 103 | 104 | 105 | -------------------------------------------------------------------------------- /src/sk_map_combiner.erl: -------------------------------------------------------------------------------- 1 | %%%---------------------------------------------------------------------------- 2 | %%% @author Sam Elliott 3 | %%% @copyright 2012 University of St Andrews (See LICENCE) 4 | %%% @headerfile "skel.hrl" 5 | %%% 6 | %%% @doc This module contains the Map skeleton combiner logic. 7 | %%% 8 | %%% The Map skeleton is a parallel map. The skeleton applies a given function 9 | %%% to the elements within one or more lists. 10 | %%% 11 | %%% The combiner receives each partite element of each input following the 12 | %%% original element's application to the given workflow. This module collects 13 | %%% each resulting partite element of all inputs and restores the structure of 14 | %%% the original inputs. 15 | %%% 16 | %%% Similar to {@link sk_map_partitioner}, this module supports both the 17 | %%% automatic creation of workers, and the ability to define the number used. 18 | %%% 19 | %%% @end 20 | %%%---------------------------------------------------------------------------- 21 | 22 | -module(sk_map_combiner). 23 | 24 | -export([start/1, start/2]). 25 | 26 | -include("skel.hrl"). 27 | 28 | -spec start(pid()) -> 'eos'. 29 | %% @doc Initialises the recomposition process for when the number of workers 30 | %% is not set by the developer. 31 | %% 32 | %% Recomposition consists of rebuilding 33 | %% a list from its elements, in the correct order. For each set of elements, a 34 | %% {@link data_message()} is produced and sent to the process given by 35 | %% `NextPid'. 36 | start(NextPid) -> 37 | sk_tracer:t(75, self(), {?MODULE, start}, [{next_pid, NextPid}]), 38 | loop(auto, 0, 0, dict:new(), recomp_with(), NextPid). 39 | 40 | 41 | -spec start(pid(), pos_integer()) -> 'eos'. 42 | %% @doc Initialises the recomposition process for when the number of workers 43 | %% is set by the developer. 44 | %% 45 | %% Recomposition similarly consists of rebuilding a list from its elements, in 46 | %% the correct order. For each set of elements a {@link data_message()} is 47 | %% produced and sent to the process given by `NextPid'. 48 | start(NextPid, NWorkers) -> 49 | sk_tracer:t(75, self(), {?MODULE, start}, [{next_pid, NextPid}]), 50 | loop(man, NWorkers, 0, dict:new(), recomp_with(), NextPid). 51 | 52 | 53 | -spec loop(atom(), non_neg_integer(), non_neg_integer(), dict:dict(), data_recomp_fun(), pid()) -> 'eos'. 54 | %% @doc Recursively receives and stores messages until groups of said messages 55 | %% may be recomposed and sent. Serves to stop all processes once all inputs 56 | %% have been processed. 57 | %% 58 | %% The first clause is used when the number of workers is left unspecified by 59 | %% the developer. The second, where it is specified. 60 | loop(auto, TotWorkers, DeadWorkers, Dict, DataCombinerFun, NextPid) -> 61 | receive 62 | {data, _, _} = PartitionMessage -> 63 | {{decomp, Ref, Idx, NPartitions}, PartitionMessage1} = sk_data:pop(PartitionMessage), 64 | Dict1 = store(Ref, Idx, NPartitions, PartitionMessage1, Dict), 65 | Dict2 = combine_and_forward(Ref, Dict1, DataCombinerFun, NextPid), 66 | TotWorkers1 = new_total_workers(TotWorkers, NPartitions), 67 | loop(auto, TotWorkers1, DeadWorkers, Dict2, DataCombinerFun, NextPid); 68 | {system, eos} when DeadWorkers+1 >= TotWorkers -> 69 | sk_tracer:t(75, self(), NextPid, {?MODULE, system}, [{msg, eos}, {total, TotWorkers}, {dead, DeadWorkers+1}]), 70 | NextPid ! {system, eos}, 71 | eos; 72 | {system, eos} -> 73 | sk_tracer:t(85, self(), {?MODULE, system}, [{msg, eos}, {total, TotWorkers}, {dead, DeadWorkers+1}]), 74 | loop(auto, TotWorkers, DeadWorkers+1, Dict, DataCombinerFun, NextPid) 75 | end; 76 | loop(man, TotWorkers, DeadWorkers, Dict, DataCombinerFun, NextPid) -> 77 | receive 78 | {data, _, _} = PartitionMessage -> 79 | {{decomp, Ref, Idx, NPartitions}, PartitionMessage1} = sk_data:pop(PartitionMessage), 80 | Dict1 = store(Ref, Idx, NPartitions, PartitionMessage1, Dict), 81 | Dict2 = combine_and_forward(Ref, Dict1, DataCombinerFun, NextPid), 82 | loop(man, TotWorkers, DeadWorkers, Dict2, DataCombinerFun, NextPid); 83 | {system, eos} when DeadWorkers+1 >= TotWorkers -> 84 | sk_tracer:t(75, self(), NextPid, {?MODULE, system}, [{msg, eos}, {total, TotWorkers}, {dead, DeadWorkers+1}]), 85 | NextPid ! {system, eos}, 86 | eos; 87 | {system, eos} -> 88 | sk_tracer:t(85, self(), {?MODULE, system}, [{msg, eos}, {total, TotWorkers}, {dead, DeadWorkers+1}]), 89 | loop(man, TotWorkers, DeadWorkers+1, Dict, DataCombinerFun, NextPid) 90 | end. 91 | 92 | 93 | -spec recomp_with() -> data_recomp_fun(). 94 | %% @doc Provides the recomposition function and means to merge many inputs 95 | %% into one. This appends each individual `DataMessage', in order, to a list. 96 | %% This list is wrapped in a single {@link data_message()}. 97 | recomp_with() -> 98 | fun([{data, _, Ids}|_] = DataMessages) -> 99 | {data, [Value || {_, Value, _} <- DataMessages], Ids} 100 | end. 101 | 102 | 103 | -spec new_total_workers(non_neg_integer(), non_neg_integer()) -> non_neg_integer(). 104 | %% @doc Returns the total number of workers used by the skeleton. Employed 105 | %% when the number of workers is automatically determined. 106 | new_total_workers(TotWorkers, NPartitions) when NPartitions > TotWorkers -> 107 | NPartitions; 108 | new_total_workers(TotWorkers, _NPartitions) -> 109 | TotWorkers. 110 | 111 | 112 | -spec store(reference(), pos_integer(), pos_integer(), data_message(), dict:dict()) -> dict:dict(). 113 | %% @doc Stores in a dictionary the total number of partitions, `NPartitions', 114 | %% expected; all messages heretofore received; and the number said received 115 | %% messages, for the original input under the reference given by `Ref'. The 116 | %% updated dictionary, using `Dict' as a base, is returned. 117 | store(Ref, Idx, NPartitions, PartitionMessage, Dict) -> 118 | Dict1 = dict:store({Ref, expecting}, NPartitions, Dict), 119 | Dict2 = dict:store({Ref, Idx}, PartitionMessage, Dict1), 120 | dict:update_counter({Ref, received}, 1, Dict2). 121 | 122 | 123 | -spec combine_and_forward(reference(), dict:dict(), data_recomp_fun(), pid()) -> dict:dict(). 124 | %% @doc Attempts to find the reference as given by `Ref' in the specified 125 | %% dictionary. 126 | %% 127 | %% If said reference is found, {@link combine_and_forward/5} is used to 128 | %% attempt a recomposition of the partite elements stored as messages in 129 | %% `Dict'. 130 | combine_and_forward(Ref, Dict, DataCombinerFun, NextPid) -> 131 | case dict:find({Ref, expecting}, Dict) of 132 | error -> Dict; 133 | {ok, NPartitions} -> combine_and_forward(Ref, Dict, NPartitions, DataCombinerFun, NextPid) 134 | end. 135 | 136 | 137 | -spec combine_and_forward(reference(), dict:dict(), pos_integer(), data_recomp_fun(), pid()) -> dict:dict(). 138 | %% @doc Inner-function for {@link combine_and_forward/4} that attempts to 139 | %% restore a decomposed list from parts in a dictionary `Dict', whose 140 | %% reference is given by `Ref'. 141 | %% 142 | %% If all decomposed elements can be found, `combine_and_forward/5' retrieves 143 | %% them and applies the recomposition function under `DataCombinerFun'. The 144 | %% resulting data message is sent to the process represented by `NextPid', 145 | %% those messages deleted from the dictionary, and the dictionary returned. 146 | combine_and_forward(Ref, Dict, NPartitions, DataCombinerFun, NextPid) -> 147 | RcvdPartitions = dict:fetch({Ref, received}, Dict), 148 | if 149 | RcvdPartitions == NPartitions -> 150 | PartitionMessages = fetch_partitions(Ref, NPartitions, Dict, []), 151 | DataMessage = apply(DataCombinerFun, [PartitionMessages]), 152 | sk_tracer:t(50, self(), NextPid, {?MODULE, data}, [{ref, Ref}, {output, DataMessage}, {partitions, PartitionMessages}]), 153 | NextPid ! DataMessage, 154 | purge_partitions(Ref, NPartitions, Dict); 155 | true -> 156 | Dict 157 | end. 158 | 159 | 160 | -spec fetch_partitions(reference(), non_neg_integer(), dict:dict(), [any()]) -> [any()]. 161 | %% @doc Returns a list of all data messages in the given dictionary, whose 162 | %% reference is `Ref'. 163 | fetch_partitions(_Ref, 0, _Dict, Acc) -> 164 | Acc; 165 | fetch_partitions(Ref, NPartitions, Dict, Acc) -> 166 | {ok, Piece} = dict:find({Ref, NPartitions}, Dict), 167 | fetch_partitions(Ref, NPartitions-1, Dict, [Piece|Acc]). 168 | 169 | 170 | -spec purge_partitions(reference(), non_neg_integer(), dict:dict()) -> dict:dict(). 171 | %% @doc Recursively removes all entries with `Ref' as their reference in the 172 | %% given dictionary. 173 | purge_partitions(Ref, 0, Dict) -> 174 | Dict1 = dict:erase({Ref, expecting}, Dict), 175 | Dict2 = dict:erase({Ref, received}, Dict1), 176 | Dict2; 177 | purge_partitions(Ref, NPartitions, Dict) -> 178 | Dict1 = dict:erase({Ref, NPartitions}, Dict), 179 | purge_partitions(Ref, NPartitions-1, Dict1). 180 | -------------------------------------------------------------------------------- /src/sk_map_partitioner.erl: -------------------------------------------------------------------------------- 1 | %%%---------------------------------------------------------------------------- 2 | %%% @author Sam Elliot 3 | %%% @copyright 2012 University of St Andrews (See LICENCE) 4 | %%% @headerfile "skel.hrl" 5 | %%% 6 | %%% @doc This module contains the simple 'map' skeleton partitioner logic. 7 | %%% 8 | %%% The Map skeleton is a parallel map. The skeleton applies a given function 9 | %%% to the elements within one or more lists. 10 | %%% 11 | %%% The partitioner takes the input list, dispatching each partite element to 12 | %%% a pool of worker processes. These worker processes apply the 13 | %%% developer-defined function to each partite element. 14 | %%% 15 | %%% This module supports both the automatic creation of worker processes, and 16 | %%% the ability to define the exact number to be used. With the former, the 17 | %%% minimal number of workers are created for all inputs. This is given by the 18 | %%% number of elements in the longest list. 19 | %%% 20 | %%% @end 21 | %%%---------------------------------------------------------------------------- 22 | 23 | -module(sk_map_partitioner). 24 | 25 | -export([ 26 | start/3, 27 | start_hyb/4 28 | ]). 29 | 30 | -include("skel.hrl"). 31 | 32 | 33 | -spec start(atom(), workflow() | [pid()], pid()) -> 'eos'. 34 | %% @doc Starts the recursive partitioning of inputs. 35 | %% 36 | %% If the number of workers to be used is specified, a list of Pids for those 37 | %% worker processes are received through `WorkerPids' and the second clause 38 | %% used. Alternatively, a workflow is received as `Workflow' and the first 39 | %% clause is used. In the case of the former, the workers have already 40 | %% been initialised with their workflows and so the inclusion of the 41 | %% `WorkFlow' argument is unneeded in this clause. 42 | %% 43 | %% The atoms `auto' and `main' are used to determine whether a list of worker 44 | %% Pids or a workflow is received. `CombinerPid' specifies the Pid of the 45 | %% process that will recompose the partite elements following their 46 | %% application to the Workflow. 47 | %% 48 | %% @todo Wait, can't this atom be gotten rid of? The types are sufficiently different. 49 | start(auto, WorkFlow, CombinerPid) -> 50 | sk_tracer:t(75, self(), {?MODULE, start}, [{combiner, CombinerPid}]), 51 | loop(decomp_by(), WorkFlow, CombinerPid, []); 52 | start(man, WorkerPids, CombinerPid) when is_pid(hd(WorkerPids)) -> 53 | sk_tracer:t(75, self(), {?MODULE, start}, [{combiner, CombinerPid}]), 54 | loop(decomp_by(), CombinerPid, WorkerPids). 55 | 56 | -spec start_hyb(atom(), [pid()], [pid()], pid()) -> 'eos'. 57 | start_hyb(man, CPUWorkerPids, GPUWorkerPids, CombinerPid) -> 58 | sk_tracer:t(75, self(), {?MODULE, start}, [{combiner, CombinerPid}]), 59 | loop_hyb(decomp_by(), CombinerPid, CPUWorkerPids, GPUWorkerPids). 60 | 61 | 62 | -spec loop(data_decomp_fun(), workflow(), pid(), [pid()]) -> 'eos'. 63 | %% @doc Recursively receives inputs as messages, which are decomposed, and the 64 | %% resulting messages sent to individual workers. `loop/4' is used in place of 65 | %% {@link loop/3} when the number of workers to be used is automatically 66 | %% determined by the total number of partite elements of an input. 67 | loop(DataPartitionerFun, WorkFlow, CombinerPid, WorkerPids) -> 68 | receive 69 | {data, _, _} = DataMessage -> 70 | PartitionMessages = DataPartitionerFun(DataMessage), 71 | WorkerPids1 = start_workers(length(PartitionMessages), WorkFlow, CombinerPid, WorkerPids), 72 | Ref = make_ref(), 73 | sk_tracer:t(60, self(), {?MODULE, data}, [{ref, Ref}, {input, DataMessage}, {partitions, PartitionMessages}]), 74 | dispatch(Ref, length(PartitionMessages), PartitionMessages, WorkerPids1), 75 | loop(DataPartitionerFun, WorkFlow, CombinerPid, WorkerPids1); 76 | {system, eos} -> 77 | sk_utils:stop_workers(?MODULE, WorkerPids), 78 | eos 79 | end. 80 | 81 | 82 | -spec loop(data_decomp_fun(), pid(), [pid()]) -> 'eos'. 83 | %% @doc Recursively receives inputs as messages, which are decomposed, and the 84 | %% resulting messages sent to individual workers. `loop/3' is used in place of 85 | %% {@link loop/4} when the number of workers is set by the developer. 86 | loop(DataPartitionerFun, CombinerPid, WorkerPids) -> 87 | receive 88 | {data, _, _} = DataMessage -> 89 | PartitionMessages = DataPartitionerFun(DataMessage), 90 | Ref = make_ref(), 91 | sk_tracer:t(60, self(), {?MODULE, data}, [{ref, Ref}, {input, DataMessage}, {partitions, PartitionMessages}]), 92 | dispatch(Ref, length(PartitionMessages), PartitionMessages, WorkerPids), 93 | loop(DataPartitionerFun, CombinerPid, WorkerPids); 94 | {system, eos} -> 95 | sk_utils:stop_workers(?MODULE, WorkerPids), 96 | eos 97 | end. 98 | 99 | loop_hyb(DataPartitionerFun, CombinerPid, CPUWorkerPids, GPUWorkerPids) -> 100 | receive 101 | {data, _, _} = DataMessage -> 102 | PartitionMessages = DataPartitionerFun(DataMessage), 103 | Ref = make_ref(), 104 | sk_tracer:t(60, self(), {?MODULE, data}, [{ref, Ref}, {input, DataMessage}, {partitions, PartitionMessages}]), 105 | hyb_dispatch(Ref, length(PartitionMessages), PartitionMessages, CPUWorkerPids, GPUWorkerPids), 106 | loop_hyb(DataPartitionerFun, CombinerPid, CPUWorkerPids, GPUWorkerPids); 107 | {system, eos} -> 108 | sk_utils:stop_workers(?MODULE, CPUWorkerPids), 109 | sk_utils:stop_workers(?MODULE, GPUWorkerPids), 110 | eos 111 | end. 112 | 113 | 114 | 115 | -spec decomp_by() -> data_decomp_fun(). 116 | %% @doc Provides the decomposition function and means to split a single input 117 | %% into many. This is based on the identity function, as the Map skeleton is 118 | %% applied to lists. 119 | decomp_by() -> 120 | fun({data, Value, Ids}) -> 121 | [{data, X, Ids} || X <- Value] 122 | end. 123 | 124 | -spec start_workers(pos_integer(), workflow(), pid(), [pid()]) -> [pid()]. 125 | %% @doc Used when the number of workers is not set by the developer. 126 | %% 127 | %% Workers are started if the number needed exceeds the number we already 128 | %% have. The total number of workers is derived from the number of partitions 129 | %% to which `WorkFlow' will be applied, as given by `NPartitions'. This 130 | %% includes 'recycled' workers from previous inputs. Both new and old worker 131 | %% processes are returned so that they might be used. Worker processes are 132 | %% represented as a list of their Pids under `WorkerPids'. 133 | start_workers(NPartitions, WorkFlow, CombinerPid, WorkerPids) when NPartitions > length(WorkerPids) -> 134 | NNewWorkers = NPartitions - length(WorkerPids), 135 | NewWorkerPids = sk_utils:start_workers(NNewWorkers, WorkFlow, CombinerPid), 136 | NewWorkerPids ++ WorkerPids; 137 | start_workers(_NPartitions, _WorkFlow, _CombinerPid, WorkerPids) -> 138 | WorkerPids. 139 | 140 | 141 | -spec dispatch(reference(), pos_integer(), [data_message(),...], [pid()]) -> 'ok'. 142 | %% @doc Partite elements of input stored in `PartitionMessages' are formatted 143 | %% and sent to a worker from `WorkerPids'. The reference argument `Ref' 144 | %% ensures that partite elements from different inputs are not incorrectly 145 | %% included. 146 | dispatch(Ref, NPartitions, PartitionMessages, WorkerPids) -> 147 | dispatch(Ref, NPartitions, 1, PartitionMessages, WorkerPids). 148 | 149 | hyb_dispatch(Ref, NPartitions, PartitionMessages, CPUWorkerPids, GPUWorkerPids) -> 150 | hyb_dispatch(Ref, NPartitions, 1, PartitionMessages, CPUWorkerPids, GPUWorkerPids). 151 | 152 | 153 | -spec dispatch(reference(), pos_integer(), pos_integer(), [data_message(),...], [pid()]) -> 'ok'. 154 | %% @doc Inner-function for {@link dispatch/4}. Recursively sends each message 155 | %% to a worker, following the addition of references to allow identification 156 | %% and recomposition. 157 | dispatch(_Ref,_NPartitions, _Idx, [], _) -> 158 | ok; 159 | dispatch(Ref, NPartitions, Idx, [PartitionMessage|PartitionMessages], [WorkerPid|WorkerPids]) -> 160 | PartitionMessage1 = sk_data:push({decomp, Ref, Idx, NPartitions}, PartitionMessage), 161 | sk_tracer:t(50, self(), WorkerPid, {?MODULE, data}, [{partition, PartitionMessage1}]), 162 | WorkerPid ! PartitionMessage1, 163 | dispatch(Ref, NPartitions, Idx+1, PartitionMessages, WorkerPids ++ [WorkerPid]). 164 | 165 | hyb_dispatch(_Ref,_NPartitions, _Idx, [], _, _) -> 166 | ok; 167 | hyb_dispatch(Ref, NPartitions, Idx, [{DataTag,{cpu,Msg},Rest}|PartitionMessages], [CPUWorkerPid|CPUWorkerPids], GPUWorkerPids) -> 168 | PartitionMessageWithoutTag = {DataTag, Msg, Rest}, 169 | PartitionMessage1 = sk_data:push({decomp, Ref, Idx, NPartitions}, PartitionMessageWithoutTag), 170 | sk_tracer:t(50, self(), CPUWorkerPid, {?MODULE, data}, [{partition, PartitionMessage1}]), 171 | CPUWorkerPid ! PartitionMessage1, 172 | hyb_dispatch(Ref, NPartitions, Idx+1, PartitionMessages, CPUWorkerPids ++ [CPUWorkerPid], GPUWorkerPids); 173 | hyb_dispatch(Ref, NPartitions, Idx, [{DataTag,{gpu,Msg},Rest}|PartitionMessages], CPUWorkerPids, [GPUWorkerPid|GPUWorkerPids]) -> 174 | PartitionMessageWithoutTag = {DataTag, Msg, Rest}, 175 | PartitionMessage1 = sk_data:push({decomp, Ref, Idx, NPartitions}, PartitionMessageWithoutTag), 176 | sk_tracer:t(50, self(), GPUWorkerPid, {?MODULE, data}, [{partition, PartitionMessage1}]), 177 | GPUWorkerPid ! PartitionMessage1, 178 | hyb_dispatch(Ref, NPartitions, Idx+1, PartitionMessages, CPUWorkerPids, GPUWorkerPids ++ [GPUWorkerPid]). 179 | -------------------------------------------------------------------------------- /src/sk_ord.erl: -------------------------------------------------------------------------------- 1 | %%%---------------------------------------------------------------------------- 2 | %%% @author Sam Elliott 3 | %%% @copyright 2012 University of St Andrews (See LICENCE) 4 | %%% @headerfile "skel.hrl" 5 | %%% 6 | %%% @doc This module contains 'ord' skeleton initialization logic. 7 | %%% 8 | %%% The 'ord' skeleton can reorder outputs from its inner skeletons such that 9 | %%% they have the same order coming out the ord skeleton as they had going into 10 | %%% it. 11 | %%% 12 | %%% This becomes useful when requiring ordering on things like a farm. 13 | %%% 14 | %%% === Example === 15 | %%% 16 | %%% ```skel:run([{ord, [{farm, [{seq, fun ?MODULE:p/1}], 10}]}], Inputs)''' 17 | %%% 18 | %%% In this example we wrap the `ord' skeleton around a farm of ten 19 | %%% workers, each of which are running the developer-defined `p/1' in 20 | %%% sequential functions. The result of this is that the returned 21 | %%% applications of the input to `p/1' is in the same order as the list 22 | %%% of inputs `Inputs' itself. 23 | %%% 24 | %%% @end 25 | %%%---------------------------------------------------------------------------- 26 | -module(sk_ord). 27 | 28 | -export([ 29 | make/1 30 | ]). 31 | 32 | -include("skel.hrl"). 33 | 34 | -ifdef(TEST). 35 | -compile(export_all). 36 | -endif. 37 | 38 | -spec make(workflow()) -> maker_fun(). 39 | %% @doc Constructs an Ord skeleton wrapper to ensure the outputs of the 40 | %% specified workflow are in the same order as that of its inputs. 41 | make(WorkFlow) -> 42 | fun(NextPid) -> 43 | ReordererPid = spawn(sk_ord_reorderer, start, [NextPid]), 44 | WorkerPid = sk_utils:start_worker(WorkFlow, ReordererPid), 45 | spawn(sk_ord_tagger, start, [WorkerPid]) 46 | end. 47 | -------------------------------------------------------------------------------- /src/sk_ord_reorderer.erl: -------------------------------------------------------------------------------- 1 | %%%---------------------------------------------------------------------------- 2 | %%% @author Sam Elliott 3 | %%% @copyright 2012 University of St Andrews (See LICENCE) 4 | %%% @headerfile "skel.hrl" 5 | %%% 6 | %%% @doc This module contains 'ord' skeleton reordering logic. 7 | %%% 8 | %%% The 'ord' skeleton can reorder outputs from its inner skeletons such that 9 | %%% they have the same order coming out the ord skeleton as they had going into 10 | %%% it. 11 | %%% 12 | %%% The reorderer takes each output from the inner skeleton and only forwards 13 | %%% it if it has already forwarded all outputs with smaller tags. 14 | %%% 15 | %%% @end 16 | %%%---------------------------------------------------------------------------- 17 | -module(sk_ord_reorderer). 18 | 19 | -export([ 20 | start/1 21 | ]). 22 | 23 | -include("skel.hrl"). 24 | 25 | -ifdef(TEST). 26 | -compile(export_all). 27 | -endif. 28 | 29 | -spec start(pid()) -> 'eos'. 30 | %% @doc Ensures that output is given in the same order as the input received. 31 | %% For each message reveived as output, that message is only released when all 32 | %% outstanding messages have been sent. 33 | start(NextPid) -> 34 | sk_tracer:t(75, self(), {?MODULE, start}, []), 35 | loop(0, dict:new(), NextPid). 36 | 37 | % receive loop until eos 38 | % opens the data message 39 | % dict1 is a dictionary of {Identifier, Data Message} key-value pairs 40 | % Seen is a count of the loop: how many things it has seen. 41 | 42 | -spec loop(non_neg_integer(), dict:dict(), pid()) -> 'eos'. 43 | %% @doc Recursively receives and stores messages until they are ready for 44 | %% release. 45 | loop(Seen, Dict, NextPid) -> 46 | receive 47 | {data, _, _} = DataMessage -> 48 | {{ord, Idx}, DataMessage1} = sk_data:pop(DataMessage), 49 | Dict1 = store(Idx, DataMessage1, Dict), 50 | {Seen1, Dict2} = forward(Seen, Dict1, NextPid), 51 | loop(Seen1, Dict2, NextPid); 52 | {system, eos} -> 53 | sk_tracer:t(75, self(), NextPid, {?MODULE, system}, [{message, eos}]), 54 | NextPid ! {system, eos}, 55 | eos 56 | end. 57 | 58 | -spec store(pos_integer(), data_message(), dict:dict()) -> dict:dict(). 59 | %% @doc Stores the given `Idx', indicating position, and message `DataMessage' 60 | %% in the dictionary `Dict'. Returns the resulting dictionary. 61 | store(Idx, DataMessage, Dict) -> 62 | dict:store(Idx, DataMessage, Dict). 63 | 64 | -spec forward(non_neg_integer(), dict:dict(), pid()) -> {non_neg_integer(), dict:dict()}. 65 | %% @doc Determines if any messages in the dictionary `Dict' can be released to 66 | %% the process `NextPid'. This decision is based upon which messages have 67 | %% already been released as indicated by the `Seen' counter. 68 | forward(Seen, Dict, NextPid) -> 69 | case dict:find(Seen+1, Dict) of 70 | error -> 71 | {Seen, Dict}; 72 | {ok, DataMessage} -> 73 | sk_tracer:t(50, self(), NextPid, {?MODULE, data}, [{output, DataMessage}]), 74 | NextPid ! DataMessage, 75 | Dict1 = dict:erase(Seen+1, Dict), 76 | forward(Seen+1, Dict1, NextPid) 77 | end. 78 | -------------------------------------------------------------------------------- /src/sk_ord_tagger.erl: -------------------------------------------------------------------------------- 1 | %%%---------------------------------------------------------------------------- 2 | %%% @author Sam Elliott 3 | %%% @copyright 2012 University of St Andrews (See LICENCE) 4 | %%% @headerfile "skel.hrl" 5 | %%% 6 | %%% @doc This module contains 'ord' skeleton tagger logic. 7 | %%% 8 | %%% The 'ord' skeleton can reorder outputs from its inner skeletons such that 9 | %%% they have the same order coming out the ord skeleton as they had going into 10 | %%% it. 11 | %%% 12 | %%% The tagger takes each input and adds information as to its place in the 13 | %%% input stream of the inner skeleton. 14 | %%% 15 | %%% 16 | %%% @end 17 | %%%---------------------------------------------------------------------------- 18 | -module(sk_ord_tagger). 19 | 20 | -export([ 21 | start/1 22 | ]). 23 | 24 | -include("skel.hrl"). 25 | 26 | %% @doc Starts the tagger, labelling each input so that the order of all 27 | %% inputs is recorded. 28 | -spec start(pid()) -> 'eos'. 29 | start(NextPid) -> 30 | sk_tracer:t(75, self(), {?MODULE, start}, []), 31 | loop(1, NextPid). 32 | 33 | -spec loop(pos_integer(), pid()) -> 'eos'. 34 | %% @doc Recursively receives input, adds an additional identifier to that 35 | %% input, and sends the input onwards. These identifiers are just a counter, 36 | %% with each input receiving the indentifier indicating how many inputs were 37 | %% seen before. 38 | loop(Idx, NextPid) -> 39 | receive 40 | {data, _, _} = DataMessage -> 41 | DataMessage1 = sk_data:push({ord, Idx}, DataMessage), 42 | sk_tracer:t(50, self(), NextPid, {?MODULE, data}, [{input, DataMessage}, {output, DataMessage1}]), 43 | NextPid ! DataMessage1, 44 | loop(Idx+1, NextPid); 45 | {system, eos} -> 46 | sk_tracer:t(75, self(), NextPid, {?MODULE, system}, [{message, eos}]), 47 | NextPid ! {system, eos}, 48 | eos 49 | end. 50 | -------------------------------------------------------------------------------- /src/sk_pipe.erl: -------------------------------------------------------------------------------- 1 | %%%---------------------------------------------------------------------------- 2 | %%% @author Sam Elliott 3 | %%% @copyright 2013 University of St Andrews (See LICENCE) 4 | %%% @headerfile "skel.hrl" 5 | %%% 6 | %%% @doc This module contains 'pipe' skeleton initialization logic. 7 | %%% 8 | %%% The Pipe skeleton is the sequential composition of several skeletons. It 9 | %%% applies those skeletons and/or functions to a sequence of independent 10 | %%% inputs where the output of one function is the input of the next. 11 | %%% 12 | %%% 13 | %%% Example 14 | %%% 15 | %%% ```skel:run([{pipe, [{seq, fun ?MODULE:f1/1}, {seq, fun ?MODULE:f2/1}, 16 | %%% {seq, fun ?MODULE:f3/1}]}], Inputs)''' 17 | %%% 18 | %%% In this example, the pipeline has three stages. Each stage uses a 19 | %%% sequential function to perform its task -- i.e. executing the 20 | %%% developer-defined `f1', `f2', and `f3' respectively. The input, 21 | %%% represented as the list `Inputs', is first passed to `f1' whose 22 | %%% results form the input for `f2', the results of which serve as input 23 | %%% to `f3'. 24 | %%% 25 | %%% @end 26 | %%%---------------------------------------------------------------------------- 27 | -module(sk_pipe). 28 | 29 | -export([ 30 | make/1 31 | ]). 32 | 33 | -include("skel.hrl"). 34 | 35 | -spec make(skel:workflow()) -> skel:maker_fun(). 36 | %% @doc Produces workers according to the specified workflow. Returns an 37 | %% anonymous function taking the Pid of the parent process `NextPid'. 38 | make(WorkFlow) -> 39 | fun(NextPid) -> 40 | sk_assembler:make(WorkFlow, NextPid) 41 | end. 42 | -------------------------------------------------------------------------------- /src/sk_profile.erl: -------------------------------------------------------------------------------- 1 | %%%---------------------------------------------------------------------------- 2 | %%% @author Sam Elliott 3 | %%% @copyright 2012 University of St Andrews (See LICENCE) 4 | %%% @headerfile "skel.hrl" 5 | %%% 6 | %%% @doc This module contains various functions to help us when profiling and 7 | %%% benchmarking. 8 | %%% 9 | %%% This allows us very simple access to benchmarking. 10 | %%% 11 | %%% === Example === 12 | %%% 13 | %%% ```sk_profile:benchmark(skel:run([{seq, fun ?MODULE:p/1}, {seq, fun ?MODULE:f/1}], Images), [Input], Ntimes)])''' 14 | %%% 15 | %%% In this example we use the {@link benchmark/3} function to record how 16 | %%% long it takes for the example seen in {@link sk_seq} to execute. 17 | %%% 18 | %%% @end 19 | %%% @todo Include eprof functionality 20 | %%%---------------------------------------------------------------------------- 21 | -module(sk_profile). 22 | 23 | -export([ 24 | benchmark/3 25 | ]). 26 | 27 | -include("skel.hrl"). 28 | 29 | -spec benchmark(fun(), list(), pos_integer()) -> list(). 30 | %% @doc Produces a list of averages for the time taken by function `Fun' to be 31 | %% evaluated `N' times, given a list of arguments `Args'. Returned times are 32 | %% in microseconds. Returns a list containing the tuples: 33 | %% 34 | %%
    35 | %%
  • N,
    • 36 | %%
  • min, the shortest time taken to perform Fun;
    • 37 | %%
  • max, the longest time taken to perform Fun;
    • 38 | %%
  • med, the median of all individual results;
    • 39 | %%
  • mean, the mean of all individual results; and
    • 40 | %%
  • std_dev, the standard deviation.
    • 41 | %%
42 | benchmark(Fun, Args, N) when N > 0 -> 43 | Timing = test_loop(Fun, Args, N, []), 44 | Mean = mean(Timing), 45 | [ 46 | {n, N}, 47 | {min, lists:min(Timing)}, 48 | {max, lists:max(Timing)}, 49 | {med, median(Timing)}, 50 | {mean, Mean}, 51 | {std_dev, std_dev(Timing, Mean)} 52 | ]. 53 | 54 | %% @doc Recursively records the length of time it takes for the function `Fun' 55 | %% to be evaluated `N' times. 56 | test_loop(_Fun, _Args, 0, Timings) -> 57 | Timings; 58 | test_loop(Fun, Args, N, Timings) -> 59 | {Timing, _} = timer:tc(Fun, Args), 60 | test_loop(Fun, Args, N-1, [Timing|Timings]). 61 | 62 | -spec median([number(),...]) -> number(). 63 | %% @doc Returns the median of the times listed. 64 | median(List) -> 65 | lists:nth(round((length(List) / 2)), lists:sort(List)). 66 | 67 | -spec mean([number(),...]) -> number(). 68 | %% @doc Returns the mean time taken for those listed. 69 | mean(List) -> 70 | lists:foldl(fun(X, Sum) -> X + Sum end, 0, List) / length(List). 71 | 72 | -spec std_dev([number(),...], number()) -> number(). 73 | %% @doc Returns the standard deviation of all times recorded. 74 | std_dev(List, Mean) -> 75 | math:pow(variance(List, Mean), 0.5). 76 | 77 | -spec variance([number(),...], number()) -> number(). 78 | %% @doc Calculates the variance of the times listed for use in calculating the 79 | %% standard deviation in {@link std_dev/2}. 80 | variance(List, _Mean) when length(List) == 1 -> 81 | 0.0; 82 | variance(List, Mean) -> 83 | lists:foldl(fun(X, Sum) -> math:pow(Mean - X, 2) + Sum end, 0, List) / (length(List) - 1). 84 | -------------------------------------------------------------------------------- /src/sk_reduce.erl: -------------------------------------------------------------------------------- 1 | %%%---------------------------------------------------------------------------- 2 | %%% @author Sam Elliott 3 | %%% @copyright 2012 University of St Andrews (See LICENCE) 4 | %%% @headerfile "skel.hrl" 5 | %%% 6 | %%% @doc This module contains Reduce skeleton initialization logic. 7 | %%% 8 | %%% A reduce skeleton is an implementation of a parallel treefold. 9 | %%% 10 | %%% === Example === 11 | %%% 12 | %%% ```skell:run([{reduce, fun?MODULE:reduce/2, fun ?MODULE:id/1}], Inputs)''' 13 | %%% 14 | %%% Here, we call upon the reduce skeleton to reduce a list of inputs, 15 | %%% denoted `Inputs', using the developer-defined functions `reduce' and `id'. In this example, we presume to sum the elements in a list. Hence, `reduce' takes two arguments and returns their total. Whilst, `id' returns its input sans transformation. We receive the answer in the form of a single-element list as a message from the sink process. 16 | %%% 17 | %%% @end 18 | %%%---------------------------------------------------------------------------- 19 | -module(sk_reduce). 20 | 21 | -export([ 22 | make/2 23 | ,fold1/2 24 | ]). 25 | 26 | -include("skel.hrl"). 27 | 28 | -spec make(decomp_fun(), reduce_fun()) -> fun((pid()) -> pid()). 29 | %% @doc Readies an instance of the Reduce skeleton. Uses the developer-defined 30 | %% decomposition and recomposition functions `Decomp' and `Reduce', 31 | %% respectively. Returns an anonymous function waiting for the sink process 32 | %% `NextPid'. 33 | make(Decomp, Reduce) -> 34 | fun(NextPid) -> 35 | spawn(sk_reduce_decomp, start, [Decomp, Reduce, NextPid]) 36 | end. 37 | 38 | % Implemented as a treefold underneath 39 | 40 | -spec fold1(fun((A, A) -> A), [A,...]) -> A when A :: term(). 41 | %% @doc Sequential `reduce' entry-point. Primarily for comparison purposes. 42 | fold1(_ReduceFun, [L1]) -> 43 | L1; 44 | fold1(ReduceFun, [L1, L2 | List]) -> 45 | fold1(ReduceFun, [ReduceFun(L1, L2) | pairs(ReduceFun, List)]). 46 | 47 | -spec pairs(fun((A, A) -> A), [A]) -> [A] when A :: term(). 48 | %% @doc Second stage to {@link fold1}'s sequential `reduce'. Recursively 49 | %% pairs the first two elements in the list and applies the given function 50 | %% `Fun'. 51 | pairs(Fun, [L1,L2|List]) -> 52 | [Fun(L1,L2) | pairs(Fun, List)]; 53 | pairs(_Fun, List) -> 54 | List. 55 | -------------------------------------------------------------------------------- /src/sk_reduce_decomp.erl: -------------------------------------------------------------------------------- 1 | %%%---------------------------------------------------------------------------- 2 | %%% @author Sam Elliott 3 | %%% @copyright 2012 University of St Andrews (See LICENCE) 4 | %%% @headerfile "skel.hrl" 5 | %%% 6 | %%% @doc This module contains Reduce skeleton decomp logic. 7 | %%% 8 | %%% The decomp process splits an input into many parts using a developer- 9 | %%% defined function, then hands it on to the tree of reducer processes to be 10 | %%% reduced. 11 | %%% 12 | %%% @end 13 | %%%---------------------------------------------------------------------------- 14 | -module(sk_reduce_decomp). 15 | 16 | -export([ 17 | start/3 18 | ]). 19 | 20 | -include("skel.hrl"). 21 | 22 | -type pid_pools() :: dict:dict(). 23 | 24 | -spec start(decomp_fun(), reduce_fun(), pid()) -> eos. 25 | %% @doc Starts the reduce process. Takes the developer-defined reduction and 26 | %% decompostion functions, `Reduce' and `Decomp', produces a tree of processes 27 | %% to handle reduction, and recursively reduces input. 28 | start(Decomp, Reduce, NextPid) -> 29 | sk_tracer:t(75, self(), {?MODULE, start}, [{next_pid, NextPid}]), 30 | DataDecompFun = sk_data:decomp_by(Decomp), 31 | DataReduceFun = sk_data:reduce_with(Reduce), 32 | PidPools = dict:new(), 33 | NewReducer = spawn(sk_reduce_reducer, start, [DataReduceFun, NextPid]), 34 | PidPools1 = dict:store(0, [NewReducer], PidPools), 35 | loop(DataDecompFun, DataReduceFun, NextPid, PidPools1). 36 | 37 | -spec loop(data_decomp_fun(), data_reduce_fun(), pid(), pid_pools()) -> eos. 38 | %% @doc Recursively receives and reduces input. In charge of dispatching work 39 | %% and input to reducers. 40 | loop(DataDecompFun, DataReduceFun, NextPid, PidPools) -> 41 | receive 42 | {data, _, _} = DataMessage -> 43 | PartitionMessages = DataDecompFun(DataMessage), 44 | PidPools1 = start_reducers(length(PartitionMessages), DataReduceFun, NextPid, PidPools), 45 | dispatch(PartitionMessages, NextPid, PidPools1), 46 | loop(DataDecompFun, DataReduceFun, NextPid, PidPools1); 47 | {system, eos} -> 48 | stop_reducers(PidPools) 49 | end. 50 | 51 | -spec start_reducers(pos_integer(), data_reduce_fun(), pid(), pid_pools()) -> pid_pools(). 52 | %% @doc Recursively produces and starts reducers. Calculates the total number 53 | %% of reducers needed based on the number of partitions `NPartitions' 54 | %% specified. If this number has not been reached, as determined by 55 | %% {@link top_pool/1}, create a new pool. 56 | start_reducers(NPartitions, DataReduceFun, NextPid, PidPools) -> 57 | TopPool = top_pool(PidPools), 58 | RequiredPool = ceiling(log2(NPartitions/2)), 59 | if 60 | TopPool < RequiredPool -> 61 | TopPoolPids = dict:fetch(TopPool, PidPools), 62 | NewPids = [spawn(sk_reduce_reducer, start, [DataReduceFun, NextPoolPid]) || NextPoolPid <- TopPoolPids, _ <- [1,2]], 63 | PidPools1 = dict:store(TopPool+1, NewPids, PidPools), 64 | start_reducers(NPartitions, DataReduceFun, NextPid, PidPools1); 65 | true -> PidPools 66 | end. 67 | 68 | -spec dispatch([data_message(),...], pid(), pid_pools()) -> ok. 69 | %% @doc Sends all input to reducers stored in `PidPools'. 70 | dispatch([DataMessage] = PartitionMessages, NextPid, _PidPools) when length(PartitionMessages) == 1 -> 71 | sk_tracer:t(75, self(), NextPid, {?MODULE, data}, [{data_message, DataMessage}]), 72 | NextPid ! DataMessage, 73 | ok; 74 | dispatch(PartitionMessages, _NextPid, PidPools) -> 75 | NPartitions = length(PartitionMessages), 76 | RequiredPool = ceiling(log2(NPartitions/2)), 77 | RequiredPoolPids = dict:fetch(RequiredPool, PidPools), 78 | ReduceCount = ceiling(log2(NPartitions)), 79 | Ref = make_ref(), 80 | dispatch(Ref, ReduceCount, PartitionMessages, RequiredPoolPids ++ RequiredPoolPids). 81 | 82 | -spec dispatch(reference(), pos_integer(), [data_message()], [pid()]) -> ok. 83 | %% @doc Recursive worker for {@link dispatch/3}. Updates messages' 84 | %% identifiers, and sends messages to reducers. 85 | dispatch(_Ref, _ReduceCount, [], []) -> 86 | ok; 87 | dispatch(Ref, ReduceCount, []=DataMessages, [NextPid|NextPids]) -> 88 | sk_tracer:t(75, self(), NextPid, {?MODULE, system}, [{message, reduce_unit}, {ref, Ref}, {reduce_count, ReduceCount}]), 89 | NextPid ! {system, {reduce_unit, Ref, ReduceCount}}, 90 | dispatch(Ref, ReduceCount, DataMessages, NextPids); 91 | dispatch(Ref, ReduceCount, [DataMessage|DataMessages], [NextPid|NextPids]) -> 92 | DataMessage1 = sk_data:push({reduce, Ref, ReduceCount}, DataMessage), 93 | sk_tracer:t(50, self(), NextPid, {?MODULE, data}, [{partition, DataMessage1}]), 94 | NextPid ! DataMessage1, 95 | dispatch(Ref, ReduceCount, DataMessages, NextPids). 96 | 97 | -spec stop_reducers(pid_pools()) -> eos. 98 | %% @doc Sends the halt command to all reducers. 99 | stop_reducers(PidPools) -> 100 | TopPoolPids = dict:fetch(top_pool(PidPools), PidPools), 101 | sk_utils:stop_workers(?MODULE, TopPoolPids ++ TopPoolPids). 102 | 103 | -spec top_pool(pid_pools()) -> number(). 104 | %% @doc Finds the index of the last added PidPool. 105 | top_pool(PidPools) -> 106 | Pools = dict:fetch_keys(PidPools), 107 | lists:max(Pools). 108 | 109 | -spec log2(number()) -> number(). 110 | %% @doc Finds the binary logarithm of `X'. 111 | log2(X) -> 112 | math:log(X) / math:log(2). 113 | 114 | -spec ceiling(number()) -> integer(). 115 | %% @doc Rounds `X' up to the nearest integer. 116 | ceiling(X) -> 117 | case trunc(X) of 118 | Y when Y < X -> Y + 1 119 | ; Z -> Z 120 | end. 121 | -------------------------------------------------------------------------------- /src/sk_reduce_reducer.erl: -------------------------------------------------------------------------------- 1 | %%%---------------------------------------------------------------------------- 2 | %%% @author Sam Elliott 3 | %%% @copyright 2012 University of St Andrews (See LICENCE) 4 | %%% @headerfile "skel.hrl" 5 | %%% 6 | %%% @doc This module contains Reduce skeleton reduce logic. 7 | %%% 8 | %%% The reduce process takes two inputs, then applies the developer-defined 9 | %%% reduce function to them, before forwarding on the results to the next step 10 | %%% in the tree of reducers. 11 | %%% 12 | %%% @end 13 | %%%---------------------------------------------------------------------------- 14 | -module(sk_reduce_reducer). 15 | 16 | -export([ 17 | start/2 18 | ]). 19 | 20 | -include("skel.hrl"). 21 | 22 | -type maybe_data() :: unit | data_message(). 23 | 24 | 25 | -spec start(data_reduce_fun(), pid()) -> eos. 26 | %% @doc Starts the reducer worker. The reducer worker recursively applies the 27 | %% developer-defined transformation `DataFun' to the input it receives. 28 | start(DataFun, NextPid) -> 29 | sk_tracer:t(75, self(), {?MODULE, start}, [{next_pid, NextPid}]), 30 | loop(dict:new(), 0, DataFun, NextPid). 31 | 32 | % Message Receiver Loop 33 | % 1st Case: data message. Stores items in a dictionary until they can be reduced (i.e. on every second call, dict1 is emptied). 34 | % 2nd Case: Occurs when you have an odd-length list as input. 35 | 36 | -spec loop(dict:dict(), integer(), data_reduce_fun(), pid()) -> eos. 37 | %% @doc The main message receiver loop. Recursively receives messages upon 38 | %% which, if said messages carry data, a reduction is attempted using 39 | %% `DataFun'. 40 | loop(Dict, EOSRecvd, DataFun, NextPid) -> 41 | receive 42 | {data, _, _} = DataMessage -> 43 | {{reduce, Reference, ReduceCount}, DataMessage1} = sk_data:pop(DataMessage), 44 | Dict1 = store(Reference, Dict, DataMessage1), 45 | Dict2 = maybe_reduce(Reference, ReduceCount-1, NextPid, DataFun, Dict1), 46 | loop(Dict2, EOSRecvd, DataFun, NextPid); 47 | {system, {reduce_unit, Reference, ReduceCount}} -> 48 | Dict1 = store(Reference, Dict, unit), 49 | Dict2 = maybe_reduce(Reference, ReduceCount-1, NextPid, DataFun, Dict1), 50 | loop(Dict2, EOSRecvd, DataFun, NextPid); 51 | {system, eos} when EOSRecvd >= 1 -> 52 | sk_tracer:t(75, self(), NextPid, {?MODULE, system}, [{msg, eos}]), 53 | NextPid ! {system, eos}, 54 | eos; 55 | {system, eos} -> 56 | sk_tracer:t(85, self(), {?MODULE, system}, [{msg, eos}]), 57 | loop(Dict, EOSRecvd+1, DataFun, NextPid) 58 | end. 59 | 60 | -spec store(reference(), dict:dict(), maybe_data()) -> dict:dict(). 61 | %% @doc Stores the given reference `Ref' and value `Value' in the dictionary 62 | %% `Dict'. Returns the resulting dictionary. 63 | store(Ref, Dict, Value) -> 64 | dict:append(Ref, Value, Dict). 65 | 66 | -spec maybe_reduce(reference(), integer(), pid(), data_reduce_fun(), dict:dict()) -> dict:dict(). 67 | %% @doc Attempts to find the reference `Ref' in the dictionary `Dict'. If 68 | %% found, a reduction shall be attempted. Otherwise, the dictionary is simply 69 | %% returned. 70 | maybe_reduce(Ref, ReduceCount, NextPid, DataFun, Dict) -> 71 | case dict:find(Ref, Dict) of 72 | {ok, DMList} -> reduce(Ref, ReduceCount, NextPid, DMList, DataFun, Dict); 73 | _ -> Dict 74 | end. 75 | 76 | % The actual reduction function. 77 | % Case 1: we are effectively empty (why would this happen?) 78 | % Case 2 & 3: reached the end of the list/found an unpopulated node, consider the data message reduced. 79 | % Case 4: Reference has two data message entries, which are then reduced. 80 | % Deletes the reference from the dictionary, result is returned. 81 | 82 | -spec reduce(reference(), integer(), pid(), [maybe_data(),...], data_reduce_fun(), dict:dict()) -> dict:dict(). 83 | %% @doc The reduction function. Given a list of length two containing specific 84 | %% data messages retreived from `Dict', all messages are reduced to a single 85 | %% message. Returns the (now-erased) dictionary. 86 | reduce(Ref, ReduceCount, NextPid, [DM1, DM2] = DMList, DataFun, Dict) when length(DMList) == 2 -> 87 | case {DM1, DM2} of 88 | {unit, unit} -> 89 | forward_unit(Ref, ReduceCount, NextPid); 90 | {unit, DM2} -> 91 | forward(Ref, ReduceCount, NextPid, DM2); 92 | {DM1, unit} -> 93 | forward(Ref, ReduceCount, NextPid, DM1); 94 | {DM1, DM2} -> 95 | DM = DataFun(DM1, DM2), 96 | forward(Ref, ReduceCount, NextPid, DM) 97 | end, 98 | dict:erase(Ref, Dict); 99 | reduce(_Ref, _ReduceCount, _NextPid, _DMList, _DataFun, Dict) -> 100 | Dict. 101 | 102 | -spec forward(reference(), integer(), pid(), data_message()) -> ok. 103 | %% @doc Formats the reduced message, then submits said message for sending. 104 | %% Adds reference and counter information to the message's identifiers. 105 | forward(_Ref, ReduceCount, NextPid, DataMessage) when ReduceCount =< 0 -> 106 | forward(NextPid, DataMessage); 107 | forward(Ref, ReduceCount, NextPid, DataMessage) -> 108 | DataMessage1 = sk_data:push({reduce, Ref, ReduceCount}, DataMessage), 109 | forward(NextPid, DataMessage1). 110 | 111 | -spec forward(pid(), data_message()) -> ok. 112 | %% @doc Sends the message to the process NextPid. 113 | forward(NextPid, DataMessage) -> 114 | sk_tracer:t(50, self(), NextPid, {?MODULE, data}, [{message, DataMessage}]), 115 | NextPid ! DataMessage, 116 | ok. 117 | 118 | -spec forward_unit(reference(), integer(), pid()) -> ok. 119 | %% @doc Sends notification of double unit reduction to the process `NextPid'. 120 | %% No message formatting is required. 121 | forward_unit(_Ref, ReduceCount, _NextPid) when ReduceCount =< 0 -> 122 | ok; 123 | forward_unit(Ref, ReduceCount, NextPid) -> 124 | sk_tracer:t(75, self(), NextPid, {?MODULE, system}, [{message, reduce_unit}, {ref, Ref}, {reduce_count, ReduceCount}]), 125 | NextPid ! {system, {reduce_unit, Ref, ReduceCount}}, 126 | ok. 127 | -------------------------------------------------------------------------------- /src/sk_seq.erl: -------------------------------------------------------------------------------- 1 | %%%---------------------------------------------------------------------------- 2 | %%% @author Sam Elliott 3 | %%% @copyright 2012 University of St Andrews (See LICENCE) 4 | %%% @headerfile "skel.hrl" 5 | %%% 6 | %%% @doc This module contains the most logic of the most basic kind of skeleton 7 | %%% - `seq'. 8 | %%% 9 | %%% A 'seq' instance is a wrapper for a sequential function, taking an input 10 | %%% from its input stream, applying the function to it and sending the result 11 | %%% on its output stream. 12 | %%% 13 | %%% === Example === 14 | %%% 15 | %%% ```skel:run([{seq, fun ?MODULE:p/1}, {seq, fun ?MODULE:f/1}], Input).''' 16 | %%% 17 | %%% In this example, Skel is run using two sequential functions. On one 18 | %%% process it runs the developer-defined `p/1' on the input `Input', 19 | %%% sending all returned results to a second process. On this second 20 | %%% process, the similarly developer-defined `f/1' is run on the passed 21 | %%% results. This will only start once `p/1' has finished processing all 22 | %%% inputs and the system message `eos' sent. Results from `f/1' are sent 23 | %%% to the sink once they are available. 24 | %%% 25 | %%% 26 | %%% @end 27 | %%%---------------------------------------------------------------------------- 28 | -module(sk_seq). 29 | 30 | -export([ 31 | start/2 32 | ,make/1 33 | ]). 34 | 35 | -include("skel.hrl"). 36 | 37 | -ifdef(TEST). 38 | -compile(export_all). 39 | -endif. 40 | 41 | -spec make(worker_fun()) -> skel:maker_fun(). 42 | %% @doc Spawns a worker process performing the function `WorkerFun'. 43 | %% Returns an anonymous function that takes the parent process `NextPid' 44 | %% as an argument. 45 | make(WorkerFun) -> 46 | fun(NextPid) -> 47 | spawn(?MODULE, start, [WorkerFun, NextPid]) 48 | end. 49 | 50 | -spec start(worker_fun(), pid()) -> eos. 51 | %% @doc Starts the worker process' task. Recursively receives the worker 52 | %% function's input, and applies it to said function. 53 | start(WorkerFun, NextPid) -> 54 | sk_tracer:t(75, self(), {?MODULE, start}, [{next_pid, NextPid}]), 55 | DataFun = sk_data:fmap(WorkerFun), 56 | loop(DataFun, NextPid). 57 | 58 | -spec loop(skel:data_fun(), pid()) -> eos. 59 | %% @doc Recursively receives and applies the input to the function `DataFun'. 60 | %% Sends the resulting data message to the process `NextPid'. 61 | loop(DataFun, NextPid) -> 62 | receive 63 | {data,_,_} = DataMessage -> 64 | DataMessage1 = DataFun(DataMessage), 65 | sk_tracer:t(50, self(), NextPid, {?MODULE, data}, [{input, DataMessage}, {output, DataMessage1}]), 66 | NextPid ! DataMessage1, 67 | loop(DataFun, NextPid); 68 | {system, eos} -> 69 | sk_tracer:t(75, self(), NextPid, {?MODULE, system}, [{message, eos}]), 70 | NextPid ! {system, eos}, 71 | eos 72 | end. 73 | 74 | -------------------------------------------------------------------------------- /src/sk_sink.erl: -------------------------------------------------------------------------------- 1 | %%%---------------------------------------------------------------------------- 2 | %%% @author Sam Elliott 3 | %%% @copyright 2012 University of St Andrews (See LICENCE) 4 | %%% @headerfile "skel.hrl" 5 | %%% 6 | %%% @doc This module contains the sink logic. 7 | %%% 8 | %%% A sink is a process that accepts inputs off the final output stream in a 9 | %%% skeleton workflow. 10 | %%% 11 | %%% Two kinds of sink are provided - a list accumulator sink (the default) and 12 | %%% a module sink, that uses a callback module to deal with the data. 13 | %%% 14 | %%% 15 | %%% @end 16 | %%%---------------------------------------------------------------------------- 17 | -module(sk_sink). 18 | 19 | -export([ 20 | make/0 21 | ,start_acc/1 22 | ,make/1 23 | ,start_mod/2 24 | ]). 25 | 26 | -include("skel.hrl"). 27 | 28 | -ifdef(TEST). 29 | -compile(export_all). 30 | -endif. 31 | 32 | -callback init() -> 33 | {ok, State :: term()} | 34 | {stop, State :: term()}. 35 | 36 | -callback next_input(NextInput :: term(), State :: term()) -> 37 | {ok, NewState :: term()} | 38 | {stop, NewState :: term()}. 39 | 40 | -callback terminate(State :: term()) -> 41 | term(). 42 | 43 | -spec make() -> maker_fun(). 44 | %% @doc Creates the process to which the final results are sent. Returns an 45 | %% anonymous function which takes the Pid of the process it is linked 46 | %% to. 47 | make() -> 48 | fun(Pid) -> 49 | spawn(?MODULE, start_acc, [Pid]) 50 | end. 51 | 52 | -spec make(module()) -> maker_fun(). 53 | %% @doc Creates the process to which the final results are sent using the 54 | %% specified module OutputMod. Returns an anonymous function, taking 55 | %% the Pid of the process it is linked to. 56 | make(OutputMod) -> 57 | fun(Pid) -> 58 | spawn(?MODULE, start_mod, [OutputMod, Pid]) 59 | end. 60 | 61 | -spec start_acc(pid()) -> 'eos'. 62 | %% @doc Sets the sink process to receive messages from other processes. 63 | start_acc(NextPid) -> 64 | loop_acc(NextPid, []). 65 | 66 | -spec loop_acc(pid(), list()) -> 'eos'. 67 | %% @doc Recursively recieves messages, collecting each result in a list. 68 | %% Returns the list of results when the system message eos is 69 | %% received. 70 | loop_acc(NextPid, Results) -> 71 | receive 72 | {data, _, _} = DataMessage -> 73 | Value = sk_data:value(DataMessage), 74 | sk_tracer:t(50, self(), {?MODULE, data}, [{input, DataMessage}, {value, Value}]), 75 | loop_acc(NextPid, Results ++ [Value]); 76 | {system, eos} -> 77 | sk_tracer:t(75, self(), {?MODULE, system}, [{msg, eos}]), 78 | forward(Results, NextPid) 79 | end. 80 | 81 | -spec start_mod(module(), pid()) -> 'eos'. 82 | %% @doc Initiates loop to receive messages from child processes, passing 83 | %% results to the given module as appropriate. 84 | start_mod(OutputMod, NextPid) -> 85 | case OutputMod:init() of 86 | {ok, State} -> loop_mod(OutputMod, State, NextPid); 87 | {stop, State} -> 88 | Result = OutputMod:terminate(State), 89 | forward(Result, NextPid) 90 | end. 91 | 92 | -spec loop_mod(module(), term(), pid()) -> 'eos'. 93 | loop_mod(OutputMod, State, NextPid) -> 94 | receive 95 | {data, _, _} = DataMessage -> 96 | Value = sk_data:value(DataMessage), 97 | sk_tracer:t(50, self(), {?MODULE, data}, [{input, DataMessage}, {value, Value}]), 98 | case OutputMod:next_input(Value, State, NextPid) of 99 | {ok, NewState} -> loop_mod(OutputMod, NewState, NextPid); 100 | {stop, NewState} -> 101 | Result = OutputMod:terminate(NewState), 102 | forward(Result, NextPid) 103 | end; 104 | {system, eos} -> 105 | sk_tracer:t(75, self(), {?MODULE, system}, [{msg, eos}]), 106 | Result = OutputMod:terminate(State), 107 | forward(Result, NextPid) 108 | end. 109 | 110 | %% @doc Forwards the final result to the process NextPid. Returns the 111 | %% system message eos denoting the sink's task finished. 112 | forward(Result, NextPid) -> 113 | NextPid ! {sink_results, Result}, 114 | eos. 115 | -------------------------------------------------------------------------------- /src/sk_source.erl: -------------------------------------------------------------------------------- 1 | %%%---------------------------------------------------------------------------- 2 | %%% @author Sam Elliott 3 | %%% @copyright 2012 University of St Andrews (See LICENCE) 4 | %%% @headerfile "skel.hrl" 5 | %%% @doc This module contains the source logic. 6 | %%% 7 | %%% A source is a process that provides the given inputs to the first process 8 | %%% in a skeleton workflow. 9 | %%% 10 | %%% Two kinds of sources are provided - a list source (the default) and 11 | %%% a module source, that uses a callback module to deal with the data. 12 | %%% 13 | %%% @end 14 | %%%---------------------------------------------------------------------------- 15 | -module(sk_source). 16 | 17 | -export([ 18 | make/1 19 | ,start/2 20 | ]). 21 | 22 | -include("skel.hrl"). 23 | 24 | -ifdef(TEST). 25 | -compile(export_all). 26 | -endif. 27 | 28 | -callback init() -> 29 | {ok, State :: term()} | 30 | {no_inputs, State :: term()}. 31 | 32 | -callback next_input(State :: term()) -> 33 | {input, NextInput :: term(), NewState :: term()} | 34 | {ignore, NewState :: term()} | 35 | {eos, NewState :: term()}. 36 | 37 | -callback terminate(State :: term()) -> 38 | ok. 39 | 40 | %% @doc Creates a new child process using Input, given the parent process 41 | %% Pid. 42 | -spec make(input()) -> maker_fun(). 43 | make(Input) -> 44 | fun(Pid) -> 45 | spawn(?MODULE, start, [Input, Pid]) 46 | end. 47 | 48 | %% @doc Transmits each input in Input to the process NextPid. 49 | %% @todo add documentation for the callback loop 50 | -spec start(input(), pid()) -> 'eos'. 51 | start(Input, NextPid) when is_list(Input) -> 52 | list_loop(Input, NextPid); 53 | start(InputMod, NextPid) when is_atom(InputMod) -> 54 | case InputMod:init() of 55 | {ok, State} -> callback_loop(InputMod, State, NextPid); 56 | {no_inputs, State} -> 57 | send_eos(NextPid), 58 | InputMod:terminate(State) 59 | end. 60 | 61 | %% @doc Recursively sends each input in a given list to the process 62 | %% NextPid. 63 | list_loop([], NextPid) -> 64 | send_eos(NextPid); 65 | list_loop([Input|Inputs], NextPid) -> 66 | send_input(Input, NextPid), 67 | list_loop(Inputs, NextPid). 68 | 69 | %% @todo doc 70 | callback_loop(InputMod, State, NextPid) -> 71 | case InputMod:next_input(State) of 72 | {input, NextInput, NewState} -> 73 | send_input(NextInput, NextPid), 74 | callback_loop(InputMod, NewState, NextPid); 75 | {ignore, NewState} -> 76 | callback_loop(InputMod, NewState, NextPid); 77 | {eos, NewState} -> 78 | send_eos(NextPid), 79 | InputMod:terminate(NewState), 80 | eos 81 | end. 82 | 83 | %% @doc Input is formatted as a data message and sent to the 84 | %% process NextPid. 85 | send_input(Input, NextPid) -> 86 | DataMessage = sk_data:pure(Input), 87 | sk_tracer:t(50, self(), NextPid, {?MODULE, data}, [{output, DataMessage}]), 88 | NextPid ! DataMessage. 89 | 90 | send_eos(NextPid) -> 91 | sk_tracer:t(75, self(), NextPid, {?MODULE, system}, [{msg, eos}]), 92 | NextPid ! {system, eos}, 93 | eos. 94 | 95 | 96 | -------------------------------------------------------------------------------- /src/sk_tracer.erl: -------------------------------------------------------------------------------- 1 | %%%---------------------------------------------------------------------------- 2 | %%% @author Sam Elliott 3 | %%% @copyright 2012 University of St Andrews (See LICENCE) 4 | %%% @headerfile "skel.hrl" 5 | %%% @doc This module contains utility functions for tracing the execution of 6 | %%% skel programs. 7 | %%% 8 | %%% Most skel processes will emit events when they send messages, allowing 9 | %%% the tracer to view them in a nice way. This is by far the easiest way to 10 | %%% debug the parallelism, rather than 'printf' statements. 11 | %%% 12 | %%% @end 13 | %%%---------------------------------------------------------------------------- 14 | -module(sk_tracer). 15 | 16 | -export([ 17 | start/0 18 | ,report_event/4 ,t/4 19 | ,report_event/5 ,t/5 20 | ]). 21 | 22 | -include("skel.hrl"). 23 | 24 | -compile(nowarn_unused_vars). 25 | 26 | -type detail_level() :: 1..100. 27 | -type actor() :: term(). 28 | -type label() :: atom() | string() | term(). 29 | -type contents() :: [{term(), term()}] | term(). 30 | 31 | -spec start() -> 'ignore' | {'error',_} | {'ok',pid()}. 32 | %% @doc Opens a viewer displaying a graphic representation of currently 33 | %% employed processes and the messages sent between them. 34 | start() -> 35 | et_viewer:start([ 36 | {detail_level, max}, 37 | {trace_pattern, {sk_tracer, max}}, 38 | {trace_global, true}, 39 | {scale, 2}, 40 | {max_actors, infinity}, 41 | {width, 1000}, 42 | {height, 800} 43 | ]). 44 | 45 | -spec report_event(detail_level(), actor(), actor(), label(), contents()) -> 'hopefully_traced'. 46 | %% @doc Reports a message between two processes. Base case; returns 47 | %% hopefully_traced where no errors are encountered. 48 | report_event(DetailLevel, From, To, Label, Contents) -> 49 | hopefully_traced. 50 | 51 | -spec t(detail_level(), actor(), actor(), label(), contents()) -> 'hopefully_traced'. 52 | %% @doc Alias for {@link report_event/5}. 53 | t(DetailLevel, From, To, Label, Contents) -> 54 | ?MODULE:report_event(DetailLevel, From, To, Label, Contents). 55 | 56 | -spec report_event(detail_level(), actor(), label(), contents()) -> 'hopefully_traced'. 57 | %% @doc Reports a process event where no message is sent to a second process. 58 | %% Primarily used in notification of a construct or process being constructed. 59 | report_event(DetailLevel, FromTo, Label, Contents) -> 60 | ?MODULE:report_event(DetailLevel, FromTo, FromTo, Label, Contents). 61 | 62 | -spec t(detail_level(), actor(), label(), contents()) -> 'hopefully_traced'. 63 | %% @doc Alias for {@link report_event/4}. 64 | t(DetailLevel, FromTo, Label, Contents) -> 65 | ?MODULE:report_event(DetailLevel, FromTo, FromTo, Label, Contents). 66 | -------------------------------------------------------------------------------- /src/sk_utils.erl: -------------------------------------------------------------------------------- 1 | %%%---------------------------------------------------------------------------- 2 | %%% @author Sam Elliott 3 | %%% @copyright 2012 University of St Andrews (See LICENCE) 4 | %%% @headerfile "skel.hrl" 5 | %%% @doc This module contains functions designed to start and stop worker 6 | %%% processes, otherwise known and referred to as simply workers. 7 | %%% 8 | %%% @end 9 | %%%---------------------------------------------------------------------------- 10 | -module(sk_utils). 11 | 12 | -export([ 13 | start_workers/3 14 | ,start_worker_hyb/4 15 | ,start_workers_hyb/5 16 | ,start_worker/2 17 | ,stop_workers/2 18 | ]). 19 | 20 | -include("skel.hrl"). 21 | 22 | -spec start_workers(pos_integer(), workflow(), pid()) -> [pid()]. 23 | %% @doc Starts a given number NWorkers of workers as children to the specified process NextPid. Returns a list of worker Pids. 24 | start_workers(NWorkers, WorkFlow, NextPid) -> 25 | start_workers(NWorkers, WorkFlow, NextPid, []). 26 | 27 | -spec start_workers_hyb(pos_integer(), pos_integer(), workflow(), workflow(), pid()) -> {[pid()],[pid()]}. 28 | start_workers_hyb(NCPUWorkers, NGPUWorkers, WorkFlowCPU, WorkFlowGPU, NextPid) -> 29 | start_workers_hyb(NCPUWorkers, NGPUWorkers, WorkFlowCPU, WorkFlowGPU, NextPid, {[],[]}). 30 | 31 | -spec start_workers(pos_integer(), workflow(), pid(), [pid()]) -> [pid()]. 32 | %% @doc Starts a given number NWorkers of workers as children to the 33 | %% specified process NextPid. Returns a list of worker Pids. Inner 34 | %% function to {@link start_workers/3}, providing storage for partial results. 35 | start_workers(NWorkers,_WorkFlow,_NextPid, WorkerPids) when NWorkers < 1 -> 36 | WorkerPids; 37 | start_workers(NWorkers, WorkFlow, NextPid, WorkerPids) -> 38 | NewWorker = start_worker(WorkFlow, NextPid), 39 | start_workers(NWorkers-1, WorkFlow, NextPid, [NewWorker|WorkerPids]). 40 | 41 | start_workers_hyb(NCPUWorkers, NGPUWorkers, _WorkFlowCPU, _WorkFlowGPU, _NextPid, Acc) 42 | when (NCPUWorkers < 1) and (NGPUWorkers < 1) -> 43 | Acc; 44 | start_workers_hyb(NCPUWorkers, NGPUWorkers, WorkFlowCPU, WorkFlowGPU, NextPid, {CPUWs,GPUWs}) 45 | when NCPUWorkers < 1 -> 46 | NewWorker = start_worker(WorkFlowGPU, NextPid), 47 | start_workers_hyb(NCPUWorkers, NGPUWorkers-1, WorkFlowCPU, WorkFlowGPU, NextPid, {CPUWs, [NewWorker|GPUWs]}); 48 | start_workers_hyb(NCPUWorkers, NGPUWorkers, WorkFlowCPU, WorkFlowGPU, NextPid, {CPUWs, GPUWs}) -> 49 | NewWorker = start_worker(WorkFlowCPU, NextPid), 50 | start_workers_hyb(NCPUWorkers-1, NGPUWorkers, WorkFlowCPU, WorkFlowGPU, NextPid, {[NewWorker|CPUWs],GPUWs}). 51 | 52 | -spec start_worker(workflow(), pid()) -> pid(). 53 | %% @doc Provides a worker with its tasks, the workflow WorkFlow. 54 | %% NextPid provides the output process to which the worker's results 55 | %% are sent. 56 | start_worker(WorkFlow, NextPid) -> 57 | sk_assembler:make(WorkFlow, NextPid). 58 | 59 | -spec start_worker_hyb(workflow(), pid(), pos_integer(), pos_integer()) -> pid(). 60 | start_worker_hyb(WorkFlow, NextPid, NCPUWorkers, NGPUWorkers) -> 61 | sk_assembler:make_hyb(WorkFlow, NextPid, NCPUWorkers, NGPUWorkers). 62 | 63 | -spec stop_workers(module(), [pid()]) -> 'eos'. 64 | %% @doc Sends the halt command to each worker in the given list of worker 65 | %% processes. 66 | stop_workers(_Mod, []) -> 67 | eos; 68 | stop_workers(Mod, [Worker|Rest]) -> 69 | sk_tracer:t(85, self(), Worker, {Mod, system}, [{msg, eos}]), 70 | Worker ! {system, eos}, 71 | stop_workers(Mod, Rest). 72 | 73 | -------------------------------------------------------------------------------- /src/skel.app.src: -------------------------------------------------------------------------------- 1 | {application, skel, 2 | [ 3 | {description, "A Streaming Process-based Skeleton Library for Erlang"}, 4 | {vsn, "0.1"} 5 | ]}. 6 | -------------------------------------------------------------------------------- /src/skel.erl: -------------------------------------------------------------------------------- 1 | %%%---------------------------------------------------------------------------- 2 | %%% @author Sam Elliott 3 | %%% @copyright 2012 University of St Andrews (See LICENCE) 4 | %%% @headerfile "skel.hrl" 5 | %%% @doc This module is the root module of the 'Skel' library, including 6 | %%% entry-point functions. 7 | %%% 8 | %%% @end 9 | %%%---------------------------------------------------------------------------- 10 | -module(skel). 11 | 12 | -export([ 13 | run/2, 14 | do/2, 15 | farm/2, 16 | farm/3, 17 | pipe/2 18 | ]). 19 | 20 | -include("skel.hrl"). 21 | 22 | -spec run(workflow(), input()) -> pid(). 23 | %% @doc Primary entry-point function to the Skel library. Runs a specified 24 | %% workflow passing Input as input. Does not receive or return any 25 | %% output from the workflow. 26 | %% 27 | %%
Example:
28 | %% ```skel:run([{seq, fun ?MODULE:p/1}], Images)''' 29 | %% 30 | %% Here, skel runs the function p on all items in the 31 | %% list Images using the Sequential Function wrapper. 32 | %% 33 | run(WorkFlow, Input) -> 34 | sk_assembler:run(WorkFlow, Input). 35 | 36 | -spec do(workflow(), list()) -> list(). 37 | %% @doc The second entry-point function to the Skel library. This function 38 | %% does receive and return the results of the given workflow. 39 | %% 40 | %%
Example:
41 | %% ```skel:do([{reduce, fun ?MODULE:reduce/2, fun ?MODULE:id/1}], Inputs)]''' 42 | %% 43 | %% In this example, Skel uses the Reduce skeleton, where reduce 44 | %% and id are given as the reduction and decomposition functions 45 | %% respectively. The result for which is returned, and so can be printed 46 | %% or otherwise used. 47 | %% 48 | do(WorkFlow, Input) -> 49 | run(WorkFlow, Input), 50 | receive 51 | {sink_results, Results} -> 52 | Results 53 | end. 54 | 55 | -spec farm(fun(), list()) -> list(). 56 | farm(Fun, Input) -> 57 | skel:do([{farm, [{seq, Fun}], erlang:system_info(schedulers_online)}], Input). 58 | 59 | -spec farm(fun(), non_neg_integer(), list()) -> list(). 60 | farm(Fun, N, Input) -> 61 | skel:do([{farm, [{seq, Fun}], N}], Input). 62 | 63 | -spec pipe([fun()], list()) -> list(). 64 | pipe(WorkflowFuns, Input) -> 65 | skel:do(lists:map(fun(Fun) -> 66 | {seq, Fun} 67 | end, WorkflowFuns), Input). 68 | -------------------------------------------------------------------------------- /tutorial/bin/style.css: -------------------------------------------------------------------------------- 1 | html, body { 2 | margin: 0; 3 | padding: 0; 4 | } 5 | 6 | * { 7 | margin: 0; 8 | padding: 0; 9 | } 10 | 11 | body { 12 | margin: 0.5em auto 0.5em auto; 13 | padding: 1em 2em; 14 | font-family: Georgia, Palatino, 'Palatino Linotype', Times, 'Times New Roman'; 15 | /*font-family: helvetica, arial, sans-serif;*/ 16 | font-size: 14px; 17 | text-align: justify; 18 | line-height: 1.5; 19 | width: 42em; 20 | 21 | } 22 | 23 | h1, h2, h3, h4, h5, h6 { 24 | margin: 2em 0 1em; 25 | padding: 0; 26 | font-family: helvetica, arial, sans-serif; 27 | font-weight: bold; 28 | -webkit-font-smoothing: subpixel-antialiased; 29 | cursor: text; 30 | } 31 | 32 | h1 { 33 | 34 | } 35 | 36 | h2 { 37 | font-size: 24px; 38 | border-bottom: 1px solid #ccc; 39 | color: #000; 40 | } 41 | 42 | h3 { 43 | font-size: 18px; 44 | color: #333; 45 | border-bottom: 1px solid #ccc; 46 | } 47 | 48 | h4 { 49 | font-size: 16px; 50 | color: #333; 51 | } 52 | 53 | p { 54 | margin: 15px 0; 55 | } 56 | 57 | h2 + p, h3 + p { 58 | margin-top: 0; 59 | } 60 | 61 | a { 62 | color: #660500; 63 | text-decoration: none; 64 | } 65 | a:hover { 66 | text-decoration: underline; 67 | } 68 | 69 | pre { 70 | border: 1px solid #CCC; 71 | border-radius: 3px; 72 | padding: 0.5em 1em; 73 | margin: 0.8em 0; 74 | font-size: 13px; 75 | line-height: 19px; 76 | overflow: auto; 77 | } 78 | 79 | code { 80 | margin: 0 2px; 81 | padding: 0 5px; 82 | white-space: nowrap; 83 | border: 1px solid #eaeaea; 84 | background-color: #f8f8f8; 85 | border-radius: 3px; 86 | font-family: Consolas, 'Liberation Mono', Courier, monospace; 87 | font-size: 12px; 88 | color: #333; 89 | } 90 | 91 | a:visited > code { 92 | border: 1px solid #eaeaea; 93 | } 94 | 95 | pre > code { 96 | margin: 0; 97 | padding: 0; 98 | white-space: pre; 99 | border: none; 100 | background: transparent; 101 | font-size: 13px 102 | } 103 | 104 | ul, ol { 105 | padding-left: 2.5em; 106 | text-align: left; 107 | } 108 | 109 | dl { 110 | text-align: left; 111 | } 112 | 113 | dt { 114 | font-weight: bold; 115 | } 116 | 117 | dd { 118 | padding-left: 2.5em; 119 | } 120 | 121 | figure { 122 | display: block; 123 | margin: 1em 0 1.2em; 124 | border-top: 1px solid #660500; 125 | border-bottom: 1px solid #660500; 126 | } 127 | 128 | figure > img { 129 | display: block; 130 | margin-top: 1em; 131 | margin-left: auto; 132 | margin-right: auto; 133 | } 134 | 135 | /*img.centred { 136 | margin-left: auto; 137 | margin-right: auto; 138 | }*/ 139 | 140 | figure > figcaption { 141 | margin: 0.5em 0 1em; 142 | font-size: small; 143 | font-style: italic; 144 | text-align: center; 145 | } 146 | 147 | figure + p { 148 | margin-top: 1em; 149 | } 150 | 151 | blockquote > p { 152 | color: gray; 153 | margin-left: 1em; 154 | font-style: italic; 155 | } -------------------------------------------------------------------------------- /tutorial/src/cluster.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ParaPhrase/skel/c8c91f5c4adf1d299012bc2ecc798cc2b5ae9e63/tutorial/src/cluster.png -------------------------------------------------------------------------------- /tutorial/src/farm.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ParaPhrase/skel/c8c91f5c4adf1d299012bc2ecc798cc2b5ae9e63/tutorial/src/farm.png -------------------------------------------------------------------------------- /tutorial/src/feedback.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ParaPhrase/skel/c8c91f5c4adf1d299012bc2ecc798cc2b5ae9e63/tutorial/src/feedback.png -------------------------------------------------------------------------------- /tutorial/src/img_merge.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ParaPhrase/skel/c8c91f5c4adf1d299012bc2ecc798cc2b5ae9e63/tutorial/src/img_merge.png -------------------------------------------------------------------------------- /tutorial/src/index.md: -------------------------------------------------------------------------------- 1 | css:./style.css 2 | 3 | # Overview of the Skel Library 4 | 5 | #### [Tutorial](tutorial.html) 6 | 7 | #### [Documentation](../../doc/index.html) -------------------------------------------------------------------------------- /tutorial/src/map.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ParaPhrase/skel/c8c91f5c4adf1d299012bc2ecc798cc2b5ae9e63/tutorial/src/map.png -------------------------------------------------------------------------------- /tutorial/src/pipe.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ParaPhrase/skel/c8c91f5c4adf1d299012bc2ecc798cc2b5ae9e63/tutorial/src/pipe.png -------------------------------------------------------------------------------- /tutorial/src/speedup.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ParaPhrase/skel/c8c91f5c4adf1d299012bc2ecc798cc2b5ae9e63/tutorial/src/speedup.png -------------------------------------------------------------------------------- /tutorial/src/style.css: -------------------------------------------------------------------------------- 1 | tutorial/bin/style.css -------------------------------------------------------------------------------- /tutorial/src/tutorial.md: -------------------------------------------------------------------------------- 1 | css:./style.css 2 | title: Skel Tutorial 3 | 4 | ## Tutorial 5 | 6 | ### Contents 7 | 8 | - [Introduction][intro] 9 | - [What Skel Is][what] 10 | - [On Parallelism and Skeletons][par_and_skel] 11 | - [Expectations of this Tutorial][expectations] 12 | - [Getting Started][start] 13 | - [Compiling the Skel Library][compile] 14 | - [Calling the Skel Library][invocation] 15 | - [Skel Workflows][workflows] 16 | - [Workflow Items][wk_itms] 17 | - [A Recurring Example][example] 18 | - [The Sequential Function][seq] 19 | - [The Pipe Skeleton][pipe] 20 | - [The Farm Skeleton][farm] 21 | - [The Ord Skeleton][ord] 22 | - [The Reduce Skeleton][reduce] 23 | - [The Map Skeleton][map] 24 | - [The Feedback Skeleton][feedback] 25 | - [Other Notes][notes] 26 | - [Schedulers][scheduler] 27 | - [Benchmarking][timings] 28 | - [Debugging][debug] 29 | - [Further Reading][read] 30 | 31 | ### Introduction [intro] 32 | 33 | #### What Skel Is [what] 34 | 35 | Skel is a library produced as part of the [Paraphrase](http://www.paraphrase-ict.eu) project to assist in the introduction of parallelism for [Erlang](http://www.erlang.com) programs. It is a collection of *algorithmic skeletons*, a structured set of common patterns of parallelism, that may be used and customised for a range of different situations. 36 | 37 | #### On Parallelism and Skeletons [par_and_skel] 38 | 39 | In recent years, the single-core processor has been usurped by its multi-core cousins, and is unlikely to make a significant return. Whilst it is reasonable to expect a change in development practices to reflect the move to multi-core, any evidence of this happening suggests changes are introduced slowly. 40 | 41 | Under single-core architectures, programs were effectively gifted a speed boost with each processor generation. The same cannot be said under multi-core architectures, however. Instead, developers must exploit parallelism if they desire to achieve ever better performance gains. 42 | 43 | On paper, *parallelising* programs is straightforward. 44 | 45 | 1. Undertake profiling to help determine what should be parallelised and where. 46 | 2. The refactoring of code to introduce the generation and management of processes. 47 | 3. Debugging and benchmarking. 48 | 49 | In practice, however, parallelising programs is difficult. It requires careful thought in coordinating shared resources, avoiding race conditions, and fiddling with synchronisation. Additionally, and in terms of language support, parallelism is often *bolted on*. This likely results in an implementation that is unwieldy at best. Even when the developer has a program that works as intended, we suggest parallelism only *effective* when the changes provide (significant) performance gains. An achievement that is not guaranteed through the manual introduction of parallelism. 50 | 51 | Algorithmic skeletons propose to simplify and speed this process by providing commonly-used patterns of parallel computation. With skeletons, the developer need only submit a sequential operation he wishes to perform in parallel. The skeleton itself handling lower-level management, and so many of the headaches that arise during implementation. 52 | 53 | The introduction of parallelism then consists of choosing a skeleton, and invoking it with (existing) sequential code. Skeletons may be fine tuned, or replaced altogether, to find the best speedups possible. As such, skeletons are simple to use. 54 | 55 | That skeletons themselves handle the low-level aspects of parallelisation, allows their implementation to be thoroughly tested, and based upon proven concepts and ideas. Hence, skeletons are also safe. 56 | 57 | It remains that, thus far, the skeletons described above are of an abstract nature. Indeed, skeletons may be adapted to, and implemented in, most languages with support for parallelism. Using Erlang's first-class concurrency primitives, and taking advantage of higher-order functions, skeletons may be thus implemented, presented, and employed as a library. Skel, that which we present here, is such a library. 58 | 59 | #### Expectations of this Tutorial [expectations] 60 | 61 | From this tutorial, you may expect to gain an understanding of how to invoke Skel, and of the skeletons it offers. 62 | 63 | For each skeleton, we include: 64 | 65 | 1. a brief summary of what that skeleton is, 66 | 2. an explanation of how to use that skeleton within Skel, and 67 | 3. one or more examples to help illustrate its use. 68 | 69 | As each skeleton featured is summarised, a detailed familiarity of the concept is not necessary. Should more information on skeletons be sought, we include a [Further Reading][read] section. 70 | 71 | A reasonable understanding of Erlang and the general concepts behind parallelism are desired, however. Similarly, and whilst we explain the steps needed to invoke Skel and each skeleton, the specific preparation of any existing Erlang program to take advantage of the library is outside the scope of both Skel and this tutorial. 72 | 73 | Lastly, please note that due to the subject nature of this tutorial, there will be a fair amount of Erlang code. 74 | 75 | io:format("It will look something like this"). 76 | 77 | ### Getting Started [start] 78 | 79 | #### Compiling Skel [compile] 80 | 81 | The Skel library includes a Makefile that may be used to build the library and examples. The most pertinent rules follow. 82 | 83 | - `make` compiles the library source and nothing else. 84 | - `make examples` compiles both the library and included examples. 85 | - `make console` compiles the library and returns an interactive console. 86 | - `make clean` cleans the `skel` directory. 87 | - `make tutorial` compiles a copy of the documentation and this tutorial. 88 | 89 | #### Invoking Skel [invocation] 90 | 91 | Having compiled Skel using one of the above `make` commands sans error, we turn our attention to its basic invocation and use. 92 | 93 | If still in the `skel` directory, the `make console` will summon an Erlang shell with access to the library. Elsewhere, the `skel/ebin` directory must be included in the Erlang path. 94 | 95 | We set this either by using the [`-pa`](http://erlang.org/doc/man/erl.html) argument when invoking the Erlang shell, or by adding Skel to the application's dependencies. Either way, and if set correctly, a shell almost identical to that produced by `make console` will allow access to the library. 96 | 97 | To test this, we might type the following at the shell prompt: 98 | 99 | skel:do([{seq, fun(X) -> X end}], [1,2,3,4,5]). 100 | 101 | This returns `[1,2,3,4,5]` if the Skel library is within your Erlang path, 102 | 103 | ** exception error: undefined function skel:do/2 104 | 105 | otherwise. 106 | 107 | Skel may be invoked using one of two functions found in the [`skel`](../../doc/skel.html) module. [`run/2`](../../doc/skel.html#run-2) simply runs a given task in the form of a *workflow* (explained shortly). Note that no result is returned, instead only a pid similar in format to `<0.36.0>`. [`do/2`](../../doc/skel.html#do-2), as seen above, runs a given workflow but also returns the result of that workflow. In the case of our previous example, this was `[1,2,3,4,5]`. 108 | 109 | Whilst the latter automatically returns the result of the specified workflow, a result can still be retrieved from the former. This can be done using a `receive` block, expecting the tuple: 110 | 111 | {sink_results, Results} 112 | 113 | Where `Results` is a variable, and may be otherwise named. 114 | 115 | Input is passed to both [`run/2`](../../doc/skel.html#run-2) and [`do/2`](../../doc/skel.html#do-2) in the form of a list, and is given as the second argument to these functions. 116 | 117 | #### Skel Workflows [workflows] 118 | 119 | As their name suggests, workflows define how certain actions should be carried out. 120 | 121 | Workflows are constructed as a list of pre-defined tuples representing specific functions and skeletons. We shall refer to these as *workflow items*. Skel currently supports a number of workflow items, the identifier atoms for which are: 122 | 123 | - [`seq`][seq], 124 | - [`pipe`][pipe], 125 | - [`farm`][farm], 126 | - [`ord`][ord], 127 | - [`reduce`][reduce], 128 | - [`map`][map], 129 | - [`cluster`][map], and 130 | - [`feedback`][feedback]. 131 | 132 | These atoms are used as the first element of each workflow item tuple, and identify the item's type. We now introduce each of these; showing how they are used, accompanied by an example. 133 | 134 | ### Workflow Items [wk_itms] 135 | 136 | In this section, we look at the workflow items that may be used in a Skel workflow. Due to the nature of workflows, we note that workflow items can be nested. Indeed, this nesting is necessary for any workflow beyond the most trivial. 137 | 138 | We cover the workflow items in a loose order of complexity. Beginning with the simplest, we build upon these -- reusing and expanding our example -- to assist in the explanation of subsequent workflow items. Firstly, however, we must define our recurring example. 139 | 140 | #### A Recurring Example [example] 141 | 142 | For all skeletons described below, we shall use the running example described herein. The use of which allows us to illustrate how skeletons may be employed in conjunction and comparison with one another. 143 | 144 | With the ubiquity of digital images, and 'Photoshop' being entered as a verb into the the Oxford English Dictionary, it is not unfair to suggest the manipulation of images useful. When the size of today's images is also taken into account, the ability to perform these manipulations *quickly* becomes highly desirable. 145 | 146 | One example of such a manipulation, and a manipulation amenable to parallelisation, is the Image Merge operation. As its name might imply, and as can be seen in the below figure, Image Merge simply combines two images -- overlaying one atop the other. 147 | 148 | ![An example of Image Merge](img_merge.png) 149 | 150 | Image Merge may be divided into three stages: reading, merging, and writing. For this example, and for reasons of simplicity, we do not consider the writing stage. From the remaining two stages, we thus define the functions `readImage/1` and `convertMerge/1`. 151 | 152 | readImage({FileName, FileName2, Output}) -> 153 | {ok, _Img=#erl_image{format=F1, pixmaps=[PM]}} = 154 | erl_img:load(FileName), 155 | #erl_pixmap{pixels=Cols} =PM, 156 | R = lists:map(fun({_A,B}) -> B end, Cols), 157 | 158 | {ok, _Img2=#erl_image{format=F2, pixmaps=[PM2]}} = 159 | erl_img:load(FileName2), 160 | #erl_pixmap{pixels=Cols2} =PM2, 161 | R2 = lists:map(fun({_A2,B2}) -> B2 end, Cols2), 162 | 163 | {R, R2, F1, F2, Output}. 164 | 165 | convertMerge({R, R2, F1, F2, Name}) -> 166 | R1_p = lists:map(fun(L) -> removeAlpha(L, F1) end, R), 167 | R2_p = lists:map(fun(L2) -> removeAlpha(L2, F2) end, R2), 168 | WhiteR = lists:map(fun(Col) -> convertToWhite(Col) end, R1_p), 169 | Result = lists:zipwith(fun(L1,L2) -> mergeTwo(L1, L2) end, 170 | WhiteR, R2_p), 171 | {Result, length(R), Name}. 172 | 173 | The former, `readImage/1`, takes three filenames in a three-element tuple, or triple, and similarly produces a five-element tuple containing two images as binaries, their representation format, and the filename of the output file. As mentioned previously, this output name will not be used in our example as we are ignoring the write phase. Regarding the images themselves, the Erlang library [`erl_img`](https://github.com/mochi/erl_img) is used to load the images, and return their binary representation and image format. 174 | 175 | The output of `readImage/1` serves as input for `convertMerge/1`, wherein the composition takes place. The method of merging is simplistic, and mostly for illustration purposes. A triple is returned, containing the resulting image as a binary, the number of columns in that image, and the output name. 176 | 177 | These two functions are representative of the first two stages of Image Merge, and perform all operations required of that stage. For our example, these functions are defined in a module titled `conversion`. To perform the Image Merge *sequentially*, we might define a function `sequential/1` in a module named `image`. 178 | 179 | sequential(ImgTuple) -> 180 | conversion:convertMerge(conversion:readImage(ImgTuple)). 181 | 182 | One key motivation for introducing parallelism is that of performance, where *effective* parallelism may introduce significant speedups. To assist in comparing the skeletons included in Skel, we shall therefore consider absolute speedups achieved for each skeleton over Image Merge. 183 | 184 | All measurements have been made on an 800Mhz, 24-core, dual AMD Opteron 6176 machine, running Centos Linux 2.6.18-274.e15 and Erlang R16B02. Speedups are calculated by dividing the execution time of the sequential variant by the execution time of the parallel variant in question. All execution times used are the average of ten runs, calculated using Skel's benchmarking function, [`profile:benchmark/3`][timings]. Twenty-five pairs of 1024x1024 images serve as input. 185 | 186 | Having defined our recurring example sequentially, we now look towards its parallelisation. We first consider that which will allow our two functions to work within the context of Skel. 187 | 188 | #### `seq` -- The Sequential Function [seq] 189 | 190 | The sequential function is the simplest element, and effective base case, in the Skel library. 191 | 192 | The `seq` skeleton formats a given function, so that it may receive input within the framework of the Skel library. Inputs are passed as messages to one or more processes that are designed to solely receive these input messages, apply said input to the given function, and send on the result. 193 | 194 | Whilst `seq` does not introduce any parallelism beyond that which is inherent in the invocation of Skel, it is nevertheless a necessary bridge between the sequential and parallel variants of a program, or parts thereof. In our example we have two functions, as such we produce two workflow items compatible with Skel. 195 | 196 | The workflow item for the read phase may be constructed thus. 197 | 198 | {seq, fun conversion:readImage/1} 199 | 200 | Similarly, for the convert phase we construct: 201 | 202 | {seq, fun conversion:convertMerge/1} 203 | 204 | Here, we observe that the passed functions are denoted by the keyword `fun`. In this example, these functions are found in the `conversion` module. Functions declared in the same module as the call to Skel may also be used; in such a case, the name of the module is replaced with the Erlang macro `?MODULE`. 205 | 206 | Irrespective of the location of the provided function, and in explanation of how these are used, Skel takes `readImage/1` -- alternatively `convertMerge/1` -- and spawns a process ready to receive input. We call such processes *worker processes*, or more simply, *workers*. When input is received, the worker will apply that input to `readImage/1`. The calculated result, returned by `readImage/1`, is then sent to the process specified by the input, ready for the next stage in the workflow. In the case of a workflow with only one `seq` item, the result of the given function is simply returned to the process that called it. 207 | 208 | If we wish to use only the read phase, for example, we might define the function `tutorialSeq/0`. 209 | 210 | tutorialSeq() -> 211 | skel:do([{seq, fun conversion:readImage/1}], Images). 212 | 213 | Here, Skel takes `readImage/1` and the imaginatively named list of image pairs, `Images`. When evaluated, `tutorialSeq/0` will return a list of all five-element tuples that are produced when each tuple in `Images` is applied to `readImage/1`. 214 | 215 | The same may also be done for `convertMerge/1`, with similar results. These two invocations might then be called one after the other, applying the output of the former as input to the latter. Alternatively, one might use both workflow items in the same invocation. We update `tutorialSeq/0` thus: 216 | 217 | tutorialSeq() -> 218 | skel:do([{seq, fun conversion:readImage/1}, 219 | {seq, fun conversion:convertMerge/1}], Images). 220 | 221 | In this instance, Skel performs both sequential functions. Image pairs are passed from `Images` to `readImage/1`. The output of which is passed as input to `convertMerge/1`. In turn, the output of this function is returned as the result of `skel:do`. 222 | 223 | As no true parallelism is invoked when applying only `seq` workflow items, we might expect to see slowdowns instead of speedups. Indeed, running `tutorialSeq/0` for twenty-five image pairs is twice as slow as the sequential variant. This 'speedup' of 0.47 is likely to be a result of the cost it takes to invoke and use Skel with only two sequential functions. These costs are made profitable in future examples as the skeletons employed save more time than Skel requires. 224 | 225 | This example illustrates the simplicity of the sequential function, and indeed, why it might be described as our base case. It is the worker item that does the basic work, as defined by the developer, and system in which Skel is being applied. Whilst little parallelism is used here, future skeletons are dependent upon it. 226 | 227 | As in Image Merge, functions rarely exist alone -- they interact with one another. This means we must somehow chain sequential functions together. 228 | 229 | #### `pipe` - The Pipe Skeleton [pipe] 230 | 231 | ![](pipe.png "Pipe") 232 | 233 | A pipeline, or pipe, is one of the simplest skeletons. For each stage within the pipeline, that stage's input is the previous stage's output. Parallelism is gained through the application of different inputs at different stages of the pipeline. Indeed, we have already seen an example of a pipe, albeit implicit. 234 | 235 | In the final version of `tutorialSeq/1` we used both sequential functions in a single invocation of Skel. We observed that the output of the first was used as input for the second. Implicitly we used a pipeline. 236 | 237 | This example may be made explicit by using the `pipe` workflow item. We define `tutorialPipe/0` to illustrate this: 238 | 239 | tutorialPipe() -> 240 | skel:do([{pipe, [{seq, fun conversion:readImage/1}, 241 | {seq, fun conversion:convertMerge/1}]}], Images). 242 | 243 | In `tutorialPipe/0`, the two `seq` workflow items are nested within the `pipe` tuple -- composed of the atom `pipe`, and a list of said nested workflow items. By default, a Skel workflow is a pipeline skeleton, and so we shall continue to use pipelines implicitly in future sections. 244 | 245 | With `tutorialPipe/0` being equivalent to `tutorialSeq/0` one might expect similar results in terms of performance. Indeed, this is not incorrect -- we receive a speedup of 0.51. 246 | 247 | Again, we find this example relatively trivial -- we are only making the implicit explicit. Nevertheless, the pipeline allows for results to be passed between two or more skeletons where nesting those skeletons is either inadvisable or not possible. 248 | 249 | #### `farm` -- The Farm Skeleton [farm] 250 | 251 | ![](farm.png "Farm") 252 | 253 | Another simple skeleton, and one more obviously parallel, is the task farm. As illustrated in the above diagram, a farm applies a given function on a number of inputs at the same time. The function is applied using a specified number of workers, is the same for all workers, and in the case of Skel, is defined in terms of a workflow. 254 | 255 | In our running example, we have two sequential functions. Whilst `convertMerge/1` is dependent on the output of `readImage/1`, within these functions, inputs are independent. This allows us to perform the same function applied to different inputs at the same time, and so enabling the use of a task farm. 256 | 257 | There are two possible ways we might apply a farm to our running example. The first means of application is almost identical to our making the pipeline explicit in the previous section. We might therefore define `tutorialFarmFirst/0` hence. 258 | 259 | tutorialFarmFirst() -> 260 | skel:do([{farm, [{seq, fun conversion:readImage/1}, 261 | {seq, fun conversion:convertMerge/1}], 10}], Images)). 262 | 263 | Here, we use the implicit pipeline workflow from [`tutorialSeq/0`][seq] nested within the `farm` workflow item. The `farm` workflow item consists of its identifier atom, a workflow, and the number of farm workers. In this example we have set the number of workers to ten, but this may be set to howsoever many workers desired. For example, we note it possible, albeit trivial, to produce a farm with a single worker -- thus effectively replicating `tutorialPipe/0`. 264 | 265 | An alternative is thus defined: 266 | 267 | tutorialFarmSecond() -> 268 | skel:do([{farm, [{seq, fun conversion:readImage/1}], 10}, 269 | {farm, [{seq, fun conversion:convertMerge/1}], 10}], Images). 270 | 271 | Under `tutorialFarmSecond/0` we have adjusted the relationship between the pipeline and farm skeletons. Now we observe that the two phases of Image Merge are themselves evaluated using two separate farms, with the output of the first piped as input to the second. The work performed by both `tutorialFarmFirst/0` and `tutorialFarmSecond/0` is equivalent, and produces similarly equivalent results. 272 | 273 | The same cannot be said of performance however. For `tutorialFarmFirst/0` and `tutorialFarmSecond/0`, our running example receives speedups of 11.59 and 9.1 respectively. But before we tackle this discrepancy, we note this a great improvement compared with both `tutorialSeq/0` and `tutorialPipe/0`. Here, we observe the workers saving sufficient time to not only cover the cost of invoking the library, but to improve upon the basic sequential Image Merge. 274 | 275 | In comparing the two task farm approaches, we remark that speedups may vary. That the degree to which performance is improved depends upon how the skeletons are used and arranged. These decisions are up to the developer, and the program on which he is working -- hence, it remains beneficial to consider alternative variations when using skeletons to introduce parallelism. Luckily, and by virtue of using skeletons, this is simple to do. We do again note, however, that either variant is a very good improvement over the sequential Image Merge. 276 | 277 | Similar to the difference between speedups, and whilst both `tutorialFarmFirst/0` and `tutorialFarmSecond/0` produce the same results for a given list of image pairs, these results need not necessarily be in the same order to either each other or the sequential variant. Indeed, the farm skeleton does not preserve ordering. 278 | 279 | #### `ord` -- The Ord Skeleton [ord] 280 | 281 | The *ord* skeleton is a wrapper that ensures output from its inner-workflow has the same ordering as the input ord receives. This is useful where other skeletons do not inherently preserve ordering, but where ordering is nevertheless desired. 282 | 283 | Similar to [pipe][pipe] and the [sequential function][seq], it is a two-element tuple consisting of its identifier atom, `ord`, and a worfklow. We note that ord does not affect the evaluation of the nested workflow in any way beyond the ordering of its results. 284 | 285 | To illustrate this, let us apply ord to our running example. We observed that the list of merged images produced by [`tutorialFarmFirst/0`][farm], or indeed [`tutorialFarmSecond/0`][farm], was not in the same order as its input list of image pairs, hence we wrap this task farm in an ord workflow item. 286 | 287 | tutorialOrd() -> 288 | skel:do([{ord, [{farm, [{seq, fun conversion:readImage/1}, 289 | {seq, fun conversion:convertMerge/1}], 10}]}], Images). 290 | 291 | When evaluated, `tutorialOrd/0` produces a list of images in the same order as the image pairs that produced them. 292 | 293 | In terms of performance, `tutorialOrd/0` produces a speedup of 9.43. This is expectedly similar to the speedups gained when using a task farm -- it is particularly similar to the speedup gained from `tutorialFarmSecond/0`. Yet, it remains that `tutorialOrd/0` uses `tutorialFarmFirst/0` for its inner-workflow. 294 | 295 | A speedup of 9.43 is reasonably less than that of 11.59. This, we observe, serves to highlight why task farms and other skeletons do not inherently preserve order. That, because the preservation of order is not always necessary, the possibly significant performance cost it entails does not always need to be paid. The ord skeleton enables the preservation of order only where it is required or desired, and thus the performance cost for this ordering need only be paid where it is applied. 296 | 297 | Regardless of their ordering, lists feature heavily in Erlang. Operations on these lists are also a common occurrence. It would be odd, then, for Skel to lack a skeleton whose focus they are. 298 | 299 | #### `reduce` -- The Reduce Skeleton [reduce] 300 | 301 | The `fold` higher-order function takes a function, a list, and a starting point, and recursively combines all elements in the list using the given function and starting point. Informally, it can be said to replace the `cons` function comprising said list with the developer-given function. The reduce skeleton is a parallelisation of `fold` -- i.e. where individual applications of the given function are run in parallel until only one element remains. 302 | 303 | To give an adequate demonstration of this skeleton and its use in Skel, we must first modify our running example. At present, Image Merge takes a list of tuples, which are, in turn, composed of three strings. These tuples are read via the function `readImage/1`, and then converted by `convertMerge/1`. To make best use of the reduce skeleton, we shall modify our running example to Batch Image Merge, in which an entire list of images are merged. 304 | 305 | Under Batch Image Merge we expect as input a two-dimensional list, with the inner lists containing the filename strings of those images to be merged. To process these, we shall again use a pipeline of two stages: the first being a task farm to read the images, and the second to merge these images using reduce. 306 | 307 | To accommodate the new input specification, we modify `readImage/1` thus. 308 | 309 | readImage([]) -> []; 310 | readImage(FileNames) -> 311 | FileName = hd(FileNames), 312 | {ok, _Img=#erl_image{format=F1, pixmaps=[PM]}} = 313 | erl_img:load(FileName), 314 | #erl_pixmap{pixels=Cols} =PM, 315 | R = lists:map(fun({_A,B}) -> B end, Cols), 316 | R_p = lists:map(fun(L) -> removeAlpha(L, F1) end, R), 317 | 318 | [R_p] ++ readImage(tl(FileNames)). 319 | 320 | As the input list of filenames is of variable length, unlike the tuples in our previous definition of `readImage/1`, we use recursion to produce the desired list of image binaries. Beyond the means to facilitate recursion, and the reading of a sole image into memory, the only other change incurred is that that the alpha channel is now removed during the read phase. This is done to simplify the input specification of `convertMerge/1`. 321 | 322 | Similarly, we modify `convertMerge/1` hence: 323 | 324 | convertMerge(R, R2) -> 325 | WhiteR = lists:map(fun(Col) -> convertToWhite(Col) end, R), 326 | Result = lists:zipwith(fun(L1,L2) -> mergeTwo(L1, L2) end, 327 | WhiteR, R2), 328 | Result. 329 | 330 | Now notably smaller, `convertMerge/2` receives as input two image binaries used for the image to be produced. The function performs only the latter two tasks of the original definition. Indeed, `convertMerge/1` also only returns the resulting binary image, in place of the triple of its previous incarnation. This facilitates the use of the output of a middling phase in the reduce operation as input for the next. 331 | 332 | We now define the function `tutorialRed/0` to invoke Skel and the reduce skeleton. 333 | 334 | tutorialRed() -> 335 | skel:do([{farm, [{seq, fun conversion:readImage/1}], 10}, 336 | {reduce, fun conversion:convertMerge/2, fun util:id/1}], 337 | ReduceImages). 338 | 339 | As one might expect, `ReduceImages` is our two-dimensional list of input images. We further observe, and as mentioned previously, that the images are loaded into memory using a task farm with ten workers. The output of which -- similarly a two-dimensional list of image binaries -- is piped as input to our reduce workflow item. 340 | 341 | The reduce workflow item is a triple composed of its identifier atom, a reduction function, and a decomposition function. The reduction function is used to combine, or *reduce*, two elements of the input list to a single element. Where the decomposition function, may be used to define how individual inputs may be decomposed and so reduced. 342 | 343 | For Batch Image Merge, `convertMerge/2` serves as our reduction function as it takes two images, returning one. As for decomposition, and for simplicity and clarity, the identity function is used. Our identity function, `id/1`, simply returns its input unchanged. 344 | 345 | That Batch Image Merge is a slightly different example to our primary, running example, makes it illogical to simply give and compare speedups between them for arbitrarily long lists of images. Yet, were we to restrict the batches of images to lists of length two, we mimic the tuples of our running example. Hence, when we quote a speedup of 9.91 for the reduce skeleton, this suggests it gained through the evaluation of twenty-five batches of two images. 346 | 347 | We might point to that of the speedups gained through the use of the Ord skeleton, 9.43, and through [`tutorialFarmSecond/0`][farm], 9.1, and consider it a good speedup. As `tutorialRed/0` is based upon the latter, even more so. We suggest, however, that for the example used here the application of the reduce skeleton is not overly desirable. We instead suggest its application desirable as an alternative to `fold`. 348 | 349 | Returning to our example, another point introduced in the change to Batch Image Merge is that we omit the means to define an output name for the similarly omitted writing phase. We suggest our modified example further modifiable to allow the inclusion of both said writing phase and output name. 350 | 351 | Keeping Batch Image Merge in mind, we consider the task farm once more. The farm skeleton works well when inputs are relatively small; but what of when this is not the case, when the inputs themselves might be better broken down and so worked on in their partite form. 352 | 353 | #### `map` and `cluster` -- The Map Skeleton [map] 354 | 355 | ![](map.png "Map") 356 | 357 | As with reduce in the [previous section][reduce], the map skeleton serves to parallelise the higher-order function with which it shares a name. The sequential map takes two arguments -- a list and a function. This function is applied to each element in that list. In Skel, and for each list passed as input, the map skeleton applies its inner-workflow to each element in each input in parallel. 358 | 359 | To achieve this parallelism, Skel again uses worker processes to which each element is sent and worked on. Unlike the task farm, map is able to automatically determine how many workers it requires. The total number of workers needed is determined by the length of its inputs. Should one input be longer than its forerunners, more workers are simply added so that the total number of workers is equal to the current input's length. 360 | 361 | Whilst the number of generated processes is not normally problematic due to the Erlang scheduler, it is also possible to specify the number of workers available to map. This might be desirable for when fine tuning performance, for example. 362 | 363 | When invoking the map skeleton, either a two- or three-element tuple is used. The first two elements of either tuple consist of the identifier atom, `map`, and an inner-workflow. The third element in the three-element tuple sets the number of workers used. 364 | 365 | The map skeleton is applied to lists, rendering it unsuitable to be used directly with our running example. The reduce skeleton in the [previous section][reduce] also used lists as input, and so we shall recycle our adjusted example here. 366 | 367 | In [`tutorialRed/0`][reduce], the task farm pertaining to the first phase of the operation takes a list of filenames as input, producing a list of the associated binary images. The function `readImage/1` is mapped to each element in each input list. Ergo, we define `tutorialMap/0` to use a map skeleton in lieu of the task farm in `tutorialRed/0`. 368 | 369 | tutorialMap() -> 370 | skel:do([{seq, fun util:makeLst/1}, 371 | {map, [{seq, fun conversion:readImage/1}], 10}, 372 | {reduce, fun conversion:convertMerge/2, fun util:id/1}], 373 | ReduceImages). 374 | 375 | In `tutorialMap/0`, we observe two things. The first is the setting of the number of workers available to the map; this may, of course, be removed. The second, and arguably, more immediate point of interest is that of the extra sequential function. 376 | 377 | As we change nothing beyond the introduction of the map skeleton, `ReduceImages` naturally remains the same. Yet, the input format required between the task farm and `readImage/1` is not the same as that between map and `readImage/1`. To bridge this disparity we include an additional sequential function to our pipeline that makes a single element list of each input. An alternative would be to again change `readImage/1`, `ReduceImages`, or both; an exercise that is left to the reader. 378 | 379 | Whilst a valid use of the map skeleton, a better example as to its employ might be during the second phase. In the merging of image pairs, and depending on image size, large amounts of binary data must be manipulated. This binary data might therefore be decomposed and each part worked on separately. To this end, we choose to employ the cluster skeleton. 380 | 381 | During map, inputs are decomposed, sent to a worker process, and then those partite elements recomposed ready to be forwarded onwards. Under a standard map, where the input is a list, this decomposition is equivalent to the identity function -- i.e. the list passes through unchanged. Similarly, recomposition is merely the creation of a list to which each decomposed element is appended. 382 | 383 | This process of decomposition and recomposition is made more generic in the cluster skeleton. The developer may define functions for both stages, this also happens to allow the manipulation of inputs amenable to map in a wider variety of formats. During execution, and having decomposed each input, cluster sends all decomposed elements to a single worker process at which a given inner-workflow will be applied to the aforementioned elements. 384 | 385 | ![](cluster.png "Cluster") 386 | 387 | To accommodate these, the cluster workflow item is a four-element tuple. It is composed of the identifier atom `cluster`, an inner-workflow, a decomposition function, and a recomposition function in that order. To demonstrate an application of the cluster skeleton, we now consider how it may be applied to the second phase in the standard version of our recurring example. 388 | 389 | The input for the second phase is a five-element tuple containing two images. These images are in the form of a list of binaries, and so may be decomposed, with each pair of elements merged separately. We thus define our imaginatively named decomposition function. 390 | 391 | decomp({[], _, _, _, _}) -> 392 | []; 393 | decomp({_, [], _, _, _}) -> 394 | []; 395 | decomp({R1, R2, F1, F2, Name}) -> 396 | [{[hd(R1)], [hd(R2)], F1, F2, Name}] ++ 397 | decomp({tl(R1), tl(R2), F1, F2, Name}). 398 | 399 | From this definition, we do highlight the caveat that both images should be of the same size. Regardless, we are now able to decompose our image pairs -- this is not altogether useful, however, unless we are able to recompose them. We define the similarly simple function `recomp/1` to do this. 400 | 401 | recomp(Parts) -> 402 | Img = lists:map(fun({[A], _B, _C}) -> A end, Parts), 403 | {_, _, Name} = hd(Parts), 404 | {Img, length(Img), Name}. 405 | 406 | Within this definition, we need not be concerned about different parts from different inputs being mixed together. The map skeleton automatically tracks the group to which a part belongs, as well as the order of the parts derived from each input. 407 | 408 | With our decomposition and recomposition functions defined, we have all components necessary to invoke Skel. For this we define the function `tutorialCluster/0`. 409 | 410 | tutorialCluster() -> 411 | skel:do([{farm, [{seq, fun conversion:readImage/1}], 10}, 412 | {cluster, [{farm, [{seq, fun conversion:convertMerge/1}], 10}], 413 | fun util:decomp/1, fun util:recomp/1}], Images). 414 | 415 | As in the standard map, we are inspired by [`tutorialFarmSecond/0`][farm]. `tutorialCluster/0` uses a task farm for the first phase, with the newly-defined `cluster` used in place of the second. We observe that a task farm is used as the inner-workflow to cluster. This is included so as to evaluate each decomposed element in parallel. 416 | 417 | For large images, it would seem the cluster skeleton quite suitable for Image Merge, with a speedup of 10.07 appearing to support this. Indeed, the use of cluster proves faster than `tutorialFarmSecond/0` which inspired it. It is likely that further adjustments, such as the use of more workers in the nested task farm, could improve these speedups further still. 418 | 419 | As with the speedup given in the [previous section][reduce], those gained for `tutorialMap/0` were the fruit of twenty-five lists of two-elements in length. Here we see the return of slowdowns, 0.4 in magnitude. We note this likely due to the sequential function introduced to the start of the pipeline. We further suggest that whilst this cost may be eliminated with further edits to the running example and the example used for reduce, the use of cluster remains more desirable for our running example due to its relatively simple application and good speedups. 420 | 421 | We suggest that, under this example, and between the two skeletons, the use of cluster is preferable to map. With the unadulterated recurring example, the merging of the image pairs is more computationally intensive than that of the loading of said images and so has the greatest opportunity for parallelism to be effective. Aside from implementation choices, we observe that under cluster the exemplar program needed little change to include the skeleton. Conversely, comparatively much greater change would be required for a truly suitable application of map. 422 | 423 | The `map` and `cluster` workflow items are similar in their execution, the difference being the latter is a wrapper skeleton. We have seen another wrapper skeleton in [`ord`][ord], and our final section and skeleton may also be thus considered. 424 | 425 | #### `feedback` -- The Feedback Skeleton [feedback] 426 | 427 | ![](feedback.png "Feedback") 428 | 429 | The feedback skeleton repeatedly applies each input to a given function until a set condition fails. Whilst the condition holds, the tested output is passed back into the skeleton as input. The overall returned result of the skeleton is a list of all transformed inputs in the form that failed the condition. 430 | 431 | Akin to [`ord`][ord] and [`cluster`][map], the feedback skeleton might be considered a wrapper as it may contain any other skeleton. In Skel, we see this in the form of a nested workflow -- itself a pipeline. 432 | 433 | One example of this is the application of feedback to a task farm. Recall that in a farm independent inputs are passed to a number of workers that apply a given function. The feedback skeleton adds an extra stage in which these returned results are then checked against the similarly given constraint-checking function, and either returned as input or output accordingly. 434 | 435 | The feedback workflow item is a triple that takes the identifying atom, `feedback`; a workflow; and a developer-defined function. As in both the [`ord`][ord] and [`cluster`][map] workflow items, the second element determines the workflow items that feedback modifies. The third element is the constraint-checking function used to check the given condition or conditions; it should return a boolean value. 436 | 437 | We might choose to apply the feedback skeleton to our running example. Firstly, we define the constraint-checking function `constraint/1` to always return false. We thus define `tutorialFeedback/0`. 438 | 439 | tutorialFeedback() -> 440 | skel:do([{farm, [{seq, fun conversion:readImage/1}], 10}, 441 | {feedback, [{farm, [{seq, 442 | fun conversion:convertMerge/1}], 10}], 443 | fun util:constraint/1}], Images). 444 | 445 | Using [`tutorialFarmSecond/0`][farm] as a base yet again, this function applies the feedback skeleton to the second task farm in the pipeline. Indeed, this example is effectively equivalent in terms of functionality to [`tutorialFarmSecond/0`][farm]. It arrives at the same result, barring perhaps ordering, due to each input only being passed through the task farm, and so `convertMerge/1`, once. Here we observe that the feedback skeleton always evaluates its workflow at least once. 446 | 447 | With `tutorialFeedback/0` we see speedups of 9.19 over our list of twenty-five image pairs. This is again similar to [`tutorialFarmSecond/0`][farm]. This result, as with the example itself, is relatively uninteresting beyond the suggestion that the feedback skeleton has less of an impact than [`ord`][ord]. 448 | 449 | Of course, it is eminently possible to apply the feedback skeleton to both stages in the pipeline, or to the sole task farm in [`tutorialFarmFirst/0`][farm]. Under the current definition of the constraint function it does not matter how the feedback skeleton is applied. As a consequence of this, such an example is relatively unsatisfying. Let us instead briefly consider the humble ant. 450 | 451 | Ant Colony Optimisation is a heuristic for solving difficult optimisation problems inspired by, rather unsurprisingly, the behaviour of ants and their colonies. Such algorithms involve a number of ants each independently solving a given problem. The ants are loosely guided by a pheromone trail laid by their predecessors. Following the ants' completion, this pheromone trail is updated using the best, newly-found solution. The ants then begin again; repeating this cycle until certain point is reached. 452 | 453 | This repetition, this *feedback* of the pheromone trail, makes Ant Colony Optimisation problems amenable to the feedback skeleton. For the constraint function, one might desire a certain quantity to be below a certain threshold. Alternatively, it could simply be a cap on the number of iterations performed. Where for the workflow, a map or farm skeleton might be used for ants to run their tasks in parallel. 454 | 455 | ### Other Notes [notes] 456 | 457 | #### Schedulers [scheduler] 458 | 459 | Being a library designed to assist in the parallisation of programs, it is worth remarking on Erlang processes. Normally, the Erlang virtual machine will start with access to all available cores and those cores will be used according to Erlang's scheduler. 460 | 461 | When performing tasks that are highly parallel, such as our recurring example and in conjunction with the Skel library, all cores will be used at near maximum capacity. It is, however, possible to restrict the number of cores available to Erlang's scheduler. 462 | 463 | To take an example, 464 | 465 | erlang:system_flag(schedulers_online, 4). 466 | 467 | will restrict the number of cores available to use to four. It is likely also worth noting that this number cannot be set higher than the maximum available number of cores on the base machine. 468 | 469 | This is commonly set in the function that calls Skel. For example, to restrict the above [`tutorialFarmFirst/0`][farm] example to use a variable number of cores one might modify the function hence. 470 | 471 | tutorialFarmFirst(X) -> 472 | erlang:system_flag(schedulers_online, X), 473 | skel:do([{farm, [{pipe, [{seq, fun conversion:readImage/1}, 474 | {seq, fun conversion:convertMerge/1}]}], 10}], Images)). 475 | 476 | 477 | #### Benchmarking [timings] 478 | 479 | Primarily designed for testing, the [`sk_profile`](../../doc/sk_profile.html) module included in Skel allows for relatively easy benchmarking of functions. [`benchmark/3`](../../doc/sk_profile.html#benchmark-3) takes a function to execute, the arguments to be passed to said function, and the number of times that function should be evaluated. It then produces a list of tuples giving a variety of averages based on the time taken to evaluate the given function the specified number of times. 480 | 481 | Using [`benchmark/3`](../../doc/sk_profile.html#benchmark-3) is similar in method to using the [ord][ord] skeleton. Indeed, to illustrate this, the below defines an exemplar function to its use. 482 | 483 | tutorialTimeOrd() -> 484 | sk_profile:benchmark(fun ?MODULE:tutorialOrd/0, [], 10). 485 | 486 | The function `tutorialTimeOrd/0` is defined in the same module as the majority of examples used throughout this tutorial and evaluates `tutorialTimeOrd/0` a total of ten times. The empty list is used here as `tutorialTimeOrd/0` takes no arguments. 487 | 488 | This function returns a list of tuples containing a number of timings in microseconds, listed below. 489 | 490 | 1. The number of times the function was evaluated. 491 | 2. The time taken by the quickest evaluation of `tutorialOrd/0`. 492 | 3. The time taken by the slowest evaluation. 493 | 4. The median of all times recorded for all evaluations. 494 | 5. The mean average of all times recorded. 495 | 6. The standard deviation for all times recorded. 496 | 497 | #### Debugging [debug] 498 | 499 | Finding errors in sequential code can often be a difficult task; regretfully parallel programs are no different. 500 | 501 | Whilst Skel is designed to avoid the common errors associated with parallelism, should any problem occur, the `sk_tracer` module provides a means of seeing a visualisation of the messages sent between processes. 502 | 503 | More generally, the tracer allows one to view the passing of information in a visual manner. Indeed, this may assist in understanding how a particular operation or skeleton works under a given set of circumstances. The tracer viewer may be displayed using the following command in the Erlang console. 504 | 505 | sk_tracer:start(). 506 | 507 | The visualiser is then displayed, and populated with the messages sent by the next call to Skel. 508 | 509 | ### Further Reading [read] 510 | 511 | The below is a list of papers related to Skel and the ideas behind it. 512 | 513 | Algorithmic Skeletons: Structured Management of Parallel Computation 514 | : Murray Cole 515 | : MIT Press (1991) 516 | : ISBN: 0-262-53086-4 517 | : URL: [Link](http://homepages.inf.ed.ac.uk/mic/Pubs/skeletonbook.ps.gz) 518 | 519 | Bringing skeletons out of the closet: a pragmatic manifesto for skeletal parallel programming 520 | : Murray Cole 521 | : Parallel Computing 30 (2004) pp. 389--406 522 | : Elsevier Science Publishers B. V., Amsterdam 523 | : DOI: 10.1016/j.parco.2003.12.002 524 | : URL: [Link](http://dx.doi.org/10.1016/j.parco.2003.12.002) 525 | 526 | A Survey of Algorithmic Skeleton Frameworks: High-level Structured Parallel Programming Enablers 527 | : Horatio González-Vélez, Mario Leyton 528 | : Software—Practice & Experience - Focus on Selected PhD Literature Reviews in the Practical Aspects of Software Technology, 40 (2010), pp. 1135--1160 529 | : John Wiley & Sons, Inc. 530 | : DOI: 10.1002/spe.v40:12 531 | : URL: [Link](http://www.comp.rgu.ac.uk/staff/hg/publications/skeletal-reviewR3.pdf) 532 | 533 | Paraphrasing: Generating Parallel Programs using Refactoring 534 | : Christopher Brown, Kevin Hammond, Marco Danelutto, Peter Kilpatrick, Holger Schöner, Tino Breddin 535 | : Formal Methods for Components and Objects 7542 (2013), pp. 237--256 536 | : Springer Berlin Heidelberg 537 | : DOI: 10.1007/978-3-642-35887-6_13 538 | : URL: [Link](http://www.paraphrase-ict.eu/paperList/fmco-refactoring/) 539 | 540 | Cost-Directed Refactoring for Parallel Erlang Programs 541 | : Christopher Brown, Marco Danelutto, Kevin Hammond, Peter Kilpatrick, Archibald Elliott 542 | : International Journal of Parallel Programming 543 | : Springer US 544 | : DOI: 10.1007/s10766-013-0266-5 545 | : URL: [Link](http://dx.doi.org/10.1007/s10766-013-0266-5) 546 | --------------------------------------------------------------------------------