├── README.md ├── lab1 ├── .gitignore ├── README.md ├── config │ └── config.exs ├── lib │ └── lab1.ex ├── mix.exs ├── solution.ex └── test │ ├── lab1_test.exs │ └── test_helper.exs ├── lab2 ├── .formatter.exs ├── .gitignore ├── README.md ├── lib │ └── lab2 │ │ ├── streams.ex │ │ └── tasks.ex ├── mix.exs ├── solutions_streams.ex ├── solutions_tasks.ex └── test │ ├── lab2_test.exs │ └── test_helper.exs ├── lab3 ├── .formatter.exs ├── .gitignore ├── README.md ├── lib │ └── lab3 │ │ ├── partitions.ex │ │ └── rate_limiting.ex ├── mix.exs ├── mix.lock ├── solution_partitions.ex ├── solutions_rate_limited_consumer.ex └── test │ ├── lab3_test.exs │ └── test_helper.exs ├── lab4 ├── .formatter.exs ├── .gitignore ├── README.md ├── config │ └── config.exs ├── lib │ └── lab4.ex ├── mix.exs ├── mix.lock ├── solution.ex └── test │ ├── lab4_test.exs │ └── test_helper.exs └── pride-and-prejudice.txt /README.md: -------------------------------------------------------------------------------- 1 | # README 2 | 3 | ### Lab requirements 4 | 5 | 1. Install git 6 | - Windows: Install via https://git-scm.com 7 | - OS X: Install with package manager or https://git-scm.com 8 | - Linux: Install via distribution package manager or https://git-scm.com 9 | 10 | 2. Install Erlang & Elixir (at least 1.6) 11 | - See https://elixir-lang.org/install.html 12 | 13 | ##### Verify your installations 14 | 15 | Verify your installation by calling the executables `git`, `erl` and `iex` in 16 | your shell. Your Erlang installation needs to be on OTP 19 and Elixir on 1.6 17 | or higher. Check the version by running `iex`: 18 | 19 | ``` 20 | ~ λ iex 21 | Erlang/OTP 20 [erts-9.3] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:10] [hipe] [kernel-poll:false] 22 | 23 | Interactive Elixir (1.6.5) - press Ctrl+C to exit (type h() ENTER for help) 24 | iex(1)> 25 | ``` 26 | 27 | 28 | ### Installing a lab 29 | 30 | The repository for all labs is found at https://github.com/whatyouhide/workshop-parallel-computation-with-elixir. 31 | 32 | 1. git clone from your console 33 | 34 | ```bash 35 | $ git clone https://github.com/whatyouhide/workshop-parallel-computation-with-elixir 36 | ``` 37 | 38 | 2. Enter the workshop directory 39 | 40 | ```bash 41 | $ cd workshop-parallel-computation-with-elixir 42 | ``` 43 | 44 | 2. Enter the directory for the current lab named `labN`. For example, for `lab1`: 45 | 46 | ```bash 47 | $ cd lab1 48 | ``` 49 | 50 | 51 | ### Running tests 52 | 53 | As you start out with a new lab all tests will be skipped with `@tag :skip`, remove this tag when 54 | you start working on a test. 55 | 56 | 57 | Tests are tagged with `@tag :skip`. When you run `mix test`, all tests are skipped so you will see 58 | all tests "passing". You should remove or comment out the `@tag :skip` line above the first test 59 | and then run tests to see that test fail. Then, implement the first function until the test 60 | passes. Repeat for all other functions. 61 | 62 | * Run all tests for a project: `$ mix test` 63 | 64 | * Run all tests in a specific file: `$ mix test test/my_test.exs` 65 | 66 | * Run all tests on a specific file and line: `$ mix test test/my_test.exs:42` 67 | 68 | 69 | ### Lab links 70 | 71 | The individual labs can be found at the following URLs: 72 | 73 | *Lab 1* - Process basics: https://github.com/whatyouhide/workshop-parallel-computation-with-elixir/tree/master/lab1 74 | 75 | *Lab 2* - Streams and tasks: https://github.com/whatyouhide/workshop-parallel-computation-with-elixir/tree/master/lab2 76 | 77 | *Lab 3* - GenStage: https://github.com/whatyouhide/workshop-parallel-computation-with-elixir/tree/master/lab3 78 | 79 | *Lab 4* - Flow: https://github.com/whatyouhide/workshop-parallel-computation-with-elixir/tree/master/lab4 80 | -------------------------------------------------------------------------------- /lab1/.gitignore: -------------------------------------------------------------------------------- 1 | # The directory Mix will write compiled artifacts to. 2 | /_build 3 | 4 | # If you run "mix test --cover", coverage assets end up here. 5 | /cover 6 | 7 | # The directory Mix downloads your dependencies sources to. 8 | /deps 9 | 10 | # Where 3rd-party dependencies like ExDoc output generated docs. 11 | /doc 12 | 13 | # If the VM crashes, it generates a dump, let's ignore it too. 14 | erl_crash.dump 15 | 16 | # Also ignore archive artifacts (built via "mix archive.build"). 17 | *.ez 18 | -------------------------------------------------------------------------------- /lab1/README.md: -------------------------------------------------------------------------------- 1 | # Lab1 2 | 3 | ### Lab purpose 4 | 5 | Learn about process basics and message passing. You'll have some simple exercises 6 | that make use of basic Elixir concurrency using primitives. 7 | 8 | ### Lab instructions 9 | 10 | The `lab1` directory has a file `lib/lab1.ex` which contains an implementation of chat 11 | room with some stubbed out functions. As usual, use that template to implement features 12 | in your chatroom until all of your tests pass. 13 | 14 | 1. Create a process that just prints the first message it receives 15 | and then dies, it should return the `pid` of the new process. The message needs to 16 | be printed to the console with `IO.write(:stderr, message)` to be able to 17 | easily capture it in tests. 18 | 19 | 2. Create a process that prints messages like in the previous task, but instead 20 | of dying after the first message, keep printing new messages with `IO.write(:stderr, message)`. 21 | 22 | 3. Create a process that waits for a message containing a list, sum the list 23 | (from lab1) and reply to the original process. The message will be in the format: 24 | `{:sum, pid, list}`, where `pid` is where the reply should be sent. 25 | 26 | 4. Spawn N number of processes doing the previous task in parallel and wait 27 | for the responses. (As an extra exercise make sure to return the results in 28 | the order the input was given.) 29 | 30 | 31 | ### Links 32 | 33 | Getting started guide: https://elixir-lang.org/getting-started/introduction.html 34 | 35 | API docs: https://hexdocs.pm/elixir/ 36 | 37 | 1. `spawn/1` https://hexdocs.pm/elixir/Kernel.html#spawn/1 38 | 39 | 2. `send/2` https://hexdocs.pm/elixir/Kernel.html#send/2 40 | 41 | 3. `receive/1` https://hexdocs.pm/elixir/Kernel.SpecialForms.html#receive/1 42 | 43 | 4. `Process` https://hexdocs.pm/elixir/Process.html 44 | 45 | 5. Processes https://elixir-lang.org/getting-started/processes.html 46 | 47 | 48 | ### Solution ( no peeking :) ) 49 | 50 | See `solution.ex` in the `lab1` directory. 51 | -------------------------------------------------------------------------------- /lab1/config/config.exs: -------------------------------------------------------------------------------- 1 | # This file is responsible for configuring your application 2 | # and its dependencies with the aid of the Mix.Config module. 3 | use Mix.Config 4 | 5 | # This configuration is loaded before any dependency and is restricted 6 | # to this project. If another project depends on this project, this 7 | # file won't be loaded nor affect the parent project. For this reason, 8 | # if you want to provide default values for your application for 9 | # 3rd-party users, it should be done in your "mix.exs" file. 10 | 11 | # You can configure for your application as: 12 | # 13 | # config :lab1, key: :value 14 | # 15 | # And access this configuration in your application as: 16 | # 17 | # Application.get_env(:lab1, :key) 18 | # 19 | # Or configure a 3rd-party app: 20 | # 21 | # config :logger, level: :info 22 | # 23 | 24 | # It is also possible to import configuration files, relative to this 25 | # directory. For example, you can emulate configuration per environment 26 | # by uncommenting the line below and defining dev.exs, test.exs and such. 27 | # Configuration from the imported file will override the ones defined 28 | # here (which is why it is important to import them last). 29 | # 30 | # import_config "#{Mix.env}.exs" 31 | -------------------------------------------------------------------------------- /lab1/lib/lab1.ex: -------------------------------------------------------------------------------- 1 | defmodule Lab1 do 2 | def print_first_message() do 3 | raise "not implemented yet" 4 | end 5 | 6 | def print_all_messages() do 7 | raise "not implemented yet" 8 | end 9 | 10 | def sum() do 11 | raise "not implemented yet" 12 | end 13 | 14 | def sum_all(_list_of_lists) do 15 | raise "not implemented yet" 16 | end 17 | end 18 | -------------------------------------------------------------------------------- /lab1/mix.exs: -------------------------------------------------------------------------------- 1 | defmodule Lab1.Mixfile do 2 | use Mix.Project 3 | 4 | def project do 5 | [app: :lab1, 6 | version: "0.1.0", 7 | elixir: "~> 1.0", 8 | build_embedded: Mix.env == :prod, 9 | start_permanent: Mix.env == :prod, 10 | deps: deps()] 11 | end 12 | 13 | # Configuration for the OTP application 14 | # 15 | # Type "mix help compile.app" for more information 16 | def application do 17 | [applications: [:logger]] 18 | end 19 | 20 | # Dependencies can be Hex packages: 21 | # 22 | # {:mydep, "~> 0.3.0"} 23 | # 24 | # Or git/path repositories: 25 | # 26 | # {:mydep, git: "https://github.com/elixir-lang/mydep.git", tag: "0.1.0"} 27 | # 28 | # Type "mix help deps" for more examples and options 29 | defp deps do 30 | end 31 | end 32 | -------------------------------------------------------------------------------- /lab1/solution.ex: -------------------------------------------------------------------------------- 1 | defmodule Lab1 do 2 | def print_first_message() do 3 | spawn(fn -> 4 | receive do 5 | message -> IO.write(:stderr, message) 6 | end 7 | end) 8 | end 9 | 10 | def print_all_messages() do 11 | spawn(&print_all_messages_recursive/0) 12 | end 13 | 14 | defp print_all_messages_recursive() do 15 | receive do 16 | message -> 17 | IO.write(:stderr, message) 18 | print_all_messages_recursive() 19 | end 20 | end 21 | 22 | def sum() do 23 | spawn(fn -> 24 | receive do 25 | {:sum, pid, list} -> send(pid, Enum.sum(list)) 26 | end 27 | end) 28 | end 29 | 30 | def sum_all(list_of_lists) do 31 | caller_pid = self() 32 | 33 | refs = 34 | Enum.map(list_of_lists, fn list -> 35 | ref = make_ref() 36 | spawn(fn -> 37 | send(caller_pid, {ref, Enum.sum(list)}) 38 | end) 39 | ref 40 | end) 41 | 42 | Enum.map(refs, fn ref -> 43 | receive do 44 | {^ref, result} -> result 45 | end 46 | end) 47 | end 48 | end 49 | -------------------------------------------------------------------------------- /lab1/test/lab1_test.exs: -------------------------------------------------------------------------------- 1 | defmodule Lab1Test do 2 | use ExUnit.Case 3 | import ExUnit.CaptureIO 4 | 5 | defp receive_once do 6 | receive do 7 | result -> result 8 | end 9 | end 10 | 11 | @tag :skip 12 | test "task 1: print_first_message" do 13 | pid = Lab1.print_first_message() 14 | 15 | assert capture_io(:stderr, fn -> 16 | send(pid, "foo") 17 | # Wait until output has been captured 18 | Process.sleep(100) 19 | end) == "foo" 20 | 21 | assert capture_io(:stderr, fn -> 22 | send(pid, "foo") 23 | # Wait until output has been captured 24 | Process.sleep(100) 25 | end) == "" 26 | 27 | refute Process.alive?(pid) 28 | end 29 | 30 | @tag :skip 31 | test "task 2: print_all_messages" do 32 | pid = Lab1.print_all_messages() 33 | 34 | assert capture_io(:stderr, fn -> 35 | send(pid, "foo") 36 | # Wait until output has been captured 37 | Process.sleep(100) 38 | end) == "foo" 39 | 40 | assert capture_io(:stderr, fn -> 41 | send(pid, "bar") 42 | # Wait until output has been captured 43 | Process.sleep(100) 44 | end) == "bar" 45 | 46 | assert Process.alive?(pid) 47 | end 48 | 49 | @tag :skip 50 | test "task 3: sum" do 51 | pid = Lab1.sum() 52 | send(pid, {:sum, self(), 1..2}) 53 | assert receive_once() == 3 54 | 55 | pid = Lab1.sum() 56 | send(pid, {:sum, self(), 1..10}) 57 | assert receive_once() == 55 58 | end 59 | 60 | @tag :skip 61 | test "task 4: sum_all" do 62 | assert Lab1.sum_all([1..2]) == [3] 63 | assert Lab1.sum_all([1..2, 3..4, 5..6]) == [3, 7, 11] 64 | 65 | assert Lab1.sum_all([[1, 2]]) == [3] 66 | assert Lab1.sum_all([[1, 2], [3, 4], [5, 6]]) == [3, 7, 11] 67 | end 68 | end 69 | -------------------------------------------------------------------------------- /lab1/test/test_helper.exs: -------------------------------------------------------------------------------- 1 | ExUnit.start(exclude: [:skip]) 2 | -------------------------------------------------------------------------------- /lab2/.formatter.exs: -------------------------------------------------------------------------------- 1 | # Used by "mix format" 2 | [ 3 | inputs: ["{mix,.formatter}.exs", "{config,lib,test}/**/*.{ex,exs}"] 4 | ] 5 | -------------------------------------------------------------------------------- /lab2/.gitignore: -------------------------------------------------------------------------------- 1 | # The directory Mix will write compiled artifacts to. 2 | /_build/ 3 | 4 | # If you run "mix test --cover", coverage assets end up here. 5 | /cover/ 6 | 7 | # The directory Mix downloads your dependencies sources to. 8 | /deps/ 9 | 10 | # Where 3rd-party dependencies like ExDoc output generated docs. 11 | /doc/ 12 | 13 | # Ignore .fetch files in case you like to edit your project deps locally. 14 | /.fetch 15 | 16 | # If the VM crashes, it generates a dump, let's ignore it too. 17 | erl_crash.dump 18 | 19 | # Also ignore archive artifacts (built via "mix archive.build"). 20 | *.ez 21 | 22 | # Ignore package tarball (built via "mix hex.build"). 23 | lab2-*.tar 24 | 25 | -------------------------------------------------------------------------------- /lab2/README.md: -------------------------------------------------------------------------------- 1 | # Lab2 2 | 3 | ### Lab purpose 4 | 5 | Learn about streams and tasks. There will be exercises for playing with both. 6 | 7 | ### Lab instructions 8 | 9 | The `lab2` directory is divided into two files: `streams.ex` and `tasks.ex`. They both contain 10 | stubbed out functions that you will implement. The job of each function to implement is described 11 | in its documentation. 12 | 13 | ### Links 14 | 15 | Getting started guide ("Enumerables and Streams" chapter): https://elixir-lang.org/getting-started/enumerables-and-streams.html 16 | 17 | API docs: https://hexdocs.pm/elixir/ 18 | 19 | 1. `Stream` https://hexdocs.pm/elixir/Stream.html 20 | 21 | 2. `Enum` https://hexdocs.pm/elixir/Enum.html 22 | 23 | 3. `Task` https://hexdocs.pm/elixir/Task.html 24 | 25 | 26 | ### Solution ( no peeking :) ) 27 | 28 | See `solutions_streams.ex` for streams exercises and `solutions_tasks.ex` for tasks exercises in 29 | the `lab2` directory. 30 | -------------------------------------------------------------------------------- /lab2/lib/lab2/streams.ex: -------------------------------------------------------------------------------- 1 | defmodule Lab2.Streams do 2 | @doc """ 3 | Counts the words in the given file until the first line that contains "stop word". 4 | 5 | This function should take all the lines of the given file until the first one that contains 6 | `stop_word` and count the occurrences of each word only on these lines. The lines should be 7 | normalized (by calliing `normalize_string/1`) before counting the words. 8 | 9 | The result should be a map like `%{word => count}` with words as keys and number of occurrences 10 | as values. 11 | 12 | ## Example 13 | 14 | Considering a file with these contents: 15 | 16 | # my_file.txt 17 | Hello world! 18 | Everything good world? 19 | Awesome, see you later. 20 | 21 | then this function should behave like follows: 22 | 23 | count_words_until_amusement("my_file.txt", "awesome") 24 | #=> %{"hello" => 1, "world" => 2, "everything" => 1, "good" => 1} 25 | 26 | """ 27 | @spec count_words_until_stop_word(Path.t(), String.t()) :: %{String.t() => pos_integer()} 28 | def count_words_until_stop_word(_file_path, _stop_word) do 29 | raise normalize_string("not implemented yet") 30 | end 31 | 32 | defp normalize_string(string) do 33 | string 34 | |> String.downcase() 35 | |> String.replace(~w(, ? . ! ; _ “ ”), "") 36 | end 37 | 38 | @doc """ 39 | Takes elements from the given `enum` while `predicate` returns true for those elements. 40 | 41 | As soon as `predicate` returns false for a given element, the enumeration stops. 42 | 43 | ## Examples 44 | 45 | stream = take_while([1, 2, 3, 4, 5], fn elem -> elem < 4 end) 46 | Enum.to_list(stream) 47 | #=> [1, 2, 3] 48 | 49 | """ 50 | @spec take_while(Enumerable.t(), (term() -> boolean())) :: Enumerable.t() 51 | def take_while(enum, predicate) do 52 | raise "not implemented yet" 53 | end 54 | 55 | @doc """ 56 | Takes an enumerable and returns that enumerable but without consecutive duplicate elements. 57 | 58 | ## Examples 59 | 60 | stream = dedup([1, 2, 1, 1, 5, 8, 8, 8, 1]) 61 | Enum.to_list(stream) 62 | #=> [1, 2, 1, 5, 8, 1] 63 | 64 | """ 65 | @spec dedup(Enumerable.t()) :: Enumerable.t() 66 | def dedup(enum) do 67 | raise "not implemented yet" 68 | end 69 | 70 | @doc """ 71 | Takes an enumerable and returns that enumerable but without duplicate elements. 72 | 73 | ## Examples 74 | 75 | stream = dedup([1, 2, 1, 1, 5, 8, 8, 8, 1]) 76 | Enum.to_list(stream) 77 | #=> [1, 2, 5, 8] 78 | 79 | """ 80 | @spec uniq(Enumerable.t()) :: Enumerable.t() 81 | def uniq(enum) do 82 | raise "not implemented yet" 83 | end 84 | end 85 | -------------------------------------------------------------------------------- /lab2/lib/lab2/tasks.ex: -------------------------------------------------------------------------------- 1 | defmodule Lab2.Tasks do 2 | @doc """ 3 | Maps the given `fun` in parallel over all elements of `enum`. 4 | 5 | The end result of this function should be the same as `Enum.map/2`, except that 6 | each element is processed in parallel with all the others. 7 | 8 | ## Examples 9 | 10 | pmap([url1, url2, ...], fn url -> expensive_request(url) end) 11 | #=> [response1, response2, ...] 12 | 13 | """ 14 | @spec pmap(Enumerable.t(), (a -> b)) :: Enumerable.t() when a: term(), b: term() 15 | def pmap(enum, fun) when is_function(fun, 1) do 16 | raise "not implenented yet" 17 | end 18 | 19 | @doc """ 20 | Takes a stream of enumerables and returns a stream of the sum of each enumerable. 21 | 22 | ## Examples 23 | 24 | iex> Enum.to_list(sum_all([1..2, 3..4, 5..6])) 25 | [3, 7, 11] 26 | 27 | """ 28 | @spec sum_all(Enumerable.t()) :: Enumerable.t() 29 | def sum_all(stream_of_enums) do 30 | raise "not implemented yet" 31 | end 32 | 33 | @doc """ 34 | Spawns a process that executes the given computation (given as a function) and that can be 35 | awaited on. 36 | 37 | Suggestion: there's a function in the `Kernel` module that can help avoiding a race condition 38 | when spawning *and then* monitoring. 39 | 40 | ## Examples 41 | 42 | See the `await/1` function. 43 | """ 44 | @spec async((() -> term())) :: term() 45 | def async(fun) when is_function(fun, 0) do 46 | raise "not implemented yet" 47 | end 48 | 49 | @doc """ 50 | Awaits on the result of a computation started with `async/1` and returns `{:ok, result}` if 51 | the result was computed or `:error` if the process crashed. 52 | 53 | Note that if you have 54 | 55 | ref = Process.monitor(pid) 56 | 57 | then if `pid` dies for some reason a message will be delivered to the monitoring process. The 58 | message would have the following shape: 59 | 60 | {:DOWN, ref, _, _, _} 61 | 62 | ## Examples 63 | 64 | iex> async = Monitor.async(fn -> 1 + 10 end) 65 | iex> Monitor.await(async) 66 | {:ok, 11} 67 | 68 | """ 69 | @spec await(term()) :: term() 70 | def await(task) do 71 | raise "not implemented yet" 72 | end 73 | end 74 | -------------------------------------------------------------------------------- /lab2/mix.exs: -------------------------------------------------------------------------------- 1 | defmodule Lab2.MixProject do 2 | use Mix.Project 3 | 4 | def project do 5 | [ 6 | app: :lab2, 7 | version: "0.1.0", 8 | elixir: "~> 1.5", 9 | start_permanent: Mix.env() == :prod, 10 | deps: deps() 11 | ] 12 | end 13 | 14 | # Run "mix help compile.app" to learn about applications. 15 | def application do 16 | [ 17 | extra_applications: [:logger] 18 | ] 19 | end 20 | 21 | # Run "mix help deps" to learn about dependencies. 22 | defp deps do 23 | [ 24 | # {:dep_from_hexpm, "~> 0.3.0"}, 25 | # {:dep_from_git, git: "https://github.com/elixir-lang/my_dep.git", tag: "0.1.0"}, 26 | ] 27 | end 28 | end 29 | -------------------------------------------------------------------------------- /lab2/solutions_streams.ex: -------------------------------------------------------------------------------- 1 | defmodule Lab2.Streams do 2 | @doc """ 3 | Counts the words in the given file until the first line that contains "stop word". 4 | 5 | This function should take all the lines of the given file until the first one that contains 6 | `stop_word` and count the occurrences of each word only on these lines. The lines should be 7 | normalized (by calliing `normalize_string/1`) before counting the words. 8 | 9 | The result should be a map like `%{word => count}` with words as keys and number of occurrences 10 | as values. 11 | 12 | ## Example 13 | 14 | Considering a file with these contents: 15 | 16 | # my_file.txt 17 | Hello world! 18 | Everything good world? 19 | Awesome, see you later. 20 | 21 | then this function should behave like follows: 22 | 23 | count_words_until_amusement("my_file.txt", "awesome") 24 | #=> %{"hello" => 1, "world" => 2, "everything" => 1, "good" => 1} 25 | 26 | """ 27 | @spec count_words_until_stop_word(Path.t(), String.t()) :: %{String.t() => pos_integer()} 28 | def count_words_until_stop_word(file_path, stop_word) do 29 | file_path 30 | |> File.stream!() 31 | |> Stream.map(&normalize_string/1) 32 | |> Stream.take_while(¬(String.contains?(&1, stop_word))) 33 | |> Stream.flat_map(&String.split/1) 34 | |> Enum.group_by(& &1) 35 | |> Map.new(fn {word, words} -> {word, length(words)} end) 36 | end 37 | 38 | defp normalize_string(string) do 39 | string 40 | |> String.downcase() 41 | |> String.replace(~w(, ? . ! ; _ “ ”), "") 42 | end 43 | 44 | @doc """ 45 | Takes elements from the given `enum` while `predicate` returns true for those elements. 46 | 47 | As soon as `predicate` returns false for a given element, the enumeration stops. 48 | 49 | ## Examples 50 | 51 | stream = take_while([1, 2, 3, 4, 5], fn elem -> elem < 4 end) 52 | Enum.to_list(stream) 53 | #=> [1, 2, 3] 54 | 55 | """ 56 | @spec take_while(Enumerable.t(), (term() -> boolean())) :: Enumerable.t() 57 | def take_while(enum, predicate) do 58 | Stream.transform(enum, nil, fn elem, nil -> 59 | if predicate.(elem) do 60 | {[elem], nil} 61 | else 62 | {:halt, nil} 63 | end 64 | end) 65 | end 66 | 67 | @doc """ 68 | Takes an enumerable and returns that enumerable but without consecutive duplicate elements. 69 | 70 | ## Examples 71 | 72 | stream = dedup([1, 2, 1, 1, 5, 8, 8, 8, 1]) 73 | Enum.to_list(stream) 74 | #=> [1, 2, 1, 5, 8, 1] 75 | 76 | """ 77 | @spec dedup(Enumerable.t()) :: Enumerable.t() 78 | def dedup(enum) do 79 | Stream.transform(enum, :none, fn 80 | elem, :none -> {[elem], {:previous, elem}} 81 | elem, {:previous, elem} -> {[], {:previous, elem}} 82 | elem, _ -> {[elem], {:previous, elem}} 83 | end) 84 | end 85 | 86 | @doc """ 87 | Takes an enumerable and returns that enumerable but without duplicate elements. 88 | 89 | ## Examples 90 | 91 | stream = dedup([1, 2, 1, 1, 5, 8, 8, 8, 1]) 92 | Enum.to_list(stream) 93 | #=> [1, 2, 5, 8] 94 | 95 | """ 96 | @spec uniq(Enumerable.t()) :: Enumerable.t() 97 | def uniq(enum) do 98 | Stream.transform(enum, MapSet.new(), fn elem, seen -> 99 | if elem in seen do 100 | {[], seen} 101 | else 102 | {[elem], MapSet.put(seen, elem)} 103 | end 104 | end) 105 | end 106 | end 107 | -------------------------------------------------------------------------------- /lab2/solutions_tasks.ex: -------------------------------------------------------------------------------- 1 | defmodule Lab2.Tasks do 2 | @doc """ 3 | Maps the given `fun` in parallel over all elements of `enum`. 4 | 5 | The end result of this function should be the same as `Enum.map/2`, except that 6 | each element is processed in parallel with all the others. 7 | 8 | ## Examples 9 | 10 | pmap([url1, url2, ...], fn url -> expensive_request(url) end) 11 | #=> [response1, response2, ...] 12 | 13 | """ 14 | @spec pmap(Enumerable.t(), (a -> b)) :: Enumerable.t() when a: term(), b: term() 15 | def pmap(enum, fun) when is_function(fun, 1) do 16 | enum 17 | |> Enum.map(&Task.async(fn -> fun.(&1) end)) 18 | |> Enum.map(&Task.await/1) 19 | end 20 | 21 | @doc """ 22 | Takes a stream of enumerables and returns a stream of the sum of each enumerable. 23 | 24 | ## Examples 25 | 26 | iex> Enum.to_list(sum_all([1..2, 3..4, 5..6])) 27 | [3, 7, 11] 28 | 29 | """ 30 | @spec sum_all(Enumerable.t()) :: Enumerable.t() 31 | def sum_all(stream_of_enums) do 32 | stream_of_enums 33 | |> Task.async_stream(&Enum.sum/1) 34 | |> Stream.map(fn {:ok, i} -> i end) 35 | end 36 | 37 | @doc """ 38 | Spawns a process that executes the given computation (given as a function) and that can be 39 | awaited on. 40 | 41 | Suggestion: there's a function in the `Kernel` module that can help avoiding a race condition 42 | when spawning *and then* monitoring. 43 | 44 | ## Examples 45 | 46 | See the `await/1` function. 47 | """ 48 | @spec async((() -> term())) :: term() 49 | def async(fun) when is_function(fun, 0) do 50 | parent = self() 51 | ref = make_ref() 52 | 53 | {_pid, monitor_ref} = 54 | spawn_monitor(fn -> 55 | result = fun.() 56 | send(parent, {ref, result}) 57 | end) 58 | 59 | {ref, monitor_ref} 60 | end 61 | 62 | @doc """ 63 | Awaits on the result of a computation started with `async/1` and returns `{:ok, result}` if 64 | the result was computed or `:error` if the process crashed. 65 | 66 | Note that if you have 67 | 68 | ref = Process.monitor(pid) 69 | 70 | then if `pid` dies for some reason a message will be delivered to the monitoring process. The 71 | message would have the following shape: 72 | 73 | {:DOWN, ref, _, _, _} 74 | 75 | ## Examples 76 | 77 | iex> async = Monitor.async(fn -> 1 + 10 end) 78 | iex> Monitor.await(async) 79 | {:ok, 11} 80 | 81 | """ 82 | @spec await(term()) :: term() 83 | def await({ref, monitor_ref} = _task) do 84 | receive do 85 | {^ref, result} -> 86 | Process.demonitor(monitor_ref, [:flush]) 87 | {:ok, result} 88 | 89 | {:DOWN, ^monitor_ref, _, _, _} -> 90 | :error 91 | end 92 | end 93 | end 94 | -------------------------------------------------------------------------------- /lab2/test/lab2_test.exs: -------------------------------------------------------------------------------- 1 | defmodule Lab2.StreamsTest do 2 | use ExUnit.Case 3 | 4 | @pride_and_prejudice Path.join(__DIR__, "../../pride-and-prejudice.txt") 5 | 6 | @tag :skip 7 | test "counting words" do 8 | count = Lab2.Streams.count_words_until_stop_word(@pride_and_prejudice, "amusement") 9 | 10 | assert count["must"] == 7 11 | assert count["twenty"] == 3 12 | assert count["end"] == 1 13 | 14 | refute Map.has_key?(count, "amusement") 15 | end 16 | 17 | @tag :skip 18 | test "take_while/2" do 19 | stream = Lab2.Streams.take_while(1..1000, &(&1 <= 5)) 20 | assert Enum.to_list(stream) == [1, 2, 3, 4, 5] 21 | 22 | assert Enum.to_list(Lab2.Streams.take_while(1..1000, &(&1 <= 0))) == [] 23 | assert Enum.to_list(Lab2.Streams.take_while(1..3, &(&1 <= 5))) == [1, 2, 3] 24 | 25 | nats = Stream.iterate(1, &(&1 + 1)) 26 | assert Enum.to_list(Lab2.Streams.take_while(nats, &(&1 <= 5))) == [1, 2, 3, 4, 5] 27 | 28 | stream = Stream.drop(1..100, 5) 29 | assert Lab2.Streams.take_while(stream, &(&1 < 11)) |> Enum.to_list() == [6, 7, 8, 9, 10] 30 | end 31 | 32 | @tag :skip 33 | test "uniq/1" do 34 | assert Lab2.Streams.uniq([1, 2, 3, 2, 1]) |> Enum.to_list() == [1, 2, 3] 35 | end 36 | 37 | @tag :skip 38 | test "dedup/1" do 39 | assert Lab2.Streams.dedup([1, 1, 2, 1, 1, 2, 1]) |> Enum.to_list() == [1, 2, 1, 2, 1] 40 | assert Lab2.Streams.dedup([2, 1, 1, 2, 1]) |> Enum.to_list() == [2, 1, 2, 1] 41 | assert Lab2.Streams.dedup([1, 2, 3, 4]) |> Enum.to_list() == [1, 2, 3, 4] 42 | assert Lab2.Streams.dedup([1, 1.0, 2.0, 2]) |> Enum.to_list() == [1, 1.0, 2.0, 2] 43 | assert Lab2.Streams.dedup([]) |> Enum.to_list() == [] 44 | 45 | assert Lab2.Streams.dedup([nil, nil, true, {:value, true}]) |> Enum.to_list() == 46 | [nil, true, {:value, true}] 47 | 48 | assert Lab2.Streams.dedup([nil]) |> Enum.to_list() == [nil] 49 | end 50 | end 51 | 52 | defmodule Lab2.TasksTest do 53 | use ExUnit.Case 54 | 55 | @tag :skip 56 | test "pmap/2" do 57 | {time_in_microsec, result} = 58 | :timer.tc(fn -> 59 | Lab2.Tasks.pmap([1000, 1000, 1000], fn ms -> 60 | Process.sleep(ms) 61 | Integer.to_string(ms) 62 | end) 63 | end) 64 | 65 | assert div(time_in_microsec, 1000) in 1000..2000 66 | assert result == ["1000", "1000", "1000"] 67 | end 68 | 69 | describe "async/await" do 70 | @tag :skip 71 | test "when everything goes fine" do 72 | async = Lab2.Tasks.async(fn -> 1 + 10 end) 73 | assert Lab2.Tasks.await(async) == {:ok, 11} 74 | end 75 | 76 | @tag :skip 77 | @tag :capture_log 78 | test "when the process crashes" do 79 | async = Lab2.Tasks.async(fn -> raise "nope!" end) 80 | assert Lab2.Tasks.await(async) == :error 81 | end 82 | 83 | @tag :skip 84 | test "no messages are leaked" do 85 | async = Lab2.Tasks.async(fn -> 1 + 10 end) 86 | assert Lab2.Tasks.await(async) == {:ok, 11} 87 | refute_receive _ 88 | end 89 | end 90 | 91 | @tag :skip 92 | test "sum_all/1" do 93 | refute is_list(Lab2.Tasks.sum_all([1..2])) 94 | 95 | assert Enum.to_list(Lab2.Tasks.sum_all([1..2])) == [3] 96 | assert Enum.to_list(Lab2.Tasks.sum_all([1..2, 3..4, 5..6])) == [3, 7, 11] 97 | 98 | assert Enum.to_list(Lab2.Tasks.sum_all([[1, 2]])) == [3] 99 | assert Enum.to_list(Lab2.Tasks.sum_all([[1, 2], [3, 4], [5, 6]])) == [3, 7, 11] 100 | end 101 | end 102 | -------------------------------------------------------------------------------- /lab2/test/test_helper.exs: -------------------------------------------------------------------------------- 1 | ExUnit.start(exclude: [:skip]) 2 | -------------------------------------------------------------------------------- /lab3/.formatter.exs: -------------------------------------------------------------------------------- 1 | # Used by "mix format" 2 | [ 3 | inputs: ["{mix,.formatter}.exs", "{config,lib,test}/**/*.{ex,exs}"] 4 | ] 5 | -------------------------------------------------------------------------------- /lab3/.gitignore: -------------------------------------------------------------------------------- 1 | # The directory Mix will write compiled artifacts to. 2 | /_build/ 3 | 4 | # If you run "mix test --cover", coverage assets end up here. 5 | /cover/ 6 | 7 | # The directory Mix downloads your dependencies sources to. 8 | /deps/ 9 | 10 | # Where 3rd-party dependencies like ExDoc output generated docs. 11 | /doc/ 12 | 13 | # Ignore .fetch files in case you like to edit your project deps locally. 14 | /.fetch 15 | 16 | # If the VM crashes, it generates a dump, let's ignore it too. 17 | erl_crash.dump 18 | 19 | # Also ignore archive artifacts (built via "mix archive.build"). 20 | *.ez 21 | 22 | # Ignore package tarball (built via "mix hex.build"). 23 | lab3-*.tar 24 | 25 | -------------------------------------------------------------------------------- /lab3/README.md: -------------------------------------------------------------------------------- 1 | # Lab3 2 | 3 | ### Lab purpose 4 | 5 | Learn about GenStage. We will work on exercises that will showcase some of GenStage's 6 | "niche" features (like custom dispatchers). 7 | 8 | ### Lab instructions 9 | 10 | 1. Implement `Lab3.InvoiceProducer`'s `init/1` callback so that this producer uses the 11 | `GenStage.PartitionDispatcher` dispatcher in order to dispatch events (that is, invoices) to 12 | the right consumer based on each invoice's region. 13 | 14 | ### Links 15 | 16 | API docs: https://hexdocs.pm/elixir/ 17 | 18 | 1. `GenStage`: https://hexdocs.pm/gen_stage 19 | 2. `GenStage.PartitionDispatcher`: https://hexdocs.pm/gen_stage/GenStage.PartitionDispatcher.html 20 | 21 | 22 | ### Solution ( no peeking :) ) 23 | 24 | See the `solution_*.ex` files in the `lab3` directory. 25 | -------------------------------------------------------------------------------- /lab3/lib/lab3/partitions.ex: -------------------------------------------------------------------------------- 1 | defmodule Lab3.InvoiceProducer do 2 | use GenStage 3 | 4 | def start_link(regions) do 5 | GenStage.start_link(__MODULE__, regions, name: __MODULE__) 6 | end 7 | 8 | def init(regions) do 9 | raise "not implemented yet" 10 | end 11 | 12 | def handle_demand(demand, regions) do 13 | events = Enum.take(invoice_generator(regions), demand) 14 | {:noreply, events, regions} 15 | end 16 | 17 | defp invoice_generator(regions) do 18 | import StreamData 19 | 20 | fixed_map(%{ 21 | region: member_of(regions), 22 | amount: positive_integer(), 23 | name: string(:alphanumeric) 24 | }) 25 | end 26 | end 27 | -------------------------------------------------------------------------------- /lab3/lib/lab3/rate_limiting.ex: -------------------------------------------------------------------------------- 1 | defmodule Lab3.RateLimitedConsumer do 2 | use GenStage 3 | 4 | @interval 1000 5 | 6 | def init(_) do 7 | {:consumer, _producers = %{}} 8 | end 9 | 10 | def handle_subscribe(:producer, opts, from, producers) do 11 | # We will only allow max_demand events every 5000 milliseconds 12 | pending = opts[:max_demand] || 1000 13 | 14 | # Register the producer in the state 15 | producers = Map.put(producers, from, pending) 16 | # Ask for the pending events and schedule the next time around 17 | producers = ask_and_schedule(producers, from) 18 | 19 | # Returns manual as we want control over the demand 20 | {:manual, producers} 21 | end 22 | 23 | def handle_cancel(_, from, producers) do 24 | # Remove the producers from the map on unsubscribe 25 | {:noreply, [], Map.delete(producers, from)} 26 | end 27 | 28 | def handle_events(events, from, producers) do 29 | # TODO: Bump the amount of pending events for the given producer. 30 | # ... 31 | 32 | # Consume the events. 33 | Enum.each(events, &process_event/1) 34 | 35 | # A producer_consumer would return the processed events here. 36 | {:noreply, [], producers} 37 | end 38 | 39 | def handle_info({:ask, from}, producers) do 40 | # This callback is invoked by the Process.send_after/3 message below. 41 | {:noreply, [], ask_and_schedule(producers, from)} 42 | end 43 | 44 | # Should ask the producer ("from") for mode demand with GenStage.ask/2, and schedule the next 45 | # time it should ask. 46 | defp ask_and_schedule(producers, from) do 47 | raise "not implemented yet" 48 | end 49 | 50 | defp process_event(event) do 51 | IO.inspect(event) 52 | end 53 | end 54 | -------------------------------------------------------------------------------- /lab3/mix.exs: -------------------------------------------------------------------------------- 1 | defmodule Lab3.MixProject do 2 | use Mix.Project 3 | 4 | def project do 5 | [ 6 | app: :lab3, 7 | version: "0.1.0", 8 | elixir: "~> 1.6", 9 | start_permanent: Mix.env() == :prod, 10 | deps: deps() 11 | ] 12 | end 13 | 14 | # Run "mix help compile.app" to learn about applications. 15 | def application do 16 | [ 17 | extra_applications: [:logger] 18 | ] 19 | end 20 | 21 | # Run "mix help deps" to learn about dependencies. 22 | defp deps do 23 | [ 24 | {:gen_stage, "~> 0.13.1"}, 25 | {:stream_data, "~> 0.4.2"} 26 | ] 27 | end 28 | end 29 | -------------------------------------------------------------------------------- /lab3/mix.lock: -------------------------------------------------------------------------------- 1 | %{ 2 | "gen_stage": {:hex, :gen_stage, "0.13.1", "edff5bca9cab22c5d03a834062515e6a1aeeb7665fb44eddae086252e39c4378", [:mix], [], "hexpm"}, 3 | "stream_data": {:hex, :stream_data, "0.4.2", "fa86b78c88ec4eaa482c0891350fcc23f19a79059a687760ddcf8680aac2799b", [:mix], [], "hexpm"}, 4 | } 5 | -------------------------------------------------------------------------------- /lab3/solution_partitions.ex: -------------------------------------------------------------------------------- 1 | defmodule Lab3.InvoiceProducer do 2 | use GenStage 3 | 4 | def start_link(regions) do 5 | GenStage.start_link(__MODULE__, regions, name: __MODULE__) 6 | end 7 | 8 | def init(regions) do 9 | dispatcher = {GenStage.PartitionDispatcher, partitions: regions, hash: &partition_hash/1} 10 | {:producer, regions, dispatcher: dispatcher} 11 | end 12 | 13 | defp partition_hash(event) do 14 | {event, event.region} 15 | end 16 | 17 | def handle_demand(demand, regions) do 18 | events = Enum.take(invoice_generator(regions), demand) 19 | {:noreply, events, regions} 20 | end 21 | 22 | defp invoice_generator(regions) do 23 | import StreamData 24 | 25 | fixed_map(%{ 26 | region: member_of(regions), 27 | amount: positive_integer(), 28 | name: string(:alphanumeric) 29 | }) 30 | end 31 | end 32 | -------------------------------------------------------------------------------- /lab3/solutions_rate_limited_consumer.ex: -------------------------------------------------------------------------------- 1 | defmodule Lab3.RateLimitedConsumer do 2 | use GenStage 3 | 4 | @interval 1000 5 | 6 | def init(_) do 7 | {:consumer, _producers = %{}} 8 | end 9 | 10 | def handle_subscribe(:producer, opts, from, producers) do 11 | # We will only allow max_demand events every 5000 milliseconds 12 | pending = opts[:max_demand] || 1000 13 | 14 | # Register the producer in the state 15 | producers = Map.put(producers, from, pending) 16 | # Ask for the pending events and schedule the next time around 17 | producers = ask_and_schedule(producers, from) 18 | 19 | # Returns manual as we want control over the demand 20 | {:manual, producers} 21 | end 22 | 23 | def handle_cancel(_, from, producers) do 24 | # Remove the producers from the map on unsubscribe 25 | {:noreply, [], Map.delete(producers, from)} 26 | end 27 | 28 | def handle_events(events, from, producers) do 29 | # Bump the amount of pending events for the given producer 30 | producers = Map.update!(producers, from, fn pending -> pending + length(events) end) 31 | 32 | # Consume the events. 33 | Enum.each(events, &process_event/1) 34 | 35 | # A producer_consumer would return the processed events here. 36 | {:noreply, [], producers} 37 | end 38 | 39 | def handle_info({:ask, from}, producers) do 40 | # This callback is invoked by the Process.send_after/3 message below. 41 | {:noreply, [], ask_and_schedule(producers, from)} 42 | end 43 | 44 | # Should ask the producer ("from") for mode demand with GenStage.ask/2, and schedule the next 45 | # time it should ask. 46 | defp ask_and_schedule(producers, from) do 47 | case producers do 48 | %{^from => pending} -> 49 | GenStage.ask(from, pending) 50 | Process.send_after(self(), {:ask, from}, @interval) 51 | Map.put(producers, from, 0) 52 | %{} -> 53 | producers 54 | end 55 | end 56 | 57 | defp process_event(event) do 58 | IO.inspect(event) 59 | end 60 | end 61 | -------------------------------------------------------------------------------- /lab3/test/lab3_test.exs: -------------------------------------------------------------------------------- 1 | defmodule Lab3.PartitionDispatcherTest do 2 | use ExUnit.Case 3 | 4 | defmodule InvoiceConsumer do 5 | use GenStage 6 | 7 | import ExUnit.Assertions 8 | 9 | def start_link(region) do 10 | GenStage.start_link(__MODULE__, region) 11 | end 12 | 13 | def init(region) do 14 | producer = {Lab3.InvoiceProducer, partition: region, max_demand: 10, min_demand: 5} 15 | {:consumer, region, subscribe_to: [producer]} 16 | end 17 | 18 | def handle_events(events, _from, region) do 19 | Enum.each(events, fn event -> 20 | assert event.region == region 21 | end) 22 | 23 | {:noreply, [], region} 24 | end 25 | end 26 | 27 | @tag :skip 28 | test "partition dispatcher" do 29 | regions = [:us, :eu, :asia] 30 | 31 | assert {:ok, _} = Lab3.InvoiceProducer.start_link(regions) 32 | 33 | for region <- regions do 34 | assert {:ok, _} = InvoiceConsumer.start_link(region) 35 | end 36 | 37 | Process.sleep(2000) 38 | end 39 | end 40 | 41 | defmodule Lab3.RateLimitedConsumerTest do 42 | use ExUnit.Case 43 | 44 | defmodule Producer do 45 | use GenStage 46 | 47 | @interval 1000 48 | 49 | def init(_) do 50 | {:producer, 0} 51 | end 52 | 53 | def handle_demand(demand, last_asked) do 54 | if now() - last_asked < (@interval / 2) do 55 | raise "failed" 56 | else 57 | {:noreply, Enum.take(Stream.repeatedly(&:rand.uniform/0), demand), now()} 58 | end 59 | end 60 | 61 | defp now() do 62 | System.system_time(:milliseconds) 63 | end 64 | end 65 | 66 | @tag :skip 67 | test "rate limiting" do 68 | {:ok, producer} = GenStage.start_link(Producer, nil) 69 | {:ok, consumer} = GenStage.start_link(Lab3.RateLimitedConsumer, nil) 70 | 71 | GenStage.sync_subscribe(consumer, to: producer, max_demand: 10) 72 | 73 | Process.sleep(2000) 74 | end 75 | end 76 | -------------------------------------------------------------------------------- /lab3/test/test_helper.exs: -------------------------------------------------------------------------------- 1 | ExUnit.start(exclude: [:skip]) 2 | -------------------------------------------------------------------------------- /lab4/.formatter.exs: -------------------------------------------------------------------------------- 1 | # Used by "mix format" 2 | [ 3 | inputs: ["{mix,.formatter}.exs", "{config,lib,test}/**/*.{ex,exs}"] 4 | ] 5 | -------------------------------------------------------------------------------- /lab4/.gitignore: -------------------------------------------------------------------------------- 1 | # The directory Mix will write compiled artifacts to. 2 | /_build/ 3 | 4 | # If you run "mix test --cover", coverage assets end up here. 5 | /cover/ 6 | 7 | # The directory Mix downloads your dependencies sources to. 8 | /deps/ 9 | 10 | # Where 3rd-party dependencies like ExDoc output generated docs. 11 | /doc/ 12 | 13 | # Ignore .fetch files in case you like to edit your project deps locally. 14 | /.fetch 15 | 16 | # If the VM crashes, it generates a dump, let's ignore it too. 17 | erl_crash.dump 18 | 19 | # Also ignore archive artifacts (built via "mix archive.build"). 20 | *.ez 21 | 22 | # Ignore package tarball (built via "mix hex.build"). 23 | lab4-*.tar 24 | 25 | -------------------------------------------------------------------------------- /lab4/README.md: -------------------------------------------------------------------------------- 1 | # Lab4 2 | 3 | ### Lab purpose 4 | 5 | Learn about Flow. We will extend on streams and GenStage to express computation flows on large 6 | collections of data. 7 | 8 | ### Lab instructions 9 | 10 | The `lab4` directory has a file `lib/lab4.ex` which contains an implementation of chat 11 | room with some stubbed out functions. As usual, use that template to implement functions 12 | until all tests pass. 13 | 14 | 1. Use Flow to a sum an enumerable of integers. Tips: Use `Enum.chunk_every/2` to split the enum 15 | and `Flow.from_enumerables/1` to build a Flow producer from the list. 16 | 17 | 2. Return N most frequent words. Tips: Use the provided `normalize_string/1` function and pass 18 | the `trim: true` option to `String.split/2`. 19 | 20 | ### Links 21 | 22 | API docs: https://hexdocs.pm/elixir/ 23 | 24 | 1. `Flow/1` https://hexdocs.pm/flow 25 | 26 | 2. `Flow.from_enumerables/1` - https://hexdocs.pm/flow/Flow.html#from_enumerables/2 27 | 28 | 3. `Flow.reduce/3` - https://hexdocs.pm/flow/Flow.html#reduce/3 29 | 30 | 4. `Flow.emit/2` - https://hexdocs.pm/flow/Flow.html#emit/2 31 | 32 | 33 | ### Solution ( no peeking :) ) 34 | 35 | See `solution.ex` in the `lab4` directory. 36 | -------------------------------------------------------------------------------- /lab4/config/config.exs: -------------------------------------------------------------------------------- 1 | # This file is responsible for configuring your application 2 | # and its dependencies with the aid of the Mix.Config module. 3 | use Mix.Config 4 | 5 | # This configuration is loaded before any dependency and is restricted 6 | # to this project. If another project depends on this project, this 7 | # file won't be loaded nor affect the parent project. For this reason, 8 | # if you want to provide default values for your application for 9 | # 3rd-party users, it should be done in your "mix.exs" file. 10 | 11 | # You can configure your application as: 12 | # 13 | # config :lab4, key: :value 14 | # 15 | # and access this configuration in your application as: 16 | # 17 | # Application.get_env(:lab4, :key) 18 | # 19 | # You can also configure a 3rd-party app: 20 | # 21 | # config :logger, level: :info 22 | # 23 | 24 | # It is also possible to import configuration files, relative to this 25 | # directory. For example, you can emulate configuration per environment 26 | # by uncommenting the line below and defining dev.exs, test.exs and such. 27 | # Configuration from the imported file will override the ones defined 28 | # here (which is why it is important to import them last). 29 | # 30 | # import_config "#{Mix.env()}.exs" 31 | -------------------------------------------------------------------------------- /lab4/lib/lab4.ex: -------------------------------------------------------------------------------- 1 | defmodule Lab4 do 2 | def sum_enumerable(_enum) do 3 | raise "not implenented yet" 4 | end 5 | 6 | def most_frequent_words(_file_path, _num) do 7 | raise "not implenented yet" 8 | end 9 | 10 | defp normalize_string(string) do 11 | string 12 | |> String.downcase() 13 | |> String.replace(~w(, ? . ! ; _ “ ”), "") 14 | end 15 | end 16 | -------------------------------------------------------------------------------- /lab4/mix.exs: -------------------------------------------------------------------------------- 1 | defmodule Lab4.MixProject do 2 | use Mix.Project 3 | 4 | def project do 5 | [ 6 | app: :lab4, 7 | version: "0.1.0", 8 | elixir: "~> 1.6", 9 | start_permanent: Mix.env() == :prod, 10 | deps: deps() 11 | ] 12 | end 13 | 14 | # Run "mix help compile.app" to learn about applications. 15 | def application do 16 | [ 17 | extra_applications: [:logger] 18 | ] 19 | end 20 | 21 | # Run "mix help deps" to learn about dependencies. 22 | defp deps do 23 | [ 24 | {:flow, "~> 0.13.0"} 25 | ] 26 | end 27 | end 28 | -------------------------------------------------------------------------------- /lab4/mix.lock: -------------------------------------------------------------------------------- 1 | %{ 2 | "flow": {:hex, :flow, "0.13.0", "7188ef047c6831ffb2f25ec0cef9bef02e7ae520632a64debdb65ae96bdc223e", [:mix], [{:gen_stage, "~> 0.13.0", [hex: :gen_stage, repo: "hexpm", optional: false]}], "hexpm"}, 3 | "gen_stage": {:hex, :gen_stage, "0.13.1", "edff5bca9cab22c5d03a834062515e6a1aeeb7665fb44eddae086252e39c4378", [:mix], [], "hexpm"}, 4 | } 5 | -------------------------------------------------------------------------------- /lab4/solution.ex: -------------------------------------------------------------------------------- 1 | defmodule Lab4 do 2 | def sum_enumerable(enum) do 3 | enum 4 | |> Enum.chunk_every(1000) 5 | |> Flow.from_enumerables() 6 | |> Flow.reduce(fn -> 0 end, &+/2) 7 | |> Flow.emit(:state) 8 | |> Enum.sum() 9 | end 10 | 11 | def most_frequent_words(file_path, num) do 12 | file_path 13 | |> File.stream!() 14 | |> Flow.from_enumerable() 15 | |> Flow.map(&String.trim/1) 16 | |> Flow.flat_map(&String.split(&1, " ", trim: true)) 17 | |> Flow.map(&normalize_string/1) 18 | |> Flow.partition() 19 | |> Flow.reduce(fn -> %{} end, fn word, acc -> 20 | Map.update(acc, word, 1, & &1 + 1) 21 | end) 22 | |> Flow.map(fn {word, count} -> {count, word} end) 23 | |> Enum.sort(&>=/2) 24 | |> Enum.take(num) 25 | |> Enum.map(fn {_count, word} -> word end) 26 | end 27 | 28 | defp normalize_string(string) do 29 | string 30 | |> String.downcase() 31 | |> String.replace(~w(, ? . ! ; _ “ ”), "") 32 | end 33 | end 34 | -------------------------------------------------------------------------------- /lab4/test/lab4_test.exs: -------------------------------------------------------------------------------- 1 | defmodule Lab4Test do 2 | use ExUnit.Case 3 | 4 | @pride_and_prejudice Path.join(__DIR__, "../../pride-and-prejudice.txt") 5 | 6 | @tag :skip 7 | test "task 1: sum enumerable" do 8 | assert Lab4.sum_enumerable(1..10) == Enum.sum(1..10) 9 | assert Lab4.sum_enumerable(1..1000) == Enum.sum(1..1000) 10 | assert Lab4.sum_enumerable(1..100000) == Enum.sum(1..100000) 11 | end 12 | 13 | @tag :skip 14 | test "task 2: most frequent words" do 15 | words = Lab4.most_frequent_words(@pride_and_prejudice, 3) 16 | assert words == ["the", "to", "of"] 17 | 18 | words = Lab4.most_frequent_words(@pride_and_prejudice, 10) 19 | assert words == ["the", "to", "of", "and", "her", "i", "a", "in", "was", "she"] 20 | end 21 | end 22 | -------------------------------------------------------------------------------- /lab4/test/test_helper.exs: -------------------------------------------------------------------------------- 1 | ExUnit.start() 2 | --------------------------------------------------------------------------------