├── .github ├── ISSUE_TEMPLATE │ └── please--open-new-issues-in-membranefranework-membrane_core.md └── workflows │ ├── on_issue_opened.yaml │ └── on_pr_opened.yaml ├── .gitignore ├── README.md ├── basic_pipeline ├── 01_Introduction.md ├── 02_SystemArchitecture.md ├── 03_Source.md ├── 04_StreamFormat.md ├── 05_Formats.md ├── 06_OrderingBuffer.md ├── 07_Redemands.md ├── 08_Depayloader.md ├── 09_Mixer.md ├── 10_Sink.md ├── 11_Pipeline.md ├── assets │ ├── diagrams │ │ ├── basic_pipeline.drawio │ │ ├── example_chat.drawio │ │ ├── example_input.drawio │ │ └── ordering_buffer.drawio │ └── images │ │ ├── Illo_basic pipeline.png │ │ ├── basic_pipeline.png │ │ ├── example_chat.drawio.png │ │ ├── example_input.drawio.png │ │ └── ordering_buffer.drawio.png └── index.md ├── basic_pipeline_extension ├── 01_Introduction.md ├── 02_Bin.md ├── 03_DynamicPads.md ├── 04_Tests.md ├── assets │ ├── diagrams │ │ ├── basic_pipeline.drawio │ │ └── basic_pipeline_bin.drawio │ └── images │ │ ├── Illo_basic pipeline extension.png │ │ ├── basic_pipeline.png │ │ └── basic_pipeline_bin.png └── index.md ├── broadcasting ├── 01_General_Introduction.md ├── 02_RTMP_Introduction.md ├── 03_RTMP_SystemArchitecture.md ├── 04_RTMP_RunningTheDemo.md ├── 05_RTMP_Pipeline.md ├── 06_WebPlayer.md ├── 07_RTSP_Introduction.md ├── 08_RTSP_Architecture.md ├── 09_RTSP_RunningDemo.md ├── 10_ConnectionManager.md ├── 11_RTSP_Pipeline.md ├── 12_Summary.md ├── 13_H264_codec.md ├── assets │ ├── OBS_settings.webp │ ├── RTMP_to_HLS_pipeline.drawio.png │ ├── au_structure.png │ ├── connection_manager.drawio.png │ ├── h264_structure.png │ ├── output_files_structure.png │ ├── rtsp_architecture.drawio.png │ └── rtsp_pipeline.drawio.png └── index.md ├── create_new_plugin ├── assets │ └── images │ │ ├── Illo_create new plugin.png │ │ └── tutorial_graphic.svg ├── create_new_plugin.md └── index.md ├── digital_video_introduction ├── 1_preface.md ├── assets │ └── images │ │ ├── Illo_ digital video introduction.png │ │ └── tutorial_graphic.svg └── index.md ├── get_started_with_membrane ├── 01_introduction.md ├── 02_pipelines.md ├── 03_elements.md ├── 04_bins.md ├── 05_pads_and_linking.md ├── 06_flow_control.md ├── 07_observability_and_logging.md ├── 08_native_code_integration.md ├── assets │ └── images │ │ ├── Illo_get_started.png │ │ ├── kino_membrane.png │ │ └── tutorial_graphic.svg └── index.md ├── glossary ├── assets │ └── images │ │ ├── Illo_glossary.png │ │ └── tutorial_graphic.svg ├── glossary.md └── index.md └── h264 ├── 1_Introduction.md ├── 2_What_is_H264.md ├── 3_H264_NAL_units.md ├── 4_H264_stream_structure.md ├── 5_Fetching_information_from_stream.md ├── 6_Appendix.md ├── assets └── images │ ├── AU_scheme.png │ ├── H264_example_stream.png │ └── VCL_vs_NAL.png └── index.md /.github/ISSUE_TEMPLATE/please--open-new-issues-in-membranefranework-membrane_core.md: -------------------------------------------------------------------------------- 1 | --- 2 | name: Please, open new issues in membranefranework/membrane_core 3 | about: New issues related to this repo should be opened there 4 | title: "[DO NOT OPEN]" 5 | labels: '' 6 | assignees: '' 7 | 8 | --- 9 | 10 | Please, do not open this issue here. Open it in the [membrane_core](https://github.com/membraneframework/membrane_core) repository instead. 11 | 12 | Thanks for helping us grow :) 13 | -------------------------------------------------------------------------------- /.github/workflows/on_issue_opened.yaml: -------------------------------------------------------------------------------- 1 | name: 'Close issue when opened' 2 | on: 3 | issues: 4 | types: 5 | - opened 6 | jobs: 7 | close: 8 | runs-on: ubuntu-latest 9 | steps: 10 | - name: Checkout membrane_core 11 | uses: actions/checkout@v3 12 | with: 13 | repository: membraneframework/membrane_core 14 | - name: Close issue 15 | uses: ./.github/actions/close_issue 16 | with: 17 | GITHUB_TOKEN: ${{ secrets.MEMBRANEFRAMEWORKADMIN_TOKEN }} 18 | ISSUE_URL: ${{ github.event.issue.html_url }} 19 | ISSUE_NUMBER: ${{ github.event.issue.number }} 20 | REPOSITORY: ${{ github.repository }} 21 | -------------------------------------------------------------------------------- /.github/workflows/on_pr_opened.yaml: -------------------------------------------------------------------------------- 1 | name: Add PR to Smackore project board, if the author is from outside Membrane Team 2 | on: 3 | pull_request_target: 4 | types: 5 | - opened 6 | jobs: 7 | maybe_add_to_project_board: 8 | runs-on: ubuntu-latest 9 | steps: 10 | - name: Checkout membrane_core 11 | uses: actions/checkout@v3 12 | with: 13 | repository: membraneframework/membrane_core 14 | - name: Puts PR in "New PRs by community" column in the Smackore project, if the author is from outside Membrane Team 15 | uses: ./.github/actions/add_pr_to_smackore_board 16 | with: 17 | GITHUB_TOKEN: ${{ secrets.MEMBRANEFRAMEWORKADMIN_TOKEN }} 18 | AUTHOR_LOGIN: ${{ github.event.pull_request.user.login }} 19 | PR_URL: ${{ github.event.pull_request.html_url }} 20 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | .DS_Store 2 | videoroom/.vscode/ 3 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Membrane Tutorials 2 | 3 | Repository which contains text and assets used in Membrane Framework tutorials. 4 | 5 | The tutorials are available at: https://membrane.stream/learn 6 | 7 | ## Copyright and License 8 | 9 | Copyright 2020, [Software Mansion](https://swmansion.com/?utm_source=git&utm_medium=readme&utm_campaign=membrane_template_plugin) 10 | 11 | [![Software Mansion](https://logo.swmansion.com/logo?color=white&variant=desktop&width=200&tag=membrane-github)](https://swmansion.com/?utm_source=git&utm_medium=readme&utm_campaign=membrane_template_plugin) -------------------------------------------------------------------------------- /basic_pipeline/01_Introduction.md: -------------------------------------------------------------------------------- 1 | In this tutorial, we will create our very own Membrane Framework consisting of our custom which will fulfill the multimedia processing task. 2 | Despite the fact that the multimedia processing task we will be facing will be really simple, we will need to deal with some problems occurring in real-life scenarios. 3 | At the same time, we will make ourselves comfortable with some concepts of multimedia streaming as well as get in touch with the nomenclature used in this field. 4 | Prepare for an adventure! 5 | 6 | ## Environment preparation 7 | 8 | In order to be able to proceed with the tutorial, you need to have Elixir installed: [How to install Elixir](https://elixir-lang.org/install.html). 9 | We assume, that you are at least slightly familiar with that language - if that is not true, we would like to strongly encourage you to take a look at the [official Elixir tutorial](https://elixir-lang.org/getting-started/introduction.html). 10 | Once you are ready with the Elixir, you can get the project template we have prepared for you: 11 | 12 | ```console 13 | git clone https://github.com/membraneframework/membrane_basic_pipeline_tutorial 14 | cd membrane_basic_pipeline_tutorial 15 | git checkout template/start 16 | mix deps.get 17 | ``` 18 | 19 | As you can see, the template's code is put on the `template/start` branch of the repository. 20 | In the repository, there is also a `template/end` branch, where you can find the completed template. 21 | If you find yourself lost during the tutorial feel free to check the implementation proposed by us, put on this branch. 22 | 23 | ## Use Case 24 | 25 | Imagine that there is a conversation occurring between two peers and the chat is based on a question-answer scheme. 26 | Each of the peers sends its part of the conversation (that means - the first peer sends questions and the second peers sends the answers to them). We refer to each sentence sent by the peers as a [**frame**](../glossary/glossary.md#frame). 27 | However, due to network limitations each frame sent by the peers is fragmented since only a few chars can be sent at the same time - that means, a single frame is sent as a sequence of [**packets**](../glossary/glossary.md#packet). 28 | In a real network there is a parameter called MTU (*Maximum Transmission Unit*) which is the maximum number of bytes which can be send in a single packet. For the Ethernet II MTU is equal to 1500 bytes. We wanted to simulate the situation, 29 | in which the data needs to be fragmented in order to be transmitted - but we decided to choose a small MTU so that this fragmentation is easy to be spotted. 30 | Each packet consists of the *header* (describing where the particular bunch of characters it transports should be put) and the *body* - aforementioned bunch of characters. 31 | Below you can see an exemplary frame sent by one peer to the other. It gets fragmented into multiple packets, which later on are sent via the network (during that process their order will probably get messed). On the second peer's side, the packets get assembled into the original frame. 32 | ![Example Chat](assets/images/example_chat.drawio.png) 33 | 34 | ### Packet format 35 | 36 | Here is how each packet looks like: 37 | 38 | ```elixir 39 | [seq:][frameid:][timestamp:] 40 | ``` 41 | 42 | where: 43 | 44 | - sequence_id - the ordering number of the packet (relative to each of the peers). It reflects the position of a particular packet in the packets' stream. 45 | - frame_id - the identifier which consists of the number of the frame to which the body of a given packet belongs, optionally followed by a single **'e'** character (meaning that the packet is the **e**nding packet of the frame). Note that frames are numbered relatively to each peer in that conversation and that frame_id does not describe the global order of the frames in the final file. 46 | - timestamp - a number indicating a time at which a given sentence was said. Timestamp describes the order of the frames from both peers. 47 | - text - the proper body of the packet, in our case - a bunch of characters which could be sent in a single packet. 48 | 49 | ### Packets generator 50 | 51 | We have equipped you with the tool which produces the packets in the format described previously, based on the input conversation. You can use it as a [mix](../glossary/glossary.md#mix) task, by typing: 52 | 53 | ```console 54 | mix generate_input --packetsPerFrame 55 | ``` 56 | 57 | where: 58 | 59 | - *input_file_path* is a path to the file which contains the text of the conversation. The input file consists of multiple lines of the following format: 60 | 61 | ``` 62 | : 63 | ``` 64 |

65 | 66 | - *packets_per_frame* is a number of packets on which each frame will be dismantled 67 | 68 | Based on the input file content, that command will create multiple files, `..` . For instance: `input.txt` with input from speakers `A` and `B` will be split into `input.A.txt` and `input.B.txt` files. 69 | The first file will contain the shuffled list of packets made from the `A` speaker's lines from the input file and the second file will contain a shuffled list of packets made out of the `B` speaker's lines of the input file. That `shuffle` is a way to simulate the imperfectness of the network - in real-life scenario, the order in which the packets are received is not always the same as the order in which they were sent. [Here](https://www.openmymind.net/How-Unreliable-Is-UDP/) you can read about this phenomenon occurring while using [UDP](../glossary/glossary.md#udp). 70 | Below you can see the steps which are taken during the input files generation:
71 | ![Example Input](assets/images/example_input.drawio.png) 72 | 73 | Create a file with your own input conversation occurring between two speakers or use the `input.txt` file where we have provided you such a conversation. Generate the files containing packets, with the use of `mix generate_input` task. 74 | 75 | ## Task description 76 | 77 | By the end of this chapter you should have generated the files containing packets (i.e. `input.A.txt` and `input.B.txt`). 78 | 79 | Your task is to assemble the conversation together from those packets. In the first step you will reproduce the sentences sent by each of the peers by gathering the words sent in packets in the appropriate order. In multimedia processing, we would say that you need to reproduce the *frame* out of the *transport layer packets*. Later on you need to put those *frames* in the correct order, producing the *raw data stream*. 80 | The resulting file should be the same as the input file from which you have created two packets list with the use of the `InputFilesGenerator`. 81 | -------------------------------------------------------------------------------- /basic_pipeline/02_SystemArchitecture.md: -------------------------------------------------------------------------------- 1 | # System Architecture 2 | 3 | Once we know what should be done, let's start thinking how should our system look like! 4 | 5 | First, let's get familiar with some terms. In the Membrane Framework, there is a concept of the [`pipeline`](../glossary/glossary.md#pipeline) which consists of multiple [`elements`](../glossary/glossary.md#element). Elements are linked with the use of the [`pads`](../glossary/glossary.md#pad), which can be input pads or output pads. 6 | Depending on what type of pads the particular element is equipped with, we distinguish the following types of elements: 7 | 8 | - Source - element with only output pads, the first element of each pipeline. It is responsible for fetching the data and transmitting it through the output pad. 9 | - Filter - element with both the input pads and the output pads. Its purpose is to get the data via the input pad, transform it and finally transmit the processed data via the output pad. 10 | - Sink - element with only input pads, the last element of the pipeline. It can be responsible for storing or playing the data received via the input pad. 11 | Between the pads, the [`buffers`](../glossary/glossary.md#buffer) are being sent. 12 | Buffers sent between the pads should be of the same type. In the Membrane Framework nomenclature, we say, that the pads should have corresponding capabilities - [`caps`](../glossary/glossary.md#caps). 13 | One might wonder when does the event of the buffers being sent occurs - well, it depends on the pad's mode! 14 | If the pad works in the *pull* mode, the buffer is sent when it is demanded by the succeeding element in the pipeline. 15 | Otherwise, the pad works in the *push* mode, which means that it is pushing the buffers once they are ready, independently from the fact if the following elements want (and are capable of processing) it. 16 | In our pipeline, all the pads will work in the *pull* mode, which inducts a flow control by a [backpressure](https://medium.com/@jayphelps/backpressure-explained-the-flow-of-data-through-software-2350b3e77ce7) mechanism. 17 | 18 | ## Scheme 19 | 20 | ![Pipeline scheme](assets/images/basic_pipeline.png) 21 | 22 | As you can see, our pipeline will consist of two twin branches, one per each of the peers. The branches will be merged with the `Mixer` element and the result produced by this element will be put in the file with the `Sink` elements. 23 | Here you can find the description of the particular elements of the system. 24 | 25 | ## Elements description 26 | 27 | - **Source** - that element is responsible for reading the list of [packets](../glossary/glossary.md#packet) from the file. 28 | - **Ordering Buffer** - this element is responsible for reordering the packets based on their sequence id. The most important part of this element is the buffer, within which there is a place for all the packets, identified by the packet's sequence id. Once the new packet is received, it is placed in the proper place in the buffer, depending on the sequence id. If there is a consistent part of packets "at the bottom of the buffer" (which means - packets with the subsequent sequence ids, which haven't already been sent yet), then this part is sent via the output pad. That is how we make sure that the next element will receive elements sorted by the sequence id. 29 | Below you can see how the Ordering Buffer is expected to work, once the message "How are you doing?" will be sent in the four packets, each with one word, and received in the following order: (are, you, how, doing?)
30 | ![Ordering Buffer](assets/images/ordering_buffer.drawio.png)
31 | 32 | - **Depayloader** - responsible for assembling the packets into the [frames](../glossary/glossary.md#frame). Since the depayloader receives the packets in the order of their sequence id (which means - the order in which they were 'sent'), it is capable of separating out the particular frames, basing on the **e** (ending packet of the frame) characters at the end of the frame id. This element also reads the timestamp from the packets' headers and puts it into buffer's 'pts' (*Presentation timestamp*) field. 33 | - **Mixer** - responsible for mixing the frames received via two input pads. The order in which the frames are mixed is based on their 'pts' (*Presentation timestamp*). 34 | - **Sink** - this element is responsible for writing the received frames to the file. Since the Mixer takes care of sorting the frames based on the timestamp, Sink can take advantage of the fact that the frames will be ordered and write them to the file in the order they are received. 35 | 36 | Now, that you have some high-level understanding of the system we can get down to implementation. 37 | -------------------------------------------------------------------------------- /basic_pipeline/03_Source.md: -------------------------------------------------------------------------------- 1 | # Source 2 | 3 | Let's get to the code! 4 | We will start where all the [pipelines](../glossary/glossary.md#pipeline) start - with the `Source` [element](../glossary/glossary.md#element). 5 | Since this will be the first element we implement, we need to find out something more about how the Membrane Framework's elements should be implemented and some concepts associated with them. 6 | The first thing you need to be aware of is that `Membrane. Element` describes a specific behavior, based on the OTP [GenServer's](https://elixir-lang.org/getting-started/mix-otp/genserver.html) behavior. 7 | Our process keeps a state which is updated in callbacks. 8 | We only need to provide an implementation of some callbacks in order to make our element act in the desired way. 9 | The set of callbacks that can be implemented depends on the type of the elements and we will get familiar with them during the implementation of these elements. 10 | However, each callback is required to return a tuple of a [specific form](https://hexdocs.pm/membrane_core/Membrane.Pipeline.html#t:callback_return/0). 11 | As you can see, we are returning an optional list of [actions](https://hexdocs.pm/membrane_core/Membrane.Pipeline.Action.html#t:t/0) to be performed, and 12 | the updated state (which later on will be passed to the next invoked callback). 13 | Take your time and read about the possible actions which can be requested to be performed while returning from the callback. Their usage is crucial for the pipeline to work. 14 | 15 | As you can judge based on the structure of the project, all the elements will be put in the `lib/elements` directory. Therefore there is a place where `source.ex` with the `Basic.Elements.Source` module's definition should be placed. 16 | 17 | ## What makes our module a Membrane Framework's element? 18 | 19 | Let's start with specifying that our module will implement the `Membrane.Source` behavior as well as alias the modules which will be used later in the module's code: 20 | 21 | **_`lib/elements/source.ex`_** 22 | 23 | ```elixir 24 | defmodule Basic.Elements.Source do 25 | use Membrane.Source 26 | alias Membrane.Buffer 27 | alias Basic.Formats.Packet 28 | ... 29 | end 30 | ``` 31 | 32 | ## Pads and options 33 | 34 | Later on, we will make use of [macros](https://elixir-lang.org/getting-started/meta/macros.html) defined in the `Membrane.Source` module: 35 | 36 | **_`lib/elements/source.ex`_** 37 | 38 | ```elixir 39 | defmodule Basic.Elements.Source do 40 | ... 41 | def_options location: [ 42 | spec: String.t(), 43 | description: "Path to the file" 44 | ] 45 | 46 | def_output_pad :output, 47 | accepted_format: %Packet{type: :custom_packets}, 48 | flow_control: :manual 49 | ... 50 | end 51 | ``` 52 | 53 | The first macro, `def_options` allows us to define the parameters which are expected to be passed while instantiating the element. The parameters will be passed as an automatically generated structure `%Basic.Elements.Source{}`. In our case, we will have a `:location` field inside of that structure. This parameter is about to be a path to the files which will contain input [packets](../glossary/glossary.md#packet). 54 | Later on, while instantiating the Source element, we will be able to write: 55 | 56 | ```elixir 57 | %Basic.Elements.Source{location: "input.A.txt"} 58 | ``` 59 | 60 | and the `:location` option will be passed during the construction of the element. 61 | 62 | The second macro, `def_output_pad`, lets us define the output pad. The pad name will be `:output` (which is a default name for the output pad). The second argument of the macro describes the `:accepted_format` - which is the type of data sent through the pad. As the code states, we want to send data in `Basic.Formats.Packet` format. 63 | What's more, we have specified that the `:output` pad will work in `:manual` mode. 64 | You can read more on pad specification [here](https://hexdocs.pm/membrane_core/Membrane.Pad.html#types). 65 | 66 | ## Initialization of the element 67 | 68 | Let's define our first callback! Why not start with [`handle_init/2`](https://hexdocs.pm/membrane_core/Membrane.Element.Base.html#c:handle_init/2), which gets called once the element is created? 69 | 70 | **_`lib/elements/source.ex`_** 71 | 72 | ```elixir 73 | defmodule Basic.Elements.Source do 74 | ... 75 | @impl true 76 | def handle_init(_context, options) do 77 | {[], 78 | %{ 79 | location: options.location, 80 | content: nil 81 | }} 82 | end 83 | ... 84 | end 85 | ``` 86 | 87 | As said before, `handle_init/2` expects a structure with the previously defined parameters to be passed as an argument. 88 | All we need to do there is to initialize the state - our state will be in a form of a map, and for now on we will put there a `location` (a path to the input file) and the `content`, where we will be holding packets read from the file, which haven't been sent yet. The recommended approach is to keep `handle_init/2` lean and defer any heavier initialization to `handle_setup/2` which runs automatically afterwards, if defined. In the next section we'll talk about more involved initialization and resources managed by our element. 89 | 90 | > ### TIP 91 | > 92 | > You might also wonder what is the purpose of the `@impl true` specifier, put just above the function signature - this is simply a way to tell the compiler 93 | > that the function defined below is about to be a callback. If we have misspelled the function name (or provided a wrong arguments list), we will be informed in the compilation time. 94 | 95 | ## Preparing our element 96 | 97 | In the `handle_setup/2` callback, we'd like to open, read and save the contents of the input file. We then save it in our state as `content`. 98 | 99 | **_`lib/elements/source.ex`_** 100 | 101 | ```elixir 102 | defmodule Basic.Elements.Source do 103 | ... 104 | @impl true 105 | def handle_setup(_context, state) do 106 | content = 107 | File.read!(state.location) 108 | |> String.split("\n") 109 | 110 | new_state = %{state | content: content} 111 | {[], new_state} 112 | end 113 | ... 114 | end 115 | ``` 116 | 117 | When the setup is complete, the element goes into `:playing` state. It can then demand buffers from previous elements and send its `:stream_format` to subsequent elements. Since we are implementing a source we do not have any previous element to demand from, but we can specify the format. We can do this, for example, in `handle_playing/2`: 118 | 119 | **_`lib/elements/source.ex`_** 120 | 121 | ```elixir 122 | defmodule Basic.Elements.Source do 123 | ... 124 | @impl true 125 | def handle_playing(_context, state) do 126 | {[stream_format: {:output, %Packet{type: :custom_packets}}], state} 127 | end 128 | ... 129 | end 130 | ``` 131 | 132 | The `:stream_format` action means that we want to transmit the information about the supported [formats](../glossary/glossary.md#stream-format-formerly-caps) through the `output` pad, to the next element in the pipeline. In [chapter 4](../basic_pipeline/04_StreamFormat.md) you will find out more about stream formats and learn why it is required to do so. 133 | 134 | ## Demands 135 | 136 | Before going any further let's stop for a moment and talk about the demands. Do you remember, that the `:output` pad is working in `:manual` mode? That means that the succeeding element has to ask the Source element for the data to be sent and our element has to take care of keeping that data in some kind of buffer until it is requested. 137 | Once the succeeding element requests for the data, the `handle_demand/4` callback will be invoked - therefore it would be good for us to define it: 138 | 139 | **_`lib/elements/source.ex`_** 140 | 141 | ```elixir 142 | defmodule Basic.Elements.Source do 143 | ... 144 | 145 | @impl true 146 | def handle_demand(:output, _size, :buffers, _context, state) do 147 | if state.content == [] do 148 | {[end_of_stream: :output], state} 149 | else 150 | [first_packet | rest] = state.content 151 | new_state = %{state | content: rest} 152 | 153 | actions = [ 154 | buffer: {:output, %Buffer{payload: first_packet}}, 155 | redemand: :output 156 | ] 157 | 158 | {actions, new_state} 159 | end 160 | end 161 | ... 162 | end 163 | ``` 164 | 165 | The callback's body describes the situation in which some buffers were requested. Then we are checking if we have any packets left in the list persisting in the state of the element. If that list is empty, we are sending an `end_of_stream` action, indicating that there will be no more buffers sent through the `:output` pad and that is why there is no point in requesting more buffers. 166 | 167 | However, in case of the `content` list of packets being non-empty, we are taking the head of that list, and storing the remaining tail of the list in the state of the element. Later on, we are defining the actions we want to take - that is, we want to return a buffer with the head packet from the original list. We make use of the [`buffer:` action](https://hexdocs.pm/membrane_core/Membrane.Element.Action.html#t:buffer/0), and specify that we want to transmit the [`%Buffer`](https://hexdocs.pm/membrane_core/Membrane.Buffer.html#t:t/0) structure through the `:output` pad. Note the fields available in the `%Buffer` structure - in our case, we make use of only the `:payload` field, which, according to the documentation, can be of `any` type - however, in almost all cases you will need to send binary data within this field. Any structured data (just like timestamps etc.) should be passed in the other fields available in the `%Buffer`, designed especially for that cases. 168 | 169 | Then there's the other action that is taken - the `:redemand` action, queued to take place on the `:output` pad. This action will simply invoke the `handle_demand/4` callback once again, which is helpful when the whole demand cannot be completely fulfilled in the single `handle_demand` invocation we are just processing. The great thing here is that the `size` of the demand will be automatically determined by the element and we do not need to specify it anyhow. Redemanding, in the context of sources, helps us simplify the logic of the `handle_demand` callback since all we need to do in that callback is to supply a single piece of data and in case this is not enough, take a [`:redemand`](https://hexdocs.pm/membrane_core/Membrane.Element.Action.html#t:redemand/0) action and invoke that callback once again. As you will see later, the process of redemanding is even more powerful in the context of [filter elements](../glossary/glossary.md#filter). 170 | 171 | > ### TIP 172 | > 173 | > Membrane also supports `:auto` demand mode, which should cover 90% of use cases. 174 | > 175 | > You can learn more about demand modes [here](https://membrane.stream/learn/get_started_with_membrane/6). 176 | 177 | In the next chapter we will explore what stream formats are in Membrane. 178 | -------------------------------------------------------------------------------- /basic_pipeline/04_StreamFormat.md: -------------------------------------------------------------------------------- 1 | # Stream format 2 | 3 | We owe you something...and we would like to pay it back as soon as possible! 4 | As promised in the [3rd chapter](03_Source.md), we will talk more about the concept of stream formats - which in fact we have used in the previous chapter, but which weren't described sufficiently. 5 | 6 | ## Why formats are important 7 | 8 | Specifying stream formats allows us to define what kind of data is flowing through the [pad](../glossary/glossary.md#pad). 9 | This isn't necessarily limited to data formats, as you'll see below. 10 | In the Membrane Framework's nomenclature, we say, that we define a stream format specification for a given [element](../glossary/glossary.md#element). 11 | 12 | We believe that an example might speak here louder than a plain definition, so we will try to describe it with a real-life scenario example. 13 | Let's say that we are connecting two elements that process the video multimedia. 14 | The link is made between the pads which are working on raw video data. 15 | Here is where stream formats come up - they can be defined with the following constraints: 16 | 17 | - data format - in our case, we are having a raw video format 18 | - some additional constraints - i.e. [frame](../glossary/glossary.md#frame) resolution (480p) , framerate (30 fps) etc. 19 | 20 | The `:stream_format` action helps us find out if the given elements are capable to communicate with each other. Not only can we not send the data between the pads if the format they are expecting is different - we need to take into consideration some other constraints! We can think of a situation in which the _data format_ would be the same (i.e. raw video data), but the element which receives the data performs a much more complex computation on that data than the sender, and therefore cannot digest such a great amount of data as the sender is capable of transmitting. Then their stream formats wouldn't be compatible, which could be expressed by adding some constraint, i.e. framerate. 21 | 22 | Stream formats help us define a contract between elements and prevent us from connecting incompatible elements. That is why it is always better to define precise constraints rather than using `stream_format: :any`. 23 | 24 | ## When are stream formats compatible? 25 | 26 | A comparison between formats is made when an input pad receives the `stream_format` action, and checks whether it matches its [accepted format](https://hexdocs.pm/membrane_core/Membrane.Pad.html#t:accepted_format/0). 27 | Here is how you define a stream format specification: 28 | 29 | 1. First you need to specify the format module 30 | 31 | ```elixir 32 | defmodule Formats.Raw do 33 | defstruct [:pixel_format, :framerate, :width, :height] 34 | end 35 | ``` 36 | 37 | Module name defines the type of the format, however it is possible to pass some other options in a form of a struct. That is why we have defined a structure with the use of `defstruct`. Our format will be described with the following options: 38 | 39 | - :pixel_format - pixel format, i.e. [I420](https://en.wikipedia.org/wiki/Chroma_subsampling) ([YUV](https://en.wikipedia.org/wiki/YUV)) or RGB888 40 | - :framerate - number of frames per second, i.e. 30 (FPS) 41 | - :width - width of the picture in pixels, i.e. 480 (px) 42 | - :height - height of the picture in pixels, i.e. 300 (px) 43 | 44 | 2. We specify the pad of the element with the format we have just defined, using the `:accepted_format` option. For the purpose of an example, let it be the `:input` pad: 45 | 46 | ```elixir 47 | def_input_pad :input, 48 | demand_unit: :buffers, 49 | accepted_format: 50 | %Format.Raw{pixel_format: pixel_format, framerate: framerate, width: 480, height: 300} 51 | when pixel_format in [:I420, :I422] and framerate >= 30 and framerate <= 60 52 | ``` 53 | As you can see, the argument of that option is simply a match pattern. The incoming stream format is later confronted against that match pattern. If it does not match, an exception is thrown at the runtime. 54 | 55 | To simplify the pattern definition, there is `any_of/1` helper function that allows to define a alternative of match patterns - the matching will succeed if the stream format received on the pad matches any of the patterns listed as `any_of/1` argument. Below you can see an example of defining alternative of match patterns: 56 | 57 | ```elixir 58 | def_input_pad :input, 59 | demand_unit: :buffers, 60 | accepted_format: 61 | any_of([ 62 | %Format.Raw{pixel_format: pixel_format, framerate: framerate, width: 480, height: 300} 63 | when pixel_format in [:I420, :I422] and framerate >= 30 and framerate <= 60, 64 | %Format.Raw{pixel_format: pixel_format, framerate: framerate, width: 720, height: 480} 65 | when pixel_format in [:I420, :I422] and framerate >= 30 and framerate <= 60 66 | ]) 67 | ``` 68 | 69 | Our journey with stream formats does not end here. We know how to describe their specification...but we also need to make our elements send the `:stream_format` events so that the following elements will be aware of what type of data our element is producing! 70 | 71 | An element can send a stream format as one of the [actions](https://hexdocs.pm/membrane_core/Membrane.Element.Action.html) it can take - the [`:stream_format` action](https://hexdocs.pm/membrane_core/Membrane.Element.Action.html#t:stream_format/0). 72 | 73 | Another thing is that we can specify the behavior of an element when it receives the stream format with the use of [`handle_stream_format/4` callback](https://hexdocs.pm/membrane_core/Membrane.Element.WithInputPads.html#c:handle_stream_format/4). 74 | 75 | For all the [filter elements](../glossary/glossary.md#filter), `handle_stream_format/4` has a default implementation, which is relaying the received format on all the output pads of that filter. 76 | However, if your filter is changing the format of data being sent, it should override the implementation of that callback to prevent formats flying through it, and send the proper spec via the output pads. 77 | 78 | For the [source element](../glossary/glossary.md#source), it is necessary to send the format as in each [pipeline](../glossary/glossary.md#pipeline) the source is the first element - formats wouldn't flow through the pipeline if the source element wouldn't have sent them. Sending can be done in the `handle_playing/2` callback. 79 | 80 | ## Example 81 | 82 | Imagine a pipeline, which starts with the source producing a video, which is then passed to the filter, responsible for reducing the quality of that video if it is too high. 83 | For the source element, we should have the `:output` pads format which would allow us to send video in the higher and in the lower quality. The same format should be specified on the input of the filter element. However, the stream format on the output of the filter should accept only video in the lower quality. 84 | Here is the definition of the source element: 85 | 86 | ```elixir 87 | # Source element 88 | 89 | defmodule Source do 90 | def_output_pad(:output, 91 | demand_unit: :buffers, 92 | stream_format: any_of([ 93 | %Format.Raw{pixel_format: pixel_format, framerate: framerate, width: 480, height: 300} 94 | when pixel_format in [:I420, :I422] and framerate >= 30 and framerate <= 60, 95 | %Format.Raw{pixel_format: pixel_format, framerate: framerate, width: 720, height: 480} 96 | when pixel_format in [:I420, :I422] and framerate >= 30 and framerate <= 60 97 | ]) 98 | ) 99 | ... 100 | def handle_playing(_context, state) do 101 | ... 102 | { {[stream_format: {:output, %Formats.Raw{pixel_format: I420, framerate: 45, width: 720, height: 300} }]}, state} 103 | end 104 | ``` 105 | 106 | While returning from the `handle_playing/2` callback, the element will send the format described by the `Formats.Raw` structure, through the `:output` pad. 107 | Will this format meet the accepted specification provided by us? Think about it! 108 | In fact, it will, as the `Formats.Raw` structure sent with `:stream_format` action matches the pattern - the value of `:pixel_format` field is one of `:I420` and `:I422`, and the `:framerate` is in the range between 30 and 60. In case the structure didn't match the pattern, a runtime exception would be thrown. 109 | 110 | Below there is the draft of the filter implementation: 111 | 112 | ```elixir 113 | # Filter 114 | 115 | defmodule Filter do 116 | def_input_pad:input, 117 | demand_unit: :buffers, 118 | accepted_format: any_of([ 119 | %Format.Raw{pixel_format: pixel_format, framerate: framerate, width: 480, height: 300} 120 | when pixel_format in [:I420, :I422] and framerate >= 30 and framerate <= 60, 121 | %Format.Raw{pixel_format: pixel_format, framerate: framerate, width: 720, height: 480} 122 | when pixel_format in [:I420, :I422] and framerate >= 30 and framerate <= 60 123 | ]) 124 | 125 | def_output_pad :output, 126 | demand_unit: :buffers, 127 | accepted_format: %Format.Raw{pixel_format: pixel_format, framerate: framerate, width: 480,height: 300} when pixel_format in [:I420, :I422] and framerate >= 30 and framerate <= 60 128 | 129 | ... 130 | 131 | def handle_stream_format(_pad, _stream_format, _context, state) do 132 | ... 133 | { {[stream_format: {:output, %Formats.Raw{pixel_format: I420, framerate: 60, width: 480, height: 300} }]}, state} 134 | end 135 | 136 | end 137 | ``` 138 | 139 | When we receive the spec on the input pad, we do not propagate it to our `:output` pad - instead, we send a different format, with reduced quality (values of the `width` and `height` fields might be lower). 140 | 141 | We hope by now you have a better understanding of what stream formats are. This knowledge will be helpful in the following chapters. 142 | -------------------------------------------------------------------------------- /basic_pipeline/05_Formats.md: -------------------------------------------------------------------------------- 1 | # Stream formats in action 2 | 3 | Since we have already discussed what stream formats are and how to use them, let's make use of them in our project! 4 | The first thing to do is to define modules responsible for describing the formats used in our pipeline. 5 | We will put them in a separate directory - `lib/formats`. Let's start with the format describing the [packets](../glossary/glossary.md#packet): 6 | 7 | **_`lib/formats/packet_format.ex`_** 8 | 9 | ```elixir 10 | defmodule Basic.Formats.Packet do 11 | @moduledoc """ 12 | A module describing the format of the packet. 13 | """ 14 | defstruct type: :custom_packets 15 | end 16 | ``` 17 | 18 | The definition of the module is not complicated, as you can see in the code snippet above - we are only defining a structure within that module, with a `:type` parameter, whose default value is `:custom_packets`. 19 | 20 | In our [pipeline](../glossary/glossary.md#pipeline) we will also send another type of data - [frames](../glossary/glossary.md#frame). Let's define a format for them: 21 | 22 | **_`lib/formats/frame_format.ex`_** 23 | 24 | ```elixir 25 | defmodule Basic.Formats.Frame do 26 | @moduledoc """ 27 | A module describing the format of the frame. 28 | """ 29 | defstruct encoding: :utf8 30 | end 31 | ``` 32 | 33 | Same as in the case of the previous format - we are defining a structure with a single field, called `:encoding`, and the default value of that field - `:utf8`. 34 | 35 | That's it! Format modules are really simple, and using them is not that hard either, as you will see in the subsequent chapters - defining the accepted stream formats and sending stream format actions! 36 | 37 | Before advancing you can test the `Source` [element](../glossary/glossary.md/#source), using the tests provided in `/test` directory. 38 | 39 | ```console 40 | mix test test/elements/source_test.exs 41 | ``` 42 | 43 | In case of errors, you may go back to the [Source chapter](/basic_pipeline/03_Source.md) or take a look how [Source](https://github.com/membraneframework/membrane_basic_pipeline_tutorial/blob/template/end/lib/elements/Source.ex) and [formats](https://github.com/membraneframework/membrane_basic_pipeline_tutorial/tree/template/end/lib/formats) should look like. 44 | -------------------------------------------------------------------------------- /basic_pipeline/06_OrderingBuffer.md: -------------------------------------------------------------------------------- 1 | # Ordering Buffer 2 | 3 | In this chapter we will deal with the next [element](../glossary/glossary.md#element) in our [pipeline](../glossary/glossary.md#pipeline) - the [Ordering Buffer](../glossary/glossary.md#jitter-buffer--ordering-buffer). 4 | As stated in the [chapter about the system architecture](02_SystemArchitecture.md), this element is responsible for ordering the incoming [packets](../glossary/glossary.md#packet), based on their sequence id. 5 | Because Ordering Buffer is a filtering element, we need to specify both the input and the output [pads](../glossary/glossary.md#pad): 6 | 7 | **_`lib/elements/ordering_buffer.ex`_** 8 | 9 | ```elixir 10 | defmodule Basic.Elements.OrderingBuffer do 11 | use Membrane.Filter 12 | alias Basic.Formats.Packet 13 | 14 | def_input_pad :input, 15 | flow_control: :manual, 16 | demand_unit: :buffers, 17 | accepted_format: %Packet{type: :custom_packets} 18 | 19 | def_output_pad :output, 20 | flow_control: :manual, 21 | accepted_format: %Packet{type: :custom_packets} 22 | ... 23 | end 24 | ``` 25 | 26 | Note the format specification definition there - we expect `Basic.Formats.Packet` of type `:custom_packets` to be sent on the input pad, and the same type of packets to be sent through the output pad. 27 | In the next step let's specify how we want the state of our element to look like: 28 | 29 | **_`lib/elements/ordering_buffer.ex`_** 30 | 31 | ```elixir 32 | defmodule Basic.Elements.OrderingBuffer do 33 | ... 34 | @impl true 35 | def handle_init(_context, _options) do 36 | {[], 37 | %{ 38 | ordered_packets: [], 39 | last_sent_seq_id: 0 40 | }} 41 | end 42 | ... 43 | end 44 | ``` 45 | 46 | If you don't remember what is the purpose of the Ordering Buffer, please refer to the [2nd chapter](02_SystemArchitecture.md). 47 | We will need to hold a list of ordered packets, as well as a sequence id of the packet, which most recently was sent through the output pad (we need to know if there are some packets missing between the last sent packet and the first packet in our ordered list). 48 | 49 | Handling demands is quite straightforward: 50 | 51 | **_`lib/elements/ordering_buffer.ex`_** 52 | 53 | ```elixir 54 | defmodule Basic.Elements.OrderingBuffer do 55 | ... 56 | @impl true 57 | def handle_demand(:output, size, _unit, _context, state) do 58 | {[demand: {:input, size}], state} 59 | end 60 | ... 61 | end 62 | ``` 63 | 64 | We simply send the `:demand` on the `:input` pad once we receive a demand on the `:output` pad. One packet on input corresponds to one packet on output so for each 1 unit of demand we send 1 unit of demand to the `:input` pad. 65 | 66 | Now we can go to the main part of the Ordering Buffer implementation - the `handle_buffer/4` callback. 67 | The purpose of this callback is to process the incoming buffer. It gets called once a new buffer is available and waiting to be processed. 68 | 69 | **_`lib/elements/ordering_buffer.ex`_** 70 | 71 | ```elixir 72 | defmodule Basic.Elements.OrderingBuffer do 73 | ... 74 | @impl true 75 | def handle_buffer(:input, buffer, _context, state) do 76 | packet = unzip_packet(buffer.payload) 77 | ordered_packets = [packet | state.ordered_packets] |> Enum.sort() 78 | state = %{state | ordered_packets: ordered_packets} 79 | [{last_seq_id, _} | _] = ordered_packets 80 | ... 81 | end 82 | 83 | defp unzip_packet(packet) do 84 | regex = ~r/^\[seq\:(?\d+)\](?.*)$/ 85 | %{"data" => data, "seq_id" => seq_id} = Regex.named_captures(regex, packet) 86 | {String.to_integer(seq_id), %Membrane.Buffer{payload: data} } 87 | end 88 | ... 89 | end 90 | ``` 91 | 92 | First, we are taking advantage of the [Regex module](https://hexdocs.pm/elixir/1.13/Regex.html) available in the Elixir. 93 | With the `Regex.named_captures` we can access the values of the fields defined within the regex description. 94 | Do you remember what our packet looks like? 95 | 96 | ``` 97 | [seq:7][frameid:2][timestamp:3]data 98 | ``` 99 | 100 | Above you can see an exemplary packet. We need to fetch the value of the sequence id (in our case it is equal to 7) and get the rest of the packet. 101 | Therefore we have defined the regex description as: 102 | 103 | ```elixir 104 | ~r/^\[seq\:(?\d+)\](?.*)$/ 105 | ``` 106 | 107 | > ### TIP - How to read the regex? 108 | > 109 | > - `~r/.../` stands for the `sigil_r/1` [sigil](https://elixir-lang.org/getting-started/sigils.html) 110 | > - `^` describes the beginning of the input 111 | > - `\[` stands for the opening square bracket ('\[') at the beginning is required to escape the char since the plain '\[' has a special meaning in the regex syntax 112 | > - `seq` is a sequence of 's', 'e', 'q' characters (we need to adjust our regex description to match the header of the packet) 113 | > - `\:` stands for the ':' character (we also need to escape that character since it is meaningful in the regex's syntax) 114 | > - `(?\d+)` allows us to define a named capture - later one, once we use the `Regex.named_captures/2`, we will retrieve the map with 'seq_id' key and the corresponding value equal to the string described by the `\d+` 'partial' regex (which means - one or more occurrences of a decimal). Generally speaking, a named capture can be specified with the following structure: `(?regex)` where `regex` is a regex description. 115 | > - `\]` is the escaped closing square bracket character 116 | > - `(?.*)` is a named capture description that allows us to get the value of a `.*` regex (any character no or any number of times) under a `data` key. 117 | > - `$` stands for the end of the input 118 | 119 | The result of `Regex.named_captures/2` applied to that regex description and the exemplary packet should be following: 120 | 121 | ```elixir 122 | {"seq_id"=>7, "data"=>"[frameid:2][timestamp:3]data"} 123 | ``` 124 | 125 | Once we unzip the header of the packet in the `handle_buffer/4` callback, we can put the incoming packet in the `ordered_packets` list and sort that list. Due to the fact, that elements of this list are tuples, whose first element is a sequence id (a value that is unique), the list will be sorted based on the sequence id. 126 | We also get the sequence id of the first element in the updated `ordered_packets` list. 127 | 128 | Here comes the rest of the `handle_buffer/4` definition: 129 | 130 | **_`lib/elements/ordering_buffer.ex`_** 131 | 132 | ```elixir 133 | defmodule Basic.Elements.OrderingBuffer do 134 | ... 135 | def handle_buffer(:input, buffer, _context, state) do 136 | ... 137 | if state.last_sent_seq_id + 1 == last_seq_id do 138 | {ready_packets_sequence, ordered_packets_left} = 139 | get_ready_packets_sequence(ordered_packets, []) 140 | 141 | {last_sent_seq_id, _} = List.last(ready_packets_sequence) 142 | 143 | state = %{ 144 | state 145 | | ordered_packets: ordered_packets_left, 146 | last_sent_seq_id: last_sent_seq_id 147 | } 148 | 149 | ready_buffers = Enum.map(ready_packets_sequence, &elem(&1, 1)) 150 | 151 | {[buffer: {:output, ready_buffers}], state} 152 | else 153 | {[redemand: :output], state} 154 | end 155 | end 156 | ... 157 | end 158 | ``` 159 | 160 | We need to distinguish between two situations: the currently processed packet can have a sequence id which is subsequent to the sequence id of the last sent packet or there might be some packets not yet delivered to us, with sequence ids in between the last sent sequence id and the sequence id of a currently processed packet. In the second case, we should store the packet and wait for the next packets to arrive. We will accomplish that using [`redemands` mechanism](../glossary/glossary.md#redemands), which will be explained in detail in the next chapter. 161 | However, in the first situation, we need to get the ready packet's sequence - that means, a consistent batch of packets from the `:ordered_packets`. This can be done in the following way: 162 | 163 | **_`lib/elements/ordering_buffer.ex`_** 164 | 165 | ```elixir 166 | defmodule Basic.Elements.OrderingBuffer do 167 | ... 168 | defp get_ready_packets_sequence([first_packet | ordered_rest], []) do 169 | get_ready_packets_sequence(ordered_rest, [first_packet]) 170 | end 171 | 172 | defp get_ready_packets_sequence( 173 | [next_seq = {next_id, _} | ordered_rest], 174 | [{last_id, _} | _] = ready_sequence 175 | ) 176 | when next_id == last_id + 1 do 177 | get_ready_packets_sequence(ordered_rest, [next_seq | ready_sequence]) 178 | end 179 | 180 | defp get_ready_packets_sequence(ordered_packets, ready_sequence) do 181 | {Enum.reverse(ready_sequence), ordered_packets} 182 | end 183 | ... 184 | end 185 | ``` 186 | 187 | Note the order of the definitions, since we are taking advantage of the pattern matching mechanism! 188 | The algorithm implemented in the snippet above is really simple - we are recursively taking the next packet out of the `:ordered_packets` buffer until it becomes empty or there is a missing packet (`next_id == last_id + 1`) between the last taken packet and the next packet in the buffer. 189 | Once we have a consistent batch of packets, we can update the state (both the`:ordered_packets` and the `:last_sent_seq_id` need to be updated) and output the ready packets by defining the `:buffer` action. 190 | 191 | Test the `OrderingBuffer`: 192 | 193 | ```console 194 | mix test test/elements/ordering_buffer_test.exs 195 | ``` 196 | 197 | Now the `OrderingBuffer` element is ready. Before we implement the next element let us introduce you to the concept of redemands. 198 | -------------------------------------------------------------------------------- /basic_pipeline/07_Redemands.md: -------------------------------------------------------------------------------- 1 | # Redemands 2 | 3 | Redemanding is a very convenient mechanism that helps you, the Membrane Developer, stick to the DRY (Don't Repeat Yourself) rule. 4 | Generally speaking, it can be used in two situations: 5 | 6 | - in the [source elements](../glossary/glossary.md#source) 7 | - in the [filter elements](../glossary/glossary.md#filter) 8 | 9 | To comprehensively understand the concept behind redemanding, you need to be aware of the typical control flow which occurs in Membrane's [elements](../glossary/glossary.md#element) - which you could have seen in the elements we have already defined. 10 | 11 | ## In Source elements 12 | 13 | In the [source elements](../glossary/glossary.md#source), there is a "side-channel", from which we can receive data. That "side-channel" can be, as in the exemplary [pipeline](../glossary/glossary.md#pipeline) we are working on, in form of a file, from which we are reading the data. In real-life scenarios it could be also, i.e. an [RTP](../glossary/glossary.md#rtp) stream received via the network. Since we have that "side-channel", there is no need to receive data via the input [pad](../glossary/glossary.md#pad) (that is why we don't have it in the source element, do we?). 14 | The whole logic of fetching the data can be put inside the `handle_demand/5` callback - once we are asked to provide the [buffers](../glossary/glossary.md#buffer), the `handle_demand/5` callback gets called and we can provide the desired number of buffers from the "side-channel", inside the body of that callback. No processing occurs here - we get asked for the buffer and we provide the buffer, simple as that. 15 | The redemand mechanism here lets you focus on providing a single buffer in the `handle_demand/5` body - later on, you can simply return the `:redemand` action and that action will invoke the `handle_demand/5` once again, with the updated number of buffers which are expected to be provided. Let's see it in an example - we could have such a `handle_demand/5` definition (and it wouldn't be a mistake!): 16 | 17 | ```elixir 18 | @impl true 19 | def handle_demand(:output, size, _unit, _context, state) do 20 | actions = for x <- 1..size do 21 | payload = Input.get_next() #Input.get_next() is an exemplary function which could be providing data 22 | {:buffer, %Membrane.Buffer(payload: payload)} 23 | end 24 | { {:ok, actions}, state} 25 | end 26 | ``` 27 | 28 | As you can see in the snippet above, we need to generate the required `size` of buffers in the single `handle_demand/5` run. The logic of supplying the demand there is quite easy - but what if you would also need to check if there is enough data to provide a sufficient number of buffer? You would need to check it in advance (or try to read as much data as possible before supplying the desired number of buffers). And what if an exception occurs during the generation, before supplying all the buffers? 29 | You would need to take under the consideration all these situations and your code would become larger and larger. 30 | 31 | Wouldn't it be better to focus on a single buffer in each `handle_demand/5` call - and let the Membrane Framework automatically update the demand's size? This can be done in the following way: 32 | 33 | ```elixir 34 | @impl true 35 | def handle_demand(:output, _size, unit, context, state) do 36 | payload = Input.get_next() #Input.get_next() is an exemplary function which could be providing data 37 | actions = [buffer: %Membrane.Buffer(payload: payload), redemand: :output] 38 | { {:ok, actions}, state} 39 | end 40 | ``` 41 | 42 | ## In Filter elements 43 | In the filter element, the situation is quite different. 44 | Since the filter's responsibility is to process the data sent via the input pads and transmit it through the output pads, there is no 'side-channel' from which we could take data. That is why in normal circumstances you would transmit the buffer through the output pad in the `handle_buffer/4` callback (which means - once your element receives a buffer, you process it, and then you 'mark' it as ready to be output with the `:buffer` action). When it comes to the `handle_demand/5` action on the output pad, all you need to do is to demand the appropriate number of buffers on the element's input pad. 45 | 46 | That behavior is easy to specify when we exactly know how many input buffers correspond to the one output buffer (recall the situation in the [Depayloader](../glossary/glossary.md#payloader-and-depayloader) of our pipeline, where we *a priori* knew, that each output buffer ([frame](../glossary/glossary.md#frame)) consists of a given number of input buffers ([packets](../glossary/glossary.md#packet))). However it becomes impossible to define if the output buffer might be a combination of a discretionary set number of input buffers. At the same time, we have dealt with an unknown number of required buffers in the OrderingBuffer implementation, where we didn't know how many input buffers do we need to demand to fulfill the missing spaces between the packets ordered in the list. How did we manage to do it? 47 | 48 | We simply used the `:redemand` action! In case there was a missing space between the packets, we returned the `:redemand` action, which immediately called the `handle_demand/5` callback (implemented in a way to request for a buffer on the input pad). The fact, that that callback invocation was immediate, which means - the callback was called synchronously, right after returning from the `handle_buffer/4` callback, before processing any other message from the element's mailbox - might be crucial in some situations, since it guarantees that the demand will be done before handling any other event. 49 | Recall the situation in the [Mixer](../glossary/glossary.md#mixer), where we were producing the output buffers right in the `handle_demand/5` callback. We needed to attempt to create the output buffer after: 50 | 51 | - updating the buffers' list in `handle_buffer/4` 52 | - updating the status of the [track](../glossary/glossary.md#track) in `handle_end_of_stream/3` 53 | Therefore, we were simply returning the `:redemand` action, and the `handle_demand/5` was called sequentially after on, trying to produce the output buffer. 54 | 55 | As you can see, redemand mechanism in filters helps us deal with situations, where we do not know how many input buffers to demand in order to be able to produce an output buffer/buffers. 56 | In case we don't provide enough buffers in the `handle_demand/5` callback (or we are not sure that we do provide), we should call `:redemand` somewhere else (usually in the `handle_buffer/4`) to make sure that the demand is not lost. 57 | 58 | With that knowledge let's carry on with the next element in our pipeline - `Depayloader`. 59 | -------------------------------------------------------------------------------- /basic_pipeline/08_Depayloader.md: -------------------------------------------------------------------------------- 1 | # Depayloader 2 | Since we have [packets](../glossary/glossary.md#packet) put in order by the [Ordering Buffer](../basic_pipeline/06_OrderingBuffer.md), we can assemble them into the original [frames](../glossary/glossary.md#frame). 3 | The Depayloader is an element responsible for this task. Specifically speaking, it unpacks the payload from the packets - 4 | and that is why it's called 'depayloader'. 5 | Let's create a new module in the `lib/elements/depayloader.ex` file: 6 | 7 | **_`lib/elements/depayloader.ex`_** 8 | 9 | ```elixir 10 | defmodule Basic.Elements.Depayloader do 11 | use Membrane.Filter 12 | 13 | alias Basic.Formats.{Packet, Frame} 14 | ... 15 | end 16 | ``` 17 | 18 | What input data do we expect? Of course in `Basic.Format.Packet` format! 19 | 20 | **_`lib/elements/depayloader.ex`_** 21 | 22 | ```elixir 23 | defmodule Basic.Elements.Depayloader do 24 | ... 25 | def_input_pad :input, 26 | flow_control: :manual, 27 | demand_unit: :buffers, 28 | accepted_format: %Packet{type: :custom_packets} 29 | ... 30 | end 31 | ``` 32 | 33 | However, our element will process that input data in a way that will change the format - on output, there will be frames instead of packets! 34 | We need to specify it while defining the `:output` pad: 35 | 36 | **_`lib/elements/depayloader.ex`_** 37 | 38 | ```elixir 39 | defmodule Basic.Elements.Depayloader do 40 | ... 41 | def_output_pad :output, 42 | flow_control: :manual, 43 | accepted_format: %Frame{encoding: :utf8} 44 | ... 45 | end 46 | ``` 47 | 48 | We will also need a parameter describing how many packets should we request once we receive a demand for a frame: 49 | 50 | **_`lib/elements/depayloader.ex`_** 51 | 52 | ```elixir 53 | defmodule Basic.Elements.Depayloader do 54 | ... 55 | def_options packets_per_frame: [ 56 | spec: pos_integer, 57 | description: 58 | "Positive integer, describing how many packets form a single frame. Used to demand the proper number of packets while assembling the frame." 59 | ] 60 | ... 61 | end 62 | ``` 63 | 64 | In the `handle_init/2` callback we are simply saving the value of that parameter in the state of our element: 65 | 66 | **_`lib/elements/depayloader.ex`_** 67 | 68 | ```elixir 69 | defmodule Basic.Elements.Depayloader do 70 | ... 71 | @impl true 72 | def handle_init(_context, options) do 73 | {[], 74 | %{ 75 | frame: [], 76 | packets_per_frame: options.packets_per_frame 77 | }} 78 | end 79 | ... 80 | end 81 | ``` 82 | 83 | Within the state, we will also hold a (potentially not complete) `:frame` - a list of packets, which form a particular frame. We will aggregate the packets in the `:frame` until the moment the frame is complete. 84 | 85 | As noted in the [chapter dedicated to stream formats](04_StreamFormat.md), since we are changing the type of data within the element, we cannot rely on the default implementation of the `handle_stream_format/4` callback. We need to explicitly send the updated version of the format: 86 | 87 | **_`lib/elements/depayloader.ex`_** 88 | 89 | ```elixir 90 | defmodule Basic.Elements.Depayloader do 91 | ... 92 | @impl true 93 | def handle_stream_format(:input, _stream_format, _context, state) do 94 | {[stream_format: {:output, %Frame{encoding: :utf8}}], state} 95 | end 96 | ... 97 | end 98 | ``` 99 | 100 | As in most elements, the `handle_demand/5` implementation is quite easy - what we do is simply make a demand on our `:input` pad once we receive a demand on the `:output` pad. However, since we are expected to produce a frame (which is formed from a particular number of packets) on the `:output` pad, we need to request a particular number of packets on the `:input` pad - that is why we have defined the `:packets_per_frame` option and now we will be making use of it. In case we would have been asked to produce 10 frames, and each frame would have been made out of 5 packets, then we would need to ask for 10\*5 = 50 packets on the `:input`. 101 | 102 | **_`lib/elements/depayloader.ex`_** 103 | 104 | ```elixir 105 | defmodule Basic.Elements.Depayloader do 106 | ... 107 | @impl true 108 | def handle_demand(:output, size, :buffers, _context, state) do 109 | {[demand: {:input, size * state.packets_per_frame}], state} 110 | end 111 | ... 112 | end 113 | ``` 114 | 115 | There is nothing left apart from processing the input data - that is - the packets. Since the packets are coming in order, we can simply hold them in the `:frame` list until all the packets forming that frame will be there. As you might remember, each packet has a frame id in its header, which can be followed by a 'b' or 'e' character, indicating the type of the packet (the one beginning a frame or the one ending the frame). We will use information about the type to find a moment in which we should produce a frame out of the packets list. 116 | 117 | **_`lib/elements/depayloader.ex`_** 118 | 119 | ```elixir 120 | defmodule Basic.Elements.Depayloader do 121 | ... 122 | @impl true 123 | def handle_buffer(:input, buffer, _context, state) do 124 | packet = buffer.payload 125 | 126 | regex = 127 | ~r/^\[frameid\:(?\d+(?[s|e]*))\]\[timestamp\:(?\d+)\](?.*)$/ 128 | 129 | %{"data" => data, "frame_id" => _frame_id, "type" => type, "timestamp" => timestamp} = 130 | Regex.named_captures(regex, packet) 131 | 132 | frame = [data | state.frame] 133 | ... 134 | end 135 | ``` 136 | 137 | Once again we are taking advantage of the `Regex.named_captures`. 138 | Once we fetch the interesting values of the header's parameters, we can update the `:frame`. 139 | 140 | **_`lib/elements/depayloader.ex`_** 141 | 142 | ```elixir 143 | defmodule Basic.Elements.Depayloader do 144 | ... 145 | @impl true 146 | def handle_buffer(:input, buffer, _context, state) do 147 | ... 148 | if type == "e" do 149 | buffer = %Membrane.Buffer{ 150 | payload: prepare_frame(frame), 151 | pts: String.to_integer(timestamp) 152 | } 153 | 154 | {[buffer: {:output, buffer}], %{state | frame: []}} 155 | else 156 | {[], %{state | frame: frame}} 157 | end 158 | ... 159 | end 160 | ``` 161 | 162 | Now, depending on the type of frame, we perform different actions. 163 | If we have the 'ending' packet, we are making the `:buffer` action with the frame made out of the packets (that's where `prepare_frame/1` function comes in handy), and clear the `:frame` buffer. Here is how the `prepare_frame/1` function can be implemented: 164 | 165 | **_`lib/elements/depayloader.ex`_** 166 | 167 | ```elixir 168 | defmodule Basic.Elements.Depayloader do 169 | ... 170 | defp prepare_frame(frame) do 171 | frame |> Enum.reverse() |> Enum.join("") 172 | end 173 | ... 174 | end 175 | ``` 176 | 177 | Otherwise, if the packet is not of the 'ending' type (that is - it can be both the 'beginning' frame or some packet in the middle), we are simply updating the state with the processed packet added to the `:frame` buffer and redemanding packet. 178 | 179 | Test the `Depayloader`: 180 | 181 | ```console 182 | mix test test/elements/depayloader_test.exs 183 | ``` 184 | 185 | With the [`Source`](../glossary/glossary.md#source), [`OrderingBuffer`](../glossary/glossary.md#jitter-buffer--ordering-buffer) and [`Depayloader`](../glossary/glossary.md#payloader-and-depayloader) elements ready we are able to read packets from file, order them based on their sequence ID and assemble them back into frames. 186 | In the next chapter we will be dealing with the [`Mixer`](../glossary/glossary.md#mixer) which will merge two message streams in order to create complete conversation. 187 | -------------------------------------------------------------------------------- /basic_pipeline/09_Mixer.md: -------------------------------------------------------------------------------- 1 | # Mixer 2 | 3 | Here comes the mixer - an [element](../glossary/glossary.md#element) responsible for mixing two streams of [frames](../glossary/glossary.md#frame), coming from two different sources. 4 | Once again we start with defining the initialization options and the pads of both types: 5 | 6 | **_`lib/elements/mixer.ex`_** 7 | 8 | ```elixir 9 | defmodule Basic.Elements.Mixer do 10 | @moduledoc """ 11 | Element responsible for mixing the frames coming from two sources, basing on their timestamps. 12 | """ 13 | use Membrane.Filter 14 | alias Basic.Formats.Frame 15 | 16 | def_input_pad :first_input, 17 | flow_control: :manual, 18 | demand_unit: :buffers, 19 | accepted_format: %Frame{encoding: :utf8} 20 | 21 | def_input_pad :second_input, 22 | flow_control: :manual, 23 | demand_unit: :buffers, 24 | accepted_format: %Frame{encoding: :utf8} 25 | 26 | def_output_pad :output, 27 | flow_control: :manual, 28 | accepted_format: %Frame{encoding: :utf8} 29 | ... 30 | end 31 | ``` 32 | 33 | Note, that we have defined two input [pads](../glossary/glossary.md#pad): `:first_input` and the `:second_input`. 34 | Each of these input pads will have a corresponding incoming [track](../glossary/glossary.md#track) in form of a [buffers](../glossary/glossary.md#buffer) stream. We need a structure that will hold the state of the track. Let's create it by defining a `Track` inside the mixer module: 35 | 36 | **_`lib/elements/mixer.ex`_** 37 | 38 | ```elixir 39 | defmodule Basic.Elements.Mixer do 40 | ... 41 | defmodule Track do 42 | @type t :: %__MODULE__{ 43 | buffer: Membrane.Buffer.t(), 44 | status: :started | :finished 45 | } 46 | defstruct buffer: nil, status: :started 47 | end 48 | ... 49 | end 50 | ``` 51 | 52 | As you can see in the code snippet above, the `Track` will consist of the `:buffer` field, holding the very last buffer received on the corresponding input pad, and the `:status` fields, indicating the status of the track - `:started`, in case we are still expecting some buffers to come (that means - in case `:end_of_stream` event hasn't been received yet) and `:finished` otherwise. 53 | 54 | It's a good practice to provide a type specification for such a custom struct since it makes the code easier to reuse and lets the compiler warn us about some misspellings (for instance in the status field atoms), which cause some hard to spot errors. 55 | 56 | A careful reader might notice, that we are holding only one buffer for each track, instead of a list of all the potentially unprocessed buffers - does it mean that we are losing some of them? Not at all, since we are taking advantage of the elements which have appeared earlier in the [pipeline](../glossary/glossary.md#pipeline) and which provide us with an ordered list of frames on each of the inputs - however, we will need to process each buffer just at the moment it comes on the pad. 57 | 58 | The logic we're going to implement can be described in the following three steps: 59 | 60 | - If all the tracks are in the 'active' state ('active' means - the ones in the `:started` state or the ones in the `:finished` state but with an unprocessed buffer in the `Track` structure) - output the one with the lower timestamp. Otherwise do nothing. 61 | - If all the tracks are in the `:finished` state and their `:buffer` is empty - send the `:end_of_stream` event. 62 | - For all the tracks which are in the `:started` state and their buffer is empty - demand on the pad corresponding to that track. 63 | 64 | The next step in our element implementation is quite an obvious one: 65 | 66 | **_`lib/elements/mixer.ex`_** 67 | 68 | ```elixir 69 | defmodule Basic.Elements.Mixer do 70 | ... 71 | @impl true 72 | def handle_init(_context, _options) do 73 | {[], 74 | %{ 75 | tracks: %{first_input: %Track{}, second_input: %Track{}} 76 | }} 77 | end 78 | ... 79 | end 80 | ``` 81 | 82 | We have provided a `handle_init/2` callback, which does not expect any options to be passed. We are simply setting up the structure of the element state. 83 | As mentioned previously, we will have a `Track` structure for each of the input pads. 84 | 85 | What's interesting is this is where the mixer having exactly two inputs stops being important. The missing functionality can be defined generically without much hassle. 86 | 87 | Following on the callbacks implementation, let's continue with the `handle_buffer/4` implementation: 88 | 89 | **_`lib/elements/mixer.ex`_** 90 | 91 | ```elixir 92 | defmodule Basic.Elements.Mixer do 93 | ... 94 | @impl true 95 | def handle_buffer(pad, buffer, _context, state) do 96 | tracks = Map.update!(state.tracks, pad, &%Track{&1 | buffer: buffer}) 97 | {tracks, buffer_actions} = get_output_buffers_actions(tracks) 98 | state = %{state | tracks: tracks} 99 | 100 | {buffer_actions ++ [redemand: :output], state} 101 | end 102 | ... 103 | end 104 | ``` 105 | 106 | In this callback we update the mixer's state by assigning the incoming buffer to its track. We can be sure no overwriting of an existing buffer happens, which will become more apparent as we delve further into the logic's implementation. 107 | 108 | Once the state is updated we gather all buffers that can be sent (might be none) in `get_output_buffers_actions/1` and return the coresponding `buffer` actions. In case any demands should be sent afterwards we also tell the output pad to redemand. 109 | 110 | **_`lib/elements/mixer.ex`_** 111 | 112 | ```elixir 113 | defmodule Basic.Elements.Mixer do 114 | ... 115 | @impl true 116 | def handle_end_of_stream(pad, _context, state) do 117 | tracks = Map.update!(state.tracks, pad, &%Track{&1 | status: :finished}) 118 | {tracks, buffer_actions} = get_output_buffers_actions(tracks) 119 | state = %{state | tracks: tracks} 120 | 121 | if Enum.all?(tracks, fn {track_id, track} -> 122 | track.status == :finished and not has_buffer?({track_id, track}) 123 | end) do 124 | {buffer_actions ++ [end_of_stream: :output], state} 125 | else 126 | {buffer_actions ++ [redemand: :output], state} 127 | end 128 | end 129 | ... 130 | end 131 | ``` 132 | 133 | What we did here was similar to the logic defined in `handle_buffer/4` - we have just updated the state of the track (in that case - by setting its status to `:finished`), gather the buffers and send them. The important difference is that in case all inputs have closed, we should forward an `end_of_stream` action instead of a `redemand`, signaling the mixer has finished its processing. 134 | 135 | Let's now implement gathering ready buffers: 136 | 137 | **_`lib/elements/mixer.ex`_** 138 | 139 | ```elixir 140 | defmodule Basic.Elements.Mixer do 141 | ... 142 | defp has_buffer?({_track_id, track}), 143 | do: track.buffer != nil 144 | 145 | defp can_send_buffer?(tracks) do 146 | started_tracks = 147 | Enum.filter( 148 | tracks, 149 | fn {_track_id, track} -> track.status != :finished end 150 | ) 151 | 152 | (started_tracks == [] and Enum.any?(tracks, &has_buffer?/1)) or 153 | (started_tracks != [] and Enum.all?(started_tracks, &has_buffer?/1)) 154 | end 155 | 156 | defp get_output_buffers_actions(tracks) do 157 | {buffers, tracks} = prepare_buffers(tracks) 158 | buffer_actions = Enum.map(buffers, fn buffer -> {:buffer, {:output, buffer}} end) 159 | {tracks, buffer_actions} 160 | end 161 | 162 | defp prepare_buffers(tracks) do 163 | if can_send_buffer?(tracks) do 164 | {next_track_id, next_track} = 165 | tracks 166 | |> Enum.filter(&has_buffer?/1) 167 | |> Enum.min_by(fn {_track_id, track} -> track.buffer.pts end) 168 | 169 | tracks = Map.put(tracks, next_track_id, %Track{next_track | buffer: nil}) 170 | {buffers, tracks} = prepare_buffers(tracks) 171 | {[next_track.buffer | buffers], tracks} 172 | else 173 | {[], tracks} 174 | end 175 | end 176 | end 177 | ``` 178 | 179 | The `prepare_buffers/1` function is the most involved here, so let's start with that. We first check whether we can send a buffer at all. The next buffer to send in order will of course be one with lowest `.pts`. We then empty the corresponding track's buffer. There might be more than one buffer ready to send and so we iterate the gathering recursively. 180 | 181 | We define `can_send_buffer?` as follows. If there's any `:started` track still waiting on a buffer we cannot send more, since whatever buffers the mixer's currently holding might come after the one that's yet to be received on this track. 182 | 183 | Otherwise, if all tracks have finished it can still be the case that some have non-empty buffers. We can happily send all of these in order since it is guaranteed there is no buffer preceding them that we would have to wait for. 184 | 185 | All that's left now is to handle redemands. 186 | 187 | **_`lib/elements/mixer.ex`_** 188 | 189 | ```elixir 190 | defmodule Basic.Elements.Mixer do 191 | ... 192 | def handle_demand(:output, _size, _unit, context, state) do 193 | demand_actions = 194 | state.tracks 195 | |> Enum.reject(&has_buffer?/1) 196 | |> Enum.filter(fn {track_id, track} -> 197 | track.status != :finished and context.pads[track_id].demand == 0 198 | end) 199 | |> Enum.map(fn {track_id, _track} -> {:demand, {track_id, 1}} end) 200 | 201 | {demand_actions, state} 202 | end 203 | ... 204 | end 205 | ``` 206 | 207 | Since it should be responsible for producing and sending `demand` actions to the corresponding input pads, we accordingly filter tracks for ones that are empty, started, and with no demands pending. 208 | It should also become clearer why in `handle_buffer/4` the receiving track is sure to have an empty buffer ready to be overwritten, since we only send demands to input pads of empty tracks. 209 | 210 | And that's all! the mixer's ready to mix, and ready to be tested: 211 | 212 | ```console 213 | mix test test/elements/mixer_test.exs 214 | ``` 215 | 216 | Now all that's left to do is to save our stream to file using [`Sink`](../glossary/glossary.md#sink). 217 | -------------------------------------------------------------------------------- /basic_pipeline/10_Sink.md: -------------------------------------------------------------------------------- 1 | # Sink 2 | 3 | The sink is the last [element](../glossary/glossary.md#element) in our [pipeline](../glossary/glossary.md#pipeline), designed to store the data processed by the pipeline. 4 | In contrast to the [filter elements](../glossary/glossary.md#filter), it won't have any output [pad](../glossary/glossary.md#pad) - that is why we need to make our element `use Membrane.Sink` and define the input pad only. 5 | Since we want to parameterize the usage of that element, it will be good to define the options structure, so that we can specify the path to the file where the output should be saved. This stuff is done in the code snippet below: 6 | 7 | **_`lib/elements/sink.ex`_** 8 | 9 | ```elixir 10 | defmodule Basic.Elements.Sink do 11 | use Membrane.Sink 12 | 13 | def_options location: [ 14 | spec: String.t(), 15 | description: "Path to the file" 16 | ] 17 | 18 | def_input_pad :input, 19 | flow_control: :manual, 20 | demand_unit: :buffers, 21 | accepted_format: _any 22 | ... 23 | end 24 | ``` 25 | 26 | No surprises there - now we need to specify the element's behavior by defining the relevant callbacks! 27 | 28 | **_`lib/elements/sink.ex`_** 29 | 30 | ```elixir 31 | 32 | defmodule Basic.Elements.Sink do 33 | ... 34 | @impl true 35 | def handle_init(_context, options) do 36 | {[], %{location: options.location}} 37 | end 38 | ... 39 | end 40 | ``` 41 | 42 | We have started with `handle_init/2`, where we are initializing the state of the element (we need to store the path to the output files). 43 | 44 | Later on, we can specify the `handle_playing/2` callback - this callback gets called once the pipeline's setup finishes - that is a moment when we can demand the [buffers](../glossary/glossary.md#buffer) for the first time (since the pipeline is already prepared to work): 45 | 46 | **_`lib/elements/sink.ex`_** 47 | 48 | ```elixir 49 | 50 | defmodule Basic.Elements.Sink do 51 | ... 52 | @impl true 53 | def handle_playing(_context, state) do 54 | {[demand: {:input, 10}], state} 55 | end 56 | ... 57 | end 58 | ``` 59 | 60 | There is only one more callback that needs to be specified - `handle_buffer/4`, which get's called once there are some buffers that can be processed (which means, that there are buffers to be written since there are no output pads through which we could be transmitting these buffers): 61 | 62 | **_`lib/elements/sink.ex`_** 63 | 64 | ```elixir 65 | 66 | defmodule Basic.Elements.Sink do 67 | ... 68 | @impl true 69 | def handle_buffer(:input, buffer, _context, state) do 70 | File.write!(state.location, buffer.payload <> "\n", [:append]) 71 | {[demand: {:input, 10}], state} 72 | end 73 | end 74 | ``` 75 | 76 | Note that after the successful writing, we are taking the `:demand` action and we ask for some more buffer. 77 | 78 | With the `Sink` completed, we have implemented all elements of our pipeline. Now let's move to the very last step - defining the actual pipeline using the elements we have created. 79 | -------------------------------------------------------------------------------- /basic_pipeline/11_Pipeline.md: -------------------------------------------------------------------------------- 1 | # Pipeline 2 | 3 | The time has come to assemble all the bricks together and create the pipeline! 4 | This task is really easy since the Membrane Framework provides a sort of DSL (*Domain Specific Language*) which allows you to link the prefabricated components together. 5 | In many real-life scenarios, this part would be the only thing you would need to do since you can take advantage of plenty of ready components (in form of [elements](../glossary/glossary.md#element) and [bins](../glossary/glossary.md#bin)) which are available as a part of the Membrane Framework. For now, we will create the pipeline out of the elements we have created during that tutorial! 6 | 7 | ## Defining the pipeline 8 | 9 | The pipeline is another behavior introduced by the Membrane Framework. To make the module a pipeline, we need to make it `use Membrane.Pipeline`. That is how we will start our implementation of the pipeline module, in the `lib/pipeline.ex` file: 10 | 11 | **_`lib/pipeline.ex`_** 12 | 13 | ```elixir 14 | 15 | defmodule Basic.Pipeline do 16 | 17 | use Membrane.Pipeline 18 | ... 19 | end 20 | ``` 21 | 22 | You could have guessed - all we need to do now is to describe the desired behavior by implementing the callbacks! In fact, the only callback we want to implement if the pipeline is the `handle_init/2` callback, called once the pipeline is initialized - of course, there are plenty of other callbacks which you might find useful while dealing with a more complex task. You can read about them [here](https://hexdocs.pm/membrane_core/Membrane.Pipeline.html#callbacks). 23 | Please add the following callback signature to our `Basic.Pipeline` module: 24 | 25 | **_`lib/pipeline.ex`_** 26 | 27 | ```elixir 28 | 29 | defmodule Basic.Pipeline do 30 | ... 31 | @impl true 32 | def handle_init(_context, _options) do 33 | 34 | end 35 | end 36 | ``` 37 | 38 | As you can see, we can initialize the pipeline with some options, but in our case, we do not need them. 39 | 40 | ## Supervising and children-parent relationship 41 | 42 | In Elixir's actor model, derived from the Erlang programming language (as well as in many other implementations of the actor system) there is a concept of the actors supervising each other. 43 | In case of the actor failing due to an error it is its supervisor's responsibility to deal with that fact - either by stopping that actor, restarting it, or performing some other action. 44 | With such a concept in mind, it's possible to create reliable and fault-tolerant actor systems. 45 | [Here](https://blog.appsignal.com/2021/08/23/using-supervisors-to-organize-your-elixir-application.html) there is a really nice article describing that concept and providing an example of the actor system. Feel free to stop here and read about the supervision mechanism in Elixir if you have never met with that concept before. 46 | Our pipeline will also be an actor system - with the `Basic.Pipeline` module being the supervisor of all its elements. 47 | As you have heard before - it is the supervisor's responsibility to launch its children's processes. 48 | In the Membrane Framework's pipeline there is a special action designed for that purpose - [`:spec`](https://hexdocs.pm/membrane_core/Membrane.Pipeline.Action.html#t:spec/0). 49 | As you can see, you need to specify the `Membrane.ChildrenSpec` for that purpose. 50 | 51 | Please stop for a moment and read about the [`Membrane.ChildrenSpec`](https://hexdocs.pm/membrane_core/Membrane.ChildrenSpec.html). 52 | We will wait for you and once you are ready, we will define our own children and links ;) 53 | 54 | Let's start with defining what children we need inside the `handle_init/2` callback! If you have forgotten what structure we want to achieve please refer to the [2nd chapter](02_SystemArchitecture.md) and recall what elements we need inside of our pipeline. 55 | 56 | **_`lib/pipeline.ex`_** 57 | 58 | ```elixir 59 | defmodule Basic.Pipeline do 60 | ... 61 | @impl true 62 | def handle_init(_context, _options) do 63 | spec = [ 64 | child(:input1, %Basic.Elements.Source{location: "input.A.txt"}) 65 | |> child(:ordering_buffer1, Basic.Elements.OrderingBuffer) 66 | |> child(:depayloader1, %Basic.Elements.Depayloader{packets_per_frame: 4}) 67 | |> via_in(:first_input) 68 | |> child(:mixer, Basic.Elements.Mixer) 69 | |> child(:output, %Basic.Elements.Sink{location: "output.txt"}), 70 | child(:input2, %Basic.Elements.Source{location: "input.B.txt"}) 71 | |> child(:ordering_buffer2, Basic.Elements.OrderingBuffer) 72 | |> child(:depayloader2, %Basic.Elements.Depayloader{packets_per_frame: 4}) 73 | |> via_in(:second_input) 74 | |> get_child(:mixer) 75 | ] 76 | 77 | {[spec: spec], %{}} 78 | end 79 | ... 80 | ``` 81 | 82 | Remember to pass the desired file paths in the `:location` option! 83 | 84 | Now... that's it! :) 85 | The spec list using Membrane's DSL is enough to describe our pipeline's topology. The child keywords spawn components of the specified type and we can use the `|>` operator to link them together. When the pads that should be linked are unamibiguous this is straightforward but for links like those with `Mixer` we can specify the pads using `via_in/1`. There also exists a `via_out/1` keyword which works similarly for output pads. 86 | 87 | As you can see the first argument to `child/2` is a component identifier. If we had omitted it Membrane would generate a unique identifier under the hood. For more about the `child` functions please refer to [the docs](https://hexdocs.pm/membrane_core/Membrane.ChildrenSpec.html#child/1). 88 | 89 | Our pipeline is ready! Let's try to launch it. 90 | We will do so by starting the pipeline, and then playing it. For the ease of use we will do it in a script. 91 | 92 | **_`start.exs`_** 93 | 94 | ```elixir 95 | {:ok, _sup, pipeline} = Membrane.Pipeline.start_link(Basic.Pipeline) 96 | Process.monitor(pipeline) 97 | 98 | # Wait for the pipeline to terminate 99 | receive do 100 | {:DOWN, _monitor, :process, ^pipeline, _reason} -> :ok 101 | end 102 | ``` 103 | 104 | You can execute it by running `mix run start.exs` in the terminal. 105 | 106 | In the output file (the one specified in the `handle_init/2` callback of the pipeline) you should see the recovered conversation. 107 | 108 | In case of any problems you can refer to the code on the `template/end` branch of `membrane_basic_pipeline_tutorial` repository. 109 | 110 | Now our solution is completed. You have acquired a basic understanding of Membrane, and you can implement a simple pipeline using elements. 111 | 112 | If you wish to extend your knowledge of Membrane's concepts we encourage you to read the [extension to this tutorial](../basic_pipeline_extension/01_Introduction.md). 113 | -------------------------------------------------------------------------------- /basic_pipeline/assets/diagrams/basic_pipeline.drawio: -------------------------------------------------------------------------------- 1 | 7Vtbb6s4EP410T6lAhMIeUzapq201ak2u+rpeVk5YC4twaxx2uT8+rXBDveEtrmhlpfg8djY428+jy/paZeL1Q2BkXePbRT0gGKvetpVDwBV0wbsh0vWqWSoaqnAJb4tlDLBzP+NhFAR0qVvo7igSDEOqB8VhRYOQ2TRggwSgt+Kag4Oil+NoIsqgpkFg6r00bepl0pNMMzkt8h3Pfll1RilOQsolUVPYg/a+C0n0q572iXBmKZvi9UlCrjxpF3SctOG3E3DCAppmwLP5OF5HKzv8PVv8vj0Sxm83j32RTdiupYdRjbrv0hiQj3s4hAG15l0QvAytBGvVWWpTOdPjCMhfEaUrsVgwiXFTOTRRSBy0cqnP3PvT+xdudBF6opjR5GJtUyElKx/5hO5UjyZFUtSslxMCX7ZDJ2a6KeySxxgkvRZU5KH5VRtKswc4yWx0BZDSmxC4iK6RU+gn1s59wExYjcILxBrPVMgKIDUfy2iEAowuxs9UXRMCFznFCLshzTO1fzABUxB+uVIgFJ4pTosYed9+uwlbYFM5bqSiRI8vgObotOvMFgKM/SAETDjThycNM6BYkiM/5bciyb3TB4j1hKayTj3QAr7vEw/JlahgEcp55Ex7wmYcpX4wsXYDRCM/PjCwgsmtmKmMnXgwg949xs+EkvqkjI1n2244jdpfxzBsOB1UjFtZFITa5OigmhVrWWWYlHUxRqYVlf8hOhOJi65edGJ3zyfolmU2vONMXnRYR0/CHLu4jgOsKw6R7KNuaEbFbdjMJxAYglCSFzwBVHLEz7a6HWviFC02uonIlcr4XMg0m8ZZ6uSiL0cXxtKs2fl0P0B8CoVk7+HWZVuMuvH+RO05E993/z5qVEGXaKoBvb4wjQBzFPThPkVWeIQ8ZfWkj8GZ8UfWvf44wexEfFDd7J0HESOwSM2RKZTyyOGZaK5U8sjEWskGyNE+GdYcwUct9CLv8it8cBgP4SzWf2dTWCiad2jnIxlngokc4zAZNCSWNThWTHLoHvMcoUiuA4wtI9EKzoy7UEdrZhgrhltwhM3gLx7Ker2HKoM1BbMoR6TOUbdI45Trmj0lsRhnBVv6N3jjY5EJIdd2ZQDjdMvbTTQPb4oBhrqUfnC6GagYXSPML4DjW2BRi1zHDXQ2LjM2TPHfhlAevZuCjDPigJku7vEAff+6jjeD4GiAF2Yorg7yp7aGGIqdsN2scIenF/XS86vnjxsGHxR5zdbOj84M+c3u+f8Mz98OZMTkNKqoOLxe/DxYfnU4+Q+Djp4bjb1ScxbtmQKf8SyuXMi8wM/ycZOMlwWC9LithCLPRjxVwsSezfGqhgpQlRcRNCq2917wFLfLGJJq0LJrEGSqhwMSh08QpkhC4d2F7B0ePzoJwdQ3Y5XOQAJ7TG/S8jtyldivlU0YzH8+Oi5xYe2LljiQR51bYlLWA+mfrCJdkJ505JPQWnE8VB/YFaZudLnU6GOtPDudU7LSCcHHr0GPFL2vjt1lUtwfbW0UtbKE2Tac1Esw2W1JqNUkVqqSAxJuaJ93awDdZs2B8H8ri20Q2L+EwBte5sAfAP0MADt4KLiLxQRbC8t5hPAgAs+04bzOOpldzKzCZ7N/2zOjNlg4/AwM3spJjz8TsJIKUJGrdlGHNXN7GAPM/uv4T8/bq5N4+5ujh7c27/7tz/+3XI5by6NPM4Zf95o+dam5kzUuFXLB3daA6Q0ZyY5KYUl0DkwdQ5NvQmcegpPpioByl5rak6HWa3uQIc4RLXgyC9jhVIDHGpA0xz7gZ1rh7pV6MHw0bxy2EBh8o2PE+KjZm1wVIA0n4XnsFCYlXbf6DfqbvSPm6/tf8PvWPAbGbsnMG0/8GPJ7H9facyU/XtOu/4f -------------------------------------------------------------------------------- /basic_pipeline/assets/diagrams/example_chat.drawio: -------------------------------------------------------------------------------- 1 | 7VtNk5s4EP0te+DolBEfxsfxTJI5JFvZmqpNctqSQdiUAXmFHOP8+pWMZIMkj5lZwGMnp0GNaNDrR6tf47Gc+6z8SOB6+RlHKLXAOCot58ECwHaBz/5wy66yBMCrDAuSRGLS0fCU/ETCOBbWTRKhojGRYpzSZN00hjjPUUgbNkgI3janxTht3nUNF0gzPIUw1a1fk4gu5SomR/sjShZLeWfbn1ZnMigni5UUSxjhbc3kvLece4IxrY6y8h6lHDyJS3XdhxNnDw9GUE7bXLDZbLMyWP3918evnz6vSurOR/+MZHh+wHQjViyelu4kBMwNQ5sNZmwJa24MU7xhXmfbZULR0xqG3LhlBGC2Jc1SNrLZoXCNCEXlyYe2D1AwDiGcIUp2bIq4wAsEeoI+nhhuj7FwpsK2rMUByOugiP/i4PoIETsQKL0AMU8D7C4vtogk+cICfsruP5sTdrTgR18QIs/geQY/WKwrSsdJiTjecZKm9zjFZO/IiYMQhSGPCyV4hWpn5oHnMqg6CYENps0Y2IYgBIYY9BYCnbN3xarCH2Ycw3xerPeLv8V4gDcWjYkWjUd2Ff6D2R5ZvuNpGPHHxRvL+aCBT/AmjziYD+PzAVDwjjwURK4J7wDMHd/viP8TBXBfB9wGBsT9vhAPjIh3Cy1EQWyksh8GaB53A62rcnncElrQF7RTDdo9kdXM8ngLYLuXBlvWeHW09wnjFtANLo6uraFbJeIbQPeA2uXQBTp3u9/gLgNu2w2uP3AdDVwNV6lF4hSVd1zoMTBQHolDJlFgUSRhE95mLFCZ0G/iDD/+zu3vPDF6KGvTHnZykLP1fasPalfx4fGy/UheVz0+ijSxqQSJLRFvSIjOV78UkgWi5+oyPei1oHqGmEobQSmkyY/m45oCLe7wBSdsITXRcOKFlS6qZYqr6qpVdaRWXxPFUYWD5mjPu8Oy/wcV3d9UPFf6n6WiPbYuyEXfaVLI9V7JRV9pRrjjYbno6HvOQeJKhRsTuMhYIBlgOJdn2d0OEzT2ciZ8gnOUKmI3TRY5Zy9zxkSzM+N7TBLC9E6cyJIo4j5mBBXJTzjf++MEW/P17xHxZpb3wB8K51S09ti++cymJbp4wpl1yBp1cj7zmp7c4kbjd67fiJ3fCbVkM1M4PWQL6QHHcYH6yUt6H+pKRaCvtJeAYyg/xoOWH3qD6WZUoIq2PdXRNjZUewNb7x9drQj0VRFoqKSHBVdvFV2tBpwEbw1cQ7PoSiVgoJbrhkbcoNhKx5couyfP191Wd/XzrdTFatqbek1HPdfFQO903dgniHGL3DfsNwj5Pvb8gvaucc9q11OhGeYdnTonyrWr0676OyrVKVeHDeL4/26wPDEq9rrxjk2wwbo8njzqWUUBM1KhbJ7Wvsc35e/BVt33Tahi4Paiiqt38jlVDJrbfiecDRo+R3LcvyYGw7SNX5mTXtPj67DWaJ3wvIvWJIo8ddSfLbTNdxNXceQMm++ALmu1TPXSBAhcUwL8E9EtJquTia1FC5AlHNqkfLOcyXGOlNpHmNrnQ1Nd1XyxOiiUJkpDzvSDAt/AXpUcnZVJTouU9LL6U2mlXqQeVep9x/M0lCdDqkWnxUeaTlEeRJSrKE8vjbLech5Zelc0JjBDbzTFtP4Q0T5K6oZl+DnesBlHb16bosRQWokq7JcMk6HtPWyYeqgP7MBUHzztsjlOi/2ULkuELqUQ3FBciIC/gHM98MRWeGL6oVFHH6PY8Pjr/6ruPP4PhfP+Pw== -------------------------------------------------------------------------------- /basic_pipeline/assets/diagrams/example_input.drawio: -------------------------------------------------------------------------------- 1 | 7VxLc+I4EP4te6Bq95AUlrGBI5DMo2pmLqndmRwNlh87xiJGJLC/fmVb8lMGBWzLJpAqgloPW936ultSSwN1sd5/DoyN8x2Z0BuAobkfqA8DAKYjnXyHhENMmAyHMcEOXDMmKSnhyf0PUiIrtnNNuM0VxAh52N3kiSvk+3CFczQjCNBbvpiFvPxTN4YNS4SnleGVqT9dEzu0F2Cc0r9A13bYkxV9GuesDVaY9mTrGCZ6y5DUx4G6CBDC8a/1fgG9kHeML3G9TxW5yYsF0MciFb7eQfP5yQavf9/9+Gc2e5kslG93ycvhA+sxNAkDaBIF2EE28g3vMaXOA7TzTRg2OySptMw3hDaEqBDivxDjA5WmscOIkBy89mgu3Lv4V1j9XqOp50zOw562HCUOLOHj4PArm3hOWwiTabUoxepZyMcL5KEg6p06jD6EHvc77GwlPylpi3bBCh5hIhuXRmBDfKSclkidoAWiNSTvSeoF0DOw+5p/D4OOWzspl4qW/KDSfYekabuvhrejTxoA3SOvO9/k5K+/7MIROfdcH945dGjPSBHlHqS55Jcd/p/FeV/IwA7xBsn3Ae0G6ifW+DJgRedx0a9hwXUIRfKAAViEjHMM//f2j3IV2vpPx8C55ok6Qa5vH3vMD4SdsAipFtX1vKR9wr4NK10a/W+Oi+HTxogE/kZUWn7kGttNrGUsdx8iYG54ru2TtAetsD3L9bzMaDM1ODFHhL7FAfoNMzkTsFR1PXmDVxhguD8+Essjh1YYUxBTFQt0mn5LFZaiUpqTVVaMWPtgS6DTAbUy7KtaUQXVykimWlEr1cqSYcz1Nzt8j/c4A8BlJQAJDHBerHns+MiHBaBREoPiinAaEvo8BJVLTPmMZqxd04xGGg/k+dFXAyqV4WlY6hxUNgdK9QbKi0E5EgSlLhOUYNQdSffWqxOV9FimpEcl9Sviih11qAoDJz8sTrhHBffHsiywWvHcH1Nf6lpN7k/R/1F4/g/gaFq9MU2rdQd/vdW0miD+JlI1rd4dSfdW04pKeipT0lpJ075nJsudjvZe02pD2ZpWlbp+pWTQl2Kxd/jTBfGnDPmjox0A6nxXRzfWITT85XYTr0sVkBc5OUUic4xqRKBpwInFRaC+msClVQ8CR6M8AseyAQimfQFgC0AaiwKpQsjtAGlcAlI8HxBCUjxdKGY1MnuQgygwkg6pSV8g1WGbNhGFIpAJxUmlU3kKjInPWcxIXND+IVHNIzFJy0Pi+IbEi5E4FUWiKhOJ0xISkylbyQ6K2Mq6J3hyIKhNZUNQlbqTWAcEQQcwyKJpToNQ6mYie83sbqI238KXaE+/sK2ocLcTtQdSwwqMNQyfxKm1ra6G3TXc4gjZ73gcJYdSHIRxSEyQaaiEalmRSEvRE/EElrUat8C2RU9pmGrGgPMY0yG+ZObsZb4U+UAppvt6mjXqeayBHeJNfk+ngj2EnOXHRYboeISLpYV/ZS0Wf7j2KvrUY6909eRyCOBFvjRnrkDfzdW4C+ZKNKJOsrnixNRVqp7RWaoHnGuuKjXdBaqHLhPVba+08zjTIcaUVsbqtVr6eQw612o1waFivObNbuXsFm/RsWXDJTU47GrmWUDUcMmNBQfdnWdV6vULtE+yqNrvqVYTrInXka9grtUEcwrhHDejlTNavPX5lo2W1DjXq5ltiR40kGy0OEcNujLbqnzcBcqH1d1uDJ/RCnsS5GnZ3AYMXLtzs7bYmJvGinGwR/O31riYP1P3Dj5+VIvJ205r2WJKjVe+kh1tRfRoiCL1bAh7Tc7ZvBirnFO/VCmEJ2gDe/knGYbDOLI58+svjnoo6r4KfbCs86F8K9NYD/nquo2O8ueLjXaUZ2CusqvH+3bh5mXZX6gwiFeBR76veIV4lKB4ZOGRv4jTPzwKbM5c4hZ/LKC3PiikGV7+FOwqkX4FllcoNKYVmEez2wZRfnqu/+GjCnodHnh66ABNExw8R4MMe7Ss1aGwhLZXsaBianA84KxWTfWxatS0WjUt3qPA29/hBX8nN93Uv1wl9SKFjp0uVETPyStSr0RQyiflO6Olb8EFt+CCrjDnvOCCd/DoCvZAy/udt41NVv2Y19wjt64R1uR3Jm8u3bELW7gbkNOmPDqlfPuAiKSpP1WSdHy35Kxwu2Re4JcIUcp9rgqYFDxxrSy2do9hKuWT6peLbX7lYtOYK9yA2EgyvTo8ysvcv64+/g8= -------------------------------------------------------------------------------- /basic_pipeline/assets/diagrams/ordering_buffer.drawio: -------------------------------------------------------------------------------- 1 | 7V1tk6I4EP41fpwpwpv6cXR2dqvOrdu7uarbvW8ZicotEgtx1f31l0CiQIODqxA8Mzu1ZRqC0E93+pVMzxovdx8jvFp8ph4Jeqbh7XrWc880h7bL/ueEfUpAfeSklHnke4J2JLz6P4kgGoK68T2yzp0YUxrE/ipPnNIwJNM4R8NRRLf502Y0yH/rCs8JILxOcQCpf/tevEipA7N/pH8i/nwhvxm5w/TIEsuTxZOsF9ij2wzJ+tCzxhGlcfppuRuTgDNP8iWd91Jx9HBjEQnjOhP+eDLf/vm6WGy9Ty+/7e3B5Mfk44O4yg8cbMQD/x55JPLDOaOONrMZicTdx3vJkvXWXwY4ZKPRjIbxqzhisPF04QfeBO/pht/SOsbT73I0WtDI/8nOxwE7hBiBHY5igbhl5M545TPFNSOyZud8kc+JCqTPeJc7cYLXsbwbGgR4tfbfkvvjE5c4mvvhiMYxXYqTBANIFJNdJWfRAS8m6IQuSRzt2SligiMQFjJui+H2KC9I0hZZWZGSgYWMzg9XPsLIPggkz0DVBKiyh38CQLInjhMcIvqdjGlAo4RuuckPh9cPggx95vB/jI4Dfx4yWkBm/AKceT5TmSdBXvqex79jtF7hKROlSXLas32k/Cm4wEmUTZ8FiWYs2ETCrjBaUT+ME644I/bL+DQ2Hp2ew+5izMboOGa//PQoHtOQPQj2E8gIE4It4YIwimiMY/x2EFEusplnSn/qykG1DkHhENJg1ZQGqylhsIAwmFoYFAmDq1oYbCAMlhYGRcIwVC0M8gYywBOPeT9iyFi3oHMa4uDDkcq4tgk94gmeHc+ZULoSVvZfEsd7YdjxJqbctMdLafbJzo+/8umPjhh9yxx5ltY8GezlIGTPm5nEh9+yx47TkpGcBwTYSH6gAL+8CHrKEc6G084A4xrdRFNyir3CXWU+BzkpJXa5mEQkwLH/I38j1xeCihXBYG5b1bqQQTPP4ZCmbmGGuYIkF4YpYyPzKU8sDduFH5NXthrw79yyqAII3RVcNstycj7bgwmV0S3RxcZcNgR9NluvzNdYmc/y3ZFyC42gv+ZoQWhfEJRbZwR9NR2bnw2rOcwv9OqDc5kA09F5Oz54hXfVlfAcuUAc0vjcYE9WWygG5ptVJhSeQwaefeNC0Rj46o19v8L71muBgrVAvckf6CBAhe9XdBI6EAXAsFxHASokQfma4MDEgA4Dzo/urHx4pz4McGCcr8OA5kx/qkbdDQMcGO3rMOBaYcA74Cu39g5MCbyfhNfgXwV89QYeJgC0099GwrfgEqh3+h2YDdBOvwpJUL8mwESAdvrPxrXvds7ph2E9d/o/MS3SVv4yKz/otn8vRVH79+2Dr9ywu7DNWvv3LYGv3Ja7MIGn/fsWvLqi9Vfv37u6tacbkqB8TRiaAHTdeZtw5CqdtwMRPr3felsuN+103g5gkPe+S3j7rbfDvDI+mA5QxrLO2+Z00b4VXVTbzW5dqCti6hduQjK5l2F5jC6vkKqwmFRA+nAXF4BfVYUzTsTkt6+DjtsxJUQGjI97phsk7GbP4s75J/QoaexLjuQr4iT8IEZwjqO/Ep3mrwiUw1PmKV0dMlRADAJmlQHmNAYYjGkhYOYdA1bsKFePGAxEIWLWHSNmD7qGGLROEDH7jhHrd25VhJV8HeK3EOK7g37ejRxASWi5g9OAdX1dw7v0tSyrbg2vQVxhlV7nc1tQ8eIuK8rTuciACRy92CuQBOXpXGTAuj6QgrzzU+YyZVyzMi+sKD6HLE96RG5P5TSz8PZrMnnYGJMRTBHoEvovVFHP3MxKuXIhmGnQTTONgl5b2RsEHSYrdMtE48quvE0KlWx/AdBeL/CKf5wFZPfE93js8eqjJz4+TwO8XvvTvEHNW19ZfZH1lqSieajE1Ky+HOxvC9UX4WFAIDNIOSVASdqFVZoHVCjTuKggAekTgDoNuFKx/GejQi6m4YIPKtlWI7EkqRNRurpwBCf4jQR5kapf00mi54zm5heF51O6LPY1FZN7hw2I3n/XGVUU7CSkxiO6jmwUdlIqAkpnszVpCEuYbdMplvP3wiqkvWvnWBrcI6VkV4wqo18MngdTMp3euGkHtuZt4NjOhXthpNrS3RZpVLIbRm1P78Cg/6M4NAe6+iRayZ4XGvRmQVefLyvZ3kLn0FvInBZNfQeS6NKH0El0xaKgflUwYaJPu/MXl8IPW92+6847zSELs3nanb/QyFeJQWfcebNGKk97dlcFvQPmHKbXNOjNgt4Bww3zcAD0zpW/f4nXqGSf+LaZXbVfjOHRpEvzpbbC6fLYmV2FjvKuQvmHx7r/ttJNvjmIzNqvDlYlWdp5efBwo+cvA7f/+pJt5mNXs6Rfod3Xl8yyBrBip75zx536hUZ9q6StqN1OffltJxFz7xgxy+wcZHVeOevfMWSu2e8aZDUSP/fYzmNWOA/ttPOYRl61bbMgAHW7eYaFNcLuFyxs0908FswxVbo/t9nGk2rQbbbxsOHx7xunpx//SrT14T8= -------------------------------------------------------------------------------- /basic_pipeline/assets/images/Illo_basic pipeline.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/membraneframework/membrane_tutorials/3e676855eeeab0da495da549133c4886b3608d16/basic_pipeline/assets/images/Illo_basic pipeline.png -------------------------------------------------------------------------------- /basic_pipeline/assets/images/basic_pipeline.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/membraneframework/membrane_tutorials/3e676855eeeab0da495da549133c4886b3608d16/basic_pipeline/assets/images/basic_pipeline.png -------------------------------------------------------------------------------- /basic_pipeline/assets/images/example_chat.drawio.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/membraneframework/membrane_tutorials/3e676855eeeab0da495da549133c4886b3608d16/basic_pipeline/assets/images/example_chat.drawio.png -------------------------------------------------------------------------------- /basic_pipeline/assets/images/example_input.drawio.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/membraneframework/membrane_tutorials/3e676855eeeab0da495da549133c4886b3608d16/basic_pipeline/assets/images/example_input.drawio.png -------------------------------------------------------------------------------- /basic_pipeline/assets/images/ordering_buffer.drawio.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/membraneframework/membrane_tutorials/3e676855eeeab0da495da549133c4886b3608d16/basic_pipeline/assets/images/ordering_buffer.drawio.png -------------------------------------------------------------------------------- /basic_pipeline/index.md: -------------------------------------------------------------------------------- 1 | --- 2 | title: All you need to know about pipelines pt 1 3 | description: >- 4 | See how pipeline works in Membrane and learn how to set it up for you 5 | part: 2 6 | graphicPath: assets/images/Illo_basic pipeline.png 7 | --- 8 | 9 | | number | title | file | 10 | | ------ | ------------------- | ------------------------ | 11 | | 1 | Introduction | 01_Introduction.md | 12 | | 2 | System Architecture | 02_SystemArchitecture.md | 13 | | 3 | Source | 03_Source.md | 14 | | 4 | Stream format | 04_StreamFormat.md | 15 | | 5 | Formats | 05_Formats.md | 16 | | 6 | Ordering Buffer | 06_OrderingBuffer.md | 17 | | 7 | Redemands | 07_Redemands.md | 18 | | 8 | Depayloader | 08_Depayloader.md | 19 | | 9 | Mixer | 09_Mixer.md | 20 | | 10 | Sink | 10_Sink.md | 21 | | 11 | Pipeline | 11_Pipeline.md | 22 | -------------------------------------------------------------------------------- /basic_pipeline_extension/01_Introduction.md: -------------------------------------------------------------------------------- 1 | 2 | In this section, we gathered some topics which should give you better insight into Membrane. We will base on the ["All you need to know about pipelines pt 1" tutorial](../basic_pipeline/01_Introduction.md), so make sure you have gone through it first. 3 | The chapters are loosely related, so feel free to read them in any order you want. 4 | 5 | **Table of contents** 6 | 7 | - [Bin](../basic_pipeline_extension/02_Bin.md) - how to create a reusable group of [elements](../glossary/glossary.md#element) 8 | - [Dynamic Pads](../basic_pipeline_extension/03_DynamicPads.md) - a more flexible way to define element's [pads](../glossary/glossary.md#pad) 9 | - [Tests](../basic_pipeline_extension/04_Tests.md) - creating unit tests for the elements 10 | -------------------------------------------------------------------------------- /basic_pipeline_extension/02_Bin.md: -------------------------------------------------------------------------------- 1 | # Bin 2 | 3 | A Membrane's bin is a container for [elements](../glossary/glossary.md#element), which allows for creating reusable groups of elements. 4 | Bin is similar to a [pipeline](../glossary/glossary.md#pipeline) in that it consists of linked elements. Such bin can then be placed inside a pipeline and linked with other entities - elements or bins. Bins can also be nested within one another. 5 | Bin also has another advantage - it manages its children, for instance by dynamically spawning or replacing them as the stream changes. 6 | 7 | ## Enclosing pipeline elements inside a bin 8 | 9 | As you can see, we have [`Source`](../glossary/glossary.md#source) -> [`Ordering Buffer`](../glossary/glossary.md#jitter-buffer--ordering-buffer) -> [`Depayloader`](../glossary/glossary.md#payloader-and-depayloader) chain, which is duplicated. 10 | ![Pipeline scheme](assets/images/basic_pipeline.png)
11 | 12 | We can encapsulate these elements inside `Bin`. 13 | ![Pipeline scheme using bin](assets/images/basic_pipeline_bin.png)
14 | 15 | Notice that there is no direct connection between `Depayloader` and [`Mixer`](../glossary/glossary.md#mixer). We have to explicitly link the `Depayloader` with `Bin`'s output [pads](../glossary/glossary.md#pad) and then we will connect the output pads to `Mixer`'s input pads. 16 | 17 | Let's define the bin's output pads and its elements. 18 | 19 | **_`lib/Bin.ex`_** 20 | 21 | ```elixir 22 | defmodule Basic.Bin do 23 | use Membrane.Bin 24 | 25 | def_output_pad :output, accepted_format: %Basic.Formats.Frame{encoding: :utf8} 26 | 27 | def_options input_filename: [ 28 | type: :string, 29 | description: "Input file for conversation." 30 | ] 31 | 32 | @impl true 33 | def handle_init(_ctx, options) do 34 | spec = [ 35 | child(:input, %Basic.Elements.Source{location: options.input_filename}) 36 | |> child(:ordering_buffer, Basic.Elements.OrderingBuffer) 37 | |> child(:depayloader, %Basic.Elements.Depayloader{packets_per_frame: 4}) 38 | |> bin_output(:output) 39 | ] 40 | 41 | {[spec: spec] %{} } 42 | end 43 | end 44 | ``` 45 | 46 | The output pads of the bin are matching the ones we [defined for depayloader](/basic_pipeline/08_Depayloader.md#libelementsdepayloaderex-2). 47 | Notice that the last link is between `depayloader` and the bin's output pads. In general, if we wanted to receive data in a bin we would need to link the first processing component in the bin with the `bin_input()`. 48 | 49 | ## Modifying pipeline using bin 50 | 51 | Using the bin we created, we can replace the elements in the pipeline. 52 | 53 | **_`lib/Pipeline.ex`_** 54 | 55 | 56 | ```elixir 57 | defmodule Basic.Pipeline do 58 | @moduledoc """ 59 | A module providing the pipeline, which aggregates and links the elements. 60 | """ 61 | use Membrane.Pipeline 62 | 63 | @impl true 64 | def handle_init(_ctx, _opts) do 65 | spec = [ 66 | child(:bin1, %Basic.Bin{input_filename: "input.A.txt"}) 67 | |> via_in(:first_input) 68 | |> child(:mixer, Basic.Elements.Mixer), 69 | child(:bin2, %Basic.Bin{input_filename: "input.B.txt"}) 70 | |> via_in(:second_input) 71 | |> get_child(:mixer), 72 | get_child(:mixer) 73 | |> child(:output, %Basic.Elements.Sink{location: "output.txt"}) 74 | ] 75 | 76 | {[spec: spec], %{}} 77 | end 78 | end 79 | ``` 80 | 81 | Combining the usage of the bin and [dynamic pads](03_DynamicPads.md) will result in an even cleaner and more scalable solution. 82 | -------------------------------------------------------------------------------- /basic_pipeline_extension/03_DynamicPads.md: -------------------------------------------------------------------------------- 1 | # Dynamic Pads 2 | 3 | The solution we have implemented along the tutorial has at least one downside - it is definitely not easily extendable. 4 | What if we needed to support mixing streams coming from three different speakers in the conversation? 5 | Well, we would need to add another input [pad](../glossary/glossary.md#pad) in the [Mixer](../glossary/glossary.md#mixer), for instance - `:third_input` pad, and then update our [pipeline](../glossary/glossary.md#pipeline) definition: 6 | 7 | **_`lib/Pipeline.ex`_** 8 | 9 | ```elixir 10 | @impl true 11 | def handle_init(_ctx, _opts) do 12 | spec = [ 13 | ... 14 | child(:input3, %Basic.Elements.Source{location: "input.C.txt"}) 15 | |> child(:ordering_buffer3, Basic.Elements.OrderingBuffer) 16 | |> child(:depayloader3, %Basic.Elements.Depayloader{packets_per_frame: 4}), 17 | ... 18 | get_child(:depayloader3) 19 | |> via_in(:third_input) 20 | |> get_child(:mixer), 21 | ] 22 | ... 23 | end 24 | ``` 25 | 26 | But what if there were 4 people in the conversation? Adding another pad to the Mixer would solve the problem, but this solution does not scale well. 27 | And what if the number of speakers was unknown at the moment of the compilation? Then we wouldn't be able to predefine the pads in the Mixer. 28 | The Membrane Framework comes with a solution for this sort of problem - and the solution is called: **dynamic pads**. 29 | 30 | ## What is the idea behind the dynamic pads? 31 | 32 | Well, the idea is quite simple! Instead of specifying a single pad with a predefined name (*static pad*) as we did in all the modules before, we specify, that we want **a set of pads** of a given type. Initially, that set will be empty, but with each link created in the parent's child specification, the [element](../glossary/glossary.md#element) will be informed, that the pad of a given type was added - and therefore it will be able to invoke the `handle_pad_added/3` callback. 33 | 34 | ## The Mixer revisited 35 | 36 | Let's try to use dynamic pads for the input in the Mixer! 37 | The very first thing we need to do is to use the `def_input_pads` appropriately. 38 | 39 | **_`lib/elements/Mixer.ex`_** 40 | 41 | ```elixir 42 | ... 43 | def_input_pad :input, 44 | demand_unit: :buffers, 45 | flow_control: :manual, 46 | availability: :on_request, 47 | accepted_format: %Basic.Formats.Frame{encoding: :utf8} 48 | ... 49 | ``` 50 | 51 | We have added the [`availability: :on_request` option](https://hexdocs.pm/membrane_core/Membrane.Pad.html#t:availability/0), which allows us to define the set of dynamic pads, identified as `:input`. 52 | 53 | No more do we have the `:first_input` and the `:second_input` pads defined, so we do not have the [tracks](../glossary/glossary.md#track) corresponding to them either! Let's update the `handle_init/2` callback: 54 | 55 | **_`lib/elements/Mixer.ex`_** 56 | 57 | ```elixir 58 | ... 59 | @impl true 60 | def handle_init(_ctx, _options) do 61 | {[], %{ tracks: %{} }} 62 | end 63 | ... 64 | ``` 65 | 66 | Tracks map is initially empty since there are no corresponding pads. 67 | The next thing we need to do is to implement the `handle_pad_added/3` callback, which will be called once the pipeline starts, with some links pointing to dynamic pads: 68 | 69 | **_`lib/elements/Mixer.ex`_** 70 | 71 | ```elixir 72 | ... 73 | @impl true 74 | def handle_pad_added(pad, _context, state) do 75 | state = %{state | tracks: Map.put(state.tracks, pad, %Track{})} 76 | {[], state} 77 | end 78 | ... 79 | ``` 80 | 81 | Once a pad is created, we add a new `Track` to the tracks map, with the pad being its key. 82 | That's it! Since we have already designed the Mixer in a way it is capable of serving more tracks, there is nothing else to do. 83 | 84 | ## Updated pipeline 85 | 86 | Below you can find the updated version of the pipeline's `handle_init/2` callback: 87 | 88 | **_`lib/Pipeline.ex`_** 89 | 90 | ```elixir 91 | ... 92 | @impl true 93 | def handle_init(_ctx, _opts) do 94 | spec = [ 95 | child(:input1, %Basic.Elements.Source{location: "input.A.txt"}) 96 | |> child(:ordering_buffer1, Basic.Elements.OrderingBuffer) 97 | |> child(:depayloader1, %Basic.Elements.Depayloader{packets_per_frame: 4}), 98 | child(:input2, %Basic.Elements.Source{location: "input.B.txt"}) 99 | |> child(:ordering_buffer2, Basic.Elements.OrderingBuffer) 100 | |> child(:depayloader2, %Basic.Elements.Depayloader{packets_per_frame: 4}), 101 | get_child(:depayloader1) 102 | |> via_in(Pad.ref(:input, :first)) 103 | |> child(:mixer, Basic.Elements.Mixer), 104 | get_child(:depayloader2) 105 | |> via_in(Pad.ref(:input, :second)) 106 | |> get_child(:mixer), 107 | get_child(:mixer) 108 | |> child(:output, %Basic.Elements.Sink{location: "output.txt"}) 109 | ] 110 | 111 | {[spec: spec], %{}} 112 | end 113 | ... 114 | ``` 115 | 116 | The crucial thing was to change the plain atom name identifying the pad (like `:first_input`) into the [Membrane.Pad.ref/2](https://hexdocs.pm/membrane_core/Membrane.Pad.html#ref/2). 117 | The first argument passed to that function is the name of the dynamic pad's set (in our case: `:input`, as we have defined the `:input` dynamic pads set in the Mixer element), and the second argument is a particular pad identifier. 118 | As you can see, we have created two `:input` pads: `:first` and `:second`. While starting the pipeline, the `handle_pad_added/3` callback will be called twice, once per each dynamic pad created. 119 | 120 | ## Further actions 121 | 122 | As an exercise, you can try to modify the `lib/pipeline.ex` file and define a pipeline consisting of three parallel branches, being mixed in a single Mixer. Later on, you can check if the pipeline works as expected, by generating the input files out of the conversation in which participate three speakers. 123 | 124 | If you combine the approach taken in the chapter about [Bin](02_Bin.md) you can simplify this solution by reducing the size of the link defintions inside the pipeline. 125 | -------------------------------------------------------------------------------- /basic_pipeline_extension/assets/diagrams/basic_pipeline.drawio: -------------------------------------------------------------------------------- 1 | 7Vtbb6s4EP410T6lAhMIeUzapq201ak2u+rpeVk5YC4twaxx2uT8+rXBDveEtrmhlpfg8djY428+jy/paZeL1Q2BkXePbRT0gGKvetpVDwBV0wbsh0vWqWSoaqnAJb4tlDLBzP+NhFAR0qVvo7igSDEOqB8VhRYOQ2TRggwSgt+Kag4Oil+NoIsqgpkFg6r00bepl0pNMMzkt8h3Pfll1RilOQsolUVPYg/a+C0n0q572iXBmKZvi9UlCrjxpF3SctOG3E3DCAppmwLP5OF5HKzv8PVv8vj0Sxm83j32RTdiupYdRjbrv0hiQj3s4hAG15l0QvAytBGvVWWpTOdPjCMhfEaUrsVgwiXFTOTRRSBy0cqnP3PvT+xdudBF6opjR5GJtUyElKx/5hO5UjyZFUtSslxMCX7ZDJ2a6KeySxxgkvRZU5KH5VRtKswc4yWx0BZDSmxC4iK6RU+gn1s59wExYjcILxBrPVMgKIDUfy2iEAowuxs9UXRMCFznFCLshzTO1fzABUxB+uVIgFJ4pTosYed9+uwlbYFM5bqSiRI8vgObotOvMFgKM/SAETDjThycNM6BYkiM/5bciyb3TB4j1hKayTj3QAr7vEw/JlahgEcp55Ex7wmYcpX4wsXYDRCM/PjCwgsmtmKmMnXgwg949xs+EkvqkjI1n2244jdpfxzBsOB1UjFtZFITa5OigmhVrWWWYlHUxRqYVlf8hOhOJi65edGJ3zyfolmU2vONMXnRYR0/CHLu4jgOsKw6R7KNuaEbFbdjMJxAYglCSFzwBVHLEz7a6HWviFC02uonIlcr4XMg0m8ZZ6uSiL0cXxtKs2fl0P0B8CoVk7+HWZVuMuvH+RO05E993/z5qVEGXaKoBvb4wjQBzFPThPkVWeIQ8ZfWkj8GZ8UfWvf44wexEfFDd7J0HESOwSM2RKZTyyOGZaK5U8sjEWskGyNE+GdYcwUct9CLv8it8cBgP4SzWf2dTWCiad2jnIxlngokc4zAZNCSWNThWTHLoHvMcoUiuA4wtI9EKzoy7UEdrZhgrhltwhM3gLx7Ker2HKoM1BbMoR6TOUbdI45Trmj0lsRhnBVv6N3jjY5EJIdd2ZQDjdMvbTTQPb4oBhrqUfnC6GagYXSPML4DjW2BRi1zHDXQ2LjM2TPHfhlAevZuCjDPigJku7vEAff+6jjeD4GiAF2Yorg7yp7aGGIqdsN2scIenF/XS86vnjxsGHxR5zdbOj84M+c3u+f8Mz98OZMTkNKqoOLxe/DxYfnU4+Q+Djp4bjb1ScxbtmQKf8SyuXMi8wM/ycZOMlwWC9LithCLPRjxVwsSezfGqhgpQlRcRNCq2917wFLfLGJJq0LJrEGSqhwMSh08QpkhC4d2F7B0ePzoJwdQ3Y5XOQAJ7TG/S8jtyldivlU0YzH8+Oi5xYe2LljiQR51bYlLWA+mfrCJdkJ505JPQWnE8VB/YFaZudLnU6GOtPDudU7LSCcHHr0GPFL2vjt1lUtwfbW0UtbKE2Tac1Esw2W1JqNUkVqqSAxJuaJ93awDdZs2B8H8ri20Q2L+EwBte5sAfAP0MADt4KLiLxQRbC8t5hPAgAs+04bzOOpldzKzCZ7N/2zOjNlg4/AwM3spJjz8TsJIKUJGrdlGHNXN7GAPM/uv4T8/bq5N4+5ujh7c27/7tz/+3XI5by6NPM4Zf95o+dam5kzUuFXLB3daA6Q0ZyY5KYUl0DkwdQ5NvQmcegpPpioByl5rak6HWa3uQIc4RLXgyC9jhVIDHGpA0xz7gZ1rh7pV6MHw0bxy2EBh8o2PE+KjZm1wVIA0n4XnsFCYlXbf6DfqbvSPm6/tf8PvWPAbGbsnMG0/8GPJ7H9facyU/XtOu/4f -------------------------------------------------------------------------------- /basic_pipeline_extension/assets/diagrams/basic_pipeline_bin.drawio: -------------------------------------------------------------------------------- 1 | 7VzbcqM4EP0a1z45xT34Mc4kk2ztVFLj3ZrLy5YMMmaCESuwY+frRwKJixEGXzB4k7zEaoSQuk+fllqCgXq7WH/GIJh/QTb0Bopkrwfqp4GiKJKmkH9UskkkhqkmAge7diKSM8HEfYNMKDHp0rVhWKgYIeRFblAUWsj3oRUVZABj9FqsNkNe8akBcGBJMLGAV5Z+c+1onkhN5TqTP0DXmfMny8YoubIAvDIbSTgHNnrNidS7gXqLEYqSX4v1LfSo8rhekvvuK66mHcPQj5rcsJzCr3+OxhNnunqytJXnPr5pQzaMMNrwAUObjJ8VEY7myEE+8O4y6RijpW9D2qpESlmdvxAKiFAmwl8wijbMmGAZISKaRwuPXSUdxpvv9P4rnRd/sObiwqd1obRhpTDC6AXeIg/huK+qFP+lV7iB5LjZZGR0OJUKY6IQLbEFd2iJAw9gB7Jbf+HnXzfe5hHdveFvP35K2urx21C+Tu1KHAKiBSTdJzdi6IHIXRU7AhgynbQeu/UGY7DJVQiQ60dhruVnKiAVmJMZ3FeYi8laAQjkR9IiL+W6lolisIiBIxxqqrvzImftRt9zv39kICKlDDa0UESNCBs1UNhh4iZYMIUAPAIcOYPuby/e7xXwluxRA8XwyAjGMxSjaQbYwI3/lpSTxl+IPIQEOlEmo0wOIjCk9wxDbBVumEcRZeUb2lPlnlYJrxyEHA+CwA2vLLQgYiskVe5nYOF61ELihxhO3AF3DTHvJWuRXyuhL8MWBcfr3I3gJEiG9EpCUxFHM9fzcjwCFElSdKaKnHwW/4mYx7gXMg+x7xhgi+F3J9JWEEdwvRMZ7KquF/07jY2vWUCSeZSZ54KRIbUEJlV7p85vNnR+pWfOb16e809c/+Ucvk8cXLEskY/bxtTQjZJHlz3+BD5+rXbo4+I5jyLAzJbeD3Biasd7bv8H6K1g5FqAXZhwB0zApOgUTjoFlF4FKT0BFanKYUV+5oHFWk76I8slQ9twBpYxykRcUSStQxiIFp4hdolVIM5Pa2lDQ+mKRJ7CHDibElfMguPSdouHTXbF3KWUSa56VnzsZLc0m1XEk9m0haRb7KYtjJ9iXnuBVPkVBhjZS4vgVDHAgvKdPw2DpEbc9Snmdck6mbBSSCyE/Kb0SlauAf1pAWzX82uRjM8wPxpJW5Axy9w5ElBnyqfHcOfP63+ePt+ZxuPjFD47D38PH57+HeqVGJpmSi6AKtM2N26CnFhRBCiSbATrsuVvxpU2TMXTSrs2NiRlGMcDFJHJrOoFRtacFQqMXibe1ildK1G6j3wohF4+9LNKp8CfUY8/VYC/U8BPTMxqfeh2iHWDipGm2TQw5dWlCj1VKmVoFpUyKuvEFPmk1Naahevgkoj93sUh7dmSVPgjLJO558aX0Sw2jUW8MmyH08uutMXyiS+qUgN+3w3Zeq87B4KErF4NoJRibz5492DePRYXPBtaJhrRuqk98tX6R749YF9RSOo5+04gsYR9CfR7GOVqFQjsEeVWoyZl14+p7qkptzku+kK5ovXWmSmX5w0Y416XVaKL1gCy3JpS6nVycNq0EnUZEZkp4ATqrsegLlb2priaqtOu2VY4Eym3t9FMCvmpCi6Ty8GO9z8MgN8oH6GI8hGMAjNGTpor5SS6zObXZaQKjL8XXvfhTKVAF1oZ0Gfe0bskQCe4ecI2xK7vjJez2Xm2iW0AzZkQXIZlwulMCK6AZ+rpY0h3GbB2YM5d5A4bKVqbKJRHPcPhceeRqM3OtLGcnWEa7LV7s7XJJAJTeqppT7PXb18n9Wp3rxkZnG7z+qBjTbJRxOb11vG2vaoffQhKqHnjKLQee3pOPgR5DU7PpcjbuZ14HBS1MhTb22PcF3mauRfydldv5/idKMnU83j9CQZg4yFgnylY69C0ywtmcsVUpqrRZCZYmQdoLR6rW1ASxWP5nPHY7ITh3kk8VhvGY+3U8fioxELtAYmzrGmb8ts+a99po4WvcCN+7OZPU5R2+euIjtBDNBCkfLcSLoIcDPBcxydFi2CSngYaU7JxLeDdsAsL17ZjtxTRZ9FVK9jqiHwNj5CCdJghoDG1tWSYIQDtmTOE2yf8epAiFJ0Jv5wcoSFWdwc5QnH/Luok2UeS8Ngk4Skx3HmSUNzb0SUh+p1mCVuAYZdZQmF/01cqP9KExyRrePqq9vVHwWsvOwjiI1FYBmw3b0v2OVPYFHw8CNaDr5OXb/uVK9ypwUsK2/+7ZGELYbnLZKEYaGonLPduw/KoITMmBNBVvlCp3av4SBi+v4ThbmrrPGGo6CWVd/hhlPRaBS317qXac36bpfEHOXrxcRb1xB9nEaO324MJwreapYsGsfCl69Mhe+c7E+2vfCqokhSzz1Ml2Mw+8qXe/QY= -------------------------------------------------------------------------------- /basic_pipeline_extension/assets/images/Illo_basic pipeline extension.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/membraneframework/membrane_tutorials/3e676855eeeab0da495da549133c4886b3608d16/basic_pipeline_extension/assets/images/Illo_basic pipeline extension.png -------------------------------------------------------------------------------- /basic_pipeline_extension/assets/images/basic_pipeline.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/membraneframework/membrane_tutorials/3e676855eeeab0da495da549133c4886b3608d16/basic_pipeline_extension/assets/images/basic_pipeline.png -------------------------------------------------------------------------------- /basic_pipeline_extension/assets/images/basic_pipeline_bin.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/membraneframework/membrane_tutorials/3e676855eeeab0da495da549133c4886b3608d16/basic_pipeline_extension/assets/images/basic_pipeline_bin.png -------------------------------------------------------------------------------- /basic_pipeline_extension/index.md: -------------------------------------------------------------------------------- 1 | --- 2 | title: All you need to know about pipelines pt 2 3 | description: >- 4 | Additional features you might find usefull while setting up your pipelines 5 | part: 3 6 | graphicPath: assets/images/Illo_basic pipeline extension.png 7 | --- 8 | 9 | | number | title | file | 10 | | ------ | ------------- | ------------------ | 11 | | 1 | Introductions | 01_Introduction.md | 12 | | 2 | Bin | 02_Bin.md | 13 | | 3 | Dynamic Pads | 03_DynamicPads.md | 14 | | 4 | Tests | 04_Tests.md | 15 | -------------------------------------------------------------------------------- /broadcasting/01_General_Introduction.md: -------------------------------------------------------------------------------- 1 | We are glad you have decided to continue your journey into the oceans of multimedia with us! 2 | In this tutorial, you will be able to see the Membrane in action - as we will prepare a pipeline responsible for converting 3 | the incoming RTMP stream into an HLS stream, ready to be easily served on the internet. 4 | Later on, we will substitute the front part of our pipeline, so that to make it compatible with another popular streaming protocol - RTSP. 5 | That is how we will try to prove the great dose of flexibility that comes with the use of the Membrane Framework! 6 | 7 | ## General tutorial's structure 8 | As mentioned earlier, the tutorial consists of two parts: 9 | * "RTMP to HLS" - this part describes how to create a pipeline capable of receiving RTMP stream, muxing it into CMAF files, and serving them with the use of HTTP 10 | * "RTSP to HLS" - this part is based on the previous one and describes how to change the pipeline from the previous part so that to make it capable of handling the RTSP stream instead of the RTMP stream. 11 | 12 | As a result of each part, you will have a web application capable of playing media streamed with the use of the appropriate protocols - RTMP and RTSP. 13 | 14 | We strongly encourage you to follow the tutorial part by part, as the different chapters are tightly coupled. However, in case you wanted just to see the result solutions or even one of them, we invite you to take a look at the appropriate directories of the `membrane_demo` repository: 15 | * [RTMP to HLS](https://github.com/membraneframework/membrane_demo/tree/master/rtmp_to_hls) 16 | * [RTSP to HLS](https://github.com/membraneframework/membrane_demo/tree/master/rtsp_to_hls) -------------------------------------------------------------------------------- /broadcasting/02_RTMP_Introduction.md: -------------------------------------------------------------------------------- 1 | ## What do we want to achieve? 2 | 3 | Imagine that we need to build a video streaming platform. That means - we want a particular system user, the streamer, to stream its multimedia on a server, where it will be available for other participants - the viewers. 4 | We need to take advantage of the fact, that there will be only one person streaming, and take into consideration, that there might be multiple, possibly many, viewers. What's the best solution for such a use case? 5 | Well, as the tutorial name suggests - we can use RTMP for the streamer to stream its multimedia to the server, and make them accessible for the viewers by broadcasting them via HLS protocol! 6 | Although it might sound quite complicated, with the Membrane Framework this will be easier than you can expect! 7 | As a final product of this tutorial, we want to have a completely working solution for broadcasting a stream. It will consist of two parts: 8 | 9 | - The pipeline, responsible for receiving an RTMP stream and preparing an HLS stream out of it 10 | - The web player, capable of playing the HLS stream 11 | 12 | The Membrane Framework will be advantageous in the first part where we will create a Membrane's Pipeline for converting the stream. 13 | When it comes to the web player, we will use an existing solution - the [HLS.js](https://github.com/video-dev/hls.js/) player. 14 | 15 | ## Why one would need such a solution? 16 | 17 | You might wonder why one would need to convert an RTMP stream into an HLS stream at all - couldn't we simply make the streamer broadcast its multimedia with the RTMP to all the viewers? 18 | Technically speaking we could...but surprisingly it wouldn't be the easiest solution, since each of the viewers would need to act as an RTMP server. And definitely, it wouldn't be a solution that would scale - RTMP is based on TCP which implies, that there is no broadcast mechanism. It would be the streamer who would need to keep the direct connection to each of the viewers. 19 | In contrast, the solution described above has plenty of advantages - the streamer needs to create a single connection with the RTMP server, and then the multimedia can be shared with the use of a regular HTTP server which is designed to serve multiple clients. 20 | 21 | ## A brief description of the technology we will use 22 | 23 | As stated previously, we will use the preexisting protocols. You might find reading about them beneficial, and that is why we provide you with some links to a brief description of the technology we will be using: 24 | 25 | - [How does RTMP work?](https://blog.stackpath.com/rtmp/) - if you are interested in how the connection in RTMP is established, take your time and read this short description! 26 | - [What is RTMP and why should we care about it?](https://www.wowza.com/blog/rtmp-streaming-real-time-messaging-protocol) - here you can find another short description of RTMP, with the focus laid on the history of the protocol and comparison with other available protocols. Whatsmore, there is a comprehensive explanation of why we need to transcode RTMP to some HTTP-based protocol, just like HLS - which is the use case in our tutorial. 27 | - [HLS behind the scenes](https://www.toptal.com/apple/introduction-to-http-live-streaming-hls) - dig into the ideas which stand behind the HLS, one of the most common HTTP-based streaming protocols. 28 | - [Have you heard about CMAF?](https://www.wowza.com/blog/what-is-cmaf) - if you haven't make sure to read what is the purpose of having a `Common Media Application Format`! 29 | -------------------------------------------------------------------------------- /broadcasting/03_RTMP_SystemArchitecture.md: -------------------------------------------------------------------------------- 1 | As noted previously, our system will be divided into two parts: 2 | 3 | - the server, which is responsible for receiving the RTMP stream, converting it into HLS, and then publishing the created files as an HTTP server 4 | - the client, responsible for playing the incoming HLS stream. 5 | 6 | Considering our point of view, the first part will be more complex, since we will need to prepare the processing pipeline on our own. Let's start with that part. 7 | 8 | ## The server 9 | 10 | Below you can find the desired flow of data in our application server: 11 | ![Pipeline scheme](assets/RTMP_to_HLS_pipeline.drawio.png) 12 | 13 | As you can see, the server architecture consists of two bigger processing units - in Membrane Framework we refer to such units as 'bins'. 14 | The following bins will be used in our solution: 15 | 16 | - RTMP Source Bin - responsible for receiving RTMP stream 17 | - HLS Sink Bin - responsible for "packing" the data into the container format 18 | 19 | Each of these bins consists of some subunits (in Membrane we refer to them as _elements_), responsible for completing an atomic multimedia processing task - i.e. parsing or payloading the incoming stream. 20 | 21 | Let's take a quick walk through the whole processing line and describe more specifically what its given parts are meant to do. 22 | 23 | ### RTMP Source 24 | 25 | The very first element, just at the beginning of the RTMP Source Bin is the RTMP Source. It acts as an RTMP server, listening for the incoming stream. It will be exposing a TCP port, and the streamer will be able to connect and start sending RTMP packets through that channel. 26 | The incoming packets will be demuxed - meaning, that the packets will be unpacked and split, based on the track (video or audio) which data they are transporting. 27 | 28 | ## Parsers 29 | 30 | Buffers containing the given track data will be sent to the appropriate parser - H264 parser for video data and AAC parser for audio data. 31 | 32 | ### H264 Parser 33 | 34 | H264 parser is quite a complex element, designed to read the incoming H264 stream. We have prepared a [separate, supplemental chapter of this tutorial](13_H264_codec.md) to describe the H264 codec and our parser's implementation - we invite you to read it, however, that knowledge is not necessary to successfully run the application. 35 | 36 | ### AAC Parser 37 | 38 | At the same time, parsing happens to the buffers containing audio data. Since the audio track has been encoded with the use of AAC, we need [AAC parser](https://github.com/membraneframework/membrane_aac_plugin) to decode it. 39 | That part of the tutorial does not describe the AAC codec itself, and in case you are interested in digging into that codec, we highly recommend visiting [that page](https://wiki.multimedia.cx/index.php/Understanding_AAC). 40 | 41 | ### HLS converter 42 | 43 | Once data reaches the HLS bin, it needs to be put into the appropriate container files. Since we will be using Common Media Application Format (CMAF) to distribute our media, we need to put all the tracks into the `fragmented MP4` container. The first step is to payload the track's data. Payloading transforms the media encoded with a given codec into a form that is suitable to be put into the container. 44 | The audio track is payloaded within the `Membrane.MP4.Payloader.AAC` module and the video track is payloaded with an H264 payloader, implemented by the `Membrane.MP4.Payloader.H264` module. 45 | The payloaded streams (both the audio stream and the video stream) are then put in the result CMAF file - and the `Membrane.MP4.Muxer.CMAF` is the element responsible for that process. The name 'muxer', relates to the process 46 | of 'muxing' - putting multiple tracks in one file, which is done by that element. Furthermore, the `CMAF muxer`, splits the streams into so-called segments - stream parts of the desired duration. Along that procedure, the manifest files, which contain metadata about the media and names of the appropriate segment file names, are generated. 47 | Finally, the output files: '.m4s' files for each of the segments, as well as manifest files, are written on the disk. 48 | Let's take a look at how do the generated files look like: 49 | ![Pipeline scheme](assets/output_files_structure.png) 50 | As we can see, there are: 51 | 52 | - index.m3u8 - the manifest file which contains metadata about the media, as well as the URI of the manifest files for both the video and audio track. 53 | 54 | ``` 55 | #EXTM3U 56 | #EXT-X-VERSION:7 57 | #EXT-X-INDEPENDENT-SEGMENTS 58 | #EXT-X-MEDIA:TYPE=AUDIO,NAME="audio_default_name",GROUP-ID="audio_default_id",AUTOSELECT=YES,DEFAULT=YES,URI="audio.m3u8" 59 | #EXT-X-STREAM-INF:BANDWIDTH=4507734,CODECS="avc1.42e00a",AUDIO="audio_default_id" 60 | video_g2QABXZpZGVv.m3u8 61 | ``` 62 | 63 | - audio.m3u8 - the manifest file for the audio track. 64 | 65 | ``` 66 | #EXTM3U 67 | #EXT-X-VERSION:7 68 | #EXT-X-TARGETDURATION:8 69 | #EXT-X-MEDIA-SEQUENCE:0 70 | #EXT-X-DISCONTINUITY-SEQUENCE:0 71 | #EXT-X-MAP:URI="audio_header_g2QABWF1ZGlv_part0_.mp4" 72 | #EXTINF:8.0, 73 | audio_segment_9_g2QABWF1ZGlv.m4s 74 | #EXTINF:8.0, 75 | audio_segment_10_g2QABWF1ZGlv.m4s 76 | #EXTINF:8.0, 77 | audio_segment_11_g2QABWF1ZGlv.m4s 78 | ``` 79 | 80 | That file contains some metadata (like the desired duration of the segment), along with the URI pointing to the '.mp4' header file of the fMP4 container, and the list of the segment files, in '.m4s' format. 81 | Each segment is described with `EXTINF` directive, indicating the duration of the segment, in seconds. 82 | 83 | - video\_.m3u8 - the manifest file for the video track. Its structure is similar to the structure of the audio.m3u8 file. However, it is worth noting, that the desired duration of each segment is equal to 10 seconds - and the '.m4s' files are holding a video stream of that length. 84 | 85 | ``` 86 | #EXTM3U 87 | #EXT-X-VERSION:7 88 | #EXT-X-TARGETDURATION:10 89 | #EXT-X-MEDIA-SEQUENCE:0 90 | #EXT-X-DISCONTINUITY-SEQUENCE:0 91 | #EXT-X-MAP:URI="video_header_g2QABXZpZGVv_part0_.mp4" 92 | #EXTINF:10.0, 93 | video_segment_8_g2QABXZpZGVv.m4s 94 | #EXTINF:10.0, 95 | video_segment_9_g2QABXZpZGVv.m4s 96 | ``` 97 | 98 | - audio_header\__part\_.mp4 - a binary file, the header meant to describe the audio track in the format required by the fragmented MP4 container. 99 | - video_header\_\_part\.mp4 - a binary file, the header meant to describe the video track in the format required by the fragmented MP4 container. 100 | - audio_segment\_\\_\<>.m4s - particular segments containing fragments of the audio stream. 101 | - video_segment\_\\_\<>.m4s - particular segments containing fragments of the video stream. 102 | 103 | ### HTTP server 104 | 105 | The HTTP server sends the requested files to the clients. Its implementation is based on the API provided by the Phoenix Framework. 106 | 107 | ## The client 108 | 109 | When it comes to our application client, we will be using the `Hls.js` library, available in the javascript ecosystem. 110 | We won't dig into the details of how the client application media player is implemented, however, we would like to point out some steps which take place while playing the media. 111 | First, the client needs to request the master manifest file. Based on that file, the client asks for the appropriate audio and video track manifest files. 112 | With these files, the client knows the MP4 header files for both the tracks, as well as knows the filenames of the particular segments, along with their duration. The client downloads the MP4 header file and starts downloading the appropriate segment files, based on which part of the media should be played in the nearest future. 113 | -------------------------------------------------------------------------------- /broadcasting/04_RTMP_RunningTheDemo.md: -------------------------------------------------------------------------------- 1 | This time we won't create the application from the scratch nor will we be based on some kind of template. 2 | Since most of the application's code is media-agnostic, we don't see a point in focusing on it. Instead, we will indicate the most crucial parts of the 3 | [RTMP to HLS demo](https://github.com/membraneframework/membrane_demo/tree/master/rtmp_to_hls), one of many publicly available [Membrane demos](https://github.com/membraneframework/membrane_demo/). 4 | The Membrane demos are supplementing the Membrane Tutorials on providing a source of knowledge in the field of multimedia processing - we warmly invite you to look at them as you may find some interesting 5 | use cases of Membrane in real-life scenarios. 6 | 7 | ## Running the application 8 | 9 | In order to run the demo, you need to clone the Membrane demos repository first: 10 | 11 | ```console 12 | git clone https://github.com/membraneframework/membrane_demo 13 | cd membrane_demo/rtmp_to_hls 14 | ``` 15 | 16 | Once in the project directory, you need to get the dependencies of the project: 17 | 18 | ```console 19 | mix deps.get 20 | ``` 21 | 22 | Finally, you can run the application with the following command: 23 | 24 | ```console 25 | mix phx.server 26 | ``` 27 | 28 | The server will be waiting for an RTMP stream on localhost:9009, and the client of the application will be available on localhost:4000. 29 | 30 | ## Exemplary stream generation with OBS 31 | 32 | You can send RTMP stream onto `localhost:9006` with your favorite streaming tool. As an example, we will show you how to generate an RTMP stream with 33 | [OBS](https://obsproject.com). 34 | Once you have OBS installed, you can perform the following steps: 35 | 36 | 1. Open the OBS application 37 | 1. Open the `Settings` windows 38 | 1. Go to the `Stream` tab and set the value in the `Server` field to: `rtmp://localhost:9006` (the address where the server is waiting for the stream) 39 | 1. Finally, you can go back to the main window and start streaming with the `Start Streaming` button. 40 | 41 | Below you can see how to set the appropriate settings (steps 2) and 3) from the list of steps above): 42 | ![OBS settings](assets/OBS_settings.webp) 43 | 44 | Once the streaming has started, you can visit `localhost:4000`, where the client application should be available. After a few seconds, you should be able to play 45 | the video you are streaming. Most likely, a kind of latency will occur and the stream played by the client application will be delayed. That latency can reach values as great as 10 seconds, which is the result 46 | of a relatively complex processing taking place in the pipeline. Note, that the latency here is not a result of the packets traveling far via the network, as the packets do not leave our computer in that setup - in the case of a real streaming server, we would need to take into consideration the time it takes a packet to reach the destination. 47 | -------------------------------------------------------------------------------- /broadcasting/05_RTMP_Pipeline.md: -------------------------------------------------------------------------------- 1 | In this chapter, we will discuss the multimedia-specific part of the application. 2 | 3 | ## The pipeline 4 | 5 | Let's start with `lib/rtmp_to_hls/pipeline.ex` file. All the logic is put inside the [`Membrane.Pipeline.handle_init/1`](https://hexdocs.pm/membrane_core/Membrane.Pipeline.html#c:handle_init/1) callback, 6 | which is invoked once the pipeline is initialized. 7 | 8 | **_`lib/rtmp_to_hls/pipeline.ex`_** 9 | 10 | ```elixir 11 | @impl true 12 | def handle_init(_opts) do 13 | ... 14 | children: %{ 15 | src: %Membrane.RTMP.SourceBin{port: 9009}, 16 | sink: %Membrane.HTTPAdaptiveStream.SinkBin{ 17 | manifest_module: Membrane.HTTPAdaptiveStream.HLS, 18 | target_window_duration: 20 |> Membrane.Time.seconds(), 19 | muxer_segment_duration: 8 |> Membrane.Time.seconds(), 20 | storage: %Membrane.HTTPAdaptiveStream.Storages.FileStorage{directory: "output"} 21 | } 22 | }, 23 | ... 24 | end 25 | ``` 26 | 27 | First, we define the list of children. The following children are defined: 28 | 29 | - `:src` - a `Membrane.RTMP.SourceBin`, an RTMP server, which, according to its `:port` configuration, will be listening on port `9009`. This bin will be acting as a source for our pipeline. For more information on RTMP Source Bin please visit [the documentation](https://hexdocs.pm/membrane_rtmp_plugin/Membrane.RTMP.SourceBin.html). 30 | - `:sink` - a `Membrane.HTTPAdaptiveStream.SinkBin`, acting as a sink of the pipeline. The full documentation of that bin is available [here](https://hexdocs.pm/membrane_http_adaptive_stream_plugin/Membrane.HTTPAdaptiveStream.SinkBin.html). We need to specify some of its options: 31 | - `:manifest_module` - a module which implements [`Membrane.HTTPAdaptiveStream.Manifest`](https://hexdocs.pm/membrane_http_adaptive_stream_plugin/Membrane.HTTPAdaptiveStream.Manifest.html#c:serialize/1) behavior. A manifest allows aggregate tracks (of a different type, i.e. an audio track and a video track as well as many tracks of the same type, i.e. a few video tracks with different resolutions). For each track, the manifest holds a reference to a list of segments, which form that track. Furthermore, the manifest module is equipped with the `serialize/1` method, which allows transforming that manifest to a string (which later on can be written to a file). In that case, we use a built-in implementation of a manifest module - the `Membrane.HTTPAdaptiveStream.HLS`, designed to serialize a manifest into a form required by HLS. 32 | - `:target_window_duriation` - that value determines the minimal manifest's duration. The oldest segments of the tracks will be removed whenever possible if persisting them would result in exceeding the manifest duration. 33 | - `:muxer_segment_duration` - the maximal duration of a segment. Each segment of each track shouldn't exceed that value. In our case, we have decided to limit the length of each segment to 8 seconds. 34 | - `:storage` - the sink element, the module responsible for writing down the HLS playlist and manifest files. In our case, we use a pre-implemented `Membrane.HTTPAdaptiveStream.FileStorage` module, designed to write the files to the local filesystem. We configure it so that the directory where the files will be put in the `output/` directory (make sure that that directory exists as the storage module won't create it itself). 35 | 36 | The fact that the configuration of a pipeline, which performs relatively complex processing, consists of just two elements, proves the power of [bins](/basic_pipeline_extension/02_Bin.md). Feel free to stop for a moment and read about them if you haven't done it yet. 37 | 38 | After providing the children's specifications, we are ready to connect the pads between these children. Take a look at that part of the code: 39 | **_`lib/rtmp_to_hls/pipeline.ex`_** 40 | 41 | ```elixir 42 | @impl true 43 | def handle_init(_opts) do 44 | ... 45 | links: [ 46 | link(:src) 47 | |> via_out(:audio) 48 | |> via_in(Pad.ref(:input, :audio), options: [encoding: :AAC]) 49 | |> to(:sink), 50 | link(:src) 51 | |> via_out(:video) 52 | |> via_in(Pad.ref(:input, :video), options: [encoding: :H264]) 53 | |> to(:sink) 54 | ] 55 | ... 56 | end 57 | ``` 58 | 59 | The structure of links reflects the desired architecture of the application, described in the [chapter about system architecture](../videoroom/3_SystemArchitecture.md). 60 | `:src` has two output pads: the `:audio` pad and the `:video` pad, transferring the appropriate media tracks. 61 | The source's `:audio` pad is linked to the input `:audio` pad of the sink - along with the `:encoding` option. That option is an atom, describing the codec which is used to encode the media data - when it comes to audio data, 62 | we will be using AAC coded. 63 | At the time of the writing, only `:H264` and `:AAC` codecs are available to be passed as an `:encoding` option - the first one is used with video data, and the second one is used with audio data. 64 | By analogy, the source's `:video` pad is linked with the sink's `:video` pad - and the `:encoding` to be used is H264. 65 | 66 | The final thing that is done in the `handle_init/1` callback's implementation is returning the desired actions: 67 | **_`lib/rtmp_to_hls/pipeline.ex`_** 68 | 69 | ```elixir 70 | @impl true 71 | def handle_init(_opts) do 72 | ... 73 | { {:ok, spec: spec, playback: :playing}, %{} } 74 | end 75 | ``` 76 | 77 | The first action is the `:spec` action, which spawns the children. The second action changes the playback state of the pipeline into the `:playing` - meaning, that data can start flowing through the pipeline. 78 | 79 | ## Starting the pipeline 80 | 81 | The pipeline is started with `Supervisor.start_link`, as a child of the application, inside the `lib/rtmp_to_hls/application.ex` file: 82 | 83 | **_`lib/rtmp_to_hls/application.ex`_** 84 | 85 | ```elixir 86 | @impl true 87 | def start(_type, _args) do 88 | children = [ 89 | # Start the Pipeline 90 | Membrane.Demo.RtmpToHls, 91 | ... 92 | ] 93 | opts = [strategy: :one_for_one, name: RtmpToHls.Supervisor] 94 | Supervisor.start_link(children, opts) 95 | end 96 | ``` 97 | 98 | ## HLS controller 99 | 100 | The files produced with the pipeline are written down to the `output/` directory. We need to make them accessible via HTTP. 101 | The Phoenix Framework provides tools to achieve that - take a look at the `RtmpToHlsWeb.Router`: 102 | **_`lib/rtmp_to_hls_web/router.ex`_** 103 | 104 | ```elixir 105 | scope "/", RtmpToHlsWeb do 106 | pipe_through :browser 107 | 108 | get "/", PageController, :index 109 | get "/video/:filename", HlsController, :index 110 | end 111 | ``` 112 | 113 | We are directing HTTP requests on `/video/:filename` to the HlsController, whose implementation is shown below: 114 | **_`lib/rtmp_to_hls_web/controllers/hls_controller.ex`_** 115 | 116 | ```elixir 117 | defmodule RtmpToHlsWeb.HlsController do 118 | use RtmpToHlsWeb, :controller 119 | 120 | alias Plug 121 | 122 | def index(conn, %{"filename" => filename}) do 123 | path = "output/#{filename}" 124 | 125 | if File.exists?(path) do 126 | conn |> Plug.Conn.send_file(200, path) 127 | else 128 | conn |> Plug.Conn.send_resp(404, "File not found") 129 | end 130 | end 131 | end 132 | ``` 133 | 134 | This part of the code is responsible for sending the file `output/` to the HTTP request sender. 135 | -------------------------------------------------------------------------------- /broadcasting/06_WebPlayer.md: -------------------------------------------------------------------------------- 1 | In this really short chapter, we will take a look at how one can play the video delivered with HLS on the website. 2 | 3 | In our demo, we are taking advantage of the HLS.js player. 4 | With such a tool onboard, the creation of the player is simple as that: 5 | **_`lib/rtmp_to_hls_web/templates/page/index.html.heex`_** 6 | 7 | ```js 8 | 9 | 10 |
11 | 12 |
13 | 24 | ``` 25 | 26 | First, we load a `hls.js` script. 27 | Then, in the DOM we add a div holding a video element of a class `Player`. 28 | Finally, we add our custom script, which decorates the video player element with the functionalities 29 | provided by the `hls.js` library. 30 | The `videoSrc` is a URL of the manifest file of the HLS playlist. As described in the previous chapter, the files from the playlist are available 31 | at `/video/`. 32 | If the HLS is supported by the client's browser, we create an object of type `Hls` (that class is a part of the `hls.js` library), then we set its source manifest file and a DOM element that acts as a video player. For more options that can be specified for the player, see the [documentation](https://github.com/video-dev/hls.js/blob/master/docs/API.md). 33 | -------------------------------------------------------------------------------- /broadcasting/07_RTSP_Introduction.md: -------------------------------------------------------------------------------- 1 | # Introduction 2 | 3 | Other than RTMP, another protocol which is sometimes used for streaming media to the server is RTSP. It isn't so popular today anymore, however, there are still areas where it is still in use, primarily by surveillance cameras. 4 | In the next couple chapters, we would like to show you a modification of the current RTMP to HLS converter, that will allow us to convert an RTSP stream instead. 5 | 6 | ## Use case 7 | 8 | Suppose that we have a surveillance camera, which is providing us with an RTSP stream. We would like to stream this video to multiple viewers, for them to be easily accessible in the web browser. 9 | 10 | Note, that we want this solution to be scalable as the number of users can be quite big. 11 | 12 | ## Solution 13 | 14 | The reasons for converting RTSP to HLS are very similar to the ones we explained with RTMP, as RTSP comes with similar shortcomings as RTMP. RTSP is very rarely supported on playback devices, has problems with traversing firewalls and proxies and doesn't support adaptive bitrate. 15 | 16 | ## What technology we will use 17 | 18 | Before going further, make sure you have some basic understanding of the protocols used: 19 | 20 | [Real Time Streaming Protocol (RTSP)](https://antmedia.io/rtsp-explained-what-is-rtsp-how-it-works) - what RTSP is and how does it work? 21 | 22 | [Real-time Transport Protocol (RTP) and RTP Control Protocol (RTCP)](https://www.techtarget.com/searchnetworking/definition/Real-Time-Transport-Protocol) are protocols used for delivering audio and video over the internet. While the RTP transports the data, the RTCP is responsible for QoS and synchronization of multiple streams. -------------------------------------------------------------------------------- /broadcasting/08_RTSP_Architecture.md: -------------------------------------------------------------------------------- 1 | # Architecture 2 | 3 | Now let's discuss how the architecture of our solution will look like. 4 | It will be a little different from the RTMP to HLS architecture. 5 | The main component will be the pipeline, which will ingest RTP stream and convert it to HLS. Beyond that we will also need a Connection Manager, which will be responsible for establishing an RTSP connection with the server. 6 | 7 | ![image](assets/rtsp_architecture.drawio.png) 8 | 9 | When initializing, the pipeline will start a Connection Manager which starts an RTSP connection with the server. Once the connection is fully established, the pipeline will be notified. 10 | 11 | Let's take a closer look on each of those components: 12 | 13 | ## Connection Manager 14 | The role of the connection manager is to initialize RTSP session and start playing the stream. 15 | It communicates with the server using the [RTSP requests](https://antmedia.io/rtsp-explained-what-is-rtsp-how-it-works/#RTSP_requests). In fact, we won't need many requests to start playing the stream - take a look at the desired message flow: 16 | 17 | ![image](assets/connection_manager.drawio.png) 18 | 19 | First we want to get the details of the video we will be playing, by sending the `DESCRIBE` method. 20 | Then we call the `SETUP` method, defining the transport protocol (RTP) and client port used for receiving the stream. 21 | Now we can start the stream using `PLAY` method. 22 | 23 | ## Pipeline 24 | 25 | The pipeline consists of a couple elements, each of them performing a specific media processing task. You can definitely notice some similarities to the pipeline described in the [RTMP architecture](02_RTMP_Introduction.md). However, we will only be processing video so only the video processing elements will be necessary. 26 | 27 | ![image](assets/rtsp_pipeline.drawio.png) 28 | 29 | We have already used the, `H264 Parser`, `MP4 H264 Payloader`, `CMAF Muxer` and `HLS Sink` elements in the RTMP pipeline, take a look at the [RTMP to HLS architecture](03_RTMP_SystemArchitecture.md) chapter for details of the purpose of those elements. 30 | 31 | Let us describe briefly what is the purpose of the other components: 32 | 33 | ### UDP Source 34 | This element is quite simple - it receives UDP packets from the network and sends their payloads to the next element. 35 | 36 | ### RTP SessionBin 37 | RTP SessionBin is a Membrane's Bin, which is a Membrane's container used for creating reusable groups of elements. In our case the Bin handles the RTP session with the server, which has been set up by the Connection Manager. 38 | -------------------------------------------------------------------------------- /broadcasting/09_RTSP_RunningDemo.md: -------------------------------------------------------------------------------- 1 | In the tutorial we won't explain how to implement the solution from the ground up - instead, we will run the existing code from [Membrane demos](https://github.com/membraneframework/membrane_demo). 2 | 3 | To run the RTSP to HLS converter first clone the demos repo: 4 | ```console 5 | git clone https://github.com/membraneframework/membrane_demo.git 6 | ``` 7 | 8 | ```console 9 | cd membrane_demo/rtsp_to_hls 10 | ``` 11 | 12 | Install the dependencies 13 | ```console 14 | mix deps.get 15 | ``` 16 | 17 | Make sure you have those libraries installed as well: 18 | - gcc 19 | - libc-dev 20 | - ffmpeg 21 | 22 | On ubuntu: 23 | ```console 24 | apt-get install gcc libc-dev ffmpeg 25 | ``` 26 | 27 | Take a look inside the `lib/application.ex` file. It's responsible for starting the pipeline. 28 | We need to give a few arguments to the pipeline: 29 | ```elixir 30 | @rtsp_stream_url "rtsp://rtsp.membrane.work:554/testsrc.264" 31 | @output_path "hls_output" 32 | @rtp_port 20000 33 | ``` 34 | 35 | The `@output_path` attribute defines the storage directory for hls files and the `@rtp_port` defines on which port we will be expecting the rtp stream, once the RTSP connection is established. 36 | 37 | The `@rtsp_stream_url` attribute contains the address of the stream, which we will be converting. It is a sample stream prepared for the purpose of the demo. 38 | 39 | Now we can start the application: 40 | ```console 41 | mix run --no-halt 42 | ``` 43 | 44 | The pipeline will start playing, after a couple of seconds the HLS files should appear in the `@output_path` directory. In order to play the stream we need to first serve them. We can do it using simple python server. 45 | 46 | ```console 47 | python3 -m http.server 8000 48 | ``` 49 | 50 | Then we can play the stream using [ffmpeg](https://ffmpeg.org/), by pointing to the location of the manifest file: 51 | ```console 52 | ffplay http://YOUR_MACHINE_IP:8000/rtsp_to_hls/hls_output/index.m3u8 53 | ``` 54 | -------------------------------------------------------------------------------- /broadcasting/10_ConnectionManager.md: -------------------------------------------------------------------------------- 1 | Now let's focus on how the Connection Manager works. As mentioned previously its role is to establish the RTSP connection with the RTSP server. 2 | 3 | The `ConnectionManager` module will use the [`Connection`](https://hexdocs.pm/connection/Connection.html) behaviour, which provides additional callbacks to [`GenServer`](https://hexdocs.pm/elixir/GenServer.html) behaviour, aiding with building a connection process. 4 | 5 | First of all we are defining the `ConnectionStatus` struct, which we will use to keep the state of the ConnectionManager: 6 | 7 | ##### lib/connection_manager.ex 8 | ```elixir 9 | defmodule ConnectionStatus do 10 | @moduledoc false 11 | @type t :: %__MODULE__{ 12 | stream_url: binary(), 13 | rtsp_session: pid(), 14 | pipeline: pid(), 15 | keep_alive: pid(), 16 | pipeline_options: keyword() 17 | } 18 | 19 | @enforce_keys [ 20 | :stream_url, 21 | :pipeline, 22 | :pipeline_options 23 | ] 24 | 25 | defstruct @enforce_keys ++ 26 | [ 27 | :rtsp_session, 28 | :keep_alive 29 | ] 30 | end 31 | ``` 32 | 33 | It holds the `rtsp_session`, which is the pid of a process started with `Membrane.RTSP.start/1`. The `Membrane.RTSP` allows us to execute RTSP client commands. You can read more about it [here](https://hexdocs.pm/membrane_rtsp/readme.html). 34 | The `pipeline` field is the pid of the pipeline, we will need it to notify the pipeline, that the RTSP connection is ready. In such notification we send `pipeline_options`, which contain necessary information about the stream. 35 | The `keep_alive` is a process which repeatedly pings the RTSP server, in order to keep the connection alive and prevent a timeout. 36 | 37 | Let's take a look at the `connect/2` callback, which is called immediately after the `init/1`: 38 | 39 | ##### lib/connection_manager.ex 40 | ```elixir 41 | def connect(_info, %ConnectionStatus{} = connection_status) do 42 | rtsp_session = start_rtsp_session(connection_status) 43 | connection_status = %{connection_status | rtsp_session: rtsp_session} 44 | 45 | if is_nil(rtsp_session) do 46 | {:backoff, @delay, connection_status} 47 | else 48 | with {:ok, connection_status} <- get_rtsp_description(connection_status), 49 | :ok <- setup_rtsp_connection(connection_status), 50 | {:ok, connection_status} <- start_keep_alive(connection_status), 51 | :ok <- play(connection_status) do 52 | 53 | send( 54 | connection_status.pipeline, 55 | {:rtsp_setup_complete, connection_status.pipeline_options} 56 | ) 57 | 58 | {:ok, connection_status} 59 | else 60 | {:error, error_message} -> 61 | {:backoff, @delay, connection_status} 62 | end 63 | end 64 | end 65 | ``` 66 | 67 | In the callback we go through the whole process of establishing RTSP connection - first starting the RTSP session, then getting the video parameters with, setting up the session, starting the keep alive process and finally playing the stream. 68 | If all those steps succeed we can notify the pipeline, otherwise we back off and try to set up the connection after a `@delay` amount of time. 69 | 70 | What might seem unclear to you is the `get_sps_pps` function. 71 | It is responsible for getting the [sps and pps](https://www.cardinalpeak.com/blog/the-h-264-sequence-parameter-set) parameters from the RTSP DESCRIBE method. In short, sps and pps are parameters used by the H.264 codec and are required to decode the stream. Once the RTSP connection is complete we are sending them to the pipeline. -------------------------------------------------------------------------------- /broadcasting/11_RTSP_Pipeline.md: -------------------------------------------------------------------------------- 1 | As explained in the [Architecture chapter](07_RTSP_Architecture.md), the pipeline will consist of a couple of elements, that will be processing the RTP stream. 2 | 3 | The flow of the pipeline will consist of three steps. First, when the pipeline is initialized we will start the Connection Manager, which will set up the RTP stream via the RTSP. 4 | Once that is finished, we will set up two initial elements in the pipeline - the `UDP Source` and `RTP SessionBin`, which will allow us to receive RTP packets and process them. 5 | When the SessionBin detects that the RTP stream has been started, it will notify the pipeline with the `:new_rtp_stream` notification. Later on, we will add the remaining elements to the pipeline, allowing for the whole conversion process to take place. 6 | 7 | Those steps take place, respectively, in the: `handle_init/1`, `handle_other/3` and `handle_notification/4` callbacks. While the `handle_init/1` is rather intuitive, we will describe in detail what's happening in the other callbacks. 8 | 9 | Let us explain what's going on in the `handle_other` callback: 10 | 11 | ##### lib/pipeline.ex 12 | ```elixir 13 | @impl true 14 | def handle_other({:rtsp_setup_complete, options}, _ctx, state) do 15 | children = %{ 16 | app_source: %Membrane.UDP.Source{ 17 | local_port_no: state[:port], 18 | recv_buffer_size: 500_000 19 | }, 20 | rtp: %Membrane.RTP.SessionBin{ 21 | fmt_mapping: %{96 => {:H264, 90_000}} 22 | }, 23 | hls: %Membrane.HTTPAdaptiveStream.Sink{ 24 | manifest_module: Membrane.HTTPAdaptiveStream.HLS, 25 | target_window_duration: 120 |> Membrane.Time.seconds(), 26 | target_segment_duration: 4 |> Membrane.Time.seconds(), 27 | storage: %Membrane.HTTPAdaptiveStream.Storages.FileStorage{ 28 | directory: state[:output_path] 29 | } 30 | } 31 | } 32 | 33 | links = [ 34 | link(:app_source) 35 | |> via_in(Pad.ref(:rtp_input, make_ref())) 36 | |> to(:rtp) 37 | ] 38 | 39 | spec = %ParentSpec{children: children, links: links} 40 | { {:ok, spec: spec}, %{state | video: %{sps: options[:sps], pps: options[:pps]}} } 41 | end 42 | ``` 43 | 44 | When we receive the `rtsp_setup_complete` message, we first define the new children for the pipeline, and links between them - the UDP Source and the RTP SessionBin. We also create the HLS Sink, however we won't be linking it just yet. With the message we receive the sps and pps inside the options, and we add them to the pipeline's state. 45 | 46 | Only after we receive the `:new_rtp_stream` notification we add the rest of the elements and link them with each other: 47 | 48 | ##### lib/pipeline.ex 49 | ```elixir 50 | @impl true 51 | def handle_notification({:new_rtp_stream, ssrc, 96, _extensions}, :rtp, _ctx, state) do 52 | actions = 53 | if Map.has_key?(state, :rtp_started) do 54 | [] 55 | else 56 | children = %{ 57 | video_nal_parser: %Membrane.H264.FFmpeg.Parser{ 58 | sps: state.video.sps, 59 | pps: state.video.pps, 60 | skip_until_keyframe?: true, 61 | framerate: {30, 1}, 62 | alignment: :au, 63 | attach_nalus?: true 64 | }, 65 | video_payloader: Membrane.MP4.Payloader.H264, 66 | video_cmaf_muxer: Membrane.MP4.Muxer.CMAF 67 | } 68 | 69 | links = [ 70 | link(:rtp) 71 | |> via_out(Pad.ref(:output, ssrc), 72 | options: [depayloader: Membrane.RTP.H264.Depayloader] 73 | ) 74 | |> to(:video_nal_parser) 75 | |> to(:video_payloader) 76 | |> to(:video_cmaf_muxer) 77 | |> via_in(:input) 78 | |> to(:hls) 79 | ] 80 | 81 | [spec: %ParentSpec{children: children, links: links}] 82 | end 83 | 84 | { {:ok, actions}, Map.put(state, :rtp_started, true) } 85 | end 86 | ``` 87 | 88 | First we check, if the stream hasn't started yet. That's because if we are restarting the pipeline there might be a previous RTP stream still being sent, so we might receive the `:new_rtp_stream` notification twice - once for the old and then for the new stream. We want to ignore any notification after the first one, as we want only a single copy of each media processing element. 89 | Notice the sps and pps being passed to the H264 parser - they are necessary for decoding the stream. 90 | -------------------------------------------------------------------------------- /broadcasting/12_Summary.md: -------------------------------------------------------------------------------- 1 | As you can see converting an RTMP or RTSP stream to HLS might not be the easiest task. However, some understanding of the protocols and the usage of a couple of prefabricated elements in the Membrane Framework make up a rather compact solution. Importantly, such solution is open for modification and easily extensible. 2 | 3 | HLS is one of the most popular media streaming protocols. Its advantages are using HTTP, allowing it for traversing over firewalls. It also allows for adaptive bitrate, i.e. it changes the resolution of the video based on the network bandwidth. Unlike RTMP or RTSP it is highly scalable. You will often come across HLS in the media streaming industry, so it's useful to have some understanding of it. 4 | 5 | We hope that you grasped the basic ideas of the RTMP and RTSP protocols as well as that you know how to create your 'to HLS' converter with the use of the Membrane Framework. 6 | 7 | If you liked this tutorial and want to extend your knowledge of the Membrane Framework, make sure to check out [other tutorials](https://membrane.stream/learn). 8 | -------------------------------------------------------------------------------- /broadcasting/13_H264_codec.md: -------------------------------------------------------------------------------- 1 | In this chapter you will learn about the H264 codec and how its processing is done with the Membrane Framework plugin - the [H264 plugin](https://github.com/membraneframework/membrane_h264_ffmpeg_plugin). 2 | 3 | In H264 we can distinguish two layers of abstraction - the video coding layer (VCL), focused on the visual representation of the video, and the network abstraction layer (NAL), focused on how to structurize the stream so that it can be sent via the network. 4 | 5 | When it comes to VCL, a video encoded with the H264 codec is represented as a sequence of pictures. Each picture consists of many **macroblocks**. 6 | The macroblock is simply a part of the picture (in the context of some spatial dependence) - i.e. the top left corner of the picture. 7 | The macroblocks are grouped together into so-called **slices**. Each slice consists of some macroblocks, which share the same **slice header**, containing some metadata common for all these macroblocks. 8 | 9 | Each slice (which in fact is a part of a video picture), can be packed into a single 10 | **NALu** (*Network Abstraction Layer units*). As the name suggests, here is where we start to deal with NAL. NALu is just an atomic piece of information sent through the network - it might contain some visual data (i.e. a list of macroblocks forming a part of a video frame), but it is not limited to the visual data - within NALus metadata used to properly decode the stream is also sent. We can distinguish two types of NAL units: 11 | 12 | - VCL NALus - which stands for "Video coding layer" NALus 13 | - Non-VCL NALus - which stands for "Non-video coding layer" NALus 14 | 15 | There are different types of both VCL and Non-VCL units - for more information on them you can refer [here](https://yumichan.net/video-processing/video-compression/introduction-to-h264-nal-unit/) 16 | > **Parse H264 stream on your own!** 17 | > 18 | > With the use of Membrane Framework, you can inspect the types of NALus in your h264 file. To do so, you need to clone the H264 parser repository with: 19 | > 20 | > ```console 21 | > git clone https://github.com/membraneframework/membrane_h264_ffmepg_plugin 22 | > ``` 23 | > 24 | > Later on, inside the repository's directory, you can launch the Elixir's interactive shell, with compiled modules from the repository's mix project, by typing: 25 | > 26 | > ```console 27 | > iex -S mix 28 | > ``` 29 | > 30 | > Once in iex, you can do the following thing: 31 | > 32 | > ```elixir 33 | > alias Membrane.H264.FFmpeg.Parser.NALu 34 | > 35 | > # Of course you can read your own file, here is just an example file from the membrane_rtmp_plugin's test directory, 36 | > # available at: https://github.com/membraneframework/membrane_rtmp_plugin/tree/master/test/fixtures/testvideo.h264 37 | > binaries = File.read!("test/fixtures/testvideo.h264") 38 | > 39 | > NALu.parse(binaries) 40 | > ``` 41 | > 42 | > You should see the following response: 43 | > 44 | > ```elixir 45 | > {[ 46 | > %{ 47 | > metadata: %{h264: %{new_access_unit: %{key_frame?: true}, type: :sps}}, 48 | > prefixed_poslen: {0, 29}, 49 | > unprefixed_poslen: {4, 25} 50 | > }, 51 | > %{ 52 | > metadata: %{h264: %{type: :pps}}, 53 | > prefixed_poslen: {29, 9}, 54 | > unprefixed_poslen: {33, 5} 55 | > }, 56 | > %{ 57 | > metadata: %{h264: %{type: :sei}}, 58 | > prefixed_poslen: {38, 690}, 59 | > unprefixed_poslen: {41, 687} 60 | > }, 61 | > %{ 62 | > metadata: %{h264: %{type: :idr}}, 63 | > prefixed_poslen: {728, 8284}, 64 | > unprefixed_poslen: {731, 8281} 65 | > }, 66 | > %{ 67 | > metadata: %{h264: %{type: :non_idr}}, 68 | > prefixed_poslen: {9012, 1536}, 69 | > unprefixed_poslen: {9016, 1532} 70 | > }, 71 | > .... 72 | > ]} 73 | > ``` 74 | > 75 | > As you can see, NALus of different types have appeared, just like: 76 | > 77 | > - SPS - *Sequence Parameter Set*, a Non-VCL NALu, with a set of parameters that rarely change, applicable to the series of consecutive coded video pictures, called the **coded video sequence**. Each coded video sequence can be decoded independently of any other coded video sequence. 78 | > - PPS - *Picture parameter Set* - a set of parameters that are applicable to some pictures from the coded video sequence. Furthermore, inside a PPS NALu there is a reference to SPS. 79 | > - SEI - *Supplemental Enhancement Information* - some additional information that enhances the usability of the decoded video, i.e. timing information. 80 | > - IDR - *Instantaneous Decoding Refresh*, a VCL unit containing the I-frame (known also as `intra frame`) - a picture that can be decoded without knowledge of any other frame, in contrast to P-frames and B-frames, which might need previously presented frames or frames that need to be presented in the future. As you might guess, the I-frames size is much greater than P-frame or B-frame size - that is because the whole information about the content of the picture needs to be encoded in such a frame. 81 | > - NON_IDR - *Non-Instantaneous Decoding Refresh* - a VCL unit containing a P-frame or B-frame or parts of such a non-key frame. Note the size of a Non-IDR NALu (1536 B), compared to the size of IDR NALu (8284 B). 82 | 83 | The sequence of NALus of a special form creates an **Access Unit**. 84 | Each access unit holds a single picture of the video. 85 | When such a picture is the keyframe, we refer to it as a **IDR Access Unit**. Otherwise, we call it **Non-IDR Access Unit**. Of course, the IDR Access Units are much bigger than Non-IDR Access Units when it comes to their binary size. 86 | Sometimes, for the convenience of the decoding, the access units are separated with the use of another Non-VCL NALu, called *AUD* (**Access Unit Delimeter**). 87 | Below you can find a diagram showing the structure of an access unit: 88 | ![Access Unit structure](assets/au_structure.png) 89 | The access unit consists of the **primary coded picture** - a set of macroblocks representing the picture, and optionally of the **redundant coded picture** - the set of macroblocks that hold the redundant information about the given areas on the picture. 90 | A parser needs to be aware that some of the NALus are optional are might not appear in the stream. 91 | 92 | The existence of a coded video sequence is determined by the presence of an IDR NALu in the first access unit. Each coded video sequence can be decoded independently of the other coded video sequences. 93 | 94 | As a summary we would like to present a diagram showing an exemplary NALus stream, along with the structures we can distinguish in that stream: 95 | ![H264 NALus stream](assets/h264_structure.png) 96 | -------------------------------------------------------------------------------- /broadcasting/assets/OBS_settings.webp: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/membraneframework/membrane_tutorials/3e676855eeeab0da495da549133c4886b3608d16/broadcasting/assets/OBS_settings.webp -------------------------------------------------------------------------------- /broadcasting/assets/RTMP_to_HLS_pipeline.drawio.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/membraneframework/membrane_tutorials/3e676855eeeab0da495da549133c4886b3608d16/broadcasting/assets/RTMP_to_HLS_pipeline.drawio.png -------------------------------------------------------------------------------- /broadcasting/assets/au_structure.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/membraneframework/membrane_tutorials/3e676855eeeab0da495da549133c4886b3608d16/broadcasting/assets/au_structure.png -------------------------------------------------------------------------------- /broadcasting/assets/connection_manager.drawio.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/membraneframework/membrane_tutorials/3e676855eeeab0da495da549133c4886b3608d16/broadcasting/assets/connection_manager.drawio.png -------------------------------------------------------------------------------- /broadcasting/assets/h264_structure.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/membraneframework/membrane_tutorials/3e676855eeeab0da495da549133c4886b3608d16/broadcasting/assets/h264_structure.png -------------------------------------------------------------------------------- /broadcasting/assets/output_files_structure.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/membraneframework/membrane_tutorials/3e676855eeeab0da495da549133c4886b3608d16/broadcasting/assets/output_files_structure.png -------------------------------------------------------------------------------- /broadcasting/assets/rtsp_architecture.drawio.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/membraneframework/membrane_tutorials/3e676855eeeab0da495da549133c4886b3608d16/broadcasting/assets/rtsp_architecture.drawio.png -------------------------------------------------------------------------------- /broadcasting/assets/rtsp_pipeline.drawio.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/membraneframework/membrane_tutorials/3e676855eeeab0da495da549133c4886b3608d16/broadcasting/assets/rtsp_pipeline.drawio.png -------------------------------------------------------------------------------- /broadcasting/index.md: -------------------------------------------------------------------------------- 1 | --- 2 | title: Membrane broadcasting tutorial 3 | description: >- 4 | Create your own media broadcasting solution! 5 | part: 8 6 | --- 7 | 8 | | number | title | file | 9 | | ------ | -------------------------------- | -------------------------------- | 10 | | 1 | General introduction | 01_General_Introduction.md | 11 | | 2 | RTMP introduction | 02_RTMP_Introduction.md | 12 | | 3 | RTMP to HLS system architecture | 03_RTMP_SystemArchitecture.md | 13 | | 4 | Running the RTMP to HLS demo | 04_RTMP_RunningTheDemo.md | 14 | | 5 | RTMP to HLS - pipeline | 05_RTMP_Pipeline.md | 15 | | 6 | Web player | 06_WebPlayer.md | 16 | | 7 | RTSP to HLS introduction | 07_RTSP_Introduction.md | 17 | | 8 | RTSP to HLS system architecture | 08_RTSP_Architecture.md | 18 | | 9 | Running the RTSP to HLS demo | 09_RTSP_RunningDemo.md | 19 | | 10 | Connection manager | 10_ConnectionManager.md | 20 | | 11 | RTSP to HLS - pipeline | 11_RTSP_Pipeline.md | 21 | | 12 | Summary | 12_Summary.md | 22 | | 13 | (Suplement) H264 codec | 13_H264_codec.md | -------------------------------------------------------------------------------- /create_new_plugin/assets/images/Illo_create new plugin.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/membraneframework/membrane_tutorials/3e676855eeeab0da495da549133c4886b3608d16/create_new_plugin/assets/images/Illo_create new plugin.png -------------------------------------------------------------------------------- /create_new_plugin/create_new_plugin.md: -------------------------------------------------------------------------------- 1 | During the development of Membrane Framework we aim at designing fewer, but higher quality plugins. However, we also kept extendability and reusability in mind. That's why it is easy for developers like you to create their own custom plugin, which satisfies their needs. 2 | 3 | In this short guide we provide you with an overview of how to create your own Membrane plugin and how to integrate it into your project. 4 | 5 | ## Membrane plugin template 6 | 7 | To create a new plugin, we recommend using the [template](https://github.com/membraneframework/membrane_template_plugin) that has been made for this very purpose and which will be the base of your plugin. 8 | It defines necessary dependencies as well as other project specs, e.g. formatting, and guarantees you compliance with other Membrane components. 9 | 10 | You can start creating a plugin by making your copy of the template. Go to the [github repo](https://github.com/membraneframework/membrane_template_plugin) and select `Use this template`. Then choose an appropriate name for the project. 11 | 12 | If you haven't already, we suggest you read [basic pipeline tutorial](/basic_pipeline/create_new_plugin.md) to get familiar with Membrane's plugin structure. In any case, as you might have guessed the code of your plugin will go into `/lib` directory and the tests belong in the `/test` directory. 13 | 14 | ## Utilizing your plugin in a project 15 | 16 | When your plugin is ready for being integrated into another project you can simply add it as a dependency in `mix.exs` as described [here](https://hexdocs.pm/mix/Mix.Tasks.Deps.html). Here's what it can look like: 17 | 18 | ```elixir 19 | defp deps do 20 | [ 21 | ... 22 | {:your_membrane_plugin, git: "https://github.com/githubuser/your_membrane_plugin", tag: "0.1"} # dependency from github 23 | {:your_membrane_plugin, ">=0.1.0"} # dependency from [hex](https://hex.pm/) 24 | {:your_membrane_plugin, path: "path/to/your_plugin"} # dependency from local file 25 | ... 26 | ] 27 | end 28 | ``` 29 | 30 | And just like this, you have added your plugin to a project. 31 | -------------------------------------------------------------------------------- /create_new_plugin/index.md: -------------------------------------------------------------------------------- 1 | --- 2 | title: How to create your own plugin 3 | description: >- 4 | In this short guide we provide you with an overview of how to create your own Membrane plugin and how to integrate it into your project. 5 | part: 4 6 | graphicPath: assets/images/Illo_create new plugin.png 7 | --- 8 | 9 | | number | title | file | 10 | | ------ | ----------------- | -------------------- | 11 | | 1 | Create New Plugin | create_new_plugin.md | 12 | -------------------------------------------------------------------------------- /digital_video_introduction/1_preface.md: -------------------------------------------------------------------------------- 1 | This tutorial is essential for anyone who wants not only create multimedia applications, but also who wants to understand concepts of digital video processing and streaming 2 |
3 | Admittedly, it's not us who created this tutorial but we found it containing all the aspects we wanted to write about, so we're glad we can share it with you here. 4 |
5 | All credits to [Leaonardo Moreira](https://github.com/leandromoreira) and his contributors. You can find original files of this tutorial here: [Digital Video Introduction on Github](https://github.com/leandromoreira/digital_video_introduction) 6 |
7 | -------------------------------------------------------------------------------- /digital_video_introduction/assets/images/Illo_ digital video introduction.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/membraneframework/membrane_tutorials/3e676855eeeab0da495da549133c4886b3608d16/digital_video_introduction/assets/images/Illo_ digital video introduction.png -------------------------------------------------------------------------------- /digital_video_introduction/index.md: -------------------------------------------------------------------------------- 1 | --- 2 | title: Digital Video Introduction 3 | description: >- 4 | All you need to know to understand digital video. Written by Leonardo Moreira. 5 | part: 6 6 | graphicPath: assets/images/Illo_ digital video introduction.png 7 | --- 8 | 9 | | number | title | file | 10 | | ------ | -------------------------- | ------------ | 11 | | 1 | Preface | 1_preface.md | 12 | -------------------------------------------------------------------------------- /get_started_with_membrane/01_introduction.md: -------------------------------------------------------------------------------- 1 | Hello there, and a warm welcome to the Membrane tutorials. We're glad you chose to learn Membrane; and we'd like to invite you on a journey around multimedia with us, where we explore how to utilize the Membrane Framework to build applications that process audio, video, and other multimedia content in interesting ways. 2 | 3 | ## What is Membrane? 4 | 5 | Membrane is a multimedia processing framework that focuses on reliability, concurrency, and scalability. It is primarily written in Elixir, while some platform-specific or time-constrained parts are written in Rust C. With a range of existing packages and an easy-to-use interface for writing your own, Membrane can be used to process almost any type of multimedia, for example: 6 | - stream via WebRTC, RTSP, RTMP, HLS, HTTP and other protocols, 7 | - transcode, mix and apply custom processing of video & audio, 8 | - accept and generate / record to MP4, MKV, FLV and other containers, 9 | - handle dynamically connecting and disconnecting streams, 10 | - seamlessly scale and recover from errors, 11 | - do whatever you imagine if you implement it yourself :D Membrane makes it easy to plug in your code at almost any point of processing. 12 | 13 | If the abbreviations above don't ring any bells, don't worry - this tutorial doesn't require any multimedia-specific knowledge. Then, [other tutorials](https://membrane.stream/learn) and [demos](https://github.com/membraneframework/membrane_demo) will introduce you to the multimedia world! 14 | 15 | ## Structure of the framework 16 | 17 | The most basic media processing entities of Membrane are `Element`s. An element might be able, for example, to mux incoming audio and video streams into MP4, or play raw audio using your sound card. You can create elements yourself, or choose from the ones provided by the framework. 18 | 19 | Elements can be organized into a pipeline - a sequence of linked elements that perform a specific task. For example, a pipeline might receive an incoming RTSP stream from a webcam and convert it to an HLS stream, or act as a selective forwarding unit (SFU) to implement your own videoconferencing room. You'll see how to create a pipeline in the subsequent chapter. 20 | 21 | ### Membrane packages 22 | 23 | To embrace modularity, Membrane is delivered to you in multiple packages, including plugins, formats, core and standalone libraries. The list of all Membrane packages is available [here](https://github.com/membraneframework/membrane_core/#All-packages). It contains all the packages maintained by the Membrane team and some third-party packages. 24 | 25 | **Plugins** 26 | 27 | Plugins provide elements that you can use in your pipeline. Each plugin lives in a `membrane_X_plugin` repository, where `X` can be a protocol, codec, container or functionality, for example [mebrane_opus_plugin](https://github.com/membraneframework/membrane_opus_plugin). Plugins wrapping a tool or library are named `membrane_X_LIBRARYNAME_plugin` or just `membrane_LIBRARYNAME_plugin`, like [membrane_mp3_mad_plugin](https://github.com/membraneframework/membrane_mp3_mad_plugin). Plugins are published on [hex.pm](https://hex.pm), for example [hex.pm/packages/membrane_opus_plugin](https://hex.pm/pakcages/membrane_opus_plugin) and docs are at [hexdocs](https://hexdocs.pm), like [hexdocs.pm/membrane_opus_plugin](https://hexdocs.pm/membrane_opus_plugin). Some plugins require native libraries installed in your OS. Those requirements, along with usage examples are outlined in each plugin's readme. 28 | 29 | **Formats** 30 | 31 | Apart from plugins, Membrane has stream formats, which live in `membrane_X_format` repositories, where `X` is usually a codec or container, for example, [mebrane_opus_format](https://github.com/membraneframework/mebrane_opus_format). Stream formats are published the same way as packages and are used by elements to define what kind of stream can be sent or received. They also provide utility functions to deal with a given codec/container. 32 | 33 | **Core** 34 | 35 | [Membrane Core](https://github.com/membraneframework/membrane_core) is the heart and soul of the Membrane Framework. It is written entirely in Elixir and provides the internal mechanisms and API that allow you to prepare processing elements and link them together in a convenient yet reliable way. Note that Membrane Core does not contain any multimedia-specific logic. 36 | The documentation for the developer's API is available at [hexdocs](https://hexdocs.pm/membrane_core/readme.html). 37 | 38 | **Standalone libraries** 39 | 40 | Last but not least, Membrane provides tools and libraries that can be used standalone and don't depend on the `membrane_core`, for example, [video_compositor](https://github.com/membraneframework/video_compositor), [ex_sdp](https://github.com/membraneframework/ex_sdp) or [unifex](https://github.com/membraneframework/unifex). 41 | 42 | ## Where can I learn Membrane? 43 | There are a number of resources available for learning about Membrane: 44 | 45 | ### This guide 46 | The following sections in that guide will introduce the main concepts of creating Membrane elements and pipelines, without focusing on the specific details of multimedia processing. 47 | 48 | ### Demos 49 | The [membrane_demo](https://github.com/membraneframework/membrane_demo) repository contains many projects, scripts and livebooks that cover different use cases of the framework. It's a good place to learn by example. 50 | 51 | ### Tutorials 52 | For a step-by-step guide to implementing a specific system using Membrane, check out our [tutorials](https://membrane.stream/learn). 53 | 54 | ### Documentation 55 | For more detailed information, you can refer to the Membrane Core documentation and the documentation for the Membrane packages maintained by the Membrane team, both of which can be accessed [here](https://hex.pm/orgs/membraneframework). 56 | 57 | If you see something requiring improvement in this guide, feel free to create an issue or open a PR in the [membrane_tutorials](https://github.com/membraneframework/membrane_tutorials) repository. 58 | -------------------------------------------------------------------------------- /get_started_with_membrane/02_pipelines.md: -------------------------------------------------------------------------------- 1 | # Pipelines 2 | 3 | Building `Pipeline`s is the way to create streaming applications with Membrane. Pipeline allows you to spawn `Element`s and establish data flow between them. Pipelines can also communicate with elements or terminate them. Elements within a pipeline are often referred to as its `children` and the pipeline is their `parent`. 4 | 5 | Connecting elements together is called `linking`. 6 | 7 | To create a pipeline, you need to implement the [Membrane.Pipeline](https://hexdocs.pm/membrane_core/Membrane.Pipeline.html) behavior. It boils down to implementing callbacks and returning actions from them. For a simple pipeline, it's sufficient to implement the [handle_init](https://hexdocs.pm/membrane_core/Membrane.Pipeline.html#c:handle_init/2) callback, which is called upon the pipeline startup, and return the [spec](https://hexdocs.pm/membrane_core/Membrane.Pipeline.Action.html#t:spec/0) action, which spawns and links elements. Let's see it in an example: 8 | 9 | ## Sample pipeline 10 | 11 | ```elixir 12 | Mix.install([ 13 | :membrane_hackney_plugin, 14 | :membrane_mp3_mad_plugin, 15 | :membrane_portaudio_plugin, 16 | ]) 17 | 18 | defmodule MyPipeline do 19 | use Membrane.Pipeline 20 | 21 | @impl true 22 | def handle_init(_ctx, mp3_url) do 23 | spec = 24 | child(%Membrane.Hackney.Source{ 25 | location: mp3_url, hackney_opts: [follow_redirect: true] 26 | }) 27 | |> child(Membrane.MP3.MAD.Decoder) 28 | |> child(Membrane.PortAudio.Sink) 29 | 30 | {[spec: spec], %{}} 31 | end 32 | end 33 | 34 | mp3_url = "https://raw.githubusercontent.com/membraneframework/membrane_demo/master/simple_pipeline/sample.mp3" 35 | 36 | Membrane.Pipeline.start_link(MyPipeline, mp3_url) 37 | ``` 38 | 39 | The code above is one of the simplest examples of Membrane usage. It plays an MP3 file through your computer's default audio playback device with the help of the [PortAudio](http://www.portaudio.com/) audio I/O library. Let's digest this code and put it to work playing some sound. 40 | > **Elixir** 41 | > 42 | > Membrane is written in Elixir. It's an awesome programming language of the functional paradigm with great fault tolerance and process management, which made it the best choice for Membrane. 43 | > If you're not familiar with it, you can use [this cheatsheet](https://devhints.io/elixir) for a quick look-up. 44 | > We encourage you to also take a [deep look into Elixir](https://elixir-lang.org/getting-started/introduction.html) and learn how to use it to take full advantage of all its awesomeness. We believe you'll fall in love with Elixir too! 45 | 46 | 47 | You have two options to run the snippet: 48 | 49 | - Option 1: Click [here](https://livebook.dev/run?url=https%3A%2F%2Fgithub.com%2Fmembraneframework%2Fmembrane_core%2Fblob%2Fmaster%2Fexample.livemd). You'll be directed to install [Livebook](https://livebook.dev), an interactive notebook similar to Jupyter, and it'll open the snippet in there for you. Then just click the 'run' button in there. 50 | 51 | - Option 2: If you don't want to use Livebook, you can [install Elixir](https://elixir-lang.org/install.html), type `iex` to run the interactive shell and paste the snippet there. 52 | 53 | 54 | ## Sample pipeline explained 55 | 56 | Let's figure out step-by-step what happens in the sample pipeline. 57 | 58 | Firstly, we install the needed dependencies. They are plugins, that contain elements that we will use in the pipeline. 59 | 60 | ```elixir 61 | Mix.install([ 62 | :membrane_hackney_plugin, 63 | :membrane_mp3_mad_plugin, 64 | :membrane_portaudio_plugin, 65 | ]) 66 | ``` 67 | 68 | Instead of creating a script and using `Mix.install`, you can also [create a Mix project](https://elixir-lang.org/getting-started/mix-otp/introduction-to-mix.html) and add these dependencies to `deps` in `mix.exs` file. 69 | 70 | After installing the dependencies, we can create a module for our pipeline: 71 | 72 | ```elixir 73 | defmodule MyPipeline do 74 | use Membrane.Pipeline 75 | 76 | end 77 | ``` 78 | 79 | Using the `Membrane.Pipeline` behaviour means we are treating our module as a Membrane Pipeline, so we will have access to functions defined in the `Membrane.Pipeline` module, and we can implement some of its callbacks. Let's implement the `handle_init/2` callback, which is a function that is invoked to initialize a pipeline during start-up: 80 | 81 | ```elixir 82 | defmodule MyPipeline do 83 | use Membrane.Pipeline 84 | 85 | @impl true 86 | def handle_init(_ctx, path_to_mp3) do 87 | 88 | end 89 | end 90 | ``` 91 | 92 | > If the concept of callbacks and behaviours is new to you, you can read more about it [here](https://elixir-lang.org/getting-started/typespecs-and-behaviours.html#behaviours), or see examples of behaviours in the Elixir standard library, for example the [GenServer](https://elixir-lang.org/getting-started/mix-otp/genserver.html) and [Supervisor](https://elixir-lang.org/getting-started/mix-otp/supervisor-and-application.html) behaviours. 93 | 94 | We'll use `handle_init` to specify all its [elements](../glossary/glossary.md#element) as children and set up links between them to define the order in which data will flow through the pipeline: 95 | 96 | ```elixir 97 | @impl true 98 | def handle_init(_ctx, path_to_mp3) do 99 | spec = 100 | child(%Membrane.Hackney.Source{ 101 | location: mp3_url, hackney_opts: [follow_redirect: true] 102 | }) 103 | |> child(Membrane.MP3.MAD.Decoder) 104 | |> child(Membrane.PortAudio.Sink) 105 | 106 | {[spec: spec], %{}} 107 | end 108 | ``` 109 | 110 | The [child](https://hexdocs.pm/membrane_core/Membrane.ChildrenSpec.html#child/2) function allows us to spawn particular elements. By piping one child to another with the `|>` operator, we can specify the order of the elements in the pipeline. Thus, the code above links `Membrane.Hackney.Source` to `Membrane.MP3.MAD.Decoder` and `Membrane.MP3.MAD.Decoder` to `Membrane.PortAudio.Sink`. Here's what they are: 111 | - Hackney source - an element based on the [Hackney HTTP library](https://github.com/benoitc/hackney), that downloads a file via HTTP chunk by chunk, and sends these chunks through its `output` pad. We pass two options to it: a URL where the MP3 is stored and a flag to make it follow HTTP redirects. 112 | - MP3 decoder - an element based on [libmad](https://github.com/markjeee/libmad), that accepts MP3 audio on the `input` pad and sends the [decoded](../glossary/glossary.md#decoding) audio through the `output` pad. 113 | - PortAudio sink - an element that accepts decoded audio on its `input` pad and uses the [PortAudio](https://github.com/PortAudio/portaudio) library to play in on the speaker. 114 | 115 | In our spec, we don't mention the names of the pads, because `input` and `output` are the defaults. However, we could explicitly specify them: 116 | 117 | ```elixir 118 | spec = 119 | child(%Membrane.Hackney.Source{ 120 | location: mp3_url, hackney_opts: [follow_redirect: true] 121 | }) 122 | |> via_out(:output) 123 | |> via_in(:input) 124 | |> child(Membrane.MP3.MAD.Decoder) 125 | |> via_out(:output) 126 | |> via_in(:input) 127 | |> child(Membrane.PortAudio.Sink) 128 | ``` 129 | 130 | Even though not necessary here, [via_in](https://hexdocs.pm/membrane_core/Membrane.ChildrenSpec.html#via_in/3) and [via_out](https://hexdocs.pm/membrane_core/Membrane.ChildrenSpec.html#via_out/3) are useful in more complex scenarios that we'll cover later. 131 | 132 | The value returned from `handle_init`: 133 | 134 | ```elixir 135 | {[spec: spec], %{}} 136 | ``` 137 | 138 | is a tuple containing the list of actions and the state. 139 | - Actions are the way to interact with Membrane. Apart from `spec`, you can for example return `terminate: reason` that will stop the elements and terminate the pipeline. Most actions, including `spec`, can be returned from multiple callbacks, allowing, for example, to spawn elements on demand. Check the [Membrane.Pipeline](https://hexdocs.pm/membrane_core/Membrane.Pipeline.html) behavior for the available callbacks and [Membrane.Pipeline.Action](https://hexdocs.pm/membrane_core/Membrane.Pipeline.Action.html) for the available actions. 140 | - State is an arbitrary data that will be passed to subsequent callbacks as the last argument. It's usually a map. As we have no use for the state in this case, we just set it to an empty map. 141 | 142 | When we have created our pipeline module, we can call [Membrane.Pipeline.start_link](https://hexdocs.pm/membrane_core/Membrane.Pipeline.html#start_link/3) to run it: 143 | 144 | ```elixir 145 | Membrane.Pipeline.start_link(MyPipeline, mp3_url) 146 | ``` 147 | 148 | We pass to it the pipeline module and options, which in our case is the `mp3_url`. The options are passed directly to the `handle_init` callback. 149 | 150 | Congratulations! You've just built and run your first Membrane application. Let's now have a deeper look at the elements. 151 | -------------------------------------------------------------------------------- /get_started_with_membrane/04_bins.md: -------------------------------------------------------------------------------- 1 | # Bins 2 | 3 | Bins, similarly to pipelines, are containers for elements. However, at the same time, they can be placed and linked within pipelines. Although a bin is a separate Membrane entity, it can be perceived as a pipeline within an element. Bins can also be nested within one another, though we don't recommend too much nesting, as it may end up hard to maintain. For an example of a bin, have a look at the [RTMP source bin](https://github.com/membraneframework/membrane_rtmp_plugin/blob/master/lib/membrane_rtmp_plugin/rtmp/source/source_bin.ex) or [HTTP adaptive stream sink bin](https://github.com/membraneframework/membrane_http_adaptive_stream_plugin/blob/master/lib/membrane_http_adaptive_stream/sink_bin.ex). 4 | 5 | The main use cases for a bin are: 6 | - creating reusable element groups, 7 | - encapsulating children's management logic, for instance, dynamically spawning or replacing elements as the stream changes. 8 | 9 | ## Bin's pads 10 | 11 | Bin's pads are defined and linked similarly to element's pads. However, their role is limited to proxy the stream to other elements and bins inside (inputs) or outside (outputs). To achieve that, each input pad of a bin needs to be linked to both an output pad from the outside of a bin and an input pad of its child inside. Accordingly, each bin's output should be linked to output inside and input outside of the bin. 12 | 13 | ## Bin and the stream 14 | 15 | Although the bin passes the stream through its pads, it does not access it directly, so callbacks such as `handle_process` or `handle_event` are not found there. This is because the responsibility of the bin is to manage its children, not to process the stream. Whatever the bin needs to know about the stream, it should get through notifications from the children. 16 | 17 | ## Bin as a black box 18 | 19 | Bins are designed to take as much responsibility for their children as possible so that pipelines (or parent bins) don't have to depend on bins' internals. That's why notifications from the children are sent to their direct parents only. Also, messages received by a bin or pipeline can be forwarded only to its direct children. 20 | -------------------------------------------------------------------------------- /get_started_with_membrane/05_pads_and_linking.md: -------------------------------------------------------------------------------- 1 | # Pads and linking 2 | 3 | You learned how to link elements in the `Pipeline` chapter and how to define pads in the `Element` chapter. Now, let's have a deeper dive into pads and their capabilities. 4 | 5 | ## Dynamic Pads 6 | 7 | A dynamic pad is a type of pad that acts as a template - each time some other pad is linked to a dynamic pad, a new instance of it is created. 8 | 9 | Dynamic pads don't have to be linked when the element is started. This needs to be handled by the element, but in return, it gives new possibilities when the number of pads can change on the fly. 10 | 11 | Another use case for dynamic pads is when the number of pads is not known at the compile time. 12 | For example, an audio mixer may have any number of inputs. 13 | 14 | 15 | ### Creating an element with dynamic pads 16 | 17 | To make a pad dynamic, you need to set its `availability` to `:on_request` in [def_input_pad](https://hexdocs.pm/membrane_core/Membrane.Element.WithInputPads.html#def_input_pad/2) or [def_output_pad](https://hexdocs.pm/membrane_core/Membrane.Element.WithOutputPads.html#def_output_pad/2). 18 | 19 | Now, each time some element is linked to this pad, a new instance of the pad is created and [handle_pad_added](https://hexdocs.pm/membrane_core/Membrane.Element.Base.html#c:handle_pad_added/3) callback is invoked. Instances of a dynamic pad can be referenced as `Pad.ref(pad_name, pad_id)`. When a dynamic pad is unlinked, the [handle_pad_removed](https://hexdocs.pm/membrane_core/Membrane.Element.Base.html#c:handle_pad_removed/3) callback is called. 20 | 21 | ### Gotchas 22 | 23 | As usual, with great power comes great responsibility. When implementing an element with dynamic pads, you need to consider what should happen when they are added or removed. Usually, you need to implement `handle_pad_added` and `handle_pad_removed` callbacks, and the logic of an element may generally become more complicated. 24 | 25 | ### Linking dynamic pads 26 | 27 | Let's see how to link dynamic pads. We'll use [membrane_file_plugin](https://github.com/membraneframework/membrane_file_plugin), a plugin that allows reading and writing to files, and [membrane_tee_plugin](https://github.com/membraneframework/membrane_tee_plugin) which allows forwarding the stream from a single input to multiple outputs. Running the pipeline below with `Membrane.Pipeline.start_link(MyPipeline)` should copy the "source" file to "target1", "target2" and "target3" files. Don't forget to create the "source" file before. 28 | 29 | ```elixir 30 | Mix.install([ 31 | :membrane_file_plugin, 32 | :membrane_tee_plugin 33 | ]) 34 | 35 | defmodule MyPipeline do 36 | use Membrane.Pipeline 37 | 38 | alias Membrane.{File, Tee} 39 | 40 | @impl true 41 | def handle_init(_ctx, _options) 42 | spec = [ 43 | child(%File.Source{location: "source"}) 44 | |> child(:tee, Tee.Parallel), 45 | get_child(:tee) |> child(%File.Sink{location: "target1"}), 46 | get_child(:tee) |> child(%File.Sink{location: "target2"}), 47 | get_child(:tee) |> child(%File.Sink{location: "target3"}) 48 | ] 49 | end 50 | end 51 | ``` 52 | 53 | The [Membrane.Tee.Parallel](https://hexdocs.pm/membrane_tee_plugin/Membrane.Tee.Parallel.html) element has a single static input and a single dynamic output pad. Because the output is dynamic, each time we link it, a new pad instance is created with a unique reference. In the example above, pad references were generated automatically. It's possible to specify them directly with [via_in](https://hexdocs.pm/membrane_core/Membrane.ChildrenSpec.html#via_in/3) or [via_out](https://hexdocs.pm/membrane_core/Membrane.ChildrenSpec.html#via_out/3) and [Membrane.Pad.ref](https://hexdocs.pm/membrane_core/Membrane.Pad.html#ref/1): 54 | 55 | ```elixir 56 | spec = [ 57 | child(%File.Source{location: "source"}) 58 | |> child(:tee, Tee.Parallel), 59 | get_child(:tee) |> via_out(Pad.ref(:output, 1)) |> child(%File.Sink{location: "target1"}), 60 | get_child(:tee) |> via_out(Pad.ref(:output, 2)) |> child(%File.Sink{location: "target2"}), 61 | get_child(:tee) |> via_out(Pad.ref(:output, 3)) |> child(%File.Sink{location: "target3"}) 62 | ] 63 | ``` 64 | 65 | In this case, it won't make a difference, but elements can rely on pad references and use them to identify the stream that should be sent or received through the given pad. 66 | 67 | ## Pad options 68 | 69 | Just like elements, pads can have options. They have to be defined with the `options` key in `def_input_pad` and `def_output_pad`, using the same syntax as the element options. For example, [Membrane.AudioMixer](https://hexdocs.pm/membrane_audio_mix_plugin/Membrane.AudioMixer.html) does it to make its `input` pad accept the `offset` option: 70 | 71 | ```elixir 72 | def_input_pad :input, 73 | availability: :on_request, 74 | options: [ 75 | offset: [ 76 | spec: Time.non_neg(), 77 | default: 0, 78 | description: "Offset of the input audio at the pad." 79 | ] 80 | ], 81 | # ... 82 | ``` 83 | 84 | Pad options can be set via a keyword list within the second argument of `via_in` or `via_out`. Here's how to provide the `offset` option to the mixer: 85 | 86 | ```elixir 87 | spec = [ 88 | # ... 89 | child(Membrane.MP3.MAD.Decoder) 90 | |> via_in(:input, options: [offset: Membrane.Time.seconds(2)]) 91 | |> child(Membrane.AudioMixer) 92 | # ... 93 | ] 94 | ``` 95 | 96 | Pad options' values can be accessed in the context of the `handle_pad_added` callback: 97 | 98 | ```elixir 99 | @impl true 100 | def handle_pad_added(pad, context, state) do 101 | %{offset: offset} = context.pad_options 102 | # ... 103 | end 104 | ``` 105 | -------------------------------------------------------------------------------- /get_started_with_membrane/06_flow_control.md: -------------------------------------------------------------------------------- 1 | # Flow Control 2 | 3 | Once we have linked the elements together, there comes a moment when they can start sending buffers through the pads. However, how to control the flow of buffers between elements? 4 | 5 | Membrane solves this problem using its own backpressure mechanism and demands. 6 | 7 | ## Types of `flow_control` 8 | 9 | When defining a pad od an element, you can pass one of 3 values in the `flow_control` field: `auto`, `push`, or `manual`. They determine how the backpressure mechanism works on the pad they relate to. 10 | 11 | - Input pad with `flow_control: :push` - an element with such a pad must always be ready to receive a new buffer on that pad. The output pad linked with such an input pad must also have `flow control` set to `:push`. It is not advised to use `flow_control: :push` in input pads, as it makes them prone to overflow. 12 | 13 | - Output pad with `flow_control: :push` - the element decides for itself when to send how many buffers via a given pad. 14 | 15 | - Input pad with `flow_control: :manual` - the element directly, using the `demand` action, requests how much data it wants to receive from this pad. Membrane guarantees that more data will never come to the input pad than was demanded. 16 | 17 | - Output pad with `flow_control: :manual` - the element gets the information, through the `c:Membrane.Element.WithOutputPads.handle_demand/5` callback, how much data a specific output pad needs. An element having an output pad with `flow_control: :manual` is obligated to satisfy the demand on it. 18 | 19 | - Input pad with `flow_control: :auto` - unlike in manual demands, the element doesn't have to directly specify the demand value on the input pad to receive buffers on it. In `:auto` `flow control`, the framework will manage the demand value itself. In the case of Sources and Endpoints, it will make sure that the demand on the input pad is positive as long as the number of buffers waiting for processing in the mailbox is not too large. In the case of filters, Membrane will take care of a positive demand under one additional condition - the demand value on all output pads with `flow_control: :auto` of this element must also be positive. 20 | 21 | - Output pad with `flow_control: :auto` - an auto output pad does not require from the element to implement the `handle_demand` callback. The demand value on this pad only matters when the framework itself calculates the value of the auto demand on the input pads. The only type of Membrane Element that can have output pads with `flow_control: :auto` is a `Membrane.Filter`. 22 | 23 | `flow_control: :manual` is useful in cases, when element wants to get a specific amount of data on each pad (e.g. to satisfy demand on 1 buffer on pad `:output`, element need 1 buffer from pad `:input_a`, 10 buffers from pad `:input_b` and 100 bytes from pad `:input_c`). Comparing to `flow_control: :auto`, `flow_control: :manual` can solve a broader class of problems, but, on the other hand, it requires more code in your element and is less performant. 24 | 25 | `flow_control: :auto` is the best solution, when you want to provide a backpressure mechanism, but your element has only one input pad or it doesn't care about the proportion of the amount of data incoming on each input pad. A good example might be a pipeline made of filters linked one after the other, with a sink or a source on each end. In such a case, using `flow_control: :auto` in filters would ensure, that the pipeline would adapt its processing pace to the slowest element, without the necessity to implement `c:Membrane.Element.WithOutputPads.handle_demand/5` in every element. Additionally, the whole pipeline will be faster, because of the lack of performance overhead caused by `flow_control: :manual`. 26 | 27 | ## Linking pads with different `flow_control` 28 | 29 | Two pads with the same type of `flow_control` can always be linked together. `flow_control: auto` can also always be linked with `flow_control: manual`. However, the situation becomes complicated when we try to link two pads with different types of `flow_control`, where one of them has `flow_control: push`. 30 | 31 | When the input pad has `flow_control: push`, the output pad linked to it must also have `flow_control: push`. An attempt to link different type of an output pad will cause an error to be thrown. 32 | 33 | Output pads with `flow_control: push` can be linked with input pads having `flow control` `:auto` or `:manual`. The demand value on such an input pad does not affect the behavior of the element with the push output pad. 34 | 35 | When an element with a push output pad sends amount of data, that exceeds demands on the input pad, when input pad `flow_control` is: 36 | - `manual`, these buffers will wait in line until the element raises the demand on its input by returning the `demand` action. 37 | - `auto`, these buffers will be processed, and the demand value on the input pad will drop below zero. 38 | 39 | ```elixir 40 | get_child(:mixer) 41 | |> via_in(:input, toilet_capacity: 500) 42 | |> get_child(:sink) 43 | ``` -------------------------------------------------------------------------------- /get_started_with_membrane/07_observability_and_logging.md: -------------------------------------------------------------------------------- 1 | # Observability & logging 2 | 3 | To observe and debug Membrane, you can use a variety of tools provided by the BEAM, like [observer](https://www.erlang.org/doc/man/observer.html) or [Process.info](https://hexdocs.pm/elixir/1.15.5/Process.html#info/1). Apart from them, Membrane provides dedicated utilities for observability and logging. 4 | 5 | ## Introspecting pipelines with kino_membrane 6 | 7 | [kino_membrane](https://github.com/membraneframework/kino_membrane) allows for introspecting Membrane pipelines in [Livebook](https://livebook.dev). You can either run the pipeline inside livebook ([see example](https://hexdocs.pm/kino_membrane/pipeline_in_livebook.html)) or connect to a running VM ([see example](https://hexdocs.pm/kino_membrane/connect_to_node.html)). Then, it's enough to install kino_membrane: 8 | 9 | ```elixir 10 | Mix.install([:kino_membrane]) 11 | ``` 12 | 13 | and run 14 | 15 | ```elixir 16 | KinoMembrane.pipeline_dashboard(pipeline_pid) 17 | ``` 18 | 19 | in Livebook, and you'll get a panel with the pipeline graph, metrics and other utilities: 20 | 21 | ![Kino Membrane](assets/images/kino_membrane.png) 22 | 23 | Check [kino_membrane](https://github.com/membraneframework/kino_membrane) for details. 24 | 25 | ## Logging 26 | 27 | Apart from the usual `Logger` configuration, logging in Membrane can be additionally configured, also via Elixir's `Config`. It allows to enable verbose mode and customize metadata, for example: 28 | 29 | ```elixir 30 | import Config 31 | config :membrane_core, :logger, verbose: true 32 | ``` 33 | 34 | For this change to take effect, it may be necessary to recompile the whole project with `mix deps.compile --force && mix compile --force`. See [`Membrane.Logger`](https://hexdocs.pm/membrane_core/Membrane.Logger.html) for details. 35 | 36 | Moreover, in a pipeline or bin, you can add logger metadata to the [children spec](https://hexdocs.pm/membrane_core/Membrane.ChildrenSpec.html): 37 | 38 | ```elixir 39 | @impl true 40 | def handle_init(_ctx, opts) do 41 | spec = { 42 | child(Some.Element) |> child(Some.Bin), 43 | log_metadata: [pipeline_id: opts.id] 44 | } 45 | 46 | {[spec: spec], state} 47 | end 48 | ``` 49 | 50 | and it will add the `pipeline_id` metadata key to all the logs coming from `Some.Element` and `Some.Bin` and all its descendants. 51 | 52 | To have the metadata displayed in the logs, you need to configure it as well: 53 | ```elixir 54 | import Config 55 | config :logger, :console, metadata: [:pipeline_id] 56 | ``` 57 | -------------------------------------------------------------------------------- /get_started_with_membrane/08_native_code_integration.md: -------------------------------------------------------------------------------- 1 | # Native code integration 2 | 3 | In Membrane we try to make use of Elixir goodness as often as possible. However, sometimes it is necessary to write some native code, for example to integrate with existing C libraries. To achieve that, we use _natives_, which are either [NIFs](http://erlang.org/doc/man/erl_nif.html) or [C nodes](http://erlang.org/doc/man/ei_connect.html). Both of them have their tradeoffs: 4 | - NIFs don't introduce significant overhead, but in case of failure they can even crash entire VM, 5 | - CNodes are isolated and can be supervised like Elixir processes, but they are slower due to the inter-process communication. 6 | 7 | That's why we use them only when necessary and created some tools to make dealing with them easier. For interoperability with C and C++, we use [Bundlex](https://github.com/membraneframework/bundlex) and [Unifex](https://github.com/membraneframework/unifex). For calling Rust, we use [Rustler](https://github.com/rusterlium/rustler). 8 | 9 | ## [Bundlex](https://github.com/membraneframework/bundlex) 10 | 11 | To simplify and unify the process of writing and compiling natives, we use our own build tool - [Bundlex](https://github.com/membraneframework/bundlex). It is a multi-platform tool that provides a convenient way of compiling and accessing natives from Elixir code. 12 | For more information, see [Bundlex's GitHub page](https://github.com/membraneframework/bundlex). 13 | 14 | ## [Unifex](https://github.com/membraneframework/unifex) 15 | 16 | Process of creating natives is not only difficult but also quite arduous, because it requires using cumbersome Erlang APIs, and thus a lot of boilerplate code. To make it more pleasant, we created [Unifex](https://github.com/membraneframework/unifex), a tool that is responsible for generating interfaces between C or C++ libraries and Elixir on the base of short `.exs` configuration files. An important feature of Unifex is that you can write the C/C++ code once and use it either as a NIF or as a C node. 17 | 18 | A quick introduction to Unifex is available [here](https://hexdocs.pm/unifex/creating_unifex_natives.html). 19 | 20 | -------------------------------------------------------------------------------- /get_started_with_membrane/assets/images/Illo_get_started.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/membraneframework/membrane_tutorials/3e676855eeeab0da495da549133c4886b3608d16/get_started_with_membrane/assets/images/Illo_get_started.png -------------------------------------------------------------------------------- /get_started_with_membrane/assets/images/kino_membrane.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/membraneframework/membrane_tutorials/3e676855eeeab0da495da549133c4886b3608d16/get_started_with_membrane/assets/images/kino_membrane.png -------------------------------------------------------------------------------- /get_started_with_membrane/index.md: -------------------------------------------------------------------------------- 1 | --- 2 | title: Get Started with Membrane 3 | description: This tutorial explains how membrane works. It covers environment preparation, pipelines, and the basic concepts behind the framework. 4 | part: 1 5 | graphicPath: assets/images/Illo_get_started.png 6 | --- 7 | 8 | | number | title | file | 9 | | ------ | ------------------------- | ------------------------------- | 10 | | 1 | Get Started with Membrane | 01_introduction.md | 11 | | 2 | Pipelines | 02_pipelines.md | 12 | | 3 | Elements | 03_elements.md | 13 | | 4 | Bins | 04_bins.md | 14 | | 5 | Pads & linking | 05_pads_and_linking.md | 15 | | 6 | Flow control | 06_flow_control.md | 16 | | 7 | Observability & logging | 07_observability_and_logging.md | 17 | | 8 | Native code integration | 08_native_code_integration.md | 18 | -------------------------------------------------------------------------------- /glossary/assets/images/Illo_glossary.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/membraneframework/membrane_tutorials/3e676855eeeab0da495da549133c4886b3608d16/glossary/assets/images/Illo_glossary.png -------------------------------------------------------------------------------- /glossary/index.md: -------------------------------------------------------------------------------- 1 | --- 2 | title: Glossary 3 | description: Set of terms we're using in our tutorials you might want to learn more about. 4 | part: 7 5 | graphicPath: assets/images/Illo_glossary.png 6 | --- 7 | 8 | | number | title | file | 9 | | ------ | -------- | ----------- | 10 | | 1 | Glossary | glossary.md | 11 | -------------------------------------------------------------------------------- /h264/1_Introduction.md: -------------------------------------------------------------------------------- 1 | # Introduction 2 | 3 | ## Once upon a time, there was a parser… 4 | 5 | In Membrane, we have a concept of an H.264 parser - a processing element that is designed to put the incoming H.264 stream "in order" by splitting the stream into packets containing the given amount of information (i.e. into separate video frames) or fetching some information about the video (like resolution or framerate). Such processing is often required before muxing the stream into a container or putting it into some transport protocol's packets. 6 | We used to use an Elixir wrapper on ffmpeg SDK that performed the type of processing described above - we have called that wrapper H264.FFMpeg.Parser, and it's still available in our repository: membrane_h264_ffmpeg_plugin, along with the H264 software decoder and encoder. 7 | Initially, such a solution perfectly fit our needs - we had a concise yet bug-free parser implementation, and we didn't need to get our hands dirty with the h264 stream structure at all. The difficult stuff was already done by the great FFmpeg team, and we were just harvesting the fruits of their labor. 8 | 9 | However, it wasn't that long until we faced a need to add some custom functionalities to the parser. Nothing particularly complicated: dropping frames from the stream before the parameters or keyframe appeared for the first time. 10 | That was the moment that we started struggling with a tremendous number of problems, which mostly originated in the fact that the state of the processing element was held in three different places: 11 | * in the Elixir's GenServer state 12 | * in the C bridge code between Elixir and FFMpeg SDK 13 | * internally, in the data structures provided by FFMpeg SDK 14 | A stack of options, parameters and behavioral patterns had been growing, some options had become more or less explicitly incompatible with others, and more and more time was being spent on debugging the problems caused by the use of our parser. A decision was made - we need to perform a refactor. Soon we realized that there was in fact nothing to refactor - the whole code would have needed to be thrown out and the element would have needed to be written from scratch. 15 | 16 | ## The new hope 17 | 18 | We realized that we were more experienced with multimedia processing than we had been when the wrapped parser was first created. So we asked ourselves - why not create the parser in plain Elixir and get rid of some native FFMpeg dependency? It shouldn't be that difficult, should it? … 19 | Long story short, we learned a lot ;) And we would like to share some of our findings with you. 20 | -------------------------------------------------------------------------------- /h264/2_What_is_H264.md: -------------------------------------------------------------------------------- 1 | The first order of business was to put our knowledge in order and supplement it. There is no better knowledge source than the official ITU-T standard defining the H264 codec: "Advanced video coding for generic audiovisual services." 2 | It might be overwhelming - the standard consists of 680 pages of printed text written in an ultra technical manner, however, for our needs, we could skip the parts concerning the video encoding and decoding processes - which make up most of the standard's volume. 3 | 4 | ## AVC, H.264 or MPEG-4 p. 10? 5 | Firstly, we would like to explain the nomenclature that we will be using, since it might be misleading. 6 | The codec, or more precisely - a coding standard, that we are describing is known under three names: 7 | * **AVC** - that stands for: Advanced Video Coding 8 | * **H.264** - the name is derived from the name of **ITU-T** (*International Telecommunication Union Standardization Sector*) (recommendation)[https://www.itu.int/rec/T-REC-H.264] that defines that coding standard 9 | * **MPEG-4 Part 10** - the name is derived from the name of a (document)[https://www.mpeg.org/standards/MPEG-4/10/] defining the standard, prepared by the **ISO/IEC JTC 1** (*Joint Technical Committee* (JTC) of the *International Organization for Standardization* (ISO) and the *International Electrotechnical Commission*) **Moving Picture Experts Group** 10 | 11 | The existence of multiple names originates from the fact that the codec has been standardized as a cooperation of two groups - Video Coding Experts Group of ITU-T and Moving Picture Experts Group of ISO/IEC JTC 1. Within the text of the article we will most frequently use the "H.264" name, as it is most commonly used. 12 | 13 | ## Why do we use H.264? 14 | There are multiple reasons! The most important is that it allows us to represent video in a format that: 15 | * supports compression - compression happens on multiple levels (for more information about some "tricks" used to compress the images and video, we invite you to read a tutorial published on [our website](https://membrane.stream/learn/digital_video_introduction)): 16 | * chroma subsampling - the compression standard, which takes advantage of the imperfections of human perception and removes the information about the pixels' color that the human's eye can't even process 17 | * DCT (*discrete cosine transform*) and rounding it's coefficients to the integers 18 | * intra-frame prediction - each video frame is divided into so called "macroblocks" (square areas consisting of 16x16 pixels) and instead of storing the information about all the pixels of each frame, only information of all of the pixels for some frames is stored (these are so called "keyframes"), and the rest of the frames are stored as vectors describing the translation of given macroblocks 19 | * allows for convenient multimedia transport - it's easy to pack an H.264 stream into multimedia containers (i.e. MP4) or packets (like RTP packets) and send it via the internet. 20 | 21 | It's good to be aware that in fact the H.264 stream is designed to be wrapped in some other data structure, like a multimedia container (i.e. MP4) or some transport protocol packets (i.e. RTP packets) - we shouldn't store the H.264 stream as a plain binary file, as it might lead to many problems. The reason for that is that the H.264 stream itself does not contain all of the information that is required for a video to be properly displayed. Examples of such information are timestamps - H.264 stream does not contain timestamps at all, so unwrapping the H.264 stream from the MP4 container (that contains timing information additionally) and storing just the H.264 stream leads to a loss of information. 22 | 23 | However, that's just theory - in practice, tools like FFmpeg allow you to read the plain H.264 files and do some "best effort" approximations of the missing information - i.e. when the framerate of the video is constant, they can simply calculate the missing timestamps by multiplying the ordering number of the given video frame by the inversion of the framerate. 24 | That defines some kind of implicit "standard" of dealing with H.264 stream dumps, which we have also needed to take into consideration while working on our parser. 25 | 26 | One might ask - why use H.264, when its successor, **H.265** standard (known also as **HEVC**, which stands for *high efficiency video compression*) is already available. The answer is - because H.264 is still widely used and still there is a need to cope with H.264 streams that are delivered to our system. At the same time, starting using H.265 comes with a great number of benefits, as it: 27 | * allows for a better compression ratio. For a given video quality, up to 50% of bandwidth can be saved with H.265, compared to H.264, which has non-negligible impact on the maintenance costs of a media processing infrastructure. 28 | * supports up to 8K UHDTV (in contrast to H.264, which supports up to 4K). 29 | * supports up to 300 FPS (in contrast to H.264, which supports up to 59.94 FPS) 30 | 31 | ## H.264 layers - VCL vs NAL 32 | 33 | In H.264 two so-called "layers" can be distinguished: 34 | * **VCL** (*Video Coding Layer*) - this layer deals mostly with the video compression standard. H.264 encoders and decoders need to work with that layer, but in the case of the H.264 parser, we won't need to dig into it. 35 | * **NAL** (*Network Abstraction Layer*) - the layer that comes as a part of the network protocol's stack. It defines an abstraction over the VCL as it describes how the stream could be split into packets - Network Abstraction Layer units (we will refer to these units as NALu). Such an abstraction layer comes in handy for network devices i.e. it delivers information about how important a given packet is for the decoding process - based on that information, a network device knows which packets are high-priority and shouldn't be dropped (i.e. the ones that hold the keyframes), as dropping them would make the decoding impossible. 36 | 37 | The NAL was the subject of our interest when it comes to the H.264 parser - the parser's goal is to read the NAL units and fetch the information that interested us out of them. 38 | 39 | 40 | ![VCL vs NAL scheme](assets/images/VCL_vs_NAL.png) 41 |
-------------------------------------------------------------------------------- /h264/3_H264_NAL_units.md: -------------------------------------------------------------------------------- 1 | As noted above, from the NAL perspective, the H.264 stream is just a sequence of NAL units. 2 | But how do we know where one NAL unit ends and another one starts? 3 | Well, the answer to that question is not straightforward, since the ITU specification does not define a single byte stream format. At the same time, one of the possible formats is described as an annex to the specification - Annex B. That format itself is usually referred to as Annex B and we will also stick to that name. 4 | We need to be aware that that format is not the only option - the other commonly used byte stream format is AVCC, known also as length prefix. 5 | We will briefly discuss both of these formats. 6 | 7 | 8 | ## Annex B 9 | 10 | Annex B is a format in which all NAL units are preceded with a so-called "start code" - one of the following byte sequences: three-byte `0x000001` or four-bytes `0x00000001`. 11 | In the case of that format, the H.264 stream might look as follows: 12 | ``` 13 | ([0x000001][first NAL unit]) | ([0x000001][second NAL unit]) | ([0x000001][third NAL unit]) … 14 | ``` 15 | The existence of two "start codes" is dictated by the fact that the first variant is more applicable in certain situations than the second one. For a four-bytes variant, it's easy to find the start of a NAL unit sent through a serial connection, as it's easy to align the stream to 31 "0" bits followed by a single "1" bit. In other cases, to save space, it's common to use a three-byte variant. 16 | 17 | The choice of `0x000001` and `0x00000001` as a NAL units separator wasn't unintentional - such a sequence of bytes does not encode much information (recall Shannon's formula for the amount of information) and that's why it's not frequent for the compression algorithm used in H.264 to produce such a sequence of bytes (the compression mechanism tries to encode as much information about video in the shortest bytes sequence possible). However, such a byte sequence might still appear in the stream (please also note, that some parts of the stream are metadata, which is not compressed) - and the format definition must take that into consideration. That's why an "emulation prevention" mechanism has been introduced. That mechanism is nothing more than a regular "escape character" mechanism - each of the following three-bytes sequences is not allowed to exist in the "escaped" byte stream: 18 | * `0x000000` 19 | * `0x000001` 20 | * `0x000002` 21 | * `0x000003` 22 | and that's why it gets replaced correspondingly with the following sequences: 23 | * `0x00000300` 24 | * `0x00000301` 25 | * `0x00000302` 26 | * `0x00000303` 27 | The "escape character" is 0x03 - also known as `emulation_prevention_three_byte`. 28 | 29 | The representation with "emulation prevention three bytes" inserted is called **SODB** (*String of Data Bits*), and the representation without the "emulation prevention" bytes is called **RBSP** (*Raw Byte Sequence Payload*). Encoders need to take care of escaping the unallowed byte sequences (converting the byte stream from RBSP to SODB), while decoders are obliged with the reverse operation (converting from SODB to RBSP). 30 | 31 | 32 | ## AVCC 33 | 34 | In this byte stream format, each NAL unit is preceded with a `nal_size` field, describing its length. That length might be stored with the use of 1, 2 or 4 bytes, and that is why a special additional header (known as "extradata", "sequence header" or "AVCDecoderConfigurationRecord") is required to be present in the stream to specify how many bytes are used to store the NAL unit's length. The stream then might look as follows: 35 | ``` 36 | ([extradata]) | ([length] NALu) | ([length] NALu) | … 37 | ``` 38 | 39 | The syntax of the extradata is described in MPEG-4 Part 15 "Advanced Video Coding (AVC) file format" section 5.2.4.1: 40 | ``` 41 | aligned(8) class AVCDecoderConfigurationRecord { 42 | unsigned int(8) configurationVersion = 1; 43 | unsigned int(8) AVCProfileIndication; 44 | unsigned int(8) profile_compatibility; 45 | unsigned int(8) AVCLevelIndication; 46 | bit(6) reserved = '111111'b; 47 | unsigned int(2) lengthSizeMinusOne; 48 | bit(3) reserved = '111'b; 49 | unsigned int(5) numOfSequenceParameterSets; 50 | for (i=0; i< numOfSequenceParameterSets; i++) { 51 | unsigned int(16) sequenceParameterSetLength; 52 | bit(8*sequenceParameterSetLength) sequenceParameterSetNALUnit; 53 | } 54 | unsigned int(8) numOfPictureParameterSets; 55 | for (i=0; i< numOfPictureParameterSets; i++) { 56 | unsigned int(16) pictureParameterSetLength; 57 | bit(8*pictureParameterSetLength) pictureParameterSetNALUnit; 58 | } 59 | } 60 | ``` 61 | The value of `lengthSizeMinusOne` field (which can be: 0, 1 or 3) determines how many bytes are used to store a length of NALu (that is, correspondingly, 1, 2 or 4 bytes). The most commonly used one is a 4-bytes size field that allows for NAL units of length up to `2^32-1` bytes. 62 | 63 | 64 | ## NAL unit syntax 65 | 66 | Each of the NAL units that we can find in the stream consists of a number of fields. Each field occupies a particular number of bytes and its value has a given semantics. ITU specifications define a syntax to describe NAL units' structure. We will become familiar with some field types defined within that syntax, as well see some exemplary NAL unit descriptions using that syntax. 67 | 68 | ### Data types 69 | 70 | These are some of the types that can be met in the NAL units syntax (for all of the types available, please refer to chapter 7.2 of the ITU Specification): 71 | * **f(N)** - "fixed pattern" sequence of N bits. I.e. f(8) might be some particular pattern of 8 bits - just like `00001111`. For information on how the pattern looks, you need to look in the semantics description of a given field. 72 | * **u(N)** - an unsigned integer written with the use of N bits - i.e. u(16) means an unsigned integer written with the use of 16 bits. 73 | * **u(v)** - an unsigned integer written with the use of some number of bits. That number is usually specified by the value of some other, previously read field. In order to find out which field that is, you need to refer to the description of the semantics of that field. 74 | * **ue(v)** - unsigned integer of variable size written with *Exponential-Golomb* coding. For more information about that coding you can refer [here](https://en.wikipedia.org/wiki/Exponential-Golomb_coding). 75 | * **se(v)** - same as above, but this time the integer is signed. 76 | 77 | 78 | ### General NAL unit structure 79 | 80 | Below there is a description of each NAL unit's structure. The structure description that you can find in chapter 7.3.1 of the ITU Specification is slightly different (and much more complicated). That is because it also covers the fact that there is a need to get rid of emulation_prevention_bytes. In order to simplify things, we assume that in the structure description we use, the byte stream has already been converted to RBSP. 81 | 82 | | nal_unit( NumBytesInNALunit ) | Descriptor | 83 | |------------------------------------------------------- |------------| 84 | |{ | | 85 | |      forbidden_zero_bit |f(1) | 86 | |      nal_ref_idc |u(2) | 87 | |      nal_unit_type |u(5) | 88 | |      nal_unit_data() | | 89 | |} | | 90 | 91 | As you can see, each NAL unit starts with a single byte being a kind of a NAL unit header. Its first bit (`forbidden_zero_bit`) is always equal to 0, then comes a `nal_ref_idc` field consisting of two bits, and finally `nal_unit_type` - an unsigned integer written with the use of 5 bits, which determines the type of NAL unit. 92 | 93 | ## NALu types 94 | 95 | Each NAL unit can be of a different type. In this chapter we will describe only some of the NAL units types, while all the possible types are listed in the table available in "Appendix/NALu types". 96 | 97 | There are two "classes" of NAL units types defined in ITU-T Specification's Annex A - **VCL** and **non-VCL** NAL units. The first one holds the encoded video, while the other does not contain video data at all. In our case, we found the following NAL unit types especially helpful: 98 | * VCL: 99 | * *Coded slice of a non-IDR picture* (**non-IDR**) - contains a part or a complete non-keyframe (that is: P-frame or a B-frame) 100 | * *Coded slice of an IDR picture* (**IDR**) - contains a part or a complete keyframe (also known as I-frame). The name IDR (that stands for *instantaneous decoding refresh*) originates from the fact that the decoder can "forget" the previous frames when the new keyframe appears, since it contains the complete information about the frame. 101 | * non-VCL: 102 | * *Sequence parameter set* (**SPS**) - contains metadata that is applicable to one or more *coded video sequences*. In that NALu you will find information allowing you to calculate the video resolution or H.264 profile. 103 | * *Picture parameter set* (**PPS**) - contains metadata applicable to one or more *coded pictures* 104 | * *Access unit delimiter* (**AUD**) - just a separator between *access units* 105 | * *Supplemental enhancement information* (**SEI**) - contains some additional metadata that "assist in processes related to decoding, display or other purposes". At the same time, information stored in SEI is not required to restore the picture during the decoding process, so the decoders are not obliged to process SEI. In fact, SEI is defined as Annex D to the ITU specification. 106 | 107 | Some of the terms used in these (especially the ones written with *curved* font) might not be known to you yet - don't worry, keep on reading as they will be explained later. Once that happens, we invite you to go back to that section to reread it ;) 108 | -------------------------------------------------------------------------------- /h264/4_H264_stream_structure.md: -------------------------------------------------------------------------------- 1 | A video is a sequence of *frames* - images. Each frame might be (but it's not obliged to be!) written with [interlacing](https://en.wikipedia.org/wiki/Interlaced_video) - which means: it might consist of two *fields*. In such a case, a *field* is a group of alternating rows of a frame (even or odd ones). The two fields that form a single frame are referred to as the *bottom field* and the *top field*. 2 | 3 | In H.264, the term *picture* refers to both a single field of a frame, or to the frame itself. 4 | 5 | As noted previously, a H.264 stream consists of NAL units. Some NAL units hold the compressed video, while the others hold some metadata. The NAL units that hold video contain it in the form of a *slice*. Each slice is a group of *macroblocks* (along with a so-called *slice header*, containing some metadata). Right now it's sufficient for you to think about a macroblock as an area of 16x16 pixels on an image. Each picture is partitioned into macroblocks and such a division allows intra-frame prediction, as the translation vectors are defined between macroblocks of two pictures. We have reached the bottom - a macroblock is an atomic term in the context of H.264. 6 | 7 | Getting back to the slices - one or more slices create a *coded picture* - a compressed representation of a picture or a part of a picture (remember that a picture is a single video frame or a field). A coded picture might be one of two types: 8 | * a *primary coded picture* - a coded representation of a whole picture - it contains all the macroblocks of a picture (that means, that it's enough for a decoder to be able to display a picture based just on the primary coded picture) 9 | * a *redundant coded picture* - a coded representation of a part of or a whole picture - as the name suggests ("redundant"), it repeats some information contained by a given primary coded picture 10 | With the primary coded picture defined, we can define another crucial term used in H.264 - *access unit*. 11 | An access unit is a sequence of NAL units that contain exactly one primary coded picture. Apart from the primary coded picture, a single access unit might also contain some *redundant coded pictures* (as well as some other video data or metadata, but let's not dig into the details). The crucial thing is that decoding a single access unit allows one to obtain a single *decoded picture*. 12 | Below there is a scheme that shows a possible structure of an access unit: 13 | ![AU Scheme](assets/images/AU_scheme.png) 14 |
15 | 16 | As shown, an access unit might or might not start with a special type of NALu - access unit delimiter. Then comes zero or more SEI NAL units. Finally, there is the only obligatory part of the access unit - the sequence of NAL units that form a primary coded picture. Then comes zero or more sequences of NAL units forming redundant coded pictures. Then there is optionally the end of sequence NAL unit (it might help in separating the coded video sequences - you will learn what a coded video sequence is very soon). Finally, the end of stream NAL unit might optionally be present if the stream ends (but in the same manner as previously, it's not obligatory - the stream might end without the End of stream NAL unit appearing). 17 | The access unit might be of two types: 18 | * IDR access unit - an access unit, in which primary coded picture is an IDR picture 19 | * non-IDR access unit - an access unit, in which primary coded picture is not an IDR picture 20 | Finally, we can define a coded video sequence - a sequence of access units that consists of exactly one IDR access unit at the beginning of the sequence, followed by zero or more non-IDR access units. 21 | 22 | ## Example - H.264 stream structure 23 | 24 | Below there is an exemplary H.264 stream consisting of multiple NAL units. The NAL units are forming Access Units, and the Access Units are gathered together in the form of a Coded Video Sequence. 25 | ![H264 Stream Example](assets/images/H264_example_stream.png) 26 |
27 | 28 | Let's thoroughly inspect that stream's structure. 29 | First, there is a Sequence Parameter Set NAL. It contains some metadata common for zero or more coded video sequences. 30 | It's followed by two Picture Parameter Set NAL units - each of them contains some metadata relevant for zero or more coded pictures. In the slice's header of each NALu holding the video data (that is - IDR NALu or non-IDR NALu), there is a reference to the particular Picture Parameter Set NALu. Note that all the slices forming a single coded picture must refer to the same Picture Parameter Set NALu. In each Picture Parameter Set NALu, there is a reference to a Sequence Parameter Set - all the Picture Parameter Sets referred to by coded pictures from the same coded video sequence must refer to the same Sequence Parameter Set. 31 | Right after the second Picture Parameter Set NALu a coded video sequence starts. It consists of three access units - the first one is an IDR access unit, while the remaining two are non-IDR access units. 32 | The first access unit starts with an optional SEI NALu, followed by two IDR NAL units. These two IDR NAL units form a single primary coded picture. The second access unit consists of two non-IDR NAL units, which also form a single primary coded video. That second access unit is probably much smaller in size compared to the first access unit - that is because it doesn't contain a keyframe. The last access unit starts with an optional SEI NALu, followed by a non-IDR NALu (which holds a complete primary coded picture) and an optional redundant coded picture that is contained in a single non-IDR NALu. 33 | -------------------------------------------------------------------------------- /h264/6_Appendix.md: -------------------------------------------------------------------------------- 1 | | nal_unit_type | Content of NAL unit & RBSP syntax structure | NAL unit type class [Annex A] | 2 | | ------------- | -------------------------------------------------------------------------------------------------------- | ------------------------------ | 3 | | 0 | Unspecified | non-VCL | 4 | | 1 | Coded slice of a non-IDR picture *slice_layer_without_partitioning_rbsp()* | VCL | 5 | | 2 | Coded slice data partition A *slice_data_partition_a_layer_rbsp()* | VCL | 6 | | 3 | Coded slice data partition B *slice_data_partition_b_layer_rbsp()* | VCL | 7 | | 4 | Coded slice data partition C *slice_data_partition_c_layer_rbsp()* | VCL | 8 | | 5 | Coded slice of an IDR picture *slice_layer_without_partitioning_rbsp()* | VCL | 9 | | 6 | Supplemental enhancement information (SEI) *sei_rbsp()* | non-VCL | 10 | | 7 | Sequence parameter set *seq_parameter_set_rbsp()* | non-VCL | 11 | | 8 | Picture parameter set *pic_parameter_set_rbsp()* | non-VCL | 12 | | 9 | Access unit delimiter *access_unit_delimiter_rbsp()* | non-VCL | 13 | | 10 | End of sequence *end_of_seq_rbsp()* | non-VCL | 14 | | 11 | End of stream *end_of_stream_rbsp()* | non-VCL | 15 | | 12 | Filler data *filler_data_rbsp()* | non-VCL | 16 | | 13 | Sequence parameter set extension *seq_parameter_set_extension_rbsp()* | non-VCL | 17 | | 14 | Prefix NAL unit *prefix_nal_unit_rbsp()* | non-VCL | 18 | | 15 | Subset sequence parameter set *subset_seq_parameter_set_rbsp()* | non-VCL | 19 | | 16 - 18 | Reserved | non-VCL | 20 | | 19 | Coded slice of an auxiliary coded picture without partitioning *slice_layer_without_partitioning_rbsp()* | non-VCL | 21 | | 20 | Coded slice extension *slice_layer_extension_rbsp()* | non-VCL | 22 | | 21 | Coded slice extension for depth view components *slice_layer_extension_rbsp()* (specified in Annex I) | non-VCL | 23 | | 22 - 23 | Reserved | non-VCL | 24 | | 24 - 31 | Unspecified | non-VCL | -------------------------------------------------------------------------------- /h264/assets/images/AU_scheme.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/membraneframework/membrane_tutorials/3e676855eeeab0da495da549133c4886b3608d16/h264/assets/images/AU_scheme.png -------------------------------------------------------------------------------- /h264/assets/images/H264_example_stream.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/membraneframework/membrane_tutorials/3e676855eeeab0da495da549133c4886b3608d16/h264/assets/images/H264_example_stream.png -------------------------------------------------------------------------------- /h264/assets/images/VCL_vs_NAL.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/membraneframework/membrane_tutorials/3e676855eeeab0da495da549133c4886b3608d16/h264/assets/images/VCL_vs_NAL.png -------------------------------------------------------------------------------- /h264/index.md: -------------------------------------------------------------------------------- 1 | --- 2 | title: H264 - what, why and how 3 | 4 | description: In this tutorial we will describe what is H264, why to use it and how to fetch interesting information from the H264 stream 5 | part: 9 6 | graphicPath: 7 | --- 8 | 9 | | number | title | file | 10 | | ------ | -------------------------------- | ------------------------------------- | 11 | | 1 | Introduction | 1_Introduction.md | 12 | | 2 | What is H264 | 2_What_is_H264.md | 13 | | 3 | H264 NAL units | 3_H264_NAL_units.md | 14 | | 4 | H264 stream structure | 4_H264_stream_structure.md | 15 | | 5 | Fetching information from stream | 5_Fetching_information_from_stream.md | 16 | | 6 | Appendix | 6_Appendix.md | 17 | 18 | --------------------------------------------------------------------------------