├── tools ├── __init__.py ├── logging_pad_probe.py ├── application_init.py └── runner.py ├── requirements.txt ├── 05-add-and-remove-network-sink.md ├── .gitignore ├── 01-add-source.py ├── 04-add-and-remove-network-source.md ├── 06-link-and-unlink-element.py ├── 03-add-and-remove-source.md ├── 01-add-source.md ├── 02-add-network-source.py ├── 03-add-and-remove-source.py ├── README.md ├── 05-add-and-remove-network-sink.py ├── 04-add-and-remove-network-source.py └── 02-add-network-source.md /tools/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | coloredlogs==10.0 2 | humanfriendly==4.18 3 | ruamel.yaml==0.15.94 4 | vext==0.7.3 5 | vext.gi==0.7.0 6 | -------------------------------------------------------------------------------- /tools/logging_pad_probe.py: -------------------------------------------------------------------------------- 1 | import datetime 2 | import logging 3 | 4 | from gi.repository import Gst 5 | 6 | log = logging.getLogger("Pad-Probe") 7 | 8 | 9 | def logging_pad_probe(pad, probeinfo, location): 10 | pts_nanpseconds = probeinfo.get_buffer().pts 11 | pts_timedelta = datetime.timedelta(microseconds=pts_nanpseconds / 1000) 12 | log.debug("PTS at %s = %s", '{:>20s}'.format(location), pts_timedelta) 13 | return Gst.PadProbeReturn.OK 14 | -------------------------------------------------------------------------------- /tools/application_init.py: -------------------------------------------------------------------------------- 1 | import logging 2 | import signal 3 | 4 | import sys 5 | 6 | log = logging.getLogger('Application Init') 7 | 8 | MIN_GST = (1, 10) 9 | MIN_PYTHON = (3, 5) 10 | 11 | 12 | def application_init(): 13 | import coloredlogs 14 | coloredlogs.install( 15 | level='DEBUG', 16 | fmt="%(asctime)s %(name)s@%(threadName)s %(levelname)s %(message)s") 17 | 18 | log.debug('importing gi') 19 | import gi 20 | 21 | log.debug('importing gi.Gst, gi.GObject') 22 | gi.require_version('Gst', '1.0') 23 | gi.require_version('GstNet', '1.0') 24 | 25 | from gi.repository import Gst, GObject 26 | 27 | log.debug('Gst.init') 28 | Gst.init([]) 29 | 30 | log.debug('version check') 31 | if Gst.version() < MIN_GST: 32 | raise Exception('GStreamer version', Gst.version(), 33 | 'is too old, at least', MIN_GST, 'is required') 34 | 35 | if sys.version_info < MIN_PYTHON: 36 | raise Exception('Python version', sys.version_info, 37 | 'is too old, at least', MIN_PYTHON, 'is required') 38 | 39 | log.debug('GObject.threads_init') 40 | GObject.threads_init() 41 | 42 | 43 | def set_sigint_handler(sigint_callback): 44 | logging.debug('setting SIGINT handler') 45 | from gi.repository import GLib 46 | GLib.unix_signal_add(GLib.PRIORITY_DEFAULT, signal.SIGINT, sigint_callback) 47 | -------------------------------------------------------------------------------- /05-add-and-remove-network-sink.md: -------------------------------------------------------------------------------- 1 | ## Adding and Removing RTP-Sinks 2 | This Example creates a Pipeline with an `audiotestsrc`, a `tee` and an internal `autoaudiosink`. 3 | After 2, 4, and 6 seconds a Bin is added and linked to the `tee`-Element. The Bin contains all Elements necessary to 4 | transmit the Audio to the Network as an RTP Stream. At 8, 10 and 12 Seconds one of the Bins is disabled and removed from 5 | the Pipeline. After this, the Process starts over again. 6 | 7 | ``` 8 | gst-launch-1.0 udpsrc port=15000 !\ 9 | application/x-rtp,clock-rate=48000,media=audio,encoding-name=L16,channels=2 ! \ 10 | rtpjitterbuffer latency=30 drop-on-latency=true ! \ 11 | rtpL16depay ! \ 12 | audio/x-raw,format=S16BE,rate=48000,channels=2 ! \ 13 | audioconvert ! \ 14 | audio/x-raw,format=S16LE,rate=48000,channels=2 ! \ 15 | autoaudiosink 16 | ``` 17 | 18 | (!) Brings Pipeline to Paused state 19 | 20 | You should read [Adding and Removing RTP-Sources](04-add-and-remove-network-source.md) before this, because important 21 | Aspects that have been explained there are not repeated here. 22 | 23 | This Experiment is very similar to [Adding and Removing RTP-Sources](04-add-and-remove-network-source.md), but the most 24 | important differences are highlighted as such: 25 | 26 | 1. tee & allow-not-linked 27 | 2. optional internal playback sink 28 | 3. queue after tee 29 | 4. network host & port 30 | 5. blocking pad probe 31 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | # Created by .ignore support plugin (hsz.mobi) 2 | ### Python template 3 | # Byte-compiled / optimized / DLL files 4 | __pycache__/ 5 | *.py[cod] 6 | *$py.class 7 | 8 | # C extensions 9 | *.so 10 | 11 | # Distribution / packaging 12 | .Python 13 | build/ 14 | develop-eggs/ 15 | dist/ 16 | downloads/ 17 | eggs/ 18 | .eggs/ 19 | lib/ 20 | lib64/ 21 | parts/ 22 | sdist/ 23 | var/ 24 | wheels/ 25 | *.egg-info/ 26 | .installed.cfg 27 | *.egg 28 | MANIFEST 29 | 30 | # PyInstaller 31 | # Usually these files are written by a python script from a template 32 | # before PyInstaller builds the exe, so as to inject date/other infos into it. 33 | *.manifest 34 | *.spec 35 | 36 | # Installer logs 37 | pip-log.txt 38 | pip-delete-this-directory.txt 39 | 40 | # Unit test / coverage reports 41 | htmlcov/ 42 | .tox/ 43 | .coverage 44 | .coverage.* 45 | .cache 46 | nosetests.xml 47 | coverage.xml 48 | *.cover 49 | .hypothesis/ 50 | .pytest_cache/ 51 | 52 | # Translations 53 | *.mo 54 | *.pot 55 | 56 | # Django stuff: 57 | *.log 58 | local_settings.py 59 | db.sqlite3 60 | 61 | # Flask stuff: 62 | instance/ 63 | .webassets-cache 64 | 65 | # Scrapy stuff: 66 | .scrapy 67 | 68 | # Sphinx documentation 69 | docs/_build/ 70 | 71 | # PyBuilder 72 | target/ 73 | 74 | # Jupyter Notebook 75 | .ipynb_checkpoints 76 | 77 | # pyenv 78 | .python-version 79 | 80 | # celery beat schedule file 81 | celerybeat-schedule 82 | 83 | # SageMath parsed files 84 | *.sage.py 85 | 86 | # Environments 87 | .env 88 | .venv 89 | env/ 90 | venv/ 91 | ENV/ 92 | env.bak/ 93 | venv.bak/ 94 | 95 | # Spyder project settings 96 | .spyderproject 97 | .spyproject 98 | 99 | # Rope project settings 100 | .ropeproject 101 | 102 | # mkdocs documentation 103 | /site 104 | 105 | # mypy 106 | .mypy_cache/ 107 | 108 | # PyCharm project folders 109 | /.idea 110 | 111 | # Dot-Files 112 | *.dot 113 | -------------------------------------------------------------------------------- /01-add-source.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | import logging 3 | from threading import Thread, Event 4 | 5 | from tools.application_init import application_init 6 | 7 | application_init() 8 | 9 | from gi.repository import Gst, GLib 10 | from tools.logging_pad_probe import logging_pad_probe 11 | from tools.runner import Runner 12 | 13 | log = logging.getLogger("main") 14 | 15 | log.info("building pipeline") 16 | pipeline = Gst.Pipeline.new() 17 | caps = Gst.Caps.from_string("audio/x-raw,format=S16LE,rate=48000,channels=2") 18 | 19 | testsrc1 = Gst.ElementFactory.make("audiotestsrc", "testsrc1") 20 | testsrc1.set_property("is-live", True) # (3) 21 | testsrc1.set_property("freq", 220) 22 | pipeline.add(testsrc1) 23 | 24 | mixer = Gst.ElementFactory.make("audiomixer") 25 | pipeline.add(mixer) 26 | testsrc1.link_filtered(mixer, caps) 27 | 28 | sink = Gst.ElementFactory.make("autoaudiosink") 29 | pipeline.add(sink) 30 | mixer.link_filtered(sink, caps) 31 | 32 | testsrc1.get_static_pad("src").add_probe( 33 | Gst.PadProbeType.BUFFER, logging_pad_probe, "testsrc1-output") 34 | 35 | mixer.get_static_pad("src").add_probe( 36 | Gst.PadProbeType.BUFFER, logging_pad_probe, "mixer-output") 37 | 38 | 39 | def add_new_src(): 40 | log.info("Adding testsrc2") 41 | testsrc2 = Gst.ElementFactory.make("audiotestsrc", "testsrc2") 42 | testsrc2.set_property("freq", 440) 43 | testsrc2.set_property("is-live", True) # (2) 44 | 45 | testsrc2.get_static_pad("src").add_probe( 46 | Gst.PadProbeType.BUFFER, logging_pad_probe, "testsrc2-output") 47 | 48 | pipeline.add(testsrc2) 49 | testsrc2.link_filtered(mixer, caps) 50 | testsrc2.sync_state_with_parent() # (4) 51 | log.info("Adding testsrc2 done") 52 | 53 | 54 | stop_event = Event() 55 | 56 | 57 | def timed_sequence(): 58 | log.info("Starting Sequence") 59 | if stop_event.wait(2): return 60 | GLib.idle_add(add_new_src) # (1) 61 | log.info("Sequence ended") 62 | 63 | 64 | t = Thread(target=timed_sequence, name="Sequence") 65 | t.start() 66 | 67 | runner = Runner(pipeline) 68 | runner.run_blocking() 69 | 70 | stop_event.set() 71 | t.join() 72 | -------------------------------------------------------------------------------- /tools/runner.py: -------------------------------------------------------------------------------- 1 | import logging 2 | 3 | from gi.repository import Gst, GObject 4 | 5 | log = logging.getLogger('Runner') 6 | 7 | 8 | class Runner(object): 9 | def __init__(self, pipeline, error_callback=None): 10 | self.mainloop = GObject.MainLoop() 11 | self.pipeline = pipeline 12 | self.error_callback = error_callback or self.quit 13 | 14 | def run_blocking(self): 15 | self.configure() 16 | self.set_playing() 17 | 18 | try: 19 | self.mainloop.run() 20 | except KeyboardInterrupt: 21 | print('Terminated via Ctrl-C') 22 | 23 | self.set_null() 24 | 25 | def configure(self): 26 | log.debug('configuring pipeline') 27 | bus = self.pipeline.bus 28 | 29 | bus.add_signal_watch() 30 | bus.connect("message::eos", self.on_eos) 31 | bus.connect("message::error", self.on_error) 32 | bus.connect("message::state-changed", self.on_state_change) 33 | 34 | def on_eos(self, _bus, message): 35 | log.error("EOS from %s (at %s)", 36 | message.src.name, message.src.get_path_string()) 37 | self.error_callback() 38 | 39 | def on_error(self, _bus, message): 40 | (error, debug) = message.parse_error() 41 | log.error("Error from %s (at %s)\n%s (%s)", 42 | message.src.name, message.src.get_path_string(), error, debug) 43 | self.error_callback() 44 | 45 | def quit(self): 46 | log.warning('quitting mainloop') 47 | self.mainloop.quit() 48 | 49 | def on_state_change(self, _bus, message): 50 | old_state, new_state, pending = message.parse_state_changed() 51 | if message.src == self.pipeline: 52 | log.info("Pipeline: State-Change from %s to %s; pending %s", 53 | old_state.value_name, new_state.value_name, pending.value_name) 54 | else: 55 | log.debug("%s: State-Change from %s to %s; pending %s", 56 | message.src.name, old_state.value_name, new_state.value_name, pending.value_name) 57 | 58 | def set_playing(self): 59 | log.info('requesting state-change to PLAYING') 60 | self.pipeline.set_state(Gst.State.PLAYING) 61 | 62 | def set_null(self): 63 | log.info('requesting state-change to NULL') 64 | self.pipeline.set_state(Gst.State.NULL) 65 | -------------------------------------------------------------------------------- /04-add-and-remove-network-source.md: -------------------------------------------------------------------------------- 1 | ## Adding and Removing RTP-Sources 2 | As a more realistic example of adding and removing Sources to a playing Pipeline, this Example creates a Pipeline with an 3 | `audiotestsrc` and an `audiomixer`. After a while, a Bin (a Cluster of Elements) which receives and decodes Audio coming 4 | from the Network via RTP is created, added to the Pipeline and linked to the `audiomixer`. After 4 and 6 seconds additional 5 | Bins of this kind are created and also linked. Then, again with 2 Seconds in between, the Source-Bins are removes again, 6 | before thw whole process starts over again. 7 | 8 | Network-Sources can, for example. look like the following Pipeline. If you want, you can start such Pipelines on different 9 | Computers across the Network, just adjust the destination IP-Address. 10 | ``` 11 | gst-launch-1.0 audiotestsrc freq=220 is-live=true ! \ 12 | audio/x-raw,format=S16BE,rate=48000,channels=2 ! \ 13 | rtpL16pay ! \ 14 | application/x-rtp,clock-rate=48000,media=audio,encoding-name=L16,channels=2 ! \ 15 | udpsink host=127.0.0.1 port=10000 16 | ``` 17 | 18 | You should read [Adding RTP Network-Sources](02-add-network-source.md) and [Adding and Removing Sources](03-add-and-remove-source.md) 19 | before this, because important Aspects that have been explained there are not repeated here. 20 | 21 | The most important new Lines have been marked as such: 22 | 23 | 1. In this Example we create a Bin to collect all the Elements that comprise the RTP-Source and manage them together. 24 | This also helps avoid some of the Problems described in [Adding and Removing Sources](03-add-and-remove-source.md). 25 | We also give the bin a unique name, so that it can be found by name again later. We could also store the Reference 26 | to the bin, depending on the use-case. 27 | 28 | 2. We selecl the Src-Pad of the last Element in the Bin and explicitly create a Ghost-Pad for it. This pad is added as 29 | Src-Pad to the Bin. 30 | 31 | 3. Because the Bin now has an unlinked Src-Pad, we can just use `Element.link` to link it to the Audio-Mixer. 32 | 33 | 4. Upon removing we select the Bin's src-Pad by its name and thr select the Pad's Peer - which we know is the Mixer-Pad. 34 | This reference is kept for releasing the Mixer-Pad later. 35 | 36 | 5. Before removing the Bin we set its State to `NULL`. This propagates the State-Change to its Children in the correct 37 | order. Then the Bin is removed, which removes all of its Children, too. Finally the requested Mixer-Pad is released. 38 | -------------------------------------------------------------------------------- /06-link-and-unlink-element.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | import logging 3 | from threading import Thread, Event 4 | 5 | from tools.application_init import application_init 6 | 7 | application_init() 8 | 9 | from gi.repository import Gst, GLib 10 | from tools.runner import Runner 11 | 12 | log = logging.getLogger("main") 13 | 14 | log.info("building pipeline") 15 | pipeline = Gst.Pipeline.new() 16 | caps_audio = Gst.Caps.from_string("audio/x-raw,format=S16LE,rate=48000,channels=2,layout=interleaved") 17 | caps_audio_be = Gst.Caps.from_string("audio/x-raw,format=S16BE,rate=48000,channels=2") 18 | caps_rtp = Gst.Caps.from_string("application/x-rtp,clock-rate=48000,media=audio,encoding-name=L16,channels=2") 19 | 20 | # audiotestsrc freq=220 ! audiomixer name=mix ! autoaudiosink 21 | # audiotestsrc freq=440 ! fakesink 22 | ## 23 | 24 | testsrc1 = Gst.ElementFactory.make("audiotestsrc", "testsrc1") 25 | testsrc1.set_property("is-live", True) 26 | testsrc1.set_property("freq", 220) 27 | pipeline.add(testsrc1) 28 | 29 | mixer = Gst.ElementFactory.make("audiomixer") 30 | pipeline.add(mixer) 31 | testsrc1.link_filtered(mixer, caps_audio) 32 | 33 | sink = Gst.ElementFactory.make("autoaudiosink") 34 | pipeline.add(sink) 35 | mixer.link(sink) 36 | 37 | testsrc2 = Gst.ElementFactory.make("audiotestsrc", "testsrc2") 38 | testsrc2.set_property("is-live", True) 39 | testsrc2.set_property("freq", 440) 40 | pipeline.add(testsrc2) 41 | 42 | tee = Gst.ElementFactory.make("tee") 43 | tee.set_property("allow-not-linked", True) 44 | pipeline.add(tee) 45 | testsrc2.link_filtered(tee, caps_audio) 46 | 47 | 48 | def link_element(): 49 | Gst.debug_bin_to_dot_file_with_ts(pipeline, Gst.DebugGraphDetails.ALL, "link_element_before") 50 | 51 | log.info("Requesting Tee-Pad") 52 | tee_pad_templ = tee.get_pad_template("src_%u") 53 | tee_pad = tee.request_pad(tee_pad_templ) 54 | log.debug(tee_pad) 55 | 56 | log.info("Requesting Mixer-Pad") 57 | mixer_pad_templ = mixer.get_pad_template("sink_%u") 58 | mixer_pad = mixer.request_pad(mixer_pad_templ) 59 | log.debug(mixer_pad) 60 | 61 | log.info("Linking Tee-Pad to Mixer-Pad") 62 | log.debug(tee_pad.link(mixer_pad)) 63 | 64 | Gst.debug_bin_to_dot_file_with_ts(pipeline, Gst.DebugGraphDetails.ALL, "link_element_after") 65 | 66 | 67 | def unlink_element(): 68 | Gst.debug_bin_to_dot_file_with_ts(pipeline, Gst.DebugGraphDetails.ALL, "link_element_before") 69 | 70 | log.info("Unlinking tee from mixer") 71 | for tee_pad in tee.srcpads: 72 | mixer_pad = tee_pad.get_peer() 73 | 74 | log.info("Unlinking pads %s and %s" % (tee_pad, mixer_pad)) 75 | log.debug(tee_pad.unlink(mixer_pad)) 76 | 77 | log.info("Releasing pad %s" % tee_pad) 78 | log.debug(tee.release_request_pad(tee_pad)) 79 | 80 | log.info("Releasing pad %s" % mixer_pad) 81 | log.debug(mixer.release_request_pad(mixer_pad)) 82 | 83 | Gst.debug_bin_to_dot_file_with_ts(pipeline, Gst.DebugGraphDetails.ALL, "link_element_after") 84 | 85 | 86 | stop_event = Event() 87 | 88 | 89 | def timed_sequence(): 90 | log.info("Starting Sequence") 91 | 92 | while True: 93 | if stop_event.wait(2): return 94 | GLib.idle_add(link_element) 95 | 96 | if stop_event.wait(2): return 97 | GLib.idle_add(unlink_element) 98 | 99 | 100 | t = Thread(target=timed_sequence, name="Sequence") 101 | t.start() 102 | 103 | runner = Runner(pipeline) 104 | runner.run_blocking() 105 | 106 | stop_event.set() 107 | t.join() 108 | -------------------------------------------------------------------------------- /03-add-and-remove-source.md: -------------------------------------------------------------------------------- 1 | # Adding and Removing Sources 2 | - [Sourcecode](03-add-and-remove-source.py) 3 | 4 | This example is based upon [01-add-source.py](01-add-source.py). A Pipeline with a live `audiotestsrc`, an `audiomixer` 5 | and an `autoaudiosink` is created. After 2 seconds, a second `audiotestsrc` is created, added to the pipeline and linked 6 | to the Mixer. Another 2 seconds later it is stopped, unlinked and removed again. The process repeats as long as the 7 | experiment runs. 8 | 9 | You should read [Adding a Source](01-add-source.md) before this, because important Aspects that have been explained there 10 | are not repeated here. 11 | 12 | The most important new Lines have been marked as such: 13 | 14 | 1. Because the Sequencing is done in a background thread and the Sequence will run over and over again, we need a way 15 | to stop the Sequence when the Main-Program terminates. In Python it is best to use the `threading.Event` Class and its 16 | `wait`-Method to achieve this. 17 | 18 | 2. We want to remove all Elements that are added. One way to do this is to keep references to the actual Elements, like 19 | we do in these global variables. A even better way would be to collect the added Elements in a `Bin`, a Cluster of 20 | Elements, which can be added and removed as a whole. See [Adding and Removing RTP-Sources](04-add-and-remove-network-source.py) 21 | for an Example of using a Bin. 22 | 23 | 3. In [Adding a Source](01-add-source.py) we used `Element.link_filtered` to create a similar link between the 24 | `audiotestsrc` and the `audiomixer`. `link_filtered` creates and adds `capsfilter`-Element for us behind the scenes 25 | and places it between the Test-Source and the Audiomixer. When we later try to remove the Test-Source, the 26 | `capsfilter`-Element would not automatically removed for us and we would need to figure out a way to get hold of a 27 | reference to this Element. 28 | 29 | To avoid that, this Example explicitly created a `capsfilter` and links it, so we already have a Reference to it. 30 | Another Option, explored in the next Experiment [Adding and Removing RTP-Sources](04-add-and-remove-network-source.py) 31 | is to use a Bin. The automatically created `capsfilter`-Element would then be a Child of that Bin and removed when 32 | the Bin is removed. 33 | 34 | For the same reason a Sink-Pad on the Mixer is requested explicitly. Usually a call to `Element.link` targeting 35 | an Element with request-Pads will automatically request a Pad for you, but then you need to figure out a way to 36 | explicitly select this pad later, when you intend to release the requested Mixer-Pad. 37 | 38 | 4. Debugging these Scenarios can be quite complex. `Gst.debug_bin_to_dot_file_with_ts` is a Utility-Method which 39 | generates a [GraphViz](https://www.graphviz.org/)-File in the directory named in the Environment-Variable 40 | `GST_DEBUG_DUMP_DOT_DIR`. To generate such files, you can run the Experiment like this: 41 | ``` 42 | GST_DEBUG_DUMP_DOT_DIR=. ./04-add-and-remove-network-source.py 43 | ``` 44 | 45 | To view theses Files, which can become rather large, I suggest [xdot](https://github.com/jrfonseca/xdot.py) which 46 | can be used on all major OSes and is really handy to view and examine these large dot-File graphs. 47 | 48 | 5. Actually removing a Source is rather straight forward. First the State of all involved Elements is set to `NULL`, 49 | starting with the Source and moving down the Buffer-Flow. This will release any Resources held by the Elements and 50 | ensures that the Sources does not try to push new Buffers into an Element that has already been stopped. 51 | 52 | Next the Elements are removed from the Pipeline in the same order. This will also remove the Links between them. 53 | 54 | Lastly the Sink-Pad will be released to the Mixer. 55 | -------------------------------------------------------------------------------- /01-add-source.md: -------------------------------------------------------------------------------- 1 | # Adding a Source 2 | → [Sourcecode](01-add-source.py) 3 | 4 | This Example creates a Pipeline like this: 5 | 6 | ``` 7 | audiotestsrc is-live=true ! audiomixer ! autoaudiosink 8 | ``` 9 | 10 | It installs Pad-Probes after the audiotestsrc and the audiomixer, which log the PTS-Timestamps of the Buffers flowing 11 | through the Element's src-Pads. 12 | 13 | After 2 seconds, another audiotestsrc is created with such a Pad-Probe and linked to the audiomixer. 14 | 15 | On the Speaker/Headphone you can hear the 220 Hz-Zone from the first audiotestsrc, which after 2 Seconds gets mixed with 16 | the 440 Hz Tone from the second audiotestsrc. 17 | 18 | On the Console you can see that the Timestamps from the second audiotestsrc start around the 0:00:02 seconds mark, 19 | because we configured it to be a true live-source. 20 | 21 | The most important Lines have been marked as such: 22 | 23 | 1. We schedule the execution on the GLib Event-Loop by using `Glib.idle_add`. On the Console you can see, that the 24 | Sequence runs in its own `Sequence`-Thread but the second Test-Source is actually added from the `MainThread`. 25 | 26 | It seems that this exact example also works without this, because some of the Methods are Thread-Safe by 27 | Documentation (ie. [Bin.add](https://lazka.github.io/pgi-docs/#Gst-1.0/classes/Bin.html#Gst.Bin.add)) and some are 28 | by luck (ie [GObject.Object.set_property](https://lazka.github.io/pgi-docs/#GObject-2.0/classes/Object.html#GObject.Object.set_property)), 29 | but not following this pattern can lead to unexpected and hard to find issues. 30 | 31 | From the console you can also see, that the Pad-Probes are called synchronously from their respective Elements' 32 | Streaming-Threads (Shown as `Dummy-[n]`). 33 | 34 | 2. Forcing the `audiotestsrc` to be a Live-Source makes it produce Buffers with timestamps starting at the current 35 | Running-Time of the Pipeline. A Non-Live-Source would start ad 0:00:00 and run faster then the other sources, 36 | until it would have caught up to the current Running-Time. 37 | 38 | Try to comment the Line marked with `(2)` out and look at the Console-Output around the 0:00:02 seconds mark 39 | 40 | 3. Furthermore it is important that all other Sources in the Pipeline are also in Live-Mode, here this is ensured 41 | by setting the `is-live` property of `testsrc1`. If one of the Sources connected to the same downstream Elements 42 | (here for example the `audsiomixer`) are not live, they will produce as many buffers as the sink allows them to. 43 | In a pipeline like the following, where neither source nor sink enforces live-behaviour, the timestamp the 44 | audiomixer is working with might be way ahead of those produced by your newly added live-source: 45 | `audiotestsrc ! audiomixer ! wavenc ! filesink …` 46 | 47 | But even if your sink is live, like an `alsasink` for example, the timestamp at the `audiomixer` might be multiple 48 | seconds ahead of your new Live-Source. GStreamer will then halt the downstream elements until you new Live-Source 49 | has caught up, possibly dropping data from other Live-Sources and not sending any Buffers out to the sink. 50 | 51 | If you are dealing with sources that do not support live behaviour, for example a `filesrc`, you should place an 52 | `identity`-Element with the `sync`-Property set to True right after it, so that it behaves like a live-source to 53 | downstream elements like `audiomixer`s. 54 | 55 | 4. An Element does not automatically take over its parent state. Also, not all Elements in a Pipeline have to have the 56 | same state. In this case the new `audiotestsrc`-Element starts in `NULL` state and is added as such to the Pipeline. 57 | Once it is added, its state is synced to the pipeline (`PLAYING`) which makes it switch from `NULL` through 58 | `READY` and `PAUSED` to `PLAYING`, where it then starts generating buffers and sending them downstream to the 59 | `audiomixer`. 60 | -------------------------------------------------------------------------------- /02-add-network-source.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | import logging 3 | from threading import Thread, Event 4 | 5 | rtp_max_jitter_mx = 30 6 | 7 | from tools.application_init import application_init 8 | 9 | application_init() 10 | 11 | from gi.repository import Gst, GLib 12 | from tools.logging_pad_probe import logging_pad_probe 13 | from tools.runner import Runner 14 | 15 | log = logging.getLogger("main") 16 | 17 | log.info("building pipeline") 18 | pipeline = Gst.Pipeline.new() 19 | caps_audio = Gst.Caps.from_string("audio/x-raw,format=S16LE,rate=48000,channels=2") # (11) 20 | caps_audio_be = Gst.Caps.from_string("audio/x-raw,format=S16BE,rate=48000,channels=2") 21 | caps_rtp = Gst.Caps.from_string("application/x-rtp,clock-rate=48000,media=audio,encoding-name=L16,channels=2") 22 | 23 | testsrc = Gst.ElementFactory.make("audiotestsrc", "testsrc") 24 | testsrc.set_property("is-live", True) # (2) 25 | testsrc.set_property("freq", 110) 26 | testsrc.set_property("volume", 0.5) 27 | pipeline.add(testsrc) 28 | 29 | mixer = Gst.ElementFactory.make("audiomixer") 30 | mixer.set_property("latency", (rtp_max_jitter_mx * 1_000_000)) # (5) 31 | pipeline.add(mixer) 32 | testsrc.link_filtered(mixer, caps_audio) 33 | 34 | sink = Gst.ElementFactory.make("autoaudiosink") 35 | pipeline.add(sink) 36 | mixer.link_filtered(sink, caps_audio) 37 | 38 | testsrc.get_static_pad("src").add_probe( 39 | Gst.PadProbeType.BUFFER, logging_pad_probe, "testsrc-output") 40 | 41 | mixer.get_static_pad("src").add_probe( 42 | Gst.PadProbeType.BUFFER, logging_pad_probe, "mixer-output") 43 | 44 | 45 | # udpsrc port=… ! {rtpcaps} ! rtpjitterbuffer latency=… ! rtpL16depay ! {rawcaps_be} ! audioconvert ! {rawcaps} ! … 46 | def create_bin(port): 47 | log.info("Creating RTP-Bin for Port %d" % port) 48 | rxbin = Gst.Bin.new("rx-bin-%d" % port) # (8) 49 | 50 | udpsrc = Gst.ElementFactory.make("udpsrc") # (3) 51 | udpsrc.set_property("port", port) 52 | rxbin.add(udpsrc) 53 | 54 | udpsrc.get_static_pad("src").add_probe( 55 | Gst.PadProbeType.BUFFER, logging_pad_probe, "udpsrc-%d-output" % port) 56 | 57 | jitterbuffer = Gst.ElementFactory.make("rtpjitterbuffer") # (4) 58 | jitterbuffer.set_property("latency", rtp_max_jitter_mx) 59 | jitterbuffer.set_property("drop-on-latency", True) 60 | rxbin.add(jitterbuffer) 61 | udpsrc.link_filtered(jitterbuffer, caps_rtp) 62 | 63 | depayload = Gst.ElementFactory.make("rtpL16depay") # (6) 64 | rxbin.add(depayload) 65 | jitterbuffer.link(depayload) 66 | 67 | depayload.get_static_pad("src").add_probe( 68 | Gst.PadProbeType.BUFFER, logging_pad_probe, "depayload-%d-output" % port) 69 | 70 | audioconvert = Gst.ElementFactory.make("audioconvert", "out-%d" % port) # (7) 71 | rxbin.add(audioconvert) 72 | depayload.link_filtered(audioconvert, caps_audio_be) 73 | 74 | return rxbin 75 | 76 | 77 | def add_bin(port): 78 | Gst.debug_bin_to_dot_file_with_ts(pipeline, Gst.DebugGraphDetails.ALL, "add_bin_%u_before" % port) 79 | bin = create_bin(port) 80 | 81 | log.info("Adding RTP-Bin for Port %d to the Pipeline" % port) 82 | pipeline.add(bin) 83 | output_element = pipeline.get_by_name("out-%d" % port) # (9) 84 | output_element.link_filtered(mixer, caps_audio) 85 | bin.sync_state_with_parent() # (10) 86 | log.info("Added RTP-Bin for Port %d to the Pipeline" % port) 87 | Gst.debug_bin_to_dot_file_with_ts(pipeline, Gst.DebugGraphDetails.ALL, "add_bin_%u_after" % port) 88 | 89 | 90 | stop_event = Event() 91 | 92 | 93 | def timed_sequence(): 94 | log.info("Starting Sequence") 95 | 96 | if stop_event.wait(2): return 97 | log.info("Scheduling adding a Bin for Port 10000") 98 | GLib.idle_add(add_bin, 10000) # (1) 99 | 100 | if stop_event.wait(2): return 101 | log.info("Scheduling adding a Bin for Port 10001") 102 | GLib.idle_add(add_bin, 10001) # (1) 103 | 104 | if stop_event.wait(2): return 105 | log.info("Scheduling adding a Bin for Port 10002") 106 | GLib.idle_add(add_bin, 10002) # (1) 107 | 108 | log.info("Sequence ended") 109 | 110 | 111 | t = Thread(target=timed_sequence, name="Sequence") 112 | t.start() 113 | 114 | runner = Runner(pipeline) 115 | runner.run_blocking() 116 | 117 | stop_event.set() 118 | t.join() 119 | -------------------------------------------------------------------------------- /03-add-and-remove-source.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | import logging 3 | from threading import Thread, Event 4 | 5 | from tools.application_init import application_init 6 | 7 | application_init() 8 | 9 | from gi.repository import Gst, GLib 10 | from tools.logging_pad_probe import logging_pad_probe 11 | from tools.runner import Runner 12 | 13 | log = logging.getLogger("main") 14 | 15 | log.info("building pipeline") 16 | pipeline = Gst.Pipeline.new() 17 | caps = Gst.Caps.from_string("audio/x-raw,format=S16LE,rate=48000,channels=2") 18 | 19 | testsrc1 = Gst.ElementFactory.make("audiotestsrc", "testsrc1") 20 | testsrc1.set_property("is-live", True) 21 | testsrc1.set_property("freq", 220) 22 | pipeline.add(testsrc1) 23 | 24 | mixer = Gst.ElementFactory.make("audiomixer") 25 | pipeline.add(mixer) 26 | testsrc1.link_filtered(mixer, caps) 27 | 28 | sink = Gst.ElementFactory.make("autoaudiosink") 29 | pipeline.add(sink) 30 | mixer.link_filtered(sink, caps) 31 | 32 | testsrc1.get_static_pad("src").add_probe( 33 | Gst.PadProbeType.BUFFER, logging_pad_probe, "testsrc1-output") 34 | 35 | mixer.get_static_pad("src").add_probe( 36 | Gst.PadProbeType.BUFFER, logging_pad_probe, "mixer-output") 37 | 38 | testsrc2 = None # (2) 39 | capsfilter2 = None 40 | mixerpad = None 41 | 42 | 43 | def add_new_src(): 44 | global testsrc2, capsfilter2, mixerpad 45 | Gst.debug_bin_to_dot_file_with_ts(pipeline, Gst.DebugGraphDetails.ALL, "adding-testsrc2-before") 46 | log.info("Adding testsrc2") 47 | 48 | log.info("Creating testsrc2") 49 | testsrc2 = Gst.ElementFactory.make("audiotestsrc", "testsrc2") 50 | testsrc2.set_property("freq", 440) 51 | testsrc2.set_property("is-live", True) 52 | 53 | testsrc2.get_static_pad("src").add_probe( 54 | Gst.PadProbeType.BUFFER, logging_pad_probe, "testsrc2-output") 55 | 56 | log.info("Adding testsrc2") 57 | log.debug(pipeline.add(testsrc2)) 58 | 59 | log.info("Creating capsfilter") 60 | capsfilter2 = Gst.ElementFactory.make("capsfilter", "capsfilter2") # (3) 61 | capsfilter2.set_property("caps", caps) 62 | 63 | log.info("Adding capsfilter") 64 | log.debug(pipeline.add(capsfilter2)) 65 | 66 | log.info("Linking testsrc2 to capsfilter2") 67 | log.debug(testsrc2.link(capsfilter2)) 68 | 69 | log.info("Requesting Pad from Mixer") 70 | mixerpad = mixer.get_request_pad("sink_%u") 71 | log.debug(mixerpad) 72 | 73 | log.info("Linking capsfilter2 to mixerpad") 74 | log.debug(capsfilter2.get_static_pad("src").link(mixerpad)) 75 | 76 | log.info("Syncing Element-States with Pipeline") 77 | log.debug(capsfilter2.sync_state_with_parent()) 78 | log.debug(testsrc2.sync_state_with_parent()) 79 | 80 | log.info("Adding testsrc2 done") 81 | Gst.debug_bin_to_dot_file_with_ts(pipeline, Gst.DebugGraphDetails.ALL, "adding-testsrc2-after") # (4) 82 | 83 | 84 | def remove_src(): 85 | Gst.debug_bin_to_dot_file_with_ts(pipeline, Gst.DebugGraphDetails.ALL, "removing-testsrc2-before") 86 | log.info("Removing testsrc2") 87 | 88 | log.info("Stopping testsrc2") 89 | log.debug(testsrc2.set_state(Gst.State.NULL)) # (5) 90 | 91 | log.info("Stopping capsfilter2") 92 | log.debug(capsfilter2.set_state(Gst.State.NULL)) 93 | 94 | log.info("Removing testsrc2") 95 | log.debug(pipeline.remove(testsrc2)) 96 | 97 | log.info("Removing capsfilter2") 98 | log.debug(pipeline.remove(capsfilter2)) 99 | 100 | log.info("Releasing mixerpad") 101 | log.debug(mixer.release_request_pad(mixerpad)) # (6) 102 | 103 | log.info("Removing testsrc2 done") 104 | Gst.debug_bin_to_dot_file_with_ts(pipeline, Gst.DebugGraphDetails.ALL, "removing-testsrc2-after") 105 | 106 | 107 | stop_event = Event() # (1) 108 | 109 | 110 | def timed_sequence(): 111 | log.info("Starting Sequence") 112 | while True: 113 | if stop_event.wait(2): return 114 | log.info("Schedule Add Source") 115 | GLib.idle_add(add_new_src) 116 | 117 | if stop_event.wait(2): return 118 | log.info("Schedule Remove Source") 119 | GLib.idle_add(remove_src) 120 | 121 | 122 | t = Thread(target=timed_sequence, name="Sequence") 123 | t.start() 124 | 125 | runner = Runner(pipeline) 126 | runner.run_blocking() 127 | 128 | stop_event.set() 129 | t.join() 130 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # GStreamer Dynamic Pipelines Cookbook 2 | 3 | Writing complex, static GStreamer Pipelines, that is Pipelines that are fully described before launching them, is easy. 4 | Ok, not trivially easy but comparably easy. One has to only work with a single State that either works or doesnt 5 | (because of various reasons that themselves are sometimes not so easy). 6 | 7 | Actually there are no static Pipelines - Depending on changing Input, Output or Properties Pipelines can re-negotiate 8 | Caps, Latency and other properties, but we can make our Pipelines quite static and well-known by placing CapsFilter 9 | between the Elements to enforce the Caps that we know beforehand and undermine any dynamic negotiation this way. 10 | Most of this is done for us and happens behind the scene. 11 | 12 | But things get quite a lot more complex when we want to change the Pipelines while they are running, without 13 | interrupting other pieces of the Pipeline. Now *we* need to take control of the State of both, our new and our existing 14 | Pipeline-Pieces and *we* need to mange their Startup-Behaviour, Latency and Running-Times. 15 | 16 | Similar when we want to remove a Piece of our Pipeline, we need to correctly shutting down the individual Pieces and 17 | manage their links. 18 | 19 | Dynamic Pipelines are a Thing I tried to ignore for as long as possible, but with this Blog-Post I want to give 20 | reproducible Examples for the most common Modifications: Adding and Removing Sinks and Sources and changing Links between 21 | existing Elements. 22 | 23 | ## Running the Experiments 24 | The Experiments require a little Test-Bed to run in, which includes a nice colorful console logger that helps a lot to 25 | find orientation in the rather long and repetitive log outputs. 26 | 27 | To run the Experiments create a virtualenv and install the dependencies in there: 28 | ``` 29 | virtualenv -ppython3 env 30 | source ./env/bin/activate 31 | ./env/bin/pip install -r requirements.txt 32 | 33 | ./01-add-source.py 34 | ``` 35 | 36 | ## Adding a Source 37 | This Example creates a Pipeline with an `audiotestsrc` and an `audiomixer`. After a while a second `audiotestsrc` is 38 | created, added to the Pipeline and linked to the `audiomixer`. 39 | 40 | - [Recipe](01-add-source.md) 41 | - [Sourcecode](01-add-source.py) 42 | 43 | ## Adding RTP-Sources 44 | As a more realistic example of adding Sources to a playing Pipeline, this Example creates a Pipeline with an 45 | `audiotestsrc` and an `audiomixer`. After a while, a Bin (a Cluster of Elements) which receives and decodes Audio coming 46 | from the Network via RTP is created, added to the Pipeline and linked to the `audiomixer`. After 4 and 6 seconds additional 47 | Bins of this kind are created and also linked. 48 | 49 | - [Recipe](02-add-network-source.md) 50 | - [Sourcecode](02-add-network-source.py) 51 | 52 | ## Adding and Removing Sources 53 | This example is based upon [Adding a Source](01-add-source.py). A Pipeline with a live `audiotestsrc`, an `audiomixer` 54 | and an `autoaudiosink` is created. After 2 seconds, a second `audiotestsrc` is created, added to the pipeline and linked 55 | to the Mixer. Another 2 seconds later it is stopped, unlinked and removed again. The process repeats as long as the 56 | experiment runs. 57 | 58 | You should read [Adding a Source](01-add-source.md) before this, because important Lines that have been explained there 59 | are not repeated here. 60 | 61 | - [Recipe](03-add-and-remove-source.md) 62 | - [Sourcecode](03-add-and-remove-source.py) 63 | 64 | ## Adding and Removing RTP-Sources 65 | As a more realistic example of adding and removing Sources to a playing Pipeline, this Example creates a Pipeline with an 66 | `audiotestsrc` and an `audiomixer`. After a while, a Bin (a Cluster of Elements) which receives and decodes Audio coming 67 | from the Network via RTP is created, added to the Pipeline and linked to the `audiomixer`. After 4 and 6 seconds additional 68 | Bins of this kind are created and also linked. Then, again with 2 Seconds in between, the Source-Bins are removes again, 69 | before thw whole process starts over again. 70 | 71 | You should read [Adding RTP-Sources](02-add-network-source.md) and [Adding and Removing Sources](03-add-and-remove-source.md) 72 | before this, because important Aspects that have been explained there are not repeated here. 73 | 74 | - [Recipe](04-add-and-remove-network-source.md) 75 | - [Sourcecode](04-add-and-remove-network-source.py) 76 | 77 | ## Help, my $Thing does not work 78 | Lalafoo Mailinglist 79 | -------------------------------------------------------------------------------- /05-add-and-remove-network-sink.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | import logging 3 | from threading import Thread, Event 4 | 5 | from tools.application_init import application_init 6 | 7 | application_init() 8 | 9 | from gi.repository import Gst, GLib 10 | from tools.runner import Runner 11 | 12 | log = logging.getLogger("main") 13 | 14 | log.info("building pipeline") 15 | pipeline = Gst.Pipeline.new() 16 | caps_audio = Gst.Caps.from_string("audio/x-raw,format=S16LE,rate=48000,channels=2") 17 | caps_audio_be = Gst.Caps.from_string("audio/x-raw,format=S16BE,rate=48000,channels=2") 18 | caps_rtp = Gst.Caps.from_string("application/x-rtp,clock-rate=48000,media=audio,encoding-name=L16,channels=2") 19 | 20 | testsrc = Gst.ElementFactory.make("audiotestsrc", "testsrc1") 21 | testsrc.set_property("is-live", True) 22 | testsrc.set_property("freq", 220) 23 | pipeline.add(testsrc) 24 | 25 | tee = Gst.ElementFactory.make("tee") # (1) 26 | tee.set_property("allow-not-linked", True) 27 | pipeline.add(tee) 28 | testsrc.link_filtered(tee, caps_audio) 29 | 30 | playback_internal = False # (2) 31 | if playback_internal: 32 | sink = Gst.ElementFactory.make("autoaudiosink") 33 | pipeline.add(sink) 34 | tee.link(sink) 35 | 36 | 37 | # audioconvert ! {rawcaps_be} ! rtpL16depay ! udpsink port=… 38 | def create_bin(port): 39 | log.info("Creating RTP-Bin for Port %d" % port) 40 | txbin = Gst.Bin.new("tx-bin-%d" % port) 41 | log.debug(txbin) 42 | 43 | log.info("Creating queue") 44 | queue = Gst.ElementFactory.make("queue") # (3) 45 | log.debug(queue) 46 | 47 | log.info("Adding queue to bin") 48 | log.debug(txbin.add(queue)) 49 | 50 | log.info("Creating audioconvert") 51 | audioconvert = Gst.ElementFactory.make("audioconvert") 52 | log.debug(audioconvert) 53 | 54 | log.info("Adding audioconvert to bin") 55 | log.debug(txbin.add(audioconvert)) 56 | 57 | log.info("Linking queue to audioconvert") 58 | log.debug(queue.link(audioconvert)) 59 | 60 | log.info("Creating payloader") 61 | payloader = Gst.ElementFactory.make("rtpL16pay") 62 | log.debug(payloader) 63 | 64 | log.info("Adding payloader to bin") 65 | log.debug(txbin.add(payloader)) 66 | 67 | log.info("Linking audioconvert to payloader") 68 | log.debug(audioconvert.link_filtered(payloader, caps_audio_be)) 69 | 70 | log.info("Creating udpsink") 71 | udpsink = Gst.ElementFactory.make("udpsink") 72 | log.debug(payloader) 73 | udpsink.set_property("host", "127.0.0.1") # (4) 74 | udpsink.set_property("port", port) 75 | 76 | log.info("Adding udpsink to bin") 77 | log.debug(txbin.add(udpsink)) 78 | 79 | log.info("Linking payloader to udpsink") 80 | log.debug(payloader.link(udpsink)) 81 | 82 | log.info("Selecting Input-Pad") 83 | sink_pad = queue.get_static_pad("sink") 84 | log.debug(sink_pad) 85 | 86 | log.info("Creating Ghost-Pad") 87 | ghost_pad = Gst.GhostPad.new("sink", sink_pad) 88 | log.debug(ghost_pad) 89 | 90 | log.info("Adding Ghost-Pad to Bin") 91 | log.debug(txbin.add_pad(ghost_pad)) 92 | 93 | return txbin 94 | 95 | 96 | def add_bin(port): 97 | log.info("Adding RTP-Bin for Port %d to the Pipeline" % port) 98 | Gst.debug_bin_to_dot_file_with_ts(pipeline, Gst.DebugGraphDetails.ALL, "add_bin_%u_before" % port) 99 | 100 | log.info("Creating Bin") 101 | txbin = create_bin(port) 102 | log.info("Created Bin") 103 | log.debug(txbin) 104 | 105 | log.info("Adding bin to pipeline") 106 | log.debug(pipeline.add(txbin)) 107 | 108 | log.info("Syncing Bin-State with Parent") 109 | log.debug(txbin.sync_state_with_parent()) 110 | 111 | log.info("Linking bin to mixer") 112 | tee.link(txbin) 113 | 114 | Gst.debug_bin_to_dot_file_with_ts(pipeline, Gst.DebugGraphDetails.ALL, "add_bin_%u_after" % port) 115 | log.info("Added RTP-Bin for Port %d to the Pipeline" % port) 116 | 117 | 118 | def remove_bin(port): 119 | log.info("Removing RTP-Bin for Port %d to the Pipeline" % port) 120 | Gst.debug_bin_to_dot_file_with_ts(pipeline, Gst.DebugGraphDetails.ALL, "remove_bin_%u_before" % port) 121 | 122 | log.info("Selecting Bin") 123 | txbin = pipeline.get_by_name("tx-bin-%d" % port) 124 | log.debug(txbin) 125 | 126 | log.info("Selecting Ghost-Pad") 127 | ghostpad = txbin.get_static_pad("sink") 128 | log.debug(ghostpad) 129 | 130 | log.info("Selecting Tee-Pad (Peer of Ghost-Pad)") 131 | teepad = ghostpad.get_peer() 132 | log.debug(teepad) 133 | 134 | def blocking_pad_probe(pad, info): 135 | log.info("Stopping Bin") 136 | log.debug(txbin.set_state(Gst.State.NULL)) 137 | 138 | log.info("Removing Bin from Pipeline") 139 | log.debug(pipeline.remove(txbin)) 140 | 141 | log.info("Releasing Tee-Pad") 142 | log.debug(tee.release_request_pad(teepad)) 143 | 144 | Gst.debug_bin_to_dot_file_with_ts(pipeline, Gst.DebugGraphDetails.ALL, "remove_bin_%u_after" % port) 145 | log.info("Removed RTP-Bin for Port %d to the Pipeline" % port) 146 | 147 | return Gst.PadProbeReturn.REMOVE 148 | 149 | log.info("Configuring Blocking Probe on teepad") 150 | teepad.add_probe(Gst.PadProbeType.BLOCK, blocking_pad_probe) # (5) 151 | 152 | 153 | stop_event = Event() 154 | 155 | 156 | def timed_sequence(): 157 | log.info("Starting Sequence") 158 | 159 | num_ports = 3 160 | timeout = 0.2 161 | while True: 162 | for i in range(0, num_ports): 163 | if stop_event.wait(timeout): return 164 | log.info("Scheduling add_bin for Port %d", 15000 + i) 165 | GLib.idle_add(add_bin, 15000 + i) 166 | 167 | for i in range(0, num_ports): 168 | if stop_event.wait(timeout): return 169 | log.info("Scheduling remove_bin for Port %d", 15000 + i) 170 | GLib.idle_add(remove_bin, 15000 + i) 171 | 172 | 173 | t = Thread(target=timed_sequence, name="Sequence") 174 | t.start() 175 | 176 | runner = Runner(pipeline) 177 | runner.run_blocking() 178 | 179 | stop_event.set() 180 | t.join() 181 | -------------------------------------------------------------------------------- /04-add-and-remove-network-source.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | import logging 3 | from threading import Thread, Event 4 | 5 | rtp_max_jitter_mx = 30 6 | 7 | from tools.application_init import application_init 8 | 9 | application_init() 10 | 11 | from gi.repository import Gst, GLib 12 | from tools.logging_pad_probe import logging_pad_probe 13 | from tools.runner import Runner 14 | 15 | log = logging.getLogger("main") 16 | 17 | log.info("building pipeline") 18 | pipeline = Gst.Pipeline.new() 19 | caps_audio = Gst.Caps.from_string("audio/x-raw,format=S16LE,rate=48000,channels=2") 20 | caps_audio_be = Gst.Caps.from_string("audio/x-raw,format=S16BE,rate=48000,channels=2") 21 | caps_rtp = Gst.Caps.from_string("application/x-rtp,clock-rate=48000,media=audio,encoding-name=L16,channels=2") 22 | 23 | testsrc = Gst.ElementFactory.make("audiotestsrc", "testsrc") 24 | testsrc.set_property("is-live", True) 25 | testsrc.set_property("freq", 220) 26 | testsrc.set_property("volume", 0.5) 27 | pipeline.add(testsrc) 28 | 29 | mixer = Gst.ElementFactory.make("audiomixer") 30 | mixer.set_property("latency", (rtp_max_jitter_mx * 1_000_000)) 31 | pipeline.add(mixer) 32 | testsrc.link_filtered(mixer, caps_audio) 33 | 34 | sink = Gst.ElementFactory.make("autoaudiosink") 35 | pipeline.add(sink) 36 | mixer.link_filtered(sink, caps_audio) 37 | 38 | testsrc.get_static_pad("src").add_probe( 39 | Gst.PadProbeType.BUFFER, logging_pad_probe, "testsrc-output") 40 | 41 | mixer.get_static_pad("src").add_probe( 42 | Gst.PadProbeType.BUFFER, logging_pad_probe, "mixer-output") 43 | 44 | 45 | # udpsrc port=… ! {rtpcaps} ! rtpjitterbuffer latency=… ! rtpL16depay ! {rawcaps_be} ! audioconvert ! {rawcaps} ! … 46 | def create_bin(port): 47 | log.info("Creating RTP-Bin for Port %d" % port) 48 | rxbin = Gst.Bin.new("rx-bin-%d" % port) # (1) 49 | 50 | log.info("Creating udpsrc") 51 | udpsrc = Gst.ElementFactory.make("udpsrc") 52 | log.debug(udpsrc) 53 | udpsrc.set_property("port", port) 54 | udpsrc.set_property("caps", caps_rtp) 55 | 56 | log.info("Adding udpsrc to bin") 57 | log.debug(rxbin.add(udpsrc)) 58 | 59 | log.info("Registering Pad-Probe after udpsrc") 60 | log.debug(udpsrc.get_static_pad("src").add_probe( 61 | Gst.PadProbeType.BUFFER, logging_pad_probe, "udpsrc-%d-output" % port)) 62 | 63 | log.info("Creating jitterbuffer") 64 | jitterbuffer = Gst.ElementFactory.make("rtpjitterbuffer") 65 | log.debug(jitterbuffer) 66 | jitterbuffer.set_property("latency", rtp_max_jitter_mx) 67 | jitterbuffer.set_property("drop-on-latency", True) 68 | 69 | log.info("Adding jitterbuffer to bin") 70 | log.debug(rxbin.add(jitterbuffer)) 71 | 72 | log.info("Linking udpsrc to jitterbuffer") 73 | log.debug(udpsrc.link(jitterbuffer)) 74 | 75 | log.info("Creating depayloader") 76 | depayload = Gst.ElementFactory.make("rtpL16depay") 77 | log.debug(depayload) 78 | 79 | log.info("Adding depayloader to bin") 80 | log.debug(rxbin.add(depayload)) 81 | 82 | log.info("Linking jitterbuffer to depayloader") 83 | log.debug(jitterbuffer.link(depayload)) 84 | 85 | log.info("Registering Pad-Probe after depayload") 86 | log.debug(depayload.get_static_pad("src").add_probe( 87 | Gst.PadProbeType.BUFFER, logging_pad_probe, "depayload-%d-output" % port)) 88 | 89 | log.info("Creating audioconvert") 90 | audioconvert = Gst.ElementFactory.make("audioconvert") 91 | log.debug(audioconvert) 92 | 93 | log.info("Adding audioconvert to bin") 94 | log.debug(rxbin.add(audioconvert)) 95 | 96 | log.info("Linking depayload to audioconvert") 97 | log.debug(depayload.link_filtered(audioconvert, caps_audio_be)) 98 | 99 | log.info("Selecting Output-Pad") 100 | src_pad = audioconvert.get_static_pad("src") # (2) ff. 101 | log.debug(src_pad) 102 | 103 | log.info("Creating Ghost-Pad") 104 | ghost_pad = Gst.GhostPad.new("src", src_pad) 105 | log.debug(ghost_pad) 106 | 107 | log.info("Adding Ghost-Pad to Bin") 108 | log.debug(rxbin.add_pad(ghost_pad)) 109 | 110 | return rxbin 111 | 112 | 113 | def add_bin(port): 114 | log.info("Adding RTP-Bin for Port %d to the Pipeline" % port) 115 | Gst.debug_bin_to_dot_file_with_ts(pipeline, Gst.DebugGraphDetails.ALL, "add_bin_%u_before" % port) 116 | 117 | log.info("Creating Bin") 118 | rxbin = create_bin(port) 119 | log.info("Created Bin") 120 | log.debug(rxbin) 121 | 122 | log.info("Adding bin to pipeline") 123 | log.debug(pipeline.add(rxbin)) 124 | 125 | log.info("Linking bin to mixer") 126 | rxbin.link(mixer) # (3) 127 | 128 | log.info("Syncing Bin-State with Parent") 129 | log.debug(rxbin.sync_state_with_parent()) 130 | 131 | Gst.debug_bin_to_dot_file_with_ts(pipeline, Gst.DebugGraphDetails.ALL, "add_bin_%u_after" % port) 132 | log.info("Added RTP-Bin for Port %d to the Pipeline" % port) 133 | 134 | 135 | def remove_bin(port): 136 | log.info("Removing RTP-Bin for Port %d to the Pipeline" % port) 137 | Gst.debug_bin_to_dot_file_with_ts(pipeline, Gst.DebugGraphDetails.ALL, "remove_bin_%u_before" % port) 138 | 139 | log.info("Selecting Bin") 140 | rxbin = pipeline.get_by_name("rx-bin-%d" % port) # (1) 141 | log.debug(rxbin) 142 | 143 | log.info("Selecting Ghost-Pad") 144 | ghostpad = rxbin.get_static_pad("src") 145 | log.debug(ghostpad) 146 | 147 | log.info("Selecting Mixerpad (Peer of Ghost-Pad)") 148 | mixerpad = ghostpad.get_peer() # (4) 149 | log.debug(mixerpad) 150 | 151 | log.info("Stopping Bin") 152 | log.debug(rxbin.set_state(Gst.State.NULL)) # (5) ff. 153 | 154 | log.info("Removing Bin from Pipeline") 155 | log.debug(pipeline.remove(rxbin)) 156 | 157 | log.info("Releasing mixerpad") 158 | log.debug(mixer.release_request_pad(mixerpad)) 159 | 160 | Gst.debug_bin_to_dot_file_with_ts(pipeline, Gst.DebugGraphDetails.ALL, "remove_bin_%u_after" % port) 161 | log.info("Removed RTP-Bin for Port %d to the Pipeline" % port) 162 | 163 | 164 | stop_event = Event() 165 | 166 | 167 | def timed_sequence(): 168 | log.info("Starting Sequence") 169 | 170 | num_ports = 3 171 | while True: 172 | for i in range(0, num_ports): 173 | if stop_event.wait(2): return 174 | log.info("Scheduling add_bin for Port %d", 10000 + i) 175 | GLib.idle_add(add_bin, 10000 + i) 176 | 177 | for i in range(0, num_ports): 178 | if stop_event.wait(2): return 179 | log.info("Scheduling remove_bin for Port %d", 10000 + i) 180 | GLib.idle_add(remove_bin, 10000 + i) 181 | 182 | 183 | t = Thread(target=timed_sequence, name="Sequence") 184 | t.start() 185 | 186 | runner = Runner(pipeline) 187 | runner.run_blocking() 188 | 189 | stop_event.set() 190 | t.join() 191 | -------------------------------------------------------------------------------- /02-add-network-source.md: -------------------------------------------------------------------------------- 1 | # Adding RTP Network-Sources 2 | → [Sourcecode](02-add-network-source.py) 3 | 4 | This Example creates a Pipeline like this: 5 | 6 | ``` 7 | audiotestsrc is-live=true ! audiomixer ! autoaudiosink 8 | ``` 9 | 10 | After 2 Seconds it adds a RTP-Receiving-Bin like this to the Pipeline: 11 | ``` 12 | udpsrc port=10000 ! 13 | application/x-rtp,clock-rate=48000,media=audio,encoding-name=L16,channels=2 ! 14 | rtpjitterbuffer latency=… ! 15 | rtpL16depay ! 16 | audio/x-raw,format=S16BE,rate=48000,channels=2 ! 17 | audioconvert ! 18 | audio/x-raw,format=S16LE,rate=48000,channels=2 ! 19 | autoaudiosink 20 | ``` 21 | 22 | After 4 and 6 seconds (from the start) another Receiving-Sink on Port `10001` and `10002` is added. 23 | 24 | The Example installs Pad-Probes in after interesting stages, which log the PTS-Timestamps 25 | of the Buffers flowing through the Elements' Pads. 26 | 27 | After the start you can hear a a low volume 110 Hz Hum on the Speaker. On the Console you can see the Buffers being 28 | generated in real time, because the `audiotestsrc` is forced to be a live source with the `is-live` property. 29 | Without that, it would run as fast as the sink allows it to. 30 | 31 | After 2 Seconds the RTP Receiving bin is added, but because there is no RTP Source yet, nothing changes. 32 | You can hear the 110 Hz Background Hum continuing without interruption. On the Console you can see that the `udpsrc` 33 | has been added and the new elements switched through all States and are now `RUNNING` , but it does not generate 34 | buffers yet because nothing is received on the UDP Port . 35 | 36 | You can now start a RDP-Source on the Same or a different Computer on the Network: 37 | ``` 38 | gst-launch-1.0 audiotestsrc freq=440 volume=0.5 is-live=true ! \ 39 | audio/x-raw,format=S16BE,rate=48000,channels=2 ! \ 40 | rtpL16pay ! \ 41 | application/x-rtp,clock-rate=48000,media=audio,encoding-name=L16,channels=2 ! \ 42 | udpsink host=… port=10000 43 | ``` 44 | 45 | On the speaker you can now hear both tones being mixed together and played back. 46 | On the Console you can see the Timestamps of the `udpsrc` starting close the the timestamps currently mixed by 47 | the mixer, because Elements of type `udpsrc` are always live. 48 | 49 | You now start an stop the RDP Source. The Background-Hum will continue and your added RDP Source will immediately start 50 | to mix with it again, as soon as you start it. The `rtpjitterbuffer`-Element can detect the loss of the source signal 51 | and re-start the timing correction when the signals starts again. 52 | 53 | The most important Lines have been marked as such: 54 | 55 | 1. We schedule the execution on the GLib Event-Loop by using `Glib.idle_add`. On the Console you can see, that the 56 | Sequence runs in its own `Sequence`-Thread but the second testsrc is actually added from the `MainThread`. 57 | 58 | It seems that this exact example also works without this, because some of the Methods are Thread-Safe by 59 | Documentation (ie. [Bin.add](https://lazka.github.io/pgi-docs/#Gst-1.0/classes/Bin.html#Gst.Bin.add)) and some are 60 | by luck (ie [Gobject.Object.set_property](https://lazka.github.io/pgi-docs/#GObject-2.0/classes/Object.html#GObject.Object.set_property)), 61 | but not following this pattern can lead to unexpected and hard to find issues. 62 | 63 | From the console you can also see, that the Pad-Probes are called synchronously from their respective Elements' 64 | Streaming-Threads (Shown as `Dummy-[n]`). 65 | 66 | 2. Forcing the `audiotestsrc` to be a Live-Source makes it produce Buffers with timestamps starting at the current 67 | Running-Time of the Pipeline. A Non-Live-Source would start ad 0:00:00 and run faster then the other sources, 68 | until it would have caught up to the current Running-Time. 69 | 70 | 3. The `udpsrc` added is always live (see [Documentation](https://gstreamer.freedesktop.org/data/doc/gstreamer/head/gst-plugins-good/html/gst-plugins-good-plugins-udpsrc.html#gst-plugins-good-plugins-udpsrc.description])) 71 | 72 | 4. Our RTP-Packets on the Network are not obviously not delivered instantly. The Source on the Source-Pipeline takes the 73 | Running-Time of the Source-Pipeline and stamps the generated Buffers into it. This timestamp is encoded into the 74 | RTP-Packets and given to the Network-Stack. 75 | 76 | The Network-Stack on both sided, Switches, Routers, WLan APs and other Network Equipment all have Buffers which delay 77 | our RTP Packets. Furthermore the exact amount of Delay is different for each and every RTP Packet. 78 | 79 | To circumvent this Jitter, an `rtpjitterbuffer` is added to the Pipeline and configured with the maximal allowed Jitter. 80 | It is also instructed to drop all late Packets. This will make a too low configured Latency obvious. 81 | 82 | The Jitter-Buffer will artificially delay each incoming Buffer, so that it plays back at its Timestamp plus the 83 | configured Latency. For example given a configured Latency of **20ms**, a Buffer with a Timestamp of **t=1000ms** 84 | will be played back at **t=1020ms**, no matter if it has been received at **t=1001ms**, **t=1005ms**, **t=1010ms** 85 | or **t=10019ms**. 86 | 87 | So with a configured Latency of **20ms** Your Network (including the Network-Stacks on both ends) is allowed to 88 | Jitter up to that amount back and forth, without the Receiving-Side starting to drop Packets. 89 | 90 | In my tests I found **30ms** to be a good start for Devices talking across multiple Switches and a Wifi-Access-Point 91 | under Linux. When all Devices are on a non-crowded GBit Ethernet Link **10ms** should be fine too. 92 | Across the Internet a Value of **200ms** or upwards might be required. 93 | 94 | Under MacOS I needed to go to **100ms** on Wifi. Your mileage may vary. 95 | 96 | 5. The Audiomixer has, apart from its obvious job of combining Audio-Samples, the task of syncing up incoming Streams 97 | and only mixing samples that are meant to be played back at the same time. When a least one of the sources linked to 98 | an `audiomixer` Element is a live-source, the Audiomixer itself is live and generates buffers stamped with the current 99 | running time of the Pipeline. 100 | 101 | Obviously in our Situation it can't mix the Samples stamped for **t=1000ms* at **t=1000ms** because they will still 102 | be waiting in the Jitter-Buffer. When going to `PLAYING` the Audiomixer queries all its sources for their latency and 103 | configured its own latency accordingly. In a dynamic Pipeline like this, the Source of Latency is not yet present when 104 | the Pipeline initially goes to `PLAYING`, so we need to configure the Audio-Mixers latency beforehand to the same or 105 | a higher value, then we configured the `rtpjitterbuffer` to buffer our RTP-Packets. 106 | 107 | 6. GStreamer comes with a metric ton of [RTP de/payloader elements](https://gstreamer.freedesktop.org/data/doc/gstreamer/head/gst-plugins-good-plugins/html/gst-plugins-good-plugins-plugin-rtp.html). 108 | This example chose L16 (specified in [RFC 3551](https://tools.ietf.org/html/rfc3551)) which encodes Audio as Uncompressed 109 | 16bit PCM in Network-Byte-Order (Big Endian). 110 | 111 | To enable Compressed Transfer a decoder-Element (ie `opusdec` can be added together with the matching depayloader, 112 | ie. `rtpopuspay`). 113 | 114 | 7. While not all Audio-Handling Elements are capable of working on Big Endian PCM (especially `autoaudiosink`, `level`, 115 | `volume` and `wavenc` aren't), all can handle Little Endian PCM. Furthermore some elements like the Audiomixer can 116 | will silently convert the Big Endian Samples to Little Endian, but every `audiomixer`-Element will do this for itself. 117 | 118 | To avoid these problem the complete main pipeline runs in Little Endian mode and the Buffers received from the Network 119 | are converted from Big to Little Endian before they are passed onto the audiomixer. 120 | 121 | 8. To Simplify State-Handling, all Elements are placed into a Bin (a Cluster of Elements) which maintains state for all 122 | its Child-Elements. 123 | 124 | 9. When Elements from Inside a Bin are linked with Elements outside the Bin or in another Bin, so called Ghost-Pads are 125 | created at the Boundary of the Bin. These Ghost-Pads can be actively managed by the Bin (for Details see the 126 | [Documentation on Ghost Pads](https://gstreamer.freedesktop.org/documentation/application-development/basics/pads.html?gi-language=c#ghost-pads)) 127 | but Ghost Pads are also automatically created, when a Cross-Bin-Link is performed, like in this example. 128 | 129 | 10. The Bin does not automatically take over its parent state. Also, not all Elements in a Pipeline have to have the 130 | same state. In this case the new Bin and all its Elements starts in `NULL` state and is added as such to the Pipeline. 131 | Once it is added, the Bins state is synced to the pipeline which in turn propagate this State-Change to all its Child- 132 | Elements. 133 | 134 | 11. At some points in the Pipeline multiple Media-Types can be processed by both Elements participating being linked together 135 | and especially in a dynamic pipeline not all requirements to these media-types are known at pipeline construction. 136 | For example the `audiotestsrc` and the `audiomixer` both share a wide range of Audio-Formats, Sample-Rates and 137 | Bit-Depths that both can support. Linking them might produce a result, that works initially, but fails when the 138 | RTP-Receiving-Bin is added later. 139 | 140 | To circumvent this, Caps are should specified at for links that can be made in multiple ways. This ensures reproducible 141 | results and avoids unexpected dynamic re-negotiation. 142 | --------------------------------------------------------------------------------