├── .gitignore ├── LICENSE.md ├── README.md ├── main.cpp ├── mainwindow.cpp ├── mainwindow.h └── noweffects.pro /.gitignore: -------------------------------------------------------------------------------- 1 | *.o 2 | *.user 3 | *.autosave 4 | *~ 5 | moc_* 6 | *.stash 7 | ui_* 8 | noweffects 9 | Makefile 10 | -------------------------------------------------------------------------------- /LICENSE.md: -------------------------------------------------------------------------------- 1 | Copyright © 2018 Michael Karl Franzl 2 | 3 | Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: 4 | 5 | The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. 6 | 7 | THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 8 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # noweffects 2 | 3 | Proof-of-concept of HTML5 + JavaScript + CSS3 RGBA video overlays on top of live GStreamer video pipelines. 4 | 5 | See related [Blog post](https://blog.michael.franzl.name/2018/04/14/html5-javascript-css3-alpha-aware-video-overlays-on-top-of-live-gstreamer-video-pipelines) and [demonstration](https://www.youtube.com/watch?v=jAtEJdnY0Mc). 6 | 7 | GStreamer comes with a number of plugins that allow rendering of text and/or graphics overlays on top of video: rsvgoverlay, subtitleoverlay, textoverlay, cairooverlay, gdkpixbufoverlay, opencvtextoverlay, etc. However, some of these plugins often allow only static graphics and text, and often do not approach the flexibility and power of dedicated video post-processing software products. 8 | 9 | "noweffects" (a play on the name of a popular video post-processing software) is a proof-of-concept of leveraging the power of a modern HTML5 + JavaScript + CSS3 web browser engine to render high-quality, programmable, alpha-aware, animated, vector- and bitmap based content, which is then rendered into an RGBA raw video stream, which can then be transferred via some kind of IPC method to separate GStreamer processeses, where it can be composited with other content via GStreamers regular `compositor` or `videomixer` plugins. 10 | 11 | Qt was chosen for its ease of integration of modern WebKit (QtWebKit) and GStreamer (qt-gstreamer), and its ability to render widgets to RGBA images. The QMainWindow widget is rendered in regular intervals to QImages in RGBA format, then inserted into a GStreamer pipeline via `appsrc` plugin. This pipeline simply uses `udpsink` to multicast the raw video RTP packets on localhost to allow for multiple 'subscribers'. A second GStreamer pipleline can then use `udpsrc` and apply the overlay. 12 | 13 | ## Usage 14 | 15 | Open `noweffects.pro` in QtCreator, or build and run from a terminal: 16 | 17 | qmake 18 | make 19 | ./noweffects 20 | 21 | The video size is `#define`d in `mainwindow.cpp` as `WIDTH` and `HEIGHT` to 1280 x 720. 22 | 23 | To view the generated video stream, a `gst-launch-1.0 ...` pipeline is printed to STDOUT when running the application. 24 | 25 | The generated video requires 1280 x 720 x 4colors x 30fps = 110 megabytes/second of UDP bandwidth on localhost. You will likely need to increase the Linux kernel's UDP buffer size, otherwise you may experience UDP packet loss (visible as flickering): 26 | 27 | sysctl -w net.core.wmem_max=2000000000 28 | sysctl -w net.core.rmem_max=2000000000 29 | 30 | Alternatively, reduce `WIDTH` and `HEIGHT` in `mainwindow.cpp`. 31 | 32 | 33 | ## Dependencies 34 | 35 | * Qt 5.11 commit 3ba94092 36 | * [QtWebkit](https://github.com/qt/qtwebkit) commit 72cfbd7 (20. Jan. 2018) 37 | * [qt-gstreamer](https://cgit.freedesktop.org/gstreamer/qt-gstreamer/) commit 0cf247bf 38 | 39 | -------------------------------------------------------------------------------- /main.cpp: -------------------------------------------------------------------------------- 1 | /* 2 | * noweffects - Proof-of-concept of HTML5 + JavaScript + CSS3 3 | * RGBA video overlays on top of live GStreamer video pipelines. 4 | * 5 | * Copyright © 2018 Michael Karl Franzl 6 | * 7 | * Permission is hereby granted, free of charge, to any person obtaining 8 | * a copy of this software and associated documentation files (the “Software”), 9 | * to deal in the Software without restriction, including without 10 | * limitation the rights to use, copy, modify, merge, publish, distribute, 11 | * sublicense, and/or sell copies of the Software, and to permit persons 12 | * to whom the Software is furnished to do so, subject to the following 13 | * conditions: 14 | * 15 | * The above copyright notice and this permission notice shall be included 16 | * in all copies or substantial portions of the Software. 17 | * 18 | * THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS 19 | * OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 20 | * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL 21 | * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR 22 | * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, 23 | * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR 24 | * OTHER DEALINGS IN THE SOFTWARE. 25 | */ 26 | 27 | #include "mainwindow.h" 28 | #include 29 | 30 | #include 31 | 32 | int main(int argc, char *argv[]) 33 | { 34 | QGst::init(&argc, &argv); 35 | 36 | QApplication a(argc, argv); 37 | MainWindow w; 38 | w.show(); 39 | 40 | return a.exec(); 41 | } 42 | -------------------------------------------------------------------------------- /mainwindow.cpp: -------------------------------------------------------------------------------- 1 | /* 2 | * noweffects - Proof-of-concept of HTML5 + JavaScript + CSS3 3 | * RGBA video overlays on top of live GStreamer video pipelines. 4 | * 5 | * Copyright © 2018 Michael Karl Franzl 6 | * 7 | * Permission is hereby granted, free of charge, to any person obtaining 8 | * a copy of this software and associated documentation files (the “Software”), 9 | * to deal in the Software without restriction, including without 10 | * limitation the rights to use, copy, modify, merge, publish, distribute, 11 | * sublicense, and/or sell copies of the Software, and to permit persons 12 | * to whom the Software is furnished to do so, subject to the following 13 | * conditions: 14 | * 15 | * The above copyright notice and this permission notice shall be included 16 | * in all copies or substantial portions of the Software. 17 | * 18 | * THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS 19 | * OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 20 | * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL 21 | * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR 22 | * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, 23 | * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR 24 | * OTHER DEALINGS IN THE SOFTWARE. 25 | */ 26 | 27 | #include 28 | 29 | #include "mainwindow.h" 30 | 31 | #include 32 | 33 | #include 34 | #include 35 | #include 36 | #include 37 | #include 38 | #include 39 | 40 | #define WIDTH 1280 41 | #define HEIGHT 720 42 | 43 | 44 | MainWindow::MainWindow(QWidget *parent) : 45 | QMainWindow(parent) 46 | { 47 | QDebug debug = qDebug(); 48 | debug.noquote(); 49 | 50 | m_webview = new QWebView(this); 51 | m_webview->setAttribute(Qt::WA_TranslucentBackground); 52 | m_webview->setStyleSheet("background:transparent"); 53 | 54 | this->setGeometry(0, 0, WIDTH, HEIGHT); 55 | m_webview->setGeometry(0, 0, WIDTH, HEIGHT); 56 | 57 | QWebPage *page = m_webview->page(); 58 | page->settings()->setAttribute(QWebSettings::DeveloperExtrasEnabled, true); // enable web inspector 59 | 60 | m_webview->setHtml("" 61 | "" 62 | " " 63 | " " 71 | " " 72 | "

Test

" 73 | "" 74 | "", QUrl("http://localhost")); 75 | 76 | m_frameshot = new QImage(this->size(), QImage::Format_RGBA8888); 77 | m_painter = new QPainter(m_frameshot); 78 | 79 | QString pipdescr = QString("appsrc name=mysrc is-live=true format=time do-timestamp=true caps=\"video/x-raw,format=RGBA,width=%1,height=%2\" ! rtpvrawpay mtu=65000 ! udpsink host=225.0.0.37 auto-multicast=true multicast-iface=lo ttl-mc=0 port=50000 bind-address=127.0.0.1 sync=false async=false buffer-size=200000000").arg(this->width()).arg(this->height()); 80 | QString viewer_cmd = QString("VIEWER COMMAND: gst-launch-1.0 compositor name=comp ! videoconvert ! autovideosink udpsrc address=225.0.0.37 auto-multicast=true multicast-iface=lo buffer-size=200000000 port=50000 caps=\"application/x-rtp,clock-rate=90000,encoding-name=RAW,sampling=(string)RGBA,width=(string)%1,height=(string)%2\" ! rtpvrawdepay ! comp.\n").arg(this->width()).arg(this->height()); 81 | debug << "\n"; 82 | debug << viewer_cmd; 83 | debug << "\n"; 84 | 85 | m_pip = QGst::Parse::launch(pipdescr).dynamicCast(); 86 | 87 | QGlib::connect(m_pip->bus(), "message", this, &MainWindow::onBusMessage); 88 | m_pip->bus()->addSignalWatch(); 89 | m_pip->setState(QGst::StatePlaying); 90 | 91 | this->m_src.setElement(m_pip->getElementByName("mysrc")); 92 | 93 | QTimer *timer = new QTimer(this); 94 | connect(timer, &QTimer::timeout, this, &MainWindow::screenshot); 95 | timer->start(30); // appox. 33 fps 96 | } 97 | 98 | void MainWindow::screenshot() 99 | { 100 | m_frameshot->fill(QColor(0,0,0,0)); 101 | this->render(m_painter, QPoint(), QRegion(), QWidget::DrawChildren); 102 | 103 | QGst::BufferPtr buf; 104 | buf = QGst::Buffer::create(WIDTH * HEIGHT * 4); 105 | 106 | QGst::MapInfo info; 107 | buf->map(info, QGst::MapFlag::MapWrite); 108 | quint8 *dataptr = info.data(); 109 | memcpy(dataptr, m_frameshot->bits(), WIDTH * HEIGHT * 4); 110 | buf->unmap(info); 111 | 112 | QGst::FlowReturn ret = this->m_src.pushBuffer(buf); 113 | // TODO: properly shut down if ret is not success 114 | } 115 | 116 | void MainWindow::onBusMessage(const QGst::MessagePtr &msg) 117 | { 118 | switch (msg->type()) { 119 | case QGst::MessageEos: 120 | qDebug() << "Received EOS"; 121 | // TODO: properly shut down 122 | break; 123 | case QGst::MessageError: 124 | qCritical() << msg.staticCast()->error(); 125 | break; 126 | case QGst::MessageWarning: 127 | qDebug() << "Warning" << msg.staticCast()->error(); 128 | break; 129 | case QGst::MessageStateChanged: 130 | //qDebug() << "state changed" << msg.staticCast()->newState(); 131 | break; 132 | default: 133 | qDebug() << "Received message type" << msg.staticCast()->typeName(); 134 | break; 135 | } 136 | } 137 | 138 | MainWindow::~MainWindow() 139 | { 140 | m_pip->setState(QGst::StateNull); 141 | } 142 | -------------------------------------------------------------------------------- /mainwindow.h: -------------------------------------------------------------------------------- 1 | /* 2 | * noweffects - Proof-of-concept of HTML5 + JavaScript + CSS3 3 | * RGBA video overlays on top of live GStreamer video pipelines. 4 | * 5 | * Copyright © 2018 Michael Karl Franzl 6 | * 7 | * Permission is hereby granted, free of charge, to any person obtaining 8 | * a copy of this software and associated documentation files (the “Software”), 9 | * to deal in the Software without restriction, including without 10 | * limitation the rights to use, copy, modify, merge, publish, distribute, 11 | * sublicense, and/or sell copies of the Software, and to permit persons 12 | * to whom the Software is furnished to do so, subject to the following 13 | * conditions: 14 | * 15 | * The above copyright notice and this permission notice shall be included 16 | * in all copies or substantial portions of the Software. 17 | * 18 | * THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS 19 | * OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 20 | * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL 21 | * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR 22 | * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, 23 | * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR 24 | * OTHER DEALINGS IN THE SOFTWARE. 25 | */ 26 | 27 | #ifndef MAINWINDOW_H 28 | #define MAINWINDOW_H 29 | 30 | #include 31 | #include 32 | #include 33 | #include 34 | 35 | #include 36 | #include 37 | #include 38 | 39 | namespace Ui { 40 | class MainWindow; 41 | } 42 | 43 | class MainWindow : public QMainWindow 44 | { 45 | Q_OBJECT 46 | 47 | public: 48 | explicit MainWindow(QWidget *parent = 0); 49 | ~MainWindow(); 50 | 51 | QGst::Utils::ApplicationSource m_src; 52 | QImage *m_frameshot; 53 | QPainter *m_painter; 54 | 55 | private: 56 | Ui::MainWindow *ui; 57 | QWebView *m_webview; 58 | QGst::PipelinePtr m_pip; 59 | 60 | 61 | private slots: 62 | void screenshot(); 63 | void onBusMessage(const QGst::MessagePtr &msg); 64 | }; 65 | 66 | #endif // MAINWINDOW_H 67 | -------------------------------------------------------------------------------- /noweffects.pro: -------------------------------------------------------------------------------- 1 | QT += core gui widgets webkit webkitwidgets 2 | 3 | PKGCONFIG += Qt5GStreamer-1.0 Qt5GStreamerUtils-1.0 4 | CONFIG += link_pkgconfig 5 | 6 | TARGET = noweffects 7 | TEMPLATE = app 8 | 9 | DEFINES += QT_DEPRECATED_WARNINGS 10 | 11 | SOURCES += main.cpp\ 12 | mainwindow.cpp 13 | 14 | HEADERS += mainwindow.h 15 | --------------------------------------------------------------------------------