├── .gitignore ├── LICENSE ├── README.md ├── ReferenceImages ├── FromUnreal.png ├── FromVue.png ├── SelectChair.png ├── SelectFloor.png ├── SelectStatue.png ├── SelectTable.png ├── UnrealFullBP.png ├── UnrealRecieveMessageBP.png └── UnrealSendMessageBP.png └── unreal-pixel-streaming-app ├── .gitignore ├── README.md ├── babel.config.js ├── package-lock.json ├── package.json ├── public ├── favicon.ico └── index.html └── src ├── App.vue ├── assets └── logo.png ├── main.js ├── router └── index.js ├── store └── index.js └── views └── Home.vue /.gitignore: -------------------------------------------------------------------------------- 1 | # Visual Studio 2015 user specific files 2 | .vs/ 3 | 4 | # Compiled Object files 5 | *.slo 6 | *.lo 7 | *.o 8 | *.obj 9 | 10 | # Precompiled Headers 11 | *.gch 12 | *.pch 13 | 14 | # Compiled Dynamic libraries 15 | *.so 16 | *.dylib 17 | *.dll 18 | 19 | # Fortran module files 20 | *.mod 21 | 22 | # Compiled Static libraries 23 | *.lai 24 | *.la 25 | *.a 26 | *.lib 27 | 28 | # Executables 29 | *.exe 30 | *.out 31 | *.app 32 | *.ipa 33 | 34 | # These project files can be generated by the engine 35 | *.xcodeproj 36 | *.xcworkspace 37 | *.sln 38 | *.suo 39 | *.opensdf 40 | *.sdf 41 | *.VC.db 42 | *.VC.opendb 43 | 44 | # Precompiled Assets 45 | SourceArt/**/*.png 46 | SourceArt/**/*.tga 47 | 48 | # Binary Files 49 | Binaries/* 50 | Plugins/*/Binaries/* 51 | 52 | # Builds 53 | Build/* 54 | 55 | # Whitelist PakBlacklist-.txt files 56 | !Build/*/ 57 | Build/*/** 58 | !Build/*/PakBlacklist*.txt 59 | 60 | # Don't ignore icon files in Build 61 | !Build/**/*.ico 62 | 63 | # Built data for maps 64 | *_BuiltData.uasset 65 | 66 | # Configuration files generated by the Editor 67 | Saved/* 68 | 69 | # Compiled source files for the engine to use 70 | Intermediate/* 71 | Plugins/*/Intermediate/* 72 | 73 | # Cache files for the editor to use 74 | DerivedDataCache/* 75 | node_modules -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2020 Slingshot Simulations 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # UnrealRenderStreamingExample 2 | 3 | Sample project demonstrating how to set up unreal render streaming within a JavaScript framework. 4 | 5 | This project is supposed to be a basic walkthrough of setting up the Unreal Pixel streaming within an iFrame rather than a fully fleshed out project. It should be used as a basis build on top of. 6 | 7 | ## Overview 8 | 9 | Setting up pixel streaming with an iFrame is a pretty straight forward process, there really isn't much to it. 10 | For demonstration this simple Vue.js app includes an iFrame to the default pixel streaming example provided by unreal, and uses the JavaScript method .postMessage() and standard event listeners to allow cross-origin communication between the Vue app and Signalling Server. In this example we're sending a custom message to the Signalling Server using a simple button click event as the trigger. We're also adding an event listener on the mounted Vue lifecycle hook to recieve messages sent from the Signalling Server to the Vue front end, this is logged out and set up as an alert, but should provide enough context to use the response in any capacity within the Vue app. 11 | 12 | The iFrame aspect ratio is maintained by using bound computed properties for the width and height. 13 | Scene interactivity is handled by the Signalling Server within the iFrame content window, this can be seen as the cursor crosses the boundary between Vue app and iFrame. This cursor behaviour is enabled by the Singalling Server using the inputOptions object with a choice between LockedMouse and HoveringMouse, using HoveringMouse removes the need to click on the iFrame to enable interactions. 14 | 15 | The unreal application itself is set up in the same way it would to handle any pixel streaming input / output. The example we use for this project simply logs a message recieved and sends the name of the object clicked. Below we include screenshots of the blueprints used to manage the scene (all of it is simply placed in the level blueprint), however for full documentation on the unreal application setup; the documentation found [here.](https://docs.unrealengine.com/en-US/Platforms/PixelStreaming/index.html) 16 | 17 | 18 | ## (Front End) Send Message via iFrame 19 | Below is an example of how to send the message via the iFrame. 20 | 21 | ```javascript 22 | // Send a json message via the iFrame 23 | document.getElementById("myIframe").contentWindow.postMessage(JSON.stringify(this.jsonMessage), "*"); 24 | ``` 25 | 26 | In our version we use a basic JSON object with a type property to allow more flexible handling of the data 27 | ```javascript 28 | window.onmessage = function(messageFromVue) { 29 | console.log("Message recieved on Signalling Server: " + messageFromVue.data); 30 | var jsonPayload = JSON.parse(messageFromVue.data); 31 | if(jsonPayload.type == "TestType") { 32 | emitUIInteraction(jsonPayload); 33 | } 34 | } 35 | ``` 36 | 37 | 38 | ## (Front End) Recieve iFrame Message 39 | To recieve data sent from the signalling server, simply add a listener to the window. The window may recieve messages from various sources, so it is worthwhile adding in checks in order to handle the data appropriately. 40 | ```javascript 41 | // Adds a listener for any messages coming from the iFrame 42 | window.onmessage = function (e) { 43 | console.log("Message Recieved from Signalling Server: " + e.data); 44 | }; 45 | ``` 46 | 47 | ## (Signalling Server) Recieve iFrame Message 48 | Recieving data from the VueJS application can be achieved in the same way as shown above, simply add in a listener to the window. As stated, the window may recieve messages from various sources so it is worthwhile adding in checks & try catches as appropriate. The content can be added in anywhere within the app.js within the Unreal Signalling server where it will be called. For a windows build with Pixel Streaming enabled, this can be found (as of version 4.25) in: **WindowsNoEditor\WindowsNoEditor\Engine\Source\Programs\PixelStreaming\WebServers\SignallingWebServer\scripts** 49 | ```javascript 50 | window.onmessage = function (messageFromVue) { 51 | // Parse the json payload 52 | var jsonPayload = JSON.parse(messageFromVue.data); 53 | // If it is a message we want to send on to the Unreal Application 54 | if (jsonPayload.type == "TestType") { 55 | // Log the contents of the payload 56 | console.log("Message from VueJS recieved on Signalling Server: " + messageFromVue.data); 57 | // Send the payload to the Unreal Application 58 | emitUIInteraction(jsonPayload); 59 | } 60 | }; 61 | ``` 62 | 63 | ## (Signalling Server) Recieve Message from Unreal and Send Message via iFrame 64 | To recieve messages from the unreal application, call the existing method within app.js called "addResponseEventListener", naming the listener and providing a method to handle the event. We simply added our call to the bottom of the load() method. From this example, the message will simply be a string related to the name of the object clicked in the unreal scene, however it can be a json object if required. 65 | ```javascript 66 | function handleUnrealMessage(message) { 67 | // Log the message recieved 68 | console.log("Message recieved on signalling server from unreal: " + message); 69 | // Post the data back to the VueJS app 70 | window.top.postMessage(message, "*"); 71 | } 72 | 73 | function load() { 74 | setupHtmlEvents(); 75 | setupFreezeFrameOverlay(); 76 | registerKeyboardEvents(); 77 | start(); 78 | addResponseEventListener("handle_responses", handleUnrealMessage); 79 | } 80 | ``` 81 | 82 | ## (Signalling Server) Cursor Behavior 83 | As mentioned above, by setting the control scheme type (found in app.js) within the inputOptions to HoveringMouse; you will not need to click on the iFrame to begin interacting if that is desired. 84 | ```javascript 85 | var inputOptions = { 86 | controlScheme: ControlSchemeType.HoveringMouse, 87 | }; 88 | ``` 89 | 90 | ## (Unreal Application) Recieve Message Blueprint 91 | The recieve message blueprint (added to the level blueprint) is very similar to what can be found in the documentation for pixel streaming on the Unreal docs, differing only by printing a message sent via VueJS to the screen. 92 | ![alt text](https://github.com/Slingshot-Simulations/UnrealRenderPixelExample/blob/IFrame-Example/ReferenceImages/UnrealRecieveMessageBP.png "Recieve Message Blueprint") 93 | 94 | ## (Unreal Application) Send Message Blueprint 95 | The send message blueprint (also added to the level blueprint) differs slightly from what is found in the Unreal docs, simply getting the name of a clicked actor within the scene and sending back its string name. As mentioned in the unreal docs, this can be a json string also if you want to be able to handle more complex object structures. 96 | ![alt text](https://github.com/Slingshot-Simulations/UnrealRenderPixelExample/blob/IFrame-Example/ReferenceImages/UnrealSendMessageBP.png "Send Message Blueprint") 97 | 98 | ## (Unreal Application)Full Blueprint 99 | Exactly the same functionality as the two blueprints above simply merged into the same blueprint to handle both sending and recieving data from the signalling server. 100 | ![alt text](https://github.com/Slingshot-Simulations/UnrealRenderPixelExample/blob/IFrame-Example/ReferenceImages/UnrealFullBP.png "Full Blueprint") 101 | 102 | ## Send Message to Unreal Example 103 | 104 | ![alt text](https://github.com/Slingshot-Simulations/UnrealRenderPixelExample/blob/IFrame-Example/ReferenceImages/FromVue.png "Full Blueprint") 105 | 106 | ## Recieve Message from Unreal Example 107 | For clearer images of the example working see [here.](https://github.com/Slingshot-Simulations/UnrealPixelStreamingExamples/tree/IFrame-Example/ReferenceImages) 108 | ![alt text](https://github.com/Slingshot-Simulations/UnrealRenderPixelExample/blob/IFrame-Example/ReferenceImages/FromUnreal.png "Full Blueprint") 109 | -------------------------------------------------------------------------------- /ReferenceImages/FromUnreal.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/digitaltwinconsortium/UnrealPixelStreamingExamples/8826613e453c46a9799de70fb44b7e72455bf4b9/ReferenceImages/FromUnreal.png -------------------------------------------------------------------------------- /ReferenceImages/FromVue.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/digitaltwinconsortium/UnrealPixelStreamingExamples/8826613e453c46a9799de70fb44b7e72455bf4b9/ReferenceImages/FromVue.png -------------------------------------------------------------------------------- /ReferenceImages/SelectChair.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/digitaltwinconsortium/UnrealPixelStreamingExamples/8826613e453c46a9799de70fb44b7e72455bf4b9/ReferenceImages/SelectChair.png -------------------------------------------------------------------------------- /ReferenceImages/SelectFloor.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/digitaltwinconsortium/UnrealPixelStreamingExamples/8826613e453c46a9799de70fb44b7e72455bf4b9/ReferenceImages/SelectFloor.png -------------------------------------------------------------------------------- /ReferenceImages/SelectStatue.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/digitaltwinconsortium/UnrealPixelStreamingExamples/8826613e453c46a9799de70fb44b7e72455bf4b9/ReferenceImages/SelectStatue.png -------------------------------------------------------------------------------- /ReferenceImages/SelectTable.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/digitaltwinconsortium/UnrealPixelStreamingExamples/8826613e453c46a9799de70fb44b7e72455bf4b9/ReferenceImages/SelectTable.png -------------------------------------------------------------------------------- /ReferenceImages/UnrealFullBP.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/digitaltwinconsortium/UnrealPixelStreamingExamples/8826613e453c46a9799de70fb44b7e72455bf4b9/ReferenceImages/UnrealFullBP.png -------------------------------------------------------------------------------- /ReferenceImages/UnrealRecieveMessageBP.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/digitaltwinconsortium/UnrealPixelStreamingExamples/8826613e453c46a9799de70fb44b7e72455bf4b9/ReferenceImages/UnrealRecieveMessageBP.png -------------------------------------------------------------------------------- /ReferenceImages/UnrealSendMessageBP.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/digitaltwinconsortium/UnrealPixelStreamingExamples/8826613e453c46a9799de70fb44b7e72455bf4b9/ReferenceImages/UnrealSendMessageBP.png -------------------------------------------------------------------------------- /unreal-pixel-streaming-app/.gitignore: -------------------------------------------------------------------------------- 1 | .DS_Store 2 | node_modules 3 | /dist 4 | 5 | 6 | # local env files 7 | .env.local 8 | .env.*.local 9 | 10 | # Log files 11 | npm-debug.log* 12 | yarn-debug.log* 13 | yarn-error.log* 14 | pnpm-debug.log* 15 | 16 | # Editor directories and files 17 | .idea 18 | .vscode 19 | *.suo 20 | *.ntvs* 21 | *.njsproj 22 | *.sln 23 | *.sw? 24 | -------------------------------------------------------------------------------- /unreal-pixel-streaming-app/README.md: -------------------------------------------------------------------------------- 1 | # unreal-pixel-streaming-app 2 | 3 | ## Project setup 4 | ``` 5 | npm install 6 | ``` 7 | 8 | ### Compiles and hot-reloads for development 9 | ``` 10 | npm run serve 11 | ``` 12 | 13 | ### Compiles and minifies for production 14 | ``` 15 | npm run build 16 | ``` 17 | 18 | ### Lints and fixes files 19 | ``` 20 | npm run lint 21 | ``` 22 | 23 | ### Customize configuration 24 | See [Configuration Reference](https://cli.vuejs.org/config/). 25 | -------------------------------------------------------------------------------- /unreal-pixel-streaming-app/babel.config.js: -------------------------------------------------------------------------------- 1 | module.exports = { 2 | presets: [ 3 | '@vue/cli-plugin-babel/preset' 4 | ] 5 | } 6 | -------------------------------------------------------------------------------- /unreal-pixel-streaming-app/package.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "unreal-pixel-streaming-app", 3 | "version": "0.1.0", 4 | "private": true, 5 | "scripts": { 6 | "serve": "vue-cli-service serve", 7 | "build": "vue-cli-service build", 8 | "lint": "vue-cli-service lint" 9 | }, 10 | "dependencies": { 11 | "core-js": "^3.6.5", 12 | "vue": "^2.6.11", 13 | "vue-router": "^3.2.0", 14 | "vuex": "^3.4.0" 15 | }, 16 | "devDependencies": { 17 | "@vue/cli-plugin-babel": "~4.5.0", 18 | "@vue/cli-plugin-eslint": "~4.5.0", 19 | "@vue/cli-plugin-router": "~4.5.0", 20 | "@vue/cli-plugin-vuex": "~4.5.0", 21 | "@vue/cli-service": "~4.5.0", 22 | "babel-eslint": "^10.1.0", 23 | "eslint": "^6.7.2", 24 | "eslint-plugin-vue": "^6.2.2", 25 | "vue-template-compiler": "^2.6.11" 26 | }, 27 | "eslintConfig": { 28 | "root": true, 29 | "env": { 30 | "node": true 31 | }, 32 | "extends": [ 33 | "plugin:vue/essential", 34 | "eslint:recommended" 35 | ], 36 | "parserOptions": { 37 | "parser": "babel-eslint" 38 | }, 39 | "rules": {} 40 | }, 41 | "browserslist": [ 42 | "> 1%", 43 | "last 2 versions", 44 | "not dead" 45 | ] 46 | } 47 | -------------------------------------------------------------------------------- /unreal-pixel-streaming-app/public/favicon.ico: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/digitaltwinconsortium/UnrealPixelStreamingExamples/8826613e453c46a9799de70fb44b7e72455bf4b9/unreal-pixel-streaming-app/public/favicon.ico -------------------------------------------------------------------------------- /unreal-pixel-streaming-app/public/index.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | <%= htmlWebpackPlugin.options.title %> 9 | 10 | 11 | 14 |
15 | 16 | 17 | 18 | -------------------------------------------------------------------------------- /unreal-pixel-streaming-app/src/App.vue: -------------------------------------------------------------------------------- 1 | 9 | 10 | 32 | -------------------------------------------------------------------------------- /unreal-pixel-streaming-app/src/assets/logo.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/digitaltwinconsortium/UnrealPixelStreamingExamples/8826613e453c46a9799de70fb44b7e72455bf4b9/unreal-pixel-streaming-app/src/assets/logo.png -------------------------------------------------------------------------------- /unreal-pixel-streaming-app/src/main.js: -------------------------------------------------------------------------------- 1 | import Vue from 'vue' 2 | import App from './App.vue' 3 | import router from './router' 4 | import store from './store' 5 | 6 | Vue.config.productionTip = false 7 | 8 | new Vue({ 9 | router, 10 | store, 11 | render: h => h(App) 12 | }).$mount('#app') 13 | -------------------------------------------------------------------------------- /unreal-pixel-streaming-app/src/router/index.js: -------------------------------------------------------------------------------- 1 | import Vue from 'vue' 2 | import VueRouter from 'vue-router' 3 | import Home from '../views/Home.vue' 4 | 5 | Vue.use(VueRouter) 6 | 7 | const routes = [ 8 | { 9 | path: '/', 10 | name: 'Home', 11 | component: Home 12 | } 13 | ] 14 | 15 | const router = new VueRouter({ 16 | routes 17 | }) 18 | 19 | export default router 20 | -------------------------------------------------------------------------------- /unreal-pixel-streaming-app/src/store/index.js: -------------------------------------------------------------------------------- 1 | import Vue from 'vue' 2 | import Vuex from 'vuex' 3 | 4 | Vue.use(Vuex) 5 | 6 | export default new Vuex.Store({ 7 | state: { 8 | }, 9 | mutations: { 10 | }, 11 | actions: { 12 | }, 13 | modules: { 14 | } 15 | }) 16 | -------------------------------------------------------------------------------- /unreal-pixel-streaming-app/src/views/Home.vue: -------------------------------------------------------------------------------- 1 | 17 | 18 | 68 | 69 | 76 | --------------------------------------------------------------------------------