├── Binaries └── Win64 │ └── place_pytorch_and_opencv_DLLs_here ├── Content ├── LibTorchMaterial.uasset ├── LibTorchTestLevel.umap └── LibTorchTestLevel_BuiltData.uasset ├── Images ├── level.jpg ├── plugin_a.jpg ├── project_settings.jpg └── viewer.jpg ├── LICENSE ├── LibTorchPlugin.uplugin ├── README.md ├── Resources └── Icon128.png ├── Source └── LibTorchPlugin │ ├── LibTorchPlugin.Build.cs │ ├── Private │ ├── LibTorchPlugin.cpp │ ├── LibTorchViewer.cpp │ ├── cDataStorageGameInstance.cpp │ └── cDataStorageWrapper.cpp │ └── Public │ ├── LibTorchPlugin.h │ ├── LibTorchViewer.h │ ├── cDataStorageGameInstance.h │ └── cDataStorageWrapper.h ├── USAGE.md └── place_export_wrapper_dll_here /Binaries/Win64/place_pytorch_and_opencv_DLLs_here: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/NeuralVFX/basic-unreal-libtorch-plugin/3bcc708816c19a0cf33f4833a7a58f15910df738/Binaries/Win64/place_pytorch_and_opencv_DLLs_here -------------------------------------------------------------------------------- /Content/LibTorchMaterial.uasset: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/NeuralVFX/basic-unreal-libtorch-plugin/3bcc708816c19a0cf33f4833a7a58f15910df738/Content/LibTorchMaterial.uasset -------------------------------------------------------------------------------- /Content/LibTorchTestLevel.umap: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/NeuralVFX/basic-unreal-libtorch-plugin/3bcc708816c19a0cf33f4833a7a58f15910df738/Content/LibTorchTestLevel.umap -------------------------------------------------------------------------------- /Content/LibTorchTestLevel_BuiltData.uasset: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/NeuralVFX/basic-unreal-libtorch-plugin/3bcc708816c19a0cf33f4833a7a58f15910df738/Content/LibTorchTestLevel_BuiltData.uasset -------------------------------------------------------------------------------- /Images/level.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/NeuralVFX/basic-unreal-libtorch-plugin/3bcc708816c19a0cf33f4833a7a58f15910df738/Images/level.jpg -------------------------------------------------------------------------------- /Images/plugin_a.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/NeuralVFX/basic-unreal-libtorch-plugin/3bcc708816c19a0cf33f4833a7a58f15910df738/Images/plugin_a.jpg -------------------------------------------------------------------------------- /Images/project_settings.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/NeuralVFX/basic-unreal-libtorch-plugin/3bcc708816c19a0cf33f4833a7a58f15910df738/Images/project_settings.jpg -------------------------------------------------------------------------------- /Images/viewer.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/NeuralVFX/basic-unreal-libtorch-plugin/3bcc708816c19a0cf33f4833a7a58f15910df738/Images/viewer.jpg -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2020 charliewinter.vfx@gmail.com 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /LibTorchPlugin.uplugin: -------------------------------------------------------------------------------- 1 | { 2 | "FileVersion": 3, 3 | "Version": 1, 4 | "VersionName": "1.0", 5 | "FriendlyName": "LibTorchPlugin", 6 | "Description": "Hello World for LibTorch", 7 | "Category": "Other", 8 | "CreatedBy": "NeuralVFX", 9 | "CreatedByURL": "", 10 | "DocsURL": "", 11 | "MarketplaceURL": "", 12 | "SupportURL": "", 13 | "CanContainContent": true, 14 | "IsBetaVersion": false, 15 | "Installed": false, 16 | "Modules": [ 17 | { 18 | "Name": "LibTorchPlugin", 19 | "Type": "Developer", 20 | "LoadingPhase": "Default" 21 | } 22 | ] 23 | } -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | ![](https://github.com/NeuralVFX/basic-libtorch-dll/blob/master/style.gif) 2 | # Basic-Unreal-Libtorch-Plugin 3 | 4 | A `"Hello World"` for running `LibTorch` inside `Unreal Engine`. 5 | 6 | ## About 7 | This is one of two repositories which are required to build this project: 8 | - [basic-unreal-libtorch-plugin](https://github.com/NeuralVFX/basic-unreal-libtorch-plugin) - You are here. 9 | - [basic-libtorch-dll](https://github.com/NeuralVFX/basic-libtorch-dll) 10 | 11 | 12 | ## Extra Info 13 | - Runs Style Transfer on a live video feed, using `OpenCV` and `LibTorch` 14 | - Neural Net Inference is performed inside a `DLL` and then the data is passed into `Unreal Engine` 15 | - A `GameInstance Class` is used to initiate the `DLL` and execute its function 16 | - An `Actor Class` queries the `GameInstance Class` each tick, and updates its texture with the resulting image 17 | 18 | 19 | ## Code Usage 20 | Usage instructions found here: [user manual page](USAGE.md). 21 | 22 | 23 | 24 | -------------------------------------------------------------------------------- /Resources/Icon128.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/NeuralVFX/basic-unreal-libtorch-plugin/3bcc708816c19a0cf33f4833a7a58f15910df738/Resources/Icon128.png -------------------------------------------------------------------------------- /Source/LibTorchPlugin/LibTorchPlugin.Build.cs: -------------------------------------------------------------------------------- 1 | // Copyright 1998-2018 Epic Games, Inc. All Rights Reserved. 2 | 3 | using UnrealBuildTool; 4 | 5 | public class LibTorchPlugin : ModuleRules 6 | { 7 | public LibTorchPlugin(ReadOnlyTargetRules Target) : base(Target) 8 | { 9 | PCHUsage = ModuleRules.PCHUsageMode.UseExplicitOrSharedPCHs; 10 | 11 | PublicIncludePaths.AddRange( 12 | new string[] { 13 | // ... add public include paths required here ... 14 | } 15 | ); 16 | 17 | 18 | PrivateIncludePaths.AddRange( 19 | new string[] { 20 | // ... add other private include paths required here ... 21 | } 22 | ); 23 | 24 | 25 | PublicDependencyModuleNames.AddRange( 26 | new string[] 27 | { 28 | "Core", 29 | // ... add other public dependencies that you statically link with here ... 30 | } 31 | ); 32 | 33 | 34 | PrivateDependencyModuleNames.AddRange( 35 | new string[] 36 | { 37 | "CoreUObject", 38 | "Engine", 39 | "Slate", 40 | "SlateCore", 41 | // ... add private dependencies that you statically link with here ... 42 | } 43 | ); 44 | 45 | 46 | DynamicallyLoadedModuleNames.AddRange( 47 | new string[] 48 | { 49 | // ... add any modules that your module loads dynamically here ... 50 | } 51 | ); 52 | } 53 | } 54 | -------------------------------------------------------------------------------- /Source/LibTorchPlugin/Private/LibTorchPlugin.cpp: -------------------------------------------------------------------------------- 1 | // Copyright 1998-2018 Epic Games, Inc. All Rights Reserved. 2 | 3 | #include "LibTorchPlugin.h" 4 | 5 | #define LOCTEXT_NAMESPACE "FLibTorchPluginModule" 6 | 7 | 8 | void FLibTorchPluginModule::StartupModule() 9 | { 10 | } 11 | 12 | 13 | void FLibTorchPluginModule::ShutdownModule() 14 | { 15 | } 16 | 17 | 18 | #undef LOCTEXT_NAMESPACE 19 | 20 | IMPLEMENT_MODULE(FLibTorchPluginModule, LibTorchPlugin) -------------------------------------------------------------------------------- /Source/LibTorchPlugin/Private/LibTorchViewer.cpp: -------------------------------------------------------------------------------- 1 | // Copyright 2020 NeuralVFX, Inc. All Rights Reserved. 2 | 3 | #include "LibTorchViewer.h" 4 | #include "cDataStorageGameInstance.h" 5 | #include "Components/StaticMeshComponent.h" 6 | #include "UObject/ConstructorHelpers.h" 7 | #include "Materials/MaterialInstance.h" 8 | #include "Materials/MaterialInstanceDynamic.h" 9 | #include "Engine/Texture2D.h" 10 | 11 | 12 | ALibTorchViewer::ALibTorchViewer() 13 | { 14 | PrimaryActorTick.bCanEverTick = true; 15 | 16 | // Create and Set Plane Mesh Component 17 | PlaneMesh = CreateDefaultSubobject(TEXT("PlaneMesh")); 18 | static ConstructorHelpers::FObjectFinderMeshAsset(TEXT("StaticMesh'/Engine/BasicShapes/Plane.Plane'")); 19 | PlaneMesh->SetStaticMesh(MeshAsset.Object); 20 | 21 | // Fetch LibTorch Material From Scene 22 | static ConstructorHelpers::FObjectFinderMaterialAsset(TEXT("Material'/LibTorchPlugin/LibTorchMaterial.LibTorchMaterial'")); 23 | PlaneMesh->SetMaterial(0, MaterialAsset.Object); 24 | } 25 | 26 | 27 | void ALibTorchViewer::BeginPlay() 28 | { 29 | // Start Camera 30 | Super::BeginPlay(); 31 | 32 | // Make Material Instance 33 | UMaterialInstance * Material = (UMaterialInstance *)PlaneMesh->GetMaterial(0); 34 | MasterMaterialRef = Material; 35 | } 36 | 37 | 38 | void ALibTorchViewer::Tick(float DeltaTime) 39 | { 40 | // Open GameInstance 41 | UcDataStorageGameInstance * GameInst = (UcDataStorageGameInstance*)GetGameInstance(); 42 | 43 | // Get Image Data From LibTorch Model 44 | TArray> Image; 45 | Image.SetNumZeroed(512 * 512 * 4); 46 | GameInst->GetImage(Image.GetData()); 47 | 48 | // Make Texture and Set with LibTorch Result 49 | UMaterialInstanceDynamic* MatInst = UMaterialInstanceDynamic::Create(MasterMaterialRef, this); 50 | UTexture2D* Texture = UTexture2D::CreateTransient(512, 512, PF_B8G8R8A8); 51 | FTexture2DMipMap& Mip = Texture->PlatformData->Mips[0]; 52 | void* Data = Mip.BulkData.Lock(LOCK_READ_WRITE); 53 | FMemory::Memcpy(Data, Image.GetData(), 512 * 512 * 4); 54 | Mip.BulkData.Unlock(); 55 | 56 | // Place Texture Into Material Parameter 57 | Texture->UpdateResource(); 58 | MatInst->SetTextureParameterValue(FName("LibTorchInput"), (UTexture*)Texture); 59 | 60 | // Set Material 61 | PlaneMesh->SetMaterial(0, MatInst); 62 | 63 | Super::Tick(DeltaTime); 64 | } 65 | 66 | -------------------------------------------------------------------------------- /Source/LibTorchPlugin/Private/cDataStorageGameInstance.cpp: -------------------------------------------------------------------------------- 1 | // Copyright 2020 NeuralVFX, Inc. All Rights Reserved. 2 | 3 | #include "cDataStorageGameInstance.h" 4 | #include "cDataStorageWrapper.h" 5 | 6 | 7 | void UcDataStorageGameInstance::Init() 8 | { 9 | // Init DLL 10 | Super::Init(); 11 | if (ImportDataStorageLibrary()) 12 | { 13 | UE_LOG(LogTemp, Log, TEXT("LibTorch DLL Loaded")); 14 | 15 | } 16 | } 17 | 18 | 19 | bool UcDataStorageGameInstance::ImportDataStorageLibrary() 20 | { 21 | // Import the DLL and Load Functions 22 | m_refDataStorageUtil = NewObject(this); 23 | if (m_refDataStorageUtil == NULL) 24 | { 25 | UE_LOG(LogTemp, Error, TEXT("Could not Create the Data Storage Object")); 26 | return false; 27 | } 28 | 29 | if (!m_refDataStorageUtil->ImportDLL("basic-unreal-libtorch-plugin", "export_wrapper.dll")) 30 | { 31 | UE_LOG(LogTemp, Error, TEXT("Failed to Import DLL")); 32 | return false; 33 | } 34 | if (!m_refDataStorageUtil->ImportMethods()) 35 | { 36 | UE_LOG(LogTemp, Error, TEXT("Failed to Import DLL Methods")); 37 | return false; 38 | } 39 | return true; 40 | } 41 | 42 | 43 | void UcDataStorageGameInstance::Shutdown() 44 | { 45 | // Calls DLL Function to ShutDown Camera 46 | m_refDataStorageUtil->CallCloseCV(); 47 | Super::Shutdown(); 48 | UE_LOG(LogTemp, Error, TEXT("Release Camera")) 49 | } 50 | 51 | 52 | void UcDataStorageGameInstance::OnStart() 53 | { 54 | // Calls DLL Function to Activate Camera 55 | m_refDataStorageUtil->CallInitCV(); 56 | UE_LOG(LogTemp, Log, TEXT("Opened Camera")); 57 | Super::OnStart(); 58 | } 59 | 60 | 61 | void UcDataStorageGameInstance::GetImage(unsigned char* image) 62 | { 63 | // Calls DLL Function to Run LibTorch Model and Return Image 64 | m_refDataStorageUtil->CallGetImageCV(image); 65 | UE_LOG(LogTemp, Log, TEXT("CV Image Raad")); 66 | } 67 | 68 | -------------------------------------------------------------------------------- /Source/LibTorchPlugin/Private/cDataStorageWrapper.cpp: -------------------------------------------------------------------------------- 1 | // Copyright 2020 NeuralVFX, Inc. All Rights Reserved. 2 | 3 | #include "cDataStorageWrapper.h" 4 | #include "Paths.h" 5 | 6 | 7 | bool UcDataStorageWrapper::ImportDLL(FString FolderName, FString DLLName) 8 | { 9 | // Init DLL from a Path 10 | FString FilePath = *FPaths::GamePluginsDir() + FolderName + "/" + DLLName; 11 | if (FPaths::FileExists(FilePath)) 12 | { 13 | v_dllHandle = FPlatformProcess::GetDllHandle(*FilePath); 14 | if (v_dllHandle != NULL) 15 | { 16 | return true; 17 | } 18 | } 19 | return false; 20 | } 21 | 22 | 23 | bool UcDataStorageWrapper::ImportMethods() 24 | { 25 | // Loop Through and Import All Functions from DLL -- Make Sure proc_name matches name of DLL method 26 | if (v_dllHandle != NULL) 27 | { 28 | FString ProcName = "InitNet"; 29 | m_funcInitCV = (__Init)FPlatformProcess::GetDllExport(v_dllHandle, *ProcName); 30 | if (m_funcInitCV == NULL) 31 | { 32 | return false; 33 | } 34 | ProcName = "CloseNet"; 35 | m_funcCloseCV = (__Close)FPlatformProcess::GetDllExport(v_dllHandle, *ProcName); 36 | if (m_funcCloseCV == NULL) 37 | { 38 | return false; 39 | } 40 | ProcName = "GetImage"; 41 | m_funcGetImageCV = (__GetImage)FPlatformProcess::GetDllExport(v_dllHandle, *ProcName); 42 | if (m_funcGetImageCV == NULL) 43 | { 44 | return false; 45 | } 46 | } 47 | return true; 48 | } 49 | 50 | 51 | void UcDataStorageWrapper::CallInitCV() 52 | { 53 | // Calls DLL Function to Activate Camera 54 | if (m_funcInitCV == NULL) 55 | { 56 | UE_LOG(LogTemp, Error, TEXT("Function Not Loaded From DLL: InitNet")); 57 | } 58 | bool init = m_funcInitCV(); 59 | UE_LOG(LogTemp, Error, TEXT("OpenCV Connection Opened %s"), init ? "true" : "false"); 60 | } 61 | 62 | 63 | void UcDataStorageWrapper::CallCloseCV() 64 | { 65 | // Calls DLL Function to ShutDown Camera 66 | if (m_funcCloseCV == NULL) 67 | { 68 | UE_LOG(LogTemp, Error, TEXT("Function Not Loaded From DLL: CloseNet")); 69 | } 70 | m_funcCloseCV(); 71 | UE_LOG(LogTemp, Error, TEXT("OpenCV Connection Close")); 72 | } 73 | 74 | 75 | void UcDataStorageWrapper::CallGetImageCV(unsigned char* Image) 76 | { 77 | // Calls DLL Function to Run LibTorch Model and Return Image 78 | if (m_funcGetImageCV == NULL) 79 | { 80 | UE_LOG(LogTemp, Error, TEXT("Function Not Loaded From DLL: Get Image ")); 81 | } 82 | 83 | bool GetImage = m_funcGetImageCV(Image); 84 | 85 | if (GetImage == false) 86 | { 87 | UE_LOG(LogTemp, Error, TEXT("OpenCV Image Could Not Be Loaded")); 88 | } 89 | } 90 | -------------------------------------------------------------------------------- /Source/LibTorchPlugin/Public/LibTorchPlugin.h: -------------------------------------------------------------------------------- 1 | // Copyright 1998-2018 Epic Games, Inc. All Rights Reserved. 2 | 3 | #pragma once 4 | 5 | #include "CoreMinimal.h" 6 | #include "Modules/ModuleManager.h" 7 | 8 | class FLibTorchPluginModule : public IModuleInterface 9 | { 10 | public: 11 | 12 | /** IModuleInterface implementation */ 13 | virtual void StartupModule() override; 14 | virtual void ShutdownModule() override; 15 | }; 16 | -------------------------------------------------------------------------------- /Source/LibTorchPlugin/Public/LibTorchViewer.h: -------------------------------------------------------------------------------- 1 | // Copyright 2020 NeuralVFX, Inc. All Rights Reserved. 2 | 3 | #pragma once 4 | 5 | #include "CoreMinimal.h" 6 | #include "GameFramework/Actor.h" 7 | #include "LibTorchViewer.generated.h" 8 | 9 | /** Just a plane mesh which displays the output of the neural net as an animated texture */ 10 | UCLASS() 11 | class LIBTORCHPLUGIN_API ALibTorchViewer : public AActor 12 | { 13 | GENERATED_BODY() 14 | 15 | public: 16 | 17 | ALibTorchViewer(); 18 | 19 | /** Mesh to Render Texture On */ 20 | UPROPERTY(EditAnywhere, BlueprintReadWrite, Category = "Output Settings") 21 | class UStaticMeshComponent* PlaneMesh; 22 | 23 | /** Material to Override Texture On */ 24 | UPROPERTY(EditDefaultsOnly, Category = Materials) 25 | class UMaterialInstance * MasterMaterialRef; 26 | 27 | protected: 28 | 29 | virtual void BeginPlay() override; 30 | 31 | public: 32 | 33 | virtual void Tick(float DeltaTime) override; 34 | 35 | 36 | 37 | 38 | }; 39 | 40 | -------------------------------------------------------------------------------- /Source/LibTorchPlugin/Public/cDataStorageGameInstance.h: -------------------------------------------------------------------------------- 1 | // Copyright 2020 NeuralVFX, Inc. All Rights Reserved. 2 | 3 | #pragma once 4 | 5 | #include "CoreMinimal.h" 6 | #include "Engine/GameInstance.h" 7 | #include "cDataStorageGameInstance.generated.h" 8 | 9 | /** Game Instance which is responsible for loading and calling DLL wrapper */ 10 | UCLASS() 11 | class LIBTORCHPLUGIN_API UcDataStorageGameInstance : public UGameInstance 12 | { 13 | GENERATED_BODY() 14 | 15 | private: 16 | 17 | /** Storage for DLL object */ 18 | UPROPERTY() 19 | class UcDataStorageWrapper* m_refDataStorageUtil; 20 | 21 | /** 22 | * Attempt to import DLL and all of its functions. 23 | * @return Whether the operation is succesfull. 24 | */ 25 | bool ImportDataStorageLibrary(); 26 | 27 | public: 28 | 29 | /** 30 | * Overridden to import DLL and DLL functions on Init. 31 | */ 32 | virtual void Init() override; 33 | 34 | /** 35 | * Overridden to open the camera with OpenCV. 36 | */ 37 | virtual void OnStart() override; 38 | 39 | /** 40 | * Overridden to close the camera with OpenCV. 41 | */ 42 | virtual void Shutdown() override; 43 | 44 | /** 45 | * Runs neural net on camera frame, then returns the result. 46 | * @param image - Raw image pointer to return information from DLL. 47 | */ 48 | void GetImage(unsigned char* image); 49 | 50 | }; 51 | -------------------------------------------------------------------------------- /Source/LibTorchPlugin/Public/cDataStorageWrapper.h: -------------------------------------------------------------------------------- 1 | // Copyright 2020 NeuralVFX, Inc. All Rights Reserved. 2 | 3 | #pragma once 4 | 5 | #include "CoreMinimal.h" 6 | #include "UObject/NoExportTypes.h" 7 | #include "cDataStorageWrapper.generated.h" 8 | 9 | typedef bool(*__Init)(); 10 | typedef void(*__Close)(); 11 | typedef bool(*__GetImage)(unsigned char* image); 12 | 13 | /** Wrapper for external DLL, allows the executing of the neural net and passes the data back to Unreal */ 14 | UCLASS() 15 | class LIBTORCHPLUGIN_API UcDataStorageWrapper : public UObject 16 | { 17 | GENERATED_BODY() 18 | private: 19 | 20 | /** DLL Handle */ 21 | void *v_dllHandle; 22 | 23 | /** DLL Functions */ 24 | __Init m_funcInitCV; 25 | __Close m_funcCloseCV; 26 | __GetImage m_funcGetImageCV; 27 | 28 | 29 | public: 30 | 31 | /** 32 | * Attempt to import DLL. 33 | * @param a_strFolderName - Folder of DLL. 34 | * @param a_strDLLName - Name of DLL file. 35 | * @return Whether the operation is succesfull. 36 | */ 37 | bool ImportDLL(FString a_strFolderName, FString a_strDLLName); 38 | 39 | /** 40 | * Attempt to import all functions of the DLL. 41 | * @return Whether the operation is succesfull. 42 | */ 43 | bool ImportMethods(); 44 | 45 | /** 46 | * Open the camera with OpenCV. 47 | */ 48 | void CallInitCV(); 49 | 50 | /** 51 | * Close the camera with OpenCV. 52 | */ 53 | void CallCloseCV(); 54 | 55 | /** 56 | * Runs neural net on camera frame, then returns the result. 57 | * @param image - Raw image pointer to return information from DLL. 58 | */ 59 | void CallGetImageCV(unsigned char* image); 60 | }; 61 | 62 | 63 | 64 | 65 | 66 | 67 | -------------------------------------------------------------------------------- /USAGE.md: -------------------------------------------------------------------------------- 1 | 2 | # Getting Started 3 | 4 | ## Requirements: 5 | - Unreal Engine 4.20.3 6 | - Visual Studio 7 | - OpenCV 4.1.2 8 | - LibTorch 1.5.1 (CPU) 9 | 10 | ## Project Setup 11 | 12 | #### New Project 13 | - Make a C++ Unreal Project 14 | - Make sure you have a `Plugins` folder inside of the project 15 | 16 | #### Compile 17 | - Clone this Repo into the `Plugins` folder: `git clone https://github.com/NeuralVFX/basic-unreal-libtorch-plugin.git` (Or download the zip file and unzip there) 18 | - Right-click your `.uproject` file and click `Generate Visual Studio Project Files` 19 | - Build the project from Visual Studio, using `Build Solution` 20 | 21 | #### DLLs 22 | - Follow [directions](https://github.com/NeuralVFX/basic-libtorch-dll ) to compile `export_wrapper.dll` and place into the main plugin directory (ie:`Plugins/basic-unreal-libtorch-plugin`) 23 | - Copy `LibTorch` and `OpenCV` `DLLs` into the plugin's `Binaries` folder (ie:`Plugins/basic-unreal-libtorch-plugin/Binaries/Win64`) 24 | 25 | #### Find Plugin 26 | - Open the compiled project in Unreal Engine 27 | - From `Edit->Plugins`, locate the `Other` Category 28 | - Find `LibTorchPlugin` and click `Enable` 29 | 30 | ![](Images/plugin_a.jpg) 31 | 32 | #### Set GameInstance 33 | 34 | - Open `Settings->Project Settings` and find `GameInstanceClass`, replace this with `cDataStorageGameInstance` 35 | 36 | ![](Images/project_settings.jpg) 37 | 38 | ## Run It 39 | 40 | #### Run Test Scene 41 | 42 | - In the Content Manager, navigate to `LibTorchPlugin Content->LibTorchTestLevel` and open this level 43 | - Play the level, and you should see the Style Transfer on the `LibTorchViewer` object 44 | 45 | ![](Images/level.jpg) 46 | 47 | 48 | #### Scene Setup From Scratch 49 | 50 | - In the Content Manager, navigate to `LibTorchPlugin C++ Classes->LibTorchPlugin->Public` 51 | - Click on `LibTorchViewer` and drag this into your scene file 52 | - Play the level, and you should see the Style Transfer on the `LibTorchViewer` object 53 | 54 | ![](Images/viewer.jpg) 55 | 56 | ## Classes 57 | 58 | #### LibTorchViewer - Actor Class 59 | - Actor Class which displays LibTorch output as a texture 60 | - On every tick, this retrieves and updates texture 61 | - The texture is queried from `UDataStorageGameInstance` 62 | 63 | #### cDataStoageWrapper - Object Class 64 | - Wrapper for `export_wrapper.dll` 65 | - Finds and initiates `DLL` 66 | - Exposes functions of `DLL` 67 | 68 | #### UDataStorageGameInstance - GameInstance Class 69 | - This is a wrapper for `cDataStoageWrapper` 70 | - Manages starting and stopping `OpenCV` based on the game state 71 | - Retrieves `LibTorch` output, to pass on to `LibTorchViewer ` 72 | 73 | ## Content 74 | 75 | #### LibTorchMaterial - Material 76 | - A Material for displaying LibTorch output 77 | - Has a `Parameter` called `LibTorchInput`, used to locate the texture of this material from the `LibTorchViewer` class in C++ 78 | - This Material is assigned by default to the `LibTorchViewer` Actor on creation 79 | 80 | #### LibTorchTestLevel - Level 81 | - A simple example level containing a `LibTorchViewer` 82 | -------------------------------------------------------------------------------- /place_export_wrapper_dll_here: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/NeuralVFX/basic-unreal-libtorch-plugin/3bcc708816c19a0cf33f4833a7a58f15910df738/place_export_wrapper_dll_here --------------------------------------------------------------------------------