Setting up Virtual Production in Rokoko Studio

Introduction

In this Virtual Production article, we will go through the set up of the following:

• the Vive tracker inside Unreal Engine
• the connection between Rokoko Studio and Unreal Engine for the Virtual Production (not livestream)

We will use the Sample project for this guide. You can find a Google Drive link with the Sample project here.

Requirements

• At least one Vive Tracker 2.0 or higher.
• At least two Vive Base Stations 2.0 or higher.
• Unreal Engine 5 up to version 5.3. For this project, which has the Rokoko plugin manually installed, we recommend 5.2 or 5.3 to avoid issues.
• A Rokoko Studio Pro subscription. 
• Rokoko Unreal Engine plugin.

Important!

• We strongly recommend using the Sample project included in this guide until further documentation is published.
• For setting up the Base Stations, please check the official Vive guide here.
• If you wish to start a Virtual Production project from scratch, you will still need to set up the livestream process depending on your character (Metahuman or a custom character)
• Extract the zipped project into this path before starting: C:\Users\username\Documents\Unreal Projects
  
• If you cannot see your project after opening Unreal Engine, make sure there is not a newly created folder in which the unzipped folder was unzipped into.
  
  Example: Project Folder Name > Project Folder Name > Project files
  It should be: Project Folder Name > Project files inside the Unreal Engine projects path
  
  

Extra Important

Since version 5.2, Unreal Engine has changed some functionalities regarding the Vive trackers. Due to this, we need to open Unreal Engine in an unorthodox way.

• Click the Windows Key or go to the Windows menu on the far left of the taskbar
• Type "run" and open the run application
• Type this command and click "OK"
  C:\Program Files\Epic Games\UE_5.2\Engine\Binaries\Win64\UnrealEditor.exe -xrtrackingonly
  
  You can type 5.1 or 5.3 as well
  
• Unreal Engine will open. Open your project by selecting it from there

This will allow the trackers to appear in Unreal Engine after following this guide. If you omit this step, even if you do everything correctly, you will not be able to see your tracker inside the Live Link

Before we start

There are certain plugins that are mandatory for this process. Please make sure you enable them inside Unreal Engine by going to Edit > Plugins before continuing with the guide.

The plugins that are needed are:

• Live Link
• LiveLinkXR
• OpenXR
• OpenXRViveTracker
• Rokoko Studio Live (can be enabled later as well)

Remember to restart Unreal Engine after enabling the above

Set up Vive Trackers in Unreal Engine

The official Vive guide for setting up the Vive Tracker in Unreal Engine can be found here. Please follow it up until the Unreal Engine setup. There are certain details that are mentioned here for the rest of that guide, so please follow along.

• In this Sample project the VRPawn Blueprint is named VP_Pawn instead. Search for VP pawn in the search bar in the All folder.
  
  
  
  
• Double-click on it and click Open Full Blueprint Editor
  
  
  
  
• In the components list on the left, click the VP_Pawn (self) and click the +Add button to add a Scene component.
  
  
  
  After this, the Vive guide is almost the same
  
  
• Add a MotionControllerComponent and make sure the parent is the newly added Scene Component.
• In the MotionControllerComponent's Details panel, scroll down to Motion Controller -> Motion Source, and select ExternalCamera.
  1. This setting defines the relationship between the assigned Tracker Role (Camera) in SteamVR, and the MotionControllerComponent.
• Add a new XRDeviceVisualizationComponent as a child to the new MotionControllerComponent.
• In the XRDeviceVisualizationComponent's Details Panel, scroll down to Visualization and set IsVisualizationActive to True, Display Model Source to Custom, and select a mesh that will represent the Vive Tracker in the game world.
  
  
  
  
• Done! Your tracker is connected to Unreal Engine. Make sure you open SteamVR and calibrate your tracker correctly.

Set up the connection between Rokoko Studio and Unreal Engine

In this step, we will connect Rokoko Studio and Unreal Engine using the Position Correction Blueprint that exists in the project.

• Search for the Position Correction Blueprint in the search bar in the All folder and open it.
  
  If you cannot find it, make sure you can see all the files in the Project. You can check that if you go to the Settings (cogwheel icon) on the right side and check all the shown folders.
  
  
  
  
  
  This Blueprint is almost ready. We just need to make some adjustments to the character's movement while it will have the tracker attached.
  This blueprint also helps with the implementaton of Rokoko Studio's Command API inside Unreal Engine. The ready part in this Blueprint is described in the Command API article under the Unreal Engine process category.
  
  
• From the Get Actor Transfrom node, drag the output and add a Break Transform Node
• From the Location output node, drag and add a Break Vector 3f node
• From that node, drag the X outpout and add a Multiply node, and type -1 in the field
• From the Mulitply node, drag the output and add a Make Vector node
• Back to the Break Vector 3f, connect the Y and the Z output to the Y and Z inputs of the Make Vector node respectively
• From the Make Vector node, drag the output (Return Value) and add a Make Transform node. It should connect to the Location input
• Back to the Break Transform node, drag Rotation and Scale outputs and connect them to the Make Transform node's Rotation and Scale inputs respectively
• Then connect the Make Transform node's output (Return Value) to the Make RokokoCommandAPI_TrackerInput node's Transform input

At this point, we are done with the connection between Rokoko Studio and the Unreal Engine.

Pair the Vive Trackers with Unreal Engine

Now that we have set up the trackers and the communication between Rokoko Studio and Unreal Engine, we just need to pair the tracker in Unreal Engine. We will do this by pairing the tracker with a Cube.

• Close this Position Correction Blueprint Blueprint
• Drag and drop the Position Correction Blueprint from the Content Drawer into the scene
• Select it inside the scene and go to the Details panel on the right side
• Change the Mobility to Moveable
  
  
  
  
• Click the PositionCorrection_Blueprint (self) and click the +Add button on the right
• Add a Live Link Controller
  
  
  
  
• Go to Window > Virtual Production > Live Link
  
  
  
  
• In the window that popped up, click the +Source button on the top left
• Hover over the LiveLinkXR Source and click Add
  
  
  
• If everything is correct, you will see the following with the Status: Receiving
  
  If you do not see this /camera input or if the green light is yellow instead, make sure the Tracker is on and that is detected by the Base Station. You can simply move it around and it will be detected
  
  
  
  
• Now that the tracker is detected, let's attach it to the Position Correction Blueprint
• Go to that Blueprint and click the Live Link Controller component that we added
  
  
  
  
• After this, you will be able to move the Position Correction Blueprint Cube inside the scene by moving the tracker in real life
  
  

Live stream into Unreal Engine using the Vive Tracker

The next step is to acitvate the livestream from Rokoko Studio into Unreal Engine. This, combined with the tracker, will provide absolute positioning and tracking of the custom character in the scene given that the calibration of the tracker is correct

This Sample project has the livestream process already set up. What we need to do is:

• activate the Rokoko plugin (if we haven't done already) from inside Unreal Engine > Edit > Plugins > Rokoko Studio Live
• Activate livestream to Unreal from inside Rokoko Studio
  
  
  
• Go to Window > Virtual Production > Live Link
  
  
  
  
• In the window that popped up, click the +Source button on the top left
• Hover over the Rokoko Studio Source
• Type the Forward IP (127.0.0.1) and Port (14043) into the field and click Studio
  
  
  
  
• Last, but not least, make sure that your Actor profile name inside Rokoko Studio matches the one in the Unreal Engine
  
  
  
  
• After you hit Play in Unreal Engine, if everything is correct, you will be able to move around with your Smartsuit Pro according to the position of the Cube. You can hide the Cube and adjust the Actor's placement in scene to improve the performance