Livestream to Custom Characters in Unreal Engine 5

Important! For any workflow-related issues, please join our Discord server.

Introduction

You have imported your Custom Character into your Unreal Engine 5 project. Next, you want to livestream your Rokoko Studio data to that Character.

Useful: A test project for Unreal Engine 5.2 (can be changed to 5.3 or newer by right-clicking on the project) with a ready custom character (plus metahumans for bonus) can be found here.

You will still need to change the Actor Profile name and the port in the livelink if needed.

Requirements

• Install Rokoko Studio here
• A Custom Character. You can find our Mime character here and its Body and Face bonemap files attached at the end of this article.
• Unreal Engine 5
• Install the Rokoko Studio Plugin in Unreal Engine 5

Make sure your Custom Character is in a proper T-pose

1. Make sure that the character's hands and fingers are modeled as close to the Newton model as possible, to get the best possible retargeting of finger animation. The fingers should be straight and the thumb pointing 45 degrees away from the other fingers. 

   Proper finger placement
   

   
   
   

    

2. Studio Live supports up to a 4 finger joint setup: Metacarpal, Proximal, Medial, and Distal finger joints.

   Finger Joint setup
   

   

3. An example of how our Mime T-Pose looks can be found in the Dropdown below. You can download the character here

   Mime's T-Pose
   

   
   
   

       

Plugin

Launch a project. Go to Edit>Plugins and search for “Rokoko Studio Live'' and in that section make sure the “Enabled” checkbox is checked. Then, restart your project.

Setting up your Custom Character

• Click Import and Import your character.
  
  
  
  A pop-up window will appear. Expand the “Advanced” tab and make sure “Use T0 as Ref Pose” and “Import Morph Targets” checkboxes are checked. 
  
  
  
  
• Right-click on your Skeletal Mesh and click Create>Anim Blueprint. Rename it to something like “character_AnimBP”
  
  
  
  
• Open the “character_AnimBP” and in the Anim Graph, right-click in the empty space and create a “Rokoko Body Pose” and a “Rokoko Face Pose” node. Connect them as shown below. (The “Component to Local” node will appear automatically)

Note: You can skip any step regarding the “Rokoko Face Pose” if you want to livestream just the body/hands.

• Right-click on the Rokoko Actor Name input in the Rokoko Body Pose node and click "Promote to Variable". You can rename it something like “Rokoko Actor Name”.
  
  
  
  
• Connect the newly created variable to the Rokoko Actor Name inputs on the “Rokoko Body Pose” and “Rokoko Face Pose” nodes. Then, compile and save.

  Note: Leave the “Rokoko Actor Name” as “None”! Do not rename it to your Actor's  Name as it's on Studio!

• In the content browser, right-click, create a Blueprint class,
  
  
  
  
• Expand the “All Classes” tab.
  
  
• Search for "Newton", select it and then create a “NewtonAlphaBoneRemappingAsset" blueprint. Name it something like “character_bonemap”.
  
  
  
  
• Then do the same and search for "Rokoko", select it and then create a “RokokoFaceMapData” blueprint. Name it something like “character_facemap”.
  
  

• Open the “character_boneMap” blueprint and write the names of the corresponding joints in the hierarchy. If you are using the Mime character, the names are already filled.
  
  It should be different for each custom character! (Unless the bone names are the same)
  
  “Mime” bone map setup:

• Compile and save the blueprint. (If you use the same joint naming on more characters you can reuse this asset).
  
  
• Open the “character_faceMap” blueprint and write the names of the blendshape names you'd like to override(first, try not making any changes). (If you use the same blendshape names on more characters you can reuse this asset). 

  When done, compile and save.
  
  

• Back in the “character_AnimBP” graph, Right-click on the Retarget Asset input in the Rokoko Body Pose node and click "Promote to Variable". You can rename it something like “Retarget Asset".
  

• Compile and set its default value to the bone map you just created.

Remember to hit Apply at the lower right corner

• Create one more variable, by doing the same for the Rokoko Face Pose. Right-click on the Retarget Asset input in the Rokoko Face Pose node and click "Promote to Variable". You can rename it something like “Retarget Asset Face".
  
  
• Compile and set its default value to the face map you just created.

Remember to hit Apply at the lower right corner

Finally, your animation graph will look like this:

Note: Ignore the warnings!!!!

• In the content browser, right-click, create a Blueprint class,
  
  
  
  
• Right-click on the content browser and create an “Actor” blueprint class.
  
  
  
• Name it something like “character_BP”. Open it and on top left, click on Add Component and select Skeletal mesh.

• To this skeletal mesh, apply your character’s skeleton and the Animation Blueprint you just created.
  
  

Hit compile and save.

• Go to the event graph of the “character BP”
  
  
  
  
• Drag the skeletal mesh to the graph. From its pin, drag and create a “Get Anim Instance” node. From its pin, drag and create a “Cast to character_AnimBP” node. From “Cast to character_AnimBP” blue pin drag and create a “Set Rokoko Actor Name”. Set the name as it is on your Actor Profile in Rokoko Studio.
  
  Connect the rest pins, Compile, and Save the Blueprint.
  
  (right click on the image below and click "open image in new tab" to zoom)
  
  

• Drag the Actor Blueprint “character_BP” into the scene.

Livestream settings

• If you are using UE 5.1, you will need to add a Rokoko Receiver. If you are using UE 5.2 or later, omit the Rokoko Receiver step and activate the plugin from Rokoko Studio next.
  
  
• On the top-left of Unreal Editor click the Add button, type "rokoko" right away", and search for "Rokoko Receiver” and drag it into the scene. Set the Rokoko Port Number as 14043, in the Details Panel.

• Open Rokoko Studio, open a scene and make sure your Smartsuit Pro is connected and paired with your Profile. You can also live stream pre-recorded animations.
  
  
• Click on Livestreaming.
  
  
  
  
• Navigate to Unreal Engine and click on the cogwheel. Make sure the port number is 14043 and enable the Livestream.
  
  
  
  
• Activate the Plugin.
  
  
  

• If you are using UE 5.2 you will need to type the IP and port in Window > Virtual Production > Livelink. The IP and port needed are found in the Rokoko Studio plugin window. In our case, it's 127.0.0.1:14043, then click "Studio" in UE as shown by the arrow.
  
  If Unreal Engine crashes, then most likely you:
  1. typed the wrong IP or port
  2. there is a node that is not connected correctly somewhere. For example, you may have forgotten to connect the rokoko actor variable in the Rokoko Face Pose as well.
  
  This basically replaces the need for the Rokoko Receiver. If you have already added it, there is nothing wrong and you can continue.
  
  
  
  

Finally, hit play and start Livestreaming with your Custom Character!