Streaming your Custom Character to Unreal Engine (Body and Face capture)

For any workflow related issues, please join our Discord server.

 

Requirements

  • Rokoko Studio 1.19/1.20
  • Unreal Engine 4.26/4.27/5
  • Rokoko Studio Subscription: Rokoko Studio Plus, Pro or Enterprise.
  • In order to livestream to Unreal Engine, you need to have installed the Unreal Engine plugin. Instructions for doing this can be found here. 

Make sure your model is in a proper T-Pose

Make sure that the character's hands and fingers are modeled as close to the Newton model as possible, to get the best possible retargeting of finger animation. The fingers should be straight and the thumb pointing 45 degrees away from the other fingers. 

mceclip0.png    mceclip1.png

Studio Live supports up to a 4 finger joint setup: Metacarpal, Proximal, Medial and Distal finger joint.

mceclip2.png

Example of Mime T-Pose (download the character here):

mceclip3.png

 

mceclip0.png

mceclip4.png

Plugin Settings

Launch a project. Go to Edit>Plugins and search for “Rokoko Studio Live'' and in that section make sure the “Enabled” checkbox is checked. Then, restart your project.

mceclip0.png

Setting up your Custom Character

  • Import your character. A pop-up window will appear. Expand the “Mesh” tab and make sure “Use T0 as Ref Pose” and “Import Morph Targets” checkboxes are checked. 

mceclip0.png

         (In this documentation, we imported our Mime character. Download it here.)

  • Right click on your Skeletal Mesh and click Create>Anim Blueprint. Rename it to something like “character_AnimBP”
  • Open the “character_AnimBP” and in the anim graph create a “Rokoko Body Pose” and a “Rokoko Face Pose” node. Connect them as shown below. (The “Component to Local” node will appear automatically)

Note: You can skip any step regarding the “Rokoko Face Pose”, if you want to livestream just the body/hands.

mceclip1.png

  • Create a variable, set it as “Name” and rename it something like “Rokoko Actor Name”.
  • Drag the “Rokoko Actor Name” variable into the graph, select “get Rokoko Actor name” and connect it to “Rokoko Body Pose” and “Rokoko Face Pose” inputs. Then, compile and save.

    Note: Leave the “Rokoko Actor Name” as “None”! Do not rename it to your Actor's  Name as it 's on Studio!

mceclip0.png

  • In the content browser, right-click, create a Blueprint class, expand the “All Classes” tab and create a “RokokoBodyMapData” and “RokokoFaceMapData” blueprint. Name them something like “character_boneMap” and “character_faceMap”.

mceclip2.png

  • Open the “character_boneMap” blueprint and write the names of the corresponding joints in the hierarchy. 

It should be different on each custom character! (Unless the bone names are the same)

“Mime” bone map setup:

mceclip4.png

mceclip5.png  mceclip6.png

  • Compile and save the blueprint.(If you use the same joint naming on more characters you can reuse this asset).
  • Open the “character_faceMap” blueprint and write the names of the blendshape names you'd like to override(first, try not making any changes). (If you use the same blendshape names on more characters you can reuse this asset). 

    When done, compile and save.

  • Back in the “character_AnimBP” graph, create a variable, set it at “Rokoko Body Map Data” class reference and name it something like “BodyMapRetargetAsset”.

mceclip7.png

Compile and set its default value to the bone map you just created.

mceclip8.png

  • Drag the “BodyMapRetargetAsset” variable to the graph, select “get BodyMapRetarget Asset” and connect it to the “Retarget Asset” input of “Rokoko Body Pose” node.

mceclip9.png

  • Create one more variable, set it at “Rokoko Face Map Data” class reference and name it something like “FaceMapRetargetAsset”.

mceclip10.png

Compile and set its default value to the face map you just created.

mceclip11.png

  • Drag the “FaceMapRetargetAsset” variable to the graph, select “get FaceMapRetarget Asset” and connect it to the “Retarget Asset” input of “Rokoko Face Pose” node.

Finally, your animation graph will look like this:

mceclip0.png

Note: Ignore the warnings!!!!

  • Right-click on the content browser and create an “Actor” blueprint class. Name it something like “character_BP”. Open it and on-top left, click on Add Component and select skeletal mesh.

mceclip14.png

  • To this skeletal mesh, apply your character’s skeleton and the Animation Blueprint you just created. 

mceclip15.png

Hit compile and save.

  • On the event graph of the “character_BP” drag the skeletal mesh to the graph. From its pin, drag and create a “Get Anim Instance” node. From its pin,drag and create a “Cast to character_AnimBP” node. From “Cast to character_AnimBP” blue pin drag and create a “Set Rokoko Actor Name”. Set the name as it is on your Actor Profile in Rokoko Studio. Connect the rest pins, Compile and Save the Blueprint.

mceclip17.png

  • Drag the Actor Blueprint “character_BP” into the scene.

Livestream settings

  • In Unreal, under Place Actors(top-left of Unreal Editor) search for “Rokoko Receiver” and drag it into the scene. Set the Rokoko Port Number as 14045, in the Details Panel.

mceclip18.png mceclip19.png

 

  • Open Rokoko Studio, open a scene and make sure your Smartsuit is connected and paired with your Profile. Click on Start Livestream, navigate to Unreal Engine and click on the cogwheel. Make sure the port number is 14045 and enable the Livestream.

mceclip20.png  mceclip21.png

mceclip22.png   mceclip23.png

  • Back in Unreal, go to Window>Live Link and then Source>Rokoko Studio Source>Studio.

mceclip24.png

mceclip25.png

 

Finally, hit play and start Livestreaming with your Custom Character!

 

 

Articles in this section

Knowledge base