Custom Streaming

Introduction

The Custom integration provides you full control of the data stream. You can configure the data stream exactly as you wish by setting the protocol and data format as seen fit.

The .txt files of the JSON fomats can also be found attached below.

Getting started

• Open Rokoko Studio Live.
• Go to the Livestreaming panel, navigate to Custom Streaming, and click the "cogwheel"  settings.

• There is already a receiver there. Let's configure it. You can add more receivers by clicking the "plus"  button.
• We can change the data format to our preference depending on what we want to achieve. To set the protocol and data format, click the arrow down button to expand the receiver configuration. 
  
  Each individual receiver can be enabled and disabled by clicking the "Include connection" tick button. You can delete a receiver by clicking the "trash" icon on the right.
   

 

• In order to enable a data stream, at least one receiver has to be enabled. Enable the data stream by clicking the "Include connection" tick button. Then exit the settings and click the "Activate" button.
   

• The green outline in the viewport indicates that the data stream is enabled. Click the "Activate" button again to disable it.

Protocols available

UDP

In computer networking, the User Datagram Protocol (UDP) is one of the core members of the Internet protocol suite. The protocol was designed by David P. Reed in 1980 and formally defined in RFC 768. With UDP, computer applications can send messages, in this case referred to as datagrams, to other hosts on an Internet Protocol (IP) network. Prior communications are not required in order to set up communication channels or data paths.

UDP uses a simple connectionless communication model with a minimum of protocol mechanisms. UDP provides checksums for data integrity, and port numbers for addressing different functions at the source and destination of the datagram. It has no handshaking dialogues, and thus exposes the user's program to any unreliability of the underlying network; there is no guarantee of delivery, ordering, or duplicate protection. If error-correction facilities are needed at the network interface level, an application may use Transmission Control Protocol (TCP) or Stream Control Transmission Protocol (SCTP) which are designed for this purpose.

Source: https://en.wikipedia.org/wiki/User_Datagram_Protocol

Data formats available

Each data format supports specific streaming capabilities from Studio Live defined in the table below:

JSON v3

{
      version: 3,
      fps: 60,
      scene: {
            timestamp: 0,
      actors: [
      {
            name: "Dimitris", //arbitary size
            color: {r, g, b}, // bytes
            meta: {
                  hasGloves: true, //Indicates if fingers are controlled
                  hasLeftGlove: true,
                  hasRightGlove: true,
                  hasBody: true, //Indicates if body is controlled
                  hasFace: true //Indicates if face is controlled
            }
            // physical proportions of the actor
            dimensions: {
                        totalHeight: 180,
                        hipHeight: 100,
                        // ... more information about the body that might be interesting 
                        // for retargetting to custom characters
            }
            body: { // optional section
                  hip: {
                        position {x: 0, y: 0, z: 0} // Y-up, Z-forward
                        rotation {x: 0, y: 0, z: 0, w: 1} // quaternion
                  },
                  spine: {...}
            },
            face: { // optional section
                  leftEyeBlink: 0, // 0-100
                  ...
            }
           },
            {...}
      ],
      props: [
      {
            name: "Box1",
            color: {r, g, b} // bytes
            type: 0/1/2 // 0: box, 1: stick, 2: camera
            position: {x: 0, y: 0, z: 0}, // Y-up, Z-forward
            rotation: {x: 0, y: 0, z: 0, w: 1}, // quaternion
       },
      ]
    }
}

JSON v4

{
  version: "3,0",              // see note 1 - v4 frames still report "3,0"
  fps: 60,
  scene: {
    timestamp: 0,              // seconds, floating point
    actors: [],                // always emitted; empty when streaming JSON v4
    newtons: [
      {
        name: "Newton 01",
        color: [r, g, b],      // integer array, 0-255
        meta: {
          hasHands: true,
          hasLeftHand: true,
          hasRightHand: true,
          hasBody: true,
          hasFace: false
        },
        dimensions: {
          totalHeight: 1.75,   // metres
          hipHeight: 0.94      // metres
        },
        joints: [              // flat list, world space, order not guaranteed
          {
            name: "Hips",
            parent: -1,        // index into this array; -1 = no parent
            position: {x: 0, y: 0.94, z: 0},
            rotation: {x: 0, y: 0, z: 0, w: 1}
          },
          {
            name: "Spine1",
            parent: 0,
            position: {x: 0, y: 1.04, z: 0},
            rotation: {x: 0, y: 0, z: 0, w: 1}
          },
          {...}                // see note 3 for the full Newton joint list
        ],
        face: {                // omitted entirely when there are no blendshapes
          leftEyeBlink: 0,
          ...
        }
      },
      {...}
    ],
    props: [],                 // always emitted; currently always empty
    characters: []             // reserved; empty unless a character is streamed
  }
}

NOTES

1. Version field
   JSON v4 frames carry version "3,0", the same value JSON v3 frames carry.
   The field cannot be used to tell the two formats apart. Detect the format
   from the payload instead: a v3 frame fills scene.actors and leaves
   scene.newtons empty; a v4 frame fills scene.newtons and leaves
   scene.actors empty. Both keys are always present.

2. Coordinate system and units
   Left-handed, Y up, Z forward - the same convention as JSON v3.
   Positions are in metres. Rotations are quaternions, ordered x, y, z, w.
   Every joint transform is in WORLD space, even though the joint carries a
   parent index. The parent index describes the skeleton hierarchy; it does
   not mean the transform is parent-relative. Do not compose parent
   transforms into children - you will double-transform the pose.

3. Joint list
   The joints array is the Newton skeleton, enumerated from the streamed
   entity's root downwards. The shipped Newton rig has 76 bones: two
   container nodes (Newton, Root), a four-segment spine (Spine1..Spine4),
   HeadTip, five joints per finger (four for the thumbs, which have no
   Medial), and toe tips.

   Newton, Root, Hips, Spine1, Spine2, Spine3, Spine4, Neck, Head, HeadTip,
   LeftShoulder, LeftArm, LeftForeArm, LeftHand,
   RightShoulder, RightArm, RightForeArm, RightHand,
   LeftThigh, LeftShin, LeftFoot, LeftToe, LeftToeTip,
   RightThigh, RightShin, RightFoot, RightToe, RightToeTip,
   and, for each hand, per finger N = 1..5
   (1 = thumb, 2 = index, 3 = middle, 4 = ring, 5 = little):
     {Left|Right}FingerN{Metacarpal, Proximal, Medial, Distal, Tip}
   The thumb (Finger1) has no Medial joint, so it contributes four names
   instead of five.

   Match joints by name and build the hierarchy from the parent index. Do not
   hardcode array positions - the enumeration starts at the streamed entity's
   root and the ordering is not part of the contract.

4. Compression
   Both formats are offered plain and LZ4-framed: "JSON v3", "JSON v3 (LZ4)",
   "JSON v4", "JSON v4 (LZ4)". An LZ4 payload starts with the LZ4 frame magic
   04 22 4D 18. The JSON itself is ASCII encoded, one complete frame per UDP
   datagram, with no length prefix or header of any kind.

5. Differences from JSON v3
   - Joints arrive as a flat array with explicit parent indices instead of a
     fixed-key body object.
   - Joint names are the native Newton skeleton names (Hips, Spine1,
     LeftFinger2Proximal) instead of the v3 humanoid names (hip, spine,
     leftIndexProximal).
   - Spine2 and Spine3, HeadTip, the finger Tip joints and the toe tips have
     no JSON v3 equivalent.
   - meta uses hasHands / hasLeftHand / hasRightHand where v3 uses
     hasGloves / hasLeftGlove / hasRightGlove.