├── LICENSE ├── README.md ├── doc ├── Protocol.md ├── fram-structure.png ├── mcp-format.odp ├── overview.png ├── seq.png ├── serialization-format.png ├── skdf-structure.png └── tran-structure.png ├── mcp_receiver ├── __init__.py ├── __main__.py ├── dummyreceiver.py ├── dumper.py ├── receiver.py ├── runner.py └── vmcsender.py ├── requirements.txt └── test ├── __init__.py └── test.py /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2022 seagetch 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # mcp-receiver 2 | Open source implementation of receiver plugin for mocopi motion tracking system. 3 | 4 | ## Install 5 | install dependency with pip command. 6 | ``` 7 | pip install -r requirements.txt 8 | ``` 9 | 10 | ## Usage 11 | Execute test application by following command. 12 | ``` 13 | python3 -m mcp_receiver 14 | ``` 15 | This command listens on port 12351, receiving packets from mocopi mobile app (and BVHSender), and then send VMC Protocol packets to localhost:39540. -------------------------------------------------------------------------------- /doc/Protocol.md: -------------------------------------------------------------------------------- 1 | # Protocol Specification 2 | **This document is UNOFFICIAL, and nothing is related to official activities around Sony's mocopi product.** 3 | 4 | **This document ensures NOTHING about the accuracy of the technical details. This is just analyzed information observed from UDP packets between BVHSender and Receiver Plugin. This document is published AS-IS and provides NO WARRANTY.** 5 | 6 | This document describes the estimated protocol between mocopi mobile application and Receiver Plugin running on the PC. 7 | 8 | - [Protocol Specification](#protocol-specification) 9 | - [Overview](#overview) 10 | - [Communication between Sender and Receiver](#communication-between-sender-and-receiver) 11 | - [Serialization Format](#serialization-format) 12 | - [Basic Field Structure](#basic-field-structure) 13 | - [Nested Field Structure](#nested-field-structure) 14 | - [Data Structure](#data-structure) 15 | - [`skdf` Packet Structure](#skdf-packet-structure) 16 | - [`head` field](#head-field) 17 | - [`sndf` field](#sndf-field) 18 | - [`skdf` field](#skdf-field) 19 | - [`bons` field](#bons-field) 20 | - [`bndt` field](#bndt-field) 21 | - [`fram` Packet Structure](#fram-packet-structure) 22 | - [`head` field](#head-field-1) 23 | - [`sndf` field](#sndf-field-1) 24 | - [`fram` field](#fram-field) 25 | - [`btrs` field](#btrs-field) 26 | - [`btdt` field](#btdt-field) 27 | - [Bone Transformation Definition in `tran` field](#bone-transformation-definition-in-tran-field) 28 | 29 | 30 | ## Overview 31 | Figure below shows the overview of the communication between mobile app and Computer. 32 | Mocopi device sends sensor data to mobile app (maybe via bluetooth protocol.) 33 | Mobile app transform raw sensors into 27 bone transformation in their application. 34 | Then mobile app sends transformed data to PC, and software (Unity, Motion Builder etc.) processes the passed transformation data. 35 | It seems this protocol is one-way communication. Packets are sent only from mobile app to computer only. 36 | 37 | Another program named "BVHSender" is the emulation of the mocopi device and mobile app data. 38 | it sends the transformation data written in BVH format to the computer. 39 | 40 | If user want to send tracking data of multiple player, it should send tracking data to another UDP socket of different port. 41 | 42 | ![overview](overview.png) 43 | 44 | 45 | ## Communication between Sender and Receiver 46 | Mobile app continues to send updated motion to specified port. No interactive communication is required. Just one-way stream from mobile app to computer. 47 | 48 | ![sequence](seq.png) 49 | 50 | Mobile app sends `skdf` packet at the beginning of the connection. 51 | It contains default definition of skeleton parameters of bones with T-pose. 52 | Mobile app seems to send `skdf` packet periodically in every few seconds. 53 | 54 | Then it continues sending sequence of `fram` packet when sensor get new value (need confirmation.) 55 | 56 | ## Serialization Format 57 | Data sent from mobile app is serialized in special serialization format. 58 | Data format is a set of field, which consists of three part, length, field name, and value data. 59 | Fields are iterated until the end of the packets. 60 | Values are serialized in little endian. 61 | 62 | ![serialization](serialization-format.png) 63 | ### Basic Field Structure 64 | - `length` specifies the size of value data in bytes. 65 | - `field name` is 4 bytes ASCII data to represent the type of data field. 66 | You can check list of fields in next section. 67 | - `value` contains actual data for field. Contents depends on the field name. 68 | 69 | ### Nested Field Structure 70 | Some field have complex structure like in C and other languages. In that kind of complex structure, all data for sub-field is contained in `value` data of parent field. 71 | 72 | If `field1` contains sub-field `field2` and `field3`, length, filed name, and value of `field2` and `field3` are packed into `value` data of `field1`. 73 | 74 | ## Data Structure 75 | Mobile app sends packets which contains following structure. 76 | 77 | ### `skdf` Packet Structure 78 | `skdf` packet is sent at the beginning of the sequence. 79 | Packet contains `head`, `sndf` and `skdf` fields, and contains several sub-fields inside. 80 | 81 | ![fields](skdf-structure.png) 82 | 83 | #### `head` field 84 | `head` field contains information about packet format and version. 85 | - `ftyp` contains the string which represent the format. Currently, "sony motion format" is used. 86 | - `vrsn` specifies the version number in 1 byte integer. Currently 1 is used. 87 | 88 | #### `sndf` field 89 | `sndf` field contains the IP addresses and port number information. 90 | - `ipad` represents the IP address in 8 bytes. Maybe containing sender and receiver. 91 | - `rcvp` represents the port number of receiver's side. "12351" is used by default. 92 | 93 | #### `skdf` field 94 | `skdf` field contains detailed information of the motion. 95 | - `bons` contains set of bone definition of the avatar. 96 | 97 | #### `bons` field 98 | `bons` field contains set of transformation definitions (`bndt` section.) Currently this section contains 27 transformation defitnion inside. 99 | 100 | #### `bndt` field 101 | `bndt` field contains every individual motion definition. 102 | - `bnid` specifies Bone ID. Currently 0 to 26 is used. 103 | For more details of Bone ID, you can consult [Technical Specification](https://www.sony.net/Products/mocopi-dev/jp/documents/Home/TechSpec.html) of mocopi developer's site. 104 | - `pbid` Parent bone ID. ? 105 | - `tran` specifies the transformation parameter values. It seems to contain 7 floating values. 106 | 107 | ### `fram` Packet Structure 108 | `fram` packet is sent continuously after first `skdf` packet is sent. 109 | Packet contains `head`, `sndf` and `fram` fields, and contains several sub-fields inside. 110 | 111 | ![fields](fram-structure.png) 112 | #### `head` field 113 | `head` field contains information about packet format and version. 114 | - `ftyp` contains the string which represent the format. Currently, "sony motion format" is used. 115 | - `vrsn` specifies the version number in 1 byte integer. Currently 1 is used. 116 | 117 | #### `sndf` field 118 | `sndf` field contains the IP addresses and port number information. 119 | - `ipad` represents the IP address in 8 bytes. Maybe containing sender and receiver. 120 | - `rcvp` represents the port number of receiver's side. "12351" is used by default. 121 | 122 | #### `fram` field 123 | `fram` field contains detailed information of the motion. 124 | - `fnum` specifies the frame number in 4 bytes integer. 125 | - `time` speicifes the time stamp of the motion in 4 bytes integer. 126 | - `btrs` contains set of transformations of the avatar. 127 | 128 | #### `btrs` field 129 | `btrs` field contains set of transformation definitions (`btdt` section.) Currently this section contains 27 transformation defitnion inside. 130 | 131 | #### `btdt` field 132 | `btdt` field contains every individual motion definition. 133 | - `bnid` specifies Bone ID. Currently 0 to 26 is used. 134 | For more details of Bone ID, you can consult [Technical Specification](https://www.sony.net/Products/mocopi-dev/jp/documents/Home/TechSpec.html) of mocopi developer's site. 135 | - `tran` specifies the transformation parameter values. It seems to contain 7 floating values. 136 | 137 | ### Bone Transformation Definition in `tran` field 138 | `tran` field contains 7 floating values. By examining the values sending VMC Performer, we figured out the meaning of the values: 139 | 140 | ![tran](tran-structure.png) 141 | 142 | - `rotation` represents the local rotation value of the bone. That means value of `rotation` is relative to angles of parent bone. 143 | `rotation` is expressed in quaternion angle. It contains four elements (x, y, z, w) 144 | - `position` represents the local position value of the bone. That means value of `position` is relative to position of parent bone. 145 | -------------------------------------------------------------------------------- /doc/fram-structure.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/seagetch/mcp-receiver/e5219b9c17666fb21248c9b5c5c0c83eb31effad/doc/fram-structure.png -------------------------------------------------------------------------------- /doc/mcp-format.odp: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/seagetch/mcp-receiver/e5219b9c17666fb21248c9b5c5c0c83eb31effad/doc/mcp-format.odp -------------------------------------------------------------------------------- /doc/overview.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/seagetch/mcp-receiver/e5219b9c17666fb21248c9b5c5c0c83eb31effad/doc/overview.png -------------------------------------------------------------------------------- /doc/seq.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/seagetch/mcp-receiver/e5219b9c17666fb21248c9b5c5c0c83eb31effad/doc/seq.png -------------------------------------------------------------------------------- /doc/serialization-format.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/seagetch/mcp-receiver/e5219b9c17666fb21248c9b5c5c0c83eb31effad/doc/serialization-format.png -------------------------------------------------------------------------------- /doc/skdf-structure.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/seagetch/mcp-receiver/e5219b9c17666fb21248c9b5c5c0c83eb31effad/doc/skdf-structure.png -------------------------------------------------------------------------------- /doc/tran-structure.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/seagetch/mcp-receiver/e5219b9c17666fb21248c9b5c5c0c83eb31effad/doc/tran-structure.png -------------------------------------------------------------------------------- /mcp_receiver/__init__.py: -------------------------------------------------------------------------------- 1 | from .receiver import Receiver 2 | from .vmcsender import VMCSender -------------------------------------------------------------------------------- /mcp_receiver/__main__.py: -------------------------------------------------------------------------------- 1 | from .receiver import Receiver 2 | from .vmcsender import VMCSender 3 | import queue 4 | 5 | q = queue.Queue() 6 | recv = Receiver() 7 | send = VMCSender() 8 | recv.run(q) 9 | send.run(q) -------------------------------------------------------------------------------- /mcp_receiver/dummyreceiver.py: -------------------------------------------------------------------------------- 1 | import time 2 | from mcp_receiver.runner import Runner 3 | 4 | class DummyReceiver(Runner): 5 | def __init__(self): 6 | pass 7 | 8 | def loop(self): 9 | 10 | # Main loop 11 | id = 0 12 | while True: 13 | data = { 14 | "head": { "ftyp": "dummy", "vrsn": 0}, 15 | "sndf": { "ipad": (0, 1), "rcvp": 12351}, 16 | "fram": { "fnum": id, "time": int(1000*time.time()), 17 | "btrs": [{ 18 | "bnid": i, 19 | "tran": (0, 0, 0, 0, 0, 0, 0)} for i in range(0, 27)]}} 20 | self.queue.put(data) 21 | time.sleep(1) 22 | -------------------------------------------------------------------------------- /mcp_receiver/dumper.py: -------------------------------------------------------------------------------- 1 | from mcp_receiver.runner import Runner 2 | 3 | 4 | class ScreenDumper(Runner): 5 | def __init__(self): 6 | pass 7 | 8 | def loop(self): 9 | 10 | # Main loop 11 | while True: 12 | data = self.queue.get() 13 | print(data) -------------------------------------------------------------------------------- /mcp_receiver/receiver.py: -------------------------------------------------------------------------------- 1 | import socket 2 | import struct 3 | from mcp_receiver.runner import Runner 4 | 5 | def is_field(name): 6 | return name.isalpha() 7 | 8 | def _deserialize(data, index, length, is_list = False): 9 | result = [] if is_list else {} 10 | end_pos = index + length 11 | while end_pos - index > 8 and is_field(data[index+4:index+8]): 12 | size = struct.unpack("@i", data[index: index+4])[0] 13 | index += 4 14 | field = data[index:index+4] 15 | index += 4 16 | value, index2 = _deserialize(data, index, size, field in [b"btrs", b"bons"]) 17 | index = index2 18 | if is_list: 19 | result.append(value) 20 | else: 21 | result[field.decode()] = value 22 | if len(result) == 0: 23 | body = data[index:index+length] 24 | return body, index + len(body) 25 | else: 26 | return result, index 27 | 28 | def _process_packet(message): 29 | data = _deserialize(message, 0, len(message), False)[0] 30 | data["head"]["ftyp"] = data["head"]["ftyp"].decode() 31 | data["head"]["vrsn"] = ord(data["head"]["vrsn"]) 32 | data["sndf"]["ipad"] = struct.unpack("@BBBBBBBB", data["sndf"]["ipad"]) 33 | data["sndf"]["rcvp"] = struct.unpack("@H", data["sndf"]["rcvp"])[0] 34 | if "skdf" in data: 35 | for item in data["skdf"]["bons"]: 36 | item["bnid"] = struct.unpack("@H", item["bnid"])[0] 37 | item["pbid"] = struct.unpack("@H", item["pbid"])[0] 38 | item["tran"] = struct.unpack("@fffffff", item["tran"]) 39 | elif "fram" in data: 40 | data["fram"]["fnum"] = struct.unpack("@I", data["fram"]["fnum"])[0] 41 | data["fram"]["time"] = struct.unpack("@I", data["fram"]["time"])[0] 42 | for item in data["fram"]["btrs"]: 43 | item["bnid"] = struct.unpack("@H", item["bnid"])[0] 44 | item["tran"] = struct.unpack("@fffffff", item["tran"]) 45 | return data 46 | 47 | 48 | class Receiver(Runner): 49 | def __init__(self, addr = "localhost", port = 12351): 50 | self.addr = addr 51 | self.port = port 52 | 53 | def loop(self): 54 | self.socket = socket.socket(family=socket.AF_INET, type=socket.SOCK_DGRAM) 55 | self.socket.bind((self.addr, self.port)) 56 | while True: 57 | try: 58 | message, client_addr = self.socket.recvfrom(2048) 59 | data = _process_packet(message) 60 | self.queue.put(data) 61 | except KeyError as e: 62 | print(e) 63 | -------------------------------------------------------------------------------- /mcp_receiver/runner.py: -------------------------------------------------------------------------------- 1 | import threading 2 | 3 | class Runner: 4 | def run(self, queue): 5 | self.queue = queue 6 | self.thread = threading.Thread(target=self.loop, args=()) 7 | self.thread.start() -------------------------------------------------------------------------------- /mcp_receiver/vmcsender.py: -------------------------------------------------------------------------------- 1 | from pythonosc import udp_client, osc_bundle_builder, osc_message_builder 2 | from mcp_receiver.runner import Runner 3 | 4 | bone_map = [ 5 | "Hips", #0 6 | "Spine", #1 7 | None, #2 8 | "Chest", #3 9 | None, #4 10 | "UpperChest", #5 11 | None, #6 12 | "-", #7 13 | "-", #8 14 | "Neck", #9 15 | "Head", #10 16 | "RightShoulder", #11 17 | "RightUpperArm", #12 18 | "RightLowerArm", #13 19 | "RightHand", #14 20 | "LeftShoulder", #15 21 | "LeftUpperArm", #16 22 | "LeftLowerArm", #17 23 | "LeftHand", #18 24 | "RightUpperLeg", #19 25 | "RightLowerLeg", #20 26 | "RightFoot", #21 27 | "RightToes", #22 28 | "LeftUpperLeg", #23 29 | "LeftLowerLeg", #24 30 | "LeftFoot", #25 31 | "LeftToes" #26 32 | ] 33 | 34 | class VMCSender(Runner): 35 | def __init__(self, host = "localhost", port = 39540): 36 | self.host = host 37 | self.port = port 38 | 39 | def loop(self): 40 | client = udp_client.UDPClient(self.host, self.port) 41 | # Do some initialization here 42 | pass 43 | 44 | # Main loop 45 | while True: 46 | try: 47 | data = self.queue.get() 48 | if "skdf" in data: 49 | msg_builder = osc_message_builder.OscMessageBuilder("/VMC/Ext/Root/Pos") 50 | tran = data["skdf"]["bons"][0]["tran"] 51 | msg_builder.add_arg("root") 52 | msg_builder.add_arg(tran[4]) 53 | msg_builder.add_arg(tran[5]) 54 | msg_builder.add_arg(tran[6]) 55 | msg_builder.add_arg(tran[0]) 56 | msg_builder.add_arg(tran[1]) 57 | msg_builder.add_arg(tran[2]) 58 | msg_builder.add_arg(tran[3]) 59 | client.send(msg_builder.build()) 60 | elif "fram" in data: 61 | bdl_builder = osc_bundle_builder.OscBundleBuilder(osc_bundle_builder.IMMEDIATELY) 62 | skipped_pos = None 63 | skipped_rot = None 64 | for btdt in data["fram"]["btrs"]: 65 | if bone_map[btdt["bnid"]] == "-": 66 | pass 67 | if bone_map[btdt["bnid"]]: 68 | msg_builder = osc_message_builder.OscMessageBuilder("/VMC/Ext/Bone/Pos") 69 | tran = btdt["tran"] 70 | if skipped_pos is not None and skipped_rot is not None: 71 | tran = [ 72 | tran[0] * skipped_rot[0], 73 | tran[1] * skipped_rot[1], 74 | tran[2] * skipped_rot[2], 75 | tran[3] * skipped_rot[3], 76 | tran[4] + skipped_pos[0], 77 | tran[5] + skipped_pos[1], 78 | tran[6] + skipped_pos[2] 79 | ] 80 | skipped_pos = None 81 | skipped_rot = None 82 | msg_builder.add_arg(bone_map[btdt["bnid"]]) 83 | msg_builder.add_arg(tran[4]) 84 | msg_builder.add_arg(tran[5]) 85 | msg_builder.add_arg(tran[6]) 86 | msg_builder.add_arg(tran[0]) 87 | msg_builder.add_arg(tran[1]) 88 | msg_builder.add_arg(tran[2]) 89 | msg_builder.add_arg(tran[3]) 90 | bdl_builder.add_content(msg_builder.build()) 91 | elif skipped_pos is None: 92 | skipped_pos = [tran[4], tran[5], tran[6]] 93 | skipped_rot = [tran[0], tran[1], tran[2], tran[3]] 94 | else: 95 | skipped_pos = [skipped_pos[0] + tran[4], skipped_pos[1] + tran[5], skipped_pos[2] + tran[6]] 96 | skipped_rot = [skipped_rot[0] * tran[0], skipped_rot[1] * tran[1], skipped_rot[2] * tran[2], skipped_rot[3] * tran[3]] 97 | client.send(bdl_builder.build()) 98 | except KeyError as e: 99 | print(e) -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | python-osc -------------------------------------------------------------------------------- /test/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/seagetch/mcp-receiver/e5219b9c17666fb21248c9b5c5c0c83eb31effad/test/__init__.py -------------------------------------------------------------------------------- /test/test.py: -------------------------------------------------------------------------------- 1 | from mcp_receiver.receiver import Receiver 2 | from mcp_receiver.vmcsender import VMCSender 3 | from mcp_receiver.dummyreceiver import DummyReceiver 4 | #from mcp_receiver.dumper import ScreenDumper 5 | import socket 6 | import queue 7 | import glob 8 | import os.path 9 | 10 | from mcp_receiver.runner import Runner 11 | 12 | class DumpServer(Runner): 13 | def __init__(self, addr = "localhost", port=39540): 14 | self.addr = addr 15 | self.port = port 16 | 17 | def loop(self): 18 | self.socket = socket.socket(family=socket.AF_INET, type=socket.SOCK_DGRAM) 19 | self.socket.bind((self.addr, self.port)) 20 | while True: 21 | try: 22 | message, client_addr = self.socket.recvfrom(2048) 23 | print("Got [%d]:"%len(message), message) 24 | except KeyError as e: 25 | print(e) 26 | 27 | class DummyBVHSender(Runner): 28 | def __init__(self, addr = "localhost", port=12351): 29 | self.addr = addr 30 | self.port = port 31 | 32 | def loop(self): 33 | self.socket = socket.socket(family=socket.AF_INET, type=socket.SOCK_DGRAM) 34 | payloads = [] 35 | for fn in sorted(glob.glob("test/stuff/*")): 36 | with open(fn, "rb") as f: 37 | payloads.append(f.read()) 38 | self.socket.sendto(payloads[0], (self.addr, self.port)) 39 | index = 1 40 | while True: 41 | try: 42 | if index == len(payloads): 43 | index = 1 44 | self.socket.sendto(payloads[index], (self.addr, self.port)) 45 | index += 1 46 | except KeyError as e: 47 | print(e) 48 | 49 | 50 | q = queue.Queue() 51 | send = VMCSender(host="127.17.0.1", port=39540) 52 | #recv = DummyReceiver() 53 | #send = ScreenDumper() 54 | #watcher = DumpServer(port=39539) 55 | if os.path.exists("test/stuff"): 56 | recv = Receiver() 57 | bvh = DummyBVHSender() 58 | else: 59 | recv = DummyReceiver() 60 | bvh = None 61 | 62 | #watcher.run(q) 63 | send.run(q) 64 | recv.run(q) 65 | if bvh: 66 | bvh.run(q) --------------------------------------------------------------------------------