├── LICENSE ├── README.md └── livesplat_realsense.py /LICENSE: -------------------------------------------------------------------------------- 1 | Software License Agreement – Academic, Personal, and Evaluation Use 2 | 3 | Copyright (c) AXBY Technologies, 2025. All rights reserved. 4 | 5 | Permission is hereby granted to use the LiveSplat software (the “Software”) solely for: 6 | 7 | - Personal entertainment 8 | - Academic research 9 | - Internal evaluation purposes, including within commercial organizations 10 | 11 | Subject to the following restrictions: 12 | 13 | 1. **Prohibited Commercial Use** 14 | Use of the Software for commercial production, deployment, or any commercial advantage beyond internal evaluation is prohibited without a separate commercial license agreement. 15 | 16 | 2. **No Redistribution** 17 | You may not copy, distribute, sublicense, or otherwise make the Software available to any third party. 18 | 19 | 3. **No Modification** 20 | You may not modify, translate, adapt, or create derivative works based on the Software. 21 | 22 | 4. **Reverse Engineering** 23 | You may not reverse engineer, decompile, disassemble, or otherwise attempt to derive the source code of the Software, **except to the extent that such activity is expressly permitted by applicable law notwithstanding this limitation**. 24 | 25 | 5. **Third-Party Software** 26 | This Software includes third-party components that are subject to separate open-source licenses. All such components and their licenses are distributed with the application .whl file in the third_party_licenses folder. Your use of those components is governed by their respective licenses. 27 | 28 | 6. **Disclaimer of Warranty** 29 | The Software is provided “as is”, without warranty of any kind, express or implied, including but not limited to the warranties of merchantability, fitness for a particular purpose, and noninfringement. 30 | 31 | 7. **Limitation of Liability** 32 | In no event shall the copyright holder be liable for any claim, damages, or other liability, whether in an action of contract, tort, or otherwise, arising from or related to the use of the Software. 33 | 34 | 8. **Contact for Commercial Licensing** 35 | For commercial production use or redistribution rights, please contact: mark@axby.cc. 36 | 37 | By using the Software, you agree to these terms. 38 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | **LiveSplat** is an algorithm for realtime Gaussian splatting using RGBD camera streams. Join our [discord](https://discord.gg/rCF5SXnc) for discussion and help. Check out the demo video below to see the the type of output that LiveSplat produces. 6 | 7 | https://github.com/user-attachments/assets/0e41f600-36e3-4482-866e-70b5c962c6f4 8 | 9 | 10 | Message from the Author 11 | ------ 12 | LiveSplat was developed as a small part of a larger proprietary VR telerobotics system. I posted a video of the Gaussian splatting component of this system on Reddit and many people expressed an interest in experimenting with it themselves. So I spun it out and I'm making it publicly available as LiveSplat (see installation instructions below). 13 | 14 | LiveSplat should be considered alpha quality. I do not have the resources to test the installation on many different machines, so let me know if the application does not run on yours (assuming your machine meets the requirements). 15 | 16 | I've decided to keep LiveSplat closed source in order to explore potential business opportunities. If you are a business wanting to license/integrate this technology, please email mark@axby.cc. 17 | 18 | I hope you have fun with LiveSplat! 19 | 20 | — Mark Liu 21 | 22 | 23 | 24 | Requirements 25 | ------------ 26 | - Python 3.12+ 27 | - Windows or Ubuntu (other Linux distributions may work, but are untested) 28 | - x86_64 CPU 29 | - Nvidia graphics card 30 | - One or more (up to four) RGBD sensors 31 | 32 | Installation 33 | ----------- 34 | Note that the application is not open source, and is covered by this [license](https://github.com/axbycc/LiveSplat/blob/main/LICENSE). 35 | 36 | **Ubuntu:** 37 | `pip3 install https://livesplat.s3.us-east-2.amazonaws.com/livesplat-0.1.0-cp312-cp312-manylinux_x86_64.whl` 38 | 39 | **Windows:** 40 | `pip install https://livesplat.s3.us-east-2.amazonaws.com/livesplat-0.1.0-cp312-cp312-win_amd64.whl` 41 | 42 | Running 43 | ------ 44 | To run LiveSplat, you will have to create an integration script that feeds your RGBD streams to the LiveSplat viewer. This repo provides an integration script for Intel Realsense devices called [livesplat_realsense.py](https://github.com/axbycc/LiveSplat/blob/main/livesplat_realsense.py). 45 | 46 | -------------------------------------------------------------------------------- /livesplat_realsense.py: -------------------------------------------------------------------------------- 1 | """ 2 | After installing LiveSplat, connect your Realsense devices and run this 3 | script to launch the application. 4 | 5 | Important: This script requires pyrealsense2. The Python 3.12+ version 6 | is currently published under the name "pyrealsense2-beta". Get it by 7 | running `pip install pyrealsense2-beta`. 8 | """ 9 | 10 | import numpy as np 11 | import livesplat 12 | import pyrealsense2 as rs # pip install pyrealsense2-beta 13 | 14 | def get_device_metadata(desired_width, desired_height, desired_fps): 15 | """This uses pyrealsense2 to query the connected RealSense 16 | devices. For each device, it pulls out the serial number, color 17 | and depth sensor objects, and color and depth stream profile 18 | objects matching the desired width, height, and fps. These are 19 | returned in a list of dict with keys serial_number, color_sensor, 20 | color_profile, depth_sensor, depth_profile. 21 | """ 22 | metadata = [] 23 | 24 | # Create context and get devices 25 | context = rs.context() 26 | devices = context.query_devices() 27 | for i, dev in enumerate(devices): 28 | color_profile = None 29 | depth_profile = None 30 | color_sensor = None 31 | depth_sensor = None 32 | 33 | serial = dev.get_info(rs.camera_info.serial_number) 34 | print(f"\nDevice {i}: Serial {serial}") 35 | 36 | for sensor in dev.query_sensors(): 37 | sensor_name = sensor.get_info(rs.camera_info.name) 38 | for profile in sensor.get_stream_profiles(): 39 | video_profile = profile.as_video_stream_profile() 40 | stream_type = video_profile.stream_type() 41 | 42 | if (video_profile.width() == desired_width and 43 | video_profile.height() == desired_height and 44 | video_profile.fps() == desired_fps): 45 | 46 | if stream_type == rs.stream.color and color_profile is None: 47 | color_profile = video_profile 48 | color_sensor = sensor 49 | print(f" Found color profile on sensor '{sensor_name}'") 50 | 51 | elif stream_type == rs.stream.depth and depth_profile is None: 52 | depth_profile = video_profile 53 | depth_sensor = sensor 54 | print(f" Found depth profile on sensor '{sensor_name}'") 55 | 56 | # Stop early if both profiles found 57 | if color_profile and depth_profile: 58 | break 59 | 60 | if color_profile and depth_profile: 61 | metadata.append({ 62 | "serial_number": serial, 63 | "color_sensor": color_sensor, 64 | "color_profile": color_profile, 65 | "depth_sensor": depth_sensor, 66 | "depth_profile": depth_profile, 67 | }) 68 | else: 69 | print(" Warning: Device is missing color or depth sensor") 70 | continue 71 | return metadata 72 | 73 | def intrinsics_to_mat3x3(fx, fy, ppx, ppy): 74 | """Converts from focal params and pixel offset to 3x3 camera matrix""" 75 | result = np.eye(3) 76 | result[0,0] = fx 77 | result[1,1] = fy 78 | result[0,2] = ppx 79 | result[1,2] = ppy 80 | return result 81 | 82 | 83 | def main(): 84 | metadata = get_device_metadata(desired_width=640, desired_height=480, desired_fps=30) 85 | 86 | # create a mapping from profile uid to serial numbers this is used 87 | # when receiving frames from the frame queue to back out which 88 | # device the frame is coming from 89 | profile_uid_to_serial_number = {} 90 | for datum in metadata: 91 | serial_number = datum["serial_number"] 92 | color_profile = datum["color_profile"].as_video_stream_profile() 93 | depth_profile = datum["depth_profile"].as_video_stream_profile() 94 | profile_uid_to_serial_number[color_profile.unique_id()] = serial_number 95 | profile_uid_to_serial_number[depth_profile.unique_id()] = serial_number 96 | 97 | # register all cameras 98 | for i, datum in enumerate(metadata): 99 | color_profile = datum["color_profile"].as_video_stream_profile() 100 | color_intrinsics = color_profile.get_intrinsics() 101 | depth_profile = datum["depth_profile"].as_video_stream_profile() 102 | depth_intrinsics = depth_profile.get_intrinsics() 103 | depth_scale = datum["depth_sensor"].get_option(rs.option.depth_units) 104 | serial_number = datum["serial_number"] 105 | 106 | pose_rgb_depth = depth_profile.get_extrinsics_to(color_profile) 107 | rot_rgb_depth = np.array(pose_rgb_depth.rotation).reshape((3,3), order="C") 108 | trans_rgb_depth = np.array(pose_rgb_depth.translation) 109 | tx_rgb_depth = np.eye(4) 110 | tx_rgb_depth[:3,:3] = rot_rgb_depth 111 | tx_rgb_depth[:3,-1] = trans_rgb_depth 112 | 113 | depth_width = depth_intrinsics.width 114 | depth_height = depth_intrinsics.height 115 | rgb_width = color_intrinsics.width 116 | rgb_height = color_intrinsics.height 117 | 118 | hm_depthimage_depth = intrinsics_to_mat3x3( 119 | depth_intrinsics.fx, 120 | depth_intrinsics.fy, 121 | depth_intrinsics.ppx, 122 | depth_intrinsics.ppy) 123 | 124 | hm_rgbimage_rgb = intrinsics_to_mat3x3( 125 | color_intrinsics.fx, 126 | color_intrinsics.fy, 127 | color_intrinsics.ppx, 128 | color_intrinsics.ppy) 129 | 130 | livesplat.register_camera_params( 131 | device_id = serial_number, 132 | rgb_width = rgb_width, 133 | rgb_height = rgb_height, 134 | depth_width = depth_width, 135 | depth_height = depth_height, 136 | hm_rgbimage_rgb = hm_rgbimage_rgb, 137 | hm_depthimage_depth = hm_depthimage_depth, 138 | tx_rgb_depth = tx_rgb_depth, 139 | depth_scale = depth_scale, 140 | ) 141 | 142 | # start all cameras 143 | frame_queue = rs.frame_queue(10) 144 | for i, datum in enumerate(metadata): 145 | color_profile = datum["color_profile"].as_video_stream_profile() 146 | depth_profile = datum["depth_profile"].as_video_stream_profile() 147 | color_sensor = datum["color_sensor"] 148 | depth_sensor = datum["depth_sensor"] 149 | color_sensor.open(color_profile) 150 | color_sensor.start(frame_queue) 151 | depth_sensor.open(depth_profile) 152 | depth_sensor.start(frame_queue) 153 | 154 | livesplat.start_viewer() 155 | while not livesplat.should_stop_all(): 156 | frame = frame_queue.poll_for_frame() 157 | if frame: 158 | profile = frame.get_profile() 159 | profile_uid = profile.unique_id() 160 | serial_number = profile_uid_to_serial_number[profile_uid] 161 | stream_type = profile.stream_type() 162 | if stream_type == rs.stream.color: 163 | data = frame.get_data() 164 | livesplat.ingest_rgb( 165 | serial_number, 166 | data) 167 | if stream_type == rs.stream.depth: 168 | data = frame.get_data() 169 | livesplat.ingest_depth( 170 | serial_number, 171 | data) 172 | 173 | for datum in metadata: 174 | color_sensor = datum["color_sensor"] 175 | depth_sensor = datum["depth_sensor"] 176 | color_sensor.stop() 177 | color_sensor.close() 178 | depth_sensor.stop() 179 | depth_sensor.close() 180 | 181 | if __name__ == '__main__': 182 | main() 183 | --------------------------------------------------------------------------------