├── .gitattributes ├── .gitignore ├── LICENSE ├── README.md ├── assets ├── block-diagram.png └── cad-drawing.png ├── electronics ├── .gitignore ├── Footprints.pretty │ └── Alps_HSFPAR003A.kicad_mod ├── ForceSensor(HSFPAR003A).STEP ├── dpoint.kicad_pcb ├── dpoint.kicad_pro ├── dpoint.kicad_sch ├── gerbers │ ├── dpoint-B_Cu.gbl │ ├── dpoint-B_Mask.gbs │ ├── dpoint-B_Paste.gbp │ ├── dpoint-B_Silkscreen.gbo │ ├── dpoint-Edge_Cuts.gm1 │ ├── dpoint-F_Cu.gtl │ ├── dpoint-F_Mask.gts │ ├── dpoint-F_Paste.gtp │ ├── dpoint-F_Silkscreen.gto │ └── dpoint.drl ├── sym-lib-table └── symbol_library.kicad_sym ├── markers ├── calib.io_charuco_297x210_8x12_24_18_DICT_4X4.pdf ├── stylus-markers.pdf └── stylus-markers.svg ├── microcontroller └── dpoint-arduino │ └── dpoint-arduino.ino ├── print ├── .gitignore ├── Compress Spring11.ipt ├── assembly.iam ├── assembly.ipn ├── batt_10440.ipt ├── dpoint.step ├── drawing.dwg ├── export │ ├── stylus_body_bottom.stl │ ├── stylus_body_top.stl │ └── stylus_nib_base.stl ├── seed_arduino.ipt ├── stylus_body.ipt ├── stylus_nib.ipt └── stylus_nib_base.ipt ├── python ├── .editorconfig ├── .gitignore ├── __init__.py ├── analysis │ ├── latencies.csv │ ├── latency-plots.ipynb │ ├── offline_ope.py │ ├── offline_playback.ipynb │ ├── offline_playback.py │ ├── requirements.txt │ ├── tip_measurement_analysis.ipynb │ └── tip_measurements │ │ ├── no2steppnp │ │ ├── predicted_20230903_104829.csv │ │ └── smoothed_20230903_104829.csv │ │ ├── nofilter │ │ ├── predicted_20230903_114230.csv │ │ ├── predicted_20230903_114250.csv │ │ ├── predicted_20230903_114351.csv │ │ ├── predicted_20230903_114505.csv │ │ ├── predicted_20230903_114549.csv │ │ ├── predicted_20230903_162552.csv │ │ ├── predicted_20230905_105507.csv │ │ ├── predicted_20230905_105713.csv │ │ ├── smoothed_20230903_114230.csv │ │ ├── smoothed_20230903_114250.csv │ │ ├── smoothed_20230903_114351.csv │ │ ├── smoothed_20230903_114505.csv │ │ ├── smoothed_20230903_114549.csv │ │ ├── smoothed_20230903_162552.csv │ │ ├── smoothed_20230905_105507.csv │ │ └── smoothed_20230905_105713.csv │ │ ├── nomarkercalib │ │ ├── predicted_20230902_191235.csv │ │ └── smoothed_20230902_191235.csv │ │ └── proposed │ │ ├── predicted_20230902_143553.csv │ │ ├── predicted_20230905_110153.csv │ │ ├── smoothed_20230902_143553.csv │ │ └── smoothed_20230905_110153.csv ├── app │ ├── __init__.py │ ├── app.py │ ├── camera_cov.py │ ├── color_button.py │ ├── dimensions.py │ ├── filter.py │ ├── filter_core.py │ ├── marker_tracker.py │ └── monitor_ble.py ├── calibrate_camera.py ├── calibrate_markers.py ├── calibration_pics │ └── f30 │ │ ├── 1687925759.jpg │ │ ├── 1687925828.jpg │ │ ├── 1687925831.jpg │ │ ├── 1687925842.jpg │ │ ├── 1687925848.jpg │ │ ├── 1687925851.jpg │ │ ├── 1687925856.jpg │ │ └── 1687925863.jpg ├── main.py ├── mesh │ ├── pen.blend │ └── pen.obj ├── params │ └── camera_params_c922_f30.yml ├── recordings │ └── 20230905_162344 │ │ ├── camera_data.json │ │ ├── camera_data_epnp.json │ │ ├── camera_data_nomarkercalib.json │ │ ├── camera_data_norsc.json │ │ ├── camera_data_original.json │ │ ├── camera_data_sqpnp.json │ │ ├── camera_extrinsics.pkl │ │ ├── scan-original.jpg │ │ ├── scan.jpg │ │ └── stylus_data.json ├── requirements.txt ├── run_marker_tracker.py └── test │ ├── test_filter.py │ ├── test_filter.test_camera_measurement.approved.txt │ ├── test_filter.test_imu_measurement.approved.txt │ ├── test_filter.test_state_transition.approved.txt │ └── test_filter.test_state_transition_jacobian.approved.txt └── setup-guide.md /.gitattributes: -------------------------------------------------------------------------------- 1 | * text=auto 2 | 3 | *.ipynb linguist-vendored -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | # Autosave files 2 | *.asv 3 | *.m~ 4 | *.autosave 5 | *.slx.r* 6 | *.mdl.r* 7 | 8 | # Derived content-obscured files 9 | *.p 10 | 11 | # Compiled MEX files 12 | *.mex* 13 | 14 | # Packaged app and toolbox files 15 | *.mlappinstall 16 | *.mltbx 17 | 18 | # Deployable archives 19 | *.ctf 20 | 21 | # Generated helpsearch folders 22 | helpsearch*/ 23 | 24 | # Code generation folders 25 | slprj/ 26 | sccprj/ 27 | codegen/ 28 | 29 | # Cache files 30 | *.slxc 31 | 32 | # Cloud based storage dotfile 33 | .MATLABDriveTag 34 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2023 Jcparkyn 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # D-POINT: Digital Pen with Optical-Inertial Tracking 2 | 3 | **D-POINT** is an open-source digital stylus that uses camera tracking and inertial measurements to achieve 6DoF (six degrees of freedom) inputs, with low latency, pressure sensitivity, and sub-millimetre accuracy. 4 | The stylus can be used on any flat surface, and works with consumer-grade webcams. 5 | 6 | 7 | 8 | This project was part of my undergraduate thesis for electrical engineering. I've open-sourced the code and design files in the hopes that they might be useful to somebody, but it's not intended to be a "plug and play" DIY project. If you want to try building it anyway, follow the [setup guide](./setup-guide.md). 9 | 10 | ## Design 11 | 12 | This is a very brief overview of how the system works. For all the details, plus literature review and lots of evaluation, read the full thesis (note: I haven't published this online yet). 13 | 14 | ![Block diagram showing how the system works](assets/block-diagram.png) 15 | 16 | ### Hardware 17 | 18 | The main body of the stylus was 3D printed as two halves, shown below. The stylus contains a force sensor, a Li-ion battery which charges over USB-C, and an Arduino-based development board for logic and Bluetooth. Eight printed [ArUco](https://www.uco.es/investiga/grupos/ava/portfolio/aruco/) markers are glued to the back of the stylus, for visual pose estimation. 19 | 20 | ![CAD drawing showing the hardware design of the stylus](assets/cad-drawing.png) 21 | 22 | ### Visual pose estimation (VPE) 23 | 24 | The VPE process involves the four main steps: 25 | 1. **Marker detection:** First, we use OpenCV to detect the corners of each visible ArUco marker on the stylus. 26 | 1. **Rolling shutter correction:** We use a simple 2D motion model to estimate and correct for the effects of [rolling shutter](https://en.wikipedia.org/wiki/Rolling_shutter) on the observed corner locations. 27 | 1. **Perspective-n-Point (PnP):** From these corner positions, we use a [PnP](https://en.wikipedia.org/wiki/Perspective-n-Point) algorithm to estimate the pose of the stylus relative to the camera. When possible, we use the pose from the previous frame as a starting point to refine with virtual visual servoing (VVS), otherwise we fall back to SQPnP. 28 | 1. **Coordinate conversion:** Using the calibrated pose of the stylus and the drawing surface relative to the camera, we calculate the position and orientation of the stylus tip relative to the drawing surface. 29 | 30 | ### Inertial fusion 31 | 32 | We use an Extended Kalman Filter (EKF) to fuse the VPE estimates with the inertial data from the accelerometer and gyroscope, and refine the estimates in real-time using the Rauch-Tung-Striebel (RTS) algorithm. To account for time delay from the camera frames, we use a negative-time measurement update algorithm. The EKF is implemented using NumPy and [Numba](https://numba.pydata.org/). 33 | 34 | Using inertial measurements allows us to dramatically reduce latency compared to a camera-only implementation, while also improving accuracy and report rate for fast movements. 35 | -------------------------------------------------------------------------------- /assets/block-diagram.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Jcparkyn/dpoint/a108c19b9b240c1531b2d30ed47ba18bc604862e/assets/block-diagram.png -------------------------------------------------------------------------------- /assets/cad-drawing.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Jcparkyn/dpoint/a108c19b9b240c1531b2d30ed47ba18bc604862e/assets/cad-drawing.png -------------------------------------------------------------------------------- /electronics/.gitignore: -------------------------------------------------------------------------------- 1 | # For PCBs designed using KiCad: https://www.kicad.org/ 2 | # Format documentation: https://kicad.org/help/file-formats/ 3 | 4 | # Temporary files 5 | *.000 6 | *.bak 7 | *.bck 8 | *.kicad_pcb-bak 9 | *.kicad_sch-bak 10 | *-backups 11 | *.kicad_prl 12 | *.sch-bak 13 | *~ 14 | _autosave-* 15 | *.tmp 16 | *-save.pro 17 | *-save.kicad_pcb 18 | fp-info-cache 19 | 20 | # Netlist files (exported from Eeschema) 21 | *.net 22 | 23 | # Autorouter files (exported from Pcbnew) 24 | *.dsn 25 | *.ses 26 | 27 | # Exported BOM files 28 | *.xml 29 | *.csv -------------------------------------------------------------------------------- /electronics/Footprints.pretty/Alps_HSFPAR003A.kicad_mod: -------------------------------------------------------------------------------- 1 | (footprint "Alps_HSFPAR003A" (version 20221018) (generator pcbnew) 2 | (layer "F.Cu") 3 | (attr smd) 4 | (fp_text reference "REF**" (at 0 2.54 unlocked) (layer "F.SilkS") 5 | (effects (font (size 1 1) (thickness 0.1))) 6 | (tstamp 220dd604-a029-4791-bcb9-f3ecd5726fe5) 7 | ) 8 | (fp_text value "Alps_HSFPAR003A" (at 0 4.04 unlocked) (layer "F.Fab") 9 | (effects (font (size 1 1) (thickness 0.15))) 10 | (tstamp ce263a11-b401-468c-9988-12d1d144144d) 11 | ) 12 | (fp_text user "${REFERENCE}" (at 0 5.54 unlocked) (layer "F.Fab") 13 | (effects (font (size 1 1) (thickness 0.15))) 14 | (tstamp ba289319-6dd9-4d39-a3e2-82d29a3f39bc) 15 | ) 16 | (pad "1" smd roundrect (at -0.6 -0.7) (size 0.3 0.5) (layers "F.Cu" "F.Paste" "F.Mask") (roundrect_rratio 0.25) 17 | (thermal_bridge_angle 45) (tstamp aead15a8-362b-4104-a382-d6fe2b46d5e1)) 18 | (pad "2" smd roundrect (at 0.6 -0.7) (size 0.3 0.5) (layers "F.Cu" "F.Paste" "F.Mask") (roundrect_rratio 0.25) 19 | (thermal_bridge_angle 45) (tstamp a54e2bb0-b5f9-426e-9161-e95e8a1bf337)) 20 | (pad "3" smd roundrect (at 0 0) (size 0.6 0.6) (layers "F.Cu" "F.Paste" "F.Mask") (roundrect_rratio 0.25) 21 | (thermal_bridge_angle 45) (tstamp 942b6f32-b73f-423f-a388-a6e1cb8ef661)) 22 | (pad "3" smd roundrect (at 0.6 0.7) (size 0.3 0.5) (layers "F.Cu" "F.Paste" "F.Mask") (roundrect_rratio 0.25) 23 | (thermal_bridge_angle 45) (tstamp d313d3ff-04f8-4519-ba4f-34cf211722b1)) 24 | (pad "4" smd roundrect (at -0.6 0.7) (size 0.3 0.5) (layers "F.Cu" "F.Paste" "F.Mask") (roundrect_rratio 0.25) 25 | (thermal_bridge_angle 45) (tstamp ac134343-1a75-4b41-aad9-7e10cd7c242c)) 26 | ) 27 | -------------------------------------------------------------------------------- /electronics/ForceSensor(HSFPAR003A).STEP: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Jcparkyn/dpoint/a108c19b9b240c1531b2d30ed47ba18bc604862e/electronics/ForceSensor(HSFPAR003A).STEP -------------------------------------------------------------------------------- /electronics/dpoint.kicad_pro: -------------------------------------------------------------------------------- 1 | { 2 | "board": { 3 | "3dviewports": [], 4 | "design_settings": { 5 | "defaults": { 6 | "board_outline_line_width": 0.09999999999999999, 7 | "copper_line_width": 0.19999999999999998, 8 | "copper_text_italic": false, 9 | "copper_text_size_h": 1.5, 10 | "copper_text_size_v": 1.5, 11 | "copper_text_thickness": 0.3, 12 | "copper_text_upright": false, 13 | "courtyard_line_width": 0.049999999999999996, 14 | "dimension_precision": 4, 15 | "dimension_units": 3, 16 | "dimensions": { 17 | "arrow_length": 1270000, 18 | "extension_offset": 500000, 19 | "keep_text_aligned": true, 20 | "suppress_zeroes": false, 21 | "text_position": 0, 22 | "units_format": 1 23 | }, 24 | "fab_line_width": 0.09999999999999999, 25 | "fab_text_italic": false, 26 | "fab_text_size_h": 1.0, 27 | "fab_text_size_v": 1.0, 28 | "fab_text_thickness": 0.15, 29 | "fab_text_upright": false, 30 | "other_line_width": 0.15, 31 | "other_text_italic": false, 32 | "other_text_size_h": 1.0, 33 | "other_text_size_v": 1.0, 34 | "other_text_thickness": 0.15, 35 | "other_text_upright": false, 36 | "pads": { 37 | "drill": 1.1, 38 | "height": 2.1, 39 | "width": 2.1 40 | }, 41 | "silk_line_width": 0.15, 42 | "silk_text_italic": false, 43 | "silk_text_size_h": 1.0, 44 | "silk_text_size_v": 1.0, 45 | "silk_text_thickness": 0.15, 46 | "silk_text_upright": false, 47 | "zones": { 48 | "min_clearance": 0.5 49 | } 50 | }, 51 | "diff_pair_dimensions": [ 52 | { 53 | "gap": 0.0, 54 | "via_gap": 0.0, 55 | "width": 0.0 56 | } 57 | ], 58 | "drc_exclusions": [], 59 | "meta": { 60 | "version": 2 61 | }, 62 | "rule_severities": { 63 | "annular_width": "error", 64 | "clearance": "error", 65 | "connection_width": "warning", 66 | "copper_edge_clearance": "error", 67 | "copper_sliver": "warning", 68 | "courtyards_overlap": "error", 69 | "diff_pair_gap_out_of_range": "error", 70 | "diff_pair_uncoupled_length_too_long": "error", 71 | "drill_out_of_range": "error", 72 | "duplicate_footprints": "warning", 73 | "extra_footprint": "warning", 74 | "footprint": "error", 75 | "footprint_type_mismatch": "warning", 76 | "hole_clearance": "error", 77 | "hole_near_hole": "error", 78 | "invalid_outline": "error", 79 | "isolated_copper": "warning", 80 | "item_on_disabled_layer": "error", 81 | "items_not_allowed": "error", 82 | "length_out_of_range": "error", 83 | "lib_footprint_issues": "warning", 84 | "lib_footprint_mismatch": "ignore", 85 | "malformed_courtyard": "error", 86 | "microvia_drill_out_of_range": "error", 87 | "missing_courtyard": "warning", 88 | "missing_footprint": "warning", 89 | "net_conflict": "warning", 90 | "npth_inside_courtyard": "warning", 91 | "padstack": "warning", 92 | "pth_inside_courtyard": "warning", 93 | "shorting_items": "error", 94 | "silk_edge_clearance": "warning", 95 | "silk_over_copper": "warning", 96 | "silk_overlap": "warning", 97 | "skew_out_of_range": "error", 98 | "solder_mask_bridge": "error", 99 | "starved_thermal": "error", 100 | "text_height": "warning", 101 | "text_thickness": "warning", 102 | "through_hole_pad_without_hole": "error", 103 | "too_many_vias": "error", 104 | "track_dangling": "warning", 105 | "track_width": "error", 106 | "tracks_crossing": "error", 107 | "unconnected_items": "error", 108 | "unresolved_variable": "error", 109 | "via_dangling": "warning", 110 | "zones_intersect": "error" 111 | }, 112 | "rules": { 113 | "max_error": 0.005, 114 | "min_clearance": 0.25, 115 | "min_connection": 0.0, 116 | "min_copper_edge_clearance": 0.0, 117 | "min_hole_clearance": 0.25, 118 | "min_hole_to_hole": 0.25, 119 | "min_microvia_diameter": 0.19999999999999998, 120 | "min_microvia_drill": 0.09999999999999999, 121 | "min_resolved_spokes": 2, 122 | "min_silk_clearance": 0.0, 123 | "min_text_height": 0.7999999999999999, 124 | "min_text_thickness": 0.08, 125 | "min_through_hole_diameter": 0.3, 126 | "min_track_width": 0.25, 127 | "min_via_annular_width": 0.09999999999999999, 128 | "min_via_diameter": 0.5, 129 | "solder_mask_clearance": 0.0, 130 | "solder_mask_min_width": 0.0, 131 | "solder_mask_to_copper_clearance": 0.0, 132 | "use_height_for_length_calcs": true 133 | }, 134 | "teardrop_options": [ 135 | { 136 | "td_allow_use_two_tracks": true, 137 | "td_curve_segcount": 5, 138 | "td_on_pad_in_zone": false, 139 | "td_onpadsmd": true, 140 | "td_onroundshapesonly": false, 141 | "td_ontrackend": false, 142 | "td_onviapad": true 143 | } 144 | ], 145 | "teardrop_parameters": [ 146 | { 147 | "td_curve_segcount": 0, 148 | "td_height_ratio": 1.0, 149 | "td_length_ratio": 0.5, 150 | "td_maxheight": 2.0, 151 | "td_maxlen": 1.0, 152 | "td_target_name": "td_round_shape", 153 | "td_width_to_size_filter_ratio": 0.9 154 | }, 155 | { 156 | "td_curve_segcount": 0, 157 | "td_height_ratio": 1.0, 158 | "td_length_ratio": 0.5, 159 | "td_maxheight": 2.0, 160 | "td_maxlen": 1.0, 161 | "td_target_name": "td_rect_shape", 162 | "td_width_to_size_filter_ratio": 0.9 163 | }, 164 | { 165 | "td_curve_segcount": 0, 166 | "td_height_ratio": 1.0, 167 | "td_length_ratio": 0.5, 168 | "td_maxheight": 2.0, 169 | "td_maxlen": 1.0, 170 | "td_target_name": "td_track_end", 171 | "td_width_to_size_filter_ratio": 0.9 172 | } 173 | ], 174 | "track_widths": [ 175 | 0.0 176 | ], 177 | "via_dimensions": [ 178 | { 179 | "diameter": 0.0, 180 | "drill": 0.0 181 | } 182 | ], 183 | "zones_allow_external_fillets": false 184 | }, 185 | "layer_presets": [], 186 | "viewports": [] 187 | }, 188 | "boards": [], 189 | "cvpcb": { 190 | "equivalence_files": [] 191 | }, 192 | "erc": { 193 | "erc_exclusions": [], 194 | "meta": { 195 | "version": 0 196 | }, 197 | "pin_map": [ 198 | [ 199 | 0, 200 | 0, 201 | 0, 202 | 0, 203 | 0, 204 | 0, 205 | 1, 206 | 0, 207 | 0, 208 | 0, 209 | 0, 210 | 2 211 | ], 212 | [ 213 | 0, 214 | 2, 215 | 0, 216 | 1, 217 | 0, 218 | 0, 219 | 1, 220 | 0, 221 | 2, 222 | 2, 223 | 2, 224 | 2 225 | ], 226 | [ 227 | 0, 228 | 0, 229 | 0, 230 | 0, 231 | 0, 232 | 0, 233 | 1, 234 | 0, 235 | 1, 236 | 0, 237 | 1, 238 | 2 239 | ], 240 | [ 241 | 0, 242 | 1, 243 | 0, 244 | 0, 245 | 0, 246 | 0, 247 | 1, 248 | 1, 249 | 2, 250 | 1, 251 | 1, 252 | 2 253 | ], 254 | [ 255 | 0, 256 | 0, 257 | 0, 258 | 0, 259 | 0, 260 | 0, 261 | 1, 262 | 0, 263 | 0, 264 | 0, 265 | 0, 266 | 2 267 | ], 268 | [ 269 | 0, 270 | 0, 271 | 0, 272 | 0, 273 | 0, 274 | 0, 275 | 0, 276 | 0, 277 | 0, 278 | 0, 279 | 0, 280 | 2 281 | ], 282 | [ 283 | 1, 284 | 1, 285 | 1, 286 | 1, 287 | 1, 288 | 0, 289 | 1, 290 | 1, 291 | 1, 292 | 1, 293 | 1, 294 | 2 295 | ], 296 | [ 297 | 0, 298 | 0, 299 | 0, 300 | 1, 301 | 0, 302 | 0, 303 | 1, 304 | 0, 305 | 0, 306 | 0, 307 | 0, 308 | 2 309 | ], 310 | [ 311 | 0, 312 | 2, 313 | 1, 314 | 2, 315 | 0, 316 | 0, 317 | 1, 318 | 0, 319 | 2, 320 | 2, 321 | 2, 322 | 2 323 | ], 324 | [ 325 | 0, 326 | 2, 327 | 0, 328 | 1, 329 | 0, 330 | 0, 331 | 1, 332 | 0, 333 | 2, 334 | 0, 335 | 0, 336 | 2 337 | ], 338 | [ 339 | 0, 340 | 2, 341 | 1, 342 | 1, 343 | 0, 344 | 0, 345 | 1, 346 | 0, 347 | 2, 348 | 0, 349 | 0, 350 | 2 351 | ], 352 | [ 353 | 2, 354 | 2, 355 | 2, 356 | 2, 357 | 2, 358 | 2, 359 | 2, 360 | 2, 361 | 2, 362 | 2, 363 | 2, 364 | 2 365 | ] 366 | ], 367 | "rule_severities": { 368 | "bus_definition_conflict": "error", 369 | "bus_entry_needed": "error", 370 | "bus_to_bus_conflict": "error", 371 | "bus_to_net_conflict": "error", 372 | "conflicting_netclasses": "error", 373 | "different_unit_footprint": "error", 374 | "different_unit_net": "error", 375 | "duplicate_reference": "error", 376 | "duplicate_sheet_names": "error", 377 | "endpoint_off_grid": "warning", 378 | "extra_units": "error", 379 | "global_label_dangling": "warning", 380 | "hier_label_mismatch": "error", 381 | "label_dangling": "error", 382 | "lib_symbol_issues": "warning", 383 | "missing_bidi_pin": "warning", 384 | "missing_input_pin": "warning", 385 | "missing_power_pin": "error", 386 | "missing_unit": "warning", 387 | "multiple_net_names": "warning", 388 | "net_not_bus_member": "warning", 389 | "no_connect_connected": "warning", 390 | "no_connect_dangling": "warning", 391 | "pin_not_connected": "error", 392 | "pin_not_driven": "error", 393 | "pin_to_pin": "warning", 394 | "power_pin_not_driven": "error", 395 | "similar_labels": "warning", 396 | "simulation_model_issue": "error", 397 | "unannotated": "error", 398 | "unit_value_mismatch": "error", 399 | "unresolved_variable": "error", 400 | "wire_dangling": "error" 401 | } 402 | }, 403 | "libraries": { 404 | "pinned_footprint_libs": [], 405 | "pinned_symbol_libs": [] 406 | }, 407 | "meta": { 408 | "filename": "dpoint.kicad_pro", 409 | "version": 1 410 | }, 411 | "net_settings": { 412 | "classes": [ 413 | { 414 | "bus_width": 12, 415 | "clearance": 0.2, 416 | "diff_pair_gap": 0.25, 417 | "diff_pair_via_gap": 0.25, 418 | "diff_pair_width": 0.2, 419 | "line_style": 0, 420 | "microvia_diameter": 0.3, 421 | "microvia_drill": 0.1, 422 | "name": "Default", 423 | "pcb_color": "rgba(0, 0, 0, 0.000)", 424 | "schematic_color": "rgba(0, 0, 0, 0.000)", 425 | "track_width": 0.25, 426 | "via_diameter": 0.8, 427 | "via_drill": 0.4, 428 | "wire_width": 6 429 | } 430 | ], 431 | "meta": { 432 | "version": 3 433 | }, 434 | "net_colors": null, 435 | "netclass_assignments": null, 436 | "netclass_patterns": [] 437 | }, 438 | "pcbnew": { 439 | "last_paths": { 440 | "gencad": "", 441 | "idf": "", 442 | "netlist": "", 443 | "specctra_dsn": "", 444 | "step": "../inventor/dpoint.step", 445 | "vrml": "" 446 | }, 447 | "page_layout_descr_file": "" 448 | }, 449 | "schematic": { 450 | "annotate_start_num": 0, 451 | "drawing": { 452 | "dashed_lines_dash_length_ratio": 12.0, 453 | "dashed_lines_gap_length_ratio": 3.0, 454 | "default_line_thickness": 6.0, 455 | "default_text_size": 50.0, 456 | "field_names": [], 457 | "intersheets_ref_own_page": false, 458 | "intersheets_ref_prefix": "", 459 | "intersheets_ref_short": false, 460 | "intersheets_ref_show": false, 461 | "intersheets_ref_suffix": "", 462 | "junction_size_choice": 3, 463 | "label_size_ratio": 0.375, 464 | "pin_symbol_size": 25.0, 465 | "text_offset_ratio": 0.15 466 | }, 467 | "legacy_lib_dir": "", 468 | "legacy_lib_list": [], 469 | "meta": { 470 | "version": 1 471 | }, 472 | "net_format_name": "", 473 | "page_layout_descr_file": "", 474 | "plot_directory": "", 475 | "spice_current_sheet_as_root": false, 476 | "spice_external_command": "spice \"%I\"", 477 | "spice_model_current_sheet_as_root": true, 478 | "spice_save_all_currents": false, 479 | "spice_save_all_voltages": false, 480 | "subpart_first_id": 65, 481 | "subpart_id_separator": 0 482 | }, 483 | "sheets": [ 484 | [ 485 | "8aa3f855-a562-4b4c-aec0-abc662e54be3", 486 | "" 487 | ] 488 | ], 489 | "text_variables": {} 490 | } 491 | -------------------------------------------------------------------------------- /electronics/dpoint.kicad_sch: -------------------------------------------------------------------------------- 1 | (kicad_sch (version 20230121) (generator eeschema) 2 | 3 | (uuid 8aa3f855-a562-4b4c-aec0-abc662e54be3) 4 | 5 | (paper "A5") 6 | 7 | (lib_symbols 8 | (symbol "Connector:Conn_01x01_Socket" (pin_names (offset 1.016) hide) (in_bom yes) (on_board yes) 9 | (property "Reference" "J" (at 0 2.54 0) 10 | (effects (font (size 1.27 1.27))) 11 | ) 12 | (property "Value" "Conn_01x01_Socket" (at 0 -2.54 0) 13 | (effects (font (size 1.27 1.27))) 14 | ) 15 | (property "Footprint" "" (at 0 0 0) 16 | (effects (font (size 1.27 1.27)) hide) 17 | ) 18 | (property "Datasheet" "~" (at 0 0 0) 19 | (effects (font (size 1.27 1.27)) hide) 20 | ) 21 | (property "ki_locked" "" (at 0 0 0) 22 | (effects (font (size 1.27 1.27))) 23 | ) 24 | (property "ki_keywords" "connector" (at 0 0 0) 25 | (effects (font (size 1.27 1.27)) hide) 26 | ) 27 | (property "ki_description" "Generic connector, single row, 01x01, script generated" (at 0 0 0) 28 | (effects (font (size 1.27 1.27)) hide) 29 | ) 30 | (property "ki_fp_filters" "Connector*:*_1x??_*" (at 0 0 0) 31 | (effects (font (size 1.27 1.27)) hide) 32 | ) 33 | (symbol "Conn_01x01_Socket_1_1" 34 | (polyline 35 | (pts 36 | (xy -1.27 0) 37 | (xy -0.508 0) 38 | ) 39 | (stroke (width 0.1524) (type default)) 40 | (fill (type none)) 41 | ) 42 | (arc (start 0 0.508) (mid -0.5058 0) (end 0 -0.508) 43 | (stroke (width 0.1524) (type default)) 44 | (fill (type none)) 45 | ) 46 | (pin passive line (at -5.08 0 0) (length 3.81) 47 | (name "Pin_1" (effects (font (size 1.27 1.27)))) 48 | (number "1" (effects (font (size 1.27 1.27)))) 49 | ) 50 | ) 51 | ) 52 | (symbol "Device:Battery_Cell" (pin_numbers hide) (pin_names (offset 0) hide) (in_bom yes) (on_board yes) 53 | (property "Reference" "BT" (at 2.54 2.54 0) 54 | (effects (font (size 1.27 1.27)) (justify left)) 55 | ) 56 | (property "Value" "Battery_Cell" (at 2.54 0 0) 57 | (effects (font (size 1.27 1.27)) (justify left)) 58 | ) 59 | (property "Footprint" "" (at 0 1.524 90) 60 | (effects (font (size 1.27 1.27)) hide) 61 | ) 62 | (property "Datasheet" "~" (at 0 1.524 90) 63 | (effects (font (size 1.27 1.27)) hide) 64 | ) 65 | (property "ki_keywords" "battery cell" (at 0 0 0) 66 | (effects (font (size 1.27 1.27)) hide) 67 | ) 68 | (property "ki_description" "Single-cell battery" (at 0 0 0) 69 | (effects (font (size 1.27 1.27)) hide) 70 | ) 71 | (symbol "Battery_Cell_0_1" 72 | (rectangle (start -2.286 1.778) (end 2.286 1.524) 73 | (stroke (width 0) (type default)) 74 | (fill (type outline)) 75 | ) 76 | (rectangle (start -1.5748 1.1938) (end 1.4732 0.6858) 77 | (stroke (width 0) (type default)) 78 | (fill (type outline)) 79 | ) 80 | (polyline 81 | (pts 82 | (xy 0 0.762) 83 | (xy 0 0) 84 | ) 85 | (stroke (width 0) (type default)) 86 | (fill (type none)) 87 | ) 88 | (polyline 89 | (pts 90 | (xy 0 1.778) 91 | (xy 0 2.54) 92 | ) 93 | (stroke (width 0) (type default)) 94 | (fill (type none)) 95 | ) 96 | (polyline 97 | (pts 98 | (xy 0.508 3.429) 99 | (xy 1.524 3.429) 100 | ) 101 | (stroke (width 0.254) (type default)) 102 | (fill (type none)) 103 | ) 104 | (polyline 105 | (pts 106 | (xy 1.016 3.937) 107 | (xy 1.016 2.921) 108 | ) 109 | (stroke (width 0.254) (type default)) 110 | (fill (type none)) 111 | ) 112 | ) 113 | (symbol "Battery_Cell_1_1" 114 | (pin passive line (at 0 5.08 270) (length 2.54) 115 | (name "+" (effects (font (size 1.27 1.27)))) 116 | (number "1" (effects (font (size 1.27 1.27)))) 117 | ) 118 | (pin passive line (at 0 -2.54 90) (length 2.54) 119 | (name "-" (effects (font (size 1.27 1.27)))) 120 | (number "2" (effects (font (size 1.27 1.27)))) 121 | ) 122 | ) 123 | ) 124 | (symbol "Device:C" (pin_numbers hide) (pin_names (offset 0.254)) (in_bom yes) (on_board yes) 125 | (property "Reference" "C" (at 0.635 2.54 0) 126 | (effects (font (size 1.27 1.27)) (justify left)) 127 | ) 128 | (property "Value" "C" (at 0.635 -2.54 0) 129 | (effects (font (size 1.27 1.27)) (justify left)) 130 | ) 131 | (property "Footprint" "" (at 0.9652 -3.81 0) 132 | (effects (font (size 1.27 1.27)) hide) 133 | ) 134 | (property "Datasheet" "~" (at 0 0 0) 135 | (effects (font (size 1.27 1.27)) hide) 136 | ) 137 | (property "ki_keywords" "cap capacitor" (at 0 0 0) 138 | (effects (font (size 1.27 1.27)) hide) 139 | ) 140 | (property "ki_description" "Unpolarized capacitor" (at 0 0 0) 141 | (effects (font (size 1.27 1.27)) hide) 142 | ) 143 | (property "ki_fp_filters" "C_*" (at 0 0 0) 144 | (effects (font (size 1.27 1.27)) hide) 145 | ) 146 | (symbol "C_0_1" 147 | (polyline 148 | (pts 149 | (xy -2.032 -0.762) 150 | (xy 2.032 -0.762) 151 | ) 152 | (stroke (width 0.508) (type default)) 153 | (fill (type none)) 154 | ) 155 | (polyline 156 | (pts 157 | (xy -2.032 0.762) 158 | (xy 2.032 0.762) 159 | ) 160 | (stroke (width 0.508) (type default)) 161 | (fill (type none)) 162 | ) 163 | ) 164 | (symbol "C_1_1" 165 | (pin passive line (at 0 3.81 270) (length 2.794) 166 | (name "~" (effects (font (size 1.27 1.27)))) 167 | (number "1" (effects (font (size 1.27 1.27)))) 168 | ) 169 | (pin passive line (at 0 -3.81 90) (length 2.794) 170 | (name "~" (effects (font (size 1.27 1.27)))) 171 | (number "2" (effects (font (size 1.27 1.27)))) 172 | ) 173 | ) 174 | ) 175 | (symbol "symbol_library:HSFPAR003A" (pin_names (offset 0.002)) (in_bom yes) (on_board yes) 176 | (property "Reference" "U1" (at -1.27 0 0) 177 | (effects (font (size 1.27 1.27)) (justify left)) 178 | ) 179 | (property "Value" "HSFPAR003A" (at -2.54 6.35 0) 180 | (effects (font (size 1.27 1.27))) 181 | ) 182 | (property "Footprint" "Footprints:Alps_HSFPAR003A" (at 0 0 0) 183 | (effects (font (size 1.27 1.27)) hide) 184 | ) 185 | (property "Datasheet" "" (at 0 0 0) 186 | (effects (font (size 1.27 1.27)) hide) 187 | ) 188 | (symbol "HSFPAR003A_0_1" 189 | (rectangle (start -3.81 5.08) (end 3.81 -5.08) 190 | (stroke (width 0) (type default)) 191 | (fill (type none)) 192 | ) 193 | ) 194 | (symbol "HSFPAR003A_1_1" 195 | (pin input line (at 6.35 3.81 180) (length 2.54) 196 | (name "Vdd" (effects (font (size 1.27 1.27)))) 197 | (number "1" (effects (font (size 1.27 1.27)))) 198 | ) 199 | (pin output line (at 6.35 -1.27 180) (length 2.54) 200 | (name "V2" (effects (font (size 1.27 1.27)))) 201 | (number "2" (effects (font (size 1.27 1.27)))) 202 | ) 203 | (pin input line (at 6.35 -3.81 180) (length 2.54) 204 | (name "GND" (effects (font (size 1.27 1.27)))) 205 | (number "3" (effects (font (size 1.27 1.27)))) 206 | ) 207 | (pin output line (at 6.35 1.27 180) (length 2.54) 208 | (name "V1" (effects (font (size 1.27 1.27)))) 209 | (number "4" (effects (font (size 1.27 1.27)))) 210 | ) 211 | ) 212 | ) 213 | (symbol "symbol_library:XIAO_nRF52840_SENSE" (pin_names (offset 0.002)) (in_bom yes) (on_board yes) 214 | (property "Reference" "A1" (at 1.27 -2.54 0) 215 | (effects (font (size 1.27 1.27)) (justify right)) 216 | ) 217 | (property "Value" "XIAO_nRF52840_SENSE" (at 12.7 10.16 0) 218 | (effects (font (size 1.27 1.27)) (justify right)) 219 | ) 220 | (property "Footprint" "" (at 0 0 0) 221 | (effects (font (size 1.27 1.27)) hide) 222 | ) 223 | (property "Datasheet" "" (at 0 0 0) 224 | (effects (font (size 1.27 1.27)) hide) 225 | ) 226 | (symbol "XIAO_nRF52840_SENSE_0_1" 227 | (rectangle (start -7.62 8.89) (end 6.35 -12.7) 228 | (stroke (width 0) (type default)) 229 | (fill (type none)) 230 | ) 231 | ) 232 | (symbol "XIAO_nRF52840_SENSE_1_1" 233 | (pin output line (at -10.16 5.08 0) (length 2.54) 234 | (name "3V3" (effects (font (size 1.27 1.27)))) 235 | (number "" (effects (font (size 1.27 1.27)))) 236 | ) 237 | (pin input line (at -10.16 2.54 0) (length 2.54) 238 | (name "AIN0" (effects (font (size 1.27 1.27)))) 239 | (number "" (effects (font (size 1.27 1.27)))) 240 | ) 241 | (pin input line (at -10.16 0 0) (length 2.54) 242 | (name "AIN1" (effects (font (size 1.27 1.27)))) 243 | (number "" (effects (font (size 1.27 1.27)))) 244 | ) 245 | (pin bidirectional line (at -5.08 -15.24 90) (length 2.54) 246 | (name "BAT+" (effects (font (size 1.27 1.27)))) 247 | (number "" (effects (font (size 1.27 1.27)))) 248 | ) 249 | (pin bidirectional line (at -1.27 -15.24 90) (length 2.54) 250 | (name "BAT-" (effects (font (size 1.27 1.27)))) 251 | (number "" (effects (font (size 1.27 1.27)))) 252 | ) 253 | (pin output line (at -10.16 -6.35 0) (length 2.54) 254 | (name "GND" (effects (font (size 1.27 1.27)))) 255 | (number "" (effects (font (size 1.27 1.27)))) 256 | ) 257 | ) 258 | ) 259 | ) 260 | 261 | (junction (at 73.66 44.45) (diameter 0) (color 0 0 0 0) 262 | (uuid 2b2cf6be-ee01-4a52-adc8-c21814f1654a) 263 | ) 264 | (junction (at 81.28 52.07) (diameter 0) (color 0 0 0 0) 265 | (uuid 3c9289ed-23b3-40a6-8379-9692ba60e51c) 266 | ) 267 | (junction (at 78.74 49.53) (diameter 0) (color 0 0 0 0) 268 | (uuid 8115d937-4d55-497d-bdc4-0471c3302a08) 269 | ) 270 | (junction (at 64.77 44.45) (diameter 0) (color 0 0 0 0) 271 | (uuid 89d7f5b6-24f8-4937-a25c-6c96b956906f) 272 | ) 273 | (junction (at 76.2 46.99) (diameter 0) (color 0 0 0 0) 274 | (uuid a45afc2e-11ca-4a1a-9f51-0c30491c2af4) 275 | ) 276 | (junction (at 64.77 52.07) (diameter 0) (color 0 0 0 0) 277 | (uuid ff18dd0a-7f08-4566-a7b5-f7c1965bf3f5) 278 | ) 279 | 280 | (wire (pts (xy 64.77 49.53) (xy 78.74 49.53)) 281 | (stroke (width 0) (type default)) 282 | (uuid 0e4107dc-1b64-4fd6-a80d-3ff8b5b47908) 283 | ) 284 | (wire (pts (xy 97.79 66.04) (xy 97.79 64.77)) 285 | (stroke (width 0) (type default)) 286 | (uuid 0fb430d2-dcac-4dbe-b2a8-f3d530fd6b8c) 287 | ) 288 | (wire (pts (xy 81.28 33.02) (xy 86.36 33.02)) 289 | (stroke (width 0) (type default)) 290 | (uuid 20757739-135c-4d91-9fa1-dea72ed7a9c0) 291 | ) 292 | (wire (pts (xy 64.77 46.99) (xy 76.2 46.99)) 293 | (stroke (width 0) (type default)) 294 | (uuid 211ced50-d7ca-477f-9b55-9adc90801389) 295 | ) 296 | (wire (pts (xy 76.2 22.86) (xy 76.2 46.99)) 297 | (stroke (width 0) (type default)) 298 | (uuid 2cf500bf-181c-4af1-86d6-4e0b109f0722) 299 | ) 300 | (wire (pts (xy 64.77 39.37) (xy 64.77 44.45)) 301 | (stroke (width 0) (type default)) 302 | (uuid 3826c176-4114-4af8-8cd9-c32bdb37bae8) 303 | ) 304 | (wire (pts (xy 73.66 44.45) (xy 73.66 17.78)) 305 | (stroke (width 0) (type default)) 306 | (uuid 3a26bc81-bb93-4b72-bf97-9ec7b7e8fb91) 307 | ) 308 | (wire (pts (xy 81.28 52.07) (xy 92.71 52.07)) 309 | (stroke (width 0) (type default)) 310 | (uuid 3ae2326e-c754-4e8d-b33c-b26c7c7be378) 311 | ) 312 | (wire (pts (xy 86.36 22.86) (xy 76.2 22.86)) 313 | (stroke (width 0) (type default)) 314 | (uuid 4784df79-29ed-48f1-831b-ecb3b5b6e35b) 315 | ) 316 | (wire (pts (xy 43.18 39.37) (xy 64.77 39.37)) 317 | (stroke (width 0) (type default)) 318 | (uuid 573d4130-ee5b-4e25-ab04-c67409b3386c) 319 | ) 320 | (wire (pts (xy 64.77 44.45) (xy 73.66 44.45)) 321 | (stroke (width 0) (type default)) 322 | (uuid 6aeaad8f-f5f8-4c9a-933b-76d2b84b346f) 323 | ) 324 | (wire (pts (xy 101.6 64.77) (xy 101.6 73.66)) 325 | (stroke (width 0) (type default)) 326 | (uuid 8a9e3f61-6ae4-4aa1-90bd-65224d931b0f) 327 | ) 328 | (wire (pts (xy 101.6 73.66) (xy 97.79 73.66)) 329 | (stroke (width 0) (type default)) 330 | (uuid 8bd9811e-a480-4f6f-bdce-9e42cec54e9e) 331 | ) 332 | (wire (pts (xy 73.66 44.45) (xy 92.71 44.45)) 333 | (stroke (width 0) (type default)) 334 | (uuid 8d96330f-071d-4122-8b5f-4b1be5ef8d80) 335 | ) 336 | (wire (pts (xy 76.2 46.99) (xy 92.71 46.99)) 337 | (stroke (width 0) (type default)) 338 | (uuid aed7a829-e155-4723-92a3-2b4aabb98cc4) 339 | ) 340 | (wire (pts (xy 64.77 55.88) (xy 64.77 52.07)) 341 | (stroke (width 0) (type default)) 342 | (uuid af21c9c3-4e1e-4afc-a651-cc373ae26452) 343 | ) 344 | (wire (pts (xy 64.77 52.07) (xy 81.28 52.07)) 345 | (stroke (width 0) (type default)) 346 | (uuid b07be88d-1d41-4cb5-baed-6eff16cdc803) 347 | ) 348 | (wire (pts (xy 73.66 17.78) (xy 86.36 17.78)) 349 | (stroke (width 0) (type default)) 350 | (uuid c1ab8b03-76d0-4cda-861b-b7853c111da7) 351 | ) 352 | (wire (pts (xy 78.74 49.53) (xy 78.74 27.94)) 353 | (stroke (width 0) (type default)) 354 | (uuid cfe09572-2285-47bb-b70b-a16860868af4) 355 | ) 356 | (wire (pts (xy 43.18 55.88) (xy 64.77 55.88)) 357 | (stroke (width 0) (type default)) 358 | (uuid d079d5d7-8b38-446f-a6cb-18e148610c73) 359 | ) 360 | (wire (pts (xy 81.28 52.07) (xy 81.28 33.02)) 361 | (stroke (width 0) (type default)) 362 | (uuid e6054ba2-e9ca-492a-97e1-3e837d827fda) 363 | ) 364 | (wire (pts (xy 78.74 49.53) (xy 92.71 49.53)) 365 | (stroke (width 0) (type default)) 366 | (uuid e9b72995-edee-4f26-b6e0-adb59289e207) 367 | ) 368 | (wire (pts (xy 43.18 52.07) (xy 43.18 55.88)) 369 | (stroke (width 0) (type default)) 370 | (uuid f072517f-3e50-48fb-b9fd-f9bd35a7bb2d) 371 | ) 372 | (wire (pts (xy 78.74 27.94) (xy 86.36 27.94)) 373 | (stroke (width 0) (type default)) 374 | (uuid f735dffe-9665-47cb-9872-6c885fea0840) 375 | ) 376 | (wire (pts (xy 43.18 44.45) (xy 43.18 39.37)) 377 | (stroke (width 0) (type default)) 378 | (uuid f83490ff-2154-4664-ad0e-1025715f4fc7) 379 | ) 380 | (wire (pts (xy 92.71 52.07) (xy 92.71 55.88)) 381 | (stroke (width 0) (type default)) 382 | (uuid ff984892-e28e-4399-a82a-f8c14df81724) 383 | ) 384 | 385 | (symbol (lib_id "Device:Battery_Cell") (at 97.79 71.12 0) (mirror y) (unit 1) 386 | (in_bom yes) (on_board yes) (dnp no) 387 | (uuid 46cdc839-2671-4b51-8151-6e693831739b) 388 | (property "Reference" "BT1" (at 93.98 68.453 0) 389 | (effects (font (size 1.27 1.27)) (justify left)) 390 | ) 391 | (property "Value" "3.7V 350mAh" (at 93.98 70.993 0) 392 | (effects (font (size 1.27 1.27)) (justify left)) 393 | ) 394 | (property "Footprint" "" (at 97.79 69.596 90) 395 | (effects (font (size 1.27 1.27)) hide) 396 | ) 397 | (property "Datasheet" "~" (at 97.79 69.596 90) 398 | (effects (font (size 1.27 1.27)) hide) 399 | ) 400 | (pin "1" (uuid 0daf011a-7aca-452b-a1ef-78ec80d3ec98)) 401 | (pin "2" (uuid 075e9304-545b-443a-a56a-e47c859e5c82)) 402 | (instances 403 | (project "dpoint" 404 | (path "/8aa3f855-a562-4b4c-aec0-abc662e54be3" 405 | (reference "BT1") (unit 1) 406 | ) 407 | ) 408 | ) 409 | ) 410 | 411 | (symbol (lib_id "Device:C") (at 43.18 48.26 0) (unit 1) 412 | (in_bom yes) (on_board yes) (dnp no) (fields_autoplaced) 413 | (uuid 5c911619-db8a-4fda-9581-40826c7de504) 414 | (property "Reference" "C1" (at 46.99 47.625 0) 415 | (effects (font (size 1.27 1.27)) (justify left)) 416 | ) 417 | (property "Value" "0.1u" (at 46.99 50.165 0) 418 | (effects (font (size 1.27 1.27)) (justify left)) 419 | ) 420 | (property "Footprint" "Capacitor_SMD:C_1206_3216Metric_Pad1.33x1.80mm_HandSolder" (at 44.1452 52.07 0) 421 | (effects (font (size 1.27 1.27)) hide) 422 | ) 423 | (property "Datasheet" "~" (at 43.18 48.26 0) 424 | (effects (font (size 1.27 1.27)) hide) 425 | ) 426 | (pin "1" (uuid e8332179-f296-4824-be19-632e486863b1)) 427 | (pin "2" (uuid 1477bf66-d3ca-4042-bd49-9d43a4677256)) 428 | (instances 429 | (project "dpoint" 430 | (path "/8aa3f855-a562-4b4c-aec0-abc662e54be3" 431 | (reference "C1") (unit 1) 432 | ) 433 | ) 434 | ) 435 | ) 436 | 437 | (symbol (lib_id "Connector:Conn_01x01_Socket") (at 91.44 27.94 0) (unit 1) 438 | (in_bom yes) (on_board yes) (dnp no) (fields_autoplaced) 439 | (uuid 9c95808a-d700-4c4e-a10a-adc24f9e1a2e) 440 | (property "Reference" "J2" (at 92.71 27.305 0) 441 | (effects (font (size 1.27 1.27)) (justify left)) 442 | ) 443 | (property "Value" "Conn_01x01_Socket" (at 92.71 29.845 0) 444 | (effects (font (size 1.27 1.27)) (justify left)) 445 | ) 446 | (property "Footprint" "Connector_Wire:SolderWire-0.5sqmm_1x01_D0.9mm_OD2.1mm" (at 91.44 27.94 0) 447 | (effects (font (size 1.27 1.27)) hide) 448 | ) 449 | (property "Datasheet" "~" (at 91.44 27.94 0) 450 | (effects (font (size 1.27 1.27)) hide) 451 | ) 452 | (pin "1" (uuid 859098dd-3b06-4ae7-8cf1-7200a17f8927)) 453 | (instances 454 | (project "dpoint" 455 | (path "/8aa3f855-a562-4b4c-aec0-abc662e54be3" 456 | (reference "J2") (unit 1) 457 | ) 458 | ) 459 | ) 460 | ) 461 | 462 | (symbol (lib_id "Connector:Conn_01x01_Socket") (at 91.44 22.86 0) (unit 1) 463 | (in_bom yes) (on_board yes) (dnp no) (fields_autoplaced) 464 | (uuid a7325506-3e4d-41be-b937-405ba14ca64b) 465 | (property "Reference" "J1" (at 92.71 22.225 0) 466 | (effects (font (size 1.27 1.27)) (justify left)) 467 | ) 468 | (property "Value" "Conn_01x01_Socket" (at 92.71 24.765 0) 469 | (effects (font (size 1.27 1.27)) (justify left)) 470 | ) 471 | (property "Footprint" "Connector_Wire:SolderWire-0.5sqmm_1x01_D0.9mm_OD2.1mm" (at 91.44 22.86 0) 472 | (effects (font (size 1.27 1.27)) hide) 473 | ) 474 | (property "Datasheet" "~" (at 91.44 22.86 0) 475 | (effects (font (size 1.27 1.27)) hide) 476 | ) 477 | (pin "1" (uuid 7c2dc479-2fe0-4dad-96af-24f221f38999)) 478 | (instances 479 | (project "dpoint" 480 | (path "/8aa3f855-a562-4b4c-aec0-abc662e54be3" 481 | (reference "J1") (unit 1) 482 | ) 483 | ) 484 | ) 485 | ) 486 | 487 | (symbol (lib_id "Connector:Conn_01x01_Socket") (at 91.44 17.78 0) (unit 1) 488 | (in_bom yes) (on_board yes) (dnp no) (fields_autoplaced) 489 | (uuid dff4fa31-0617-4d88-bf49-f8ba40927d9b) 490 | (property "Reference" "J4" (at 92.71 17.145 0) 491 | (effects (font (size 1.27 1.27)) (justify left)) 492 | ) 493 | (property "Value" "Conn_01x01_Socket" (at 92.71 19.685 0) 494 | (effects (font (size 1.27 1.27)) (justify left)) 495 | ) 496 | (property "Footprint" "Connector_Wire:SolderWire-0.5sqmm_1x01_D0.9mm_OD2.1mm" (at 91.44 17.78 0) 497 | (effects (font (size 1.27 1.27)) hide) 498 | ) 499 | (property "Datasheet" "~" (at 91.44 17.78 0) 500 | (effects (font (size 1.27 1.27)) hide) 501 | ) 502 | (pin "1" (uuid 02d3ee9d-3847-49ea-8822-ead7455e1e90)) 503 | (instances 504 | (project "dpoint" 505 | (path "/8aa3f855-a562-4b4c-aec0-abc662e54be3" 506 | (reference "J4") (unit 1) 507 | ) 508 | ) 509 | ) 510 | ) 511 | 512 | (symbol (lib_id "Connector:Conn_01x01_Socket") (at 91.44 33.02 0) (unit 1) 513 | (in_bom yes) (on_board yes) (dnp no) (fields_autoplaced) 514 | (uuid e506e65c-72ee-4e39-a179-b3bc7abc4e33) 515 | (property "Reference" "J3" (at 92.71 32.385 0) 516 | (effects (font (size 1.27 1.27)) (justify left)) 517 | ) 518 | (property "Value" "Conn_01x01_Socket" (at 92.71 34.925 0) 519 | (effects (font (size 1.27 1.27)) (justify left)) 520 | ) 521 | (property "Footprint" "Connector_Wire:SolderWire-0.5sqmm_1x01_D0.9mm_OD2.1mm" (at 91.44 33.02 0) 522 | (effects (font (size 1.27 1.27)) hide) 523 | ) 524 | (property "Datasheet" "~" (at 91.44 33.02 0) 525 | (effects (font (size 1.27 1.27)) hide) 526 | ) 527 | (pin "1" (uuid 44464904-cd1d-4c0f-9ce5-6800e1b0a3ee)) 528 | (instances 529 | (project "dpoint" 530 | (path "/8aa3f855-a562-4b4c-aec0-abc662e54be3" 531 | (reference "J3") (unit 1) 532 | ) 533 | ) 534 | ) 535 | ) 536 | 537 | (symbol (lib_id "symbol_library:HSFPAR003A") (at 58.42 48.26 0) (unit 1) 538 | (in_bom yes) (on_board yes) (dnp no) 539 | (uuid f1db5f07-0d1c-4db2-beca-fd46e77633e1) 540 | (property "Reference" "U1" (at 57.15 48.26 0) 541 | (effects (font (size 1.27 1.27)) (justify left)) 542 | ) 543 | (property "Value" "HSFPAR003A" (at 55.88 41.91 0) 544 | (effects (font (size 1.27 1.27))) 545 | ) 546 | (property "Footprint" "Footprints:Alps_HSFPAR003A" (at 58.42 48.26 0) 547 | (effects (font (size 1.27 1.27)) hide) 548 | ) 549 | (property "Datasheet" "" (at 58.42 48.26 0) 550 | (effects (font (size 1.27 1.27)) hide) 551 | ) 552 | (pin "1" (uuid de3c83d7-8e56-4beb-9715-518bbfe1c95b)) 553 | (pin "2" (uuid 16fef40a-2471-4132-ac9e-1a3349e06c66)) 554 | (pin "3" (uuid 24eefd23-eccd-4b15-bf89-69b89ca4f634)) 555 | (pin "4" (uuid 1a4ade05-a713-4302-928a-6483c9c0d7a4)) 556 | (instances 557 | (project "dpoint" 558 | (path "/8aa3f855-a562-4b4c-aec0-abc662e54be3" 559 | (reference "U1") (unit 1) 560 | ) 561 | ) 562 | ) 563 | ) 564 | 565 | (symbol (lib_id "symbol_library:XIAO_nRF52840_SENSE") (at 102.87 49.53 0) (unit 1) 566 | (in_bom yes) (on_board yes) (dnp no) 567 | (uuid f55432a2-e6f0-4511-bc9f-c7c5376ebc8f) 568 | (property "Reference" "A1" (at 104.14 52.07 0) 569 | (effects (font (size 1.27 1.27)) (justify right)) 570 | ) 571 | (property "Value" "XIAO_nRF52840_SENSE" (at 115.57 39.37 0) 572 | (effects (font (size 1.27 1.27)) (justify right)) 573 | ) 574 | (property "Footprint" "" (at 102.87 49.53 0) 575 | (effects (font (size 1.27 1.27)) hide) 576 | ) 577 | (property "Datasheet" "" (at 102.87 49.53 0) 578 | (effects (font (size 1.27 1.27)) hide) 579 | ) 580 | (pin "" (uuid df200d95-d686-4bee-a95d-fac73f0997db)) 581 | (pin "" (uuid df200d95-d686-4bee-a95d-fac73f0997db)) 582 | (pin "" (uuid df200d95-d686-4bee-a95d-fac73f0997db)) 583 | (pin "" (uuid df200d95-d686-4bee-a95d-fac73f0997db)) 584 | (pin "" (uuid df200d95-d686-4bee-a95d-fac73f0997db)) 585 | (pin "" (uuid df200d95-d686-4bee-a95d-fac73f0997db)) 586 | (instances 587 | (project "dpoint" 588 | (path "/8aa3f855-a562-4b4c-aec0-abc662e54be3" 589 | (reference "A1") (unit 1) 590 | ) 591 | ) 592 | ) 593 | ) 594 | 595 | (sheet_instances 596 | (path "/" (page "1")) 597 | ) 598 | ) 599 | -------------------------------------------------------------------------------- /electronics/gerbers/dpoint-B_Cu.gbl: -------------------------------------------------------------------------------- 1 | G04 #@! TF.GenerationSoftware,KiCad,Pcbnew,7.0.2* 2 | G04 #@! TF.CreationDate,2023-07-02T13:04:34+10:00* 3 | G04 #@! TF.ProjectId,dpoint,64706f69-6e74-42e6-9b69-6361645f7063,rev?* 4 | G04 #@! TF.SameCoordinates,Original* 5 | G04 #@! TF.FileFunction,Copper,L2,Bot* 6 | G04 #@! TF.FilePolarity,Positive* 7 | %FSLAX46Y46*% 8 | G04 Gerber Fmt 4.6, Leading zero omitted, Abs format (unit mm)* 9 | G04 Created by KiCad (PCBNEW 7.0.2) date 2023-07-02 13:04:34* 10 | %MOMM*% 11 | %LPD*% 12 | G01* 13 | G04 APERTURE LIST* 14 | G04 Aperture macros list* 15 | %AMRoundRect* 16 | 0 Rectangle with rounded corners* 17 | 0 $1 Rounding radius* 18 | 0 $2 $3 $4 $5 $6 $7 $8 $9 X,Y pos of 4 corners* 19 | 0 Add a 4 corners polygon primitive as box body* 20 | 4,1,4,$2,$3,$4,$5,$6,$7,$8,$9,$2,$3,0* 21 | 0 Add four circle primitives for the rounded corners* 22 | 1,1,$1+$1,$2,$3* 23 | 1,1,$1+$1,$4,$5* 24 | 1,1,$1+$1,$6,$7* 25 | 1,1,$1+$1,$8,$9* 26 | 0 Add four rect primitives between the rounded corners* 27 | 20,1,$1+$1,$2,$3,$4,$5,0* 28 | 20,1,$1+$1,$4,$5,$6,$7,0* 29 | 20,1,$1+$1,$6,$7,$8,$9,0* 30 | 20,1,$1+$1,$8,$9,$2,$3,0*% 31 | G04 Aperture macros list end* 32 | G04 #@! TA.AperFunction,SMDPad,CuDef* 33 | %ADD10RoundRect,0.250000X-0.574524X-0.097227X-0.097227X-0.574524X0.574524X0.097227X0.097227X0.574524X0*% 34 | G04 #@! TD* 35 | G04 #@! TA.AperFunction,ComponentPad* 36 | %ADD11RoundRect,1.050000X0.000000X0.000000X0.000000X0.000000X0.000000X0.000000X0.000000X0.000000X0*% 37 | G04 #@! TD* 38 | G04 #@! TA.AperFunction,Conductor* 39 | %ADD12C,0.250000*% 40 | G04 #@! TD* 41 | G04 APERTURE END LIST* 42 | D10* 43 | X63703877Y-23766377D03* 44 | X65171123Y-25233623D03* 45 | D11* 46 | X62250000Y-26750000D03* 47 | X66750000Y-22250000D03* 48 | X62250000Y-22250000D03* 49 | X66750000Y-26750000D03* 50 | D12* 51 | X66687500Y-26750000D02* 52 | X65171123Y-25233623D01* 53 | X66750000Y-26750000D02* 54 | X66687500Y-26750000D01* 55 | X62250000Y-22312500D02* 56 | X63703877Y-23766377D01* 57 | X62250000Y-22250000D02* 58 | X62250000Y-22312500D01* 59 | M02* 60 | -------------------------------------------------------------------------------- /electronics/gerbers/dpoint-B_Mask.gbs: -------------------------------------------------------------------------------- 1 | G04 #@! TF.GenerationSoftware,KiCad,Pcbnew,7.0.2* 2 | G04 #@! TF.CreationDate,2023-07-02T13:04:34+10:00* 3 | G04 #@! TF.ProjectId,dpoint,64706f69-6e74-42e6-9b69-6361645f7063,rev?* 4 | G04 #@! TF.SameCoordinates,Original* 5 | G04 #@! TF.FileFunction,Soldermask,Bot* 6 | G04 #@! TF.FilePolarity,Negative* 7 | %FSLAX46Y46*% 8 | G04 Gerber Fmt 4.6, Leading zero omitted, Abs format (unit mm)* 9 | G04 Created by KiCad (PCBNEW 7.0.2) date 2023-07-02 13:04:34* 10 | %MOMM*% 11 | %LPD*% 12 | G01* 13 | G04 APERTURE LIST* 14 | G04 Aperture macros list* 15 | %AMRoundRect* 16 | 0 Rectangle with rounded corners* 17 | 0 $1 Rounding radius* 18 | 0 $2 $3 $4 $5 $6 $7 $8 $9 X,Y pos of 4 corners* 19 | 0 Add a 4 corners polygon primitive as box body* 20 | 4,1,4,$2,$3,$4,$5,$6,$7,$8,$9,$2,$3,0* 21 | 0 Add four circle primitives for the rounded corners* 22 | 1,1,$1+$1,$2,$3* 23 | 1,1,$1+$1,$4,$5* 24 | 1,1,$1+$1,$6,$7* 25 | 1,1,$1+$1,$8,$9* 26 | 0 Add four rect primitives between the rounded corners* 27 | 20,1,$1+$1,$2,$3,$4,$5,0* 28 | 20,1,$1+$1,$4,$5,$6,$7,0* 29 | 20,1,$1+$1,$6,$7,$8,$9,0* 30 | 20,1,$1+$1,$8,$9,$2,$3,0*% 31 | G04 Aperture macros list end* 32 | %ADD10RoundRect,0.250000X-0.574524X-0.097227X-0.097227X-0.574524X0.574524X0.097227X0.097227X0.574524X0*% 33 | %ADD11RoundRect,1.050000X0.000000X0.000000X0.000000X0.000000X0.000000X0.000000X0.000000X0.000000X0*% 34 | G04 APERTURE END LIST* 35 | D10* 36 | X63703877Y-23766377D03* 37 | X65171123Y-25233623D03* 38 | D11* 39 | X62250000Y-26750000D03* 40 | X66750000Y-22250000D03* 41 | X62250000Y-22250000D03* 42 | X66750000Y-26750000D03* 43 | M02* 44 | -------------------------------------------------------------------------------- /electronics/gerbers/dpoint-B_Paste.gbp: -------------------------------------------------------------------------------- 1 | G04 #@! TF.GenerationSoftware,KiCad,Pcbnew,7.0.2* 2 | G04 #@! TF.CreationDate,2023-07-02T13:04:34+10:00* 3 | G04 #@! TF.ProjectId,dpoint,64706f69-6e74-42e6-9b69-6361645f7063,rev?* 4 | G04 #@! TF.SameCoordinates,Original* 5 | G04 #@! TF.FileFunction,Paste,Bot* 6 | G04 #@! TF.FilePolarity,Positive* 7 | %FSLAX46Y46*% 8 | G04 Gerber Fmt 4.6, Leading zero omitted, Abs format (unit mm)* 9 | G04 Created by KiCad (PCBNEW 7.0.2) date 2023-07-02 13:04:34* 10 | %MOMM*% 11 | %LPD*% 12 | G01* 13 | G04 APERTURE LIST* 14 | G04 Aperture macros list* 15 | %AMRoundRect* 16 | 0 Rectangle with rounded corners* 17 | 0 $1 Rounding radius* 18 | 0 $2 $3 $4 $5 $6 $7 $8 $9 X,Y pos of 4 corners* 19 | 0 Add a 4 corners polygon primitive as box body* 20 | 4,1,4,$2,$3,$4,$5,$6,$7,$8,$9,$2,$3,0* 21 | 0 Add four circle primitives for the rounded corners* 22 | 1,1,$1+$1,$2,$3* 23 | 1,1,$1+$1,$4,$5* 24 | 1,1,$1+$1,$6,$7* 25 | 1,1,$1+$1,$8,$9* 26 | 0 Add four rect primitives between the rounded corners* 27 | 20,1,$1+$1,$2,$3,$4,$5,0* 28 | 20,1,$1+$1,$4,$5,$6,$7,0* 29 | 20,1,$1+$1,$6,$7,$8,$9,0* 30 | 20,1,$1+$1,$8,$9,$2,$3,0*% 31 | G04 Aperture macros list end* 32 | %ADD10RoundRect,0.250000X-0.574524X-0.097227X-0.097227X-0.574524X0.574524X0.097227X0.097227X0.574524X0*% 33 | G04 APERTURE END LIST* 34 | D10* 35 | X63703877Y-23766377D03* 36 | X65171123Y-25233623D03* 37 | M02* 38 | -------------------------------------------------------------------------------- /electronics/gerbers/dpoint-B_Silkscreen.gbo: -------------------------------------------------------------------------------- 1 | G04 #@! TF.GenerationSoftware,KiCad,Pcbnew,7.0.2* 2 | G04 #@! TF.CreationDate,2023-07-02T13:04:34+10:00* 3 | G04 #@! TF.ProjectId,dpoint,64706f69-6e74-42e6-9b69-6361645f7063,rev?* 4 | G04 #@! TF.SameCoordinates,Original* 5 | G04 #@! TF.FileFunction,Legend,Bot* 6 | G04 #@! TF.FilePolarity,Positive* 7 | %FSLAX46Y46*% 8 | G04 Gerber Fmt 4.6, Leading zero omitted, Abs format (unit mm)* 9 | G04 Created by KiCad (PCBNEW 7.0.2) date 2023-07-02 13:04:34* 10 | %MOMM*% 11 | %LPD*% 12 | G01* 13 | G04 APERTURE LIST* 14 | %ADD10C,0.120000*% 15 | G04 APERTURE END LIST* 16 | D10* 17 | X63733043Y-24834990D02* 18 | X64102510Y-25204457D01* 19 | X64772490Y-23795543D02* 20 | X65141957Y-24165010D01* 21 | M02* 22 | -------------------------------------------------------------------------------- /electronics/gerbers/dpoint-Edge_Cuts.gm1: -------------------------------------------------------------------------------- 1 | G04 #@! TF.GenerationSoftware,KiCad,Pcbnew,7.0.2* 2 | G04 #@! TF.CreationDate,2023-07-02T13:04:34+10:00* 3 | G04 #@! TF.ProjectId,dpoint,64706f69-6e74-42e6-9b69-6361645f7063,rev?* 4 | G04 #@! TF.SameCoordinates,Original* 5 | G04 #@! TF.FileFunction,Profile,NP* 6 | %FSLAX46Y46*% 7 | G04 Gerber Fmt 4.6, Leading zero omitted, Abs format (unit mm)* 8 | G04 Created by KiCad (PCBNEW 7.0.2) date 2023-07-02 13:04:34* 9 | %MOMM*% 10 | %LPD*% 11 | G01* 12 | G04 APERTURE LIST* 13 | G04 #@! TA.AperFunction,Profile* 14 | %ADD10C,0.100000*% 15 | G04 #@! TD* 16 | G04 APERTURE END LIST* 17 | D10* 18 | X69000000Y-27000000D02* 19 | X67000000Y-29000000D01* 20 | X69000000Y-22000000D02* 21 | X69000000Y-27000000D01* 22 | X60000000Y-27000000D02* 23 | X60000000Y-22000000D01* 24 | X62000000Y-29000000D02* 25 | X60000000Y-27000000D01* 26 | X62000000Y-20000000D02* 27 | X67000000Y-20000000D01* 28 | X60000000Y-22000000D02* 29 | X62000000Y-20000000D01* 30 | X67000000Y-20000000D02* 31 | X69000000Y-22000000D01* 32 | X67000000Y-29000000D02* 33 | X62000000Y-29000000D01* 34 | M02* 35 | -------------------------------------------------------------------------------- /electronics/gerbers/dpoint-F_Cu.gtl: -------------------------------------------------------------------------------- 1 | G04 #@! TF.GenerationSoftware,KiCad,Pcbnew,7.0.2* 2 | G04 #@! TF.CreationDate,2023-07-02T13:04:34+10:00* 3 | G04 #@! TF.ProjectId,dpoint,64706f69-6e74-42e6-9b69-6361645f7063,rev?* 4 | G04 #@! TF.SameCoordinates,Original* 5 | G04 #@! TF.FileFunction,Copper,L1,Top* 6 | G04 #@! TF.FilePolarity,Positive* 7 | %FSLAX46Y46*% 8 | G04 Gerber Fmt 4.6, Leading zero omitted, Abs format (unit mm)* 9 | G04 Created by KiCad (PCBNEW 7.0.2) date 2023-07-02 13:04:34* 10 | %MOMM*% 11 | %LPD*% 12 | G01* 13 | G04 APERTURE LIST* 14 | G04 Aperture macros list* 15 | %AMRoundRect* 16 | 0 Rectangle with rounded corners* 17 | 0 $1 Rounding radius* 18 | 0 $2 $3 $4 $5 $6 $7 $8 $9 X,Y pos of 4 corners* 19 | 0 Add a 4 corners polygon primitive as box body* 20 | 4,1,4,$2,$3,$4,$5,$6,$7,$8,$9,$2,$3,0* 21 | 0 Add four circle primitives for the rounded corners* 22 | 1,1,$1+$1,$2,$3* 23 | 1,1,$1+$1,$4,$5* 24 | 1,1,$1+$1,$6,$7* 25 | 1,1,$1+$1,$8,$9* 26 | 0 Add four rect primitives between the rounded corners* 27 | 20,1,$1+$1,$2,$3,$4,$5,0* 28 | 20,1,$1+$1,$4,$5,$6,$7,0* 29 | 20,1,$1+$1,$6,$7,$8,$9,0* 30 | 20,1,$1+$1,$8,$9,$2,$3,0*% 31 | G04 Aperture macros list end* 32 | G04 #@! TA.AperFunction,SMDPad,CuDef* 33 | %ADD10RoundRect,0.075000X-0.075000X-0.175000X0.075000X-0.175000X0.075000X0.175000X-0.075000X0.175000X0*% 34 | G04 #@! TD* 35 | G04 #@! TA.AperFunction,SMDPad,CuDef* 36 | %ADD11RoundRect,0.150000X-0.150000X-0.150000X0.150000X-0.150000X0.150000X0.150000X-0.150000X0.150000X0*% 37 | G04 #@! TD* 38 | G04 #@! TA.AperFunction,ComponentPad* 39 | %ADD12RoundRect,1.050000X0.000000X0.000000X0.000000X0.000000X0.000000X0.000000X0.000000X0.000000X0*% 40 | G04 #@! TD* 41 | G04 #@! TA.AperFunction,Conductor* 42 | %ADD13C,0.250000*% 43 | G04 #@! TD* 44 | G04 APERTURE END LIST* 45 | D10* 46 | X63900000Y-23800000D03* 47 | X65100000Y-23800000D03* 48 | D11* 49 | X64500000Y-24500000D03* 50 | D10* 51 | X65100000Y-25200000D03* 52 | X63900000Y-25200000D03* 53 | D12* 54 | X62250000Y-26750000D03* 55 | X66750000Y-22250000D03* 56 | X62250000Y-22250000D03* 57 | X66750000Y-26750000D03* 58 | D13* 59 | X63800000Y-23800000D02* 60 | X62250000Y-22250000D01* 61 | X63900000Y-23800000D02* 62 | X63800000Y-23800000D01* 63 | X65200000Y-23800000D02* 64 | X66750000Y-22250000D01* 65 | X65100000Y-23800000D02* 66 | X65200000Y-23800000D01* 67 | X63800000Y-25200000D02* 68 | X62250000Y-26750000D01* 69 | X63900000Y-25200000D02* 70 | X63800000Y-25200000D01* 71 | X65200000Y-25200000D02* 72 | X66750000Y-26750000D01* 73 | X65100000Y-25200000D02* 74 | X65200000Y-25200000D01* 75 | X65100000Y-25100000D02* 76 | X64500000Y-24500000D01* 77 | X65100000Y-25200000D02* 78 | X65100000Y-25100000D01* 79 | M02* 80 | -------------------------------------------------------------------------------- /electronics/gerbers/dpoint-F_Mask.gts: -------------------------------------------------------------------------------- 1 | G04 #@! TF.GenerationSoftware,KiCad,Pcbnew,7.0.2* 2 | G04 #@! TF.CreationDate,2023-07-02T13:04:34+10:00* 3 | G04 #@! TF.ProjectId,dpoint,64706f69-6e74-42e6-9b69-6361645f7063,rev?* 4 | G04 #@! TF.SameCoordinates,Original* 5 | G04 #@! TF.FileFunction,Soldermask,Top* 6 | G04 #@! TF.FilePolarity,Negative* 7 | %FSLAX46Y46*% 8 | G04 Gerber Fmt 4.6, Leading zero omitted, Abs format (unit mm)* 9 | G04 Created by KiCad (PCBNEW 7.0.2) date 2023-07-02 13:04:34* 10 | %MOMM*% 11 | %LPD*% 12 | G01* 13 | G04 APERTURE LIST* 14 | G04 Aperture macros list* 15 | %AMRoundRect* 16 | 0 Rectangle with rounded corners* 17 | 0 $1 Rounding radius* 18 | 0 $2 $3 $4 $5 $6 $7 $8 $9 X,Y pos of 4 corners* 19 | 0 Add a 4 corners polygon primitive as box body* 20 | 4,1,4,$2,$3,$4,$5,$6,$7,$8,$9,$2,$3,0* 21 | 0 Add four circle primitives for the rounded corners* 22 | 1,1,$1+$1,$2,$3* 23 | 1,1,$1+$1,$4,$5* 24 | 1,1,$1+$1,$6,$7* 25 | 1,1,$1+$1,$8,$9* 26 | 0 Add four rect primitives between the rounded corners* 27 | 20,1,$1+$1,$2,$3,$4,$5,0* 28 | 20,1,$1+$1,$4,$5,$6,$7,0* 29 | 20,1,$1+$1,$6,$7,$8,$9,0* 30 | 20,1,$1+$1,$8,$9,$2,$3,0*% 31 | G04 Aperture macros list end* 32 | %ADD10RoundRect,0.075000X-0.075000X-0.175000X0.075000X-0.175000X0.075000X0.175000X-0.075000X0.175000X0*% 33 | %ADD11RoundRect,0.150000X-0.150000X-0.150000X0.150000X-0.150000X0.150000X0.150000X-0.150000X0.150000X0*% 34 | %ADD12RoundRect,1.050000X0.000000X0.000000X0.000000X0.000000X0.000000X0.000000X0.000000X0.000000X0*% 35 | G04 APERTURE END LIST* 36 | D10* 37 | X63900000Y-23800000D03* 38 | X65100000Y-23800000D03* 39 | D11* 40 | X64500000Y-24500000D03* 41 | D10* 42 | X65100000Y-25200000D03* 43 | X63900000Y-25200000D03* 44 | D12* 45 | X62250000Y-26750000D03* 46 | X66750000Y-22250000D03* 47 | X62250000Y-22250000D03* 48 | X66750000Y-26750000D03* 49 | M02* 50 | -------------------------------------------------------------------------------- /electronics/gerbers/dpoint-F_Paste.gtp: -------------------------------------------------------------------------------- 1 | G04 #@! TF.GenerationSoftware,KiCad,Pcbnew,7.0.2* 2 | G04 #@! TF.CreationDate,2023-07-02T13:04:34+10:00* 3 | G04 #@! TF.ProjectId,dpoint,64706f69-6e74-42e6-9b69-6361645f7063,rev?* 4 | G04 #@! TF.SameCoordinates,Original* 5 | G04 #@! TF.FileFunction,Paste,Top* 6 | G04 #@! TF.FilePolarity,Positive* 7 | %FSLAX46Y46*% 8 | G04 Gerber Fmt 4.6, Leading zero omitted, Abs format (unit mm)* 9 | G04 Created by KiCad (PCBNEW 7.0.2) date 2023-07-02 13:04:34* 10 | %MOMM*% 11 | %LPD*% 12 | G01* 13 | G04 APERTURE LIST* 14 | G04 Aperture macros list* 15 | %AMRoundRect* 16 | 0 Rectangle with rounded corners* 17 | 0 $1 Rounding radius* 18 | 0 $2 $3 $4 $5 $6 $7 $8 $9 X,Y pos of 4 corners* 19 | 0 Add a 4 corners polygon primitive as box body* 20 | 4,1,4,$2,$3,$4,$5,$6,$7,$8,$9,$2,$3,0* 21 | 0 Add four circle primitives for the rounded corners* 22 | 1,1,$1+$1,$2,$3* 23 | 1,1,$1+$1,$4,$5* 24 | 1,1,$1+$1,$6,$7* 25 | 1,1,$1+$1,$8,$9* 26 | 0 Add four rect primitives between the rounded corners* 27 | 20,1,$1+$1,$2,$3,$4,$5,0* 28 | 20,1,$1+$1,$4,$5,$6,$7,0* 29 | 20,1,$1+$1,$6,$7,$8,$9,0* 30 | 20,1,$1+$1,$8,$9,$2,$3,0*% 31 | G04 Aperture macros list end* 32 | %ADD10RoundRect,0.075000X-0.075000X-0.175000X0.075000X-0.175000X0.075000X0.175000X-0.075000X0.175000X0*% 33 | %ADD11RoundRect,0.150000X-0.150000X-0.150000X0.150000X-0.150000X0.150000X0.150000X-0.150000X0.150000X0*% 34 | G04 APERTURE END LIST* 35 | D10* 36 | X63900000Y-23800000D03* 37 | X65100000Y-23800000D03* 38 | D11* 39 | X64500000Y-24500000D03* 40 | D10* 41 | X65100000Y-25200000D03* 42 | X63900000Y-25200000D03* 43 | M02* 44 | -------------------------------------------------------------------------------- /electronics/gerbers/dpoint-F_Silkscreen.gto: -------------------------------------------------------------------------------- 1 | G04 #@! TF.GenerationSoftware,KiCad,Pcbnew,7.0.2* 2 | G04 #@! TF.CreationDate,2023-07-02T13:04:34+10:00* 3 | G04 #@! TF.ProjectId,dpoint,64706f69-6e74-42e6-9b69-6361645f7063,rev?* 4 | G04 #@! TF.SameCoordinates,Original* 5 | G04 #@! TF.FileFunction,Legend,Top* 6 | G04 #@! TF.FilePolarity,Positive* 7 | %FSLAX46Y46*% 8 | G04 Gerber Fmt 4.6, Leading zero omitted, Abs format (unit mm)* 9 | G04 Created by KiCad (PCBNEW 7.0.2) date 2023-07-02 13:04:34* 10 | %MOMM*% 11 | %LPD*% 12 | G01* 13 | G04 APERTURE LIST* 14 | %ADD10C,0.150000*% 15 | G04 APERTURE END LIST* 16 | D10* 17 | X60591851Y-23470834D02* 18 | X60858518Y-24270834D01* 19 | X60858518Y-24270834D02* 20 | X61125184Y-23470834D01* 21 | X61734708Y-24232739D02* 22 | X61658517Y-24270834D01* 23 | X61658517Y-24270834D02* 24 | X61506136Y-24270834D01* 25 | X61506136Y-24270834D02* 26 | X61429946Y-24232739D01* 27 | X61429946Y-24232739D02* 28 | X61391851Y-24194643D01* 29 | X61391851Y-24194643D02* 30 | X61353755Y-24118453D01* 31 | X61353755Y-24118453D02* 32 | X61353755Y-23889881D01* 33 | X61353755Y-23889881D02* 34 | X61391851Y-23813691D01* 35 | X61391851Y-23813691D02* 36 | X61429946Y-23775596D01* 37 | X61429946Y-23775596D02* 38 | X61506136Y-23737500D01* 39 | X61506136Y-23737500D02* 40 | X61658517Y-23737500D01* 41 | X61658517Y-23737500D02* 42 | X61734708Y-23775596D01* 43 | X62420422Y-24232739D02* 44 | X62344231Y-24270834D01* 45 | X62344231Y-24270834D02* 46 | X62191850Y-24270834D01* 47 | X62191850Y-24270834D02* 48 | X62115660Y-24232739D01* 49 | X62115660Y-24232739D02* 50 | X62077565Y-24194643D01* 51 | X62077565Y-24194643D02* 52 | X62039469Y-24118453D01* 53 | X62039469Y-24118453D02* 54 | X62039469Y-23889881D01* 55 | X62039469Y-23889881D02* 56 | X62077565Y-23813691D01* 57 | X62077565Y-23813691D02* 58 | X62115660Y-23775596D01* 59 | X62115660Y-23775596D02* 60 | X62191850Y-23737500D01* 61 | X62191850Y-23737500D02* 62 | X62344231Y-23737500D01* 63 | X62344231Y-23737500D02* 64 | X62420422Y-23775596D01* 65 | X66883166Y-24480295D02* 66 | X66806976Y-24442200D01* 67 | X66806976Y-24442200D02* 68 | X66692690Y-24442200D01* 69 | X66692690Y-24442200D02* 70 | X66578404Y-24480295D01* 71 | X66578404Y-24480295D02* 72 | X66502214Y-24556485D01* 73 | X66502214Y-24556485D02* 74 | X66464119Y-24632676D01* 75 | X66464119Y-24632676D02* 76 | X66426023Y-24785057D01* 77 | X66426023Y-24785057D02* 78 | X66426023Y-24899343D01* 79 | X66426023Y-24899343D02* 80 | X66464119Y-25051724D01* 81 | X66464119Y-25051724D02* 82 | X66502214Y-25127914D01* 83 | X66502214Y-25127914D02* 84 | X66578404Y-25204105D01* 85 | X66578404Y-25204105D02* 86 | X66692690Y-25242200D01* 87 | X66692690Y-25242200D02* 88 | X66768881Y-25242200D01* 89 | X66768881Y-25242200D02* 90 | X66883166Y-25204105D01* 91 | X66883166Y-25204105D02* 92 | X66921262Y-25166009D01* 93 | X66921262Y-25166009D02* 94 | X66921262Y-24899343D01* 95 | X66921262Y-24899343D02* 96 | X66768881Y-24899343D01* 97 | X67264119Y-25242200D02* 98 | X67264119Y-24442200D01* 99 | X67264119Y-24442200D02* 100 | X67721262Y-25242200D01* 101 | X67721262Y-25242200D02* 102 | X67721262Y-24442200D01* 103 | X68102214Y-25242200D02* 104 | X68102214Y-24442200D01* 105 | X68102214Y-24442200D02* 106 | X68292690Y-24442200D01* 107 | X68292690Y-24442200D02* 108 | X68406976Y-24480295D01* 109 | X68406976Y-24480295D02* 110 | X68483166Y-24556485D01* 111 | X68483166Y-24556485D02* 112 | X68521261Y-24632676D01* 113 | X68521261Y-24632676D02* 114 | X68559357Y-24785057D01* 115 | X68559357Y-24785057D02* 116 | X68559357Y-24899343D01* 117 | X68559357Y-24899343D02* 118 | X68521261Y-25051724D01* 119 | X68521261Y-25051724D02* 120 | X68483166Y-25127914D01* 121 | X68483166Y-25127914D02* 122 | X68406976Y-25204105D01* 123 | X68406976Y-25204105D02* 124 | X68292690Y-25242200D01* 125 | X68292690Y-25242200D02* 126 | X68102214Y-25242200D01* 127 | X63260184Y-27871242D02* 128 | X63526851Y-28671242D01* 129 | X63526851Y-28671242D02* 130 | X63793517Y-27871242D01* 131 | X64060184Y-28366480D02* 132 | X64669708Y-28366480D01* 133 | X64364946Y-28671242D02* 134 | X64364946Y-28061718D01* 135 | X64652866Y-20392888D02* 136 | X64919533Y-21192888D01* 137 | X64919533Y-21192888D02* 138 | X65186199Y-20392888D01* 139 | X65452866Y-20888126D02* 140 | X66062390Y-20888126D01* 141 | X63498345Y-20647117D02* 142 | G75* 143 | G03* 144 | X63498345Y-20647117I-232428J0D01* 145 | G01* 146 | M02* 147 | -------------------------------------------------------------------------------- /electronics/gerbers/dpoint.drl: -------------------------------------------------------------------------------- 1 | M48 2 | ; DRILL file {KiCad 7.0.2} date Sun Jul 2 13:05:47 2023 3 | ; FORMAT={-:-/ absolute / inch / decimal} 4 | ; #@! TF.CreationDate,2023-07-02T13:05:47+10:00 5 | ; #@! TF.GenerationSoftware,Kicad,Pcbnew,7.0.2 6 | ; #@! TF.FileFunction,MixedPlating,1,2 7 | FMAT,2 8 | INCH 9 | ; #@! TA.AperFunction,Plated,PTH,ComponentDrill 10 | T1C0.0433 11 | % 12 | G90 13 | G05 14 | T1 15 | X2.4508Y-0.876 16 | X2.4508Y-1.0531 17 | X2.628Y-0.876 18 | X2.628Y-1.0531 19 | T0 20 | M30 21 | -------------------------------------------------------------------------------- /electronics/sym-lib-table: -------------------------------------------------------------------------------- 1 | (sym_lib_table 2 | (version 7) 3 | (lib (name "symbol_library")(type "KiCad")(uri "${KIPRJMOD}/symbol_library.kicad_sym")(options "")(descr "")) 4 | ) 5 | -------------------------------------------------------------------------------- /electronics/symbol_library.kicad_sym: -------------------------------------------------------------------------------- 1 | (kicad_symbol_lib (version 20220914) (generator kicad_symbol_editor) 2 | (symbol "HSFPAR003A" (pin_names (offset 0.002)) (in_bom yes) (on_board yes) 3 | (property "Reference" "U" (at 0 0 0) 4 | (effects (font (size 1.27 1.27))) 5 | ) 6 | (property "Value" "HSFPAR003A" (at 0 6.35 0) 7 | (effects (font (size 1.27 1.27))) 8 | ) 9 | (property "Footprint" "" (at 0 0 0) 10 | (effects (font (size 1.27 1.27)) hide) 11 | ) 12 | (property "Datasheet" "" (at 0 0 0) 13 | (effects (font (size 1.27 1.27)) hide) 14 | ) 15 | (symbol "HSFPAR003A_0_1" 16 | (rectangle (start -3.81 5.08) (end 3.81 -5.08) 17 | (stroke (width 0) (type default)) 18 | (fill (type none)) 19 | ) 20 | ) 21 | (symbol "HSFPAR003A_1_1" 22 | (pin input line (at 6.35 -3.81 180) (length 2.54) 23 | (name "GND" (effects (font (size 1.27 1.27)))) 24 | (number "" (effects (font (size 1.27 1.27)))) 25 | ) 26 | (pin output line (at 6.35 1.27 180) (length 2.54) 27 | (name "V1" (effects (font (size 1.27 1.27)))) 28 | (number "" (effects (font (size 1.27 1.27)))) 29 | ) 30 | (pin output line (at 6.35 -1.27 180) (length 2.54) 31 | (name "V2" (effects (font (size 1.27 1.27)))) 32 | (number "" (effects (font (size 1.27 1.27)))) 33 | ) 34 | (pin input line (at 6.35 3.81 180) (length 2.54) 35 | (name "Vdd" (effects (font (size 1.27 1.27)))) 36 | (number "" (effects (font (size 1.27 1.27)))) 37 | ) 38 | ) 39 | ) 40 | (symbol "XIAO_nRF52840_SENSE" (pin_names (offset 0)) (in_bom yes) (on_board yes) 41 | (property "Reference" "A" (at 5.08 7.62 0) 42 | (effects (font (size 1.27 1.27)) (justify right)) 43 | ) 44 | (property "Value" "XIAO_nRF52840_SENSE" (at 2.54 10.16 0) 45 | (effects (font (size 1.27 1.27))) 46 | ) 47 | (property "Footprint" "" (at 0 0 0) 48 | (effects (font (size 1.27 1.27)) hide) 49 | ) 50 | (property "Datasheet" "" (at 0 0 0) 51 | (effects (font (size 1.27 1.27)) hide) 52 | ) 53 | (symbol "XIAO_nRF52840_SENSE_0_1" 54 | (rectangle (start -7.62 8.89) (end 6.35 -12.7) 55 | (stroke (width 0) (type default)) 56 | (fill (type none)) 57 | ) 58 | ) 59 | (symbol "XIAO_nRF52840_SENSE_1_1" 60 | (pin output line (at -7.62 6.35 180) (length 2.54) 61 | (name "3V3" (effects (font (size 1.27 1.27)))) 62 | (number "" (effects (font (size 1.27 1.27)))) 63 | ) 64 | (pin input line (at -7.62 2.54 180) (length 2.54) 65 | (name "AIN0" (effects (font (size 1.27 1.27)))) 66 | (number "" (effects (font (size 1.27 1.27)))) 67 | ) 68 | (pin input line (at -7.62 0 180) (length 2.54) 69 | (name "AIN1" (effects (font (size 1.27 1.27)))) 70 | (number "" (effects (font (size 1.27 1.27)))) 71 | ) 72 | (pin bidirectional line (at -5.08 -12.7 270) (length 2.54) 73 | (name "BAT+" (effects (font (size 1.27 1.27)))) 74 | (number "" (effects (font (size 1.27 1.27)))) 75 | ) 76 | (pin bidirectional line (at -1.27 -12.7 270) (length 2.54) 77 | (name "BAT-" (effects (font (size 1.27 1.27)))) 78 | (number "" (effects (font (size 1.27 1.27)))) 79 | ) 80 | (pin output line (at -7.62 -6.35 180) (length 2.54) 81 | (name "GND" (effects (font (size 1.27 1.27)))) 82 | (number "" (effects (font (size 1.27 1.27)))) 83 | ) 84 | ) 85 | ) 86 | ) 87 | -------------------------------------------------------------------------------- /markers/calib.io_charuco_297x210_8x12_24_18_DICT_4X4.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Jcparkyn/dpoint/a108c19b9b240c1531b2d30ed47ba18bc604862e/markers/calib.io_charuco_297x210_8x12_24_18_DICT_4X4.pdf -------------------------------------------------------------------------------- /markers/stylus-markers.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Jcparkyn/dpoint/a108c19b9b240c1531b2d30ed47ba18bc604862e/markers/stylus-markers.pdf -------------------------------------------------------------------------------- /microcontroller/dpoint-arduino/dpoint-arduino.ino: -------------------------------------------------------------------------------- 1 | #include 2 | #include 3 | #include 4 | #include 5 | #include 6 | #include 7 | #include 8 | 9 | #define LEDR LED_RED 10 | #define LEDG LED_GREEN 11 | #define LEDB LED_BLUE 12 | 13 | #define PRESSURE_SENSOR_VCC_PIN D1 14 | #define PRESSURE_SENSOR_SAADC_CHANNEL 0 15 | 16 | const unsigned long delayMs = 7; 17 | const unsigned long rate = 1000/delayMs; 18 | const unsigned long stayAwakeTimeMs = 1000*60; 19 | 20 | struct IMUDataPacket { 21 | int16_t accel[3]; 22 | int16_t gyro[3]; 23 | uint16_t pressure; 24 | }; 25 | 26 | const uint8_t LBS_UUID_SERVICE[] = 27 | { 28 | 0x19, 0xB1, 0x00, 0x10, 0xE8, 0xF2, 0x53, 0x7E, 0x4F, 0x6C, 0xD1, 0x04, 0x76, 0x8A, 0x12, 0x14 29 | }; 30 | 31 | // 19B10013-E8F2-537E-4F6C-D104768A1214 32 | const uint8_t LBS_UUID_CHR_IMU[] = 33 | { 34 | 0x14, 0x12, 0x8A, 0x76, 0x04, 0xD1, 0x6C, 0x4F, 0x7E, 0x53, 0xF2, 0xE8, 0x13, 0x00, 0xB1, 0x19 35 | }; 36 | 37 | BLEService bleService(LBS_UUID_SERVICE); 38 | BLECharacteristic imuCharacteristic(LBS_UUID_CHR_IMU); 39 | 40 | Adafruit_FlashTransport_QSPI flashTransport; 41 | LSM6DS3 imu(I2C_MODE, 0x6A); //I2C device address 0x6A 42 | 43 | const nrf_saadc_channel_config_t adcConfig = { 44 | .resistor_p = NRF_SAADC_RESISTOR_DISABLED, 45 | .resistor_n = NRF_SAADC_RESISTOR_DISABLED, 46 | .gain = NRF_SAADC_GAIN4, 47 | .reference = NRF_SAADC_REFERENCE_VDD4, 48 | .acq_time = NRF_SAADC_ACQTIME_20US, 49 | .mode = NRF_SAADC_MODE_DIFFERENTIAL, 50 | .burst = NRF_SAADC_BURST_ENABLED, 51 | }; 52 | 53 | // Disable QSPI flash to save power 54 | void QSPIF_sleep(void) 55 | { 56 | flashTransport.begin(); 57 | flashTransport.runCommand(0xB9); 58 | flashTransport.end(); 59 | } 60 | 61 | void imuISR() { 62 | // Interrupt triggers for both single and double taps, so we need to check which one it was. 63 | uint8_t tapSrc; 64 | status_t status = imu.readRegister(&tapSrc, LSM6DS3_ACC_GYRO_TAP_SRC); 65 | bool wasDoubleTap = (tapSrc & LSM6DS3_ACC_GYRO_DOUBLE_TAP_EV_STATUS_DETECTED) > 0; 66 | if (!wasDoubleTap) { 67 | nrf_power_system_off(NRF_POWER); 68 | } 69 | } 70 | 71 | void setupWakeUpInterrupt() 72 | { 73 | // Tap interrupt code is based on code by daCoder 74 | // https://forum.seeedstudio.com/t/xiao-sense-accelerometer-examples-and-low-power/270801 75 | imu.settings.gyroEnabled = 0; 76 | imu.settings.accelEnabled = 1; 77 | imu.settings.accelSampleRate = 104; 78 | imu.settings.accelRange = 2; 79 | imu.begin(); 80 | 81 | //https://www.st.com/resource/en/datasheet/lsm6ds3tr-c.pdf 82 | imu.writeRegister(LSM6DS3_ACC_GYRO_TAP_CFG1, 0b10001000); // Enable interrupts and tap detection on X-axis 83 | imu.writeRegister(LSM6DS3_ACC_GYRO_TAP_THS_6D, 0b10001000); // Set tap threshold 84 | const int duration = 0b0010 << 4; // 1LSB corresponds to 32*ODR_XL time 85 | const int quietTime = 0b10 << 2; // 1LSB corresponds to 4*ODR_XL time 86 | const int shockTime = 0b01 << 0; // 1LSB corresponds to 8*ODR_XL time 87 | imu.writeRegister(LSM6DS3_ACC_GYRO_INT_DUR2, duration | quietTime | shockTime); // Set Duration, Quiet and Shock time windows 88 | imu.writeRegister(LSM6DS3_ACC_GYRO_WAKE_UP_THS, 0x80); // Single & double-tap enabled (SINGLE_DOUBLE_TAP = 1) 89 | imu.writeRegister(LSM6DS3_ACC_GYRO_MD1_CFG, 0x08); // Double-tap interrupt driven to INT1 pin 90 | imu.writeRegister(LSM6DS3_ACC_GYRO_CTRL6_G, 0x10); // High-performance operating mode disabled for accelerometer 91 | 92 | // Set up the sense mechanism to generate the DETECT signal to wake from system_off 93 | pinMode(PIN_LSM6DS3TR_C_INT1, INPUT_PULLDOWN_SENSE); 94 | attachInterrupt(digitalPinToInterrupt(PIN_LSM6DS3TR_C_INT1), imuISR, CHANGE); 95 | 96 | return; 97 | } 98 | 99 | void setupPressureSensor() { 100 | pinMode(PRESSURE_SENSOR_VCC_PIN, OUTPUT); 101 | digitalWrite(PRESSURE_SENSOR_VCC_PIN, HIGH); 102 | 103 | nrf_saadc_enable(NRF_SAADC); 104 | 105 | nrf_saadc_oversample_set(NRF_SAADC, NRF_SAADC_OVERSAMPLE_32X); 106 | nrf_saadc_resolution_set(NRF_SAADC, NRF_SAADC_RESOLUTION_12BIT); 107 | nrf_saadc_channel_input_set(NRF_SAADC, PRESSURE_SENSOR_SAADC_CHANNEL, NRF_SAADC_INPUT_AIN2, NRF_SAADC_INPUT_AIN4); 108 | nrf_saadc_channel_init(NRF_SAADC, PRESSURE_SENSOR_SAADC_CHANNEL, &adcConfig); 109 | 110 | NRF_SAADC->TASKS_CALIBRATEOFFSET = 1; 111 | while (NRF_SAADC->EVENTS_CALIBRATEDONE == 0); 112 | NRF_SAADC->EVENTS_CALIBRATEDONE = 0; 113 | while (NRF_SAADC->STATUS == (SAADC_STATUS_STATUS_Busy << SAADC_STATUS_STATUS_Pos)); 114 | 115 | nrf_saadc_disable(NRF_SAADC); 116 | } 117 | 118 | int16_t readPressure() { 119 | volatile nrf_saadc_value_t result = -1; 120 | 121 | nrf_saadc_enable(NRF_SAADC); 122 | nrf_saadc_continuous_mode_disable(NRF_SAADC); 123 | 124 | NRF_SAADC->RESULT.MAXCNT = 1; 125 | NRF_SAADC->RESULT.PTR = (uint32_t)&result; 126 | 127 | // Start the SAADC and wait for the started event. 128 | NRF_SAADC->TASKS_START = 1; 129 | while (NRF_SAADC->EVENTS_STARTED == 0); 130 | NRF_SAADC->EVENTS_STARTED = 0; 131 | 132 | // Do a SAADC sample, will put the result in the configured RAM buffer. 133 | NRF_SAADC->TASKS_SAMPLE = 1; 134 | while (NRF_SAADC->EVENTS_END == 0); 135 | NRF_SAADC->EVENTS_END = 0; 136 | 137 | NRF_SAADC->TASKS_STOP = 1; 138 | while (NRF_SAADC->EVENTS_STOPPED == 0); 139 | NRF_SAADC->EVENTS_STOPPED = 0; 140 | 141 | nrf_saadc_disable(NRF_SAADC); 142 | 143 | // Scaling to match previous mbed implementation 144 | return int16_t((uint32_t(result) * 0xFFFF) / 0x0FFF); 145 | } 146 | 147 | void setupImu() { 148 | imu.settings.accelRange = 4; // Can be: 2, 4, 8, 16 149 | imu.settings.gyroRange = 500; // Can be: 125, 245, 500, 1000, 2000 150 | imu.settings.accelSampleRate = 416; //Hz. Can be: 13, 26, 52, 104, 208, 416, 833, 1666, 3332, 6664, 13330 151 | imu.settings.gyroSampleRate = 416; //Hz. Can be: 13, 26, 52, 104, 208, 416, 833, 1666 152 | imu.settings.accelBandWidth = 200; 153 | imu.settings.gyroBandWidth = 200; 154 | imu.begin(); 155 | } 156 | 157 | void runBle() { 158 | while (Bluefruit.connected(0)) { 159 | unsigned long startTime = millis(); 160 | 161 | IMUDataPacket packet; 162 | // This could be optimised by reading all values as a block. 163 | packet.accel[0] = imu.readRawAccelX(); 164 | packet.accel[1] = imu.readRawAccelY(); 165 | packet.accel[2] = imu.readRawAccelZ(); 166 | packet.gyro[0] = imu.readRawGyroX(); 167 | packet.gyro[1] = imu.readRawGyroY(); 168 | packet.gyro[2] = imu.readRawGyroZ(); 169 | 170 | packet.pressure = readPressure(); 171 | 172 | imuCharacteristic.notify(&packet, sizeof(packet)); 173 | 174 | // Inaccurate but usable way to throttle the rate of measurements. 175 | unsigned long time = millis() - startTime; 176 | unsigned long waitPeriod = delayMs - time; 177 | if (waitPeriod > 0 && waitPeriod < 500) { // protection against overflow issues 178 | delay(waitPeriod); 179 | } 180 | } 181 | } 182 | 183 | void startAdvertising() 184 | { 185 | Bluefruit.Advertising.addFlags(BLE_GAP_ADV_FLAGS_LE_ONLY_GENERAL_DISC_MODE); 186 | Bluefruit.Advertising.addTxPower(); 187 | Bluefruit.Advertising.addService(bleService); 188 | 189 | Bluefruit.ScanResponse.addName(); 190 | 191 | Bluefruit.Advertising.restartOnDisconnect(true); 192 | Bluefruit.Advertising.setInterval(128, 488); // in unit of 0.625 ms 193 | Bluefruit.Advertising.setFastTimeout(30); // number of seconds in fast mode 194 | Bluefruit.Advertising.start(0); // 0 = Don't stop advertising after n seconds 195 | } 196 | 197 | void sleepUntilDoubleTap() { 198 | digitalWrite(LEDR, HIGH); 199 | digitalWrite(LEDG, HIGH); 200 | digitalWrite(LEDB, HIGH); 201 | 202 | Serial.println("Setting up interrupt"); 203 | // Setup up double tap interrupt to wake back up 204 | setupWakeUpInterrupt(); 205 | 206 | Serial.println("Entering sleep"); 207 | Serial.flush(); 208 | 209 | // Execution should not go beyond this 210 | nrf_power_system_off(NRF_POWER); 211 | } 212 | 213 | void setup() { 214 | Serial.begin(9600); 215 | // while (!Serial && millis() < 1000); // Timeout in case serial disconnected. 216 | pinMode(LEDR, OUTPUT); 217 | pinMode(LEDG, OUTPUT); 218 | pinMode(LEDB, OUTPUT); 219 | digitalWrite(LEDR, LOW); 220 | digitalWrite(LEDG, HIGH); 221 | digitalWrite(LEDB, HIGH); 222 | 223 | QSPIF_sleep(); 224 | 225 | Bluefruit.autoConnLed(false); 226 | Serial.println("Initialise the Bluefruit nRF52 module"); 227 | Bluefruit.configPrphBandwidth(4); // max is BANDWIDTH_MAX = 4 228 | Serial.print("Begin Bluefruit: "); 229 | Serial.println(Bluefruit.begin(1, 0)); 230 | Bluefruit.Periph.setConnInterval(6, 6); 231 | Bluefruit.setName("DPOINT"); 232 | Serial.println("Begin bleService"); 233 | bleService.begin(); 234 | 235 | imuCharacteristic.setProperties(CHR_PROPS_READ | CHR_PROPS_NOTIFY); 236 | imuCharacteristic.setPermission(SECMODE_OPEN, SECMODE_NO_ACCESS); 237 | imuCharacteristic.setFixedLen(sizeof(IMUDataPacket)); 238 | Serial.println("Begin imuCharacteristic"); 239 | imuCharacteristic.begin(); 240 | IMUDataPacket initialPacket = { 0 }; 241 | imuCharacteristic.write(&initialPacket, sizeof(initialPacket)); 242 | 243 | Serial.println("Setup finished"); 244 | } 245 | 246 | void loop() { 247 | Serial.print("Starting advertising..."); 248 | startAdvertising(); 249 | digitalWrite(LEDR, HIGH); 250 | digitalWrite(LEDG, HIGH); 251 | digitalWrite(LEDB, LOW); 252 | 253 | Serial.print("Starting IMU..."); 254 | setupImu(); 255 | 256 | unsigned long wakeUpTime = millis(); 257 | 258 | while (millis() - wakeUpTime < stayAwakeTimeMs) { 259 | if (Bluefruit.connected(0)) { 260 | Serial.println("Connected"); 261 | setupPressureSensor(); 262 | digitalWrite(LEDR, HIGH); 263 | digitalWrite(LEDG, LOW); 264 | digitalWrite(LEDB, HIGH); 265 | runBle(); 266 | digitalWrite(LEDR, HIGH); 267 | digitalWrite(LEDG, HIGH); 268 | digitalWrite(LEDB, LOW); 269 | digitalWrite(PRESSURE_SENSOR_VCC_PIN, LOW); 270 | wakeUpTime = millis(); 271 | } 272 | // Don't sleep if USB connected, to make code upload easier. 273 | if (Serial) wakeUpTime = millis(); 274 | delay(100); 275 | } 276 | Serial.println("Stopping advertising"); 277 | Bluefruit.Advertising.stop(); 278 | sleepUntilDoubleTap(); 279 | } 280 | -------------------------------------------------------------------------------- /print/.gitignore: -------------------------------------------------------------------------------- 1 | *.tmp 2 | OldVersions -------------------------------------------------------------------------------- /print/Compress Spring11.ipt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Jcparkyn/dpoint/a108c19b9b240c1531b2d30ed47ba18bc604862e/print/Compress Spring11.ipt -------------------------------------------------------------------------------- /print/assembly.iam: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Jcparkyn/dpoint/a108c19b9b240c1531b2d30ed47ba18bc604862e/print/assembly.iam -------------------------------------------------------------------------------- /print/assembly.ipn: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Jcparkyn/dpoint/a108c19b9b240c1531b2d30ed47ba18bc604862e/print/assembly.ipn -------------------------------------------------------------------------------- /print/batt_10440.ipt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Jcparkyn/dpoint/a108c19b9b240c1531b2d30ed47ba18bc604862e/print/batt_10440.ipt -------------------------------------------------------------------------------- /print/drawing.dwg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Jcparkyn/dpoint/a108c19b9b240c1531b2d30ed47ba18bc604862e/print/drawing.dwg -------------------------------------------------------------------------------- /print/export/stylus_body_bottom.stl: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Jcparkyn/dpoint/a108c19b9b240c1531b2d30ed47ba18bc604862e/print/export/stylus_body_bottom.stl -------------------------------------------------------------------------------- /print/export/stylus_body_top.stl: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Jcparkyn/dpoint/a108c19b9b240c1531b2d30ed47ba18bc604862e/print/export/stylus_body_top.stl -------------------------------------------------------------------------------- /print/export/stylus_nib_base.stl: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Jcparkyn/dpoint/a108c19b9b240c1531b2d30ed47ba18bc604862e/print/export/stylus_nib_base.stl -------------------------------------------------------------------------------- /print/seed_arduino.ipt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Jcparkyn/dpoint/a108c19b9b240c1531b2d30ed47ba18bc604862e/print/seed_arduino.ipt -------------------------------------------------------------------------------- /print/stylus_body.ipt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Jcparkyn/dpoint/a108c19b9b240c1531b2d30ed47ba18bc604862e/print/stylus_body.ipt -------------------------------------------------------------------------------- /print/stylus_nib.ipt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Jcparkyn/dpoint/a108c19b9b240c1531b2d30ed47ba18bc604862e/print/stylus_nib.ipt -------------------------------------------------------------------------------- /print/stylus_nib_base.ipt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Jcparkyn/dpoint/a108c19b9b240c1531b2d30ed47ba18bc604862e/print/stylus_nib_base.ipt -------------------------------------------------------------------------------- /python/.editorconfig: -------------------------------------------------------------------------------- 1 | 2 | [*.{py,ipynb}] 3 | indent_style = space 4 | indent_size = 4 -------------------------------------------------------------------------------- /python/.gitignore: -------------------------------------------------------------------------------- 1 | /venv 2 | /.venv 3 | .vscode 4 | __pycache__ 5 | .pytest_cache 6 | 7 | *.blend1 8 | 9 | *.received.txt 10 | 11 | !pen.obj 12 | marker_calibration_pics/ 13 | params/calibrated_marker_positions.json 14 | 15 | recordings/*/frames 16 | recordings/*/*.pdf 17 | recordings/*/*.pkl 18 | !recordings/*/camera_extrinsics.pkl 19 | 20 | analysis/tip_measurements/**/*.pdf 21 | analysis/*.png 22 | analysis/latency-histogram.pdf -------------------------------------------------------------------------------- /python/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Jcparkyn/dpoint/a108c19b9b240c1531b2d30ed47ba18bc604862e/python/__init__.py -------------------------------------------------------------------------------- /python/analysis/latencies.csv: -------------------------------------------------------------------------------- 1 | type,start,end 2 | prediction,1.062,1.095 3 | prediction,1.945,1.987 4 | prediction,2.761,2.799 5 | prediction,3.578,3.611 6 | prediction,4.448,4.494 7 | prediction,7.950,7.988 8 | prediction,8.663,8.708 9 | prediction,9.508,9.545 10 | prediction,10.299,10.333 11 | prediction,11.011,11.041 12 | prediction,11.728,11.769 13 | prediction,12.444,12.486 14 | prediction,13.127,13.160 15 | prediction,13.756,13.806 16 | prediction,14.626,14.664 17 | prediction,15.330,15.372 18 | prediction,16.084,16.121 19 | prediction,16.717,16.754 20 | prediction,17.433,17.471 21 | prediction,18.158,18.191 22 | prediction,21.694,21.731 23 | prediction,22.573,22.610 24 | prediction,23.255,23.289 25 | prediction,23.997,24.038 26 | prediction,24.751,24.792 27 | prediction,25.425,25.467 28 | prediction,26.108,26.146 29 | prediction,26.795,26.841 30 | prediction,27.474,27.512 31 | prediction,30.785,30.835 32 | delay,0.821,0.905 33 | delay,1.554,1.633 34 | delay,2.233,2.308 35 | delay,2.941,3.024 36 | delay,3.657,3.737 37 | delay,9.317,9.392 38 | delay,10.004,10.088 39 | delay,10.579,10.654 40 | delay,11.187,11.266 41 | delay,11.758,11.833 42 | delay,14.677,14.760 43 | delay,15.556,15.635 44 | delay,16.180,16.264 45 | delay,16.768,16.843 46 | delay,17.384,17.467 47 | delay,18.009,18.084 48 | delay,19.504,19.587 49 | delay,20.162,20.241 50 | delay,20.837,20.916 51 | delay,21.532,21.615 52 | delay,24.085,24.172 53 | delay,24.672,24.760 54 | delay,25.243,25.326 55 | delay,25.797,25.884 56 | delay,26.346,26.426 57 | delay,26.938,27.017 58 | delay,27.546,27.621 59 | delay,28.141,28.225 60 | delay,28.804,28.883 61 | delay,29.637,29.707 62 | cameraonly,4.373,4.477 63 | cameraonly,6.076,6.180 64 | cameraonly,8.092,8.204 65 | cameraonly,9.279,9.404 66 | cameraonly,11.644,11.774 67 | cameraonly,12.490,12.607 68 | cameraonly,13.406,13.502 69 | cameraonly,14.360,14.476 70 | cameraonly,15.309,15.438 71 | cameraonly,16.238,16.330 72 | cameraonly,17.462,17.567 73 | cameraonly,18.054,18.179 74 | cameraonly,18.776,18.878 75 | cameraonly,19.345,19.470 76 | cameraonly,20.003,20.107 77 | cameraonly,20.836,20.940 78 | cameraonly,21.648,21.773 79 | cameraonly,22.806,22.939 80 | cameraonly,23.505,23.643 81 | cameraonly,25.434,25.529 82 | cameraonly,26.112,26.237 83 | cameraonly,26.795,26.912 84 | cameraonly,27.537,27.645 85 | cameraonly,29.032,29.136 86 | cameraonly,29.748,29.869 87 | cameraonly,30.539,30.635 88 | cameraonly,32.272,32.397 89 | cameraonly,33.496,33.600 90 | cameraonly,34.117,34.238 91 | cameraonly,34.796,34.904 -------------------------------------------------------------------------------- /python/analysis/offline_ope.py: -------------------------------------------------------------------------------- 1 | import sys 2 | import time 3 | sys.path.append('.') 4 | 5 | import glob 6 | import json 7 | import ntpath 8 | import pickle 9 | 10 | import cv2 11 | from app.marker_tracker import ( 12 | CameraReading, 13 | MarkerTracker, 14 | load_marker_positions, 15 | read_camera_parameters, 16 | relative_transform, 17 | ) 18 | 19 | # This script pre-computes a list of estimated stylus poses from a set of frames. 20 | 21 | 22 | def main(): 23 | recording_timestamp = "20230905_162344" 24 | files = glob.glob(f"recordings/{recording_timestamp}/frames/*.bmp") 25 | with open( 26 | f"recordings/{recording_timestamp}/camera_extrinsics.pkl", "rb" 27 | ) as pickle_file: 28 | baseRvec, baseTvec = pickle.load(pickle_file) 29 | camera_data = [] 30 | cameraMatrix, distCoeffs = read_camera_parameters("params/camera_params_c922_f30.yml") 31 | markerPositions = load_marker_positions() 32 | 33 | tracker = MarkerTracker(cameraMatrix, distCoeffs, markerPositions) 34 | startTime = time.perf_counter() 35 | processingTimes = [] 36 | for file in files: 37 | frame_time = int(ntpath.basename(file).split(".")[0]) 38 | frame = cv2.imread(file) 39 | processingStartTime = time.perf_counter() 40 | result = tracker.process_frame(frame) 41 | if result is not None: 42 | rvec, tvec = result 43 | rvecRelative, tvecRelative = relative_transform( 44 | rvec, tvec, baseRvec, baseTvec 45 | ) 46 | Rrelative = cv2.Rodrigues(rvecRelative)[0] 47 | camera_data.append((frame_time, CameraReading(tvecRelative, Rrelative))) 48 | processingEndTime = time.perf_counter() 49 | processingTimes.append(processingEndTime - processingStartTime) 50 | endTime = time.perf_counter() 51 | totalProcessingTime = sum(processingTimes) 52 | totalTime = endTime - startTime 53 | 54 | print(f"Time per frame: {totalTime / len(files)}, processing only: {totalProcessingTime / len(files)}") 55 | # with open(f"recordings/{recording_timestamp}/camera_timing.json", "w") as f: 56 | # json.dump(processingTimes, f) 57 | 58 | with open( 59 | f"recordings/{recording_timestamp}/camera_data.json", "w" 60 | ) as f: 61 | json.dump([dict(t=t, data=reading.to_json()) for t, reading in camera_data], f, indent=2) 62 | 63 | 64 | if __name__ == "__main__": 65 | main() 66 | -------------------------------------------------------------------------------- /python/analysis/offline_playback.py: -------------------------------------------------------------------------------- 1 | import time 2 | from typing import NamedTuple 3 | 4 | import numpy as np 5 | import cv2 as cv 6 | from scipy.spatial import KDTree 7 | from pyquaternion import Quaternion 8 | 9 | from app.filter import DpointFilter, blend_new_data 10 | from app.marker_tracker import CameraReading 11 | from app.monitor_ble import StylusReading 12 | from app.dimensions import IMU_OFFSET, STYLUS_LENGTH 13 | 14 | INCH_TO_METRE = 0.0254 15 | 16 | 17 | def binarize(image: np.ndarray) -> np.ndarray: 18 | ret, threshold = cv.threshold(image, 0, 255, cv.THRESH_BINARY + cv.THRESH_OTSU) 19 | return threshold 20 | 21 | 22 | def reject_outliers_2d(x: np.ndarray, y: np.ndarray, m=2.0): 23 | d = np.sqrt((x - np.median(x)) ** 2 + (y - np.median(y)) ** 2) 24 | mdev = np.median(d) 25 | s = d / mdev if mdev else np.zeros(len(d)) 26 | indices = s < m 27 | return x[indices], y[indices] 28 | 29 | 30 | def get_black_points(image: np.ndarray, dpi: float): 31 | points_0, points_1 = (binarize(image) == 0).nonzero() 32 | points_x = points_1 * (INCH_TO_METRE / dpi) 33 | points_y = (image.shape[0] - points_0) * (INCH_TO_METRE / dpi) 34 | points_x, points_y = reject_outliers_2d(points_x, points_y) 35 | return points_x, points_y 36 | 37 | 38 | def normalize_points(xy: np.ndarray): 39 | return xy - np.mean(xy, axis=0) 40 | 41 | 42 | def camera_reading_to_tip_pos(reading: CameraReading): 43 | orientation_quat = Quaternion(matrix=reading.orientation_mat) 44 | tip_pos = reading.position.flatten() - orientation_quat.rotate( 45 | np.array([0, STYLUS_LENGTH, 0]) + IMU_OFFSET 46 | ).flatten() 47 | return tip_pos 48 | 49 | 50 | def replay_data(recorded_data: list[tuple[float, CameraReading | StylusReading]], dt, smoothing_length, camera_delay): 51 | filter = DpointFilter(dt=dt, smoothing_length=smoothing_length, camera_delay=camera_delay) 52 | sample_count = sum( 53 | isinstance(reading, StylusReading) for _, reading in recorded_data 54 | ) 55 | print(f"sample_count: {sample_count}") 56 | 57 | tip_pos_predicted = np.zeros((sample_count, 3)) 58 | tip_pos_smoothed = np.zeros((sample_count, 3)) 59 | pressure = np.zeros(sample_count) 60 | 61 | tip_pos_cameraonly = [] 62 | pressure_cameraonly = [] 63 | 64 | camera_fuse_times = [] 65 | stylus_fuse_times = [] 66 | 67 | pressure_baseline = 0.017 # Approximate measured value for initial estimate 68 | pressure_avg_factor = 0.1 # Factor for exponential moving average 69 | pressure_range = 0.02 70 | pressure_offset = 0.003 # Offset so that small positive numbers are treated as zero 71 | sample = 0 72 | for t, reading in recorded_data: 73 | t0 = time.perf_counter() 74 | match reading: 75 | case CameraReading(pos, or_mat): 76 | # print(f"t: {t}, pos: {pos}, or_mat: {or_mat}") 77 | tip_pos_cameraonly.append(camera_reading_to_tip_pos(reading)) 78 | pressure_cameraonly.append(pressure[sample - 4]) 79 | smoothed_tip_pos = filter.update_camera(pos.flatten(), or_mat) 80 | if smoothed_tip_pos: 81 | start = sample - len(smoothed_tip_pos) + 1 82 | tps_view = tip_pos_smoothed[start : sample + 1, :] 83 | tps_view[:,:] = blend_new_data(tps_view, smoothed_tip_pos, 0.5) 84 | camera_fuse_times.append(time.perf_counter() - t0) 85 | case StylusReading(accel=accel, gyro=gyro, t=_, pressure=p): 86 | filter.update_imu(accel, gyro) 87 | position, orientation = filter.get_tip_pose() 88 | zpos = position[2] 89 | if zpos > 0.005: 90 | # calibrate pressure baseline using current pressure reading 91 | pressure_baseline = ( 92 | pressure_baseline * (1 - pressure_avg_factor) 93 | + reading.pressure * pressure_avg_factor 94 | ) 95 | tip_pos_predicted[sample, :] = position.flatten() 96 | tip_pos_smoothed[sample, :] = position.flatten() 97 | pressure[sample] = ( 98 | p - pressure_baseline - pressure_offset 99 | ) / pressure_range 100 | stylus_fuse_times.append(time.perf_counter() - t0) 101 | sample += 1 102 | case _: 103 | print("Invalid reading", reading) 104 | camera_fuse_times = np.array(camera_fuse_times)*1000 105 | stylus_fuse_times = np.array(stylus_fuse_times)*1000 106 | print(f"Camera: {np.mean(camera_fuse_times):.3f}ms +- {np.std(camera_fuse_times):.3f}") 107 | print(f"Stylus: {np.mean(stylus_fuse_times):.3f}ms +- {np.std(stylus_fuse_times):.3f}") 108 | return tip_pos_predicted, tip_pos_smoothed, pressure, np.row_stack(tip_pos_cameraonly), np.array(pressure_cameraonly) 109 | 110 | 111 | def minimise_chamfer_distance(a: np.ndarray, b: np.ndarray, iterations=3): 112 | """Finds the optimal translation of b to minimise the chamfer distance to a.""" 113 | tree = KDTree(a) 114 | offset = np.mean(a, axis=0) - np.mean(b, axis=0) 115 | for i in range(iterations): 116 | _, indices = tree.query(b + offset) 117 | errors = (a[indices, :] - b) - offset 118 | error = np.mean(errors, axis=0) 119 | offset += error 120 | # print(f"Iteration {i}, offset: {offset}, error: {error}") 121 | dist, _ = tree.query(b + offset) 122 | return offset, dist 123 | 124 | 125 | def resample_line(points, desired_distance, mask): 126 | assert points.shape[1] == 2 127 | if points.shape[0] < 2: 128 | raise ValueError("The input array should contain at least 2 points.") 129 | 130 | # Calculate the total length of the line 131 | lengths = np.sqrt(np.sum(np.diff(points, axis=0) ** 2, axis=1)) 132 | total_length = np.sum(lengths) 133 | 134 | # Calculate the number of new points to be added 135 | num_points = int(np.ceil(total_length / desired_distance)) 136 | 137 | # Calculate the distances between the original points 138 | distances = np.zeros(len(points)) 139 | distances[1:] = np.cumsum(lengths) 140 | 141 | # Interpolate new points along the line 142 | new_distances = np.linspace(0, total_length, num_points) 143 | resampled_points = np.zeros((num_points, 2)) 144 | resampled_mask = np.interp(new_distances, distances, mask) 145 | for i in range(2): 146 | resampled_points[:, i] = np.interp(new_distances, distances, points[:, i]) 147 | 148 | return resampled_points[resampled_mask > 0.5, :] 149 | 150 | 151 | def merge_data( 152 | stylus_data: list, camera_data: list 153 | ) -> list[tuple[float, CameraReading | StylusReading]]: 154 | result = stylus_data + camera_data 155 | result.sort(key=lambda x: x[0]) 156 | return result 157 | 158 | 159 | class ProcessedStroke(NamedTuple): 160 | position: np.ndarray 161 | pressure: np.ndarray 162 | dist_mean: float 163 | 164 | 165 | class ProcessResult(NamedTuple): 166 | pressure: np.ndarray 167 | paths: dict[str, ProcessedStroke] 168 | 169 | 170 | def process_stroke( 171 | stroke: np.ndarray, scan_points: np.ndarray, pressure: np.ndarray 172 | ) -> ProcessedStroke: 173 | resample_dist = 0.001 * 0.5 # 0.5mm 174 | stroke_resampled = resample_line( 175 | stroke[:, :2], resample_dist, mask=pressure > 0.1 176 | ) 177 | offset, dist = minimise_chamfer_distance( 178 | scan_points, stroke_resampled, iterations=8 179 | ) 180 | offset3d = np.append(offset, 0) 181 | dist_mean = np.mean(dist) 182 | return ProcessedStroke( 183 | position=stroke + offset3d, pressure=pressure, dist_mean=dist_mean 184 | ) 185 | -------------------------------------------------------------------------------- /python/analysis/requirements.txt: -------------------------------------------------------------------------------- 1 | allpairspy==2.5.1 2 | approval-utilities==8.4.1 3 | approvaltests==8.4.1 4 | asttokens==2.2.1 5 | backcall==0.2.0 6 | beautifulsoup4==4.12.2 7 | bleak==0.20.2 8 | bleak-winrt==1.2.0 9 | certifi==2023.7.22 10 | charset-normalizer==3.2.0 11 | colorama==0.4.6 12 | comm==0.1.3 13 | contourpy==1.1.1 14 | cycler==0.11.0 15 | debugpy==1.6.7 16 | decorator==5.1.1 17 | empty-files==0.0.9 18 | executing==1.2.0 19 | fonttools==4.42.1 20 | freetype-py==2.4.0 21 | hsluv==5.0.3 22 | idna==3.4 23 | iniconfig==2.0.0 24 | ipykernel==6.25.0 25 | ipython==8.14.0 26 | jedi==0.19.0 27 | jupyter_client==8.3.0 28 | jupyter_core==5.3.1 29 | kiwisolver==1.4.4 30 | llvmlite==0.40.1 31 | matplotlib==3.8.0 32 | matplotlib-inline==0.1.6 33 | mock==5.1.0 34 | mrjob==0.7.4 35 | nest-asyncio==1.5.7 36 | numba==0.57.1 37 | numpy==1.24.4 38 | opencv-contrib-python==4.8.0.74 39 | packaging==23.1 40 | pandas==2.1.0 41 | parso==0.8.3 42 | pickleshare==0.7.5 43 | Pillow==10.0.1 44 | platformdirs==3.10.0 45 | pluggy==1.2.0 46 | prompt-toolkit==3.0.39 47 | psutil==5.9.5 48 | pure-eval==0.2.2 49 | Pygments==2.15.1 50 | pyparsing==3.1.1 51 | pyperclip==1.8.2 52 | PyQt6==6.5.2 53 | PyQt6-Qt6==6.5.2 54 | PyQt6-sip==13.5.2 55 | pyquaternion==0.9.9 56 | pytest==7.4.0 57 | python-dateutil==2.8.2 58 | pytz==2023.3.post1 59 | pywin32==306 60 | PyYAML==6.0.1 61 | pyzmq==25.1.0 62 | requests==2.31.0 63 | scipy==1.11.1 64 | seaborn==0.13.0 65 | six==1.16.0 66 | soupsieve==2.4.1 67 | stack-data==0.6.2 68 | testfixtures==7.1.0 69 | tornado==6.3.2 70 | traitlets==5.9.0 71 | typing_extensions==4.7.1 72 | tzdata==2023.3 73 | urllib3==2.0.4 74 | vispy==0.13.0 75 | wcwidth==0.2.6 76 | -------------------------------------------------------------------------------- /python/analysis/tip_measurements/no2steppnp/predicted_20230903_104829.csv: -------------------------------------------------------------------------------- 1 | 0.000988719,0.025962645,1.5081755e-05 2 | 0.0003994761,0.024874503,-0.00031959818 3 | -5.0939714e-05,0.02520879,-0.00093396544 4 | -0.00061105535,0.025585424,-0.0004126248 5 | 0.0006081067,0.025196314,-0.0011514479 6 | -5.0023114e-05,0.026214939,-0.000998219 7 | -0.0011631214,0.025433293,-0.0007946275 8 | -0.0015963342,0.024322614,0.0003280963 9 | -0.0009729329,0.024746282,-0.0004860177 10 | 8.127195e-05,0.024458982,-0.0009467948 11 | -2.3815928e-05,0.14483851,-0.00057031884 12 | 0.00033275783,0.14404711,-0.00069739425 13 | 0.00021030256,0.14434724,-0.0005502042 14 | -0.00033741855,0.14444739,-0.00045044706 15 | -0.00017841718,0.14336857,-0.0003970959 16 | -0.00019879515,0.1433702,-0.00028272715 17 | -7.8505254e-05,0.14329271,-0.00032725665 18 | 0.0009281744,0.1466855,-0.0006902015 19 | 0.00079110765,0.14337167,-0.00027919322 20 | -0.00011933327,0.14356677,-0.00014511406 21 | 0.0003138622,0.26272592,-0.0006527511 22 | 0.0008508221,0.26250455,-0.0011773211 23 | -0.00033299095,0.26208067,-0.000637439 24 | 0.001032939,0.26248762,-0.00091222115 25 | 0.0006040647,0.26181027,-0.0008861628 26 | -0.00066502753,0.2594022,-0.0012327172 27 | 0.0010597635,0.26068798,-0.0006799782 28 | 0.0019525177,0.2626261,-0.0010097004 29 | 0.0005207619,0.26283896,-0.0008916401 30 | 0.001070538,0.2622675,-0.00061251427 31 | 0.09492589,0.024722079,-8.5400636e-05 32 | 0.095832,0.024133459,-0.00010052981 33 | 0.095761985,0.023547998,0.00056504586 34 | 0.0960951,0.025120022,-1.3396539e-05 35 | 0.0952271,0.02586168,0.000106358566 36 | 0.09488825,0.024952106,-0.0004952357 37 | 0.0950601,0.024049852,-0.0003130844 38 | 0.09386161,0.025848027,4.7252677e-05 39 | 0.09519399,0.02547072,-0.00076031924 40 | 0.09406173,0.027258186,0.0004209919 41 | 0.094823,0.14457907,0.00019679111 42 | 0.09558538,0.1447823,-0.00068617845 43 | 0.094029784,0.14568321,-9.12776e-05 44 | 0.09504291,0.1460362,-0.00010456951 45 | 0.09608646,0.14480154,0.0003297889 46 | 0.09557549,0.14458649,-0.0002648358 47 | 0.09446535,0.14419982,-8.495944e-05 48 | 0.095227644,0.14324686,0.0005018025 49 | 0.096517034,0.1432865,-0.00010494671 50 | 0.09611771,0.14419416,-0.0005114417 51 | 0.0963988,0.26370344,-0.00087623025 52 | 0.095015116,0.2634781,-0.00030870474 53 | 0.09613192,0.26474422,0.000115171104 54 | 0.09425624,0.26479843,-0.00048319373 55 | 0.09357489,0.2636067,5.6957677e-05 56 | 0.09437593,0.26320663,-9.460475e-06 57 | 0.09580485,0.26297423,-0.00022743114 58 | 0.09581933,0.2622729,-0.00048876303 59 | 0.095444225,0.26394725,-0.0005529528 60 | 0.09510974,0.26347497,-0.00066065294 61 | 0.18939099,0.025985237,-0.0005219293 62 | 0.18947466,0.028367124,0.00017277253 63 | 0.19007428,0.025448686,-0.0001441444 64 | 0.18736312,0.027320907,0.0010028755 65 | 0.1903354,0.028091181,0.00037045623 66 | 0.19168937,0.02692486,0.000397679 67 | 0.19236335,0.024950674,6.8966554e-05 68 | 0.1900665,0.026515389,-1.0404849e-05 69 | 0.19075114,0.02590097,0.00021115613 70 | 0.19424054,0.02384048,-0.0011699426 71 | 0.19330671,0.1444463,3.9447856e-05 72 | 0.19282752,0.1441966,-0.0011976975 73 | 0.19015673,0.142958,-1.3288309e-05 74 | 0.19500647,0.14343539,-0.0015711272 75 | 0.18792063,0.14061308,0.000530528 76 | 0.19019817,0.14570163,9.7265234e-05 77 | 0.19158553,0.14499313,-0.00162018 78 | 0.18719406,0.14581305,0.00026951803 79 | 0.18855317,0.14732645,0.0002463784 80 | 0.19090433,0.14608654,-0.000964022 81 | 0.19525892,0.26132914,-0.0020496808 82 | 0.19346146,0.26615855,-0.003072057 83 | 0.18699059,0.26506528,-0.000111822716 84 | 0.1903323,0.26748955,-0.0015677344 85 | 0.1927074,0.27041486,-0.00021202167 86 | 0.18999496,0.26439083,-0.00072888314 87 | 0.19159253,0.2622448,-0.00031477242 88 | 0.19681786,0.2633422,-0.0012314451 89 | 0.18920311,0.2660419,-0.0013396696 90 | 0.18900657,0.26533997,-0.00043856064 91 | -------------------------------------------------------------------------------- /python/analysis/tip_measurements/no2steppnp/smoothed_20230903_104829.csv: -------------------------------------------------------------------------------- 1 | 0.0003029037,0.026124887,-0.00038792857 2 | -0.001052946,0.02572987,-0.00045298584 3 | 0.00011348617,0.025905248,-0.00044026048 4 | -0.0007754457,0.025671616,-0.00031798895 5 | -0.00064677995,0.026210725,-0.00063609815 6 | -0.0012028031,0.02691174,-0.00062740454 7 | -0.0017446805,0.025672894,-0.00069873367 8 | -0.0012768888,0.025752326,-0.0002810726 9 | -0.0011805723,0.02521732,-0.00047375992 10 | -0.001231021,0.025764927,-0.0009953702 11 | -5.9836507e-06,0.14403836,-0.00035342935 12 | 0.00068437366,0.14400148,-0.00042447937 13 | -0.00015843505,0.14391096,-0.0004190837 14 | -0.0005771922,0.1443779,-0.00025596944 15 | -0.00092761393,0.1435442,-0.00027081888 16 | -0.00042823228,0.14357018,-0.00028238236 17 | -0.00059934915,0.14314133,-0.00034614024 18 | 0.00015510479,0.14496689,-0.0005573671 19 | 0.00022371206,0.14337794,-0.00029782037 20 | 0.00030960966,0.1433399,-0.00020659596 21 | 0.00033175087,0.26280993,-0.0006616595 22 | -0.000350907,0.2623525,-0.001006212 23 | -0.0009338194,0.26190466,-0.0007988133 24 | 0.00021175547,0.26241046,-0.0009635579 25 | -0.00022810505,0.26190683,-0.0008262783 26 | -0.0001336066,0.26151332,-0.00075239147 27 | 0.00047459325,0.26134726,-0.0007491166 28 | 0.00071296963,0.26261288,-0.0008774234 29 | 0.00010606515,0.26285514,-0.0008909265 30 | 0.0010084867,0.26227823,-0.0007975081 31 | 0.094690986,0.025469553,0.0002160375 32 | 0.094996214,0.024695488,1.4169841e-05 33 | 0.0946957,0.02431208,0.00020573974 34 | 0.095499285,0.02492263,5.420864e-05 35 | 0.09509793,0.02601804,-5.6596466e-05 36 | 0.093638346,0.025533061,-0.00035860264 37 | 0.094617695,0.023755554,-0.00043957643 38 | 0.09408366,0.025516093,0.00014866053 39 | 0.09414008,0.025192862,-0.0006449127 40 | 0.09385338,0.026737664,0.00010556249 41 | 0.09480763,0.14374341,0.00010610072 42 | 0.093403116,0.14425017,-0.00035999823 43 | 0.09358379,0.14505857,-0.00010196113 44 | 0.09453129,0.1449519,0.00014135706 45 | 0.0955249,0.14499749,5.830465e-05 46 | 0.094864614,0.14414653,2.7083815e-05 47 | 0.09470779,0.14375255,-8.6825865e-05 48 | 0.09683015,0.14294484,-9.5358686e-05 49 | 0.095475905,0.14340195,2.4179682e-05 50 | 0.09590666,0.14324956,-2.71263e-05 51 | 0.09563055,0.26309144,-0.0004829816 52 | 0.09489561,0.2628177,-0.00045768733 53 | 0.09428167,0.2632619,-0.0010135614 54 | 0.0948563,0.26405644,-0.00022524038 55 | 0.093830004,0.26328948,-0.00018766553 56 | 0.095001996,0.263564,-0.00018494352 57 | 0.09406633,0.26257986,-0.0004267205 58 | 0.09501165,0.26293015,-0.00017497865 59 | 0.09472152,0.26383045,-0.0003891815 60 | 0.09555234,0.26441476,-0.0010284244 61 | 0.18958457,0.025684351,-0.00049250596 62 | 0.19051786,0.026755212,0.00035976642 63 | 0.18820366,0.02590249,-0.00055268774 64 | 0.18902405,0.02589255,0.00016659251 65 | 0.19054858,0.026719926,0.0004311742 66 | 0.19097196,0.025618508,0.00043487415 67 | 0.19035056,0.024276404,-0.00015637808 68 | 0.19056287,0.02556085,0.0005016625 69 | 0.1899339,0.025004027,0.00053905096 70 | 0.19059014,0.025810024,-0.0004918068 71 | 0.19260874,0.14355847,7.7863515e-05 72 | 0.19091488,0.14372565,-0.0007284378 73 | 0.1894745,0.14341351,-0.000107135966 74 | 0.18869756,0.14384574,-0.0013851335 75 | 0.18968284,0.14346999,-1.3099532e-05 76 | 0.19157448,0.14482112,-0.00016724561 77 | 0.18951273,0.14438912,-0.00034119 78 | 0.1899693,0.14642562,0.00011041405 79 | 0.18971026,0.14466925,-6.132253e-05 80 | 0.18990538,0.1451521,-0.00093172025 81 | 0.19231433,0.26015037,-0.0020597202 82 | 0.1882963,0.26290247,-0.002608478 83 | 0.18779233,0.26333404,-0.0004110477 84 | 0.18952097,0.26556584,-0.0021265114 85 | 0.18870215,0.26464856,-0.0007199466 86 | 0.1912773,0.26430196,-7.657381e-05 87 | 0.19154988,0.2626481,-0.00029812727 88 | 0.19273269,0.26177233,-0.0022844195 89 | 0.1887233,0.26339206,-0.00076654385 90 | 0.18903883,0.26324576,-0.00080106733 91 | -------------------------------------------------------------------------------- /python/analysis/tip_measurements/nofilter/predicted_20230903_114230.csv: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Jcparkyn/dpoint/a108c19b9b240c1531b2d30ed47ba18bc604862e/python/analysis/tip_measurements/nofilter/predicted_20230903_114230.csv -------------------------------------------------------------------------------- /python/analysis/tip_measurements/nofilter/predicted_20230903_114250.csv: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Jcparkyn/dpoint/a108c19b9b240c1531b2d30ed47ba18bc604862e/python/analysis/tip_measurements/nofilter/predicted_20230903_114250.csv -------------------------------------------------------------------------------- /python/analysis/tip_measurements/nofilter/predicted_20230903_114351.csv: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Jcparkyn/dpoint/a108c19b9b240c1531b2d30ed47ba18bc604862e/python/analysis/tip_measurements/nofilter/predicted_20230903_114351.csv -------------------------------------------------------------------------------- /python/analysis/tip_measurements/nofilter/predicted_20230903_114505.csv: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Jcparkyn/dpoint/a108c19b9b240c1531b2d30ed47ba18bc604862e/python/analysis/tip_measurements/nofilter/predicted_20230903_114505.csv -------------------------------------------------------------------------------- /python/analysis/tip_measurements/nofilter/predicted_20230903_114549.csv: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Jcparkyn/dpoint/a108c19b9b240c1531b2d30ed47ba18bc604862e/python/analysis/tip_measurements/nofilter/predicted_20230903_114549.csv -------------------------------------------------------------------------------- /python/analysis/tip_measurements/nofilter/predicted_20230903_162552.csv: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Jcparkyn/dpoint/a108c19b9b240c1531b2d30ed47ba18bc604862e/python/analysis/tip_measurements/nofilter/predicted_20230903_162552.csv -------------------------------------------------------------------------------- /python/analysis/tip_measurements/nofilter/predicted_20230905_105507.csv: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Jcparkyn/dpoint/a108c19b9b240c1531b2d30ed47ba18bc604862e/python/analysis/tip_measurements/nofilter/predicted_20230905_105507.csv -------------------------------------------------------------------------------- /python/analysis/tip_measurements/nofilter/predicted_20230905_105713.csv: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Jcparkyn/dpoint/a108c19b9b240c1531b2d30ed47ba18bc604862e/python/analysis/tip_measurements/nofilter/predicted_20230905_105713.csv -------------------------------------------------------------------------------- /python/analysis/tip_measurements/nofilter/smoothed_20230903_114230.csv: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Jcparkyn/dpoint/a108c19b9b240c1531b2d30ed47ba18bc604862e/python/analysis/tip_measurements/nofilter/smoothed_20230903_114230.csv -------------------------------------------------------------------------------- /python/analysis/tip_measurements/nofilter/smoothed_20230903_114250.csv: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Jcparkyn/dpoint/a108c19b9b240c1531b2d30ed47ba18bc604862e/python/analysis/tip_measurements/nofilter/smoothed_20230903_114250.csv -------------------------------------------------------------------------------- /python/analysis/tip_measurements/nofilter/smoothed_20230903_114351.csv: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Jcparkyn/dpoint/a108c19b9b240c1531b2d30ed47ba18bc604862e/python/analysis/tip_measurements/nofilter/smoothed_20230903_114351.csv -------------------------------------------------------------------------------- /python/analysis/tip_measurements/nofilter/smoothed_20230903_114505.csv: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Jcparkyn/dpoint/a108c19b9b240c1531b2d30ed47ba18bc604862e/python/analysis/tip_measurements/nofilter/smoothed_20230903_114505.csv -------------------------------------------------------------------------------- /python/analysis/tip_measurements/nofilter/smoothed_20230903_114549.csv: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Jcparkyn/dpoint/a108c19b9b240c1531b2d30ed47ba18bc604862e/python/analysis/tip_measurements/nofilter/smoothed_20230903_114549.csv -------------------------------------------------------------------------------- /python/analysis/tip_measurements/nofilter/smoothed_20230903_162552.csv: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Jcparkyn/dpoint/a108c19b9b240c1531b2d30ed47ba18bc604862e/python/analysis/tip_measurements/nofilter/smoothed_20230903_162552.csv -------------------------------------------------------------------------------- /python/analysis/tip_measurements/nofilter/smoothed_20230905_105507.csv: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Jcparkyn/dpoint/a108c19b9b240c1531b2d30ed47ba18bc604862e/python/analysis/tip_measurements/nofilter/smoothed_20230905_105507.csv -------------------------------------------------------------------------------- /python/analysis/tip_measurements/nofilter/smoothed_20230905_105713.csv: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Jcparkyn/dpoint/a108c19b9b240c1531b2d30ed47ba18bc604862e/python/analysis/tip_measurements/nofilter/smoothed_20230905_105713.csv -------------------------------------------------------------------------------- /python/analysis/tip_measurements/nomarkercalib/predicted_20230902_191235.csv: -------------------------------------------------------------------------------- 1 | 0.0014141263,0.024553621,5.8834168e-05 2 | 0.0023236386,0.02523166,-8.966219e-05 3 | 0.0015790227,0.026338127,-0.0005602371 4 | 0.00024782165,0.025628017,-0.00014026067 5 | 0.00094174786,0.025512496,6.263566e-05 6 | 0.0004122328,0.14360198,6.1106846e-05 7 | 0.00093417586,0.14414872,-0.0002516737 8 | 0.0010865261,0.14363435,-0.00031100702 9 | 0.0006521406,0.14537634,-2.2045893e-05 10 | 0.0015282324,0.14462824,-0.0006252238 11 | -0.00018699863,0.1444019,0.00024900382 12 | -0.0006599318,0.14404143,0.0004414377 13 | 0.0007905522,0.14370933,-0.00021530614 14 | 0.0014647718,0.14423636,-0.00011577799 15 | 0.0014899628,0.14449194,-7.174196e-05 16 | 0.0009383219,0.024725506,0.00010030966 17 | 0.0013567837,0.02567877,0.00013983343 18 | -6.125108e-06,0.024196347,-0.00048550405 19 | 0.0016938101,0.025339687,-6.674391e-05 20 | -0.0007620022,0.026337849,0.0004691193 21 | 0.00052437466,0.2635748,2.0806963e-05 22 | 0.00069621496,0.2642158,0.0001255706 23 | 0.0007566231,0.26430744,-0.00048155617 24 | 0.00046326534,0.26376262,-0.00016481453 25 | -0.00022522412,0.26451325,0.00019521681 26 | -0.001210059,0.26380408,0.00024083661 27 | -0.00036001968,0.2630329,-5.96105e-06 28 | 0.0010297245,0.2633452,-0.00033040126 29 | -0.0009308426,0.26375875,0.00013014463 30 | 9.9201505e-05,0.26415598,-0.00014911406 31 | 0.095108055,0.02540753,0.001286934 32 | 0.093866184,0.026415033,0.0009525902 33 | 0.093540125,0.025127772,0.00085199176 34 | 0.09409052,0.024969239,0.0008776788 35 | 0.09524574,0.025335329,0.00022904373 36 | 0.09527476,0.025405852,0.00026898735 37 | 0.09604711,0.024995929,0.000547316 38 | 0.095564246,0.02521837,0.00052249775 39 | 0.09626292,0.025134498,0.00021314094 40 | 0.09483486,0.02425636,0.0003570666 41 | 0.09279551,0.14370286,0.0009795092 42 | 0.094494835,0.14454105,0.0007278104 43 | 0.0963431,0.14470556,0.0002168878 44 | 0.09640227,0.14480023,0.00014539885 45 | 0.09620094,0.1441551,0.00032324286 46 | 0.09309177,0.14431421,0.00066786644 47 | 0.09557443,0.14318891,0.00062690396 48 | 0.09693668,0.14373246,0.00032220784 49 | 0.09692055,0.14441967,0.00043259474 50 | 0.09699072,0.14379825,0.0002851701 51 | 0.09328393,0.26248536,0.0007245879 52 | 0.09397911,0.26321393,0.00066373544 53 | 0.09527973,0.26413715,0.00016590669 54 | 0.09556047,0.2645901,3.9212126e-05 55 | 0.09596496,0.2647391,-6.398888e-05 56 | 0.09487629,0.26405013,0.0003256067 57 | 0.09514767,0.2640183,0.00034259443 58 | 0.094128266,0.26203778,0.00035151033 59 | 0.095529504,0.26344803,0.0004910525 60 | 0.0961544,0.26408327,0.00020840399 61 | 0.18851468,0.024927123,0.0012913002 62 | 0.18924648,0.025118023,0.0010975122 63 | 0.19145264,0.024387496,0.0006120445 64 | 0.1912112,0.024535254,0.00071092014 65 | 0.19229472,0.024753597,0.0007781301 66 | 0.18764432,0.024827039,0.0011922958 67 | 0.19195339,0.024857935,0.0004372337 68 | 0.1918496,0.025342654,0.0005597637 69 | 0.19194104,0.025353055,0.000607816 70 | 0.19072741,0.025481496,0.0010974174 71 | 0.19306317,0.14487848,0.00080942275 72 | 0.19159517,0.14509669,0.00069814787 73 | 0.19195421,0.1453229,0.00056199566 74 | 0.18956313,0.14529717,0.0010219781 75 | 0.18902889,0.14519426,0.0008637112 76 | 0.18947606,0.1440185,0.001040534 77 | 0.19291842,0.14391968,0.000532954 78 | 0.19239157,0.14457811,0.00056055875 79 | 0.1926794,0.14525092,0.0005602673 80 | 0.19181369,0.14481436,0.00042369426 81 | 0.1919087,0.264159,0.00082222314 82 | 0.19348586,0.26488313,0.00035599718 83 | 0.19153582,0.26474246,0.00070081954 84 | 0.1920809,0.26510948,0.00027656797 85 | 0.19147088,0.2642635,0.0006681696 86 | 0.19254503,0.26511133,0.0001699207 87 | 0.19094908,0.26446274,0.0005982977 88 | 0.18875144,0.2626403,0.00082455116 89 | 0.18870497,0.2625977,0.0007618844 90 | 0.18866962,0.26197138,0.00046986833 91 | -------------------------------------------------------------------------------- /python/analysis/tip_measurements/nomarkercalib/smoothed_20230902_191235.csv: -------------------------------------------------------------------------------- 1 | 0.0016742414,0.024450984,5.7748148e-05 2 | 0.0018236212,0.02506619,-2.1763663e-05 3 | 0.0013008809,0.025562134,-0.0003653401 4 | 0.00043026748,0.025447594,-0.00015804893 5 | 0.0011996034,0.024831023,-0.00016407818 6 | 0.00019753393,0.14342597,-0.00015245653 7 | 0.0010958357,0.14434528,-0.00013294857 8 | 0.0015905233,0.14361745,-0.00016152325 9 | 0.0010257148,0.14490484,-9.178967e-05 10 | 0.0014786121,0.14416517,-0.0003871435 11 | -2.3638711e-05,0.14406408,0.00023682184 12 | -0.00079716014,0.14315985,0.00028282715 13 | 0.0006409988,0.14352158,-0.0002710929 14 | 0.0014790358,0.14449689,-0.00020674539 15 | 0.0020802314,0.14439535,-3.66339e-05 16 | 0.00094449334,0.024958648,-6.465695e-05 17 | 0.0018162183,0.02506369,-4.1450796e-05 18 | 0.00023790926,0.025245018,-0.0002989177 19 | 0.001561813,0.024836395,-0.00011486184 20 | -0.0005173678,0.025556246,0.0002932321 21 | 0.0009506509,0.26418257,-0.000264499 22 | 0.00097116426,0.26452184,3.196023e-05 23 | 0.00017269785,0.26383412,-0.0005285385 24 | 0.00014483406,0.2641118,-0.00022902 25 | 0.0005305559,0.26477817,0.00014491081 26 | -0.00069827586,0.262862,0.00018413611 27 | -0.00038334014,0.2625968,-6.1392406e-05 28 | 0.00055882183,0.2635401,-0.00037180036 29 | -0.00095802196,0.26357326,0.00014981606 30 | 8.741793e-05,0.26426166,-0.0001548456 31 | 0.09456011,0.025508912,0.0006284731 32 | 0.09345871,0.02577536,0.0007520009 33 | 0.0929058,0.024853513,0.00087231037 34 | 0.09348231,0.02452863,0.00064143975 35 | 0.095027044,0.02543407,0.00014186703 36 | 0.09509178,0.025466612,0.00013187174 37 | 0.096022464,0.024986988,0.00046819268 38 | 0.095837325,0.024920646,0.0005222589 39 | 0.09620382,0.0248446,0.00046578588 40 | 0.09470654,0.024159646,0.00028762774 41 | 0.09320546,0.14363018,0.0009084084 42 | 0.09539348,0.14462635,0.0006012288 43 | 0.095605366,0.14497335,0.0003540626 44 | 0.09691892,0.14464012,0.0002504444 45 | 0.09649732,0.14419879,0.00040537948 46 | 0.09347063,0.1436673,0.00077373895 47 | 0.09545446,0.1433235,0.00039387788 48 | 0.09663431,0.14409609,0.00024423553 49 | 0.09728987,0.14422964,0.00036316187 50 | 0.096996896,0.1440511,0.00023105528 51 | 0.0934568,0.26260597,0.00068744284 52 | 0.09420618,0.26344413,0.0006089196 53 | 0.094764,0.264118,0.0003303971 54 | 0.09547642,0.26462984,8.534162e-06 55 | 0.09592958,0.26485404,2.792039e-05 56 | 0.095571406,0.2640699,0.0003039872 57 | 0.09559082,0.26425573,0.00046897493 58 | 0.093736336,0.26175737,0.000386149 59 | 0.09616159,0.2640581,0.00037225176 60 | 0.095975846,0.26397675,0.00018437247 61 | 0.18901335,0.024897601,0.001181618 62 | 0.18911101,0.02464396,0.0009474721 63 | 0.19118449,0.023960618,0.0007760441 64 | 0.19224723,0.024443947,0.0007612691 65 | 0.19225186,0.0245565,0.00058543973 66 | 0.18902755,0.024565913,0.0011989367 67 | 0.19206384,0.024644481,0.00086160994 68 | 0.19164932,0.024442488,0.0006455089 69 | 0.19219771,0.025298046,0.00061507075 70 | 0.19122937,0.025152044,0.00077125727 71 | 0.19334546,0.14503545,0.00035788756 72 | 0.1918212,0.14518662,0.00059523457 73 | 0.19240656,0.14517023,0.0004745198 74 | 0.19038887,0.14467749,0.00090923335 75 | 0.1895161,0.14430262,0.00083580316 76 | 0.1898966,0.14404444,0.00087618723 77 | 0.19365174,0.14410308,0.00048099423 78 | 0.1926844,0.14465554,0.00045314114 79 | 0.19287615,0.14502232,0.00031447486 80 | 0.19250257,0.14488517,0.00054910785 81 | 0.19191155,0.26393825,0.00060887536 82 | 0.19257924,0.26437795,0.00019533587 83 | 0.19274543,0.2647369,0.00052994 84 | 0.19182123,0.2641288,0.00045311646 85 | 0.19220954,0.26406664,0.00037455163 86 | 0.19220239,0.26507488,-4.5665158e-05 87 | 0.19084986,0.26438865,0.0004741754 88 | 0.1898558,0.26273724,0.00093159097 89 | 0.18873028,0.2621991,0.0006998352 90 | 0.19020921,0.26199508,0.00045207812 91 | -------------------------------------------------------------------------------- /python/analysis/tip_measurements/proposed/predicted_20230902_143553.csv: -------------------------------------------------------------------------------- 1 | 0.00032874156,0.024639767,0.00037977862 2 | 0.0004107419,0.02654422,7.582638e-05 3 | 5.244249e-05,0.025494585,0.00029519774 4 | 0.00024176874,0.025069105,0.00047572824 5 | 0.00035694588,0.025512842,0.00044271923 6 | 0.00107052,0.1438271,2.8047009e-05 7 | 0.001297817,0.14461328,-0.00017561066 8 | 0.0010324698,0.14507793,-0.00011216778 9 | 0.00071807666,0.14443636,0.00026429887 10 | -0.00016893385,0.14446223,0.00017460785 11 | 0.0012000812,0.26438057,-0.0001916316 12 | 0.001036596,0.26404592,-9.297059e-05 13 | 0.00055597606,0.26402816,-0.0002906892 14 | 0.0009932386,0.2642272,-0.00018969756 15 | 0.0011747368,0.26419437,-0.00036998963 16 | 0.19228357,0.26265723,0.00010821389 17 | 0.19219826,0.26472375,1.7893432e-05 18 | 0.19189559,0.26370695,-0.00034027695 19 | 0.19175684,0.26274,0.00044879914 20 | 0.19313107,0.26377386,0.00011239306 21 | 0.19206819,0.14432694,0.00064085826 22 | 0.19248736,0.14509542,0.0005407294 23 | 0.1925948,0.14485349,0.00022139233 24 | 0.19195208,0.14417517,0.00058673025 25 | 0.19130173,0.14420752,0.00043146336 26 | 0.19171642,0.023599315,0.0008885909 27 | 0.19201753,0.023962842,0.0010351547 28 | 0.1914966,0.025198007,0.0007306639 29 | 0.19065863,0.0245675,0.0010087142 30 | 0.19044556,0.02351389,0.0006293979 31 | 0.09485423,0.024278065,0.00093570485 32 | 0.09556866,0.025437994,0.0009720397 33 | 0.094392195,0.026367467,0.0007460577 34 | 0.09443945,0.026147867,0.00024375856 35 | 0.094812945,0.02455391,0.00073277246 36 | 0.09580119,0.14343704,0.0003720763 37 | 0.09624556,0.14421488,0.0004206123 38 | 0.097075656,0.14418593,0.00023912008 39 | 0.09637429,0.14486182,0.00031492652 40 | 0.09563772,0.14448975,0.00033009698 41 | 0.09513594,0.2636721,0.00015230301 42 | 0.09506857,0.2637067,1.53299e-05 43 | 0.09576233,0.26404396,7.7392164e-05 44 | 0.096013226,0.26374575,0.00023026584 45 | 0.09657,0.2646233,1.950596e-05 46 | -------------------------------------------------------------------------------- /python/analysis/tip_measurements/proposed/predicted_20230905_110153.csv: -------------------------------------------------------------------------------- 1 | 0.0009997784,0.02512118,-0.0005015245 2 | 0.0007429597,0.025184589,-0.00042142 3 | 0.00035176557,0.025129989,-0.00018397416 4 | 0.00024846246,0.024928654,-0.00027862977 5 | -0.00046580192,0.024477046,9.374543e-06 6 | -0.00012363099,0.024130875,-0.00014001511 7 | 9.863161e-05,0.023817126,-0.00012920072 8 | 0.001254367,0.024479896,-0.00025453095 9 | 0.0018331313,0.024736876,-0.00042172516 10 | 0.0015955486,0.024733184,-0.00021500977 11 | 0.0006837301,0.14387622,-2.9133307e-05 12 | 0.0014003325,0.14401183,-6.480018e-05 13 | 0.0015753349,0.1441283,-8.9901005e-05 14 | 0.0017479993,0.14427197,-0.00016383977 15 | 0.0016355896,0.14471258,-9.242722e-05 16 | 0.0017179738,0.14477392,-0.00013706971 17 | 0.0016799346,0.14514242,-0.00026854945 18 | 0.0011834117,0.14480394,-3.7118325e-06 19 | 0.0014333734,0.1451081,-0.000118163516 20 | 0.001653366,0.14517185,-0.00038269622 21 | 0.00087875087,0.26440617,-0.00026855752 22 | 0.00032401658,0.2638699,-8.717761e-05 23 | 0.0006280413,0.2638735,-0.0001897036 24 | 0.00037926424,0.26374334,-5.1726143e-05 25 | 0.00055152766,0.2635548,-0.00014655331 26 | 0.0006828024,0.26340348,-0.0001938328 27 | 0.00021504918,0.26301494,-0.00032214387 28 | 0.0003725043,0.26333407,-0.00020942478 29 | 0.00022327914,0.26390016,-0.00047375614 30 | 0.00031446596,0.26406452,-0.00036116297 31 | 0.096512884,0.024392316,6.721291e-05 32 | 0.095893174,0.024252348,0.00022328775 33 | 0.0949442,0.024003964,0.00032565423 34 | 0.09523285,0.024006207,0.0003718664 35 | 0.094560176,0.024205225,0.0002971978 36 | 0.09467302,0.024798587,0.00029815402 37 | 0.09520011,0.024795113,0.00020406391 38 | 0.095532835,0.025432715,0.00012547662 39 | 0.09681072,0.025333663,0.00013833425 40 | 0.096318975,0.025708253,0.00019151531 41 | 0.09643178,0.14425983,0.00023974205 42 | 0.09678866,0.14560264,0.00019774967 43 | 0.09570292,0.14577566,0.00014707603 44 | 0.09552383,0.14397468,0.00023520907 45 | 0.09552138,0.14472009,0.00029509817 46 | 0.0960262,0.1442036,0.0002681898 47 | 0.09688014,0.14451465,0.00023701676 48 | 0.096068546,0.14488485,0.00016907402 49 | 0.095856756,0.1443141,0.0004051994 50 | 0.096435465,0.14423782,0.0005421295 51 | 0.09608336,0.26421908,1.1006184e-05 52 | 0.09603362,0.26368803,4.8442314e-05 53 | 0.09540272,0.26338056,0.00010160715 54 | 0.095228456,0.26379815,2.4578467e-05 55 | 0.09506421,0.2648685,0.00022902379 56 | 0.09599631,0.26411855,1.4767834e-05 57 | 0.09594598,0.2635492,9.104174e-05 58 | 0.09524821,0.263046,5.8713493e-05 59 | 0.09516139,0.26333585,0.00016544583 60 | 0.09529274,0.26427677,0.00033925765 61 | 0.19233769,0.02408611,0.0004901497 62 | 0.19231904,0.02359802,0.000535451 63 | 0.19149669,0.023700835,0.0006679848 64 | 0.19096678,0.02369556,0.0008173144 65 | 0.19029017,0.023996841,0.00050470984 66 | 0.19098379,0.02437836,0.00066505047 67 | 0.19082564,0.024802828,0.00068560266 68 | 0.19142798,0.02456313,0.00046348764 69 | 0.19087017,0.024420293,0.0006162198 70 | 0.19208679,0.02418221,0.0006207118 71 | 0.19155715,0.14424086,0.0004401101 72 | 0.1909066,0.14500144,0.00057762227 73 | 0.19183269,0.14446592,0.0003526113 74 | 0.19127052,0.14388621,0.00059779995 75 | 0.19181012,0.14451176,0.00042041278 76 | 0.19260676,0.1446051,0.00026413047 77 | 0.19162872,0.1440996,0.0003227189 78 | 0.19172035,0.1443614,0.000360284 79 | 0.19145837,0.14561015,0.00025433992 80 | 0.19255956,0.14476745,0.00012490233 81 | 0.19370551,0.26480645,3.4257328e-06 82 | 0.19196706,0.26309514,0.00043122272 83 | 0.19151983,0.26499802,0.00019928809 84 | 0.19097556,0.26392636,0.00010445788 85 | 0.19197148,0.26394215,1.0884478e-06 86 | 0.19128709,0.263627,0.00014752611 87 | 0.19187929,0.26342118,0.0001418935 88 | 0.19233689,0.26392218,-1.3392764e-05 89 | 0.19042483,0.26400483,0.00018808583 90 | 0.19090375,0.26349095,4.2225285e-05 91 | -------------------------------------------------------------------------------- /python/analysis/tip_measurements/proposed/smoothed_20230902_143553.csv: -------------------------------------------------------------------------------- 1 | 0.0005136792,0.0248245,0.0002436325 2 | -0.00016716105,0.025893241,0.00027493376 3 | -0.00041878186,0.025240505,0.00029829226 4 | 5.4618224e-05,0.024650697,0.000324275 5 | 0.0001889985,0.025219873,0.0003910549 6 | 0.0011725007,0.14394456,-3.4110642e-05 7 | 0.0014012489,0.14458807,-5.119401e-05 8 | 0.0011046835,0.14496078,-8.993144e-05 9 | 0.00047569422,0.14371455,8.6683554e-05 10 | 0.00019246759,0.14434986,6.8080924e-05 11 | 0.0014589538,0.26467305,-0.0003967218 12 | 0.0010504605,0.2638823,-0.0003044818 13 | 0.0004088853,0.26398063,-0.00033943405 14 | 0.00095476845,0.26396525,-0.00035267608 15 | 0.0014255863,0.26419115,-0.00043133163 16 | 0.19239233,0.2628468,2.0520405e-05 17 | 0.19156687,0.26310483,-1.6559916e-05 18 | 0.1912779,0.26352853,4.728982e-05 19 | 0.1921002,0.26278546,-0.000103150225 20 | 0.19214855,0.26307517,-4.28695e-05 21 | 0.19218366,0.14407787,0.00034795224 22 | 0.19233704,0.14421575,0.00028321907 23 | 0.19242334,0.14465095,0.0003622344 24 | 0.19229478,0.14416286,0.00035579663 25 | 0.19154218,0.14417244,0.00040597905 26 | 0.19203107,0.023431595,0.0007149464 27 | 0.1901807,0.024611797,0.0008259759 28 | 0.19138956,0.02478959,0.0007182556 29 | 0.19118783,0.024259701,0.00082906627 30 | 0.19063203,0.023131892,0.0006547625 31 | 0.094832346,0.024052318,0.00079600967 32 | 0.09604413,0.025193723,0.0006086488 33 | 0.095023006,0.025435679,0.00066639873 34 | 0.09474443,0.025234625,0.000646005 35 | 0.095347874,0.024461549,0.00060199405 36 | 0.096356735,0.14376254,0.0002904637 37 | 0.0969816,0.1439056,0.00029087538 38 | 0.0969837,0.14404628,0.00024197705 39 | 0.09652693,0.1446002,0.00029998398 40 | 0.09557045,0.14411497,0.00038726017 41 | 0.09568813,0.26437923,-6.447244e-06 42 | 0.09529926,0.2635737,-5.0267613e-06 43 | 0.09608613,0.26377124,-4.6850386e-05 44 | 0.096217774,0.26380867,9.299294e-05 45 | 0.09691346,0.26485366,-8.1798435e-06 46 | -------------------------------------------------------------------------------- /python/analysis/tip_measurements/proposed/smoothed_20230905_110153.csv: -------------------------------------------------------------------------------- 1 | 0.0010215777,0.025124138,-0.0005428987 2 | 0.00059949665,0.025065225,-0.00041157586 3 | 0.00038564028,0.025076183,-0.00021990981 4 | -2.266084e-06,0.024812283,-0.00028843217 5 | -0.00043359783,0.024454957,-8.184023e-05 6 | -0.00017676884,0.02408763,-0.00020720367 7 | 0.00020855613,0.02377143,-0.00017927254 8 | 0.0009868471,0.024363004,-0.00012833413 9 | 0.0012752691,0.024398012,-0.00041983815 10 | 0.0012808394,0.024588795,-0.00028179618 11 | 0.0009947352,0.14371766,-0.00010988748 12 | 0.0015902272,0.14404108,-0.00016171079 13 | 0.0015539584,0.1439978,-0.00018070453 14 | 0.0016140887,0.14415947,-0.00015641005 15 | 0.0017372874,0.14447471,-9.381264e-05 16 | 0.0015515785,0.14442496,-0.0001473804 17 | 0.0018258027,0.14476399,-0.00015320338 18 | 0.0014147981,0.14467204,-0.00022556387 19 | 0.0016390103,0.14482798,-9.1129834e-05 20 | 0.001547974,0.14496621,-0.00027691992 21 | 0.00068504986,0.26441437,-0.00032969934 22 | 0.00057501585,0.2643339,-0.00013446997 23 | 0.0007027237,0.26380417,-0.00033129705 24 | 0.0007273415,0.26408786,-0.00013477298 25 | 0.0007586612,0.26380444,-0.00020361181 26 | 0.0008997356,0.26349106,-0.00024318395 27 | 0.0003152377,0.26313135,-0.00015242852 28 | 0.00028739154,0.26320875,-0.0002514069 29 | 0.00014095978,0.2639711,-0.00036702055 30 | 0.00041771893,0.26422298,-0.00025911425 31 | 0.096402004,0.024378488,0.00022679007 32 | 0.09616906,0.024117867,0.00025180454 33 | 0.09546294,0.024128024,0.00024075258 34 | 0.09497493,0.023742871,0.00025256176 35 | 0.09475855,0.02418723,0.00025191807 36 | 0.09531757,0.024756936,0.0002714412 37 | 0.09537713,0.024810271,0.00023217272 38 | 0.09561962,0.024780998,0.00022589866 39 | 0.09654302,0.024782034,7.652173e-05 40 | 0.09630073,0.025298612,9.21163e-05 41 | 0.09662149,0.14426856,0.00026029212 42 | 0.09686728,0.14513715,0.00022999448 43 | 0.09609901,0.14518209,0.00024841027 44 | 0.09598154,0.14368315,0.00024572093 45 | 0.09605242,0.14480826,0.00016160343 46 | 0.096517466,0.14432777,0.00024620374 47 | 0.09671872,0.14440295,0.00027114875 48 | 0.096409105,0.14464577,0.00018537932 49 | 0.096482836,0.14384456,0.0002947874 50 | 0.096630834,0.14411199,0.0003828561 51 | 0.09606406,0.26421747,0.000105197774 52 | 0.0961379,0.26396,8.837015e-05 53 | 0.095568545,0.26338413,5.094339e-05 54 | 0.0953888,0.26398072,8.470584e-05 55 | 0.09591115,0.26449782,0.00015155993 56 | 0.095831096,0.26395538,9.223971e-05 57 | 0.09625959,0.26374617,9.788009e-05 58 | 0.095573045,0.26327854,4.660676e-05 59 | 0.095469184,0.26400805,8.5914464e-05 60 | 0.09596244,0.26447156,0.00014411936 61 | 0.19286832,0.023980092,0.0006100533 62 | 0.1921519,0.0237871,0.00052581826 63 | 0.19172983,0.023773324,0.00059017946 64 | 0.1918397,0.023530303,0.000585391 65 | 0.1909068,0.023909008,0.0006791229 66 | 0.19174942,0.02432319,0.00063004636 67 | 0.19196041,0.024752246,0.00065182004 68 | 0.1916629,0.02455928,0.0005138707 69 | 0.19146986,0.023918405,0.00057124806 70 | 0.19218914,0.024072448,0.0004985125 71 | 0.191978,0.14454694,0.00042451662 72 | 0.19187297,0.14505923,0.00047598078 73 | 0.19233167,0.14467636,0.0003658522 74 | 0.19201368,0.14395253,0.00045362732 75 | 0.19143821,0.14464411,0.00038438087 76 | 0.19272213,0.14452827,0.0004995405 77 | 0.1922299,0.14405116,0.0005299263 78 | 0.1916603,0.14380988,0.00044485778 79 | 0.19251756,0.14511679,0.00037223744 80 | 0.19219904,0.14471291,0.00035352426 81 | 0.19375563,0.26468393,8.3092775e-05 82 | 0.19267076,0.26360333,9.083191e-05 83 | 0.191991,0.2645487,0.0002793484 84 | 0.19194305,0.26417413,0.00017547775 85 | 0.19245298,0.263725,9.6389456e-05 86 | 0.19235131,0.2640992,0.00012581251 87 | 0.19186836,0.26329505,4.2619035e-05 88 | 0.19188404,0.26371077,0.0001746356 89 | 0.19167182,0.2645364,0.00017780936 90 | 0.19124538,0.2637934,0.0002173817 91 | -------------------------------------------------------------------------------- /python/app/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Jcparkyn/dpoint/a108c19b9b240c1531b2d30ed47ba18bc604862e/python/app/__init__.py -------------------------------------------------------------------------------- /python/app/app.py: -------------------------------------------------------------------------------- 1 | # Parts of this file were scaffolded from https://github.com/vispy/vispy/blob/main/examples/scene/realtime_data/ex03b_data_sources_threaded_loop.py 2 | import datetime 3 | import json 4 | from pathlib import Path 5 | import time 6 | from typing import NamedTuple 7 | from PyQt6 import QtWidgets, QtCore, QtGui 8 | from PyQt6.QtCore import Qt 9 | 10 | import vispy 11 | from vispy import scene 12 | from vispy.io import read_mesh 13 | from vispy.scene import SceneCanvas, visuals 14 | import vispy.app 15 | from vispy.app import use_app 16 | from vispy.util import quaternion 17 | from vispy.visuals import transforms 18 | 19 | import numpy as np 20 | import queue 21 | import multiprocessing as mp 22 | from app.color_button import ColorButton 23 | from app.filter import DpointFilter, blend_new_data 24 | 25 | from app.marker_tracker import CameraReading, run_tracker 26 | from app.monitor_ble import StopCommand, StylusReading, monitor_ble 27 | 28 | CANVAS_SIZE = (1080, 1080) # (width, height) 29 | TRAIL_POINTS = 12000 30 | USE_3D_LINE = ( 31 | False # If true, uses a lower quality GL line renderer that supports 3D lines 32 | ) 33 | 34 | # Recording is only used for testing and evaluation of the system. 35 | # When enabled the data from the IMU and camera are saved to disk, so they can be replayed 36 | # offline with offline_ope.py and offline_playback,py. 37 | recording_enabled = mp.Value("b", False) 38 | app_start_datetime = datetime.datetime.now() 39 | recording_timestamp = app_start_datetime.strftime("%Y%m%d_%H%M%S") 40 | 41 | 42 | def append_line_point(line: np.ndarray, new_point: np.array): 43 | """Append new points to a line.""" 44 | # There are many faster ways to do this, but this solution works well enough 45 | line[:-1, :] = line[1:, :] 46 | line[-1, :] = new_point 47 | 48 | 49 | def get_line_color(line: np.ndarray): 50 | base_col = np.array([[0.0, 0.0, 0.0]], dtype=np.float32) 51 | pos_z = line[:, [2]] 52 | return np.hstack( 53 | [ 54 | np.tile(base_col, (line.shape[0], 1)), 55 | 1 - np.clip(pos_z * 400, 0, 1), 56 | ] 57 | ) 58 | 59 | 60 | def get_line_color_from_pressure(pressure: float, color=(0, 0, 0, 1)): 61 | col = np.array(color, dtype=np.float32) 62 | col[3] *= np.clip(pressure, 0, 1) 63 | return col 64 | 65 | 66 | class CameraUpdateData(NamedTuple): 67 | position_replace: list[np.ndarray] 68 | 69 | 70 | class StylusUpdateData(NamedTuple): 71 | position: np.ndarray 72 | orientation: np.ndarray 73 | pressure: float 74 | 75 | 76 | ViewUpdateData = CameraUpdateData | StylusUpdateData 77 | 78 | 79 | class CanvasWrapper: 80 | def __init__(self): 81 | self.canvas = SceneCanvas(size=CANVAS_SIZE, vsync=False) 82 | self.canvas.measure_fps() 83 | self.canvas.connect(self.on_key_press) 84 | self.grid = self.canvas.central_widget.add_grid() 85 | 86 | self.view_top = self.grid.add_view(0, 0, bgcolor="white") 87 | self.view_top.camera = scene.TurntableCamera( 88 | up="z", 89 | fov=0, 90 | center=(0.10, 0.13, 0), 91 | elevation=90, 92 | azimuth=0, 93 | scale_factor=0.3, 94 | ) 95 | vertices, faces, normals, texcoords = read_mesh("./mesh/pen.obj") 96 | self.pen_mesh = visuals.Mesh( 97 | vertices, faces, color=(0.8, 0.8, 0.8, 1), parent=self.view_top.scene 98 | ) 99 | self.pen_mesh.transform = transforms.MatrixTransform() 100 | self.line_color = (0, 0, 0, 1) 101 | 102 | pen_tip = visuals.XYZAxis(parent=self.pen_mesh) 103 | pen_tip.transform = transforms.MatrixTransform( 104 | vispy.util.transforms.scale([0.01, 0.01, 0.01]) 105 | ) 106 | 107 | self.line_data_pos = np.zeros((TRAIL_POINTS, 3), dtype=np.float32) 108 | self.line_data_col = np.zeros((TRAIL_POINTS, 4), dtype=np.float32) 109 | # agg looks much better than gl, but only works with 2D data. 110 | if USE_3D_LINE: 111 | self.trail_line = visuals.Line( 112 | width=1, 113 | parent=self.view_top.scene, 114 | method="gl", 115 | ) 116 | else: 117 | self.trail_line = visuals.Line( 118 | width=3, parent=self.view_top.scene, method="agg", antialias=False 119 | ) 120 | 121 | axis = scene.visuals.XYZAxis(parent=self.view_top.scene) 122 | axis.transform = transforms.MatrixTransform() 123 | axis.transform.scale([0.02, 0.02, 0.02]) 124 | # This is broken for now, see https://github.com/vispy/vispy/issues/2363 125 | # grid = scene.visuals.GridLines(parent=self.view_top.scene) 126 | 127 | def update_data(self, new_data: ViewUpdateData): 128 | match new_data: 129 | case StylusUpdateData( 130 | position=pos, orientation=orientation, pressure=pressure 131 | ): 132 | orientation_quat = quaternion.Quaternion(*orientation).inverse() 133 | self.pen_mesh.transform.matrix = ( 134 | orientation_quat.get_matrix() @ vispy.util.transforms.translate(pos) 135 | ) 136 | col = get_line_color_from_pressure(pressure, self.line_color) 137 | append_line_point(self.line_data_pos, pos) 138 | append_line_point(self.line_data_col, col) 139 | case CameraUpdateData(position_replace): 140 | if len(position_replace) == 0: 141 | return 142 | view = self.line_data_pos[-len(position_replace) :, :] 143 | view[:, :] = blend_new_data(view, position_replace, 0.5) 144 | self.refresh_line() 145 | 146 | def refresh_line(self): 147 | # Skip rendering points where both ends have zero alpha 148 | pressure_mask = self.line_data_col[:, 3] > 0 149 | pressure_mask = ( 150 | pressure_mask | np.roll(pressure_mask, 1) | np.roll(pressure_mask, -1) 151 | ) 152 | pressure_mask[0:2] = True # To ensure we always have at least one line segment 153 | pos = self.line_data_pos[pressure_mask, :] 154 | col = self.line_data_col[pressure_mask, :] 155 | self.trail_line.set_data( 156 | pos if USE_3D_LINE else pos[:, 0:2], 157 | color=col, 158 | ) 159 | 160 | def clear_line(self): 161 | self.line_data_col[:, 3] *= 0 162 | self.refresh_line() 163 | 164 | def set_line_color(self, col: QtGui.QColor): 165 | self.line_color = col.getRgbF() # (col.redF, col.greenF, col.blueF) 166 | 167 | def set_line_width(self, width: float): 168 | self.trail_line.set_data(width=width) 169 | 170 | def clear_last_stroke(self): 171 | diff = np.diff( 172 | (self.line_data_col[:, 3] > 0).astype(np.int8) 173 | ) # 1 when stroke starts, -1 when it ends 174 | start_indices = np.where(diff == 1)[0] 175 | if len(start_indices) > 0: 176 | last_stroke_index = start_indices[-1] 177 | print(TRAIL_POINTS - last_stroke_index) 178 | self.line_data_col[last_stroke_index:, 3] *= 0 179 | self.refresh_line() 180 | 181 | def on_key_press(self, e: vispy.app.canvas.KeyEvent): 182 | # if e.key == "R": 183 | # if "Control" in e.modifiers: 184 | # recording_enabled.value = True 185 | # print("Recording enabled") 186 | # else: 187 | # recording_enabled.value = False 188 | # print("Recording disabled") 189 | if e.key == "C": 190 | self.clear_line() 191 | elif e.key == "Z" and "Control" in e.modifiers: 192 | self.clear_last_stroke() 193 | 194 | 195 | class MainWindow(QtWidgets.QMainWindow): 196 | closing = QtCore.pyqtSignal() 197 | 198 | def __init__(self, canvas_wrapper: CanvasWrapper, *args, **kwargs): 199 | super().__init__(*args, **kwargs) 200 | 201 | central_widget = QtWidgets.QWidget() 202 | main_layout = QtWidgets.QVBoxLayout() 203 | 204 | self._canvas_wrapper = canvas_wrapper 205 | main_layout.addWidget(self._canvas_wrapper.canvas.native) 206 | 207 | color_button = ColorButton("Line color") 208 | color_button.colorChanged.connect(canvas_wrapper.set_line_color) 209 | color_button.setFocusPolicy(Qt.FocusPolicy.NoFocus) 210 | clear_button = QtWidgets.QPushButton("Clear (C)") 211 | clear_button.clicked.connect(canvas_wrapper.clear_line) 212 | clear_button.setFocusPolicy(Qt.FocusPolicy.NoFocus) 213 | undo_button = QtWidgets.QPushButton("Undo (Ctrl+Z)") 214 | undo_button.clicked.connect(canvas_wrapper.clear_last_stroke) 215 | undo_button.setFocusPolicy(Qt.FocusPolicy.NoFocus) 216 | self.line_width_label = QtWidgets.QLabel("") 217 | self.line_width_label.setMinimumWidth(80) 218 | self.line_width_slider = QtWidgets.QSlider(QtCore.Qt.Orientation.Horizontal) 219 | self.line_width_slider.setRange(1, 20) 220 | self.line_width_slider.valueChanged.connect(self.line_width_changed) 221 | self.line_width_slider.setValue(2) 222 | bottom_toolbar = QtWidgets.QHBoxLayout() 223 | bottom_toolbar.addWidget(clear_button) 224 | bottom_toolbar.addWidget(undo_button) 225 | bottom_toolbar.addWidget(color_button) 226 | bottom_toolbar.addWidget(QtWidgets.QLabel("Thickness:")) 227 | bottom_toolbar.addWidget(self.line_width_slider) 228 | bottom_toolbar.addWidget(self.line_width_label) 229 | main_layout.addLayout(bottom_toolbar) 230 | 231 | central_widget.setLayout(main_layout) 232 | self.setCentralWidget(central_widget) 233 | 234 | def line_width_changed(self, width): 235 | self.line_width_label.setText(str(width)) 236 | self._canvas_wrapper.set_line_width(width) 237 | 238 | def closeEvent(self, event): 239 | print("Closing main window!") 240 | self.closing.emit() 241 | return super().closeEvent(event) 242 | 243 | 244 | class QueueConsumer(QtCore.QObject): 245 | new_data = QtCore.pyqtSignal(object) 246 | finished = QtCore.pyqtSignal() 247 | 248 | def __init__( 249 | self, 250 | tracker_queue: "mp.Queue[CameraReading]", 251 | imu_queue: "mp.Queue[StylusReading]", 252 | parent=None, 253 | ): 254 | super().__init__(parent) 255 | self._should_end = False 256 | self._tracker_queue = tracker_queue 257 | self._imu_queue = imu_queue 258 | self._filter = DpointFilter(dt=1 / 145, smoothing_length=8, camera_delay=5) 259 | self._recorded_data_stylus = [] 260 | self._recorded_data_camera = [] 261 | 262 | def run_queue_consumer(self): 263 | print("Queue consumer is starting") 264 | samples_since_camera = 1000 265 | pressure_baseline = 0.017 # Approximate measured value for initial estimate 266 | pressure_avg_factor = 0.1 # Factor for exponential moving average 267 | pressure_range = 0.02 268 | pressure_offset = ( 269 | 0.002 # Offset so that small positive numbers are treated as zero 270 | ) 271 | while True: 272 | if self._should_end: 273 | print("Data source saw that it was told to stop") 274 | break 275 | 276 | try: 277 | while self._tracker_queue.qsize() > 2: 278 | self._tracker_queue.get() 279 | reading = self._tracker_queue.get_nowait() 280 | if recording_enabled.value: 281 | self._recorded_data_camera.append( 282 | (time.time_ns() // 1_000_000, reading) 283 | ) 284 | samples_since_camera = 0 285 | smoothed_tip_pos = self._filter.update_camera( 286 | reading.position.flatten(), reading.orientation_mat 287 | ) 288 | self.new_data.emit(CameraUpdateData(position_replace=smoothed_tip_pos)) 289 | except queue.Empty: 290 | pass 291 | 292 | while self._imu_queue.qsize() > 0: 293 | reading = self._imu_queue.get() 294 | samples_since_camera += 1 295 | if samples_since_camera > 10: 296 | continue 297 | if recording_enabled.value: 298 | self._recorded_data_stylus.append( 299 | (time.time_ns() // 1_000_000, reading) 300 | ) 301 | self._filter.update_imu(reading.accel, reading.gyro) 302 | position, orientation = self._filter.get_tip_pose() 303 | zpos = position[2] 304 | if zpos > 0.007: 305 | # calibrate pressure baseline using current pressure reading 306 | pressure_baseline = ( 307 | pressure_baseline * (1 - pressure_avg_factor) 308 | + reading.pressure * pressure_avg_factor 309 | ) 310 | self.new_data.emit( 311 | StylusUpdateData( 312 | position=position, 313 | orientation=orientation, 314 | pressure=( 315 | reading.pressure - pressure_baseline - pressure_offset 316 | ) 317 | / pressure_range, 318 | ) 319 | ) 320 | 321 | print("Queue consumer finishing") 322 | 323 | if self._recorded_data_stylus: 324 | file1 = Path(f"recordings/{recording_timestamp}/stylus_data.json") 325 | file1.parent.mkdir(parents=True, exist_ok=True) 326 | with file1.open("x") as f: 327 | json.dump( 328 | [ 329 | dict(t=t, data=reading.to_json()) 330 | for t, reading in self._recorded_data_stylus 331 | ], 332 | f, 333 | ) 334 | file2 = Path(f"recordings/{recording_timestamp}/camera_data_original.json") 335 | with file2.open("x") as f: 336 | json.dump( 337 | [ 338 | dict(t=t, data=reading.to_json()) 339 | for t, reading in self._recorded_data_camera 340 | ], 341 | f, 342 | ) 343 | 344 | self.finished.emit() 345 | 346 | def stop_data(self): 347 | print("Data source is quitting...") 348 | self._should_end = True 349 | 350 | 351 | def run_tracker_with_queue(queue: mp.Queue, *args): 352 | run_tracker(lambda reading: queue.put(reading, block=False), *args) 353 | 354 | 355 | def main(): 356 | np.set_printoptions( 357 | precision=3, suppress=True, formatter={"float": "{: >5.2f}".format} 358 | ) 359 | app = use_app("pyqt6") 360 | app.create() 361 | 362 | tracker_queue = mp.Queue() 363 | ble_queue = mp.Queue() 364 | ble_command_queue = mp.Queue() 365 | canvas_wrapper = CanvasWrapper() 366 | win = MainWindow(canvas_wrapper) 367 | win.resize(*CANVAS_SIZE) 368 | data_thread = QtCore.QThread(parent=win) 369 | 370 | queue_consumer = QueueConsumer(tracker_queue, ble_queue) 371 | queue_consumer.moveToThread(data_thread) 372 | 373 | camera_process = mp.Process( 374 | target=run_tracker_with_queue, 375 | args=(tracker_queue, recording_enabled, recording_timestamp), 376 | daemon=False, 377 | ) 378 | camera_process.start() 379 | 380 | ble_process = mp.Process( 381 | target=monitor_ble, args=(ble_queue, ble_command_queue), daemon=False 382 | ) 383 | ble_process.start() 384 | 385 | # update the visualization when there is new data 386 | queue_consumer.new_data.connect(canvas_wrapper.update_data) 387 | # start data generation when the thread is started 388 | data_thread.started.connect(queue_consumer.run_queue_consumer) 389 | # if the data source finishes before the window is closed, kill the thread 390 | queue_consumer.finished.connect( 391 | data_thread.quit, QtCore.Qt.ConnectionType.DirectConnection 392 | ) 393 | # if the window is closed, tell the data source to stop 394 | win.closing.connect( 395 | queue_consumer.stop_data, QtCore.Qt.ConnectionType.DirectConnection 396 | ) 397 | win.closing.connect( 398 | lambda: ble_command_queue.put(StopCommand()), 399 | QtCore.Qt.ConnectionType.DirectConnection, 400 | ) 401 | # when the thread has ended, delete the data source from memory 402 | data_thread.finished.connect(queue_consumer.deleteLater) 403 | 404 | try: 405 | win.show() 406 | data_thread.start() 407 | app.run() 408 | finally: 409 | camera_process.terminate() 410 | ble_process.terminate() 411 | print("Waiting for data source to close gracefully...") 412 | data_thread.wait(1000) 413 | -------------------------------------------------------------------------------- /python/app/camera_cov.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | import cv2 as cv 3 | from pyquaternion import Quaternion 4 | 5 | 6 | def projection_matrix(rvec, tvec, camera_matrix): 7 | R = cv.Rodrigues(rvec)[0] 8 | result = np.matmul(camera_matrix, np.hstack((R, tvec))) 9 | assert result.shape == (3, 4) 10 | return result 11 | 12 | 13 | def point_dWorld_dPose(q_object: np.ndarray, object_point: np.ndarray): 14 | assert q_object.shape == (4,) 15 | assert len(object_point) == 3 16 | o1, o2, o3 = object_point.flatten() 17 | q1, q2, q3, q4 = q_object 18 | 19 | # fmt: off 20 | return np.array([ 21 | [2*q3*o3 - 2*q4*o2, 2*q3*o2 + 2*q4*o3, 2*q1*o3 + 2*q2*o2 - 4*q3*o1, 2*q2*o3 - 2*q1*o2 - 4*q4*o1, 1, 0, 0], 22 | [2*q4*o1 - 2*q2*o3, 2*q3*o1 - 4*q2*o2 - 2*q1*o3, 2*q2*o1 + 2*q4*o3, 2*q1*o1 + 2*q3*o3 - 4*q4*o2, 0, 1, 0], 23 | [2*q2*o2 - 2*q3*o1, 2*q1*o2 - 4*q2*o3 + 2*q4*o1, 2*q4*o2 - 4*q3*o3 - 2*q1*o1, 2*q2*o1 + 2*q3*o2, 0, 0, 1], 24 | [0] * 7, 25 | ]) 26 | # fmt: on 27 | 28 | 29 | def duv_dxyz(xyz): 30 | x, y, z = xyz.flatten() 31 | return np.array([[1 / z, 0, -x / z**2], [0, 1 / z, -y / z**2]]) 32 | 33 | 34 | def point_dUV_dPose( 35 | q_object: np.ndarray, 36 | t_object: np.ndarray, 37 | object_point: np.ndarray, 38 | proj_matrix: np.ndarray, 39 | ): 40 | # print("p", object_point) 41 | dxyz_dPose = proj_matrix @ point_dWorld_dPose(q_object, object_point) 42 | R = Quaternion(q_object).rotation_matrix 43 | world_point = R @ object_point.reshape(3, 1) + t_object.reshape(3, 1) 44 | assert world_point.shape == (3, 1) 45 | xyz = proj_matrix @ np.vstack((world_point, 1)) 46 | return duv_dxyz(xyz) @ dxyz_dPose 47 | 48 | 49 | def df_dPose( 50 | q_object: np.ndarray, 51 | t_object: np.ndarray, 52 | object_points: list[np.ndarray], 53 | proj_matrix: np.ndarray, 54 | ): 55 | assert q_object.shape == (4,) 56 | result = np.vstack([ 57 | point_dUV_dPose(q_object, t_object, p, proj_matrix) for p in object_points 58 | ]) 59 | assert result.shape == (2 * len(object_points), 7) 60 | return result 61 | 62 | 63 | def camera_measurement_cov( 64 | q_object: np.ndarray, 65 | t_object: np.ndarray, 66 | object_points: np.ndarray, # one row per point 67 | camera_rvec: np.ndarray, 68 | camera_tvec: np.ndarray, 69 | camera_matrix: np.ndarray, 70 | corner_stdev: float, 71 | ) -> np.ndarray: 72 | proj_matrix = projection_matrix(camera_rvec, camera_tvec, camera_matrix) 73 | J = df_dPose(q_object, t_object, object_points, proj_matrix) 74 | variance = corner_stdev**2 75 | return variance * np.linalg.pinv(J.T @ J) 76 | -------------------------------------------------------------------------------- /python/app/color_button.py: -------------------------------------------------------------------------------- 1 | from PyQt6 import QtCore, QtGui, QtWidgets 2 | from PyQt6.QtCore import Qt, pyqtSignal 3 | 4 | # This file is from https://www.pythonguis.com/widgets/qcolorbutton-a-color-selector-tool-for-pyqt/ 5 | # Used under MIT license 6 | 7 | class ColorButton(QtWidgets.QPushButton): 8 | ''' 9 | Custom Qt Widget to show a chosen color. 10 | 11 | Left-clicking the button shows the color-chooser, while 12 | right-clicking resets the color to None (no-color). 13 | ''' 14 | 15 | colorChanged = pyqtSignal(object) 16 | 17 | def __init__(self, *args, color=None, **kwargs): 18 | super(ColorButton, self).__init__(*args, **kwargs) 19 | 20 | self._color = None 21 | self._default = color 22 | self.pressed.connect(self.onColorPicker) 23 | 24 | # Set the initial/default state. 25 | self.setColor(self._default) 26 | 27 | def setColor(self, color): 28 | if color != self._color: 29 | self._color = color 30 | self.colorChanged.emit(color) 31 | 32 | def color(self): 33 | return self._color 34 | 35 | def onColorPicker(self): 36 | ''' 37 | Show color-picker dialog to select color. 38 | 39 | Qt will use the native dialog by default. 40 | 41 | ''' 42 | dlg = QtWidgets.QColorDialog(self) 43 | if self._color: 44 | dlg.setCurrentColor(QtGui.QColor(self._color)) 45 | 46 | if dlg.exec(): 47 | self.setColor(dlg.currentColor()) 48 | 49 | def mousePressEvent(self, e): 50 | if e.button() == Qt.MouseButton.RightButton: 51 | self.setColor(self._default) 52 | 53 | return super(ColorButton, self).mousePressEvent(e) -------------------------------------------------------------------------------- /python/app/dimensions.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | 3 | 4 | IMU_OFFSET = (0.0, -0.01, 0.004) # position of IMU relative to the top of the stylus 5 | STYLUS_LENGTH = 0.1686 # length from the tip to the top of the stylus 6 | 7 | def rotateY(angle: float, point: np.ndarray) -> np.ndarray: 8 | c, s = np.cos(angle), np.sin(angle) 9 | rotation_matrix = np.array([[c, 0, s], [0, 1, 0], [-s, 0, c]], dtype=np.float32) 10 | return np.dot(rotation_matrix, point) 11 | 12 | 13 | # markerLength = 0.015 14 | def getMarkerCorners(markerLength: float): 15 | return np.array( 16 | [ 17 | [-markerLength / 2, markerLength / 2, 0], 18 | [markerLength / 2, markerLength / 2, 0], 19 | [markerLength / 2, -markerLength / 2, 0], 20 | [-markerLength / 2, -markerLength / 2, 0], 21 | ], 22 | dtype=np.float32, 23 | ) 24 | 25 | 26 | def getCornersPS( 27 | origin: np.ndarray, angleY: float, markerLength: float = 0.013*0.97 28 | ) -> np.ndarray: 29 | cornersWS = getMarkerCorners(markerLength) + origin 30 | rotated_corners = np.apply_along_axis(lambda x: rotateY(angleY, x), 1, cornersWS) 31 | return rotated_corners - IMU_OFFSET 32 | 33 | 34 | def deg2rad(deg: float) -> float: 35 | return deg * np.pi / 180 36 | 37 | idealMarkerPositions = { 38 | 99: getCornersPS(np.array([0, -0.01, 0.01], dtype=np.float32), deg2rad(135)), 39 | 98: getCornersPS(np.array([0, -0.01, 0.01], dtype=np.float32), deg2rad(225)), 40 | 97: getCornersPS(np.array([0, -0.01, 0.01], dtype=np.float32), deg2rad(315)), 41 | 96: getCornersPS(np.array([0, -0.01, 0.01], dtype=np.float32), deg2rad(45)), 42 | 95: getCornersPS(np.array([0, -0.0395, 0.01], dtype=np.float32), deg2rad(90)), 43 | 94: getCornersPS(np.array([0, -0.0395, 0.01], dtype=np.float32), deg2rad(180)), 44 | 93: getCornersPS(np.array([0, -0.0395, 0.01], dtype=np.float32), deg2rad(270)), 45 | 92: getCornersPS(np.array([0, -0.0395, 0.01], dtype=np.float32), deg2rad(0)), 46 | } -------------------------------------------------------------------------------- /python/app/filter.py: -------------------------------------------------------------------------------- 1 | from collections import deque 2 | from typing import Deque, Tuple 3 | import numpy as np 4 | from numpy import typing as npt 5 | from pyquaternion import Quaternion 6 | from numba.typed.typedlist import List 7 | 8 | from app.dimensions import IMU_OFFSET, STYLUS_LENGTH 9 | from app.filter_core import ( 10 | STATE_SIZE, 11 | FilterState, 12 | HistoryItem, 13 | SmoothingHistoryItem, 14 | ekf_predict, 15 | ekf_smooth, 16 | fuse_camera, 17 | fuse_imu, 18 | i_acc, 19 | i_accbias, 20 | i_av, 21 | i_gyrobias, 22 | i_pos, 23 | i_quat, 24 | i_vel, 25 | ) 26 | 27 | Mat = npt.NDArray[np.float64] 28 | 29 | additive_noise = np.zeros(STATE_SIZE) 30 | additive_noise[i_pos] = 1e-6 31 | additive_noise[i_vel] = 4e-4 32 | additive_noise[i_acc] = 100 33 | additive_noise[i_av] = 50 34 | additive_noise[i_quat] = 1e-5 35 | additive_noise[i_accbias] = 0.5e-4 36 | additive_noise[i_gyrobias] = 1e-5 37 | Q = np.diag(additive_noise) 38 | 39 | accel_noise = 2e-3 40 | gyro_noise = 5e-4 41 | imu_noise = np.diag([accel_noise] * 3 + [gyro_noise] * 3) 42 | camera_noise_pos = 1e-6 43 | camera_noise_or = 1e-4 44 | camera_noise = np.diag([camera_noise_pos] * 3 + [camera_noise_or] * 4) 45 | 46 | 47 | def initial_state(position=None, orientation=None): 48 | state = np.zeros(STATE_SIZE, dtype=np.float64) 49 | state[i_quat] = [1, 0, 0, 0] 50 | if position is not None: 51 | state[i_pos] = position.flatten() 52 | if orientation is not None: 53 | state[i_quat] = orientation.flatten() 54 | covdiag = np.ones(STATE_SIZE, dtype=np.float64) * 0.0001 55 | covdiag[i_accbias] = 1e-2 56 | covdiag[i_gyrobias] = 1e-4 57 | statecov = np.diag(covdiag) 58 | return FilterState(state, statecov) 59 | 60 | 61 | def get_tip_pose(state: Mat) -> Tuple[Mat, Mat]: 62 | pos = state[i_pos] 63 | orientation = state[i_quat] 64 | orientation_quat = Quaternion(array=orientation) 65 | tip_pos = pos - orientation_quat.rotate( 66 | np.array([0, STYLUS_LENGTH, 0]) + IMU_OFFSET 67 | ) 68 | return (tip_pos, orientation) 69 | 70 | 71 | def get_orientation_quat(orientation_mat_opencv: Mat): 72 | return Quaternion(matrix=orientation_mat_opencv).normalised 73 | 74 | 75 | def nearest_quaternion(reference: Mat, new: Mat): 76 | """ 77 | Find the sign for new that makes it as close to reference as possible. 78 | Changing the sign of a quaternion does not change its rotation, but affects 79 | the difference from the reference quaternion. 80 | """ 81 | error1 = np.linalg.norm(reference - new) 82 | error2 = np.linalg.norm(reference + new) 83 | return (new, error1) if error1 < error2 else (-new, error2) 84 | 85 | 86 | def blend_new_data(old: np.ndarray, new: np.ndarray, alpha: float): 87 | """Blends between old and new based on a power curve. 88 | Abruptly stopping smoothing can sometimes cause jumps, so we fade out the correction. 89 | This isn't mathematically optimal, but it looks a bit nicer. 90 | """ 91 | N = old.shape[0] 92 | # This is just an arbitrary function that starts close to zero and ends at one. 93 | mix_factor = np.linspace(1 / 2 / N, 1, N)[:, np.newaxis] ** alpha 94 | return old * (1 - mix_factor) + new * mix_factor 95 | 96 | 97 | class DpointFilter: 98 | history: Deque[HistoryItem] 99 | 100 | def __init__(self, dt, smoothing_length: int, camera_delay: int): 101 | self.history = deque() 102 | self.fs = initial_state() 103 | self.dt = dt 104 | self.smoothing_length = smoothing_length 105 | self.camera_delay = camera_delay 106 | 107 | def update_imu(self, accel: np.ndarray, gyro: np.ndarray): 108 | predicted = ekf_predict(self.fs, self.dt, Q) 109 | self.fs = fuse_imu(predicted, accel, gyro, imu_noise) 110 | self.history.append( 111 | HistoryItem( 112 | self.fs.state, 113 | self.fs.statecov, 114 | predicted.state, 115 | predicted.statecov, 116 | accel=accel, 117 | gyro=gyro, 118 | ) 119 | ) 120 | max_history_len = self.smoothing_length + self.camera_delay + 1 121 | if len(self.history) > max_history_len: 122 | self.history.popleft() 123 | 124 | def update_camera( 125 | self, imu_pos: np.ndarray, orientation_mat: np.ndarray 126 | ) -> list[np.ndarray]: 127 | if len(self.history) == 0: 128 | return [] 129 | 130 | # Rollback and store recent IMU measurements 131 | replay: Deque[HistoryItem] = deque() 132 | for _ in range(min(len(self.history) - 1, self.camera_delay)): 133 | replay.appendleft(self.history.pop()) 134 | 135 | # Fuse camera in its rightful place 136 | h = self.history[-1] 137 | fs = FilterState(h.updated_state, h.updated_statecov) 138 | or_quat = get_orientation_quat(orientation_mat) 139 | or_quat_smoothed, or_error = nearest_quaternion( 140 | fs.state[i_quat], or_quat.elements 141 | ) 142 | pos_error = np.linalg.norm(imu_pos - fs.state[i_pos]) 143 | if pos_error > 0.05 or or_error > 0.4: 144 | print(f"Resetting state, errors: {pos_error}, {or_error}") 145 | self.fs = initial_state(imu_pos, or_quat_smoothed) 146 | self.history = deque() 147 | return [] 148 | self.fs = fuse_camera(fs, imu_pos, or_quat_smoothed, camera_noise) 149 | previous = self.history.pop() # Replace last item 150 | self.history.append( 151 | HistoryItem( 152 | self.fs.state, 153 | self.fs.statecov, 154 | previous.predicted_state, 155 | previous.predicted_statecov, 156 | # accel=previous.accel, 157 | # gyro=previous.gyro, 158 | ) 159 | ) 160 | 161 | # Apply smoothing to the rest of the history. 162 | # We could also smooth the future measurements, but that would be slower 163 | # for very little benefit (the final estimate won't change). 164 | smoothed_estimates = ekf_smooth( 165 | List( 166 | [ 167 | SmoothingHistoryItem( 168 | h.updated_state, 169 | h.updated_statecov, 170 | h.predicted_state, 171 | h.predicted_statecov, 172 | ) 173 | for h in self.history 174 | ] 175 | ), 176 | self.dt, 177 | ) 178 | 179 | # Replay the IMU measurements 180 | predicted_estimates = [] 181 | for item in replay: 182 | assert item.accel is not None 183 | assert item.gyro is not None 184 | self.update_imu(item.accel, item.gyro) 185 | predicted_estimates.append(self.fs.state) 186 | return [ 187 | get_tip_pose(state)[0] for state in smoothed_estimates + predicted_estimates 188 | ] 189 | 190 | def get_tip_pose(self) -> Tuple[Mat, Mat]: 191 | return get_tip_pose(self.fs.state) 192 | -------------------------------------------------------------------------------- /python/app/filter_core.py: -------------------------------------------------------------------------------- 1 | from typing import NamedTuple, Optional 2 | 3 | import numpy as np 4 | from numba import njit 5 | from numpy import typing as npt 6 | from pyquaternion import Quaternion 7 | 8 | # This file is separate from filter.py, so that it doesn't need to be re-compiled so often. 9 | 10 | Mat = npt.NDArray[np.float64] 11 | 12 | i_quat = np.array([0, 1, 2, 3]) 13 | i_av = np.array([4, 5, 6]) 14 | i_pos = np.array([7, 8, 9]) 15 | i_vel = np.array([10, 11, 12]) 16 | i_acc = np.array([13, 14, 15]) 17 | i_accbias = np.array([16, 17, 18]) 18 | i_gyrobias = np.array([19, 20, 21]) 19 | 20 | STATE_SIZE = 22 21 | GRAVITY_VECTOR = np.array([0, 0, -9.81]) 22 | 23 | 24 | class FilterState(NamedTuple): 25 | state: Mat 26 | statecov: Mat 27 | 28 | 29 | class SmoothingHistoryItem(NamedTuple): 30 | updated_state: Mat 31 | updated_statecov: Mat 32 | predicted_state: Mat 33 | predicted_statecov: Mat 34 | 35 | 36 | class HistoryItem(NamedTuple): 37 | updated_state: Mat 38 | updated_statecov: Mat 39 | predicted_state: Mat 40 | predicted_statecov: Mat 41 | accel: Optional[Mat] = None 42 | gyro: Optional[Mat] = None 43 | 44 | 45 | @njit(cache=True) 46 | def state_transition(state: Mat = np.array([])): 47 | av = state[i_av] 48 | quat = state[i_quat] 49 | q0, q1, q2, q3 = quat 50 | acc = state[i_acc] 51 | vel = state[i_vel] 52 | 53 | qdot = np.array( 54 | [ 55 | np.dot(av, np.array([-q1, -q2, -q3])) / 2, 56 | np.dot(av, np.array([q0, -q3, q2])) / 2, 57 | np.dot(av, np.array([q3, q0, -q1])) / 2, 58 | np.dot(av, np.array([-q2, q1, q0])) / 2, 59 | ] 60 | ) 61 | pdot = vel 62 | vdot = acc 63 | statedot = np.zeros_like(state) 64 | statedot[i_quat] = qdot 65 | statedot[i_av] = 0 66 | statedot[i_pos] = pdot 67 | statedot[i_vel] = vdot 68 | statedot[i_acc] = 0 69 | statedot[i_accbias] = 0 70 | statedot[i_gyrobias] = 0 71 | return statedot 72 | 73 | 74 | @njit(cache=True) 75 | def state_transition_jacobian(state: Mat): 76 | av = state[i_av] 77 | avx, avy, avz = av 78 | quat = state[i_quat] 79 | q0, q1, q2, q3 = quat 80 | 81 | N = len(state) 82 | 83 | # Orientation 84 | dorientfuncdx = np.zeros((4, N), dtype=state.dtype) 85 | dorientfuncdx[0, i_quat] = [0, -avx / 2, -avy / 2, -avz / 2] 86 | dorientfuncdx[0, i_av] = [-q1 / 2, -q2 / 2, -q3 / 2] 87 | dorientfuncdx[1, i_quat] = [avx / 2, 0, avz / 2, -avy / 2] 88 | dorientfuncdx[1, i_av] = [q0 / 2, -q3 / 2, q2 / 2] 89 | dorientfuncdx[2, i_quat] = [avy / 2, -avz / 2, 0, avx / 2] 90 | dorientfuncdx[2, i_av] = [q3 / 2, q0 / 2, -q1 / 2] 91 | dorientfuncdx[3, i_quat] = [avz / 2, avy / 2, -avx / 2, 0] 92 | dorientfuncdx[3, i_av] = [-q2 / 2, q1 / 2, q0 / 2] 93 | 94 | # Position 95 | dposfuncdx = np.zeros((3, N), dtype=state.dtype) 96 | dposfuncdx[0, i_vel] = [1, 0, 0] 97 | dposfuncdx[1, i_vel] = [0, 1, 0] 98 | dposfuncdx[2, i_vel] = [0, 0, 1] 99 | 100 | # Velocity 101 | dvelfuncdx = np.zeros((3, N), dtype=state.dtype) 102 | dvelfuncdx[0, i_acc] = [1, 0, 0] 103 | dvelfuncdx[1, i_acc] = [0, 1, 0] 104 | dvelfuncdx[2, i_acc] = [0, 0, 1] 105 | 106 | dfdx = np.zeros((STATE_SIZE, STATE_SIZE), dtype=state.dtype) 107 | dfdx[i_quat, :] = dorientfuncdx 108 | dfdx[i_pos, :] = dposfuncdx 109 | dfdx[i_vel, :] = dvelfuncdx 110 | return dfdx 111 | 112 | 113 | @njit(cache=True) 114 | def imu_measurement(state: Mat): 115 | av = state[i_av] 116 | acc = state[i_acc] 117 | quat = state[i_quat] 118 | q0, q1, q2, q3 = quat 119 | accbias = state[i_accbias] 120 | 121 | m_gyro = av + state[i_gyrobias] 122 | mj_gyro = np.zeros((3, len(state))) 123 | mj_gyro[:, i_av] = np.eye(3) 124 | mj_gyro[:, i_gyrobias] = np.eye(3) 125 | 126 | grav = GRAVITY_VECTOR 127 | 128 | m_accel = np.array( 129 | [ 130 | accbias[0] 131 | - (acc[0] - grav[0]) * (2 * q0**2 + 2 * q1**2 - 1) 132 | - (acc[1] - grav[1]) * (2 * q0 * q3 + 2 * q1 * q2) 133 | + (acc[2] - grav[2]) * (2 * q0 * q2 - 2 * q1 * q3), 134 | accbias[1] 135 | - (acc[1] - grav[1]) * (2 * q0**2 + 2 * q2**2 - 1) 136 | + (acc[0] - grav[0]) * (2 * q0 * q3 - 2 * q1 * q2) 137 | - (acc[2] - grav[2]) * (2 * q0 * q1 + 2 * q2 * q3), 138 | accbias[2] 139 | - (acc[2] - grav[2]) * (2 * q0**2 + 2 * q3**2 - 1) 140 | - (acc[0] - grav[0]) * (2 * q0 * q2 + 2 * q1 * q3) 141 | + (acc[1] - grav[1]) * (2 * q0 * q1 - 2 * q2 * q3), 142 | ] 143 | ) 144 | 145 | mj_accel = np.zeros((3, len(state))) 146 | mj_accel[0, i_quat] = [ 147 | 2 * q2 * (acc[2] - grav[2]) 148 | - 2 * q3 * (acc[1] - grav[1]) 149 | - 4 * q0 * (acc[0] - grav[0]), 150 | -4 * q1 * (acc[0] - grav[0]) 151 | - 2 * q2 * (acc[1] - grav[1]) 152 | - 2 * q3 * (acc[2] - grav[2]), 153 | 2 * q0 * (acc[2] - grav[2]) - 2 * q1 * (acc[1] - grav[1]), 154 | -2 * q0 * (acc[1] - grav[1]) - 2 * q1 * (acc[2] - grav[2]), 155 | ] 156 | mj_accel[1, i_quat] = [ 157 | 2 * q3 * (acc[0] - grav[0]) 158 | - 4 * q0 * (acc[1] - grav[1]) 159 | - 2 * q1 * (acc[2] - grav[2]), 160 | -2 * q2 * (acc[0] - grav[0]) - 2 * q0 * (acc[2] - grav[2]), 161 | -2 * q1 * (acc[0] - grav[0]) 162 | - 4 * q2 * (acc[1] - grav[1]) 163 | - 2 * q3 * (acc[2] - grav[2]), 164 | 2 * q0 * (acc[0] - grav[0]) - 2 * q2 * (acc[2] - grav[2]), 165 | ] 166 | mj_accel[2, i_quat] = [ 167 | 2 * q1 * (acc[1] - grav[1]) 168 | - 2 * q2 * (acc[0] - grav[0]) 169 | - 4 * q0 * (acc[2] - grav[2]), 170 | 2 * q0 * (acc[1] - grav[1]) - 2 * q3 * (acc[0] - grav[0]), 171 | -2 * q0 * (acc[0] - grav[0]) - 2 * q3 * (acc[1] - grav[1]), 172 | -2 * q1 * (acc[0] - grav[0]) 173 | - 2 * q2 * (acc[1] - grav[1]) 174 | - 4 * q3 * (acc[2] - grav[2]), 175 | ] 176 | mj_accel[0, i_acc] = [ 177 | 1 - 2 * q1**2 - 2 * q0**2, 178 | -2 * q0 * q3 - 2 * q1 * q2, 179 | 2 * q0 * q2 - 2 * q1 * q3, 180 | ] 181 | mj_accel[1, i_acc] = [ 182 | 2 * q0 * q3 - 2 * q1 * q2, 183 | 1 - 2 * q2**2 - 2 * q0**2, 184 | -2 * q0 * q1 - 2 * q2 * q3, 185 | ] 186 | mj_accel[2, i_acc] = [ 187 | -2 * q0 * q2 - 2 * q1 * q3, 188 | 2 * q0 * q1 - 2 * q2 * q3, 189 | 1 - 2 * q3**2 - 2 * q0**2, 190 | ] 191 | mj_accel[:, i_accbias] = np.eye(3) 192 | 193 | m_combined = np.concatenate((m_accel, m_gyro)) 194 | mj_combined = np.vstack((mj_accel, mj_gyro)) 195 | return (m_combined, mj_combined) 196 | 197 | 198 | @njit(cache=True) 199 | def camera_measurement(state: Mat): 200 | pos = state[i_pos] 201 | orientation = state[i_quat] 202 | m_camera = np.concatenate((pos, orientation)) 203 | # m_camera = state[i_pos + i_quat] 204 | mj_camera = np.zeros((7, len(state))) 205 | mj_camera[0:3, i_pos] = np.eye(3) 206 | mj_camera[3:7, i_quat] = np.eye(4) 207 | return (m_camera, mj_camera) 208 | 209 | 210 | @njit(cache=True) 211 | def repair_quaternion(q: Mat): 212 | # Note: we can't change the sign here, because it will affect the smoothing 213 | return q / np.linalg.norm(q) 214 | 215 | 216 | @njit(cache=True) 217 | def predict_cov_derivative(P: Mat, dfdx: Mat, Q: Mat): 218 | pDot = dfdx @ P + P @ (dfdx.T) + Q 219 | pDot = 0.5 * (pDot + pDot.T) 220 | return pDot 221 | 222 | 223 | @njit(cache=True) 224 | def euler_integrate(x: Mat, xdot: Mat, dt: float): 225 | return x + xdot * dt 226 | 227 | 228 | @njit(cache=True) 229 | def ekf_predict(fs: FilterState, dt: float, Q: np.ndarray): 230 | xdot = state_transition(fs.state) 231 | dfdx = state_transition_jacobian(fs.state) 232 | P = fs.statecov 233 | Pdot = predict_cov_derivative(P, dfdx, Q) 234 | state = euler_integrate(fs.state, xdot, dt) 235 | state[i_quat] = repair_quaternion(state[i_quat]) 236 | statecov = euler_integrate(P, Pdot, dt) 237 | # statecov = P + dt * (dfdx @ P + P @ dfdx.T + Q) + dt**2 * (dfdx @ P @ dfdx.T) 238 | # statecov = 0.5 * (statecov + statecov.T) 239 | 240 | return FilterState(state, statecov) 241 | 242 | 243 | @njit(cache=True) 244 | def ekf_correct(x: Mat, P: Mat, h: Mat, H: Mat, z: Mat, R: Mat): 245 | S = H @ P @ H.T + R # innovation covariance 246 | W = P @ H.T @ np.linalg.inv(S) 247 | x2 = x + W @ (z - h) 248 | P2 = P - W @ H @ P 249 | return x2, P2 250 | 251 | 252 | @njit(cache=True) 253 | def fuse_imu( 254 | fs: FilterState, accel: np.ndarray, gyro: np.ndarray, meas_noise: np.ndarray 255 | ): 256 | h, H = imu_measurement(fs.state) 257 | accel2 = np.array([-accel[2], accel[0], accel[1]]) 258 | gyro2 = np.array([gyro[2], -gyro[0], -gyro[1]]) 259 | z = np.concatenate((accel2, gyro2)) # actual measurement 260 | state, statecov = ekf_correct(fs.state, fs.statecov, h, H, z, meas_noise) 261 | state[i_quat] = repair_quaternion(state[i_quat]) 262 | return FilterState(state, statecov) 263 | 264 | 265 | def fuse_camera( 266 | fs: FilterState, 267 | imu_pos: np.ndarray, 268 | orientation_quat: np.ndarray, 269 | meas_noise: np.ndarray, 270 | ): 271 | h, H = camera_measurement(fs.state) 272 | z = np.concatenate((imu_pos.flatten(), orientation_quat)) # actual measurement 273 | state, statecov = ekf_correct(fs.state, fs.statecov, h, H, z, meas_noise) 274 | state[i_quat] = repair_quaternion(state[i_quat]) 275 | return FilterState(state, statecov) 276 | 277 | 278 | @njit(cache=True) 279 | def ekf_smooth(history: list[HistoryItem], dt: float): 280 | # We only need the last item to be set, but we do all of them to make numba happy 281 | smoothed_state = [h.updated_state for h in history] 282 | 283 | for i in range(len(history) - 2, -1, -1): 284 | h = history[i] 285 | F = np.eye(STATE_SIZE) + state_transition_jacobian(h.updated_state) * dt 286 | A = h.updated_statecov @ F.T @ np.linalg.inv(history[i + 1].predicted_statecov) 287 | correction = A @ (smoothed_state[i + 1] - history[i + 1].predicted_state) 288 | smoothed_state[i] = h.updated_state + correction 289 | return smoothed_state 290 | -------------------------------------------------------------------------------- /python/app/marker_tracker.py: -------------------------------------------------------------------------------- 1 | import json 2 | import math 3 | from pathlib import Path 4 | import os 5 | import pickle 6 | import cv2 7 | from cv2 import aruco 8 | import numpy as np 9 | from typing import NamedTuple, Tuple, Callable, Optional 10 | import time 11 | import sys 12 | import multiprocessing as mp 13 | 14 | from app.dimensions import IMU_OFFSET, STYLUS_LENGTH, idealMarkerPositions 15 | 16 | RECORD_DATA = True 17 | FPS = 30 18 | FRAME_WIDTH = 1920 19 | FRAME_HEIGHT = 1080 20 | TEXT_COL = (0, 0, 255) 21 | 22 | 23 | class CameraReading(NamedTuple): 24 | position: np.ndarray 25 | orientation_mat: np.ndarray 26 | 27 | def to_json(self): 28 | return { 29 | "position": self.position.tolist(), 30 | "orientation_mat": self.orientation_mat.tolist(), 31 | } 32 | 33 | def from_json(dict): 34 | return CameraReading( 35 | np.array(dict["position"]), np.array(dict["orientation_mat"]) 36 | ) 37 | 38 | 39 | def read_camera_parameters(filename: str) -> Tuple[np.ndarray, np.ndarray]: 40 | fs = cv2.FileStorage(filename, cv2.FILE_STORAGE_READ) 41 | if not fs.isOpened(): 42 | raise Exception("Couldn't open file") 43 | camera_matrix = fs.getNode("camera_matrix").mat() 44 | dist_coeffs = fs.getNode("distortion_coefficients").mat() 45 | fs.release() 46 | return (camera_matrix, dist_coeffs) 47 | 48 | 49 | def get_webcam(): 50 | webcam = cv2.VideoCapture(0, cv2.CAP_DSHOW) 51 | webcam.set(cv2.CAP_PROP_FPS, 60) 52 | webcam.set(cv2.CAP_PROP_FRAME_WIDTH, FRAME_WIDTH) 53 | webcam.set(cv2.CAP_PROP_FRAME_HEIGHT, FRAME_HEIGHT) 54 | if not webcam.set(cv2.CAP_PROP_FOURCC, cv2.VideoWriter_fourcc(*"MJPG")): 55 | raise Exception("Couldn't set FourCC") 56 | webcam.set(cv2.CAP_PROP_AUTOFOCUS, 0) 57 | webcam.set(cv2.CAP_PROP_FOCUS, 30) 58 | webcam.set(cv2.CAP_PROP_AUTO_EXPOSURE, 1) 59 | webcam.set(cv2.CAP_PROP_EXPOSURE, -9) 60 | webcam.set(cv2.CAP_PROP_BRIGHTNESS, 127) 61 | webcam.set(cv2.CAP_PROP_CONTRAST, 140) 62 | webcam.set(cv2.CAP_PROP_GAIN, 200) 63 | webcam.set(cv2.CAP_PROP_BUFFERSIZE, 1) 64 | return webcam 65 | 66 | 67 | def inverse_RT(rvec, tvec) -> Tuple[np.ndarray, np.ndarray]: 68 | R, _ = cv2.Rodrigues(rvec) 69 | Rt = np.transpose(R) 70 | return (cv2.Rodrigues(Rt)[0], -Rt @ tvec) 71 | 72 | 73 | def relative_transform(rvec1, tvec1, rvec2, tvec2) -> Tuple[np.ndarray, np.ndarray]: 74 | rvec2inv, tvec2inv = inverse_RT(rvec2, tvec2) 75 | rvec, tvec, *_ = cv2.composeRT(rvec1, tvec1, rvec2inv, tvec2inv) 76 | return (rvec, tvec) 77 | 78 | 79 | def get_aruco_params(): 80 | p = aruco.DetectorParameters() 81 | p.cornerRefinementMethod = cv2.aruco.CORNER_REFINE_CONTOUR 82 | p.cornerRefinementWinSize = 2 83 | # Reduce the number of threshold steps, which significantly improves performance 84 | p.adaptiveThreshWinSizeMin = 15 85 | p.adaptiveThreshWinSizeMax = 15 86 | p.useAruco3Detection = False 87 | p.minMarkerPerimeterRate = 0.02 88 | p.maxMarkerPerimeterRate = 2 89 | p.minSideLengthCanonicalImg = 16 90 | p.adaptiveThreshConstant = 7 91 | return p 92 | 93 | 94 | aruco_dic = aruco.getPredefinedDictionary(cv2.aruco.DICT_4X4_100) 95 | aruco_params = get_aruco_params() 96 | detector = aruco.ArucoDetector(aruco_dic, aruco_params) 97 | 98 | reprojection_error_threshold = 3 # px 99 | 100 | 101 | def array_to_str(arr): 102 | return ",".join(map(lambda x: f"{x:+2.2f}", list(arr.flat))) 103 | 104 | 105 | # charuco_dic = aruco.getPredefinedDictionary(cv2.aruco.DICT_5X5_100) 106 | # charuco_board = aruco.CharucoBoard((10, 7), 0.028, 0.022, charuco_dic) 107 | charuco_dic = aruco.getPredefinedDictionary(cv2.aruco.DICT_4X4_100) 108 | charuco_board = aruco.CharucoBoard((12, 8), 0.024, 0.018, charuco_dic) 109 | charuco_board.setLegacyPattern(True) 110 | charuco_params = aruco.DetectorParameters() 111 | charuco_detector = aruco.ArucoDetector(charuco_dic, charuco_params) 112 | 113 | 114 | def estimate_camera_pose_charuco(frame, camera_matrix, dist_coeffs): 115 | corners, ids, rejected = charuco_detector.detectMarkers(frame) 116 | if len(corners) == 0: 117 | raise Exception("No markers detected") 118 | display_frame = aruco.drawDetectedMarkers(image=frame, corners=corners) 119 | num_corners, charuco_corners, charuco_ids = aruco.interpolateCornersCharuco( 120 | markerCorners=corners, markerIds=ids, image=frame, board=charuco_board 121 | ) 122 | if num_corners < 5: 123 | raise Exception("Not enough corners detected") 124 | display_frame = aruco.drawDetectedCornersCharuco( 125 | image=display_frame, charucoCorners=charuco_corners, charucoIds=charuco_ids 126 | ) 127 | success, rvec, tvec = aruco.estimatePoseCharucoBoard( 128 | charuco_corners, 129 | charuco_ids, 130 | charuco_board, 131 | camera_matrix, 132 | dist_coeffs, 133 | None, 134 | None, 135 | False, 136 | ) 137 | if not success: 138 | raise Exception("Failed to estimate camera pose") 139 | # The rvec from charuco is z-down for some reason. 140 | # This is a hack to convert back to z-up. 141 | rvec, *_ = cv2.composeRT(np.array([0, 0, -np.pi / 2]), tvec * 0, rvec, tvec) 142 | rvec, *_ = cv2.composeRT(np.array([0, np.pi, 0]), tvec * 0, rvec, tvec) 143 | display_frame = cv2.drawFrameAxes( 144 | display_frame, camera_matrix, dist_coeffs, rvec, tvec, 0.2 145 | ) 146 | # cv2.imshow("Charuco", display_frame) 147 | return (rvec, tvec) 148 | 149 | 150 | def vector_rms(arr: np.ndarray, axis: int): 151 | """Computes the RMS magnitude of an array of vectors.""" 152 | return math.sqrt(np.mean(np.sum(np.square(arr), axis=axis))) 153 | 154 | 155 | def solve_pnp( 156 | initialized, 157 | prev_rvec, 158 | prev_tvec, 159 | object_points, 160 | image_points, 161 | camera_matrix, 162 | dist_coeffs, 163 | ) -> Tuple[bool, np.ndarray, np.ndarray]: 164 | """Attempt to refine the previous pose. If this fails, fall back to SQPnP.""" 165 | if initialized: 166 | rvec, tvec = cv2.solvePnPRefineVVS( 167 | object_points, 168 | image_points, 169 | cameraMatrix=camera_matrix, 170 | distCoeffs=dist_coeffs, 171 | # OpenCV mutates these arguments, which we don't want. 172 | rvec=prev_rvec.copy(), 173 | tvec=prev_tvec.copy(), 174 | ) 175 | projected_image_points, _ = cv2.projectPoints( 176 | object_points, rvec, tvec, camera_matrix, dist_coeffs, None 177 | ) 178 | projected_image_points = projected_image_points[:, 0, :] 179 | reprojection_error = vector_rms(projected_image_points - image_points, axis=1) 180 | 181 | if reprojection_error < reprojection_error_threshold: 182 | # print(f"Reprojection error: {reprojectionError}") 183 | return (True, rvec, tvec) 184 | else: 185 | print(f"Reprojection error too high: {reprojection_error}") 186 | 187 | success, rvec, tvec = cv2.solvePnP( 188 | object_points, 189 | image_points, 190 | cameraMatrix=camera_matrix, 191 | distCoeffs=dist_coeffs, 192 | flags=cv2.SOLVEPNP_SQPNP, 193 | ) 194 | return (success, rvec, tvec) 195 | 196 | 197 | MarkerDict = dict[int, tuple[np.ndarray, np.ndarray]] 198 | 199 | 200 | def detect_markers_bounded(frame: np.ndarray, x0: int, x1: int, y0: int, y1: int): 201 | x0, y0 = max(x0, 0), max(y0, 0) 202 | frame_view = frame[y0:y1, x0:x1] 203 | ids = None 204 | allCornersIS = [] 205 | rejected = [] 206 | try: 207 | allCornersIS, ids, rejected = detector.detectMarkers(frame_view) 208 | except cv2.error as e: 209 | # OpenCV threw an error here once for some reason, but we'd rather ignore it. 210 | # D:\a\opencv-python\opencv-python\opencv\modules\objdetect\src\aruco\aruco_detector.cpp:698: error: (-215:Assertion failed) nContours.size() >= 2 in function 'cv::aruco::_interpolate2Dline' 211 | print(e) 212 | pass 213 | if ids is not None: 214 | for i in range(ids.shape[0]): 215 | allCornersIS[i][0, :, 0] += x0 216 | allCornersIS[i][0, :, 1] += y0 217 | return allCornersIS, ids, rejected 218 | 219 | 220 | def bounds(x): 221 | return np.min(x), np.max(x) 222 | 223 | 224 | def clamp(x, xmin, xmax): 225 | return max(min(x, xmax), xmin) 226 | 227 | 228 | class MarkerTracker: 229 | def __init__( 230 | self, 231 | camera_matrix: np.ndarray, 232 | dist_coeffs: np.ndarray, 233 | marker_positions: dict[int, np.ndarray], 234 | ): 235 | self.cameraMatrix = camera_matrix 236 | self.distCoeffs = dist_coeffs 237 | self.rvec: Optional[np.ndarray] = None 238 | self.tvec: Optional[np.ndarray] = None 239 | self.initialized = False 240 | self.markerPositions = marker_positions 241 | self.allObjectPoints = np.concatenate(list(marker_positions.values())) 242 | self.lastValidMarkers: MarkerDict = {} 243 | self.lastVelocity = np.zeros(2) 244 | 245 | def get_search_area(self, rvec: np.ndarray, tvec: np.ndarray, velocity: np.array): 246 | """Returns a bounding box to search in the next frame, based on the current marker positions and velocity.""" 247 | 248 | # Re-project all object points, to avoid cases where some markers were missed in the previous frame. 249 | projected_image_points, _ = cv2.projectPoints( 250 | self.allObjectPoints, rvec, tvec, self.cameraMatrix, self.distCoeffs, None 251 | ) 252 | projected_image_points = projected_image_points[:, 0, :] 253 | 254 | x0, x1 = bounds(projected_image_points[:, 0] + velocity[0] / FPS) 255 | y0, y1 = bounds(projected_image_points[:, 1] + velocity[1] / FPS) 256 | w = x1 - x0 257 | h = y1 - y0 258 | 259 | # Amount to expand each axis by, in pixels. This is just a rough heuristic, and the constants are arbitrary. 260 | expand = max(0.5 * (w + h), 200) + 1.0 * np.abs(velocity) / FPS 261 | 262 | # Values are sometimes extremely large if tvec is wrong, clamp is a workaround to stop cv2.rectangle from breaking. 263 | return ( 264 | int(clamp(x0 - expand[0], 0, FRAME_WIDTH)), 265 | int(clamp(x1 + expand[0], 0, FRAME_WIDTH)), 266 | int(clamp(y0 - expand[1], 0, FRAME_HEIGHT)), 267 | int(clamp(y1 + expand[1], 0, FRAME_HEIGHT)), 268 | ) 269 | 270 | def process_frame(self, frame: np.ndarray): 271 | ids: np.ndarray 272 | if self.initialized: 273 | x0, x1, y0, y1 = self.get_search_area( 274 | self.rvec, self.tvec, self.lastVelocity 275 | ) 276 | cv2.rectangle(frame, (x0, y0), (x1, y1), (0, 100, 0, 0.3), 2) 277 | allCornersIS, ids, rejected = detect_markers_bounded(frame, x0, x1, y0, y1) 278 | else: 279 | allCornersIS, ids, rejected = detector.detectMarkers(frame) 280 | aruco.drawDetectedMarkers(frame, allCornersIS, ids) 281 | valid_markers: MarkerDict = {} 282 | if ids is not None: 283 | for i in range(ids.shape[0]): 284 | # cornersIS is 4x2 285 | id, cornersIS = (ids[i, 0], allCornersIS[i][0, :, :]) 286 | if id in self.markerPositions: 287 | cornersPS = self.markerPositions[id] 288 | valid_markers[id] = (cornersPS, cornersIS) 289 | 290 | if len(valid_markers) < 1: 291 | self.initialized = False 292 | self.lastValidMarkers = {} 293 | self.next_search_area = None 294 | return None 295 | 296 | point_deltas = [] 297 | for id, (cornersPS, cornersIS) in valid_markers.items(): 298 | if id in self.lastValidMarkers: 299 | velocity = cornersIS - self.lastValidMarkers[id][1] 300 | point_deltas.append(np.mean(velocity, axis=0)) 301 | 302 | if point_deltas: 303 | meanVelocity = np.mean(point_deltas, axis=0) * 30 # px/second 304 | else: 305 | meanVelocity = np.zeros(2) 306 | 307 | mean_position_IS = np.mean( 308 | [cornersIS for _, cornersIS in valid_markers.values()], 309 | axis=(0, 1), 310 | ) 311 | 312 | screen_corners = [] 313 | pen_corners = [] 314 | delay_per_image_row = 1 / 30 / 1080 # seconds/row 315 | 316 | for id, (cornersPS, cornersIS) in valid_markers.items(): 317 | pen_corners.append(cornersPS) 318 | if point_deltas: 319 | # Compensate for rolling shutter 320 | timeDelay = ( 321 | cornersIS[:, 1] - mean_position_IS[1] 322 | ) * delay_per_image_row # seconds, relative to centroid 323 | cornersISCompensated = ( 324 | cornersIS - meanVelocity * timeDelay[:, np.newaxis] 325 | ) 326 | screen_corners.append(cornersISCompensated) 327 | else: 328 | screen_corners.append(cornersIS) 329 | 330 | self.initialized, self.rvec, self.tvec = solve_pnp( 331 | self.initialized, 332 | self.rvec, 333 | self.tvec, 334 | object_points=np.concatenate(pen_corners), 335 | image_points=np.concatenate(screen_corners), 336 | camera_matrix=self.cameraMatrix, 337 | dist_coeffs=self.distCoeffs, 338 | ) 339 | 340 | self.lastValidMarkers = valid_markers 341 | self.lastVelocity = meanVelocity 342 | return (self.rvec, self.tvec) 343 | 344 | 345 | focus_interval = 30 # frames 346 | # Map from distance to optimal focus value, measured manually. 347 | # These don't need to be very precise. 348 | focus_targets = np.array( 349 | [ 350 | [0.1, 75], 351 | [0.15, 50], 352 | [0.2, 40], 353 | [0.3, 30], 354 | [0.5, 25], 355 | ] 356 | ) 357 | 358 | 359 | def load_marker_positions(): 360 | try: 361 | with open("./params/calibrated_marker_positions.json", "r") as f: 362 | pos_json = json.load(f) 363 | return {int(k): np.array(v) for k, v in pos_json.items()} 364 | except: 365 | print("Couldn't load calibrated marker positions, using ideal positions") 366 | return idealMarkerPositions 367 | 368 | 369 | def get_focus_target(dist_to_camera): 370 | f = np.interp([dist_to_camera], focus_targets[:, 0], focus_targets[:, 1])[0] 371 | return 5 * round(f / 5) # Webcam only supports multiples of 5 372 | 373 | 374 | def run_tracker( 375 | on_estimate: Optional[Callable[[CameraReading], None]], 376 | recording_enabled: Optional[mp.Value] = None, 377 | recording_timestamp: str = "", 378 | ): 379 | cv2.namedWindow("Tracker", cv2.WINDOW_KEEPRATIO) 380 | cv2.resizeWindow("Tracker", 1050, int(1050 * 1080 / 1920)) 381 | camera_matrix, dist_coeffs = read_camera_parameters( 382 | "params/camera_params_c922_f30.yml" 383 | ) 384 | marker_positions = load_marker_positions() 385 | print("Opening webcam..") 386 | webcam = get_webcam() 387 | 388 | calibrated = False 389 | baseRvec = np.zeros([3, 1]) 390 | baseTvec = np.zeros([3, 1]) 391 | avg_fps = 30 392 | 393 | tracker = MarkerTracker(camera_matrix, dist_coeffs, marker_positions) 394 | frame_count = 0 395 | auto_focus = True 396 | current_focus = 30 397 | while True: 398 | frame_start_time = time.perf_counter() 399 | keypress = cv2.waitKey(1) & 0xFF 400 | if keypress == ord("q"): 401 | break 402 | elif keypress == ord("u"): 403 | auto_focus = False 404 | current_focus += 5 405 | webcam.set(cv2.CAP_PROP_FOCUS, current_focus) 406 | elif keypress == ord("d"): 407 | auto_focus = False 408 | current_focus -= 5 409 | webcam.set(cv2.CAP_PROP_FOCUS, current_focus) 410 | elif keypress == ord("a"): 411 | auto_focus = True 412 | 413 | frame: np.ndarray 414 | ret, frame = webcam.read() 415 | frame_original = frame.copy() 416 | 417 | if keypress == ord("s"): 418 | focus = round(webcam.get(cv2.CAP_PROP_FOCUS)) 419 | filepath = f"calibration_pics/f{focus}/{round(time.time())}.jpg" 420 | os.makedirs(os.path.dirname(filepath), exist_ok=True) 421 | success = cv2.imwrite(filepath, frame) 422 | print(f"save: {success}, {filepath}") 423 | 424 | processing_start_time = time.perf_counter() 425 | 426 | if not calibrated or keypress == ord("c"): 427 | print("Calibrating...") 428 | try: 429 | baseRvec, baseTvec = estimate_camera_pose_charuco( 430 | frame, camera_matrix, dist_coeffs 431 | ) 432 | calibrated = True 433 | except Exception as e: 434 | print("Error calibrating camera, press C to retry.", e) 435 | 436 | result = tracker.process_frame(frame) 437 | processing_end_time = time.perf_counter() 438 | if result is not None: 439 | rvec, tvec = result 440 | rvec_relative, tvec_relative = relative_transform( 441 | rvec, tvec, baseRvec, baseTvec 442 | ) 443 | tip_to_imu_offset = -np.array(IMU_OFFSET) - [0, STYLUS_LENGTH, 0] 444 | rvec_tip, tvec_tip, *_ = cv2.composeRT( 445 | np.zeros(3), tip_to_imu_offset, rvec, tvec 446 | ) 447 | _, tvec_tip_relative = relative_transform( 448 | rvec_tip, tvec_tip, baseRvec, baseTvec 449 | ) 450 | R_relative = cv2.Rodrigues(rvec_relative)[0] # TODO: use Rodrigues directly 451 | cv2.drawFrameAxes(frame, camera_matrix, dist_coeffs, rvec, tvec, 0.01) 452 | cv2.drawFrameAxes( 453 | frame, camera_matrix, dist_coeffs, rvec_tip, tvec_tip, 0.01 454 | ) 455 | cv2.putText( 456 | frame, 457 | f"IMU: [{array_to_str(tvec_relative*100)}]cm", 458 | (10, 120), 459 | cv2.FONT_HERSHEY_DUPLEX, 460 | 1, 461 | TEXT_COL, 462 | ) 463 | cv2.putText( 464 | frame, 465 | f"Tip: [{array_to_str(tvec_tip_relative*100)}]cm", 466 | (10, 150), 467 | cv2.FONT_HERSHEY_DUPLEX, 468 | 1, 469 | TEXT_COL, 470 | ) 471 | 472 | if on_estimate is not None: 473 | on_estimate(CameraReading(tvec_relative, R_relative)) 474 | 475 | frame_end_time = time.perf_counter() 476 | fps = 1 / (frame_end_time - frame_start_time) 477 | avg_fps = 0.9 * avg_fps + 0.1 * fps 478 | cv2.putText( 479 | frame, 480 | f"FPS: {avg_fps:.1f}", 481 | (10, 30), 482 | cv2.FONT_HERSHEY_DUPLEX, 483 | 1, 484 | TEXT_COL, 485 | ) 486 | cv2.putText( 487 | frame, 488 | f"Processing: {(processing_end_time - processing_start_time)*1000:.1f}ms", 489 | (10, 60), 490 | cv2.FONT_HERSHEY_DUPLEX, 491 | 1, 492 | TEXT_COL, 493 | ) 494 | cv2.putText( 495 | frame, 496 | f"Focus: {current_focus}", 497 | (10, 180), 498 | cv2.FONT_HERSHEY_DUPLEX, 499 | 1, 500 | TEXT_COL, 501 | ) 502 | 503 | cv2.imshow("Tracker", frame) 504 | 505 | if recording_enabled and recording_enabled.value: 506 | timestamp = time.time_ns() // 1_000_000 507 | dir = f"recordings/{recording_timestamp}/frames" 508 | Path(dir).mkdir(parents=True, exist_ok=True) 509 | filepath = f"{dir}/{timestamp}.bmp" 510 | cv2.imwrite(filepath, frame_original) 511 | with open( 512 | f"./recordings/{recording_timestamp}/camera_extrinsics.pkl", "wb" 513 | ) as pickle_file: 514 | pickle.dump((baseRvec, baseTvec), pickle_file) 515 | 516 | # Adjust focus periodically 517 | if auto_focus and calibrated and frame_count % focus_interval == 0: 518 | if result is None: 519 | focus = 30 520 | else: 521 | dist_to_camera = np.linalg.norm(tvec) 522 | focus = get_focus_target(dist_to_camera) 523 | if focus != current_focus: 524 | current_focus = focus 525 | webcam.set(cv2.CAP_PROP_FOCUS, focus) 526 | 527 | frame_count += 1 528 | 529 | 530 | if __name__ == "__main__" and sys.flags.interactive == 0: 531 | run_tracker(None) 532 | -------------------------------------------------------------------------------- /python/app/monitor_ble.py: -------------------------------------------------------------------------------- 1 | import struct 2 | from typing import NamedTuple 3 | from bleak import BleakClient, BleakScanner 4 | from bleak.backends.characteristic import BleakGATTCharacteristic 5 | import asyncio 6 | import multiprocessing as mp 7 | 8 | import numpy as np 9 | 10 | 11 | class StylusReading(NamedTuple): 12 | accel: np.ndarray 13 | gyro: np.ndarray 14 | t: int 15 | pressure: float 16 | 17 | def format_aligned(self): 18 | return f"p={self.pressure:<8.5f}: |a|={np.linalg.norm(self.accel):<7.3f} a={self.accel}, g={self.gyro}" 19 | 20 | def to_json(self): 21 | return { 22 | "accel": self.accel.tolist(), 23 | "gyro": self.gyro.tolist(), 24 | "t": self.t, 25 | "pressure": self.pressure 26 | } 27 | 28 | def from_json(dict): 29 | return StylusReading( 30 | np.array(dict["accel"]), 31 | np.array(dict["gyro"]), 32 | dict["t"], 33 | dict["pressure"] 34 | ) 35 | 36 | 37 | class StopCommand(NamedTuple): 38 | pass 39 | 40 | 41 | def calc_accel(a): 42 | """Remap raw accelerometer measurements to Gs.""" 43 | accel_range = 4 # Should match settings.accelRange in microcontroller code 44 | return a * 0.061 * (accel_range / 2) / 1000 45 | 46 | 47 | def calc_gyro(g): 48 | """Remap raw gyro measurements to degrees per second.""" 49 | gyro_range = 500 # Should match settings.gyroRange in microcontroller code 50 | return g * 4.375 * (gyro_range / 125) / 1000 51 | 52 | 53 | def unpack_imu_data_packet(data: bytearray): 54 | """Unpacks an IMUDataPacket struct from the given data buffer.""" 55 | ax, ay, az, gx, gy, gz, pressure = struct.unpack("<3h3hH", data) 56 | accel = calc_accel(np.array([ax, ay, az], dtype=np.float64) * 9.8) 57 | gyro = calc_gyro(np.array([gx, gy, gz], dtype=np.float64) * np.pi / 180.0) 58 | return StylusReading(accel, gyro, 0, pressure / 2**16) 59 | 60 | 61 | characteristic = "19B10013-E8F2-537E-4F6C-D104768A1214" 62 | 63 | 64 | async def monitor_ble_async(data_queue: mp.Queue, command_queue: mp.Queue): 65 | while True: 66 | device = await BleakScanner.find_device_by_name("DPOINT", timeout=5) 67 | if device is None: 68 | print("could not find device with name DPOINT. Retrying in 1 second...") 69 | await asyncio.sleep(1) 70 | continue 71 | 72 | def queue_notification_handler(_: BleakGATTCharacteristic, data: bytearray): 73 | reading = unpack_imu_data_packet(data) 74 | data_queue.put(reading) 75 | 76 | disconnected_event = asyncio.Event() 77 | print("Connecting to BLE device...") 78 | try: 79 | async with BleakClient( 80 | device, disconnected_callback=lambda _: disconnected_event.set() 81 | ) as client: 82 | print("Connected!") 83 | await client.start_notify(characteristic, queue_notification_handler) 84 | command = asyncio.create_task( 85 | asyncio.to_thread(lambda: command_queue.get()) 86 | ) 87 | disconnected_task = asyncio.create_task(disconnected_event.wait()) 88 | await asyncio.wait( 89 | [disconnected_task, command], return_when=asyncio.FIRST_COMPLETED 90 | ) 91 | if command.done(): 92 | print("Quitting BLE process") 93 | return 94 | print("Disconnected from BLE") 95 | except Exception as e: 96 | print(f"BLE Exception: {e}") 97 | print("Retrying in 1 second...") 98 | await asyncio.sleep(1) 99 | 100 | 101 | def monitor_ble(data_queue: mp.Queue, command_queue: mp.Queue): 102 | asyncio.run(monitor_ble_async(data_queue, command_queue)) 103 | -------------------------------------------------------------------------------- /python/calibrate_camera.py: -------------------------------------------------------------------------------- 1 | # based on https://github.com/kyle-bersani/opencv-examples/blob/master/CalibrationByCharucoBoard/CalibrateCamera.py 2 | 3 | from pathlib import Path 4 | import numpy 5 | import cv2 6 | from cv2 import aruco 7 | import glob 8 | 9 | 10 | # ChAruco board variables 11 | CHARUCOBOARD_ROWCOUNT = 8 12 | CHARUCOBOARD_COLCOUNT = 12 13 | ARUCO_DICT = aruco.getPredefinedDictionary(aruco.DICT_4X4_100) 14 | 15 | # Create constants to be passed into OpenCV and Aruco methods 16 | CHARUCO_BOARD = aruco.CharucoBoard( 17 | (CHARUCOBOARD_COLCOUNT, CHARUCOBOARD_ROWCOUNT), 18 | 0.024, 19 | 0.018, 20 | ARUCO_DICT) 21 | CHARUCO_BOARD.setLegacyPattern(True) 22 | 23 | # Create the arrays and variables we'll use to store info like corners and IDs from images processed 24 | corners_all = [] # Corners discovered in all images processed 25 | ids_all = [] # Aruco ids corresponding to corners discovered 26 | image_size = None # Determined at runtime 27 | 28 | images = glob.glob('./calibration_pics/f30/*.jpg') 29 | 30 | # Loop through images glob'ed 31 | for iname in images: 32 | # Open the image 33 | img = cv2.imread(iname) 34 | # Grayscale the image 35 | gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY) 36 | 37 | # Find aruco markers in the query image 38 | corners, ids, _ = aruco.detectMarkers( 39 | image=gray, 40 | dictionary=ARUCO_DICT) 41 | 42 | # Outline the aruco markers found in our query image 43 | img = aruco.drawDetectedMarkers( 44 | image=img, 45 | corners=corners) 46 | 47 | # Get charuco corners and ids from detected aruco markers 48 | response, charuco_corners, charuco_ids = aruco.interpolateCornersCharuco( 49 | markerCorners=corners, 50 | markerIds=ids, 51 | image=gray, 52 | board=CHARUCO_BOARD) 53 | 54 | # If a Charuco board was found, let's collect image/corner points 55 | # Requiring at least 20 squares 56 | if response > 20: 57 | # Add these corners and ids to our calibration arrays 58 | corners_all.append(charuco_corners) 59 | ids_all.append(charuco_ids) 60 | 61 | # Draw the Charuco board we've detected to show our calibrator the board was properly detected 62 | img = aruco.drawDetectedCornersCharuco( 63 | image=img, 64 | charucoCorners=charuco_corners, 65 | charucoIds=charuco_ids) 66 | 67 | # If our image size is unknown, set it now 68 | if not image_size: 69 | image_size = gray.shape[::-1] 70 | 71 | # Reproportion the image, maxing width or height at 1000 72 | proportion = max(img.shape) / 1000.0 73 | img = cv2.resize(img, (int(img.shape[1]/proportion), int(img.shape[0]/proportion))) 74 | # Pause to display each image, waiting for key press 75 | cv2.imshow('Charuco board', img) 76 | cv2.waitKey(0) 77 | else: 78 | print("Not able to detect a charuco board in image: {}".format(iname)) 79 | 80 | # Destroy any open CV windows 81 | cv2.destroyAllWindows() 82 | 83 | # Make sure at least one image was found 84 | if len(images) < 1: 85 | # Calibration failed because there were no images, warn the user 86 | print("Calibration was unsuccessful. No images of charucoboards were found. Add images of charucoboards and use or alter the naming conventions used in this file.") 87 | # Exit for failure 88 | exit() 89 | 90 | # Make sure we were able to calibrate on at least one charucoboard by checking 91 | # if we ever determined the image size 92 | if not image_size: 93 | # Calibration failed because we didn't see any charucoboards of the PatternSize used 94 | print("Calibration was unsuccessful. We couldn't detect charucoboards in any of the images supplied. Try changing the patternSize passed into Charucoboard_create(), or try different pictures of charucoboards.") 95 | # Exit for failure 96 | exit() 97 | 98 | # Now that we've seen all of our images, perform the camera calibration 99 | # based on the set of points we've discovered 100 | calibration, cameraMatrix, distCoeffs, rvecs, tvecs = aruco.calibrateCameraCharuco( 101 | charucoCorners=corners_all, 102 | charucoIds=ids_all, 103 | board=CHARUCO_BOARD, 104 | imageSize=image_size, 105 | cameraMatrix=None, 106 | distCoeffs=None) 107 | 108 | # Print matrix and distortion coefficient to the console 109 | print(cameraMatrix) 110 | print(distCoeffs) 111 | 112 | output_path = "./params/camera_params_c922_f30.yml" 113 | Path(output_path).parent.mkdir(parents=True, exist_ok=True) 114 | fs = cv2.FileStorage(output_path, cv2.FILE_STORAGE_WRITE) 115 | if not fs.isOpened(): 116 | raise Exception("Couldn't open file") 117 | fs.write("camera_matrix", cameraMatrix) 118 | fs.write("distortion_coefficients", distCoeffs) 119 | fs.release() 120 | 121 | print(f"Calibration successful. Calibration file used: {output_path}") 122 | -------------------------------------------------------------------------------- /python/calibrate_markers.py: -------------------------------------------------------------------------------- 1 | # Computes the 3D positions of the markers on the pen, given a set of images of the pen. 2 | # The images should be taken with the pen in a variety of poses, and each image should contain at least two markers. 3 | # The calibrated positions are stored OUTPUT_PATH 4 | 5 | import json 6 | from pathlib import Path 7 | import numpy as np 8 | import cv2 9 | import glob 10 | import scipy.optimize 11 | from cv2 import aruco 12 | from app.marker_tracker import aruco_params, read_camera_parameters 13 | from app.dimensions import idealMarkerPositions 14 | 15 | MARKER_COUNT = 8 # The number of markers on the pen 16 | FIRST_MARKER_ID = 92 # ID of the first marker on the pen, used to convert IDs to 0-n. 17 | IMAGE_PATH = "./marker_calibration_pics/f30/*.jpg" 18 | OUTPUT_PATH = "./params/calibrated_marker_positions.json" 19 | 20 | 21 | Observation = dict[int, np.ndarray] # Map from marker id to 4x3 array of corners 22 | 23 | 24 | def residual( 25 | x: np.ndarray, 26 | camera_matrix, 27 | dist_coeffs, 28 | observations: list[Observation], 29 | marker0_pose: np.ndarray, 30 | ): 31 | marker_poses = np.concatenate( 32 | (marker0_pose, x[0 : (MARKER_COUNT - 1) * 4 * 3]) 33 | ).reshape((MARKER_COUNT, 4, 3)) 34 | camera_poses = x[(MARKER_COUNT - 1) * 4 * 3 :].reshape((-1, 6)) 35 | res_all = [] 36 | for img_id in range(len(observations)): 37 | img = observations[img_id] 38 | rvec = camera_poses[img_id, 0:3] 39 | tvec = camera_poses[img_id, 3:6] 40 | 41 | for marker_id, marker_corners_observed in img.items(): 42 | projected: np.ndarray 43 | projected, jac = cv2.projectPoints( 44 | objectPoints=marker_poses[marker_id - FIRST_MARKER_ID, :, :], # 4x3 45 | rvec=rvec, 46 | tvec=tvec, 47 | cameraMatrix=camera_matrix, 48 | distCoeffs=dist_coeffs, 49 | ) 50 | 51 | res = projected.flatten() - marker_corners_observed.flatten() 52 | res_all.append(res) 53 | 54 | return np.concatenate(res_all) 55 | 56 | 57 | def get_observed_points(pathname, arucoParams): 58 | """Reads in the set of images, and detects any markers in them.""" 59 | observed_points: list[Observation] = [] 60 | for iname in glob.glob(pathname): 61 | img = cv2.imread(iname) 62 | corners_all, ids, _ = aruco.detectMarkers( 63 | image=img, 64 | dictionary=aruco.getPredefinedDictionary(aruco.DICT_4X4_100), 65 | parameters=arucoParams, 66 | ) 67 | if ids is None: 68 | print("No markers found in image", iname) 69 | continue 70 | if ids.shape[0] < 2: 71 | print("Not enough markers found in image", iname) 72 | continue 73 | corner_dict: Observation = {} 74 | for i in range(ids.shape[0]): 75 | if ids[i, 0] in idealMarkerPositions: 76 | corner_dict[ids[i, 0]] = corners_all[i][0, :, :] 77 | observed_points.append(corner_dict) 78 | 79 | if not observed_points: 80 | raise RuntimeError("No valid images found in path", pathname) 81 | return observed_points 82 | 83 | 84 | def get_initial_estimate(observations: list[Observation], camera_matrix, dist_coeffs): 85 | """Computes an initial state vector based on: 86 | - The ideal marker positions 87 | - Camera poses for each image, estimated using PnP 88 | """ 89 | marker_poses = np.zeros((MARKER_COUNT, 4, 3), dtype=np.float32) 90 | for mid, corners in idealMarkerPositions.items(): 91 | marker_poses[mid - FIRST_MARKER_ID, :, :] = corners 92 | 93 | camera_poses = np.zeros((len(observations), 6), dtype=np.float32) 94 | for img_id, img in enumerate(observations): 95 | validMarkers = [] 96 | for marker_id, marker_corners_observed in img.items(): 97 | if marker_id in idealMarkerPositions: 98 | validMarkers.append( 99 | (idealMarkerPositions[marker_id], marker_corners_observed) 100 | ) 101 | 102 | screenCorners = np.concatenate([cornersIS for _, cornersIS in validMarkers]) 103 | penCorners = np.concatenate([cornersPS for cornersPS, _ in validMarkers]) 104 | success, rvec, tvec = cv2.solvePnP( 105 | penCorners, 106 | screenCorners, 107 | camera_matrix, 108 | dist_coeffs, 109 | flags=cv2.SOLVEPNP_SQPNP, 110 | ) 111 | camera_poses[img_id, :] = np.hstack((rvec.flatten(), tvec.flatten())) 112 | 113 | result = np.concatenate((marker_poses.flatten(), camera_poses.flatten())) 114 | return result[0:12], result[12:] 115 | 116 | 117 | def calibrate_markers(camera_matrix, dist_coeffs, observations: list[Observation]): 118 | # We fix the pose of the first marker, so that the problem is properly constrained. 119 | marker0_pose, x0 = get_initial_estimate(observations, camera_matrix, dist_coeffs) 120 | 121 | def fun(x): 122 | return residual(x, camera_matrix, dist_coeffs, observations, marker0_pose) 123 | 124 | opt_result = scipy.optimize.least_squares( 125 | fun=fun, 126 | x0=x0, 127 | max_nfev=1000, 128 | verbose=2, 129 | ) 130 | return np.concatenate((marker0_pose, opt_result.x[0 : 7 * 4 * 3])).reshape( 131 | (MARKER_COUNT, 4, 3) 132 | ) 133 | 134 | 135 | def main(): 136 | observed_points = get_observed_points(IMAGE_PATH, aruco_params) 137 | camera_matrix, dist_coeffs = read_camera_parameters("./params/camera_params_c922_f30.yml") 138 | result = calibrate_markers(camera_matrix, dist_coeffs, observed_points) 139 | 140 | calibratedMarkerPositions = { 141 | i + FIRST_MARKER_ID: result[i, :, :].tolist() for i in range(MARKER_COUNT) 142 | } 143 | file = Path(OUTPUT_PATH) 144 | file.parent.mkdir(parents=True, exist_ok=True) 145 | with file.open("w") as f: 146 | json.dump(calibratedMarkerPositions, f, indent=2) 147 | 148 | 149 | if __name__ == "__main__": 150 | main() 151 | -------------------------------------------------------------------------------- /python/calibration_pics/f30/1687925759.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Jcparkyn/dpoint/a108c19b9b240c1531b2d30ed47ba18bc604862e/python/calibration_pics/f30/1687925759.jpg -------------------------------------------------------------------------------- /python/calibration_pics/f30/1687925828.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Jcparkyn/dpoint/a108c19b9b240c1531b2d30ed47ba18bc604862e/python/calibration_pics/f30/1687925828.jpg -------------------------------------------------------------------------------- /python/calibration_pics/f30/1687925831.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Jcparkyn/dpoint/a108c19b9b240c1531b2d30ed47ba18bc604862e/python/calibration_pics/f30/1687925831.jpg -------------------------------------------------------------------------------- /python/calibration_pics/f30/1687925842.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Jcparkyn/dpoint/a108c19b9b240c1531b2d30ed47ba18bc604862e/python/calibration_pics/f30/1687925842.jpg -------------------------------------------------------------------------------- /python/calibration_pics/f30/1687925848.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Jcparkyn/dpoint/a108c19b9b240c1531b2d30ed47ba18bc604862e/python/calibration_pics/f30/1687925848.jpg -------------------------------------------------------------------------------- /python/calibration_pics/f30/1687925851.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Jcparkyn/dpoint/a108c19b9b240c1531b2d30ed47ba18bc604862e/python/calibration_pics/f30/1687925851.jpg -------------------------------------------------------------------------------- /python/calibration_pics/f30/1687925856.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Jcparkyn/dpoint/a108c19b9b240c1531b2d30ed47ba18bc604862e/python/calibration_pics/f30/1687925856.jpg -------------------------------------------------------------------------------- /python/calibration_pics/f30/1687925863.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Jcparkyn/dpoint/a108c19b9b240c1531b2d30ed47ba18bc604862e/python/calibration_pics/f30/1687925863.jpg -------------------------------------------------------------------------------- /python/main.py: -------------------------------------------------------------------------------- 1 | from app.app import main 2 | 3 | """Runs the main application.""" 4 | 5 | if __name__ == "__main__": 6 | main() 7 | -------------------------------------------------------------------------------- /python/mesh/pen.blend: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Jcparkyn/dpoint/a108c19b9b240c1531b2d30ed47ba18bc604862e/python/mesh/pen.blend -------------------------------------------------------------------------------- /python/mesh/pen.obj: -------------------------------------------------------------------------------- 1 | # Blender v2.93.1 OBJ File: 'pen.blend' 2 | # www.blender.org 3 | o Cylinder 4 | v -0.007000 0.018472 0.000000 5 | v -0.007000 0.148472 0.000000 6 | v -0.004950 0.018472 -0.004950 7 | v -0.004950 0.148472 -0.004950 8 | v 0.000000 0.018472 -0.007000 9 | v 0.000000 0.148472 -0.007000 10 | v 0.004950 0.018472 -0.004950 11 | v 0.004950 0.148472 -0.004950 12 | v 0.007000 0.018472 0.000000 13 | v 0.007000 0.148472 0.000000 14 | v 0.004950 0.018472 0.004950 15 | v 0.004950 0.148472 0.004950 16 | v -0.000000 0.018472 0.007000 17 | v 0.000000 0.148472 0.007000 18 | v -0.004950 0.018472 0.004950 19 | v -0.004950 0.148472 0.004950 20 | v 0.000000 0.000000 -0.000000 21 | v 0.000000 0.148472 0.000000 22 | vt 1.000000 1.000000 23 | vt 0.875000 0.500000 24 | vt 1.000000 0.500000 25 | vt 0.875000 1.000000 26 | vt 0.750000 0.500000 27 | vt 0.750000 1.000000 28 | vt 0.625000 0.500000 29 | vt 0.625000 1.000000 30 | vt 0.500000 0.500000 31 | vt 0.500000 1.000000 32 | vt 0.375000 0.500000 33 | vt 0.375000 1.000000 34 | vt 0.250000 0.500000 35 | vt 0.250000 1.000000 36 | vt 0.125000 0.500000 37 | vt 0.125000 1.000000 38 | vt 0.000000 0.500000 39 | vt 0.500000 0.500000 40 | vt 0.125000 0.500000 41 | vt 1.000000 0.500000 42 | vt 0.750000 0.500000 43 | vt 0.375000 0.500000 44 | vt 0.625000 0.500000 45 | vt 0.250000 0.500000 46 | vt 0.875000 0.500000 47 | vt 0.419706 0.419706 48 | vt 0.250000 0.490000 49 | vt 0.250000 0.250000 50 | vt 0.080294 0.419706 51 | vt 0.010000 0.250000 52 | vt 0.080294 0.080294 53 | vt 0.250000 0.010000 54 | vt 0.419706 0.080294 55 | vt 0.490000 0.250000 56 | vt 0.000000 1.000000 57 | vn -0.9239 -0.0000 -0.3827 58 | vn -0.3827 0.0000 -0.9239 59 | vn 0.3827 0.0000 -0.9239 60 | vn 0.9239 0.0000 -0.3827 61 | vn 0.9239 0.0000 0.3827 62 | vn 0.3827 -0.0000 0.9239 63 | vn -0.3827 0.0000 0.9239 64 | vn -0.9239 0.0000 0.3827 65 | vn 0.8720 -0.3304 0.3612 66 | vn -0.8720 -0.3304 0.3612 67 | vn -0.8720 -0.3304 -0.3612 68 | vn 0.3612 -0.3304 -0.8720 69 | vn 0.3612 -0.3304 0.8720 70 | vn 0.8720 -0.3304 -0.3612 71 | vn -0.3612 -0.3304 0.8720 72 | vn -0.3612 -0.3304 -0.8720 73 | vn -0.0000 1.0000 0.0000 74 | s off 75 | f 2/1/1 3/2/1 1/3/1 76 | f 4/4/2 5/5/2 3/2/2 77 | f 6/6/3 7/7/3 5/5/3 78 | f 8/8/4 9/9/4 7/7/4 79 | f 10/10/5 11/11/5 9/9/5 80 | f 12/12/6 13/13/6 11/11/6 81 | f 14/14/7 15/15/7 13/13/7 82 | f 16/16/8 1/17/8 15/15/8 83 | f 9/9/9 11/11/9 17/18/9 84 | f 15/15/10 1/17/10 17/19/10 85 | f 1/3/11 3/2/11 17/20/11 86 | f 5/5/12 7/7/12 17/21/12 87 | f 11/11/13 13/13/13 17/22/13 88 | f 7/7/14 9/9/14 17/23/14 89 | f 13/13/15 15/15/15 17/24/15 90 | f 3/2/16 5/5/16 17/25/16 91 | f 4/26/17 2/27/17 18/28/17 92 | f 2/27/17 16/29/17 18/28/17 93 | f 16/29/17 14/30/17 18/28/17 94 | f 14/30/17 12/31/17 18/28/17 95 | f 12/31/17 10/32/17 18/28/17 96 | f 10/32/17 8/33/17 18/28/17 97 | f 8/33/17 6/34/17 18/28/17 98 | f 6/34/17 4/26/17 18/28/17 99 | f 2/1/1 4/4/1 3/2/1 100 | f 4/4/2 6/6/2 5/5/2 101 | f 6/6/3 8/8/3 7/7/3 102 | f 8/8/4 10/10/4 9/9/4 103 | f 10/10/5 12/12/5 11/11/5 104 | f 12/12/6 14/14/6 13/13/6 105 | f 14/14/7 16/16/7 15/15/7 106 | f 16/16/8 2/35/8 1/17/8 107 | -------------------------------------------------------------------------------- /python/params/camera_params_c922_f30.yml: -------------------------------------------------------------------------------- 1 | %YAML:1.0 2 | --- 3 | camera_matrix: !!opencv-matrix 4 | rows: 3 5 | cols: 3 6 | dt: d 7 | data: [ 1.4246961327437521e+03, 0., 9.6088177498278958e+02, 0., 8 | 1.4238595252528614e+03, 5.2150903733130576e+02, 0., 0., 1. ] 9 | distortion_coefficients: !!opencv-matrix 10 | rows: 1 11 | cols: 5 12 | dt: d 13 | data: [ 4.7280097179329364e-02, -2.0575594743540951e-01, 14 | 1.5160660383935016e-03, -1.8201496328580638e-03, 15 | 2.1305075891978098e-01 ] 16 | -------------------------------------------------------------------------------- /python/recordings/20230905_162344/camera_extrinsics.pkl: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Jcparkyn/dpoint/a108c19b9b240c1531b2d30ed47ba18bc604862e/python/recordings/20230905_162344/camera_extrinsics.pkl -------------------------------------------------------------------------------- /python/recordings/20230905_162344/scan-original.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Jcparkyn/dpoint/a108c19b9b240c1531b2d30ed47ba18bc604862e/python/recordings/20230905_162344/scan-original.jpg -------------------------------------------------------------------------------- /python/recordings/20230905_162344/scan.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Jcparkyn/dpoint/a108c19b9b240c1531b2d30ed47ba18bc604862e/python/recordings/20230905_162344/scan.jpg -------------------------------------------------------------------------------- /python/requirements.txt: -------------------------------------------------------------------------------- 1 | # Only add direct dependencies to this file. 2 | # Some libraries (e.g. bleak) need different transitive dependencies per platform. 3 | bleak==0.21.1 4 | numba==0.58.1 5 | numpy==1.26.2 6 | PyQt6==6.6.0 7 | pyquaternion==0.9.9 8 | vispy==0.14.1 9 | opencv-contrib-python==4.8.1.78 10 | scipy==1.11.4 -------------------------------------------------------------------------------- /python/run_marker_tracker.py: -------------------------------------------------------------------------------- 1 | from app.marker_tracker import run_tracker 2 | 3 | """ 4 | This is a utility script to run the marker tracker without any of the other parts (BLE, filtering, GUI, etc). 5 | """ 6 | 7 | if __name__ == "__main__": 8 | run_tracker(None) -------------------------------------------------------------------------------- /python/test/test_filter.py: -------------------------------------------------------------------------------- 1 | from approvaltests.approvals import verify 2 | import numpy as np 3 | from filter import ( 4 | initial_state, 5 | state_transition, 6 | state_transition_jacobian, 7 | imu_measurement, 8 | camera_measurement, 9 | i_vel, 10 | i_quat, 11 | i_pos, 12 | i_av, 13 | i_acc, 14 | state_size, 15 | FilterState, 16 | ) 17 | 18 | 19 | def initial_state_for_tests(): 20 | state = np.zeros(state_size, dtype=np.float64) 21 | state[i_quat] = [1, 0, 0, 0] 22 | statecov = np.eye(state_size) * 0.01 23 | return FilterState(state, statecov) 24 | 25 | 26 | def test_initial_state(): 27 | fs = initial_state() 28 | assert len(fs.state) == 22 29 | assert fs.statecov.shape == (22, 22) 30 | 31 | 32 | def test_state_transition_jacobian(): 33 | fs = initial_state_for_tests() 34 | fs.state[i_vel] = [1, 2, 3] 35 | fs.state[i_av] = [4, 5, 6] 36 | fs.state[i_acc] = [7, 8, 9] 37 | jacobian = state_transition_jacobian(fs.state) 38 | verify(jacobian) 39 | 40 | 41 | def test_state_transition(): 42 | fs = initial_state_for_tests() 43 | fs.state[i_vel] = [1, 2, 3] 44 | fs.state[i_av] = [4, 5, 6] 45 | fs.state[i_acc] = [7, 8, 9] 46 | st = state_transition(fs.state) 47 | verify(st) 48 | 49 | 50 | def test_imu_measurement(): 51 | fs = initial_state_for_tests() 52 | fs.state[i_vel] = [1, 2, 3] 53 | fs.state[i_av] = [4, 5, 6] 54 | fs.state[i_acc] = [7, 8, 9] 55 | measurement = imu_measurement(fs.state) 56 | verify(measurement) 57 | 58 | 59 | def test_camera_measurement(): 60 | fs = initial_state_for_tests() 61 | fs.state[i_pos] = [1, 2, 3] 62 | fs.state[i_quat] = [4, 5, 6, 7] 63 | measurement = camera_measurement(fs.state) 64 | verify(measurement) 65 | -------------------------------------------------------------------------------- /python/test/test_filter.test_camera_measurement.approved.txt: -------------------------------------------------------------------------------- 1 | (array([1., 2., 3., 4., 5., 6., 7.]), array([[0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 2 | 0., 0., 0., 0., 0., 0.], 3 | [0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 4 | 0., 0., 0., 0., 0., 0.], 5 | [0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 6 | 0., 0., 0., 0., 0., 0.], 7 | [1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 8 | 0., 0., 0., 0., 0., 0.], 9 | [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 10 | 0., 0., 0., 0., 0., 0.], 11 | [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 12 | 0., 0., 0., 0., 0., 0.], 13 | [0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 14 | 0., 0., 0., 0., 0., 0.]])) 15 | -------------------------------------------------------------------------------- /python/test/test_filter.test_imu_measurement.approved.txt: -------------------------------------------------------------------------------- 1 | (array([ -7. , -8. , -18.81, 4. , 5. , 6. ]), array([[-28. , -0. , 37.62, -16. , 0. , 0. , 0. , 0. , 2 | 0. , 0. , 0. , 0. , 0. , -1. , -0. , 0. , 3 | 1. , 0. , 0. , 0. , 0. , 0. ], 4 | [-32. , -37.62, -0. , 14. , 0. , 0. , 0. , 0. , 5 | 0. , 0. , 0. , 0. , 0. , 0. , -1. , -0. , 6 | 0. , 1. , 0. , 0. , 0. , 0. ], 7 | [-75.24, 16. , -14. , -0. , 0. , 0. , 0. , 0. , 8 | 0. , 0. , 0. , 0. , 0. , -0. , 0. , -1. , 9 | 0. , 0. , 1. , 0. , 0. , 0. ], 10 | [ 0. , 0. , 0. , 0. , 1. , 0. , 0. , 0. , 11 | 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 12 | 0. , 0. , 0. , 1. , 0. , 0. ], 13 | [ 0. , 0. , 0. , 0. , 0. , 1. , 0. , 0. , 14 | 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 15 | 0. , 0. , 0. , 0. , 1. , 0. ], 16 | [ 0. , 0. , 0. , 0. , 0. , 0. , 1. , 0. , 17 | 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 18 | 0. , 0. , 0. , 0. , 0. , 1. ]])) 19 | -------------------------------------------------------------------------------- /python/test/test_filter.test_state_transition.approved.txt: -------------------------------------------------------------------------------- 1 | [0. 2. 2.5 3. 0. 0. 0. 1. 2. 3. 7. 8. 9. 0. 0. 0. 0. 0. 2 | 0. 0. 0. 0. ] 3 | -------------------------------------------------------------------------------- /python/test/test_filter.test_state_transition_jacobian.approved.txt: -------------------------------------------------------------------------------- 1 | [[ 0. -2. -2.5 -3. -0. -0. -0. 0. 0. 0. 0. 0. 0. 0. 2 | 0. 0. 0. 0. 0. 0. 0. 0. ] 3 | [ 2. 0. 3. -2.5 0.5 -0. 0. 0. 0. 0. 0. 0. 0. 0. 4 | 0. 0. 0. 0. 0. 0. 0. 0. ] 5 | [ 2.5 -3. 0. 2. 0. 0.5 -0. 0. 0. 0. 0. 0. 0. 0. 6 | 0. 0. 0. 0. 0. 0. 0. 0. ] 7 | [ 3. 2.5 -2. 0. -0. 0. 0.5 0. 0. 0. 0. 0. 0. 0. 8 | 0. 0. 0. 0. 0. 0. 0. 0. ] 9 | [ 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 10 | 0. 0. 0. 0. 0. 0. 0. 0. ] 11 | [ 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 12 | 0. 0. 0. 0. 0. 0. 0. 0. ] 13 | [ 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 14 | 0. 0. 0. 0. 0. 0. 0. 0. ] 15 | [ 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 1. 0. 0. 0. 16 | 0. 0. 0. 0. 0. 0. 0. 0. ] 17 | [ 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 1. 0. 0. 18 | 0. 0. 0. 0. 0. 0. 0. 0. ] 19 | [ 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 1. 0. 20 | 0. 0. 0. 0. 0. 0. 0. 0. ] 21 | [ 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 1. 22 | 0. 0. 0. 0. 0. 0. 0. 0. ] 23 | [ 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 24 | 1. 0. 0. 0. 0. 0. 0. 0. ] 25 | [ 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 26 | 0. 1. 0. 0. 0. 0. 0. 0. ] 27 | [ 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 28 | 0. 0. 0. 0. 0. 0. 0. 0. ] 29 | [ 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 30 | 0. 0. 0. 0. 0. 0. 0. 0. ] 31 | [ 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 32 | 0. 0. 0. 0. 0. 0. 0. 0. ] 33 | [ 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 34 | 0. 0. 0. 0. 0. 0. 0. 0. ] 35 | [ 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 36 | 0. 0. 0. 0. 0. 0. 0. 0. ] 37 | [ 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 38 | 0. 0. 0. 0. 0. 0. 0. 0. ] 39 | [ 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 40 | 0. 0. 0. 0. 0. 0. 0. 0. ] 41 | [ 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 42 | 0. 0. 0. 0. 0. 0. 0. 0. ] 43 | [ 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 44 | 0. 0. 0. 0. 0. 0. 0. 0. ]] 45 | -------------------------------------------------------------------------------- /setup-guide.md: -------------------------------------------------------------------------------- 1 | # Setup guide 2 | 3 | As mentioned in the [readme](./README.md), the project is intended as a proof-of-concept (not a "plug-and-play" DIY project), so I can't guarantee that all steps will be easy or perfectly documented. If that's not enough to discourage you, follow the steps below to build and run D-POINT on your own hardware. 4 | 5 | Before continuing, [clone](https://docs.github.com/en/repositories/creating-and-managing-repositories/cloning-a-repository) this repository using git: 6 | 7 | ```bash 8 | git clone https://github.com/jcparkyn/dpoint.git 9 | ``` 10 | 11 | ## Building the stylus 12 | 13 |
14 | Tools required 15 | 16 | - A 3D printer, for printing the stylus body. I used a Creality Ender 3, but most types of 3D printer should work. 17 | - Hot air rework station, for soldering the force sensor. You might be able to make do with a soldering iron, but I wouldn't recommend it. 18 | - A soldering iron, for soldering the other electronics. 19 | - An inkjet or laser printer, for printing the ArUco markers. 20 | 21 |
22 | 23 |
24 | Parts required 25 | 26 | - Development board: [Seeed Studio XIAO nRF52840 Sense](https://www.seeedstudio.com/Seeed-XIAO-BLE-Sense-nRF52840-p-5253.html). 27 | - Force sensor: Alps Alpine [HSFPAR003A](https://tech.alpsalpine.com/e/products/detail/HSFPAR003A/) or [HSFPAR004A](https://tech.alpsalpine.com/e/products/detail/HSFPAR004A/) (preferred). These can be a little hard to find, but unfortunately, there aren't many other off-the-shelf options aside from FSRs. The [HSFPAR007A](https://tech.alpsalpine.com/e/products/detail/HSFPAR007A/) will also work, but you'll have to modify the PCB footprint. 28 | - Custom PCB: Order this from your PCB manufacturer of choice, using the provided [gerber files](./electronics/gerbers/). Use a PCB thickness of 1.6mm. 29 | - Printer filament. I used PLA, but most hard plastics should work. 30 | - A 6.5x10mm compression spring. I used one from a spring kit (like [this one](https://www.ebay.com.au/itm/175482659706)), but he spring tension does not need to match exactly. 31 | - A 10440 lithium-ion battery (e.g., [this one](https://www.ebay.com/itm/194025159718)). 32 | - Wire, 22-24 AWG for soldering. 33 | - Something to use as a nib for the stylus. I used a [Wacom replacement nib](https://estore.wacom.com/en-au/wacom-standard-replacement-nibs-previous-gen.html), but you might be able to substitute this with something else (e.g., 3D printer filament). If you do, make sure to adjust the dimensions on the 3D model to match. 34 | 35 |
36 | 37 | ### Micro-controller code 38 | 39 | To upload the code to the development board: 40 | 1. Follow [these instructions](https://wiki.seeedstudio.com/XIAO_BLE/#getting-started) to download the necessary software. 41 | 1. Open [microcontroller/dpoint-arduino](./microcontroller/dpoint-arduino/) in the Arduino IDE. 42 | 1. Connect the board to your computer using a USB-C cable. 43 | 1. Click **Tools > Board > Seeed nRF52 Boards > Seeed XIAO nRF52840 Sense**. 44 | 1. Upload the code. 45 | 46 | ### Build steps 47 | 48 | 1. Print all three STL files from [print/export/](./print/export/) on your 3D printer. 49 | 1. Solder the force sensor to the custom PCB. Make sure to align the top-left of the force sensor (marked with a very small circle) with the circle on the PCB. 50 | 1. Solder the force sensor PCB, battery, and development board together using wires, following the schematic below. Make sure to cut wires the right length to fit the stylus body. 51 | 1. Place the force sensor PCB, battery, and microcontroller into the bottom half of the 3D printed stylus. The stylus body also has a hole for a switch to disconnect the battery during development, but you can ignore this. 52 | 1. Find a small, flat piece of metal or hard plastic, and glue it to the back of the 3D printed nib base where it contacts the force sensor. 53 | 1. Insert the nib into the 3D printed nib base, then fit it into the front of the stylus using the spring. You may need to hold it in place until the stylus is fully assembled. 54 | 1. Add the top half of the stylus, and secure it in place using tape or glue. 55 | 1. Print out the ArUco markers from [markers/stylus-markers.pdf](./markers/stylus-markers.pdf) at 100% scale. This has two copies of each marker (and some spares), but you only need one. 56 | 1. Cut carefully along each of the lines around the markers. The white borders are important, so don't cut them off. 57 | 1. Glue the markers to the stylus using a glue stick (or your glue of choice). Place them according to the diagram below. 58 | 1. Finally, you'll also need a ChArUco board for camera calibration, which you can print using the pattern from [markers/](./markers/). You can either attach this to a flat board, or leave it on your desk. 59 | 60 | 61 | Wiring diagram: 62 | 63 | 64 | 65 | Marker placement: 66 | 67 | 68 | 69 | ## Software setup 70 | 71 | The python code requires Python 3.11. I've tried to make sure the codebase works cross-platform, but it's currently only been tested on Windows 10. 72 | You'll also need a relatively powerful CPU to run the application with its current settings. I used an i5-13600KF, but you should be able to get away with something a couple of generations older. If you have performance problems, try changing the settings in [configuration and tuning](#configuration-and-tuning). 73 | 74 | 1. Open the [python/](./python/) directory (`cd python`). 75 | 1. Create and activate a Python [virtual environment](https://docs.python.org/3/tutorial/venv.html#creating-virtual-environments) (optional, but recommended). 76 | 1. Install dependencies: 77 | ```bash 78 | python -m pip install -r requirements.txt 79 | ``` 80 | 1. Run the camera calibration script (see steps [below](#camera-calibration)). If you're using a Logitech C922 webcam like mine, you may be able to skip this step at the cost of some accuracy. 81 | 1. Optional, for better accuracy: Run the [marker calibration script](./python/calibrate_markers.py). Take at least 15 photos of the stylus from different angles (using your calibrated webcam), put them in the `IMAGE_PATH` directory for the script, then run the script. 82 | 83 | ## Running the application 84 | 85 | Run [main.py](./python/main.py). If you have the `python` directory open in VSCode, you can do this by pressing F5, or running: 86 | ```bash 87 | python -m main 88 | ``` 89 | 90 | Put the ChArUco board on your desk in view of the webcam, then press C (with the camera window focused) to calibrate the camera position. 91 | 92 | Make sure the stylus is on - there should be a blue light (if not connected) or green light (if connected) near the USB port. If the stylus is asleep, shake it twice vertically to wake it up. 93 | 94 | ## Camera calibration 95 | To calibrate your camera: 96 | 1. Take 5-10 photos of the ChArUco board from different angles using your webcam. 97 | 2. Put the images into `calibration_pics/f30/`. 98 | 3. Run `calibrate_camera.py`. 99 | 100 | This calibrates the camera intrinsic parameters, but not the camera position or rotation. These are calibrated at run-time. 101 | 102 | ## Configuration and tuning 103 | Most of the codebase is set up for my own hardware and requirements, so it might not work perfectly for other setups. Here are a list of things to check: 104 | 105 | - **Camera exposure:** Auto-exposure is disabled, so you might need to adjust the exposure level in `marker_tracker.py`. Make it as low as you can without the markers failing to detect. Adding a light source will help. 106 | - **Camera time delay:** The code tries to account for the time delay between the camera and IMU readings, which may be different on different systems. Adjust the value of `camera_delay` in `app.py` until you find the value that gives the best accuracy. 107 | - **Camera focus:** The code tries to maintain focus on the stylus markers, but this relies on pre-measured focus values for my camera. If the focus looks wrong for your camera, try adjusting the values in `focus_targets` (`marker_tracker.py`). 108 | - **RTS smoothing:** The filter uses RTS smoothing, which slightly improves accuracy but can be computationally intensive. If you have performance issues, set `smoothing_length` in `app.py` to a smaller value (or zero). 109 | - **Marker search area:** The marker tracker tries to reduce the area of the frame where it looks for markers (this is the green box on the camera preview). If you have performance issues, adjust the value of `expand` in `marker_tracker.py` to reduce the search area for markers in each frame. 110 | - **Camera frame rate:** This isn't directly exposed in the code, and depends on the camera you're using (OpenCV can't reliably set it), but you may want to try running your webcam at 15FPS (or another frame rate) instead of 30FPS. This should give relatively similar accuracy down to about 10-15FPS, and reduce computational requirements, but it might affect the `camera_delay`. 111 | - **IMU update rate:** You can try adjusting the IMU update rate by changing `delayMs` in the [Arduino code](./microcontroller/dpoint-arduino/dpoint-arduino.ino). Make sure to update the corresponing `dt` in `app.py`, and test that the rate you chose is actually being achieved (this is limited by BLE settings and some other things). Ideally, `dt` should be computed automatically from timestamps in the BLE packets, but I removed this at one point and didn't get around to re-implementing it. 112 | - **VSync:** For minimum latency, make sure to disable VSync in your graphics card settings. 113 | - **Camera resolution:** If you try running this at a resolution other than 1920x1080, you'll probably have to tweak some other settings (and re-calibrate your camera). 114 | - **Stylus length:** Depending on the nib you use, the length of your stylus might be slightly different than mine. Adjust `STYLUS_LENGTH` in `dimensions.py` to correct this. 115 | - **Pressure sensor:** With the current wiring, the force sensor gets lots of interference when the USB cable is connected. Make sure to run it wirelessly for full accuracy. The code also tries to account for drift by re-calibrating the sensor whenever it's away from the page, but if you want to draw on 3D objects you'll need to disable this. --------------------------------------------------------------------------------