├── merppg.pdf ├── model.onnx ├── example.py ├── LICENSE ├── main.py └── README.md /merppg.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/KegangWangCCNU/ME-rPPG/HEAD/merppg.pdf -------------------------------------------------------------------------------- /model.onnx: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/KegangWangCCNU/ME-rPPG/HEAD/model.onnx -------------------------------------------------------------------------------- /example.py: -------------------------------------------------------------------------------- 1 | from main import * 2 | 3 | state = load_state('state.json') 4 | model = load_model('model.onnx') 5 | 6 | bvp = [] 7 | video_path = r"C:\Users\16252\Downloads\29_3.mp4" 8 | cap = cv2.VideoCapture(video_path) 9 | while 1: 10 | _, frame = cap.read() 11 | if not _: 12 | break 13 | facial_img = crop_face(frame) 14 | output, state = model(facial_img, state) 15 | bvp.append(output) 16 | cap.release() 17 | 18 | print(f'Heart Rate is {get_hr(bvp):.2f}') 19 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2025 汪汪 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /main.py: -------------------------------------------------------------------------------- 1 | import mediapipe as mp 2 | import cv2, json 3 | import onnxruntime as ort 4 | import numpy as np 5 | from scipy.signal import welch 6 | 7 | def crop_face(frame): 8 | with mp.solutions.face_mesh.FaceMesh(max_num_faces=1, static_image_mode=True) as fm: 9 | landmarks = fm.process(frame).multi_face_landmarks[0].landmark 10 | xaxis = [i.x for i in landmarks if i.x>0] 11 | yaxis = [i.y for i in landmarks if i.y>0] 12 | xmin, xmax = min(xaxis)*frame.shape[1], max(xaxis)*frame.shape[1] 13 | ymin, ymax = min(yaxis)*frame.shape[0], max(yaxis)*frame.shape[0] 14 | img = frame[round(ymin):round(ymax), round(xmin):round(xmax), ::-1].astype('float32')/255 15 | return cv2.resize(img, (36, 36), interpolation=cv2.INTER_AREA) 16 | 17 | def load_state(path): 18 | with open(path, 'r') as f: 19 | return json.load(f) 20 | 21 | def load_model(path): 22 | model = ort.InferenceSession(path) 23 | def run(img, state, dt=1/30): 24 | result = model.run(None, {"arg_0.1": img[None, None], "onnx::Mul_37": [dt], **state}) 25 | bvp, new_state = result[0][0, 0], result[1:] 26 | return bvp, dict(zip(state, new_state)) 27 | return run 28 | 29 | def get_hr(y, sr=30, hr_min=30, hr_max=180): 30 | p, q = welch(y, sr, nfft=int(1e5 / sr), nperseg=np.min((len(y) - 1, 256))) 31 | return p[(p > hr_min / 60) & (p < hr_max / 60)][np.argmax( 32 | q[(p > hr_min / 60) & (p < hr_max / 60)])] * 60 33 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # ME-rPPG 2 | The official code of [this paper](https://rppgdemo.kegang.wang/merppg.pdf). We implemented memory-efficient real-time step-by-step inference using ME-rPPG. 3 | ![image](https://github.com/user-attachments/assets/ca75b574-834f-41f2-b08e-7b2dd2602d67) 4 | 5 | ## Usage 6 | 7 | The ME model is now available in [open-rppg](https://github.com/KegangWangCCNU/open-rppg). 8 | 9 | ## Inference Example 10 | 11 | Performing step-by-step inference through the following code. 12 | 13 | ```python 14 | state = load_state('state.json') 15 | model = load_model('model.onnx') 16 | 17 | while True: 18 | ........ 19 | facial_img = crop_face(frame) # Cropping to a 36×36x3 RGB facial image 20 | output, state = model(facial_img, state) # Computing the BVP and updating the state 21 | ........ 22 | ``` 23 | 24 | ## Web Browser Inference Demo 25 | The following is our implemented web browser inference demo, which operates directly within the browser without requiring video uploads or GPU acceleration. 26 | Demo URL: [https://rppgdemo.kegang.wang/](https://rppgdemo.kegang.wang/) 27 | Source code: [https://github.com/Health-HCI-Group/ME-rPPG-demo](https://github.com/Health-HCI-Group/ME-rPPG-demo) 28 | 29 | ## Training Framework 30 | The ME-rPPG was trained on RLAP using PhysBench, with the full code coming soon. 31 | 32 | ## Citation 33 | ``` 34 | @article{wang2025memory, 35 | title={Memory-efficient Low-latency Remote Photoplethysmography through Temporal-Spatial State Space Duality}, 36 | author={Wang, Kegang and Tang, Jiankai and Fan, Yuxuan and Ji, Jiatong and Shi, Yuanchun and Wang, Yuntao}, 37 | journal={arXiv preprint arXiv:2504.01774}, 38 | year={2025} 39 | } 40 | ``` 41 | --------------------------------------------------------------------------------