├── README.md
├── TIPA_basic_run.ipynb
├── TIPA_library
    ├── main
    │   ├── data_preparation.py
    │   └── thermal_image_processing.py
    └── utils
    │   ├── compute_frequency_grid.py
    │   ├── gausswin.py
    │   ├── overlap_matrix.py
    │   ├── overlap_windows.py
    │   ├── pca_basic.py
    │   ├── rvs.py
    │   ├── simpleDRQ.py
    │   └── timshow.py
├── data
    └── readme.md
└── figures
    ├── figure1.png
    ├── figure2.png
    ├── figure3.png
    └── readme.md


/README.md:
--------------------------------------------------------------------------------
 1 | # TIPA opensource toolkit project
 2 | TIPA: Thermal Imaging-based Physiological and Affective computing
 3 | 
 4 | Lead contributor: Dr. Youngjun Cho, Department of Computer Science, University College London (UCL)
 5 | 
 6 | 
 7 | 
 8 | ## Brief guideline
 9 | 
10 | 1. Download Anaconda (latest version) - Python 3.7 (recommended)
11 | 
12 |     https://www.anaconda.com/distribution/
13 | 
14 | 
15 | 2. Install basic libraries on the Conda console.
16 | 
17 |     conda install -c conda-forge opencv
18 |     
19 |     conda install scikit-learn
20 |     
21 |     pip install --upgrade numpy
22 |     
23 |     pip install --upgrade matplotlib
24 |     
25 |     pip install packaging
26 |     
27 |     conda install -c anaconda scipy
28 | 
29 | 
30 |     * For your information
31 |     
32 |         print(python_version()) 
33 |         
34 |         3.7.3
35 |         
36 |         print(np.version.version)
37 |         
38 |         1.16.4
39 |         
40 |         print(cv2.__version__)
41 |         
42 |         3.4.2
43 |         
44 |         scipy (1.3.1)
45 | 
46 | 3. Run "<a href="https://github.com/deepneuroscience/TIPA/blob/master/TIPA_basic_run.ipynb">TIPA_basic_run.ipynb</a>" on the Jupyter notebook 
47 | 
48 |     You can find a basic instruction on the notebook.
49 |     
50 | 
51 | 
52 |     
53 |     
54 | ## Key Reference
55 | [1] Youngjun Cho and Nadia Bianchi-Berthouze. 2019. Physiological and Affective Computing through Thermal Imaging: A Survey. arXiv:1908.10307 [cs], http://arxiv.org/abs/1908.10307
56 | 
57 | ### Further Technical References
58 | [2] Cho, Y., Julier, S.J., Marquardt, N. and Bianchi-Berthouze, N., 2017. Robust tracking of respiratory rate in high-dynamic range scenes using mobile thermal imaging. Biomedical optics express, 8(10), pp.4480-4503. https://doi.org/10.1364/BOE.8.004480
59 | 
60 | [3] Cho, Y., Julier, S.J. and Bianchi-Berthouze, N., 2019. Instant Stress: Detection of Perceived Mental Stress Through Smartphone Photoplethysmography and Thermal Imaging. JMIR mental health, 6(4), p.e10140. https://doi.org/10.2196/10140
61 | 
62 | [4] Cho, Y., Bianchi-Berthouze, N. and Julier, S.J., 2017. DeepBreath: Deep learning of breathing patterns for automatic stress recognition using low-cost thermal imaging in unconstrained settings. In 2017 Seventh International Conference on Affective Computing and Intelligent Interaction (ACII) (pp. 456-463). IEEE. https://doi.org/10.1109/ACII.2017.8273639
63 | 
64 | [5] Cho, Y., Bianchi-Berthouze, N., Marquardt, N. and Julier, S.J., 2018. Deep Thermal Imaging: Proximate Material Type Recognition in the Wild through Deep Learning of Spatial Surface Temperature Patterns. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, ACM. https://doi.org/10.1145/3173574.3173576
65 | 


--------------------------------------------------------------------------------
/TIPA_library/main/data_preparation.py:
--------------------------------------------------------------------------------
 1 | # TIPA (Thermal Imaging-based Physiological and Affective computing) open-source project
 2 | 
 3 | ## Author(s): Dr. Youngjun Cho*(Assistant Professor, UCL Computer Science)
 4 | ## * http://youngjuncho.com
 5 | 
 6 | import numpy as np
 7 | import matplotlib.pyplot as plt
 8 | from ipywidgets import widgets
 9 | from TIPA_library.main.thermal_image_processing import optimal_quantization_range as oq_range
10 |     
11 | class data_preparation_TIPA_protocol:
12 |     def __init__(self, file_name, known_height, known_width):
13 |         self.frame_height = known_height
14 |         self.frame_width = known_width
15 |         self.file = np.fromfile(file_name, dtype='uint16')
16 |         self.file = np.reshape(self.file, [-1, self.frame_height*self.frame_width + 4])#
17 |         self.file = (np.transpose(self.file).astype('uint16'))
18 |         
19 |         self.tracked_matrix=[]
20 |         self.thermal_matrix = self.get_thermal_matrix(self.file)
21 |         self.time_stamp = self.get_time_stamp(self.file)
22 |         self.time_stamp[0]=0
23 |         
24 |     def get_thermal_matrix(self, file):
25 |         # assigning pixels to image-frame matrix 'img_frames'
26 |         num_frames = np.size(file, 1)
27 |         file_segment=file[0:self.frame_height*self.frame_width, 0:num_frames]; 
28 |         img_frames = np.reshape(file_segment[0:self.frame_height*self.frame_width, :], [self.frame_height, self.frame_width, num_frames])
29 |         #img_frames = np.reshape(file[0:320*240, :], [frame_height, frame_width, num_frames])
30 |         img_frames = (img_frames - 27315) / 100 # celcius
31 |         return img_frames
32 | 
33 | 
34 |     def get_time_stamp(self, file):
35 |         # getting time array from the last two rows of read-in file
36 |         pre_time_arr = file[self.frame_height*self.frame_width+2:, :]
37 |         pre_time_arr_high = (np.round(pre_time_arr[:, :]) & 255) << 8
38 |         pre_time_arr_low = np.round(pre_time_arr[:, :]) >> 8
39 |         time_stamp = (pre_time_arr_high + pre_time_arr_low).astype('uint32')
40 |         new_time_stamp = ((np.round(time_stamp[0, :]) << 16) + time_stamp[1, :]) / 1000
41 |         return new_time_stamp
42 | 
43 | 
44 |     # To show thermal 2D sequences interactively
45 |     def interactive_imshow_cond(self, frame_number):
46 |         plt.figure(2)
47 |         plt.imshow(self.thermal_matrix[:,:,frame_number], cmap=plt.get_cmap('hot'))
48 |         plt.colorbar()
49 |     #     plt.show()
50 |     
51 |     def interactive_imshow_cond2(self, frame_number, thermal_range):
52 |         plt.figure(2)
53 |         plt.imshow(self.thermal_matrix[:,:,frame_number], cmap=plt.get_cmap('hot'), vmin=thermal_range[0], vmax=thermal_range[1])
54 |         plt.colorbar()
55 |     #     plt.show()
56 |     
57 |     def interactive_imshow_cond3(self, frame_number):
58 |         plt.figure(2)
59 |         min_T, max_T=oq_range(self.thermal_matrix[:,:,frame_number])
60 |         plt.imshow(self.thermal_matrix[:,:,frame_number], cmap=plt.get_cmap('hot'), vmin=min_T, vmax=max_T)
61 |         plt.colorbar()
62 |     #     plt.show()
63 |     
64 |     def interactive_imshow_cond4(self, frame_number):
65 |         plt.figure(2)
66 |         plt.imshow(self.tracked_matrix[:,:,frame_number], cmap=plt.get_cmap('hot'))
67 |         plt.colorbar()
68 |     #     plt.show()    
69 |     
70 | class data_preparation_raw_matrix:
71 |     def __init__(self, matrix, framerate):
72 |         self.frame_height = matrix.shape[0]
73 |         self.frame_width = matrix.shape[1]
74 |         self.frame_length = matrix.shape[2]
75 |         self.thermal_matrix = matrix
76 |         self.time_stamp = np.arange(0,self.frame_length/framerate,1/framerate)


--------------------------------------------------------------------------------
/TIPA_library/main/thermal_image_processing.py:
--------------------------------------------------------------------------------
  1 | # TIPA (Thermal Imaging-based Physiological and Affective computing) open-source project
  2 | 
  3 | ## Author(s): Dr. Youngjun Cho*(Assistant Professor, UCL Computer Science)
  4 | ## * http://youngjuncho.com
  5 | 
  6 | import numpy as np
  7 | import matplotlib.pyplot as plt
  8 | import copy
  9 | import cv2
 10 | from packaging import version
 11 |     
 12 | ''' Optimal Quantization: Optimal Quantization technique adaptively constructs a color mapping of absolute temperature to improve segmentation, classification and tracking
 13 | 
 14 | Reference: Cho, Y., Julier, S. J., Marquardt, N., & Bianchi-Berthouze, N. (2017). Robust tracking of respiratory rate in high-dynamic range scenes using mobile thermal imaging. Biomedical optics express, 8(10), 4480-4503.
 15 | '''
 16 | # output = optimal_quantization(mat, True)  
 17 | def optimal_quantization(t2d_data_original, print_mode = False):
 18 | 
 19 |     t2d_data= np.copy(t2d_data_original)
 20 |     
 21 |     vector_data= np.reshape(t2d_data, t2d_data.shape[0]*t2d_data.shape[1])
 22 |     min_T=np.percentile(vector_data, 2.5)  #2.5 percentile point
 23 |     max_T=np.percentile(vector_data, 97.5) #97.5 percentile point
 24 | 
 25 |     t2d_data[np.where(t2d_data > max_T)]=max_T
 26 |     t2d_data[np.where(t2d_data < min_T)]=min_T
 27 | 
 28 |     opt_T=min_T
 29 |     count=0
 30 |     while True:
 31 |         count+=1
 32 |         mean_back=np.mean(t2d_data[np.where(t2d_data <= opt_T)])
 33 |         mean_obj=np.mean(t2d_data[np.where(t2d_data > opt_T)])
 34 |         if np.abs(opt_T - (mean_back+mean_obj)/2 )<0.005:
 35 |             break;
 36 |         else:
 37 |             opt_T=(mean_back+mean_obj)/2
 38 | 
 39 |     min_T=opt_T
 40 | 
 41 |     if print_mode:
 42 |         print("optimal thermal range is [%f, %f]"%(min_T,max_T))
 43 | 
 44 |     t2d_data[np.where(t2d_data < min_T)]=min_T
 45 | 
 46 |     quantized_t_img=255*(t2d_data-min_T)/(max_T-min_T)
 47 |     quantized_t_img = quantized_t_img.astype(np.uint8)
 48 | 
 49 |     return quantized_t_img                  
 50 |                          
 51 | def optimal_quantization_range(t2d_data):
 52 |     
 53 |     vector_data= np.reshape(t2d_data, t2d_data.shape[0]*t2d_data.shape[1])
 54 |     min_T=np.percentile(vector_data, 2.5)  #2.5 percentile point
 55 |     max_T=np.percentile(vector_data, 97.5) #97.5 percentile point
 56 | 
 57 |     t2d_data[np.where(t2d_data > max_T)]=max_T
 58 |     t2d_data[np.where(t2d_data < min_T)]=min_T
 59 | 
 60 |     opt_T=min_T
 61 |     count=0
 62 |     while True:
 63 |         count+=1
 64 |         mean_back=np.mean(t2d_data[np.where(t2d_data <= opt_T)])
 65 |         mean_obj=np.mean(t2d_data[np.where(t2d_data > opt_T)])
 66 |         if np.abs(opt_T - (mean_back+mean_obj)/2 )<0.005:
 67 |             break;
 68 |         else:
 69 |             opt_T=(mean_back+mean_obj)/2
 70 | 
 71 |     min_T=opt_T
 72 | 
 73 |     return min_T,max_T     
 74 |     
 75 | '''
 76 |  This is the classical quantization method using fixed points (temperature range of interest).
 77 | '''
 78 | # output = nonoptimal_quantization(mat, T1,Tk, True)  
 79 | def nonoptimal_quantization(t2d_data, min_T, max_T, print_mode = False):
 80 |     if print_mode:
 81 |         print("your fixed thermal range of interest is [%f, %f]"%(min_T,max_T))
 82 | 
 83 |     quantized_t_img= np.copy(t2d_data)
 84 |     
 85 |     quantized_t_img[np.where(quantized_t_img > max_T)]=max_T
 86 |     quantized_t_img[np.where(quantized_t_img < min_T)]=min_T
 87 | 
 88 |     quantized_t_img=255*(quantized_t_img-min_T)/(max_T-min_T)
 89 |     quantized_t_img = quantized_t_img.astype(np.uint8)
 90 | 
 91 |     return quantized_t_img    
 92 | 
 93 | 
 94 | 
 95 |     
 96 | ''' Quantization-enabled thermal image tracking
 97 | 
 98 | Reference [1]: Cho, Y., Julier, S. J., Marquardt, N., & Bianchi-Berthouze, N. (2017). Robust tracking of respiratory rate in high-dynamic range scenes using mobile thermal imaging. Biomedical optics express, 8(10), 4480-4503.
 99 | Reference [2]: Cho, Y., Julier, S. J., & Bianchi-Berthouze, N. (2019). Instant Stress: Detection of Perceived Mental Stress Through Smartphone Photoplethysmography and Thermal Imaging. JMIR mental health, 6(4), e10140.
100 | Reference [3]: Cho, Y., Bianchi-Berthouze, N., Oliveira, M., Holloway, C., & Julier, S. (2019). Nose Heat: Exploring Stress-induced Nasal Thermal Variability through Mobile Thermal Imaging. In 2019 eighth International Conference on Affective Computing and Intelligent Interaction (ACII). IEEE.
101 | '''
102 | 
103 | ## eg. ROI_seq, tracked_img=thermal_tracker(data.thermal_matrix, 'MEDIANFLOW', False)     
104 | def thermal_tracker(mat_3d, quantization_method='optimal', im_tracker_name = 'TLD', print_mode=False, nonfixed_ROI_size=True, img_viewer=True, min_T=28, max_T=38):
105 | 
106 |     # If using a version of openCV prior to 4.5.1, include all standard
107 |     # trackers
108 |     if version.parse(cv2.__version__) < version.parse("4.5.1"):
109 |         trackers = {"TLD": cv2.TrackerTLD_create,
110 |                     "MEDIANFLOW": cv2.TrackerMedianFlow_create,
111 |                     "GOTURN": cv2.TrackerGOTURN_create,
112 |                     "MOSSE": cv2.TrackerMOSSE_create,
113 |                     "CSRT": cv2.TrackerCSRT_create,
114 |                     "BOOSTING": cv2.TrackerBoosting_create,
115 |                     "MIL": cv2.TrackerMIL_create,
116 |                     "KCF": cv2.TrackerKCF_create}
117 |     # If using a more recent version of OpenCV include legacy trackers
118 |     else:
119 |         trackers = {"TLD": cv2.legacy.TrackerTLD_create,
120 |                     "MEDIANFLOW": cv2.legacy.TrackerMedianFlow_create,
121 |                     "GOTURN": cv2.TrackerGOTURN_create,
122 |                     "MOSSE": cv2.legacy.TrackerMOSSE_create,
123 |                     "CSRT": cv2.TrackerCSRT_create,
124 |                     "BOOSTING": cv2.legacy.TrackerBoosting_create,
125 |                     "MIL": cv2.TrackerMIL_create,
126 |                     "KCF": cv2.TrackerKCF_create}
127 |     # If using a version of OpenCV newer than 4.5.2 then include the
128 |     # DaSiamRPN tracker
129 |     if version.parse(cv2.__version__) > version.parse("4.5.2"):
130 |         trackers["DASIAMRPN"] = cv2.TrackerDaSiamRPN_create
131 | 
132 |     try:
133 |         im_tracker = trackers[im_tracker_name]()
134 |     except KeyError as E:
135 |         print("Tracker could not be found, please check it is included in "
136 |               "your version of OpenCV!")
137 |         raise E
138 |     greyscale = True
139 |     if im_tracker_name in ["GOTURN", "BOOSTING", "DASIAMRPN"]:
140 |         greyscale = False
141 | 
142 |     temp_mat = copy.deepcopy(mat_3d[:,:,0])    
143 |     tracked_imgs = np.zeros(mat_3d.shape,np.uint8)
144 |     
145 |     if quantization_method == 'optimal':
146 |         tracked_imgs[:,:,0] = optimal_quantization(temp_mat, print_mode)
147 |     elif quantization_method == 'non-optimal':
148 |         tracked_imgs[:,:,0] = nonoptimal_quantization(temp_mat, min_T, max_T, print_mode)
149 |         
150 |         
151 |     # ROI selection 
152 |     mROI = np.zeros((4, mat_3d.shape[2]), dtype=int)
153 |     mROI[:,0] = cv2.selectROI('ROI selection', tracked_imgs[:,:,0], False)
154 |     # mROI.shape
155 |     cv2.destroyWindow('ROI selection')
156 | 
157 | #     cv2.namedWindow('TrackViewer')
158 |     
159 |     # Initialization of a tracker
160 | 
161 |     if greyscale:
162 |         out = im_tracker.init(tracked_imgs[:,:,0], tuple(mROI[:,0]))
163 |     else:
164 |         stacked_img = np.stack((tracked_imgs[:, :, 0],) * 3, axis=-1)
165 |         out = im_tracker.init(stacked_img, tuple(mROI[:, 0]))
166 | 
167 | 
168 |     for i in range(1,mat_3d.shape[2]):
169 |         if np.mod(i,20)==0:
170 |             print("frame: %d"%i)
171 |         temp_mat = copy.deepcopy(mat_3d[:,:,i])
172 |         current_frame = np.zeros(tracked_imgs[:,:,0].shape)
173 |         
174 |         if quantization_method == 'optimal':
175 |             current_frame = optimal_quantization(temp_mat, print_mode)
176 |         elif quantization_method == 'non-optimal':
177 |             current_frame = nonoptimal_quantization(temp_mat, min_T, max_T, print_mode)        
178 |         
179 |         
180 |         # Updating the tracker
181 |         if nonfixed_ROI_size:
182 |             if greyscale:
183 |                 result, mROI[:,i] = im_tracker.update(current_frame)
184 |             else:
185 |                 stacked_img = np.stack((current_frame,) * 3, axis=-1)
186 |                 result, mROI[:, i] = im_tracker.update(stacked_img)
187 |         else:
188 |             if greyscale:
189 |                 result, bbox = im_tracker.update(current_frame)
190 |             else:
191 |                 stacked_img = np.stack((current_frame,) * 3, axis=-1)
192 |                 result, bbox = im_tracker.update(stacked_img)
193 |                 
194 |             mROI[2,i]=mROI[2,0]
195 |             mROI[3,i]=mROI[3,0]
196 |             mROI[0,i]=bbox[0]+np.round(bbox[2]/2) -np.round(mROI[2,0]/2)
197 |             mROI[1,i]=bbox[1]+np.round(bbox[3]/2) -np.round(mROI[3,0]/2)
198 |             
199 | 
200 |         # Visualise the tracked ROI
201 |         if result==True:        
202 |             point1 = (int(mROI[0,i]), int(mROI[1,i]))
203 |             point2 = (int(mROI[0,i] + mROI[2,i]), int(mROI[1,i] + mROI[3,i]))
204 |             
205 |             cv2.rectangle(current_frame, point1, point2, (255,255,0), 3, 0)
206 |             tracked_imgs[:,:,i]=current_frame
207 |         else :
208 |             print("tracking error occurred at %d frame"%(i))
209 |             tracked_imgs[:,:,i]=current_frame
210 | 
211 |         if img_viewer:
212 |             cv2.imshow('TrackViewer',current_frame)
213 |                     # Exit if ESC pressed
214 |             k = cv2.waitKey(1) & 0xff
215 |             if k == 27 : break
216 | 
217 | #     cv2.namedWindow('TrackViewer')
218 | #     count=0
219 | #     while True:
220 | #         cv2.imshow('TrackViewer',tracked_imgs[:,:,count])
221 | #         if count<tracked_imgs.shape[2]-1:
222 | #             count+=1
223 | #         # Press ESC to exit
224 | #         k = cv2.waitKey(1) & 0xff
225 | #         if k == 27 : break
226 | #     cv2.destroyAllWindows()
227 | 
228 |             
229 |     return mROI, tracked_imgs
230 | 


--------------------------------------------------------------------------------
/TIPA_library/utils/compute_frequency_grid.py:
--------------------------------------------------------------------------------
 1 | ## Contributors
 2 | ## Dr. Youngjun Cho (Assistant Professor, UCL Computer Science), Chang Liu (MSc in Machine Learning, UCL Computer Science)
 3 | 
 4 | import numpy as np
 5 | 
 6 | ''' 
 7 | Signal processing toolkits for Respiration Variability Spectrogram
 8 | 
 9 | [Reference] 
10 | Cho, Y., Bianchi-Berthouze, N. and Julier, S.J., 2017. 
11 | DeepBreath: Deep learning of breathing patterns for automatic stress recognition using low-cost thermal imaging in unconstrained settings. In 2017 Seventh International Conference on Affective Computing and Intelligent Interaction (ACII) (pp. 456-463). IEEE. https://doi.org/10.1109/ACII.2017.8273639
12 | '''
13 | 
14 | def compute_frequency_grid(oversampling=50):
15 |     T=20 #T = t.max() - t.min()#20
16 |     N=5120#N = len(t)#5120
17 |     ts = T/(N-1)
18 |     df = 1 / (oversampling*ts*N)
19 |     fmax = 1/ (2 * ts)
20 |     return np.arange(df,fmax,df)
21 | 


--------------------------------------------------------------------------------
/TIPA_library/utils/gausswin.py:
--------------------------------------------------------------------------------
 1 | ## Contributors
 2 | ## Dr. Youngjun Cho (Assistant Professor, UCL Computer Science), Chang Liu (MSc in Machine Learning, UCL Computer Science)
 3 | 
 4 | import numpy as np
 5 | 
 6 | ''' 
 7 | Signal processing toolkits for Respiration Variability Spectrogram
 8 | 
 9 | [Reference] 
10 | Cho, Y., Bianchi-Berthouze, N. and Julier, S.J., 2017. 
11 | DeepBreath: Deep learning of breathing patterns for automatic stress recognition using low-cost thermal imaging in unconstrained settings. In 2017 Seventh International Conference on Affective Computing and Intelligent Interaction (ACII) (pp. 456-463). IEEE. https://doi.org/10.1109/ACII.2017.8273639
12 | '''
13 | 
14 |         
15 | def gausswin(L, alpha=2.5):
16 |     N = L - 1
17 |     n = np.arange(0,N+1)-N/2
18 |     w = np.exp(-(1/2)*(alpha*n /(N /2))**2)
19 |     return w
20 | 


--------------------------------------------------------------------------------
/TIPA_library/utils/overlap_matrix.py:
--------------------------------------------------------------------------------
 1 | ## Contributors
 2 | ## Dr. Youngjun Cho (Assistant Professor, UCL Computer Science), Chang Liu (MSc in Machine Learning, UCL Computer Science)
 3 | 
 4 | import numpy as np
 5 | 
 6 | ''' 
 7 | Signal processing toolkits for Respiration Variability Spectrogram
 8 | 
 9 | [Reference] 
10 | Cho, Y., Bianchi-Berthouze, N. and Julier, S.J., 2017. 
11 | DeepBreath: Deep learning of breathing patterns for automatic stress recognition using low-cost thermal imaging in unconstrained settings. In 2017 Seventh International Conference on Affective Computing and Intelligent Interaction (ACII) (pp. 456-463). IEEE. https://doi.org/10.1109/ACII.2017.8273639
12 | '''
13 | 
14 | def overlap_matrix(data,overlap_rate1,window_size):
15 |     matrix_list = []
16 |     start = 0
17 |     end = window_size
18 |     remain = (np.shape(data)[1])
19 |     while remain>=window_size:
20 |         matrix_list.append(data[:,int(round(start+0.01)):int(round(end+0.01))])
21 |         start += overlap_rate1*window_size#start and end overlap 1
22 |         end += overlap_rate1*window_size
23 |         remain -= overlap_rate1*window_size 
24 |     return matrix_list
25 |      


--------------------------------------------------------------------------------
/TIPA_library/utils/overlap_windows.py:
--------------------------------------------------------------------------------
 1 | ## Contributors
 2 | ## Dr. Youngjun Cho (Assistant Professor, UCL Computer Science), Chang Liu (MSc in Machine Learning, UCL Computer Science)
 3 | 
 4 | 
 5 | ''' 
 6 | Signal processing toolkits for Respiration Variability Spectrogram
 7 | 
 8 | [Reference] 
 9 | Cho, Y., Bianchi-Berthouze, N. and Julier, S.J., 2017. 
10 | DeepBreath: Deep learning of breathing patterns for automatic stress recognition using low-cost thermal imaging in unconstrained settings. In 2017 Seventh International Conference on Affective Computing and Intelligent Interaction (ACII) (pp. 456-463). IEEE. https://doi.org/10.1109/ACII.2017.8273639
11 | '''
12 | 
13 | def overlap_windows(data,overlap_rate,window_size):
14 |     window_list = []
15 |     start = 0
16 |     end = window_size
17 |     remain_length = len(data)
18 |     
19 |     while remain_length>=window_size:
20 |         window_list.append(data[int(round(start+0.01)):int(round(end+0.01))])
21 |         start += overlap_rate*window_size#start and end overlap 1
22 |         end += overlap_rate*window_size
23 |         remain_length -= overlap_rate*window_size 
24 |     return window_list
25 | 


--------------------------------------------------------------------------------
/TIPA_library/utils/pca_basic.py:
--------------------------------------------------------------------------------
 1 | import numpy as np
 2 | from sklearn.decomposition import PCA
 3 | 
 4 | 
 5 | ''' This is an example code for PCA projection
 6 | '''
 7 | def pca_basic(t2d_data_sequence):
 8 |     # pca_input: 2184*(240*320)
 9 |     
10 |     t2d_data_height = t2d_data_sequence.shape[0]
11 |     t2d_data_width = t2d_data_sequence.shape[1]
12 |     t2d_data_sequence = t2d_data_sequence.reshape((t2d_data_height*t2d_data_width, np.size(t2d_data_sequence, 2)))
13 |     t2d_data_sequence = np.transpose(t2d_data_sequence)
14 | 
15 |     # Eigenfaces
16 |     output_pca = PCA(n_components=0.95)
17 |     
18 |     # PCA projection
19 |     _ = output_pca.fit(t2d_data_sequence)
20 |     t2d_data_sequence = output_pca.transform(t2d_data_sequence)
21 | 
22 |     # Eigenfaces
23 |     eigen_faces = output_pca.components_
24 |     var_percent = output_pca.explained_variance_ratio_
25 |     
26 |     return eigen_faces, var_percent


--------------------------------------------------------------------------------
/TIPA_library/utils/rvs.py:
--------------------------------------------------------------------------------
 1 | ## Contributors
 2 | ## Dr. Youngjun Cho (Assistant Professor, UCL Computer Science), Chang Liu (MSc in Machine Learning, UCL Computer Science)
 3 | ### currently, this codes needs to be debugged.
 4 | 
 5 | 
 6 | import numpy as np
 7 | from TIPA_library.utils import overlap_windows, overlap_matrix, gausswin, compute_frequency_grid
 8 | from scipy import signal
 9 | 
10 | ''' 
11 | Signal processing toolkits for Respiration Variability Spectrogram
12 | 
13 | [Reference] 
14 | Cho, Y., Bianchi-Berthouze, N. and Julier, S.J., 2017. 
15 | DeepBreath: Deep learning of breathing patterns for automatic stress recognition using low-cost thermal imaging in unconstrained settings. In 2017 Seventh International Conference on Affective Computing and Intelligent Interaction (ACII) (pp. 456-463). IEEE. https://doi.org/10.1109/ACII.2017.8273639
16 | '''
17 | # This code does have an issue with the limited resolution of outputs from scipy.signal.periodogram. (needs to improve)
18 | 
19 | def rvs(Fss,x0):#t0:
20 |     Tmax = 20
21 |     step_length = Fss*1
22 |     lag = Tmax*Fss
23 |     window_list = overlap_windows.overlap_windows(x0,0.05,lag)
24 |     #time_list = overlap_windows(t0,0.05,lag)
25 |     s_PSD_1 = []
26 |     for i in range(len(window_list)):
27 |         data_use = window_list[i]
28 |         data_info = data_use.copy()
29 |         #time_info = time_list[i]
30 |         max_data = max(data_info)
31 |         min_data = min(data_info)
32 |         for j in range(len(data_info)):
33 |             data_info[j] = (data_info[j]-min_data)/(max_data-min_data)
34 |         
35 |         filterN = 3
36 |         Wn1=0.1
37 |         Wn2=0.85
38 |         Fn=Fss/2
39 |         filter_b, filter_a = scipy.signal.ellip(filterN,1,2,[Wn1/Fn,Wn2/Fn],btype='bandpass')
40 |         filtered_featurescaled_data = scipy.signal.lfilter(filter_b,filter_a,data_info,axis=0)
41 |         w = gausswin.gausswin(5120)
42 |         gaussian_final_window = np.array(filtered_featurescaled_data*(w.T),dtype='float64')
43 | #######
44 |         freq_index,freq_amplitude = scipy.signal.periodogram(gaussian_final_window,Fss)
45 | #         freqs = compute_frequency_grid(oversampling=10)
46 | #         ang_freqs = 2 * np.pi * freqs
47 | #         #t = list(t)
48 | #         freq_amplitude = scipy.signal.lombscargle(time_info,gaussian_final_window,ang_freqs)
49 |         
50 |         #STACK PSD IN SLIDING WNNDOW
51 |         s_PSD_1.append(freq_amplitude)
52 |     s_PSD_1 = np.array(s_PSD_1).T
53 | #     ss_PSD_1=s_PSD_1[1:19,:]
54 |     #return ss_PSD_1
55 | #     seperate_data = overlap_matrix.overlap_matrix(ss_PSD_1,1/120,120)
56 |     return s_PSD_1


--------------------------------------------------------------------------------
/TIPA_library/utils/simpleDRQ.py:
--------------------------------------------------------------------------------
 1 | # TIPA (Thermal Imaging-based Physiological and Affective computing) open-source project
 2 | 
 3 | ## Author(s): Dr. Youngjun Cho*(Assistant Professor, UCL Computer Science)
 4 | ## * http://youngjuncho.com
 5 | 
 6 | import numpy as np
 7 | import cv2
 8 | 
 9 | ''' 
10 | Simple Dynamic-Range Quantisation.
11 | 
12 | [Reference] 
13 | Cho, Y., Bianchi-Berthouze, N., Marquardt, N. and Julier, S.J., 2018. Deep Thermal Imaging: Proximate Material Type Recognition in the Wild through Deep Learning of Spatial Surface Temperature Patterns. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, ACM. https://doi.org/10.1145/3173574.3173576
14 | '''
15 | def simpleDRQ(t2d_data, desired_height, desired_width):
16 |     
17 |     (height, width) = t2d_data.shape
18 | 
19 |     x_start = int((height - width * 0.925) / 2 + 1)
20 |     x_end = int((height + width * 0.925) / 2)
21 |     y_start = int((width - width * 0.925) / 2 + 1)
22 |     y_end = int((width + width * 0.925) / 2)
23 |     t2d_data = t2d_data[x_start:x_end, y_start:y_end]
24 |     
25 |     [adjusted_h, adjusted_w] = t2d_data.shape
26 | 
27 |     data = np.zeros((leng, 60, 60))
28 | 
29 | 
30 |     temp_A = t2d_data
31 |     mmin = temp_A.min()
32 |     mmax = temp_A.max()
33 |     for cho1 in range(0, adjusted_w):
34 |         for cho2 in range(0, adjusted_h):
35 |             temp_A[cho2, cho1] = (temp_A[cho2, cho1] - mmin) / (mmax - mmin)
36 | 
37 |     return cv2.resize(temp_A, (desired_height, desired_width))
38 | 


--------------------------------------------------------------------------------
/TIPA_library/utils/timshow.py:
--------------------------------------------------------------------------------
 1 | # TIPA (Thermal Imaging-based Physiological and Affective computing) open-source project
 2 | 
 3 | ## Author(s): Dr. Youngjun Cho*(Assistant Professor, UCL Computer Science)
 4 | ## * http://youngjuncho.com
 5 | 
 6 | 
 7 | # from ipywidgets import interact, interactive, fixed, interact_manual
 8 | # import ipywidgets as widgets
 9 | import matplotlib.pyplot as plt
10 | 
11 | ## for python jupyter users, type %matplotlib notebook before calling this.
12 | # Thermal image show.
13 | def timshow(matrix):
14 |     fig= plt.figure()
15 |     ax = fig.add_subplot(1,1,1)
16 |     plt.imshow(matrix, cmap=plt.get_cmap('hot'))
17 |     plt.colorbar()
18 | 
19 |     
20 | 


--------------------------------------------------------------------------------
/data/readme.md:
--------------------------------------------------------------------------------
1 | Place your data here.
2 | 


--------------------------------------------------------------------------------
/figures/figure1.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/deepneuroscience/TIPA/c14bbd3355be38b5f8e48f9ba1497ac8aea62032/figures/figure1.png


--------------------------------------------------------------------------------
/figures/figure2.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/deepneuroscience/TIPA/c14bbd3355be38b5f8e48f9ba1497ac8aea62032/figures/figure2.png


--------------------------------------------------------------------------------
/figures/figure3.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/deepneuroscience/TIPA/c14bbd3355be38b5f8e48f9ba1497ac8aea62032/figures/figure3.png


--------------------------------------------------------------------------------
/figures/readme.md:
--------------------------------------------------------------------------------
1 | 
2 | 


--------------------------------------------------------------------------------