├── README.md
├── media
    ├── data
    └── therm.png
├── mlx90640_driver.cpp
└── thermalcam.py


/README.md:
--------------------------------------------------------------------------------
 1 | Based on code from: https://github.com/pimoroni/mlx90640-library
 2 | 
 3 | An ugly but working hack for the raspberry pi 3 so we can spit thermal data out to a file to be read by other programs.
 4 | 
 5 | A thermal cam can easily be implemented in python and openCV by reading this file.
 6 | 
 7 | ![Screenshot](media/therm.png)
 8 | 
 9 | **Above:** Screenshot of myself having got up from the sofa. Note the thermal signature left behind.
10 | 
11 | **mlx90640_driver.cpp**
12 | 
13 | When compiled, the executable can be called with: sudo ./mlx90640_driver 8
14 | Where 8 is the required FPS (4,8 and 16 work) 8 FPS is optimal
15 | 
16 | This program continually overwrites frame data to /tmp/heatmap.csv, where it can be read by other programs.
17 | 
18 | Note: Modify /etc/fstab to mount /tmp in to RAM, else this program will hammer your SD card!
19 | 
20 | **thermalcam.py**
21 | 
22 | An example program that reads data from /tmp/heatmap.csv and generates an image from it.
23 | 
24 | There are a number of Features and hacks implemented this program:
25 | 
26 | When I physically mounted the cameras, I mounted them side by side. This means the video and thermal images do not overlap correctly. Originally I considered mounting one camera on a kinematic mount, however the simplest solution was to simply re-align them in software.
27 | The original video camera frame dimensions are set to 288 368 (slightly larger than 240 * 320 that we end up with)
28 | The video data is then cropped in opencv like this: frame = [5:325,10:250] where 5 and 10 are the offsets to crop from respectively.
29 | 
30 | Ocassionally data is being written whilst we are trying to read and a read error occurs, in that case the last good frame is displayed, rather than just dropping it which is visually annoying.
31 | 
32 | For an as of yet undetermined reason i2C randomly hangs on the raspberry pi 3, which means the program can no longer retrieve i2c data! Fortunately it turns out that if we merely probe the i2c bus, suddenly everything wakes back up again.The python script checks to see if the current frame of thermal data is different from the last one. If it is not, it just probes the i2c bus at 0x33 (rather than the whole bus which is slow!).
33 | 
34 | Thermal data is cubic interpolated to give an impression of a higher resolution. The sensor is only 32 by 24 and is scaled to 320 by 240.
35 | 
36 | The visible and thermal images are combined to provide a meamingful image. Before this occurs edges in the visible image are enhanced to aid viewing.
37 | 
38 | The temperature displayed is that of the area under the crosshairs.
39 | 
40 | Keys:
41 | 
42 | a & z: Alter nmin (normalization limits) changes color mapping. Lower nmin reduces background thermal noise displayed.
43 | 
44 | s & x: Alters nmax (normalization limits) changes upper limit color mapping.
45 | 
46 | d & c: Alters the alpha ratio between the Thermal and the video image.
47 | 
48 | 
49 | 
50 | 
51 | 
52 | 
53 | 
54 | 


--------------------------------------------------------------------------------
/media/data:
--------------------------------------------------------------------------------
1 | screenshots
2 | 


--------------------------------------------------------------------------------
/media/therm.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/leswright1977/mlx90640_python/c20aa5c16947fa223d640566bbefc3a5c1d7c1fc/media/therm.png


--------------------------------------------------------------------------------
/mlx90640_driver.cpp:
--------------------------------------------------------------------------------
  1 | #include <stdint.h>
  2 | #include <iostream>
  3 | #include <cstring>
  4 | #include <fstream>
  5 | #include <chrono>
  6 | #include <thread>
  7 | #include <math.h>
  8 | #include "headers/MLX90640_API.h"
  9 | 
 10 | 
 11 | #define MLX_I2C_ADDR 0x33
 12 | 
 13 | 
 14 | int main(int argc, char* argv[]){
 15 |     int state = 0;
 16 |     //printf("Starting...\n");
 17 |     static uint16_t eeMLX90640[832];
 18 |     float emissivity = 1;
 19 |     uint16_t frame[834];
 20 |     static float image[768];
 21 |     float eTa;
 22 |     static uint16_t data[768*sizeof(float)];
 23 | 
 24 |     float outputArray = 768;
 25 | 
 26 | 	int FPS = 8;
 27 | 
 28 | 	if(argc > 1){
 29 |     	FPS = strtol(argv[1], nullptr, 0);
 30 | 	}
 31 | 
 32 |     std::cout << "capturing at: " << FPS << " FPS\n"; //print the val
 33 | 
 34 |     std::ofstream outfile;
 35 | 
 36 | 
 37 |     MLX90640_SetDeviceMode(MLX_I2C_ADDR, 0);
 38 |     MLX90640_SetSubPageRepeat(MLX_I2C_ADDR, 0);
 39 |     switch(FPS){
 40 |         case 1:
 41 |             MLX90640_SetRefreshRate(MLX_I2C_ADDR, 0b001);
 42 |             break;
 43 |         case 2:
 44 |             MLX90640_SetRefreshRate(MLX_I2C_ADDR, 0b010);
 45 |             break;
 46 |         case 4:
 47 |             MLX90640_SetRefreshRate(MLX_I2C_ADDR, 0b011);
 48 |             break;
 49 |         case 8:
 50 |             MLX90640_SetRefreshRate(MLX_I2C_ADDR, 0b100);
 51 |             break;
 52 |         case 16:
 53 |             MLX90640_SetRefreshRate(MLX_I2C_ADDR, 0b101);
 54 |             break;
 55 |         case 32:
 56 |             MLX90640_SetRefreshRate(MLX_I2C_ADDR, 0b110);
 57 |             break;
 58 |         case 64:
 59 |             MLX90640_SetRefreshRate(MLX_I2C_ADDR, 0b111);
 60 |             break;
 61 |         default:
 62 |             printf("Unsupported framerate: %d", FPS);
 63 |             return 1;
 64 |     }
 65 |     //MLX90640_SetChessMode(MLX_I2C_ADDR); //interesting...
 66 |     //MLX90640_SetSubPage(MLX_I2C_ADDR, 0);
 67 |     //printf("Configured...\n");
 68 | 
 69 |     paramsMLX90640 mlx90640;
 70 |     MLX90640_DumpEE(MLX_I2C_ADDR, eeMLX90640);
 71 |     MLX90640_ExtractParameters(eeMLX90640, &mlx90640);
 72 | 
 73 |     int refresh = MLX90640_GetRefreshRate(MLX_I2C_ADDR);
 74 |     //printf("EE Dumped...\n");
 75 | 
 76 |     int frames = 30;
 77 |     int subpage;
 78 |     static float mlx90640To[768];
 79 | 
 80 | 	int fcount = 0;
 81 | 	int wcount = 0;
 82 | 	int loopcount = 0;
 83 | 
 84 | 	//outfile.open ("/tmp/heatmap.csv");
 85 | 
 86 |     while (1){
 87 | 		//outfile.clear();
 88 | 		//outfile.seekp(0, std::ofstream::beg);
 89 | 
 90 |         state = !state;
 91 |         printf("State: %d \n", state);
 92 | 
 93 | 		std::cout << "Trying for a frame\n";
 94 |         int t = MLX90640_GetFrameData(MLX_I2C_ADDR, frame);
 95 | 		std::cout << "Frame status: ";
 96 | 		std::cout << t;
 97 | 		std::cout << "\n";
 98 | 
 99 | 		printf("Frame retrieved\n");
100 |         MLX90640_InterpolateOutliers(frame, eeMLX90640);
101 |         eTa = MLX90640_GetTa(frame, &mlx90640);
102 |         subpage = MLX90640_GetSubPageNumber(frame);
103 |         MLX90640_CalculateTo(frame, &mlx90640, emissivity, eTa, mlx90640To);
104 |         //printf("Subpage: %d\n", subpage);
105 |         //MLX90640_SetSubPage(MLX_I2C_ADDR,!subpage);
106 | 
107 | 		outfile.open ("/tmp/heatmap.csv");
108 | 
109 | 		if (outfile.is_open()){
110 | 			int counter = 0;
111 | 		    for(int x = 0; x < 32; x++){
112 | 		        for(int y = 0; y < 24; y++){
113 | 		            //std::cout << image[32 * y + x] << ",";
114 | 		            float val = mlx90640To[32 * (23-y) + x];
115 | 		            if(val > 99.99) val = 99.99;
116 | 					counter++;
117 | 					//std::cout << val; //print the val
118 | 					val = roundf(val * 100) / 100;  //round to 2 dp
119 | 					//val = roundf(val); //just round
120 | 					outfile << val;
121 | 					//std::this_thread::sleep_for(std::chrono::microseconds(50)); //chill...
122 | 					//std::cout << "data write: "; //print the val
123 | 					//std::cout << wcount; //print the val
124 | 					//std::cout << "\n"; //print the val
125 | 					wcount++;
126 | 						    
127 | 					if(counter < 768) outfile << ",";//print the comma, omit the last one
128 | 				    }
129 | 		    }
130 | 		}
131 | 	outfile.close();
132 | 	//std::this_thread::sleep_for(std::chrono::milliseconds(100)); //chill...
133 |     }
134 | 	//outfile.close();
135 |     return 0;
136 | }
137 | 


--------------------------------------------------------------------------------
/thermalcam.py:
--------------------------------------------------------------------------------
  1 | import cv2
  2 | import time
  3 | import numpy
  4 | import random
  5 | import math
  6 | 
  7 | #a hack to wake our bus if it hangs....
  8 | import subprocess
  9 | p = subprocess.run(['i2cdetect', '-y','1', '0x33', '0x33'])
 10 | #######################################
 11 | 
 12 | from picamera.array import PiRGBArray
 13 | from picamera import PiCamera
 14 | import numpy as np
 15 | 
 16 | 
 17 | camera = PiCamera()
 18 | camera.resolution = (288, 368) #start with a slightly larger image so we can crop and align later!
 19 | camera.framerate = 20
 20 | rawCapture = PiRGBArray(camera, size=(288, 368)) 
 21 | 
 22 | # allow the camera to warmup
 23 | time.sleep(0.1)
 24 | 
 25 | nmin = 0
 26 | nmax = 255
 27 | alpha1 = 0.5
 28 | alpha2 = 0.5
 29 | 
 30 | prevData = []
 31 | 
 32 | for frame in camera.capture_continuous(rawCapture, format="rgb", use_video_port=True):
 33 | 	# Capture frame-by-frame
 34 | 	frame = frame.array
 35 | 	frame = cv2.flip( frame, 0 ) #flip if neccesary
 36 | 
 37 | 	#crop and align visible image...
 38 | 	#crop image y start yend, xstart xend
 39 | 	frame = frame[5:325, 10:250]
 40 | 
 41 | 	heatmap = np.zeros((32,24,3), np.uint8) #create the blank image to work from
 42 | 
 43 | 	data = np.fromfile('/tmp/heatmap.csv', dtype=float, count=-1, sep=',') #get the data
 44 | 	if np.array_equal(data,prevData):
 45 | 		print('Data stall...Probing i2c')
 46 | 		p = subprocess.run(['i2cdetect', '-y','1', '0x33', '0x33'])
 47 | 				
 48 | 	prevData = data
 49 | 
 50 | 	index = 0
 51 | 	#add to the image
 52 | 	if len(data) == 768:
 53 | 		for y in range (0,32):
 54 | 			for x in range (0,24):
 55 | 				val = (data[index]*10)-100
 56 | 				if math.isnan(val):
 57 | 					val = 0
 58 | 				if val > 255:
 59 | 					val=255
 60 | 				#print(index)
 61 | 				#print(data)
 62 | 
 63 | 				heatmap[y,x] = (val,val,val)
 64 | 			
 65 | 				if(y == 16) and (x == 12):
 66 | 					temp = data[index]
 67 | 				index+=1
 68 | 		heatmap = cv2.flip(heatmap, -1 ) #flip heatmap to match image
 69 | 		prev_heatmap = heatmap #save the heatmap in case we get a data miss
 70 | 	
 71 | 	else:
 72 | 		print("Data miss...Loading previous thermal image")
 73 | 	try:
 74 | 		heatmap = prev_heatmap
 75 | 	except:
 76 | 		print("Previous heatmap does not exist!")
 77 | 
 78 | 	heatmap = cv2.normalize(heatmap,None,nmin,nmax,cv2.NORM_MINMAX)
 79 | 	heatmap = cv2.applyColorMap(heatmap, cv2.COLORMAP_JET)
 80 | 	heatmap = cv2.resize(heatmap,(240,320),interpolation=cv2.INTER_CUBIC)
 81 | 
 82 | 
 83 | 	# Display the resulting frame
 84 | 	cv2.namedWindow('Thermal',cv2.WINDOW_NORMAL)
 85 | 
 86 | 	#Sharpen the image up so we can see edges under the heatmap
 87 | 	kernel = np.array([[-1,-1,-1], [-1,9,-1], [-1,-1,-1]])
 88 | 	frame = cv2.filter2D(frame, -1, kernel)
 89 | 
 90 | 	frame = cv2.addWeighted(frame,alpha1,heatmap,alpha2,0) #combine the images
 91 | 
 92 | 	cv2.line(frame,(120,150),(120,170),(0,0,0),1) #vline
 93 | 	cv2.line(frame,(110,160),(130,160),(0,0,0),1) #hline
 94 | 
 95 | 	cv2.putText(frame,'Temp: '+str(temp), (10, 10),\
 96 | 	cv2.FONT_HERSHEY_SIMPLEX, 0.3,(0, 255, 255), 1, cv2.LINE_AA)
 97 | 
 98 | 	cv2.imshow('Thermal',frame)
 99 | 	
100 | 	# clear the stream in preparation for the next frame
101 | 	rawCapture.truncate(0)
102 | 
103 | 	res = cv2.waitKey(1)
104 | 	#print(res)
105 | 	
106 | 	if res == 113: #q
107 | 		break
108 | 	if res == 97: #a
109 | 		nmin += 10
110 | 		print(nmin)
111 | 	if res == 122: #z
112 | 		nmin -= 10
113 | 		print(nmin)
114 | 	if res == 115: #s
115 | 		nmax += 10
116 | 		print(nmax)
117 | 	if res == 120: #x
118 | 		nmax -= 10
119 | 		print(nmax)
120 | 	if res == 100: #d
121 | 		alpha1 += 0.1
122 | 		alpha2 -= 0.1
123 | 	if res == 99: #c
124 | 		alpha1 -= 0.1
125 | 		alpha2 += 0.1
126 | 
127 | 
128 | 
129 | cv2.destroyAllWindows()
130 | 
131 | 
132 | 
133 | 
134 | 


--------------------------------------------------------------------------------