├── 1.png ├── 3.png ├── MAIN.png ├── README.md ├── bipartite.png ├── detect.png ├── final.ipynb ├── output_13_0.png ├── output_13_1.png ├── output_13_2.png ├── output_13_3.png ├── output_19_0.png ├── output_19_1.png ├── output_19_2.png └── output_19_3.png /1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/NickNair/Multiple-Object-Tracking-using-Kalman-Filter/8e9ceaacd0188e55280606a30a45c787cca0a046/1.png -------------------------------------------------------------------------------- /3.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/NickNair/Multiple-Object-Tracking-using-Kalman-Filter/8e9ceaacd0188e55280606a30a45c787cca0a046/3.png -------------------------------------------------------------------------------- /MAIN.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/NickNair/Multiple-Object-Tracking-using-Kalman-Filter/8e9ceaacd0188e55280606a30a45c787cca0a046/MAIN.png -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Multi-Object Tracking using Kalman Filters and the Hungarian Algorithm 2 | 3 | 4 | # What is a Kalman Filter ? 5 | 6 | It is an iterative mathematical process that uses a set of equations and consecutive data inputs to quickly
estimate the true value, position ,velocity ,etc of a the object being measured, when the measurements contain
unprecedented or random error , uncertainity or variation 7 | 8 | The general use of Kalman filters are usually for things that require smoothing of noisy measurement data and estimation. 9 | 10 | Global Positioning Systems , Laptop Trackpads , trajectory optimization in robotics applications involving control and motion planning. 11 | 12 | The most famous early use of the Kalman filter was in the Apollo navigation computer 13 | that took Neil Armstrong to the moon, and (most importantly) brought him back. Today, Kalman filters are at work 14 | in every satellite navigation device, every smart phone, and many computer games. 15 | 16 | # Understanding the Kalman Filter ? 17 | 18 | The very first tenet that is considered in the usage of a Kalman filter in a system is that there is error in both the prediction and measurement of a system variable that we are concerned with. 19 | 20 | The Kalman filter will be dealt with in the context of tracking the position of a certain object. 21 | 22 | A 1-D Kalman Filter to track an object moving along the x-axis will be implemented in order to gain an understanding. 23 | 24 | Assume a car moving along a road with it's position being measured and estimated. 25 | 26 | In Kalman filters μ and σ2 representing a Gaussian distibution is taken as the best estimate for representing the probality distribution for the measured and estimated position of the car along the x-axis . 27 | 28 | In dealing with a Gaussian distribution it is clear that a distribution with a lower variance and hence narrower distribution is preferred as it imbues it with more certainty. 29 | 30 | The Kalman filter represents all distributions by Gaussians and has two major iterative steps: Update and Predict. 31 | 32 | ![](1.png) 33 | 34 | # Update 35 | 36 |
37 | 38 | 39 | Simply put the Update step takes in the predicted postion of the car in terms of the mean ,$\nu$ and variance ,$r^2$ of the Gaussian distribution representing it and updates the position based on the measured position represnted by it's mean,$\mu$ and variance,$\sigma^2$ 40 | 41 | The intuition behind the update is that the mean needs to be weighted by the reciprocal of the variance. As a lower variance hints at higher confidence in the position. 42 | 43 | The variance needs to be updated based on the harmonic sum of the two variances as intuitively it makes more sense for the distribution to get peakier as there is more information about the system and hence allowing for a more certain estimate of the position. 44 | 45 | 46 | $$\mu^{'} = \frac{\mu r^2 + \nu\sigma^2}{r^2 + \sigma^2}$$ 47 | $$\sigma^{'} = \frac{1}{\frac{1}{r^2}+\frac{1}{\sigma^2}} $$ 48 | 49 | 50 | 51 | 52 | 53 | ```python 54 | def update(mean1, var1, mean2, var2): 55 | 56 | new_mean= (mean1*var2+mean2*var1)/(var1+var2) 57 | new_var = 1/(1/var1 +1/var2) 58 | return [new_mean,new_var] 59 | 60 | 61 | ``` 62 | 63 | # Predict 64 |
65 | 66 | 67 | The predict step is straightforward. When the car moves it's new mean and variance will be same as the measurement but we also need to add in the the uncertainity of the motion which is ($u$ , $r^2$). 68 | 69 | $$\mu^{'} = \mu+u$$ 70 | $$\sigma^{'} = \sigma^2 + r^2 $$ 71 | 72 | 73 | 74 | 75 | 76 | ```python 77 | def predict(mean1, var1, mean2, var2): 78 | 79 | new_mean= mean1 +mean2 80 | new_var = var1 +var2 81 | return [new_mean,new_var] 82 | 83 | ``` 84 | 85 | # A Simple Kalman Filter 86 | 87 |
88 | 89 | 90 | A simple Kalman filter is just these two steps performed for the set of measurements of $x$ and $\dot{x}. $ 91 | 92 | 93 | 94 | 95 | ```python 96 | measurements =[5, 6, 7, 9, 10] 97 | motion = [1 ,1 , 2, 1, 1] 98 | measurement_sig = 4 99 | motion_sig =2 100 | mu = 0 101 | sig = 1000 102 | 103 | for i in range(5): 104 | mu,sig=update(mu,sig,measurements[i],measurement_sig) 105 | mu,sig=predict(mu,sig,motion[i],motion_sig) 106 | print([mu, sig]) 107 | ``` 108 | 109 | [5.9800796812749, 5.98406374501992] 110 | [6.992019154030327, 4.397446129289705] 111 | [8.996198441360958, 4.094658810112146] 112 | [9.99812144836331, 4.023387967876767] 113 | [10.99906346214631, 4.005829948139216] 114 | 115 | 116 | # A more General Approach 117 | 118 |
119 | 120 | Now that a general intuition for what a Kalman Filter does is established, the 1-D Filter can be revisited with a more general design. 121 | 122 | 123 | ## State Updation 124 |
125 | 126 | 127 | The Kalman filter model assumes that the state of a system at a time $t$ evolved from the previous state at time $t-1$ according to the equation. 128 | 129 | 130 | $$ X_t = A_t X_{t-1} + Bu_t + w_t$$ 131 | 132 | Where ,
133 | * $\bf{X_t}$ : State Vector and in our car example the [ position ; velocity ] vector.
134 | 135 | * $\bf{A_t} $: The State Transition Matrix. It's defines how the current state depends on the previous. In the case of
the car example it is dictated by the kinematics of the system.
136 | 137 | * $\bf{u_t}$ : The Vector holding the control inputs. For the car it could be the braking, steering,etc
138 | 139 | * $\bf{B_t}$ : The Control Input matrix that maps the control input to the coresponding State Vector
represntation. 140 | * $\bf{w_t}$ : Process noise with a covariance Q. 141 | 142 | 143 | 144 | 145 | ## System Measurements 146 | 147 | 148 | $$ Y_t = H_tX_t + \nu_t$$ 149 | 150 | Where ,
151 | * $\bf{Y_t}$ : Vector of measurements
152 | 153 | * $\bf{H_t} $: Matrix that maos $X_t$ to measurement space
154 | 155 | * $\bf{\nu_t}$ : Measurement Noise with a covariance R
156 | 157 | Note : $\textbf{w}_k and \textbf{v}_k$ represent the process noise vector with the covariance Q and the measurement noise vector with the covariance R, respectively. They are assumed statistically independent Gaussian noise with the normal probability distribution. 158 | $$p(w) = N(0,Q)$$ 159 | $$p(\nu)=N(0,R)$$ 160 | 161 | 162 | 163 | # Kalman Filter Equations 164 | 165 | ![](3.png) 166 |
167 | 168 | The equations are divided into:
169 | 170 | * $\bf{Predictor}$ $ \bf{Equations }$ : To find the current state using the previous state.

171 | 172 | * $\bf{Corrector}$ $ \bf{Equations}$ : To improve the curent estimated state by incorporating a new measurement into the model 173 | 174 | 175 | 176 | # Predictor Equations 177 | 178 |
179 | 180 | 181 | The state is predicted as follows using the Dynamic(Kinematic) modelling of the system disregarding the noise 182 | 183 | $$ X_t = A_t X_{t-1} + Bu_t $$ 184 | 185 | Now the error covariance matrix is given by 186 | $$P_k = E[(X^{true}_t -X_t)(X^{true}_t -X_t)^T]$$ 187 | 188 | $$ {P}_k = A {P}_{k-1}A^T+ {Q} $$ 189 | 190 | For an object whose postion on the x-axis varies as: 191 | $$X(t)= 3t - 0.75t^2$$ 192 | 193 | The equation for updating $X_t$ would look like 194 | $$ X_t = X_{t-1} + \dot{X_{t-1}}\Delta t + \frac{1}{2}\dot{\dot{X_t}}{\Delta t}^2$$ 195 | 196 | And $\dot{X_t}$ is 197 | $$\dot{X_t} = \dot{X_{t-1}} + \dot{\dot{X_{t-1}}}\Delta t $$ 198 | 199 | And $\dot{\dot{X_{t-1}}}$ is a constant in this case 200 | 201 | 202 | Thus the state vector , which looks like, 203 | $$\bf{X_t}=\begin{bmatrix} X_t \\ \dot{X_t}\end{bmatrix}= \begin{bmatrix} X_{t-1} + \dot{X_{t-1}}\Delta t + \frac{1}{2}\dot{\dot{X_t}}{\Delta t}^2 \\ \dot{X_{t-1}} + \dot{\dot{X_{t-1}}}\Delta t \end{bmatrix} $$ 204 | 205 | 206 | This can be rewriiten as 207 | 208 | 209 | $$\bf{X_t}=\begin{bmatrix} X_t \\ \dot{X_t}\end{bmatrix}= \begin{bmatrix} 1 & \Delta t \\ 0 & 1 \end{bmatrix} \begin{bmatrix} X_{t-1} \\ \dot{X_{t-1}}\end{bmatrix} + \begin{bmatrix}\frac{1}{2}{\Delta t}^2 \\ \Delta t\end{bmatrix} \dot{\dot{X_{t-1}}}$$ 210 | 211 | So comparing this with the general equation allows us to conclude 212 | 213 | $$A =\begin{bmatrix} 1 & \Delta t \\ 0 & 1 \end{bmatrix} $$ 214 | $$B = \begin{bmatrix}\frac{1}{2}{\Delta t}^2 \\ \Delta t\end{bmatrix}$$ 215 | 216 | Here acceleration can be thought of as the control input and B is the matrix that maps it to position and velocity. 217 | 218 | 219 | 220 | # Corrector Equations 221 | 222 |
223 | 224 | 225 | $$Y_t = H_tX_t + \nu_t$$ 226 | 227 | Here the error in measurement is 228 | 229 | $$\nu_t = Y_t-H_tX_t$$ 230 | 231 | Knowing this the X_t is updated based on the equation 232 | 233 | $$ X_t = X_t + K_t\nu_t$$
234 | $$ X_t = X_t + K_t(Y_t-H_tX_t)$$ 235 | 236 | Here $K_t$ is called the $\bf{Kalman}$ $\bf{ Gain}$.
237 | The Kalman gain is used to update $X_t$ after knowing the measurement $Y_t$. 238 | 239 | It can be thought of as the ratio 240 | $$K = \frac{E1}{E1+E2}$$ 241 | where,
242 | 243 | * $\bf{E1}$: Error in Estimate from Predictor Equations 244 | 245 | * $\bf{E2}$: Error in Measurement of the the state vector 246 | 247 | If E2 is low then the Kalman Gain is closer to 1 thereby implying the measurement ought to be given more weight. And if E2 is high then the Estimate ought to be trusted more. 248 | 249 | In the general representation $K_t$ can be calculated as 250 | 251 | $${K}_t = \frac{{P}_tH^T}{H{P}_tH^T+ {R} }$$ 252 | 253 | Where $R$ is the measurement covariance matrix. 254 | 255 | The error covariance gets updated with the following equation that's derived based on least mean square approach 256 | 257 | $$ P_t = (I - K_tH_t)P_t $$ 258 | 259 | 260 | In the case of our car we measure only the position so the measurement $Y_t$ is just 261 | 262 | $$Y_t = X_t + \nu_t$$ 263 | 264 | $$Y_t = \begin{bmatrix} 1 & 0 \end{bmatrix}\begin{bmatrix} X_t \\ \dot{X_t} \end{bmatrix} + \nu_t$$ 265 | 266 | $$Y_t = \begin{bmatrix} 1 & 0 \end{bmatrix}\bf{X_t} + \nu_t$$ 267 | 268 | Thus $H$ is just 269 | 270 | $$H = \begin{bmatrix} 1 & 0 \end{bmatrix}$$ 271 | 272 | 273 | 274 | ## Process noise covariance Q and Measurement noise covariance R 275 | 276 |
277 | 278 | 279 | If $\sigma_x$ and $\sigma_{\dot{x}}$ are the standard deviations of the position and the velocity then the Process Noise Covariance Q is 280 | 281 | $$Q=\begin{bmatrix} {\sigma_x}^2 & \sigma_x\sigma_{\dot{x}}\\ \sigma_x\sigma_{\dot{x}} & \sigma_{\dot{x}}^2\end{bmatrix}$$ 282 | 283 | If $\sigma_{\dot{\dot{x}}}$ is the standard deviation of acceleration we can write 284 | $$ \sigma_x = \frac{\Delta t^2}{2} \sigma_{\dot{\dot{x}}}$$ 285 | $$ \sigma_{\dot{x}} = \Delta t \sigma_{\dot{\dot{x}}}$$ 286 | 287 | Thus 288 | $$ Q=\begin{bmatrix} \frac{\Delta t^4}{4} & \frac{\Delta t^3}{2}\\ \frac{\Delta t^3}{2} & \Delta t^2\end{bmatrix} \sigma_{\dot{\dot{x}}}^2 289 | $$ 290 | 291 | And R is just a scalar as we measure only one variable ,that is position. 292 | 293 | 294 | $$R = \sigma_m^2$$ 295 | 296 | Q and R are essential for correct updation of the error covariance matrix $P_t$ 297 | 298 | 299 | 300 | 301 | 302 | ```python 303 | import numpy as np 304 | import matplotlib.pyplot as plt 305 | 306 | class KalmanFilterClass(): 307 | def __init__(self, dt, acceleration, sd_acceleration , sd_measurement): 308 | 309 | self.dt = dt 310 | 311 | # The acceleration which is essentially u from the state update equation 312 | self.a = acceleration 313 | 314 | # The standard deviation of the acceleration variable in order get t 315 | self.sd_a = sd_acceleration 316 | 317 | # The state transition matrix 318 | self.A = np.matrix([ [1 , self.dt], 319 | [0 , 1 ]] ) 320 | 321 | 322 | # The control input transition matrix 323 | self.B = np.matrix([ [(self.dt**2)/2], 324 | [ self.dt ]]) 325 | 326 | # The matrix that maps x to measurement 327 | self.H = np.matrix( [[1,0]] ) 328 | 329 | 330 | 331 | # Processs Covariance that for our case depends solely on the acceleration 332 | self.Q = np.matrix([[(self.dt**4)/4, (self.dt**3)/2], 333 | [(self.dt**3)/2, self.dt**2]]) * self.sd_a**2 334 | 335 | 336 | # Measurement Covariance. It's a scalar in this case because only position is measured 337 | self.R = sd_measurement**2 338 | 339 | # The error covariance matrix that is Identity for now. It's get updated based on Q, A and R. 340 | self.P = np.eye(self.A.shape[1]) 341 | 342 | 343 | # Finally the vector in consideration it's [ position ; velocity ] 344 | self.x = np.matrix([[0],[0]]) 345 | 346 | def predict(self): 347 | 348 | # The state update : X_t = A*X_t-1 + B*u 349 | # here u is acceleration,a 350 | self.x = np.dot(self.A, self.x) + np.dot(self.B, self.a) 351 | 352 | # Updation of the error covariance matrix 353 | self.P = np.dot(np.dot(self.A, self.P), self.A.T) + self.Q 354 | 355 | return self.x 356 | 357 | def update(self, z): 358 | 359 | S = np.dot(self.H, np.dot(self.P, self.H.T)) + self.R 360 | 361 | K = np.dot(np.dot(self.P, self.H.T), np.linalg.inv(S)) 362 | 363 | self.x = np.round(self.x + np.dot(K, (z - np.dot(self.H, self.x)))) 364 | 365 | I = np.eye(self.H.shape[1]) 366 | 367 | self.P = (I - (K * self.H)) * self.P 368 | ``` 369 | 370 | 371 | ```python 372 | def main(): 373 | dt = 0.10 374 | 375 | t =np.arange(0, 100, dt) 376 | 377 | # create Waypoints 378 | waypoints = 0.01*(3*t - 4*t**2) 379 | 380 | # acceleration is set to 0.08 m/s^2 381 | a= 0.08 382 | 383 | # we assume that the standard deviation of the acceleration is 1.9 (m/s^2) 384 | sd_a = 0.1 385 | 386 | # and standard deviation of the measurement is 1.2 (m) 387 | sd_m = 1.2 388 | 389 | # create KalmanFilter object 390 | 391 | kf = KalmanFilterClass(dt, a, sd_a, sd_m) 392 | 393 | prediction_points = [] 394 | 395 | measurement_points = [] 396 | 397 | for x in waypoints: 398 | 399 | # Mesurement is given as waypoint + normal error in the range(0,50) 400 | z = kf.H * x + np.random.normal(0, 25) 401 | 402 | # Append measurements 403 | measurement_points.append(z.item(0)) 404 | 405 | # Call predict() to predict the next state and append the output 406 | prediction_points.append(kf.predict()[0]) 407 | 408 | # Update to take in the measurement to update the parameters 409 | kf.update(z.item(0)) 410 | 411 | fig = plt.figure() 412 | fig.suptitle(' Waypoints ', fontsize=20) 413 | 414 | plt.plot(t, np.array(waypoints), label='waypoints') 415 | plt.xlabel('T', fontsize=20) 416 | plt.ylabel('X', fontsize=20) 417 | plt.show() 418 | fig = plt.figure() 419 | fig.suptitle('Measurements ', fontsize=20) 420 | plt.plot(t, measurement_points, label='Measurements') 421 | plt.xlabel('T', fontsize=20) 422 | plt.ylabel('X', fontsize=20) 423 | plt.show() 424 | 425 | fig = plt.figure() 426 | fig.suptitle('Kalman Filter Prediction ', fontsize=20) 427 | plt.plot(t, np.squeeze(prediction_points), label='Kalman Filter ') 428 | plt.xlabel('T', fontsize=20) 429 | plt.ylabel('X', fontsize=20) 430 | plt.show() 431 | 432 | fig = plt.figure() 433 | fig.suptitle('', fontsize=20) 434 | plt.plot(t, np.array(waypoints), label='waypoints') 435 | plt.plot(t, measurement_points, label='Measurements') 436 | plt.plot(t, np.squeeze(prediction_points), label='Kalman Filter ') 437 | plt.xlabel('T', fontsize=20) 438 | plt.ylabel('X', fontsize=20) 439 | plt.legend() 440 | plt.show() 441 | 442 | 443 | if __name__ == "__main__": 444 | main() 445 | 446 | 447 | 448 | ``` 449 | 450 | 451 | 452 | ![png](output_13_0.png) 453 | 454 | 455 | 456 | 457 | 458 | ![png](output_13_1.png) 459 | 460 | 461 | 462 | 463 | 464 | ![png](output_13_2.png) 465 | 466 | 467 | 468 | 469 | 470 | ![png](output_13_3.png) 471 | 472 | 473 | 474 | # Kalman Filter Equations for 2D tracking 475 | 476 |
477 | 478 | The basic equations should remain the same as the 1D case with additional considerations for the other variable. 479 | Considering $X$ and $Y$ to be the concerned variables denoting the position the state vector should look like 480 | 481 | $$\bf{X_t}=\begin{bmatrix} X_t \\ Y_t \\\dot{X_t}\\ \dot{Y_t}\end{bmatrix}$$ 482 | 483 | And by extension the can be related to the previous state vector as follows 484 | 485 | $$\bf{X_t}=\begin{bmatrix} X_t \\ Y_t \\\dot{X_t}\\ \dot{Y_t}\end{bmatrix}= \begin{bmatrix} X_{t-1} + \dot{X_{t-1}}\Delta t + \frac{1}{2}\dot{\dot{X_t}}{\Delta t}^2 \\ Y_{t-1} + \dot{Y_{t-1}}\Delta t + \frac{1}{2}\dot{\dot{Y_t}}{\Delta t}^2 \\ \dot{X_{t-1}} + \dot{\dot{X_{t-1}}}\Delta t \\ \dot{Y_{t-1}} + \dot{\dot{Y_{t-1}}}\Delta t \end{bmatrix} $$ 486 | 487 | This can be rewritten as 488 | 489 | $$\bf{X_t}=\begin{bmatrix} X_t \\ Y_t \\\dot{X_t}\\ \dot{Y_t}\end{bmatrix}= \begin{bmatrix} 1 & 0 &\Delta t & 0\\ 0 & 1 & 0 &\Delta t \\ 0 &0 &1 &0 \\ 0 & 0 & 0 & 1\end{bmatrix} \begin{bmatrix} X_{t-1} \\ Y_{t-1}\\\dot{X_{t-1}}\\\dot{Y_{t-1}}\end{bmatrix} + \begin{bmatrix}\frac{1}{2}{\Delta t}^2 & 0 \\0 & \frac{1}{2}{\Delta t}^2 \\ \Delta t & 0 \\0 & \Delta t \end{bmatrix} \begin{bmatrix}\dot{\dot{X_{t-1}}}\\\dot{\dot{Y_{t-1}}}\end{bmatrix} $$ 490 | 491 | Which gives 492 | 493 | $$ A =\begin{bmatrix} 1 & 0 &\Delta t & 0\\ 0 & 1 & 0 &\Delta t \\ 0 &0 &1 &0 \\ 0 & 0 & 0 & 1\end{bmatrix} ;B=\begin{bmatrix}\frac{1}{2}{\Delta t}^2 & 0 \\0 & \frac{1}{2}{\Delta t}^2 \\ \Delta t & 0 \\0 & \Delta t \end{bmatrix} $$ 494 | 495 | Since We're considering the case where velocity isn't being measured the $H$ matrix will look like 496 | 497 | $$H = \begin{bmatrix} 1 & 0 & 0 & 0 \\ 0 & 1 &0& 0 \end{bmatrix} $$ 498 | 499 | 500 | Also Process noise covariance Q and Measurement noise covariance R will look like 501 | 502 | $$Q=\begin{bmatrix} {\sigma_x}^2 & 0 &\sigma_x\sigma_{\dot{x}}& 0\\0& {\sigma_y}^2 & 0 &\sigma_y\sigma_{\dot{y}}\\\sigma_x\sigma_{\dot{x}} & 0 &\sigma_{\dot{x}}^2 & 0\\ 0&\sigma_y\sigma_{\dot{y}} & 0 &\sigma_{\dot{y}}^2 \end{bmatrix}$$ 503 | 504 | This is merely an extension of the 1D Q matrix where 0 is added in places where an x variable and y variable intersect as they are independednt of each other 505 | 506 | Now extending the same logic as last time 507 | 508 | $$ \sigma_x = \frac{\Delta t^2}{2} \sigma_{\dot{\dot{x}}}$$ 509 | $$ \sigma_{\dot{x}} = \Delta t \sigma_{\dot{\dot{x}}}$$ 510 | 511 | Thus 512 | $$ Q=\begin{bmatrix} \frac{\Delta t^4}{4} & 0 &\frac{\Delta t^3}{2}& 0\\ 0 & \frac{\Delta t^4}{4} & 0 &\frac{\Delta t^3}{2}\\\frac{\Delta t^3}{2} &0 &\Delta t^2 & 0 \\ 0& \frac{\Delta t^3}{2} &0 &\Delta t^2 \end{bmatrix} \sigma_{\dot{\dot{x}}}^2 513 | $$ 514 | Here the same process noise is considered for x and y. 515 | 516 | The measurement noise R extends itself to 2D 517 | 518 | $$R = \begin{bmatrix}\sigma_x^2 & 0 \\ 0 &\sigma_y^2\end{bmatrix}$$ 519 | 520 | Q and R are essential for correct updation of the error covariance matrix $P_t$ 521 | 522 | 523 | 524 | 525 | ```python 526 | import numpy as np 527 | import matplotlib.pyplot as plt 528 | 529 | class KalmanFilterClass2D(object): 530 | 531 | def __init__(self, dt, a_x, a_y, sd_acceleration, x_sd, y_sd): 532 | 533 | self.dt = dt 534 | 535 | # the acceleration which is essentially u from the state update equation 536 | self.a = np.matrix([[a_x],[a_y]]) 537 | 538 | 539 | 540 | # The state transition matrix 541 | self.A = np.matrix([[1, 0, self.dt, 0],[0, 1, 0, self.dt],[0, 0, 1, 0],[0, 0, 0, 1]]) 542 | 543 | # The control input transition matrix 544 | self.B = np.matrix([[(self.dt**2)/2, 0],[0,(self.dt**2)/2],[self.dt,0],[0,self.dt]]) 545 | 546 | # The matrix that maps state vector to measurement 547 | self.H = np.matrix([[1, 0, 0, 0],[0, 1, 0, 0]]) 548 | 549 | # Processs Covariance that for our case depends solely on the acceleration 550 | self.Q = np.matrix([[(self.dt**4)/4, 0, (self.dt**3)/2, 0],[0, (self.dt**4)/4, 0, (self.dt**3)/2], 551 | [(self.dt**3)/2, 0, self.dt**2, 0],[0, (self.dt**3)/2, 0, self.dt**2]]) * sd_acceleration**2 552 | 553 | # Measurement Covariance 554 | self.R = np.matrix([[x_sd**2,0], 555 | [0, y_sd**2]]) 556 | 557 | # The error covariance matrix that is Identity for now. It gets updated based on Q, A and R. 558 | self.P = np.eye(self.A.shape[1]) 559 | 560 | # Finally the vector in consideration ; it's [ x position ; y position ; x velocity ; y velocity ; ] 561 | self.x = np.matrix([[0], [0], [0], [0]]) 562 | 563 | def predict(self): 564 | 565 | # The state update : X_t = A*X_t-1 + B*u 566 | # here u is acceleration,a 567 | self.x = np.dot(self.A, self.x) + np.dot(self.B, self.a) 568 | 569 | # Updation of the error covariance matrix 570 | self.P = np.dot(np.dot(self.A, self.P), self.A.T) + self.Q 571 | return self.x[0:2] 572 | 573 | def update(self, z): 574 | 575 | 576 | S = np.dot(self.H, np.dot(self.P, self.H.T)) + self.R 577 | 578 | 579 | K = np.dot(np.dot(self.P, self.H.T), np.linalg.inv(S)) 580 | 581 | self.x = np.round(self.x + np.dot(K, (z - np.dot(self.H, self.x)))) 582 | 583 | I = np.eye(self.H.shape[1]) 584 | 585 | self.P = (I -(K*self.H))*self.P 586 | 587 | return self.x[0:2] 588 | ``` 589 | 590 | # Detection Pipeline 591 | ![](detect.png) 592 |
593 | 594 | 595 | 596 | We use basic blob detection to find the circles in the video on which we will test the 2D filter. 597 | 598 | The exact steps involved in the preprocessing can be understood from the diagram 599 | 600 | 601 | 602 | 603 | ```python 604 | import numpy as np 605 | import cv2 606 | 607 | def calc_centroid(image,thresh): 608 | contours, _ = cv2.findContours(thresh, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE) 609 | centroids = [] 610 | blob_thresh = 4 611 | for i in contours: 612 | try: 613 | (x, y), r = cv2.minEnclosingCircle(i) 614 | centeroid = (int(x), int(y)) 615 | r = int(r) 616 | if (r > blob_thresh): 617 | cv2.circle(image, centeroid, r, (0, 0, 255), 2) 618 | coords = np.array([[x], [y]]) 619 | centroids.append(np.round(coords)) 620 | except ZeroDivisionError: 621 | pass 622 | return centroids 623 | 624 | class Detect(object): 625 | def __init__(self): 626 | self.bgd = cv2.createBackgroundSubtractorMOG2() 627 | 628 | def get_centroid(self, image): 629 | g = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY) 630 | 631 | try: 632 | cv2.imshow('Gray Scaled', g) 633 | except: 634 | print("End") 635 | 636 | f = self.bgd.apply(g) 637 | e = cv2.Canny(f, 50, 190, 3) 638 | _, thresh = cv2.threshold(e, 127, 255, 0) 639 | return calc_centroid(image, thresh) 640 | ``` 641 | 642 | # Testing the 2D Kalman Filter for a single object 643 | 644 |
645 | 646 | 647 | 648 | The Kalman Filter is put to test by running the detection and Kalman Filter code to detect a single object in a video. 649 | 650 | 651 | 652 | 653 | ```python 654 | 655 | def main(): 656 | 657 | 658 | # dt, a_x, a_y, sd_acceleration, x_sd, y_sd 659 | KF = KalmanFilterClass2D(0.1, 1, 1, 1, 0.1,0.1) 660 | 661 | Video = cv2.VideoCapture('42.mp4') 662 | 663 | detection = Detect() 664 | 665 | E=[] 666 | t=[] 667 | G=[[] , [] , [] , [], [], []] 668 | X=[[]] 669 | i=0 670 | 671 | writer=None 672 | 673 | while(True): 674 | 675 | ret, frame = Video.read() 676 | if not ret: 677 | break 678 | 679 | centers = detection.get_centroid(frame) 680 | 681 | (x, y) = KF.predict() 682 | X.append([x,y]) 683 | 684 | if (len(centers) > 0): 685 | 686 | 687 | cv2.circle(frame, (int(centers[0][0]), int(centers[0][1])), 10, (0, 191, 255), 2) 688 | 689 | 690 | x,y=int(x),int(y) 691 | G[4].append(x) 692 | G[5].append(y) 693 | 694 | (x1, y1) = KF.update(centers[0]) 695 | 696 | 697 | x1=int(x1) 698 | y1=int(y1) 699 | G[0].append(x1) 700 | G[1].append(y1) 701 | x,y=int(x),int(y) 702 | 703 | G[2].append(int(centers[0][0])) 704 | G[3].append(int(centers[0][1])) 705 | E.append(((int(centers[0][0])-x1)**2 + (int(centers[0][1])-y1)**2)**0.5) 706 | t.append(i) 707 | cv2.rectangle(frame, (x1 - 14, y1 - 14), (x1 + 14, y1 + 14), (255, 0, 0), 2) 708 | cv2.putText(frame, "Estimated Position", (x1 + 14, y1 + 9), 0, 0.5, (0, 100, 255), 2) 709 | 710 | cv2.putText(frame, "Measured Position", (int(centers[0][0]) + 15, int(centers[0][1]) - 15), 0, 0.5, (0,255,100), 2) 711 | if i>3: 712 | cv2.line(frame, (int(X[i][0]), int(X[i][1])), (int(X[i-3][0]), int(X[i-3][1])),(0,0,255), 2) 713 | i+=1 714 | 715 | cv2.imshow('image', frame) 716 | 717 | if writer is None: 718 | writer = cv2.VideoWriter('./output6.avi', cv2.VideoWriter_fourcc(*"MJPG") , 30, (frame.shape[1], frame.shape[0]), True) 719 | writer.write(frame) 720 | 721 | cv2.waitKey(70) 722 | Video.release() 723 | cv2.destroyAllWindows() 724 | 725 | plt.plot(G[0],G[1],label='Estimated') 726 | plt.plot(G[2],G[3],label='Measurement') 727 | 728 | plt.xlabel('X', fontsize=20) 729 | plt.ylabel('Y', fontsize=20) 730 | plt.legend() 731 | plt.show() 732 | 733 | plt.plot(G[4],G[5],label='Predicted') 734 | plt.plot(G[2],G[3],label='Measurement') 735 | 736 | plt.xlabel('X', fontsize=20) 737 | plt.ylabel('Y', fontsize=20) 738 | plt.legend() 739 | plt.show() 740 | 741 | plt.plot(G[0],G[1],label='Estimated') 742 | plt.plot(G[4],G[5],label='Predicted') 743 | plt.plot(G[2],G[3],label='Measurement') 744 | 745 | plt.xlabel('X', fontsize=20) 746 | plt.ylabel('Y', fontsize=20) 747 | plt.legend() 748 | plt.show() 749 | 750 | plt.plot(t,E) 751 | S= str(sum(E)/len(E)) 752 | plt.title("Average Deviation from Measured "+ S[:3]) 753 | plt.ylabel('Deviation', fontsize=20) 754 | plt.xlabel('time step', fontsize=20) 755 | plt.show() 756 | 757 | if __name__ == "__main__": 758 | main() 759 | ``` 760 | 761 | 762 | 763 | ![png](output_19_0.png) 764 | 765 | 766 | 767 | 768 | 769 | ![png](output_19_1.png) 770 | 771 | 772 | 773 | 774 | 775 | ![png](output_19_2.png) 776 | 777 | 778 | 779 | 780 | 781 | ![png](output_19_3.png) 782 | 783 | 784 | 785 | # Problem with Multiple Object tracking 786 | 787 |
788 | 789 | ## Detection doesn't track the object 790 | 791 | 792 |
793 | 794 | 795 | 796 | Whenver we try to detect multiple objects in a frame the corresponding image processing algorithm treats each frame like a separate problem and outputs the centroids in the order the objects are detected. Since this assumes no previous knowlegde of the object's position , the centroids returned need to be mapped to the correct object. 797 | 798 | 799 | 800 | ## Solution to the problem 801 | 802 | 803 |
804 | 805 | 806 | 807 | To solve this problem we need to assign an ID to each object once it is detected. Having done that the next problem is to correctly assign the IDs to the objects detected in the next frame. To solve that we need to model this as an assignment problem. 808 | 809 | This picture shows a bipartite graphs where vertices represent objects with their (x,y) co-ordinates. The edges represent the possible matchings. The edges have a cost which is the euclidian distance between the two co-ordinates / vertices. 810 | 811 | The objects on the left are from a previous detection and the objects on the right are the current detections. 812 | 813 | 814 | The minimum weight/cost matching will allow us to correctly ID the objects from left to right. 815 | 816 | 817 | 818 | 819 | 820 | ![](bipartite.png) 821 | 822 | 823 | ## Greedy Solution 824 | 825 |
826 | 827 | 828 | 829 | The greedy solution to the problem is to check the min dist cost match among all the possible matchings object and then assign the ID. 830 | Considering there are N object IDs and N detections the number of iterations are 831 | 832 | $$Iterations = N(N-1)(N-2) ....... 1$$ 833 | 834 | Thus the complexity becomes 835 | 836 | $$Complexity = O(N(N-1)(N-2) ....... 1) = O(N!)$$ 837 | 838 | This is highly undesirable as the assignment needs to be done for every frame. Thereby rendering it very computationally intensive. 839 | 840 | 841 | 842 | 843 | 844 | # The Hungarian Algorithm 845 | 846 | 847 |
848 | 849 | The bipartite graph can be represnted as an adjacency matrix as shown 850 | 851 | $$ Adjacency =\begin{bmatrix} dist_{11} & dist_{12} & dist_{13} & .... & dist_{1M} \\ 852 | dist_{21} & dist_{22} & dist_{23} & .... & dist_{2M} 853 | \\. & . & . & ..... & . 854 | \\. & . & . & ..... & . 855 | \\. & . & . & ..... & . 856 | \\ dist_{N1} & dist_{N2} & dist_{N3} & .... & dist_{NM}\end{bmatrix} 857 | $$ 858 | 859 | Now the Hungarian Algorithm helps us arrive at the final optimal assignment using a few steps that involve row and column reduction on the adjacency matrix. 860 | 861 | The Hungarian Algorithm can be captured in the following steps 862 | 863 | 864 | 865 | 866 | 867 | 868 | 869 | 870 | 871 | 872 | 873 | ```python 874 | class Hungarian(object): 875 | 876 | def __init__(self, arr_costs): 877 | self.X = arr_costs.copy() 878 | 879 | n, m = self.X.shape 880 | self.u_row = np.ones(n, dtype=bool) 881 | self.u_column = np.ones(m, dtype=bool) 882 | self.r_0Z = 0 883 | self.c_0Z = 0 884 | self.course = np.zeros((n + m, 2), dtype=int) 885 | self.check = np.zeros((n, m), dtype=int) 886 | 887 | def clear(self): 888 | self.u_row[:] = True 889 | self.u_column[:] = True 890 | ``` 891 | 892 | ### Step 1 893 |
894 | 895 | Find the smallest element in each row and subtract every elemnt in that row with that element. 896 | 897 | 898 | 899 | 900 | ```python 901 | def row_reduction(assignment): 902 | assignment.X -= assignment.X.min(axis=1)[:, np.newaxis] 903 | for i, j in zip(*np.where(assignment.X == 0)): 904 | if assignment.u_column[j] and assignment.u_row[i]: 905 | assignment.check[i, j] = 1 906 | assignment.u_column[j] = False 907 | assignment.u_row[i] = False 908 | 909 | assignment.clear() 910 | return cover_columns 911 | ``` 912 | 913 | ### Step 2 914 | 915 |
916 | 917 | 918 | Draw lines so that all the zeros are covered with minimal number of lines 919 | 920 | 921 | 922 | 923 | 924 | ```python 925 | def cover_columns(assignment): 926 | check = (assignment.check == 1) 927 | assignment.u_column[np.any(check, axis=0)] = False 928 | 929 | if check.sum() < assignment.X.shape[0]: 930 | return cover_zeros 931 | ``` 932 | 933 | ### Step 3 934 | 935 |
936 | 937 | If number of line is N then algorithm is done and all we need to do is generate assignments based on the zero elements. 938 | 939 | 940 | 941 | 942 | 943 | ```python 944 | def cover_zeros(assignment): 945 | X = (assignment.X == 0).astype(int) 946 | covered = X * assignment.u_row[:, np.newaxis] 947 | covered *= np.asarray(assignment.u_column, dtype=int) 948 | n = assignment.X.shape[0] 949 | m = assignment.X.shape[1] 950 | 951 | while True: 952 | row, col = np.unravel_index(np.argmax(covered), (n, m)) 953 | if covered[row, col] == 0: 954 | return generate_zeros 955 | else: 956 | assignment.check[row, col] = 2 957 | star_col = np.argmax(assignment.check[row] == 1) 958 | if assignment.check[row, star_col] != 1: 959 | assignment.r_0Z = row 960 | assignment.c_0Z = col 961 | count = 0 962 | course = assignment.course 963 | course[count, 0] = assignment.r_0Z 964 | course[count, 1] = assignment.c_0Z 965 | 966 | while True: 967 | row = np.argmax(assignment.check[:, course[count, 1]] == 1) 968 | if assignment.check[row, course[count, 1]] != 1: 969 | break 970 | else: 971 | count += 1 972 | course[count, 0] = row 973 | course[count, 1] = course[count - 1, 1] 974 | 975 | col = np.argmax(assignment.check[course[count, 0]] == 2) 976 | if assignment.check[row, col] != 2: 977 | col = -1 978 | count += 1 979 | course[count, 0] = course[count - 1, 0] 980 | course[count, 1] = col 981 | 982 | for i in range(count + 1): 983 | if assignment.check[course[i, 0], course[i, 1]] == 1: 984 | assignment.check[course[i, 0], course[i, 1]] = 0 985 | else: 986 | assignment.check[course[i, 0], course[i, 1]] = 1 987 | 988 | assignment.clear() 989 | assignment.check[assignment.check == 2] = 0 990 | return cover_columns 991 | else: 992 | col = star_col 993 | assignment.u_row[row] = False 994 | assignment.u_column[col] = True 995 | covered[:, col] = X[:, col] * ( 996 | np.asarray(assignment.u_row, dtype=int)) 997 | covered[row] = 0 998 | ``` 999 | 1000 | ### Step 4 1001 | 1002 |
1003 | 1004 | Find the smallest entry not covered by any line. Now subtract this from every uncovered algorithm and add it to the elements that have the horizontal and vertical lines cross each other. Now try from step 3 again. 1005 | 1006 | 1007 | 1008 | 1009 | 1010 | ```python 1011 | def generate_zeros(assignment): 1012 | if np.any(assignment.u_row) and np.any(assignment.u_column): 1013 | minimum_value = np.min(assignment.X[assignment.u_row], axis=0) 1014 | minimum_value = np.min(minimum_value[assignment.u_column]) 1015 | assignment.X[~assignment.u_row] += minimum_value 1016 | assignment.X[:, assignment.u_column] -= minimum_value 1017 | return cover_zeros 1018 | ``` 1019 | 1020 | ## Main function for assignment 1021 | 1022 | 1023 | ```python 1024 | def get_minimum_cost_assignment(arr_costs): 1025 | arr_costs = np.asarray(arr_costs) 1026 | 1027 | 1028 | if arr_costs.shape[1] < arr_costs.shape[0]: 1029 | arr_costs = arr_costs.T 1030 | is_T = True 1031 | else: 1032 | is_T = False 1033 | 1034 | assignment = Hungarian(arr_costs) 1035 | 1036 | 1037 | run = None if 0 in arr_costs.shape else row_reduction 1038 | 1039 | while run is not None: 1040 | run = run(assignment) 1041 | 1042 | if is_T: 1043 | check = assignment.check.T 1044 | else: 1045 | check = assignment.check 1046 | return np.where(check == 1) 1047 | ``` 1048 | 1049 | ## ObjectTracker 1050 | 1051 |
1052 | 1053 | Code that instantiates every detcted object and keeps track of it's ID , position and other relevant information. 1054 | Each object has a Kalman Filter object that helps the Hungarian algorithm assign effectively. 1055 | ' 1056 | 1057 | 1058 | 1059 | ```python 1060 | class Object(object): 1061 | 1062 | def __init__(self, detect , ID): 1063 | 1064 | self.prediction = np.asarray(detect) 1065 | self.object_id = ID 1066 | self.KF = KalmanFilterClass2D(0.1, 1, 1, 1, 0.1,0.1) 1067 | self.skip_count = 0 1068 | self.line = [] 1069 | 1070 | class ObjectTracker(object): 1071 | 1072 | def __init__(self, min_dist , max_skip ,line_length , object_id): 1073 | 1074 | self.min_dist = min_dist 1075 | 1076 | self.max_skip = max_skip 1077 | 1078 | self.line_length = line_length 1079 | 1080 | self.objects = [] 1081 | 1082 | self.object_id = object_id 1083 | 1084 | def Update(self, detections): 1085 | 1086 | if self.objects ==[]: 1087 | 1088 | for i in range(len(detections)): 1089 | 1090 | self.objects.append( Object(detections[i], self.object_id) ) 1091 | 1092 | self.object_id += 1 1093 | 1094 | N , M = len(self.objects), len(detections) 1095 | 1096 | cost_matrix = np.zeros(shape=(N, M)) 1097 | 1098 | for i in range(N): 1099 | 1100 | for j in range(M): 1101 | 1102 | diff = self.objects[i].prediction - detections[j] 1103 | 1104 | cost_matrix[i][j] = np.sqrt(diff[0][0]*diff[0][0] +diff[1][0]*diff[1][0]) 1105 | 1106 | cost_matrix = (0.5) * cost_matrix 1107 | 1108 | assign = [] 1109 | for _ in range(N): 1110 | assign.append(-1) 1111 | 1112 | rows, cols = get_minimum_cost_assignment(cost_matrix) 1113 | 1114 | for i in range(len(rows)): 1115 | assign[rows[i]] = cols[i] 1116 | 1117 | unassign = [] 1118 | 1119 | for i in range(len(assign)): 1120 | 1121 | if (assign[i] != -1): 1122 | 1123 | if (cost_matrix[i][assign[i]] > self.min_dist): 1124 | 1125 | assign[i] = -1 1126 | unassign.append(i) 1127 | else: 1128 | self.objects[i].skip_count += 1 1129 | 1130 | del_objects = [] 1131 | for i in range(len(self.objects)): 1132 | if (self.objects[i].skip_count > self.max_skip): 1133 | del_objects.append(i) 1134 | if len(del_objects) > 0: 1135 | for id in del_objects: 1136 | if id < len(self.objects): 1137 | del self.objects[id] 1138 | del assign[id] 1139 | 1140 | for i in range(len(detections)): 1141 | if i not in assign: 1142 | self.objects.append( Object( detections[i], self.object_id ) ) 1143 | self.object_id += 1 1144 | 1145 | 1146 | 1147 | for i in range(len(assign)): 1148 | self.objects[i].KF.predict() 1149 | 1150 | if(assign[i] != -1): 1151 | self.objects[i].skip_count = 0 1152 | self.objects[i].prediction = self.objects[i].KF.update( detections[assign[i]]) 1153 | else: 1154 | self.objects[i].prediction = self.objects[i].KF.update( np.array([[0], [0]])) 1155 | 1156 | if(len(self.objects[i].line) > self.line_length): 1157 | for j in range( len(self.objects[i].line) - self.line_length): 1158 | del self.objects[i].line[j] 1159 | 1160 | self.objects[i].line.append(self.objects[i].prediction) 1161 | self.objects[i].KF.lastResult = self.objects[i].prediction 1162 | 1163 | ``` 1164 | 1165 | ## Main Code 1166 | 1167 |
1168 | 1169 | The Detector and ObjectTracker are called here to detect objects in the video input. The code can be understood using this diagram. 1170 | 1171 | 1172 | ![](MAIN.png) 1173 | 1174 | 1175 | ```python 1176 | import copy 1177 | 1178 | def draw_line(tracker,image): 1179 | for i in range(len(tracker.objects)): 1180 | if (len(tracker.objects[i].line) > 1): 1181 | for j in range(len(tracker.objects[i].line)-1): 1182 | x1 = tracker.objects[i].line[j][0][0] 1183 | y1 = tracker.objects[i].line[j][1][0] 1184 | x2 = tracker.objects[i].line[j+1][0][0] 1185 | y2 = tracker.objects[i].line[j+1][1][0] 1186 | clr = tracker.objects[i].object_id 1187 | font = cv2.FONT_HERSHEY_SIMPLEX 1188 | cv2.line(image, (int(x1), int(y1)), (int(x2), int(y2)),(0,0,0), 2) 1189 | cv2.putText(image,str(clr), (int(x1)-10,int(y1)-20),0, 0.5, (0,0,0),2) 1190 | 1191 | def save_video(writer,image): 1192 | if writer is None: 1193 | fourcc = cv2.VideoWriter_fourcc(*"MJPG") 1194 | writer = cv2.VideoWriter('./output5.avi', fourcc, 30, (image.shape[1], image.shape[0]), True) 1195 | writer.write(image) 1196 | 1197 | if __name__ == "__main__": 1198 | 1199 | video = cv2.VideoCapture('11.avi') 1200 | 1201 | detect_object = Detect() 1202 | 1203 | tracker = ObjectTracker(160, 8, 3, 1) 1204 | 1205 | 1206 | writer=None 1207 | 1208 | while(True): 1209 | ret, image = video.read() 1210 | 1211 | original_video = copy.copy(image) 1212 | 1213 | if not ret: 1214 | break 1215 | 1216 | centroids = detect_object.get_centroid(image) 1217 | 1218 | if (len(centroids) > 0): 1219 | 1220 | tracker.Update(centroids) 1221 | 1222 | draw_line(tracker,image) 1223 | 1224 | cv2.imshow('Tracking', image) 1225 | 1226 | save_video(writer,image) 1227 | 1228 | 1229 | cv2.imshow('Original', original_video) 1230 | 1231 | cv2.waitKey(50) 1232 | 1233 | video.release() 1234 | cv2.destroyAllWindows() 1235 | ``` 1236 | 1237 | ## Testing the Kalman Filter and the Hungarian algorithm on a real Video 1238 | 1239 |
1240 | 1241 | We tested the above code on a real video. Since our detection code won't work on a real video we decided to use pre-trained weights and implemented a detection based on a DNN using opencv. 1242 | 1243 | 1244 | 1245 | ```python 1246 | def draw_boxes(img, boxes, confidences, classids, idxs, labels): 1247 | if len(idxs) > 0: 1248 | for i in idxs.flatten(): 1249 | x, y = boxes[i][0], boxes[i][1] 1250 | w, h = boxes[i][2], boxes[i][3] 1251 | 1252 | color = (255,0,0) 1253 | 1254 | 1255 | cv2.rectangle(img, (x, y), (x+w, y+h), color, 2) 1256 | return img 1257 | ``` 1258 | 1259 | 1260 | ```python 1261 | def generate_boxes(outs, height, width, tconf,labels): 1262 | boxes = [] 1263 | confidences = [] 1264 | classids = [] 1265 | center=[] 1266 | 1267 | for out in outs: 1268 | for detection in out: 1269 | scores = detection[5:] 1270 | classid = np.argmax(scores) 1271 | confidence = scores[classid] 1272 | 1273 | if confidence > tconf and labels[classid]=='person': 1274 | 1275 | box = detection[0:4] * np.array([width, height, width, height]) 1276 | centerX, centerY, bwidth, bheight = box.astype('int') 1277 | 1278 | 1279 | x = int(centerX - (bwidth / 2)) 1280 | y = int(centerY - (bheight / 2)) 1281 | 1282 | 1283 | boxes.append([x, y, int(bwidth), int(bheight)]) 1284 | center.append(np.array([[x], [y]])) 1285 | confidences.append(float(confidence)) 1286 | classids.append(classid) 1287 | 1288 | return boxes, confidences, classids,center 1289 | ``` 1290 | 1291 | 1292 | ```python 1293 | def infer_image(net, layer_names, height, width, img, labels, boxes=None, confidences=None, classids=None, idxs=None): 1294 | 1295 | 1296 | 1297 | blob = cv2.dnn.blobFromImage(img, 1 / 255.0, (416, 416), swapRB=True, crop=False) 1298 | 1299 | 1300 | net.setInput(blob) 1301 | outs = net.forward(layer_names) 1302 | 1303 | boxes, confidences, classids,center = generate_boxes(outs, height, width, 0.5,labels) 1304 | 1305 | 1306 | idxs = cv2.dnn.NMSBoxes(boxes, confidences, 0.5, 0.3) 1307 | 1308 | 1309 | 1310 | img = draw_boxes(img, boxes, confidences, classids, idxs, labels) 1311 | 1312 | return center 1313 | ``` 1314 | 1315 | # Multiple Player Tracking on a Tennis video 1316 | 1317 | 1318 | ```python 1319 | import copy 1320 | 1321 | def main(): 1322 | 1323 | cap = cv2.VideoCapture("123.mp4") 1324 | 1325 | tracker = ObjectTracker(160, 8, 3,1) 1326 | 1327 | labels = open('./yolov3-coco/coco-labels').read().strip().split('\n') 1328 | net = cv2.dnn.readNetFromDarknet('./yolov3-coco/yolov3.cfg', './yolov3-coco/yolov3.weights') 1329 | layer_names = net.getLayerNames() 1330 | layer_names = [layer_names[i[0] - 1] for i in net.getUnconnectedOutLayers()] 1331 | 1332 | height, width = None, None 1333 | writer = None 1334 | 1335 | while(True): 1336 | 1337 | ret, frame = cap.read() 1338 | if width is None or height is None: 1339 | height, width = frame.shape[:2] 1340 | 1341 | orig_frame = copy.copy(frame) 1342 | 1343 | if not ret: 1344 | break 1345 | 1346 | centers = infer_image(net, layer_names, height, width, frame, labels) 1347 | 1348 | if (len(centers) > 0): 1349 | 1350 | tracker.Update(centers) 1351 | 1352 | draw_line(tracker,frame) 1353 | 1354 | cv2.imshow('Tracking', frame) 1355 | 1356 | save_video(writer,frame) 1357 | 1358 | cv2.imshow('Original', orig_frame) 1359 | 1360 | cv2.waitKey(50) 1361 | 1362 | 1363 | writer.release() 1364 | cap.release() 1365 | cv2.destroyAllWindows() 1366 | 1367 | 1368 | if __name__ == "__main__": 1369 | 1370 | main() 1371 | ``` 1372 | -------------------------------------------------------------------------------- /bipartite.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/NickNair/Multiple-Object-Tracking-using-Kalman-Filter/8e9ceaacd0188e55280606a30a45c787cca0a046/bipartite.png -------------------------------------------------------------------------------- /detect.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/NickNair/Multiple-Object-Tracking-using-Kalman-Filter/8e9ceaacd0188e55280606a30a45c787cca0a046/detect.png -------------------------------------------------------------------------------- /output_13_0.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/NickNair/Multiple-Object-Tracking-using-Kalman-Filter/8e9ceaacd0188e55280606a30a45c787cca0a046/output_13_0.png -------------------------------------------------------------------------------- /output_13_1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/NickNair/Multiple-Object-Tracking-using-Kalman-Filter/8e9ceaacd0188e55280606a30a45c787cca0a046/output_13_1.png -------------------------------------------------------------------------------- /output_13_2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/NickNair/Multiple-Object-Tracking-using-Kalman-Filter/8e9ceaacd0188e55280606a30a45c787cca0a046/output_13_2.png -------------------------------------------------------------------------------- /output_13_3.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/NickNair/Multiple-Object-Tracking-using-Kalman-Filter/8e9ceaacd0188e55280606a30a45c787cca0a046/output_13_3.png -------------------------------------------------------------------------------- /output_19_0.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/NickNair/Multiple-Object-Tracking-using-Kalman-Filter/8e9ceaacd0188e55280606a30a45c787cca0a046/output_19_0.png -------------------------------------------------------------------------------- /output_19_1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/NickNair/Multiple-Object-Tracking-using-Kalman-Filter/8e9ceaacd0188e55280606a30a45c787cca0a046/output_19_1.png -------------------------------------------------------------------------------- /output_19_2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/NickNair/Multiple-Object-Tracking-using-Kalman-Filter/8e9ceaacd0188e55280606a30a45c787cca0a046/output_19_2.png -------------------------------------------------------------------------------- /output_19_3.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/NickNair/Multiple-Object-Tracking-using-Kalman-Filter/8e9ceaacd0188e55280606a30a45c787cca0a046/output_19_3.png --------------------------------------------------------------------------------