├── README.md ├── images ├── .ipynb_checkpoints │ ├── test_horizon-checkpoint.ipynb │ └── two_objectives_horizon_detection_notebook-checkpoint.ipynb ├── v1.jpeg ├── v10.jpeg ├── v11.jpeg ├── v12.jpeg ├── v13.jpeg ├── v14.jpeg ├── v15.jpeg ├── v16.jpeg ├── v17.jpeg ├── v18.jpeg ├── v19.jpeg ├── v2.jpeg ├── v20.jpeg ├── v21.jpeg ├── v22.jpeg ├── v23.jpeg ├── v24.jpeg ├── v25.jpeg ├── v26.jpeg ├── v27.jpeg ├── v28.jpeg ├── v29.jpeg ├── v3.jpeg ├── v30.jpeg ├── v31.jpeg ├── v32.jpeg ├── v33.jpeg ├── v34.jpeg ├── v35.jpeg ├── v36.jpeg ├── v4.jpeg ├── v5.jpeg ├── v6.jpeg ├── v7.jpeg ├── v8.jpeg └── v9.jpeg ├── two_objectives_horizon_detection.py └── two_objectives_horizon_detection_notebook.ipynb /README.md: -------------------------------------------------------------------------------- 1 | # Vision-Based Maritime Horizon Detection with Two-Stage Global and Local Objectives 2 | 3 | ![Sample](http://i.imgur.com/4UlnRxY.jpg) 4 | 5 | The following notebook walks the user through an implementation of a computer vision-based horizon detection that uses a two-stage objective that greatly reduces computational overhead compared to an exhaustive search approach. 6 | 7 | The first objective ("global") attempts to find a narrow range of combinations of "pitch" and "roll" (attitude and angle) corresponding to a halfplane that likely subdivdes the sky from the rest of the image. The second objective ("local") searches exhaustively through these combinations to find the halfplane that maximizes the difference in average intensity of the two halfplanes in the immediate viscinity of the halfplane. 8 | 9 | Compared with an exhaustive search of pitch and roll combinations performed at the outset, this method obtains perfect accuracy on our datset with a full order-of-magnitute less computations. This method benefits from the assumption that a "sky" as represented by image data has higher intensity values than the ground pixels (higher mean), and that the sky has higher consistency of representation (lower variance). A "coefficient of variance" calculation (ratio of mean to variance) performed on the full range of angle/attitude for all images in the set suggests that--at least for our data--the optimization surface is approximately convex, allowing for confident use of subsampling in the global objective. 10 | 11 | Exhaustive search over the second objective for our dataset in the data exploration phase however reveals numerous local maxima- therefore we must employ exhaustive search over the subregion identified in the global objective in runtime. Regardless, we observe no loss of accuracy on our output. 12 | 13 | The method is demonstrated on a selection of maritime images whose time-of-day, glare effects, and ground-truth horizon location within the frame and varied. Our method also allows us to ignore color information that is typically used to booster similar vision-based horizon detection methods. 14 | 15 | ### Software and Library Requirements 16 | * Python 2.7.11 17 | * Jupyter Notebook 4.2.2 18 | * Numpy 1.11.2 19 | * matplotlib 1.5.2 20 | * OpenCV 3.2.0 21 | 22 | ## Goals 23 | This repository demonstrates a novel solution to horizon detection for video streams of maritime images that reduces computational requirements as compared with common exhaustive approaches, such as that outlined by [Ettinger et al](https://www.researchgate.net/profile/Martin_Waszak/publication/2494734_Towards_Flight_Autonomy_Vision-Based_Horizon_Detection_for_Micro_Air_Vehicles/links/5441579b0cf2a76a3cc7de60.pdf). 24 | 25 | ## Key Processes 26 | 1. Solve global objective by optimizing attitute/angle objective surface. 27 | 2. Solve local objective by exhaustive search over range defined in (1). 28 | 29 | ## Illustration and Explanation of Routine 30 | We begin with the assumption that the horizon line lies in the exhaustive search space that contains all pitch and angle combinations. In order to better work with the pitch/angle nomenclature, we reparameterize the cartesian version of a horizon line equation into a set of polar coordinates with the caveat that the 'distance' measure is a percentage of pitch from 0% to 100% that "fits" into the rectangular views of most video streams. 31 | 32 | In the global objective, we perform our global search on a downsampled image, using a coarse resolution that samples both angle and pitch at a factor of 5 (5%, 10%, 15%, ..., 100% pitch, and -90°, -85°, ..., -5°, 0°, 5°, ..., 90° angle) for a total search space size of 20 x 40 = 800. Our approach assumes the "sky" has higher intensity value mean than the "ground", and that the sky has less information, which we represent in the objective as its variance. Thus, the global objective maximizes the difference in the sky and ground mean, while minimizing the sky's variance. 33 | 34 | ![Objective 1](http://i.imgur.com/aXQpS2W.jpg) 35 | 36 | ![Objective 1 Overlay](http://i.imgur.com/ia9RScV.jpg) 37 | 38 | To define a region for the second objective, we simply select the outlying combinations of angle in pitch from the global search. Again, we are able to do this due to the approximetely convex nature of the global optimization surface for our dataset (for other approaches, see "Challenges" below). Note that our subsample method does not ensure consistent search space in the second objective across images. 39 | 40 | In the second objective, we exhaustively search over the subregion identified in the global objective. Our optimization objective in the "local search" is to maximize the squared difference in the average sky and ground intensity values in the region immediately surrounding the candidate horizon line, again adjusted by the sky's variance. 41 | 42 | ![Objective 2 Overlay](http://i.imgur.com/WKVWdO9.jpg) 43 | 44 | Using the global objective in front of the local objective, we were able to reduce our search to 1,700 pitch/angle combinations for this sample, compared to an exhuastive optimization of the local objective at the outset which would have yielded the evaluation of 18,000 combinations. 45 | 46 | ## Code Organization 47 | 48 | File | Purpose 49 | ------------ | ------------- 50 | two_objectives_horizon_detection.ipynb | iPython Notebook for user-friendly implementation and data exploration. 51 | two_objectives_horizon_detection.py | Python file containing main, preprocessing steps and subroutines. 52 | 53 | ## Getting Up and Running 54 | 55 | While in the `two_objectives_horizon_detection` directory, enter the following in the command line: 56 | 57 | > ipython notebook two_objectives_horizon_detection.ipynb 58 | -------------------------------------------------------------------------------- /images/.ipynb_checkpoints/two_objectives_horizon_detection_notebook-checkpoint.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Vision-Based Horizion Detection with Dual Global and Local Objectives\n", 8 | "\n", 9 | "The following notebook walks you through an implementation of a computer vision-based horizon detection that uses a two-stage objective that greatly reduces computational overhead compared to the typical exhaustive search approach.\n", 10 | "\n", 11 | "The first objective (\"global\") attempts to find the range of combinations of \"pitch\" and \"roll\" (attitude and angle) corresponding to a halfplane that likely subdivdes the sky from the rest of the image. The second objective (\"local\") searches exhaustively through these combinations to find the halfplane that maximizes the difference in average intensity of the two halfplanes in the immediate viscinity of the halfplane.\n", 12 | "\n", 13 | "Compared with an exhaustive search of pitch and roll combinations performed at the outset, this method obtains perfect accuracy on our datset with a full order-of-magnitute less computations. This method benefits from the assumption that a \"sky\" as represented by image data has higher intensity values than the ground pixels (higher mean), and that the sky has higher consistency of representation (lower variance). A \"coefficient of variance\" calculation (ratio of mean to variance) performed on the full range of angle/attitude for all images in the set suggests that--at least for our data--the optimization surface is approximately convex, allowing for confident use of subsampling in the global objective.\n", 14 | "\n", 15 | "Exhaustive search over the second objective for our dataset in the data exploration phase however reveals numerous local maxima- therefore we must employ exhaustive search over the subregion identified in the global objective in runtime. Regardless, we observe no loss of accuracy on out output.\n", 16 | "\n", 17 | "The method is demonstrated on a selection of maritime images whose time-of-day, glare effects, and ground-truth horizon location within the frame and varied.\n", 18 | "\n", 19 | "*Main()* and associated functions are located in *two_objectives_horizon_detection.py*." 20 | ] 21 | }, 22 | { 23 | "cell_type": "markdown", 24 | "metadata": {}, 25 | "source": [ 26 | "## Header" 27 | ] 28 | }, 29 | { 30 | "cell_type": "code", 31 | "execution_count": null, 32 | "metadata": { 33 | "collapsed": false 34 | }, 35 | "outputs": [], 36 | "source": [ 37 | "%matplotlib tk\n", 38 | "\n", 39 | "from __future__ import division\n", 40 | "\n", 41 | "import numpy as np\n", 42 | "import cv2\n", 43 | "from matplotlib import pyplot as plt\n", 44 | "from mpl_toolkits.mplot3d import Axes3D\n", 45 | "from matplotlib import path\n", 46 | "import os\n", 47 | "\n", 48 | "import two_objectives_horizon_detection as tohd\n", 49 | "\n", 50 | "current_dir = os.getcwd()\n", 51 | "files = [x for x in os.listdir(current_dir) if x.endswith('.jpeg')]" 52 | ] 53 | }, 54 | { 55 | "cell_type": "markdown", 56 | "metadata": {}, 57 | "source": [ 58 | "## GLOBAL OBJECTIVE" 59 | ] 60 | }, 61 | { 62 | "cell_type": "code", 63 | "execution_count": null, 64 | "metadata": { 65 | "collapsed": false 66 | }, 67 | "outputs": [], 68 | "source": [ 69 | "#GLOBAL OBJECTIVE SETTINGS\n", 70 | "img_file = files[14]\n", 71 | "global_img_reduction = 0.1\n", 72 | "global_angles = (-90,91,5)\n", 73 | "global_distances = (5,100,5) \n", 74 | "global_buffer_size = 3\n", 75 | "\n", 76 | "#GLOBAL OBJECTIVE MAIN ROUTINE\n", 77 | "global_search = tohd.main(img_file, \n", 78 | " img_reduction = global_img_reduction,\n", 79 | " angles = global_angles, \n", 80 | " distances = global_distances, \n", 81 | " buffer_size = global_buffer_size, \n", 82 | " local_objective = 0) #2m5s\n", 83 | "\n", 84 | "#GLOBAL OBJECTIVE OPTIMIZATION SURFACE\n", 85 | "objective_1 = np.max((global_search[:,:,0] - global_search[:,:,1]),0) / (global_search[:,:,2])" 86 | ] 87 | }, 88 | { 89 | "cell_type": "markdown", 90 | "metadata": {}, 91 | "source": [ 92 | "### Global Objective Optimization Surface" 93 | ] 94 | }, 95 | { 96 | "cell_type": "code", 97 | "execution_count": null, 98 | "metadata": { 99 | "collapsed": true 100 | }, 101 | "outputs": [], 102 | "source": [ 103 | "Y = np.arange(0, len(range(*global_distances)), 1)\n", 104 | "X, Y = np.meshgrid(Y, X)\n", 105 | "Z = objective_1\n", 106 | "\n", 107 | "fig = plt.figure()\n", 108 | "ax = fig.add_subplot(111, projection='3d')\n", 109 | "ax.plot_wireframe(X,Y,Z, rstride=1, cstride=1)\n", 110 | "plt.draw()" 111 | ] 112 | }, 113 | { 114 | "cell_type": "markdown", 115 | "metadata": {}, 116 | "source": [ 117 | "## LOCAL OBJECTIVE" 118 | ] 119 | }, 120 | { 121 | "cell_type": "code", 122 | "execution_count": null, 123 | "metadata": { 124 | "collapsed": false 125 | }, 126 | "outputs": [], 127 | "source": [ 128 | "#LOCAL OBJECTIVE SETTINGS\n", 129 | "above_two_sigma = (2* np.nanstd(objective_1)) + np.nanmean(objective_1)\n", 130 | "\n", 131 | "local_angles = global_search[np.where(objective_1 > above_two_sigma)[0],\n", 132 | " np.where(objective_1 > above_two_sigma)[1]][:,6]\n", 133 | "local_distances = global_search[np.where(objective_1 > above_two_sigma)[0],\n", 134 | " np.where(objective_1 > above_two_sigma)[1]][:,7]\n", 135 | "local_angle_range = (int(np.min(local_angles))-2,\n", 136 | " int(np.max(local_angles))+3,1)\n", 137 | "local_distance_range = (int(np.min(local_distances))-2,\n", 138 | " int(np.max(local_distances))+3,1)\n", 139 | "\n", 140 | "local_img_reduction = 0.25\n", 141 | "local_angles = local_angle_range\n", 142 | "local_distances = local_distance_range\n", 143 | "local_buffer_size = 5\n", 144 | "\n", 145 | "#LOCAL OBJECTIVE MAIN ROUTINE\n", 146 | "print(\"Evaluating\",(len(range(*local_angle_range))*len(range(*local_distance_range))),\"candidates...\")\n", 147 | "\n", 148 | "local_search = main(img_file, \n", 149 | " img_reduction = local_img_reduction,\n", 150 | " angles = local_angles, \n", 151 | " distances = local_distances, \n", 152 | " buffer_size = local_buffer_size, \n", 153 | " local_objective = 1) #2m5s\n", 154 | "\n", 155 | "#LOCAL OBJECTIVE OPTIMIZATION SURFACE\n", 156 | "objective_2 = np.abs(local_search[:,:,4] - local_search[:,:,5])**2 / local_search[:,:,2] # objective 1" 157 | ] 158 | }, 159 | { 160 | "cell_type": "markdown", 161 | "metadata": {}, 162 | "source": [ 163 | "## DETECTED HORIZON LINE" 164 | ] 165 | }, 166 | { 167 | "cell_type": "code", 168 | "execution_count": null, 169 | "metadata": { 170 | "collapsed": true 171 | }, 172 | "outputs": [], 173 | "source": [ 174 | "horizon_line = local_search[np.unravel_index(objective_2.argmax(), objective_2.shape)]\n", 175 | "\n", 176 | "print(\"For \",img_file,\", best predicted line is\",horizon_line[6],\"degrees and a distance of\",horizon_line[7])" 177 | ] 178 | }, 179 | { 180 | "cell_type": "markdown", 181 | "metadata": {}, 182 | "source": [ 183 | "### OVERLAY HORIZON LINE ON INPUT IMAGE AND DISPLAY" 184 | ] 185 | }, 186 | { 187 | "cell_type": "code", 188 | "execution_count": null, 189 | "metadata": { 190 | "collapsed": false 191 | }, 192 | "outputs": [], 193 | "source": [ 194 | "img = cv2.imread(img_file,0)\n", 195 | "line_coordinates = get_plane_indicator_coord(img,int(horizon_line[6]),horizon_line[7]/100,0)[2:4]\n", 196 | "cv2.line(img,(line_coordinates[0][0],line_coordinates[0][1]),(line_coordinates[1][0],line_coordinates[1][1]),(0,0,255),2)\n", 197 | "\n", 198 | "plt.imshow(img)\n", 199 | "plt.show()" 200 | ] 201 | } 202 | ], 203 | "metadata": { 204 | "kernelspec": { 205 | "display_name": "Python 2", 206 | "language": "python", 207 | "name": "python2" 208 | }, 209 | "language_info": { 210 | "codemirror_mode": { 211 | "name": "ipython", 212 | "version": 2 213 | }, 214 | "file_extension": ".py", 215 | "mimetype": "text/x-python", 216 | "name": "python", 217 | "nbconvert_exporter": "python", 218 | "pygments_lexer": "ipython2", 219 | "version": "2.7.11" 220 | } 221 | }, 222 | "nbformat": 4, 223 | "nbformat_minor": 1 224 | } 225 | -------------------------------------------------------------------------------- /images/v1.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/citrusvanilla/horizon_detection/be4409ee031f79b5c4661ebff5c1acc8b3171d7a/images/v1.jpeg -------------------------------------------------------------------------------- /images/v10.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/citrusvanilla/horizon_detection/be4409ee031f79b5c4661ebff5c1acc8b3171d7a/images/v10.jpeg -------------------------------------------------------------------------------- /images/v11.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/citrusvanilla/horizon_detection/be4409ee031f79b5c4661ebff5c1acc8b3171d7a/images/v11.jpeg -------------------------------------------------------------------------------- /images/v12.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/citrusvanilla/horizon_detection/be4409ee031f79b5c4661ebff5c1acc8b3171d7a/images/v12.jpeg -------------------------------------------------------------------------------- /images/v13.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/citrusvanilla/horizon_detection/be4409ee031f79b5c4661ebff5c1acc8b3171d7a/images/v13.jpeg -------------------------------------------------------------------------------- /images/v14.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/citrusvanilla/horizon_detection/be4409ee031f79b5c4661ebff5c1acc8b3171d7a/images/v14.jpeg -------------------------------------------------------------------------------- /images/v15.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/citrusvanilla/horizon_detection/be4409ee031f79b5c4661ebff5c1acc8b3171d7a/images/v15.jpeg -------------------------------------------------------------------------------- /images/v16.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/citrusvanilla/horizon_detection/be4409ee031f79b5c4661ebff5c1acc8b3171d7a/images/v16.jpeg -------------------------------------------------------------------------------- /images/v17.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/citrusvanilla/horizon_detection/be4409ee031f79b5c4661ebff5c1acc8b3171d7a/images/v17.jpeg -------------------------------------------------------------------------------- /images/v18.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/citrusvanilla/horizon_detection/be4409ee031f79b5c4661ebff5c1acc8b3171d7a/images/v18.jpeg -------------------------------------------------------------------------------- /images/v19.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/citrusvanilla/horizon_detection/be4409ee031f79b5c4661ebff5c1acc8b3171d7a/images/v19.jpeg -------------------------------------------------------------------------------- /images/v2.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/citrusvanilla/horizon_detection/be4409ee031f79b5c4661ebff5c1acc8b3171d7a/images/v2.jpeg -------------------------------------------------------------------------------- /images/v20.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/citrusvanilla/horizon_detection/be4409ee031f79b5c4661ebff5c1acc8b3171d7a/images/v20.jpeg -------------------------------------------------------------------------------- /images/v21.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/citrusvanilla/horizon_detection/be4409ee031f79b5c4661ebff5c1acc8b3171d7a/images/v21.jpeg -------------------------------------------------------------------------------- /images/v22.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/citrusvanilla/horizon_detection/be4409ee031f79b5c4661ebff5c1acc8b3171d7a/images/v22.jpeg -------------------------------------------------------------------------------- /images/v23.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/citrusvanilla/horizon_detection/be4409ee031f79b5c4661ebff5c1acc8b3171d7a/images/v23.jpeg -------------------------------------------------------------------------------- /images/v24.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/citrusvanilla/horizon_detection/be4409ee031f79b5c4661ebff5c1acc8b3171d7a/images/v24.jpeg -------------------------------------------------------------------------------- /images/v25.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/citrusvanilla/horizon_detection/be4409ee031f79b5c4661ebff5c1acc8b3171d7a/images/v25.jpeg -------------------------------------------------------------------------------- /images/v26.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/citrusvanilla/horizon_detection/be4409ee031f79b5c4661ebff5c1acc8b3171d7a/images/v26.jpeg -------------------------------------------------------------------------------- /images/v27.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/citrusvanilla/horizon_detection/be4409ee031f79b5c4661ebff5c1acc8b3171d7a/images/v27.jpeg -------------------------------------------------------------------------------- /images/v28.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/citrusvanilla/horizon_detection/be4409ee031f79b5c4661ebff5c1acc8b3171d7a/images/v28.jpeg -------------------------------------------------------------------------------- /images/v29.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/citrusvanilla/horizon_detection/be4409ee031f79b5c4661ebff5c1acc8b3171d7a/images/v29.jpeg -------------------------------------------------------------------------------- /images/v3.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/citrusvanilla/horizon_detection/be4409ee031f79b5c4661ebff5c1acc8b3171d7a/images/v3.jpeg -------------------------------------------------------------------------------- /images/v30.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/citrusvanilla/horizon_detection/be4409ee031f79b5c4661ebff5c1acc8b3171d7a/images/v30.jpeg -------------------------------------------------------------------------------- /images/v31.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/citrusvanilla/horizon_detection/be4409ee031f79b5c4661ebff5c1acc8b3171d7a/images/v31.jpeg -------------------------------------------------------------------------------- /images/v32.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/citrusvanilla/horizon_detection/be4409ee031f79b5c4661ebff5c1acc8b3171d7a/images/v32.jpeg -------------------------------------------------------------------------------- /images/v33.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/citrusvanilla/horizon_detection/be4409ee031f79b5c4661ebff5c1acc8b3171d7a/images/v33.jpeg -------------------------------------------------------------------------------- /images/v34.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/citrusvanilla/horizon_detection/be4409ee031f79b5c4661ebff5c1acc8b3171d7a/images/v34.jpeg -------------------------------------------------------------------------------- /images/v35.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/citrusvanilla/horizon_detection/be4409ee031f79b5c4661ebff5c1acc8b3171d7a/images/v35.jpeg -------------------------------------------------------------------------------- /images/v36.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/citrusvanilla/horizon_detection/be4409ee031f79b5c4661ebff5c1acc8b3171d7a/images/v36.jpeg -------------------------------------------------------------------------------- /images/v4.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/citrusvanilla/horizon_detection/be4409ee031f79b5c4661ebff5c1acc8b3171d7a/images/v4.jpeg -------------------------------------------------------------------------------- /images/v5.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/citrusvanilla/horizon_detection/be4409ee031f79b5c4661ebff5c1acc8b3171d7a/images/v5.jpeg -------------------------------------------------------------------------------- /images/v6.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/citrusvanilla/horizon_detection/be4409ee031f79b5c4661ebff5c1acc8b3171d7a/images/v6.jpeg -------------------------------------------------------------------------------- /images/v7.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/citrusvanilla/horizon_detection/be4409ee031f79b5c4661ebff5c1acc8b3171d7a/images/v7.jpeg -------------------------------------------------------------------------------- /images/v8.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/citrusvanilla/horizon_detection/be4409ee031f79b5c4661ebff5c1acc8b3171d7a/images/v8.jpeg -------------------------------------------------------------------------------- /images/v9.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/citrusvanilla/horizon_detection/be4409ee031f79b5c4661ebff5c1acc8b3171d7a/images/v9.jpeg -------------------------------------------------------------------------------- /two_objectives_horizon_detection.py: -------------------------------------------------------------------------------- 1 | from __future__ import division 2 | 3 | import numpy as np 4 | import cv2 5 | import math 6 | 7 | 8 | def get_plane_indicator_coord(img,angle,dist_as_perc,buffer_size): 9 | 10 | heading = angle + 90 11 | heading_from_hor = min(math.radians(180-heading),math.radians(heading)) 12 | 13 | radius = int(math.ceil(math.sqrt((img.shape[1]*0.5)**2 + (img.shape[0]*0.5)**2))) 14 | 15 | x0 = img.shape[1]*0.5 16 | y0 = img.shape[0]*0.5 17 | 18 | x1 = (img.shape[1]*0.5) - radius 19 | x2 = (img.shape[1]*0.5) + radius 20 | 21 | y1 = (img.shape[0]*0.5) 22 | y2 = (img.shape[0]*0.5) 23 | 24 | x1_n = x0 + math.cos(math.radians(angle)) * (x1 - x0) - math.sin(math.radians(angle)) * (y1 - y0) 25 | y1_n = y0 + math.sin(math.radians(angle)) * (x2 - x0) + math.cos(math.radians(angle)) * (y2 - y0) 26 | 27 | x2_n = x0 + math.cos(math.radians(angle)) * (x2 - x0) - math.sin(math.radians(angle)) * (y2 - y0) 28 | y2_n = y0 + math.sin(math.radians(angle)) * (x1 - x0) + math.cos(math.radians(angle)) * (y1 - y0) 29 | 30 | 31 | if heading_from_hor == 0: 32 | avail_dist = img.shape[1] 33 | elif abs(heading_from_hor) < math.atan(img.shape[0]/img.shape[1]): 34 | avail_dist = abs(int(img.shape[1]/math.cos(math.radians(heading)))) 35 | else: #heading_from_hor >= np.arctan(img.shape[0]/img.shape[1]) 36 | avail_dist = abs(int(img.shape[0]/math.sin(math.radians(heading)))) 37 | 38 | origin = avail_dist *0.5 39 | heading_transform = (dist_as_perc * avail_dist) - origin 40 | sky_buffer_transform = heading_transform + buffer_size 41 | sea_buffer_transform = heading_transform - buffer_size 42 | 43 | x_transform = heading_transform * math.cos(math.radians(heading)) 44 | y_transform = heading_transform * math.sin(math.radians(heading)) 45 | 46 | pos_x_transform = sky_buffer_transform * math.cos(math.radians(heading)) 47 | pos_y_transform = sky_buffer_transform * math.sin(math.radians(heading)) 48 | 49 | neg_x_transform = sea_buffer_transform * math.cos(math.radians(heading)) 50 | neg_y_transform = sea_buffer_transform * math.sin(math.radians(heading)) 51 | 52 | return [ (int(x1_n+pos_x_transform),int(y1_n+pos_y_transform)), 53 | (int(x2_n+pos_x_transform),int(y2_n+pos_y_transform)), 54 | (int(x1_n+x_transform),int(y1_n+y_transform)), 55 | (int(x2_n+x_transform),int(y2_n+y_transform)), 56 | (int(x1_n+neg_x_transform),int(y1_n+neg_y_transform)), 57 | (int(x2_n+neg_x_transform),int(y2_n+neg_y_transform))] 58 | 59 | 60 | def get_line(start, end): 61 | """Bresenham's Line Algorithm 62 | Produces a list of tuples from start and end. 63 | From http://www.roguebasin.com/index.php?title=Bresenham%27s_Line_Algorithm#Python. 64 | 65 | >>> points1 = get_line((0, 0), (3, 4)) 66 | >>> points2 = get_line((3, 4), (0, 0)) 67 | >>> assert(set(points1) == set(points2)) 68 | >>> print points1 69 | [(0, 0), (1, 1), (1, 2), (2, 3), (3, 4)] 70 | >>> print points2 71 | [(3, 4), (2, 3), (1, 2), (1, 1), (0, 0)] 72 | """ 73 | # Setup initial conditions 74 | x1, y1 = start 75 | x2, y2 = end 76 | dx = x2 - x1 77 | dy = y2 - y1 78 | 79 | # Determine how steep the line is 80 | is_steep = abs(dy) > abs(dx) 81 | 82 | # Rotate line 83 | if is_steep: 84 | x1, y1 = y1, x1 85 | x2, y2 = y2, x2 86 | 87 | # Swap start and end points if necessary and store swap state 88 | swapped = False 89 | if x1 > x2: 90 | x1, x2 = x2, x1 91 | y1, y2 = y2, y1 92 | swapped = True 93 | 94 | # Recalculate differentials 95 | dx = x2 - x1 96 | dy = y2 - y1 97 | 98 | # Calculate error 99 | error = int(dx / 2.0) 100 | ystep = 1 if y1 < y2 else -1 101 | 102 | # Iterate over bounding box generating points between start and end 103 | y = y1 104 | points = [] 105 | for x in range(x1, x2 + 1): 106 | coord = (y, x) if is_steep else (x, y) 107 | points.append(coord) 108 | error -= abs(dy) 109 | if error < 0: 110 | y += ystep 111 | error += dx 112 | 113 | # Reverse the list if the coordinates were swapped 114 | if swapped: 115 | points.reverse() 116 | 117 | return points 118 | 119 | 120 | def get_local_objective_buffer_means(img,plane_coordinates,angle,buffer_size): 121 | 122 | line_pixels = np.array(get_line(plane_coordinates[0],plane_coordinates[1])) 123 | origin = plane_coordinates[0] 124 | 125 | pos_x_transform = int(buffer_size * math.cos(math.radians(angle+90))) 126 | pos_y_transform = int(buffer_size * math.sin(math.radians(angle+90))) 127 | neg_x_transform = int(buffer_size * math.cos(math.radians(angle-90))) 128 | neg_y_transform = int(buffer_size * math.sin(math.radians(angle-90))) 129 | 130 | pos_buffer_pixels = np.array(get_line((0,0),(pos_x_transform,pos_y_transform))) 131 | neg_buffer_pixels = np.array(get_line((0,0),(neg_x_transform,neg_y_transform))) 132 | 133 | local_pos_mat = 0 134 | local_pos_count = 0 135 | local_neg_mat = 0 136 | local_neg_count = 0 137 | 138 | for pixel in line_pixels: 139 | relevant_pos_pixels = pixel - pos_buffer_pixels #(x,y) 140 | for i in relevant_pos_pixels: 141 | 142 | if 0 < i[0] < img.shape[1] and 0 < i[1] < img.shape[0]: 143 | 144 | local_pos_mat += img[(i[1],i[0])] 145 | local_pos_count += 1 146 | 147 | relevant_neg_pixels = pixel - neg_buffer_pixels 148 | for j in relevant_neg_pixels: 149 | if 0 < j[0] < img.shape[1] and 0 < j[1] < img.shape[0]: 150 | local_neg_mat += img[(j[1],j[0])] 151 | local_neg_count += 1 152 | 153 | return int(local_pos_mat/local_pos_count), int(local_neg_mat/local_neg_count) 154 | 155 | 156 | def main(img_file, img_reduction, angles, distances, buffer_size, local_objective = 0): 157 | ''' 158 | # 0 pos_half_mean --> should have higher values 159 | # 1 neg_half_mean --> should have lower values 160 | # 2 pos_half_var --> should have low values 161 | # 3 neg_half_var --> should have low values 162 | # 4 pos_local_mean --> should have higher values 163 | # 5 pos_local_mean --> should have low values 164 | ''' 165 | 166 | img = cv2.imread(img_file,0) 167 | img = cv2.resize(img, dsize = None, fx = img_reduction, fy = img_reduction) 168 | vals = np.zeros((len(range(*angles)), 169 | len(range(*distances)), 170 | 8)) #rows, columns 171 | i = 0 172 | 173 | for angle in xrange(*angles): 174 | #print(angle) 175 | j = 0 176 | for distance in xrange(*distances): 177 | 178 | points = get_plane_indicator_coord(img, angle, distance/100, buffer_size) 179 | endpoints = points[2:4] 180 | 181 | global_pos_mat = np.empty(0,dtype=np.uint8) 182 | global_neg_mat = np.empty(0,dtype=np.uint8) 183 | 184 | local_pos_points = points[0:4] 185 | local_neg_points = points[2:6] 186 | 187 | for x_coor in xrange(0,img.shape[1]): 188 | 189 | if x_coor < endpoints[0][0]: 190 | if angle < 0: 191 | global_neg_mat = np.append(global_neg_mat,img[:,x_coor]) 192 | else: # angle > 0 193 | global_pos_mat = np.append(global_pos_mat, img[:,x_coor]) 194 | 195 | elif x_coor >= endpoints[0][0] and x_coor < endpoints[1][0]: 196 | y_d = int(endpoints[0][1] - ((x_coor-endpoints[0][0]) * np.tan(math.radians(angle)))) 197 | 198 | if y_d < 0: 199 | global_neg_mat = np.append(global_neg_mat,img[:,x_coor]) 200 | elif y_d > img.shape[0]: 201 | global_pos_mat = np.append(global_pos_mat,img[:,x_coor]) 202 | else: 203 | global_pos_mat = np.append(global_pos_mat, img[0:y_d,x_coor] ) 204 | global_neg_mat = np.append(global_neg_mat, img[y_d:img.shape[0], x_coor]) 205 | 206 | else: #elif x_coor >= endpoints[1][0]: 207 | if angle < 0: 208 | global_pos_mat = np.append(global_pos_mat, img[:,x_coor]) 209 | else: # angle > 0 210 | global_neg_mat = np.append(global_neg_mat,img[:,x_coor]) 211 | 212 | if len(global_pos_mat)==0: 213 | vals[i,j,0]= 0 214 | vals[i,j,2] = 0 215 | else: 216 | vals[i,j,0] = int(np.mean(global_pos_mat)) 217 | vals[i,j,2] = int(np.var(global_pos_mat)) 218 | 219 | if len(global_neg_mat)==0: 220 | vals[i,j,1]= 0 221 | vals[i,j,3] = 0 222 | else: 223 | vals[i,j,1]= int(np.mean(global_neg_mat)) 224 | vals[i,j,3]= int(np.var(global_neg_mat)) 225 | 226 | if local_objective == 1: 227 | vals[i,j,4], vals[i,j,5] = get_local_objective_buffer_means(img,endpoints,angle,buffer_size) 228 | 229 | vals[i,j,6] = angle 230 | vals[i,j,7] = distance 231 | 232 | j+=1 233 | 234 | i += 1 235 | 236 | return vals -------------------------------------------------------------------------------- /two_objectives_horizon_detection_notebook.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Vision-Based Horizon Detection with Dual Global and Local Objectives\n", 8 | "\n", 9 | "The following notebook walks you through an implementation of a computer vision-based horizon detection that uses a two-stage objective that greatly reduces computational overhead compared to the typical exhaustive search approach.\n", 10 | "\n", 11 | "The first objective (\"global\") attempts to find the range of combinations of \"pitch\" and \"roll\" (attitude and angle) corresponding to a halfplane that likely subdivdes the sky from the rest of the image. The second objective (\"local\") searches exhaustively through these combinations to find the halfplane that maximizes the difference in average intensity of the two halfplanes in the immediate viscinity of the halfplane.\n", 12 | "\n", 13 | "Compared with an exhaustive search of pitch and roll combinations performed at the outset, this method obtains perfect accuracy on our datset with a full order-of-magnitute less computations. This method benefits from the assumption that a \"sky\" as represented by image data has higher intensity values than the ground pixels (higher mean), and that the sky has higher consistency of representation (lower variance). A \"coefficient of variance\" calculation (ratio of mean to variance) performed on the full range of angle/attitude for all images in the set suggests that--at least for our data--the optimization surface is approximately convex, allowing for confident use of subsampling in the global objective.\n", 14 | "\n", 15 | "Exhaustive search over the second objective for our dataset in the data exploration phase however reveals numerous local maxima- therefore we must employ exhaustive search over the subregion identified in the global objective in runtime. Regardless, we observe no loss of accuracy on out output.\n", 16 | "\n", 17 | "The method is demonstrated on a selection of maritime images whose time-of-day, glare effects, and ground-truth horizon location within the frame and varied.\n", 18 | "\n", 19 | "*Main()* and associated functions are located in *two_objectives_horizon_detection.py*." 20 | ] 21 | }, 22 | { 23 | "cell_type": "markdown", 24 | "metadata": {}, 25 | "source": [ 26 | "## Header" 27 | ] 28 | }, 29 | { 30 | "cell_type": "code", 31 | "execution_count": null, 32 | "metadata": { 33 | "collapsed": false 34 | }, 35 | "outputs": [], 36 | "source": [ 37 | "%matplotlib tk\n", 38 | "\n", 39 | "from __future__ import division\n", 40 | "\n", 41 | "import numpy as np\n", 42 | "import cv2\n", 43 | "from matplotlib import pyplot as plt\n", 44 | "from mpl_toolkits.mplot3d import Axes3D\n", 45 | "from matplotlib import path\n", 46 | "import os\n", 47 | "\n", 48 | "import two_objectives_horizon_detection as tohd\n", 49 | "\n", 50 | "current_dir = os.getcwd()\n", 51 | "img_dir = os.path.join(current_dir,\"images\")\n", 52 | "files = [os.path.join(\"images\",x) for x in os.listdir(img_dir) if x.endswith('.jpeg')]" 53 | ] 54 | }, 55 | { 56 | "cell_type": "markdown", 57 | "metadata": {}, 58 | "source": [ 59 | "## GLOBAL OBJECTIVE" 60 | ] 61 | }, 62 | { 63 | "cell_type": "code", 64 | "execution_count": null, 65 | "metadata": { 66 | "collapsed": false 67 | }, 68 | "outputs": [], 69 | "source": [ 70 | "#GLOBAL OBJECTIVE SETTINGS\n", 71 | "img_file = files[10]\n", 72 | "global_img_reduction = 0.1\n", 73 | "global_angles = (-90,91,5)\n", 74 | "global_distances = (5,100,5) \n", 75 | "global_buffer_size = 3\n", 76 | "\n", 77 | "#GLOBAL OBJECTIVE MAIN ROUTINE\n", 78 | "global_search = tohd.main(img_file, \n", 79 | " img_reduction = global_img_reduction,\n", 80 | " angles = global_angles, \n", 81 | " distances = global_distances, \n", 82 | " buffer_size = global_buffer_size, \n", 83 | " local_objective = 0) #2m5s\n", 84 | "\n", 85 | "#GLOBAL OBJECTIVE OPTIMIZATION SURFACE\n", 86 | "objective_1 = np.max((global_search[:,:,0] - global_search[:,:,1]),0) / (global_search[:,:,2])" 87 | ] 88 | }, 89 | { 90 | "cell_type": "markdown", 91 | "metadata": {}, 92 | "source": [ 93 | "### Global Objective Optimization Surface" 94 | ] 95 | }, 96 | { 97 | "cell_type": "code", 98 | "execution_count": null, 99 | "metadata": { 100 | "collapsed": false 101 | }, 102 | "outputs": [], 103 | "source": [ 104 | "X = np.arange(0, len(range(*global_angles)), 1)\n", 105 | "Y = np.arange(0, len(range(*global_distances)), 1)\n", 106 | "X, Y = np.meshgrid(Y, X)\n", 107 | "Z = objective_1\n", 108 | "\n", 109 | "fig = plt.figure()\n", 110 | "ax = fig.add_subplot(111, projection='3d')\n", 111 | "ax.plot_wireframe(X,Y,Z, rstride=1, cstride=1)\n", 112 | "plt.draw()" 113 | ] 114 | }, 115 | { 116 | "cell_type": "markdown", 117 | "metadata": {}, 118 | "source": [ 119 | "## LOCAL OBJECTIVE" 120 | ] 121 | }, 122 | { 123 | "cell_type": "code", 124 | "execution_count": null, 125 | "metadata": { 126 | "collapsed": false 127 | }, 128 | "outputs": [], 129 | "source": [ 130 | "#LOCAL OBJECTIVE SETTINGS\n", 131 | "above_two_sigma = (2* np.nanstd(objective_1)) + np.nanmean(objective_1)\n", 132 | "\n", 133 | "local_angles = global_search[np.where(objective_1 > above_two_sigma)[0],\n", 134 | " np.where(objective_1 > above_two_sigma)[1]][:,6]\n", 135 | "local_distances = global_search[np.where(objective_1 > above_two_sigma)[0],\n", 136 | " np.where(objective_1 > above_two_sigma)[1]][:,7]\n", 137 | "local_angle_range = (int(np.min(local_angles))-2,\n", 138 | " int(np.max(local_angles))+3,1)\n", 139 | "local_distance_range = (int(np.min(local_distances))-2,\n", 140 | " int(np.max(local_distances))+3,1)\n", 141 | "\n", 142 | "local_img_reduction = 0.25\n", 143 | "local_angles = local_angle_range\n", 144 | "local_distances = local_distance_range\n", 145 | "local_buffer_size = 5\n", 146 | "\n", 147 | "#LOCAL OBJECTIVE MAIN ROUTINE\n", 148 | "print(\"Evaluating\",(len(range(*local_angle_range))*len(range(*local_distance_range))),\"candidates...\")\n", 149 | "\n", 150 | "local_search = tohd.main(img_file, \n", 151 | " img_reduction = local_img_reduction,\n", 152 | " angles = local_angles, \n", 153 | " distances = local_distances, \n", 154 | " buffer_size = local_buffer_size, \n", 155 | " local_objective = 1) #2m5s\n", 156 | "\n", 157 | "#LOCAL OBJECTIVE OPTIMIZATION SURFACE\n", 158 | "objective_2 = (local_search[:,:,4] - local_search[:,:,5])**2 / local_search[:,:,2]\n", 159 | "\n", 160 | "#DETECTED HORIZON LINE\n", 161 | "horizon_line = local_search[np.unravel_index(objective_2.argmax(), objective_2.shape)]\n", 162 | "print(\"For \",img_file,\", best predicted line is\",horizon_line[6],\"degrees and a distance of\",horizon_line[7])" 163 | ] 164 | }, 165 | { 166 | "cell_type": "markdown", 167 | "metadata": {}, 168 | "source": [ 169 | "## DETECTED HORIZON LINE" 170 | ] 171 | }, 172 | { 173 | "cell_type": "markdown", 174 | "metadata": {}, 175 | "source": [ 176 | "### OVERLAY HORIZON LINE ON INPUT IMAGE AND DISPLAY" 177 | ] 178 | }, 179 | { 180 | "cell_type": "code", 181 | "execution_count": null, 182 | "metadata": { 183 | "collapsed": false 184 | }, 185 | "outputs": [], 186 | "source": [ 187 | "img = cv2.imread(img_file,0)\n", 188 | "line_coordinates = tohd.get_plane_indicator_coord(img,int(horizon_line[6]),horizon_line[7]/100,0)[2:4]\n", 189 | "cv2.line(img,(line_coordinates[0][0],line_coordinates[0][1]),(line_coordinates[1][0],line_coordinates[1][1]),(0,0,255),2)\n", 190 | "\n", 191 | "plt.imshow(img)\n", 192 | "plt.show()" 193 | ] 194 | } 195 | ], 196 | "metadata": { 197 | "kernelspec": { 198 | "display_name": "Python 2", 199 | "language": "python", 200 | "name": "python2" 201 | }, 202 | "language_info": { 203 | "codemirror_mode": { 204 | "name": "ipython", 205 | "version": 2 206 | }, 207 | "file_extension": ".py", 208 | "mimetype": "text/x-python", 209 | "name": "python", 210 | "nbconvert_exporter": "python", 211 | "pygments_lexer": "ipython2", 212 | "version": "2.7.11" 213 | } 214 | }, 215 | "nbformat": 4, 216 | "nbformat_minor": 1 217 | } 218 | --------------------------------------------------------------------------------