├── README.md
├── eval
├── eval_img_process_tf.py
├── eval_ssd
├── np_methods.py
├── show_ssd_network.py
└── timg.jpeg
├── preprocess_img_tf.py
├── ssd_vgg300_tf.py
├── tfr_data_process.py
├── tfr_generate.ipynb
├── train_ssd_network.py
└── util_tf.py
/README.md:
--------------------------------------------------------------------------------
1 | # SSD_Realization_TensorFlow
2 | [『TensorFlow × MXNet』SSD项目复现经验](https://www.cnblogs.com/hellcat/p/9540591.html)
3 | ## 使用
4 | #### 数据准备
5 | 将VOC2012数据解压到文件夹VOC2012中，注意检查下一级目录包含Annotations文件夹和JPEGImages文件夹。
6 | #### 生成TFR压缩数据
7 | 使用jupyter运行`tfr_generate.ipynb`脚本，使用TFR格式压缩图片对于提升训练速度大有裨益
8 | #### 训练模型
9 | 调用脚本即可
10 | ```bash
11 | python train_ssd_network.py
12 | ```
13 | #### 测试运行
14 | 进入eval目录，运行:
15 | ```bash
16 | python show_ssd_network.py
17 | ```
18 | 在该脚本中有设置图片路径的位置，替换为想要检测的图片即可。
19 | ## 实验说明
20 | 本工程17w steps后结果（框体过多的可以将NMS阈值减小）：
21 | 
22 | 
23 | 损失函数变化情况如图：
24 | 
25 | ## 简要说明
26 | SSD架构主要有四个部分，网络设计、搜索框设计、学习目标处理、损失函数实现。
27 | ###### 网络设计
28 | 重点在于正常前向网络中挑选出的特征层分别添加两个卷积出口：分类和回归出口，用于对应后面的每个搜索框的各个类别得分、以及4个坐标值。
29 | ###### 搜索框设计
30 | 对应网络的特征层：每个层有若干搜索框，我们需要搜索框位置形状信息。对于tf版本我们保存了每个框的中心点以及HW信息，而mx版本我们保存的是左上右下两个的4个坐标数值，mx更为直观，但是tf版本节省空间：一组框对应同一个中心点，不过搜索框信息量不大，b无伤大雅。
31 | ###### 学习目标处理
32 | 个人感觉最为繁琐，我们需要的的信息包含（此时已经获得了）：一组搜索框（实际上指的是全部搜索框的n*4个坐标值），图片的label、图片的真实框坐标(对应label数目*4)，我们需要的就是找到搜索框和真是图片的标签联系，
33 | 获取：
34 | 每个搜索框对应的分类（和哪个真实框的IOU最大就选真实框的类别标注给该搜索，也就是说会出现大量的0 class搜索框）
35 | 每个搜索框的坐标的回归目标（同上的寻找方法，空位也为0）
36 | 负类掩码，虽然每张图片里面通常只有几个标注的边框，但SSD会生成大量的锚框。可以想象很多锚框都不会框住感兴趣的物体，就是说跟任何对应感兴趣物体的表框的IoU都小于某个阈值。这样就会产生大量的负类锚框，或者说对应标号为0的锚框。对于这类锚框有两点要考虑的：
37 | 1、边框预测的损失函数不应该包括负类锚框，因为它们并没有对应的真实边框
38 | 2、因为负类锚框数目可能远多于其他，我们可以只保留其中的一些。而且是保留那些目前预测最不确信它是负类的，就是对类0预测值排序，选取数值最小的哪一些困难的负类锚框
39 | 所以需要使用掩码，抑制一部分计算出来的loss。
40 | 损失函数可讲的不多，按照公式实现即可，重点也在上一步计算出来的掩码处理损失函数值一步。
41 | ## 日志
42 | #### 18.8.27
43 | 截止目前，本改版网络已经可以正常运行稳定收敛，之前的问题及解决思路如下：
44 | ###### 1.解决了之前版本Loss值过大且不收敛的问题
45 | 这个问题实际上是因为我个人的疏忽，将未预处理的图片送入ssd.bboxes_encode函数中，修正后如下，
46 | ```python
47 | image, glabels, gbboxes = \
48 | tfr_data_process.tfr_read(dataset)
49 |
50 | image, glabels, gbboxes = \
51 | preprocess_img_tf.preprocess_image(image, glabels, gbboxes, out_shape=(300, 300))
52 |
53 | gclasses, glocalisations, gscores = \
54 | ssd.bboxes_encode(glabels, gbboxes, ssd_anchors)
55 | ```
56 | 这个疏忽导致Loss值维持在200～400之间且不收敛，修改后经过300左右steps损失函数会稳定到60左右，和原SSD网络一致(示意如下)。
57 | 
58 | ###### 2.解决了训练速度过慢的问题
59 | 原SSD模型训练速度（CPU：E5-2690，GPU：1080Ti）大概50 samples/sec（实际上略高与此），我的训练速度仅仅22-24 samples/sec，经对比查验，应该是节点分配硬件设备的配置优化问题，涉及队列（主要是数据输入）、优化器设定的节点分派给CPU后（其他节点会默认优先分配给GPU），速度明显提升，大概到达44-46 samples/sec。
60 | 另外，tfr数据解析过程放在GPU下，生成队列过程放在CPU下有不明显加速，理想情况下能提升0-1 samples/sec。
61 | 综上，现阶段的程序比原程序训练阶段还是要慢5 samples/sec左右，原因还在排查中。
62 | #### 18.8.31
63 | ###### 添加了测试脚本
64 | 参考资料见博客，由于该作者写的很好，稍作调整即可使用。
65 |
--------------------------------------------------------------------------------
/eval/eval_img_process_tf.py:
--------------------------------------------------------------------------------
1 | # Author : hellcat
2 | # Time : 18-8-30
3 |
4 | """
5 | import os
6 | os.environ["CUDA_VISIBLE_DEVICES"]="-1"
7 |
8 | import numpy as np
9 | np.set_printoptions(threshold=np.inf)
10 |
11 | import tensorflow as tf
12 | config = tf.ConfigProto()
13 | config.gpu_options.allow_growth = True
14 | sess = tf.Session(config=config)
15 | """
16 | import tensorflow as tf
17 | from tensorflow.python.framework import ops
18 | from tensorflow.python.ops import variables
19 | from tensorflow.python.ops import control_flow_ops
20 | from tensorflow.python.ops import array_ops
21 | from tensorflow.python.ops import check_ops
22 | from tensorflow.python.ops import math_ops
23 | import util_tf
24 |
25 |
26 | from enum import Enum, IntEnum
27 | # Resizing strategies.
28 | Resize = IntEnum('Resize', ('NONE', # Nothing!
29 | 'CENTRAL_CROP', # Crop (and pad if necessary).
30 | 'PAD_AND_RESIZE', # Pad, and resize to output shape.
31 | 'WARP_RESIZE')) # Warp resize.
32 | # VGG mean parameters.
33 | _R_MEAN = 123.
34 | _G_MEAN = 117.
35 | _B_MEAN = 104.
36 | EVAL_SIZE = (300, 300)
37 |
38 |
39 | def _is_tensor(x):
40 | """Returns `True` if `x` is a symbolic tensor-like object.
41 | Args:
42 | x: A python object to check.
43 | Returns:
44 | `True` if `x` is a `tf.Tensor` or `tf.Variable`, otherwise `False`.
45 | """
46 | return isinstance(x, (ops.Tensor, variables.Variable))
47 |
48 |
49 | def _assert(cond, ex_type, msg):
50 | """A polymorphic assert, works with tensors and boolean expressions.
51 | If `cond` is not a tensor, behave like an ordinary assert statement, except
52 | that a empty list is returned. If `cond` is a tensor, return a list
53 | containing a single TensorFlow assert op.
54 | Args:
55 | cond: Something evaluates to a boolean value. May be a tensor.
56 | ex_type: The exception class to use.
57 | msg: The error message.
58 | Returns:
59 | A list, containing at most one assert op.
60 | """
61 | if _is_tensor(cond):
62 | return [control_flow_ops.Assert(cond, [msg])]
63 | else:
64 | if not cond:
65 | raise ex_type(msg)
66 | else:
67 | return []
68 |
69 |
70 | def _Check3DImage(image, require_static=True):
71 | """Assert that we are working with properly shaped image.
72 | Args:
73 | image: 3-D Tensor of shape [height, width, channels]
74 | require_static: If `True`, requires that all dimensions of `image` are
75 | known and non-zero.
76 | Raises:
77 | ValueError: if `image.shape` is not a 3-vector.
78 | Returns:
79 | An empty list, if `image` has fully defined dimensions. Otherwise, a list
80 | containing an assert op is returned.
81 | """
82 | try:
83 | image_shape = image.get_shape().with_rank(3)
84 | except ValueError:
85 | raise ValueError("'image' must be three-dimensional.")
86 | if require_static and not image_shape.is_fully_defined():
87 | raise ValueError("'image' must be fully defined.")
88 | if any(x == 0 for x in image_shape):
89 | raise ValueError("all dims of 'image.shape' must be > 0: %s" %
90 | image_shape)
91 | if not image_shape.is_fully_defined():
92 | return [check_ops.assert_positive(array_ops.shape(image),
93 | ["all dims of 'image.shape' "
94 | "must be > 0."])]
95 | else:
96 | return []
97 |
98 |
99 | def bboxes_crop_or_pad(bboxes,
100 | height, width,
101 | offset_y, offset_x,
102 | target_height, target_width):
103 | """Adapt bounding boxes to crop or pad operations.
104 | Coordinates are always supposed to be relative to the image.
105 |
106 | Arguments:
107 | bboxes: Tensor Nx4 with bboxes coordinates [y_min, x_min, y_max, x_max];
108 | height, width: Original image dimension;
109 | offset_y, offset_x: Offset to apply,
110 | negative if cropping, positive if padding;
111 | target_height, target_width: Target dimension after cropping / padding.
112 | """
113 | with tf.name_scope('bboxes_crop_or_pad'):
114 | # Rescale bounding boxes in pixels.
115 | scale = tf.cast(tf.stack([height, width, height, width]), bboxes.dtype)
116 | bboxes = bboxes * scale
117 | # Add offset.
118 | offset = tf.cast(tf.stack([offset_y, offset_x, offset_y, offset_x]), bboxes.dtype)
119 | bboxes = bboxes + offset
120 | # Rescale to target dimension.
121 | scale = tf.cast(tf.stack([target_height, target_width,
122 | target_height, target_width]), bboxes.dtype)
123 | bboxes = bboxes / scale
124 | return bboxes
125 |
126 |
127 | def _ImageDimensions(image):
128 | """Returns the dimensions of an image tensor.
129 | Args:
130 | image: A 3-D Tensor of shape `[height, width, channels]`.
131 | Returns:
132 | A list of `[height, width, channels]` corresponding to the dimensions of the
133 | input image. Dimensions that are statically known are python integers,
134 | otherwise they are integer scalar tensors.
135 | """
136 | if image.get_shape().is_fully_defined():
137 | return image.get_shape().as_list()
138 | else:
139 | static_shape = image.get_shape().with_rank(3).as_list()
140 | dynamic_shape = array_ops.unstack(array_ops.shape(image), 3)
141 | return [s if s is not None else d
142 | for s, d in zip(static_shape, dynamic_shape)]
143 |
144 |
145 | def resize_image_bboxes_with_crop_or_pad(image, bboxes,
146 | target_height, target_width):
147 | """Crops and/or pads an image to a target width and height.
148 | Resizes an image to a target width and height by either centrally
149 | cropping the image or padding it evenly with zeros.
150 |
151 | If `width` or `height` is greater than the specified `target_width` or
152 | `target_height` respectively, this op centrally crops along that dimension.
153 | If `width` or `height` is smaller than the specified `target_width` or
154 | `target_height` respectively, this op centrally pads with 0 along that
155 | dimension.
156 | Args:
157 | image: 3-D tensor of shape `[height, width, channels]`
158 | target_height: Target height.
159 | target_width: Target width.
160 | Raises:
161 | ValueError: if `target_height` or `target_width` are zero or negative.
162 | Returns:
163 | Cropped and/or padded image of shape
164 | `[target_height, target_width, channels]`
165 | """
166 | with tf.name_scope('resize_with_crop_or_pad'):
167 | image = ops.convert_to_tensor(image, name='image')
168 |
169 | assert_ops = []
170 | assert_ops += _Check3DImage(image, require_static=False)
171 | assert_ops += _assert(target_width > 0, ValueError,
172 | 'target_width must be > 0.')
173 | assert_ops += _assert(target_height > 0, ValueError,
174 | 'target_height must be > 0.')
175 |
176 | image = control_flow_ops.with_dependencies(assert_ops, image)
177 | # `crop_to_bounding_box` and `pad_to_bounding_box` have their own checks.
178 | # Make sure our checks come first, so that error messages are clearer.
179 | if _is_tensor(target_height):
180 | target_height = control_flow_ops.with_dependencies(
181 | assert_ops, target_height)
182 | if _is_tensor(target_width):
183 | target_width = control_flow_ops.with_dependencies(assert_ops, target_width)
184 |
185 | def max_(x, y):
186 | if _is_tensor(x) or _is_tensor(y):
187 | return math_ops.maximum(x, y)
188 | else:
189 | return max(x, y)
190 |
191 | def min_(x, y):
192 | if _is_tensor(x) or _is_tensor(y):
193 | return math_ops.minimum(x, y)
194 | else:
195 | return min(x, y)
196 |
197 | def equal_(x, y):
198 | if _is_tensor(x) or _is_tensor(y):
199 | return math_ops.equal(x, y)
200 | else:
201 | return x == y
202 |
203 | height, width, _ = _ImageDimensions(image)
204 | width_diff = target_width - width
205 | offset_crop_width = max_(-width_diff // 2, 0)
206 | offset_pad_width = max_(width_diff // 2, 0)
207 |
208 | height_diff = target_height - height
209 | offset_crop_height = max_(-height_diff // 2, 0)
210 | offset_pad_height = max_(height_diff // 2, 0)
211 |
212 | # Maybe crop if needed.
213 | height_crop = min_(target_height, height)
214 | width_crop = min_(target_width, width)
215 | cropped = tf.image.crop_to_bounding_box(image, offset_crop_height, offset_crop_width,
216 | height_crop, width_crop)
217 | bboxes = bboxes_crop_or_pad(bboxes,
218 | height, width,
219 | -offset_crop_height, -offset_crop_width,
220 | height_crop, width_crop)
221 | # Maybe pad if needed.
222 | resized = tf.image.pad_to_bounding_box(cropped, offset_pad_height, offset_pad_width,
223 | target_height, target_width)
224 | bboxes = bboxes_crop_or_pad(bboxes,
225 | height_crop, width_crop,
226 | offset_pad_height, offset_pad_width,
227 | target_height, target_width)
228 |
229 | # In theory all the checks below are redundant.
230 | if resized.get_shape().ndims is None:
231 | raise ValueError('resized contains no shape.')
232 |
233 | resized_height, resized_width, _ = _ImageDimensions(resized)
234 |
235 | assert_ops = []
236 | assert_ops += _assert(equal_(resized_height, target_height), ValueError,
237 | 'resized height is not correct.')
238 | assert_ops += _assert(equal_(resized_width, target_width), ValueError,
239 | 'resized width is not correct.')
240 |
241 | resized = control_flow_ops.with_dependencies(assert_ops, resized)
242 | return resized, bboxes
243 |
244 |
245 | def tf_image_whitened(image, means=(_R_MEAN, _G_MEAN, _B_MEAN)):
246 | """Subtracts the given means from each image channel.
247 |
248 | Returns:
249 | the centered image.
250 | """
251 | if image.get_shape().ndims != 3:
252 | raise ValueError('Input must be of size [height, width, C>0]')
253 | num_channels = image.get_shape().as_list()[-1]
254 | if len(means) != num_channels:
255 | raise ValueError('len(means) must match the number of channels')
256 |
257 | mean = tf.constant(means, dtype=image.dtype)
258 | image = image - mean
259 | return image
260 |
261 |
262 | def preprocess_for_eval(image, labels, bboxes,
263 | out_shape=EVAL_SIZE, data_format='NHWC',
264 | difficults=None, resize=Resize.WARP_RESIZE,
265 | scope='ssd_preprocessing_train'):
266 | """Preprocess an image for evaluation.
267 | Returns:
268 | A preprocessed image.
269 | """
270 | with tf.name_scope(scope):
271 | if image.get_shape().ndims != 3:
272 | raise ValueError('Input must be of size [height, width, C>0]')
273 |
274 | image = tf.to_float(image)
275 | image = tf_image_whitened(image, [_R_MEAN, _G_MEAN, _B_MEAN])
276 |
277 | # Add image rectangle to bboxes.
278 | bbox_img = tf.constant([[0., 0., 1., 1.]])
279 | if bboxes is None:
280 | bboxes = bbox_img
281 | else:
282 | bboxes = tf.concat([bbox_img, bboxes], axis=0)
283 |
284 | if resize == Resize.NONE:
285 | # No resizing...
286 | pass
287 | elif resize == Resize.CENTRAL_CROP:
288 | # Central cropping of the image.
289 | image, bboxes = resize_image_bboxes_with_crop_or_pad(
290 | image, bboxes, out_shape[0], out_shape[1])
291 | elif resize == Resize.PAD_AND_RESIZE:
292 | # Resize image first: find the correct factor...
293 | shape = tf.shape(image)
294 | factor = tf.minimum(tf.to_double(1.0),
295 | tf.minimum(tf.to_double(out_shape[0] / shape[0]),
296 | tf.to_double(out_shape[1] / shape[1])))
297 | resize_shape = factor * tf.to_double(shape[0:2])
298 | resize_shape = tf.cast(tf.floor(resize_shape), tf.int32)
299 |
300 | image = util_tf.resize_image(image, resize_shape,
301 | method=tf.image.ResizeMethod.BILINEAR,
302 | align_corners=False)
303 | # Pad to expected size.
304 | image, bboxes = resize_image_bboxes_with_crop_or_pad(
305 | image, bboxes, out_shape[0], out_shape[1])
306 | elif resize == Resize.WARP_RESIZE:
307 | # Warp resize of the image.
308 | image = util_tf.resize_image(image, out_shape,
309 | method=tf.image.ResizeMethod.BILINEAR,
310 | align_corners=False)
311 |
312 | # Split back bounding boxes.
313 | bbox_img = bboxes[0]
314 | bboxes = bboxes[1:]
315 | # Remove difficult boxes.
316 | if difficults is not None:
317 | mask = tf.logical_not(tf.cast(difficults, tf.bool))
318 | labels = tf.boolean_mask(labels, mask)
319 | bboxes = tf.boolean_mask(bboxes, mask)
320 | # Image data format.
321 | if data_format == 'NCHW':
322 | image = tf.transpose(image, perm=(2, 0, 1))
323 | return image, labels, bboxes, bbox_img
--------------------------------------------------------------------------------
/eval/eval_ssd:
--------------------------------------------------------------------------------
1 | eval
2 |
--------------------------------------------------------------------------------
/eval/np_methods.py:
--------------------------------------------------------------------------------
1 | # Copyright 2017 Paul Balanca. All Rights Reserved.
2 | #
3 | # Licensed under the Apache License, Version 2.0 (the "License");
4 | # you may not use this file except in compliance with the License.
5 | # You may obtain a copy of the License at
6 | #
7 | # http://www.apache.org/licenses/LICENSE-2.0
8 | #
9 | # Unless required by applicable law or agreed to in writing, software
10 | # distributed under the License is distributed on an "AS IS" BASIS,
11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12 | # See the License for the specific language governing permissions and
13 | # limitations under the License.
14 | # ==============================================================================
15 | """Additional Numpy methods. Big mess of many things!
16 | """
17 | import numpy as np
18 |
19 |
20 | # =========================================================================== #
21 | # Numpy implementations of SSD boxes functions.
22 | # =========================================================================== #
23 | def ssd_bboxes_decode(feat_localizations,
24 | anchor_bboxes,
25 | prior_scaling=[0.1, 0.1, 0.2, 0.2]):
26 | """Compute the relative bounding boxes from the layer features and
27 | reference anchor bounding boxes.
28 |
29 | Return:
30 | numpy array Nx4: ymin, xmin, ymax, xmax
31 | """
32 | # Reshape for easier broadcasting.
33 | l_shape = feat_localizations.shape
34 | feat_localizations = np.reshape(feat_localizations,
35 | (-1, l_shape[-2], l_shape[-1]))
36 | yref, xref, href, wref = anchor_bboxes
37 | xref = np.reshape(xref, [-1, 1])
38 | yref = np.reshape(yref, [-1, 1])
39 |
40 | # Compute center, height and width
41 | cx = feat_localizations[:, :, 0] * wref * prior_scaling[0] + xref
42 | cy = feat_localizations[:, :, 1] * href * prior_scaling[1] + yref
43 | w = wref * np.exp(feat_localizations[:, :, 2] * prior_scaling[2])
44 | h = href * np.exp(feat_localizations[:, :, 3] * prior_scaling[3])
45 | # bboxes: ymin, xmin, xmax, ymax.
46 | bboxes = np.zeros_like(feat_localizations)
47 | bboxes[:, :, 0] = cy - h / 2.
48 | bboxes[:, :, 1] = cx - w / 2.
49 | bboxes[:, :, 2] = cy + h / 2.
50 | bboxes[:, :, 3] = cx + w / 2.
51 | # Back to original shape.
52 | bboxes = np.reshape(bboxes, l_shape)
53 | return bboxes
54 |
55 |
56 | def ssd_bboxes_select_layer(predictions_layer,
57 | localizations_layer,
58 | anchors_layer,
59 | select_threshold=0.5,
60 | img_shape=(300, 300),
61 | num_classes=21,
62 | decode=True):
63 | """Extract classes, scores and bounding boxes from features in one layer.
64 |
65 | Return:
66 | classes, scores, bboxes: Numpy arrays...
67 | """
68 | # First decode localizations features if necessary.
69 | if decode:
70 | localizations_layer = ssd_bboxes_decode(localizations_layer, anchors_layer)
71 |
72 | # Reshape features to: Batches x N x N_labels | 4.
73 | p_shape = predictions_layer.shape
74 | batch_size = p_shape[0] if len(p_shape) == 5 else 1
75 | predictions_layer = np.reshape(predictions_layer,
76 | (batch_size, -1, p_shape[-1]))
77 | l_shape = localizations_layer.shape
78 | localizations_layer = np.reshape(localizations_layer,
79 | (batch_size, -1, l_shape[-1]))
80 |
81 | # Boxes selection: use threshold or score > no-label criteria.
82 | if select_threshold is None or select_threshold == 0:
83 | # Class prediction and scores: assign 0. to 0-class
84 | classes = np.argmax(predictions_layer, axis=2)
85 | scores = np.amax(predictions_layer, axis=2)
86 | mask = (classes > 0)
87 | classes = classes[mask]
88 | scores = scores[mask]
89 | bboxes = localizations_layer[mask]
90 | else:
91 | sub_predictions = predictions_layer[:, :, 1:]
92 | idxes = np.where(sub_predictions > select_threshold)
93 | classes = idxes[-1]+1
94 | scores = sub_predictions[idxes]
95 | bboxes = localizations_layer[idxes[:-1]]
96 |
97 | return classes, scores, bboxes
98 |
99 |
100 | def ssd_bboxes_select(predictions_net,
101 | localizations_net,
102 | anchors_net,
103 | select_threshold=0.5,
104 | img_shape=(300, 300),
105 | num_classes=21,
106 | decode=True):
107 | """Extract classes, scores and bounding boxes from network output layers.
108 |
109 | Return:
110 | classes, scores, bboxes: Numpy arrays...
111 | """
112 | l_classes = []
113 | l_scores = []
114 | l_bboxes = []
115 | # l_layers = []
116 | # l_idxes = []
117 | for i in range(len(predictions_net)):
118 | classes, scores, bboxes = ssd_bboxes_select_layer(
119 | predictions_net[i], localizations_net[i], anchors_net[i],
120 | select_threshold, img_shape, num_classes, decode)
121 | l_classes.append(classes)
122 | l_scores.append(scores)
123 | l_bboxes.append(bboxes)
124 | # Debug information.
125 | # l_layers.append(i)
126 | # l_idxes.append((i, idxes))
127 |
128 | classes = np.concatenate(l_classes, 0)
129 | scores = np.concatenate(l_scores, 0)
130 | bboxes = np.concatenate(l_bboxes, 0)
131 | return classes, scores, bboxes
132 |
133 |
134 | # =========================================================================== #
135 | # Common functions for bboxes handling and selection.
136 | # =========================================================================== #
137 | def bboxes_sort(classes, scores, bboxes, top_k=400):
138 | """Sort bounding boxes by decreasing order and keep only the top_k
139 | """
140 | # if priority_inside:
141 | # inside = (bboxes[:, 0] > margin) & (bboxes[:, 1] > margin) & \
142 | # (bboxes[:, 2] < 1-margin) & (bboxes[:, 3] < 1-margin)
143 | # idxes = np.argsort(-scores)
144 | # inside = inside[idxes]
145 | # idxes = np.concatenate([idxes[inside], idxes[~inside]])
146 | idxes = np.argsort(-scores)
147 | classes = classes[idxes][:top_k]
148 | scores = scores[idxes][:top_k]
149 | bboxes = bboxes[idxes][:top_k]
150 | return classes, scores, bboxes
151 |
152 |
153 | def bboxes_clip(bbox_ref, bboxes):
154 | """Clip bounding boxes with respect to reference bbox.
155 | """
156 | bboxes = np.copy(bboxes)
157 | bboxes = np.transpose(bboxes)
158 | bbox_ref = np.transpose(bbox_ref)
159 | bboxes[0] = np.maximum(bboxes[0], bbox_ref[0])
160 | bboxes[1] = np.maximum(bboxes[1], bbox_ref[1])
161 | bboxes[2] = np.minimum(bboxes[2], bbox_ref[2])
162 | bboxes[3] = np.minimum(bboxes[3], bbox_ref[3])
163 | bboxes = np.transpose(bboxes)
164 | return bboxes
165 |
166 |
167 | def bboxes_resize(bbox_ref, bboxes):
168 | """Resize bounding boxes based on a reference bounding box,
169 | assuming that the latter is [0, 0, 1, 1] after transform.
170 | """
171 | bboxes = np.copy(bboxes)
172 | # Translate.
173 | bboxes[:, 0] -= bbox_ref[0]
174 | bboxes[:, 1] -= bbox_ref[1]
175 | bboxes[:, 2] -= bbox_ref[0]
176 | bboxes[:, 3] -= bbox_ref[1]
177 | # Resize.
178 | resize = [bbox_ref[2] - bbox_ref[0], bbox_ref[3] - bbox_ref[1]]
179 | bboxes[:, 0] /= resize[0]
180 | bboxes[:, 1] /= resize[1]
181 | bboxes[:, 2] /= resize[0]
182 | bboxes[:, 3] /= resize[1]
183 | return bboxes
184 |
185 |
186 | def bboxes_jaccard(bboxes1, bboxes2):
187 | """Computing jaccard index between bboxes1 and bboxes2.
188 | Note: bboxes1 and bboxes2 can be multi-dimensional, but should broacastable.
189 | """
190 | bboxes1 = np.transpose(bboxes1)
191 | bboxes2 = np.transpose(bboxes2)
192 | # Intersection bbox and volume.
193 | int_ymin = np.maximum(bboxes1[0], bboxes2[0])
194 | int_xmin = np.maximum(bboxes1[1], bboxes2[1])
195 | int_ymax = np.minimum(bboxes1[2], bboxes2[2])
196 | int_xmax = np.minimum(bboxes1[3], bboxes2[3])
197 |
198 | int_h = np.maximum(int_ymax - int_ymin, 0.)
199 | int_w = np.maximum(int_xmax - int_xmin, 0.)
200 | int_vol = int_h * int_w
201 | # Union volume.
202 | vol1 = (bboxes1[2] - bboxes1[0]) * (bboxes1[3] - bboxes1[1])
203 | vol2 = (bboxes2[2] - bboxes2[0]) * (bboxes2[3] - bboxes2[1])
204 | jaccard = int_vol / (vol1 + vol2 - int_vol)
205 | return jaccard
206 |
207 |
208 | def bboxes_intersection(bboxes_ref, bboxes2):
209 | """Computing jaccard index between bboxes1 and bboxes2.
210 | Note: bboxes1 and bboxes2 can be multi-dimensional, but should broacastable.
211 | """
212 | bboxes_ref = np.transpose(bboxes_ref)
213 | bboxes2 = np.transpose(bboxes2)
214 | # Intersection bbox and volume.
215 | int_ymin = np.maximum(bboxes_ref[0], bboxes2[0])
216 | int_xmin = np.maximum(bboxes_ref[1], bboxes2[1])
217 | int_ymax = np.minimum(bboxes_ref[2], bboxes2[2])
218 | int_xmax = np.minimum(bboxes_ref[3], bboxes2[3])
219 |
220 | int_h = np.maximum(int_ymax - int_ymin, 0.)
221 | int_w = np.maximum(int_xmax - int_xmin, 0.)
222 | int_vol = int_h * int_w
223 | # Union volume.
224 | vol = (bboxes_ref[2] - bboxes_ref[0]) * (bboxes_ref[3] - bboxes_ref[1])
225 | score = int_vol / vol
226 | return score
227 |
228 |
229 | def bboxes_nms(classes, scores, bboxes, nms_threshold=0.45):
230 | """Apply non-maximum selection to bounding boxes.
231 | """
232 | keep_bboxes = np.ones(scores.shape, dtype=np.bool)
233 | for i in range(scores.size-1):
234 | if keep_bboxes[i]:
235 | # Computer overlap with bboxes which are following.
236 | overlap = bboxes_jaccard(bboxes[i], bboxes[(i+1):])
237 | # Overlap threshold for keeping + checking part of the same class
238 | keep_overlap = np.logical_or(overlap < nms_threshold, classes[(i+1):] != classes[i])
239 | keep_bboxes[(i+1):] = np.logical_and(keep_bboxes[(i+1):], keep_overlap)
240 |
241 | idxes = np.where(keep_bboxes)
242 | return classes[idxes], scores[idxes], bboxes[idxes]
243 |
244 |
245 | def bboxes_nms_fast(classes, scores, bboxes, threshold=0.45):
246 | """Apply non-maximum selection to bounding boxes.
247 | """
248 | pass
249 |
250 |
251 |
252 |
253 |
--------------------------------------------------------------------------------
/eval/show_ssd_network.py:
--------------------------------------------------------------------------------
1 | # Author : hellcat
2 | # Time : 18-8-30
3 |
4 | """
5 | import os
6 | os.environ["CUDA_VISIBLE_DEVICES"]="-1"
7 |
8 | import numpy as np
9 | np.set_printoptions(threshold=np.inf)
10 |
11 | import tensorflow as tf
12 | config = tf.ConfigProto()
13 | config.gpu_options.allow_growth = True
14 | sess = tf.Session(config=config)
15 | """
16 |
17 | import tensorflow as tf
18 |
19 | from eval.eval_img_process_tf import preprocess_for_eval, Resize
20 | from ssd_vgg300_tf import SSDNet
21 |
22 | tf.logging.set_verbosity(tf.logging.INFO)
23 | slim = tf.contrib.slim
24 |
25 | gpu_options = tf.GPUOptions(allow_growth=True)
26 | config = tf.ConfigProto(log_device_placement=False, gpu_options=gpu_options)
27 | isess = tf.InteractiveSession(config=config)
28 |
29 |
30 | img_input = tf.placeholder(tf.uint8, shape=(None, None, 3))
31 | image_pre, labels_pre, bboxes_pre, bbox_img = preprocess_for_eval(
32 | img_input,
33 | None,
34 | None,
35 | (300, 300),
36 | 'NHWC',
37 | resize=Resize.WARP_RESIZE
38 | )
39 | image_4d = tf.expand_dims(image_pre, 0)
40 |
41 | ssd = SSDNet()
42 | predictions, localisations, _, _ =\
43 | ssd.net(image_4d, is_training=False)
44 |
45 | isess.run(tf.global_variables_initializer())
46 |
47 | ckpt = tf.train.get_checkpoint_state('./logs/model/')
48 | # saver = tf.train.import_meta_graph(ckpt.model_checkpoint_path + '.meta')
49 | saver = tf.train.Saver()
50 | saver.restore(isess, ckpt.model_checkpoint_path)
51 |
52 |
53 | # 在网络模型结构中，提取搜索网格的位置
54 | ssd_anchors = ssd.anchors
55 |
56 |
57 | import cv2
58 | from eval import np_methods
59 | import matplotlib.pyplot as plt
60 | import matplotlib.cm as mpcm
61 | l_VOC_CLASS = [
62 | 'aeroplane', 'bicycle', 'bird', 'boat', 'bottle',
63 | 'bus', 'car', 'cat', 'chair', 'cow',
64 | 'diningTable', 'dog', 'horse', 'motorbike', 'person',
65 | 'pottedPlant', 'sheep', 'sofa', 'train', 'TV'
66 | ]
67 |
68 |
69 | def colors_subselect(colors, num_classes=21):
70 | dt = len(colors) // num_classes
71 | sub_colors = []
72 | for i in range(num_classes):
73 | color = colors[i*dt]
74 | if isinstance(color[0], float):
75 | sub_colors.append([int(c * 255) for c in color])
76 | else:
77 | sub_colors.append([c for c in color])
78 | return sub_colors
79 |
80 |
81 | def bboxes_draw_on_img(img, classes, scores, bboxes, colors, thickness=2):
82 | shape = img.shape
83 | for i in range(bboxes.shape[0]):
84 | bbox = bboxes[i]
85 | color = colors[classes[i]]
86 | # Draw bounding box...
87 | p1 = (int(bbox[0] * shape[0]), int(bbox[1] * shape[1]))
88 | p2 = (int(bbox[2] * shape[0]), int(bbox[3] * shape[1]))
89 | cv2.rectangle(img, p1[::-1], p2[::-1], color, thickness)
90 | # Draw text...
91 | s = '%s/%.3f' % ( l_VOC_CLASS[int(classes[i])-1], scores[i])
92 | p1 = (p1[0]-5, p1[1])
93 | #cv2.putText(img, s, p1[::-1], cv2.FONT_HERSHEY_DUPLEX, 1.5, color, 3)
94 |
95 | colors_plasma = colors_subselect(mpcm.plasma.colors, num_classes=21)
96 |
97 |
98 | # Main image processing routine.
99 | def process_image(img, select_threshold=0.3, nms_threshold=.8, net_shape=(300, 300)):
100 | # Run SSD network.
101 | rimg, rpredictions, rlocalisations, rbbox_img = isess.run([image_4d, predictions, localisations, bbox_img],
102 | feed_dict={img_input: img})
103 |
104 | # Get classes and bboxes from the net outputs.
105 | rclasses, rscores, rbboxes = np_methods.ssd_bboxes_select(
106 | rpredictions, rlocalisations, ssd_anchors,
107 | select_threshold=select_threshold, img_shape=net_shape, num_classes=21, decode=True)
108 |
109 | rbboxes = np_methods.bboxes_clip(rbbox_img, rbboxes)
110 | rclasses, rscores, rbboxes = np_methods.bboxes_sort(rclasses, rscores, rbboxes, top_k=400)
111 | rclasses, rscores, rbboxes = np_methods.bboxes_nms(rclasses, rscores, rbboxes, nms_threshold=nms_threshold)
112 | # Resize bboxes to original image shape. Note: useless for Resize.WARP!
113 | rbboxes = np_methods.bboxes_resize(rbbox_img, rbboxes)
114 | bboxes_draw_on_img(img, rclasses, rscores, rbboxes, colors_plasma, thickness=8)
115 | return img
116 |
117 | img = cv2.imread("./eval/timg.jpeg")
118 | img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
119 | plt.imshow(process_image(img))
120 | plt.show()
121 |
122 |
--------------------------------------------------------------------------------
/eval/timg.jpeg:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/Hellcatzm/SSD_Realization_TensorFlow/628924987fd4ff1d5bc50db5238257bba9357dac/eval/timg.jpeg
--------------------------------------------------------------------------------
/preprocess_img_tf.py:
--------------------------------------------------------------------------------
1 | import tensorflow as tf
2 | import util_tf
3 |
4 |
5 | def bboxes_intersection(bbox_ref, bboxes, name=None):
6 | """Compute relative intersection between a reference box and a
7 | collection of bounding boxes. Namely, compute the quotient between
8 | intersection area and box area.
9 |
10 | Args:
11 | bbox_ref: (N, 4) or (4,) Tensor with reference bounding box(es).
12 | bboxes: (N, 4) Tensor, collection of bounding boxes.
13 | Return:
14 | (N,) Tensor with relative intersection.
15 | """
16 | with tf.name_scope(name, 'bboxes_intersection'):
17 | # Should be more efficient to first transpose.
18 | bboxes = tf.transpose(bboxes)
19 | bbox_ref = tf.transpose(bbox_ref)
20 | # Intersection bbox and volume.
21 | int_ymin = tf.maximum(bboxes[0], bbox_ref[0])
22 | int_xmin = tf.maximum(bboxes[1], bbox_ref[1])
23 | int_ymax = tf.minimum(bboxes[2], bbox_ref[2])
24 | int_xmax = tf.minimum(bboxes[3], bbox_ref[3])
25 | h = tf.maximum(int_ymax - int_ymin, 0.)
26 | w = tf.maximum(int_xmax - int_xmin, 0.)
27 | # Volumes.
28 | inter_vol = h * w # 各个框在[0,0,1,1]内的面积
29 | bboxes_vol = (bboxes[2] - bboxes[0]) * (bboxes[3] - bboxes[1]) # 各个框面积
30 | scores = tf.where(
31 | tf.greater(bboxes_vol, 0),
32 | tf.divide(inter_vol, bboxes_vol),
33 | tf.zeros_like(inter_vol),
34 | name='intersection')
35 | return scores
36 |
37 |
38 | def bboxes_filter_overlap(labels, bboxes,
39 | threshold=0.5, assign_negative=False,
40 | scope=None):
41 | """Filter out bounding boxes based on (relative )overlap with reference
42 | box [0, 0, 1, 1]. Remove completely bounding boxes, or assign negative
43 | labels to the one outside (useful for latter processing...).
44 |
45 | Return:
46 | labels, bboxes: Filtered (or newly assigned) elements.
47 | """
48 | with tf.name_scope(scope, 'bboxes_filter', [labels, bboxes]):
49 | # (N,) Tensor：和[0,0,1,1]相交面积大于0的位置返回面积比（相交/原本），小于0的位置返回0
50 | scores = bboxes_intersection(tf.constant([0, 0, 1, 1], bboxes.dtype),
51 | bboxes)
52 | mask = scores > threshold
53 | if assign_negative: # 保留所有的label和框，重叠区不够的label置负
54 | labels = tf.where(mask, labels, -labels) # 交叉满足的标记为正，否则为负
55 | else: # 删除重叠区不够的label和框
56 | labels = tf.boolean_mask(labels, mask) # bool掩码，类似于array的bool切片
57 | bboxes = tf.boolean_mask(bboxes, mask)
58 | return labels, bboxes
59 |
60 |
61 | def bboxes_resize(bbox_ref, bboxes, name=None):
62 | """
63 | 使用新的参考点和基底长度（bbox_ref）重置bboxes的表示
64 | :param bbox_ref: 参考框，左上角点为新的参考点，hw为新的参考基
65 | :param bboxes: 目标框
66 | :param name: 域名
67 | :return: 目标框重新表示后的写法
68 | """
69 | # Tensors inputs.
70 | with tf.name_scope(name, 'bboxes_resize'):
71 | # Translate.
72 | # bbox_ref:['ymin', 'xmin', 'ymax', 'xmax']
73 | v = tf.stack([bbox_ref[0], bbox_ref[1], bbox_ref[0], bbox_ref[1]])
74 | bboxes = bboxes - v
75 | # Scale.
76 | s = tf.stack([bbox_ref[2] - bbox_ref[0], # h
77 | bbox_ref[3] - bbox_ref[1], # w
78 | bbox_ref[2] - bbox_ref[0],
79 | bbox_ref[3] - bbox_ref[1]])
80 | bboxes = bboxes / s
81 | return bboxes
82 |
83 |
84 | def distorted_bounding_box_crop(image,
85 | labels,
86 | bboxes,
87 | min_object_covered=0.3,
88 | aspect_ratio_range=(0.9, 1.1),
89 | area_range=(0.1, 1.0),
90 | max_attempts=200,
91 | scope=None):
92 | """Generates cropped_image using a one of the bboxes randomly distorted.
93 |
94 | See `tf.image.sample_distorted_bounding_box` for more documentation.
95 |
96 | Args:
97 | image: 3-D Tensor of image (it will be converted to floats in [0, 1]).
98 | bbox: 3-D float Tensor of bounding boxes arranged [1, num_boxes, coords]
99 | where each coordinate is [0, 1) and the coordinates are arranged
100 | as [ymin, xmin, ymax, xmax]. If num_boxes is 0 then it would use the whole
101 | image.
102 | min_object_covered: An optional `float`. Defaults to `0.1`. The cropped
103 | area of the image must contain at least this fraction of any bounding box
104 | supplied.
105 | aspect_ratio_range: An optional list of `floats`. The cropped area of the
106 | image must have an aspect ratio = width / height within this range.
107 | area_range: An optional list of `floats`. The cropped area of the image
108 | must contain a fraction of the supplied image within in this range.
109 | max_attempts: An optional `int`. Number of attempts at generating a cropped
110 | region of the image of the specified constraints. After `max_attempts`
111 | failures, return the entire image.
112 | scope: Optional scope for name_scope.
113 | Returns:
114 | A tuple, a 3-D Tensor cropped_image and the distorted bbox
115 | """
116 | with tf.name_scope(scope, 'distorted_bounding_box_crop', [image, bboxes]):
117 | # 高级的随机裁剪
118 | # The bounding box coordinates are floats in `[0.0, 1.0]` relative to the width
119 | # and height of the underlying image.
120 | # 1-D, 1-D, [1, 1, 4]
121 | bbox_begin, bbox_size, distort_bbox = tf.image.sample_distorted_bounding_box(
122 | tf.shape(image),
123 | bounding_boxes=tf.expand_dims(bboxes, 0), # [1, n, 4]
124 | min_object_covered=min_object_covered,
125 | aspect_ratio_range=aspect_ratio_range,
126 | area_range=area_range,
127 | max_attempts=max_attempts, # 最大尝试裁剪次数，失败则返回原图
128 | use_image_if_no_bounding_boxes=True)
129 | '''
130 | Returns:
131 | A tuple of `Tensor` objects (begin, size, bboxes).
132 |
133 | begin: A `Tensor`. Has the same type as `image_size`. 1-D, containing `[offset_height, offset_width, 0]`.
134 | Provide as input to `tf.slice`.
135 | size: A `Tensor`. Has the same type as `image_size`. 1-D, containing `[target_height, target_width, -1]`.
136 | Provide as input to `tf.slice`.
137 | bboxes: A `Tensor` of type `float32`. 3-D with shape `[1, 1, 4]` containing the distorted bounding box.
138 | Provide as input to `tf.image.draw_bounding_boxes`.
139 | '''
140 | # [4]，裁剪结果相对原图的(y, x, h, w)
141 | distort_bbox = distort_bbox[0, 0]
142 |
143 | # Crop the image to the specified bounding box.
144 | cropped_image = tf.slice(image, bbox_begin, bbox_size)
145 | # Restore the shape since the dynamic slice loses 3rd dimension.
146 | cropped_image.set_shape([None, None, 3]) # <-----设置了尺寸了哈
147 |
148 | # Update bounding boxes: resize and filter out.
149 | # 以裁剪子图为参考，将bboxes更换参考点和基长度
150 | bboxes = bboxes_resize(distort_bbox, bboxes) # [4], [n, 4]
151 | # 筛选变换后的bboxes和裁剪子图交集大于阈值的图bboxes
152 | labels, bboxes = bboxes_filter_overlap(labels, bboxes,
153 | threshold=0.5,
154 | assign_negative=False)
155 | # 返回随机裁剪的图片，筛选调整后的labels(n,)、bboxes(n, 4)，裁剪图片对应原图坐标(4,)
156 | return cropped_image, labels, bboxes, distort_bbox
157 |
158 |
159 | def preprocess_image(image, labels, bboxes, out_shape,
160 | scope='ssd_preprocessing_train'):
161 |
162 | with tf.name_scope(scope, 'ssd_preprocessing_train', [image, labels, bboxes]):
163 | if image.get_shape().ndims != 3:
164 | raise ValueError('Input must be of size [height, width, C>0]')
165 | # Convert to float scaled [0, 1].
166 | # 并不单单是float化，而是将255像素表示放缩为01表示
167 | if image.dtype != tf.float32:
168 | image = tf.image.convert_image_dtype(image, dtype=tf.float32)
169 |
170 | # （有条件的）随机裁剪，筛选调整后的labels(n,)、bboxes(n, 4)，裁剪图片对应原图坐标(4,)
171 | dst_image, labels, bboxes, distort_bbox = \
172 | distorted_bounding_box_crop(image, labels, bboxes,
173 | min_object_covered=0.25,
174 | aspect_ratio_range=(0.6, 1.67))
175 | # Resize image to output size.
176 | dst_image = util_tf.resize_image(dst_image, out_shape,
177 | method=tf.image.ResizeMethod.BILINEAR,
178 | align_corners=False)
179 |
180 | # Randomly flip the image horizontally.
181 | dst_image, bboxes = util_tf.random_flip_left_right(dst_image, bboxes)
182 |
183 | # Randomly distort the colors. There are 4 ways to do it.
184 | dst_image = util_tf.apply_with_random_selector(
185 | dst_image,
186 | lambda x, ordering: util_tf.distort_color(x, ordering, False),
187 | num_cases=4)
188 |
189 | # Rescale to VGG input scale.
190 | image = dst_image * 255.
191 | image = util_tf.tf_image_whitened(image)
192 | # mean = tf.constant(means, dtype=image.dtype)
193 | # image = image - mean
194 |
195 | # 'NHWC' (n,) (n, 4)
196 | return image, labels, bboxes
197 |
198 |
--------------------------------------------------------------------------------
/ssd_vgg300_tf.py:
--------------------------------------------------------------------------------
1 | import sys
2 | import math
3 | import numpy as np
4 | import tensorflow as tf
5 | from collections import namedtuple
6 |
7 | from util_tf import *
8 |
9 | slim = tf.contrib.slim
10 |
11 | SSDParams = namedtuple('SSDParameters', ['img_shape',
12 | 'num_classes',
13 | 'no_annotation_label',
14 | 'feat_layers',
15 | 'feat_shapes',
16 | 'anchor_size_bounds',
17 | 'anchor_sizes',
18 | 'anchor_ratios',
19 | 'anchor_steps',
20 | 'anchor_offset',
21 | 'normalizations',
22 | 'prior_scaling'
23 | ])
24 |
25 |
26 | class SSDNet(object):
27 | default_params = SSDParams(
28 | img_shape=(300, 300),
29 | num_classes=21,
30 | no_annotation_label=21,
31 | feat_layers=['block4', 'block7', 'block8', 'block9', 'block10', 'block11'],
32 | feat_shapes=[(38, 38), (19, 19), (10, 10), (5, 5), (3, 3), (1, 1)],
33 | anchor_size_bounds=[0.15, 0.90],
34 | anchor_sizes=[(21., 45.),
35 | (45., 99.),
36 | (99., 153.),
37 | (153., 207.),
38 | (207., 261.),
39 | (261., 315.)],
40 | anchor_ratios=[[2, .5],
41 | [2, .5, 3, 1. / 3],
42 | [2, .5, 3, 1. / 3],
43 | [2, .5, 3, 1. / 3],
44 | [2, .5],
45 | [2, .5]],
46 | anchor_steps=[8, 16, 32, 64, 100, 300],
47 | anchor_offset=0.5,
48 | normalizations=[1, -1, -1, -1, -1, -1], # 控制SSD层处理时是否预先沿着HW正则化
49 | prior_scaling=[0.1, 0.1, 0.2, 0.2]
50 | )
51 |
52 | def __init__(self, params=None):
53 | if isinstance(params, SSDParams):
54 | self.params = params
55 | else:
56 | self.params = SSDNet.default_params
57 |
58 | def net(self, input_data, weight_decay, update_feat_shapes=True, is_training=True):
59 | with slim.arg_scope(self._ssd_arg_scope(weight_decay)):
60 | output = self._ssd_net(input_data, is_training=is_training)
61 | # Update feature shapes (try at least!)
62 | if update_feat_shapes:
63 | feat_shapes = []
64 | # 获取各个中间层shape（不含0维），如果含有None则返回默认的feat_shapes
65 | for l in output[0]:
66 | if isinstance(l, np.ndarray):
67 | shape = l.shape
68 | else:
69 | shape = l.get_shape().as_list()
70 | shape = shape[1:4]
71 | if None in shape:
72 | feat_shapes = self.params.feat_shapes
73 | break
74 | else:
75 | feat_shapes.append(shape)
76 | self.params = self.params._replace(feat_shapes=feat_shapes)
77 | sys.stdout.write('[*] Report: variable feat_shapes is {}\n'.format(self.params.feat_shapes))
78 | return output
79 |
80 | @property
81 | def anchors(self):
82 | return self._ssd_anchors_all_layers(self.params.img_shape,
83 | self.params.feat_shapes,
84 | self.params.anchor_sizes,
85 | self.params.anchor_ratios,
86 | self.params.anchor_steps, # [8, 16, 32, 64, 100, 300]
87 | self.params.anchor_offset # 0.5
88 | )
89 |
90 | def _ssd_net(self, inputs,
91 | scope='ssd_net',
92 | reuse=False,
93 | is_training=True,
94 | dropout_keep_prob=0.5):
95 | with tf.variable_scope(scope, 'ssd_net', [inputs], reuse=reuse) as sc:
96 | end_points_collection = sc.original_name_scope + '_end_points'
97 | # Collect outputs for conv2d, fully_connected and max_pool2d.
98 | with slim.arg_scope(
99 | [slim.conv2d, slim.max_pool2d],
100 | outputs_collections=end_points_collection):
101 | end_points = {}
102 | # ——————————————————Original VGG-16 blocks.———————————————————
103 | # Block 1.
104 | net = slim.repeat(inputs, 2, slim.conv2d, 64, [3, 3], scope='conv1')
105 | end_points['block1'] = net
106 | net = slim.max_pool2d(net, [2, 2], scope='pool1')
107 | # Block 2.
108 | net = slim.repeat(net, 2, slim.conv2d, 128, [3, 3], scope='conv2')
109 | end_points['block2'] = net
110 | net = slim.max_pool2d(net, [2, 2], scope='pool2')
111 | # Block 3.
112 | net = slim.repeat(net, 3, slim.conv2d, 256, [3, 3], scope='conv3')
113 | end_points['block3'] = net
114 | net = slim.max_pool2d(net, [2, 2], scope='pool3')
115 | # Block 4.
116 | net = slim.repeat(net, 3, slim.conv2d, 512, [3, 3], scope='conv4')
117 | end_points['block4'] = net
118 | net = slim.max_pool2d(net, [2, 2], scope='pool4')
119 | # Block 5.
120 | net = slim.repeat(net, 3, slim.conv2d, 512, [3, 3], scope='conv5')
121 | end_points['block5'] = net
122 | net = slim.max_pool2d(net, [3, 3], stride=1, scope='pool5')
123 | # ————————————Additional SSD blocks.——————————————————————
124 | # Block 6
125 | net = slim.conv2d(net, 1024, [3, 3], rate=6, scope='conv6')
126 | end_points['block6'] = net
127 | net = tf.layers.dropout(net, rate=dropout_keep_prob, training=is_training)
128 | # Block 7
129 | net = slim.conv2d(net, 1024, [1, 1], scope='conv7')
130 | end_points['block7'] = net
131 | net = tf.layers.dropout(net, rate=dropout_keep_prob, training=is_training)
132 | # Block 8/9/10/11: 1x1 and 3x3 convolutions stride 2 (except lasts).
133 | end_point = 'block8'
134 | with tf.variable_scope(end_point):
135 | net = slim.conv2d(net, 256, [1, 1], scope='conv1x1')
136 | net = tf.pad(net, ([0, 0], [1, 1], [1, 1], [0, 0]), mode='CONSTANT')
137 | net = slim.conv2d(net, 512, [3, 3], stride=2, scope='conv3x3', padding='VALID')
138 | end_points[end_point] = net
139 | end_point = 'block9'
140 | with tf.variable_scope(end_point):
141 | net = slim.conv2d(net, 128, [1, 1], scope='conv1x1')
142 | net = tf.pad(net, ([0, 0], [1, 1], [1, 1], [0, 0]), mode='CONSTANT')
143 | net = slim.conv2d(net, 256, [3, 3], stride=2, scope='conv3x3', padding='VALID')
144 | end_points[end_point] = net
145 | end_point = 'block10'
146 | with tf.variable_scope(end_point):
147 | net = slim.conv2d(net, 128, [1, 1], scope='conv1x1')
148 | net = slim.conv2d(net, 256, [3, 3], scope='conv3x3', padding='VALID')
149 | end_points[end_point] = net
150 | end_point = 'block11'
151 | with tf.variable_scope(end_point):
152 | net = slim.conv2d(net, 128, [1, 1], scope='conv1x1')
153 | net = slim.conv2d(net, 256, [3, 3], scope='conv3x3', padding='VALID')
154 | end_points[end_point] = net
155 |
156 | predictions = []
157 | logits = []
158 | localisations = []
159 | # 对于每一feat层进行特征输出
160 | for i, layer in enumerate(self.default_params.feat_layers):
161 | with tf.variable_scope(layer + '_box'):
162 | p, l = self._ssd_multibox_layer(end_points[layer], # <-----SSD处理
163 | self.params.num_classes,
164 | self.params.anchor_sizes[i],
165 | self.params.anchor_ratios[i],
166 | self.params.normalizations[i])
167 | predictions.append(slim.softmax(p)) # prediction_fn=slim.softmax
168 | logits.append(p)
169 | localisations.append(l)
170 |
171 | # import pprint as pp
172 | # from tensorflow.contrib.layers.python.layers import utils
173 | # pp.pprint(end_points)
174 | end_points_total = slim.utils.convert_collection_to_dict(end_points_collection)
175 | return predictions, localisations, logits, end_points
176 |
177 | @staticmethod
178 | def _ssd_arg_scope(weight_decay=0.0005):
179 | with slim.arg_scope([slim.conv2d, slim.fully_connected],
180 | activation_fn=tf.nn.relu,
181 | weights_regularizer=slim.l2_regularizer(weight_decay),
182 | weights_initializer=tf.contrib.layers.xavier_initializer(),
183 | biases_initializer=tf.zeros_initializer()):
184 | with slim.arg_scope([slim.conv2d, slim.max_pool2d],
185 | padding='SAME') as sc:
186 | return sc
187 |
188 | @staticmethod
189 | def _ssd_multibox_layer(net,
190 | num_classes,
191 | sizes,
192 | ratios=(1,),
193 | normalization=-1):
194 | # l2 normalize layer
195 | if normalization > 0:
196 | scale = tf.Variable(dtype=tf.float32, initial_value=tf.ones(shape=(net.get_shape()[-1],)), trainable=True)
197 | net = tf.multiply(tf.nn.l2_normalize(net, net.get_shape().ndims-1, epsilon=1e-12), scale)
198 |
199 | # Number of anchors.
200 | num_anchors = len(sizes) + len(ratios)
201 |
202 | # Location.
203 | num_loc_pred = num_anchors * 4 # 每一个框有四个坐标
204 | loc_pred = slim.conv2d(net, num_loc_pred, [3, 3], activation_fn=None,
205 | scope='conv_loc') # 输出C表示不同框的某个坐标
206 | loc_shape = tensor_shape(loc_pred, rank=4)
207 | loc_pred = tf.reshape(loc_pred, loc_shape[0:-1]+[loc_shape[-1]//4, 4])
208 |
209 | # Class prediction.
210 | num_cls_pred = num_anchors * num_classes # 每一个框都要计算所有的类别
211 | cls_pred = slim.conv2d(net, num_cls_pred, [3, 3], activation_fn=None,
212 | scope='conv_cls') # 输出C表示不同框的对某个类的预测
213 | cls_shape = tensor_shape(cls_pred, rank=4)
214 | cls_pred = tf.reshape(cls_pred, cls_shape[0:-1] + [cls_shape[-1] // num_classes, num_classes])
215 | return cls_pred, loc_pred
216 |
217 | @staticmethod
218 | def _ssd_anchors_all_layers(img_shape,
219 | layers_shape,
220 | anchor_sizes,
221 | anchor_ratios,
222 | anchor_steps, # [8, 16, 32, 64, 100, 300]
223 | offset=0.5,
224 | dtype=np.float32):
225 | layers_anchors = []
226 | for i, feat_shape in enumerate(layers_shape):
227 | # anchor_bboxes = ssd_anchor_one_layer(img_shape,
228 | # feat_shape,
229 | # anchor_sizes[i],
230 | # anchor_ratios[i],
231 | # anchor_steps[i],
232 | # offset=offset, dtype=dtype)
233 | # layers_anchors.append(anchor_bboxes)
234 | # 生成feat_shape中HW对应的网格坐标
235 | y, x = np.mgrid[0:feat_shape[0], 0:feat_shape[1]]
236 | # step*feat_shape 约等于img_shape，这使得网格点坐标介于0~1，放缩一下即可到图像大小
237 | y = (y.astype(dtype) + offset) * anchor_steps[i] / img_shape[0]
238 | x = (x.astype(dtype) + offset) * anchor_steps[i] / img_shape[1]
239 |
240 | # Expand dims to support easy broadcasting.
241 | y = np.expand_dims(y, axis=-1)
242 | x = np.expand_dims(x, axis=-1)
243 |
244 | # Compute relative height and width.
245 | # Tries to follow the original implementation of SSD for the order.
246 | num_anchors = len(anchor_sizes[i]) + len(anchor_ratios[i])
247 | h = np.zeros((num_anchors,), dtype=dtype)
248 | w = np.zeros((num_anchors,), dtype=dtype)
249 | # Add first anchor boxes with ratio=1.
250 | h[0] = anchor_sizes[i][0] / img_shape[0]
251 | w[0] = anchor_sizes[i][0] / img_shape[1]
252 | di = 1
253 | if len(anchor_sizes[i]) > 1:
254 | h[1] = math.sqrt(anchor_sizes[i][0] * anchor_sizes[i][1]) / img_shape[0]
255 | w[1] = math.sqrt(anchor_sizes[i][0] * anchor_sizes[i][1]) / img_shape[1]
256 | di += 1
257 | for l, r in enumerate(anchor_ratios[i]):
258 | h[l + di] = anchor_sizes[l][0] / img_shape[0] / math.sqrt(r)
259 | w[l + di] = anchor_sizes[l][0] / img_shape[1] * math.sqrt(r)
260 | layers_anchors.append((y, x, h, w))
261 | return layers_anchors
262 |
263 | # 绘制各层中心点示意
264 | # import matplotlib.pyplot as plt
265 | # plt.scatter(y, x, c='r', marker='.')
266 | # plt.grid(True)
267 | # plt.show()
268 | # print(h, w)
269 |
270 | def bboxes_encode(self, labels, bboxes, anchors, scope=None):
271 | return tf_ssd_bboxes_encode(
272 | labels, bboxes, anchors,
273 | self.params.num_classes,
274 | self.params.no_annotation_label, # 21
275 | ignore_threshold=0.5,
276 | prior_scaling=self.params.prior_scaling,
277 | scope=scope)
278 |
279 | @staticmethod
280 | def losses(logits, localisations,
281 | gclasses, glocalisations, gscores,
282 | match_threshold=0.5,
283 | negative_ratio=3.,
284 | alpha=1.,
285 | label_smoothing=0.,
286 | scope='ssd_losses'):
287 | """Define the SSD network losses.
288 | """
289 | return ssd_losses(logits, localisations,
290 | gclasses, glocalisations, gscores,
291 | match_threshold=match_threshold,
292 | negative_ratio=negative_ratio,
293 | alpha=alpha,
294 | label_smoothing=label_smoothing,
295 | scope=scope)
296 |
297 |
298 | def tf_ssd_bboxes_encode(labels,
299 | bboxes,
300 | anchors,
301 | num_classes,
302 | no_annotation_label,
303 | ignore_threshold=0.5,
304 | prior_scaling=(0.1, 0.1, 0.2, 0.2),
305 | dtype=tf.float32,
306 | scope='ssd_bboxes_encode'):
307 | with tf.name_scope(scope):
308 | target_labels = []
309 | target_localizations = []
310 | target_scores = []
311 | # anchors_layer: (y, x, h, w)
312 | for i, anchors_layer in enumerate(anchors):
313 | with tf.name_scope('bboxes_encode_block_%i' % i):
314 | # (m,m,k)，xywh(m,m,4k)，(m,m,k)
315 | t_labels, t_loc, t_scores = \
316 | tf_ssd_bboxes_encode_layer(labels, bboxes, anchors_layer,
317 | num_classes, no_annotation_label,
318 | ignore_threshold,
319 | prior_scaling, dtype)
320 | target_labels.append(t_labels)
321 | target_localizations.append(t_loc)
322 | target_scores.append(t_scores)
323 | return target_labels, target_localizations, target_scores
324 |
325 |
326 | # 为了有助理解，m表示该层中心点行列数，k为每个中心点对应的框数，n为图像上的目标数
327 | def tf_ssd_bboxes_encode_layer(labels, # (n,)
328 | bboxes, # (n, 4)
329 | anchors_layer, # y(m, m, 1), x(m, m, 1), h(k,), w(k,)
330 | num_classes,
331 | no_annotation_label,
332 | ignore_threshold=0.5,
333 | prior_scaling=(0.1, 0.1, 0.2, 0.2),
334 | dtype=tf.float32):
335 | """Encode groundtruth labels and bounding boxes using SSD anchors from
336 | one layer.
337 |
338 | Arguments:
339 | labels: 1D Tensor(int64) containing groundtruth labels;
340 | bboxes: Nx4 Tensor(float) with bboxes relative coordinates;
341 | anchors_layer: Numpy array with layer anchors;
342 | matching_threshold: Threshold for positive match with groundtruth bboxes;
343 | prior_scaling: Scaling of encoded coordinates.
344 |
345 | Return:
346 | (target_labels, target_localizations, target_scores): Target Tensors.
347 | """
348 | # Anchors coordinates and volume.
349 | yref, xref, href, wref = anchors_layer # y(m, m, 1), x(m, m, 1), h(k,), w(k,)
350 | ymin = yref - href / 2. # (m, m, k)
351 | xmin = xref - wref / 2.
352 | ymax = yref + href / 2.
353 | xmax = xref + wref / 2.
354 | vol_anchors = (xmax - xmin) * (ymax - ymin) # 搜索框面积(m, m, k)
355 |
356 | # Initialize tensors...
357 | # 下面各个Tensor矩阵的shape等于中心点坐标矩阵的shape
358 | shape = (yref.shape[0], yref.shape[1], href.size) # (m, m, k)
359 | feat_labels = tf.zeros(shape, dtype=tf.int64) # (m, m, k)
360 | feat_scores = tf.zeros(shape, dtype=dtype)
361 |
362 | feat_ymin = tf.zeros(shape, dtype=dtype)
363 | feat_xmin = tf.zeros(shape, dtype=dtype)
364 | feat_ymax = tf.ones(shape, dtype=dtype)
365 | feat_xmax = tf.ones(shape, dtype=dtype)
366 |
367 | def jaccard_with_anchors(bbox):
368 | """Compute jaccard score between a box and the anchors.
369 | """
370 | int_ymin = tf.maximum(ymin, bbox[0]) # (m, m, k)
371 | int_xmin = tf.maximum(xmin, bbox[1])
372 | int_ymax = tf.minimum(ymax, bbox[2])
373 | int_xmax = tf.minimum(xmax, bbox[3])
374 | h = tf.maximum(int_ymax - int_ymin, 0.)
375 | w = tf.maximum(int_xmax - int_xmin, 0.)
376 | # Volumes.
377 | # 处理搜索框和bbox之间的联系
378 | inter_vol = h * w # 交集面积
379 | union_vol = vol_anchors - inter_vol \
380 | + (bbox[2] - bbox[0]) * (bbox[3] - bbox[1]) # 并集面积
381 | jaccard = tf.div(inter_vol, union_vol) # 交集/并集，即IOU
382 | return jaccard # (m, m, k)
383 |
384 | def condition(i, feat_labels, feat_scores,
385 | feat_ymin, feat_xmin, feat_ymax, feat_xmax):
386 | """Condition: check label index.
387 | """
388 | r = tf.less(i, tf.shape(labels))
389 | return r[0] # tf.shape(labels)有维度，所以r有维度
390 |
391 | def body(i, feat_labels, feat_scores,
392 | feat_ymin, feat_xmin, feat_ymax, feat_xmax):
393 | """Body: update feature labels, scores and bboxes.
394 | Follow the original SSD paper for that purpose:
395 | - assign values when jaccard > 0.5;
396 | - only update if beat the score of other bboxes.
397 | """
398 | # Jaccard score.
399 | label = labels[i] # 当前图片上第i个对象的标签
400 | bbox = bboxes[i] # 当前图片上第i个对象的真实框bbox
401 | jaccard = jaccard_with_anchors(bbox) # 当前对象的bbox和当前层的搜索网格IOU，(m, m, k)
402 | # Mask: check threshold + scores + no annotations + num_classes.
403 | mask = tf.greater(jaccard, feat_scores) # 掩码矩阵，IOU大于历史得分的为True，(m, m, k)
404 | # mask = tf.logical_and(mask, tf.greater(jaccard, matching_threshold))
405 | mask = tf.logical_and(mask, feat_scores > -0.5)
406 | mask = tf.logical_and(mask, label < num_classes) # 不太懂，label应该必定小于类别数
407 | imask = tf.cast(mask, tf.int64) # 整形mask
408 | fmask = tf.cast(mask, dtype) # 浮点型mask
409 |
410 | # Update values using mask.
411 | # 保证feat_labels存储对应位置得分最大对象标签，feat_scores存储那个得分
412 | # (m, m, k) × 当前类别scalar + (1 - (m, m, k)) × (m, m, k)
413 | # 更新label记录，此时的imask已经保证了True位置当前对像得分高于之前的对象得分，其他位置值不变
414 | feat_labels = imask * label + (1 - imask) * feat_labels
415 | # 更新score记录，mask为True使用本类别IOU，否则不变
416 | feat_scores = tf.where(mask, jaccard, feat_scores)
417 |
418 | # 下面四个矩阵存储对应label的真实框坐标
419 | # (m, m, k) × 当前框坐标scalar + (1 - (m, m, k)) × (m, m, k)
420 | feat_ymin = fmask * bbox[0] + (1 - fmask) * feat_ymin
421 | feat_xmin = fmask * bbox[1] + (1 - fmask) * feat_xmin
422 | feat_ymax = fmask * bbox[2] + (1 - fmask) * feat_ymax
423 | feat_xmax = fmask * bbox[3] + (1 - fmask) * feat_xmax
424 |
425 | return [i + 1, feat_labels, feat_scores,
426 | feat_ymin, feat_xmin, feat_ymax, feat_xmax]
427 |
428 | # Main loop definition.
429 | # 对当前图像上每一个目标进行循环
430 | i = 0
431 | (i,
432 | feat_labels, feat_scores,
433 | feat_ymin, feat_xmin,
434 | feat_ymax, feat_xmax) = tf.while_loop(condition, body,
435 | [i,
436 | feat_labels, feat_scores,
437 | feat_ymin, feat_xmin,
438 | feat_ymax, feat_xmax])
439 | # Transform to center / size.
440 | # 这里的y、x、h、w指的是对应位置所属真实框的相关属性
441 | feat_cy = (feat_ymax + feat_ymin) / 2.
442 | feat_cx = (feat_xmax + feat_xmin) / 2.
443 | feat_h = feat_ymax - feat_ymin
444 | feat_w = feat_xmax - feat_xmin
445 |
446 | # Encode features.
447 | # prior_scaling: [0.1, 0.1, 0.2, 0.2]，放缩意义不明
448 | # ((m, m, k) - (m, m, 1)) / (k,) * 10
449 | # 以搜索网格中心点为参考，真实框中心的偏移，单位长度为网格hw
450 | feat_cy = (feat_cy - yref) / href / prior_scaling[0]
451 | feat_cx = (feat_cx - xref) / wref / prior_scaling[1]
452 | # log((m, m, k) / (m, m, 1)) * 5
453 | # 真实框宽高/搜索网格宽高，取对
454 | feat_h = tf.log(feat_h / href) / prior_scaling[2]
455 | feat_w = tf.log(feat_w / wref) / prior_scaling[3]
456 | # Use SSD ordering: x / y / w / h instead of ours.(m, m, k, 4)
457 | feat_localizations = tf.stack([feat_cx, feat_cy, feat_w, feat_h], axis=-1) # -1会扩维，故有4
458 |
459 | return feat_labels, feat_localizations, feat_scores
460 |
461 |
462 | def ssd_losses(logits, localisations, # 预测类别，位置
463 | gclasses, glocalisations, gscores, # ground truth类别，位置，得分
464 | match_threshold=0.5,
465 | negative_ratio=3.,
466 | alpha=1.,
467 | label_smoothing=0.,
468 | scope=None):
469 | with tf.name_scope(scope, 'ssd_losses'):
470 | # 提取类别数和batch_size
471 | lshape = tensor_shape(logits[0], 5) # tensor_shape函数可以取代
472 | num_classes = lshape[-1]
473 | batch_size = lshape[0]
474 |
475 | # Flatten out all vectors!
476 | flogits = []
477 | fgclasses = []
478 | fgscores = []
479 | flocalisations = []
480 | fglocalisations = []
481 | for i in range(len(logits)): # 按照图片循环
482 | flogits.append(tf.reshape(logits[i], [-1, num_classes]))
483 | fgclasses.append(tf.reshape(gclasses[i], [-1]))
484 | fgscores.append(tf.reshape(gscores[i], [-1]))
485 | flocalisations.append(tf.reshape(localisations[i], [-1, 4]))
486 | fglocalisations.append(tf.reshape(glocalisations[i], [-1, 4]))
487 | # And concat the crap!
488 | logits = tf.concat(flogits, axis=0) # 全部的搜索框，对应的21类别的输出
489 | gclasses = tf.concat(fgclasses, axis=0) # 全部的搜索框，真实的类别数字
490 | gscores = tf.concat(fgscores, axis=0) # 全部的搜索框，和真实框的IOU
491 | localisations = tf.concat(flocalisations, axis=0)
492 | glocalisations = tf.concat(fglocalisations, axis=0)
493 |
494 | dtype = logits.dtype
495 | pmask = gscores > match_threshold # (全部搜索框数目, 21)，类别搜索框和真实框IOU大于阈值
496 | fpmask = tf.cast(pmask, dtype) # 浮点型前景掩码（前景假定为含有对象的IOU足够的搜索框标号）
497 | n_positives = tf.reduce_sum(fpmask) # 前景总数
498 |
499 | # Hard negative mining...
500 | no_classes = tf.cast(pmask, tf.int32)
501 | predictions = slim.softmax(logits) # 此时每一行的21个数转化为概率
502 | nmask = tf.logical_and(tf.logical_not(pmask),
503 | gscores > -0.5) # IOU达不到阈值的类别搜索框位置记1
504 | fnmask = tf.cast(nmask, dtype)
505 | nvalues = tf.where(nmask,
506 | predictions[:, 0], # 框内无物体标记为背景预测概率
507 | 1. - fnmask) # 框内有物体位置标记为1
508 | nvalues_flat = tf.reshape(nvalues, [-1])
509 |
510 | # Number of negative entries to select.
511 | # 在nmask中剔除n_neg个最不可能背景点(对应的class0概率最低)
512 | max_neg_entries = tf.cast(tf.reduce_sum(fnmask), tf.int32)
513 | # 3 × 前景掩码数量 + batch_size
514 | n_neg = tf.cast(negative_ratio * n_positives, tf.int32) + batch_size
515 | n_neg = tf.minimum(n_neg, max_neg_entries)
516 | val, idxes = tf.nn.top_k(-nvalues_flat, k=n_neg) # 最不可能为背景的n_neg个点
517 | max_hard_pred = -val[-1]
518 | # Final negative mask.
519 | nmask = tf.logical_and(nmask, nvalues < max_hard_pred) # 不是前景，又最不像背景的n_neg个点
520 | fnmask = tf.cast(nmask, dtype)
521 |
522 | # Add cross-entropy loss.
523 | with tf.name_scope('cross_entropy_pos'):
524 | loss = tf.nn.sparse_softmax_cross_entropy_with_logits(logits=logits,
525 | labels=gclasses) # 0-20
526 | loss = tf.div(tf.reduce_sum(loss * fpmask), batch_size, name='value')
527 | tf.losses.add_loss(loss)
528 |
529 | with tf.name_scope('cross_entropy_neg'):
530 | loss = tf.nn.sparse_softmax_cross_entropy_with_logits(logits=logits,
531 | labels=no_classes) # {0,1}
532 | loss = tf.div(tf.reduce_sum(loss * fnmask), batch_size, name='value')
533 | tf.losses.add_loss(loss)
534 |
535 | # Add localization loss: smooth L1, L2, ...
536 | with tf.name_scope('localization'):
537 | # Weights Tensor: positive mask + random negative.
538 | weights = tf.expand_dims(alpha * fpmask, axis=-1)
539 | loss = abs_smooth(localisations - glocalisations)
540 | loss = tf.div(tf.reduce_sum(loss * weights), batch_size, name='value')
541 | tf.losses.add_loss(loss)
542 |
543 |
544 | if __name__ == '__main__':
545 | ssd = SSDNet()
546 | ssd.net(tf.placeholder(dtype=tf.float32, shape=[5, 304, 304, 3]))
547 |
548 |
--------------------------------------------------------------------------------
/tfr_data_process.py:
--------------------------------------------------------------------------------
1 | import os
2 | import tensorflow as tf
3 |
4 | slim = tf.contrib.slim
5 |
6 |
7 | def get_split(tfr_path, tfr_pattren, num_classes=21, num_samples=17125):
8 | # ===============TFR文件名匹配模板===============
9 | tfr_pattren = os.path.join(tfr_path, tfr_pattren)
10 |
11 | # =========阅读器=========
12 | reader = tf.TFRecordReader
13 |
14 | # ===================解码器===================
15 | keys_to_features = { # 解码TFR文件方式
16 | 'image/encoded': tf.FixedLenFeature((), tf.string, default_value=''),
17 | 'image/format': tf.FixedLenFeature((), tf.string, default_value='jpeg'),
18 | 'image/object/bbox/xmin': tf.VarLenFeature(dtype=tf.float32),
19 | 'image/object/bbox/ymin': tf.VarLenFeature(dtype=tf.float32),
20 | 'image/object/bbox/xmax': tf.VarLenFeature(dtype=tf.float32),
21 | 'image/object/bbox/ymax': tf.VarLenFeature(dtype=tf.float32),
22 | 'image/object/bbox/label': tf.VarLenFeature(dtype=tf.int64),
23 | }
24 | items_to_handlers = { # 解码二进制数据
25 | # 图像解码设置蛮有意思的
26 | 'image': slim.tfexample_decoder.Image('image/encoded', 'image/format'),
27 | 'object/bbox': slim.tfexample_decoder.BoundingBox(
28 | ['ymin', 'xmin', 'ymax', 'xmax'], 'image/object/bbox/'),
29 | 'object/label': slim.tfexample_decoder.Tensor('image/object/bbox/label'),
30 | }
31 | decoder = slim.tfexample_decoder.TFExampleDecoder(keys_to_features, items_to_handlers)
32 |
33 | # =======描述字段=======
34 | items_to_descriptions = {
35 | 'image': 'A color image of varying height and width.',
36 | 'shape': 'Shape of the image',
37 | 'object/bbox': 'A list of bounding boxes, one per each object.',
38 | 'object/label': 'A list of labels, one per each object.',
39 | }
40 |
41 | return slim.dataset.Dataset(
42 | data_sources=tfr_pattren, # TFR文件名
43 | reader=reader, # 阅读器
44 | decoder=decoder, # 解码器
45 | num_samples=num_samples, # 数目
46 | items_to_descriptions=items_to_descriptions, # decoder条目描述字段
47 | num_classes=num_classes, # 类别数
48 | labels_to_names=None # 字典{图片:类别,……}
49 | )
50 |
51 |
52 | def tfr_read(dataset):
53 | # 涉及队列操作，本部使用CPU设备
54 | provider = slim.dataset_data_provider.DatasetDataProvider(
55 | dataset, # DatasetDataProvider 需要 slim.dataset.Dataset 做参数
56 | num_readers=2,
57 | common_queue_capacity=20 * 5,
58 | common_queue_min=10 * 5,
59 | shuffle=True)
60 | image, glabels, gbboxes = provider.get(['image',
61 | 'object/label',
62 | 'object/bbox'])
63 | return image, glabels, gbboxes
64 |
65 |
--------------------------------------------------------------------------------
/tfr_generate.ipynb:
--------------------------------------------------------------------------------
1 | {
2 | "cells": [
3 | {
4 | "cell_type": "markdown",
5 | "metadata": {},
6 | "source": [
7 | "## 数据格式转化脚本\n",
8 | "\n",
9 | "#### 将voc目标检测数据转换为TFRecord格式，方便TensorFlow读取"
10 | ]
11 | },
12 | {
13 | "cell_type": "code",
14 | "execution_count": 1,
15 | "metadata": {
16 | "collapsed": false,
17 | "scrolled": true
18 | },
19 | "outputs": [
20 | {
21 | "name": "stderr",
22 | "output_type": "stream",
23 | "text": [
24 | "/home/hellcat/anaconda3/lib/python3.6/importlib/_bootstrap.py:219: RuntimeWarning: compiletime version 3.5 of module 'tensorflow.python.framework.fast_tensor_util' does not match runtime version 3.6\n",
25 | " return f(*args, **kwds)\n"
26 | ]
27 | },
28 | {
29 | "name": "stdout",
30 | "output_type": "stream",
31 | "text": [
32 | "Number of images is 17125"
33 | ]
34 | }
35 | ],
36 | "source": [
37 | "import os\n",
38 | "import sys\n",
39 | "import random\n",
40 | "from math import ceil\n",
41 | "import tensorflow as tf\n",
42 | "import xml.etree.ElementTree as ET\n",
43 | "\n",
44 | "TFR_NAME = './TFR_Data/voc2012'\n",
45 | "IMAGE_PATH = './VOC2012/JPEGImages'\n",
46 | "ANNOTATION_PATH = './VOC2012/Annotations'\n",
47 | "SAMPLES_PER_FILES = 2000\n",
48 | "VOC_LABELS = {\n",
49 | " 'none': (0, 'Background'),\n",
50 | " 'aeroplane': (1, 'Vehicle'),\n",
51 | " 'bicycle': (2, 'Vehicle'),\n",
52 | " 'bird': (3, 'Animal'),\n",
53 | " 'boat': (4, 'Vehicle'),\n",
54 | " 'bottle': (5, 'Indoor'),\n",
55 | " 'bus': (6, 'Vehicle'),\n",
56 | " 'car': (7, 'Vehicle'),\n",
57 | " 'cat': (8, 'Animal'),\n",
58 | " 'chair': (9, 'Indoor'),\n",
59 | " 'cow': (10, 'Animal'),\n",
60 | " 'diningtable': (11, 'Indoor'),\n",
61 | " 'dog': (12, 'Animal'),\n",
62 | " 'horse': (13, 'Animal'),\n",
63 | " 'motorbike': (14, 'Vehicle'),\n",
64 | " 'person': (15, 'Person'),\n",
65 | " 'pottedplant': (16, 'Indoor'),\n",
66 | " 'sheep': (17, 'Animal'),\n",
67 | " 'sofa': (18, 'Indoor'),\n",
68 | " 'train': (19, 'Vehicle'),\n",
69 | " 'tvmonitor': (20, 'Indoor'),\n",
70 | "}\n",
71 | "tfr_dir = os.path.split(TFR_NAME)[0]\n",
72 | "if not os.path.exists(tfr_dir):\n",
73 | " os.makedirs(tfr_dir)\n",
74 | "if not os.path.exists(IMAGE_PATH):\n",
75 | " raise BaseException('file {} is not exists'.format(IMAGE_PATH))\n",
76 | "file_names = sorted(os.listdir(IMAGE_PATH))\n",
77 | "random.seed = 10\n",
78 | "random.shuffle(file_names)\n",
79 | "sys.stdout.write('Number of images is {}'.format(len(file_names)))"
80 | ]
81 | },
82 | {
83 | "cell_type": "code",
84 | "execution_count": 2,
85 | "metadata": {
86 | "collapsed": true
87 | },
88 | "outputs": [],
89 | "source": [
90 | "def int64_feature(value):\n",
91 | " \"\"\"Wrapper for inserting int64 features into Example proto.\n",
92 | " \"\"\"\n",
93 | " if not isinstance(value, list):\n",
94 | " value = [value]\n",
95 | " return tf.train.Feature(int64_list=tf.train.Int64List(value=value))\n",
96 | " \n",
97 | "def float_feature(value):\n",
98 | " \"\"\"Wrapper for inserting float features into Example proto.\n",
99 | " \"\"\"\n",
100 | " if not isinstance(value, list):\n",
101 | " value = [value]\n",
102 | " return tf.train.Feature(float_list=tf.train.FloatList(value=value))\n",
103 | " \n",
104 | "def bytes_feature(value):\n",
105 | " \"\"\"Wrapper for inserting bytes features into Example proto.\n",
106 | " \"\"\"\n",
107 | " if not isinstance(value, list):\n",
108 | " value = [value]\n",
109 | " return tf.train.Feature(bytes_list=tf.train.BytesList(value=value))\n",
110 | "\n",
111 | "def xml_parse(xml_path):\n",
112 | " tree = ET.parse(xml_path)\n",
113 | " root = tree.getroot()\n",
114 | " # Image shape.\n",
115 | " size = root.find('size')\n",
116 | " shape = [int(size.find('height').text),\n",
117 | " int(size.find('width').text),\n",
118 | " int(size.find('depth').text)]\n",
119 | " # Find annotations.\n",
120 | " bboxes = []\n",
121 | " labels = []\n",
122 | " labels_text = []\n",
123 | " difficult = []\n",
124 | " truncated = []\n",
125 | " for obj in root.findall('object'):\n",
126 | " label = obj.find('name').text\n",
127 | " labels.append(VOC_LABELS[label][0])\n",
128 | " \n",
129 | " if obj.find('difficult'):\n",
130 | " difficult.append(int(obj.find('difficult').text))\n",
131 | " else:\n",
132 | " difficult.append(0)\n",
133 | " if obj.find('truncated'):\n",
134 | " truncated.append(int(obj.find('truncated').text))\n",
135 | " else:\n",
136 | " truncated.append(0)\n",
137 | " bbox = obj.find('bndbox')\n",
138 | " bboxes.append((float(bbox.find('ymin').text) / shape[0],\n",
139 | " float(bbox.find('xmin').text) / shape[1],\n",
140 | " float(bbox.find('ymax').text) / shape[0],\n",
141 | " float(bbox.find('xmax').text) / shape[1]\n",
142 | " ))\n",
143 | " return shape, bboxes, labels, labels_text, difficult, truncated"
144 | ]
145 | },
146 | {
147 | "cell_type": "code",
148 | "execution_count": 3,
149 | "metadata": {
150 | "collapsed": false
151 | },
152 | "outputs": [
153 | {
154 | "name": "stdout",
155 | "output_type": "stream",
156 | "text": [
157 | "Writing file './TFR_Data/voc2012_000.tfrecord'......\n",
158 | "Writing file './TFR_Data/voc2012_001.tfrecord'......\n",
159 | "Writing file './TFR_Data/voc2012_002.tfrecord'......\n",
160 | "Writing file './TFR_Data/voc2012_003.tfrecord'......\n",
161 | "Writing file './TFR_Data/voc2012_004.tfrecord'......\n",
162 | "Writing file './TFR_Data/voc2012_005.tfrecord'......\n",
163 | "Writing file './TFR_Data/voc2012_006.tfrecord'......\n",
164 | "Writing file './TFR_Data/voc2012_007.tfrecord'......\n",
165 | "Writing file './TFR_Data/voc2012_008.tfrecord'......\n"
166 | ]
167 | }
168 | ],
169 | "source": [
170 | "num_tfr = ceil(len(file_names)/SAMPLES_PER_FILES)\n",
171 | "i = 0\n",
172 | "for idx in range(num_tfr):\n",
173 | " tfr_file = '{}_{:03d}.tfrecord'.format(TFR_NAME, idx)\n",
174 | " sys.stdout.write(\"Writing file '{}'......\\n\".format(tfr_file))\n",
175 | " # 建立书写器\n",
176 | " with tf.python_io.TFRecordWriter(tfr_file) as writer:\n",
177 | " while i < SAMPLES_PER_FILES * (idx + 1) and i < len(file_names):\n",
178 | " xml_file = os.path.join(ANNOTATION_PATH, \n",
179 | " file_names[i].strip('.jpg') + '.xml')\n",
180 | " image_file = os.path.join(IMAGE_PATH, file_names[i])\n",
181 | " _, box, label, _, _, _ = xml_parse(xml_file)\n",
182 | " image_data = tf.gfile.FastGFile(image_file, 'rb').read()\n",
183 | " i += 1\n",
184 | " \n",
185 | " xmin, ymin, xmax, ymax = ([] for _ in range(4))\n",
186 | " for b in box:\n",
187 | " assert len(b) == 4\n",
188 | " [coord.append(point) for coord, point in zip([ymin, xmin, ymax, xmax], b)]\n",
189 | " image_format = b'JPEG'\n",
190 | " # 建立example\n",
191 | " example = tf.train.Example(features=tf.train.Features(feature={\n",
192 | " 'image/object/bbox/xmin': float_feature(xmin),\n",
193 | " 'image/object/bbox/xmax': float_feature(xmax),\n",
194 | " 'image/object/bbox/ymin': float_feature(ymin),\n",
195 | " 'image/object/bbox/ymax': float_feature(ymax),\n",
196 | " 'image/object/bbox/label': int64_feature(label),\n",
197 | " 'image/format': bytes_feature(image_format), # 图像编码格式\n",
198 | " 'image/encoded': bytes_feature(image_data)})) # 二进制图像数据\n",
199 | " # 书写入文件\n",
200 | " writer.write(example.SerializeToString())"
201 | ]
202 | },
203 | {
204 | "cell_type": "code",
205 | "execution_count": 6,
206 | "metadata": {
207 | "collapsed": false
208 | },
209 | "outputs": [
210 | {
211 | "data": {
212 | "text/plain": [
213 | "(,\n",
214 | " ,\n",
215 | " )"
216 | ]
217 | },
218 | "execution_count": 6,
219 | "metadata": {},
220 | "output_type": "execute_result"
221 | }
222 | ],
223 | "source": [
224 | "import tensorflow.contrib.slim as slim\n",
225 | "\n",
226 | "def get_split(tfr_path, tfr_pattren, num_classes=21):\n",
227 | " \n",
228 | " # ===============TFR文件名匹配模板===============\n",
229 | " tfr_pattren = os.path.join(tfr_path, tfr_pattren)\n",
230 | " \n",
231 | " # =========阅读器=========\n",
232 | " reader = tf.TFRecordReader\n",
233 | " \n",
234 | " # ===================解码器===================\n",
235 | " keys_to_features = { # 解码TFR文件方式\n",
236 | " 'image/encoded': tf.FixedLenFeature((), tf.string, default_value=''),\n",
237 | " 'image/format': tf.FixedLenFeature((), tf.string, default_value='jpeg'),\n",
238 | " 'image/object/bbox/xmin': tf.VarLenFeature(dtype=tf.float32),\n",
239 | " 'image/object/bbox/ymin': tf.VarLenFeature(dtype=tf.float32),\n",
240 | " 'image/object/bbox/xmax': tf.VarLenFeature(dtype=tf.float32),\n",
241 | " 'image/object/bbox/ymax': tf.VarLenFeature(dtype=tf.float32),\n",
242 | " 'image/object/bbox/label': tf.VarLenFeature(dtype=tf.int64),\n",
243 | " }\n",
244 | " items_to_handlers = { # 解码二进制数据\n",
245 | " # 图像解码设置蛮有意思的\n",
246 | " 'image': slim.tfexample_decoder.Image('image/encoded', 'image/format'),\n",
247 | " 'object/bbox': slim.tfexample_decoder.BoundingBox(\n",
248 | " ['ymin', 'xmin', 'ymax', 'xmax'], 'image/object/bbox/'),\n",
249 | " 'object/label': slim.tfexample_decoder.Tensor('image/object/bbox/label'),\n",
250 | " }\n",
251 | " decoder = slim.tfexample_decoder.TFExampleDecoder(keys_to_features, items_to_handlers)\n",
252 | " \n",
253 | " # =======描述字段=======\n",
254 | " items_to_descriptions={\n",
255 | " 'image': 'A color image of varying height and width.',\n",
256 | " 'shape': 'Shape of the image',\n",
257 | " 'object/bbox': 'A list of bounding boxes, one per each object.',\n",
258 | " 'object/label': 'A list of labels, one per each object.',\n",
259 | " }\n",
260 | " \n",
261 | " return slim.dataset.Dataset(\n",
262 | " data_sources=tfr_pattren, # TFR文件名\n",
263 | " reader=reader, # 阅读器\n",
264 | " decoder=decoder, # 解码器\n",
265 | " num_samples=len(file_names), # 数目\n",
266 | " items_to_descriptions=items_to_descriptions, # decoder条目描述字段\n",
267 | " num_classes=num_classes, # 类别数\n",
268 | " labels_to_names=None # 字典{图片:类别,……}\n",
269 | " )\n",
270 | "\n",
271 | "pattren = 'voc2012_*.tfrecord'\n",
272 | "dataset = get_split(tfr_dir, pattren, num_classes=21)\n",
273 | "provider = slim.dataset_data_provider.DatasetDataProvider(\n",
274 | " dataset, # DatasetDataProvider 需要 slim.dataset.Dataset 做参数\n",
275 | " num_readers=2,\n",
276 | " common_queue_capacity=20 * 5,\n",
277 | " common_queue_min=10 * 5,\n",
278 | " shuffle=True)\n",
279 | "image, glabels, gbboxes = provider.get(['image',\n",
280 | " 'object/label',\n",
281 | " 'object/bbox'])\n",
282 | "image, glabels, gbboxes"
283 | ]
284 | },
285 | {
286 | "cell_type": "code",
287 | "execution_count": 5,
288 | "metadata": {
289 | "collapsed": false
290 | },
291 | "outputs": [
292 | {
293 | "name": "stdout",
294 | "output_type": "stream",
295 | "text": [
296 | "[array([15]), array([[ 0.40266666, 0.252 , 1. , 0.76999998]], dtype=float32)]\n"
297 | ]
298 | }
299 | ],
300 | "source": [
301 | "with tf.Session() as sess:\n",
302 | " init_op = tf.group(tf.global_variables_initializer(),\n",
303 | " tf.local_variables_initializer())\n",
304 | " sess.run(init_op)\n",
305 | " coord = tf.train.Coordinator()\n",
306 | " threads = tf.train.start_queue_runners(sess=sess, coord=coord)\n",
307 | " print(sess.run([glabels, gbboxes]))\n",
308 | " coord.request_stop()\n",
309 | " coord.join(threads)"
310 | ]
311 | },
312 | {
313 | "cell_type": "code",
314 | "execution_count": null,
315 | "metadata": {
316 | "collapsed": true
317 | },
318 | "outputs": [],
319 | "source": [
320 | "class SSD:\n",
321 | " def __init__(self):\n",
322 | " pass\n",
323 | " def inference(self):\n",
324 | " pass"
325 | ]
326 | }
327 | ],
328 | "metadata": {
329 | "kernelspec": {
330 | "display_name": "Python 3",
331 | "language": "python",
332 | "name": "python3"
333 | },
334 | "language_info": {
335 | "codemirror_mode": {
336 | "name": "ipython",
337 | "version": 3
338 | },
339 | "file_extension": ".py",
340 | "mimetype": "text/x-python",
341 | "name": "python",
342 | "nbconvert_exporter": "python",
343 | "pygments_lexer": "ipython3",
344 | "version": "3.6.3"
345 | }
346 | },
347 | "nbformat": 4,
348 | "nbformat_minor": 2
349 | }
350 |
--------------------------------------------------------------------------------
/train_ssd_network.py:
--------------------------------------------------------------------------------
1 | # Author : hellcat
2 | # Time : 18-8-24
3 |
4 | """
5 | import os
6 | os.environ["CUDA_VISIBLE_DEVICES"]="-1"
7 |
8 | import numpy as np
9 | np.set_printoptions(threshold=np.inf)
10 |
11 | import tensorflow as tf
12 | config = tf.ConfigProto()
13 | config.gpu_options.allow_growth = True
14 | sess = tf.Session(config=config)
15 | """
16 | import tensorflow as tf
17 |
18 |
19 | import util_tf
20 | import tfr_data_process
21 | import preprocess_img_tf
22 | from ssd_vgg300_tf import SSDNet
23 |
24 |
25 | slim = tf.contrib.slim
26 |
27 |
28 | # =========================================================================== #
29 | # Main training routine.
30 | # =========================================================================== #
31 | def main():
32 |
33 | max_steps = 1500
34 | batch_size = 32
35 | adam_beta1 = 0.9
36 | adam_beta2 = 0.999
37 | opt_epsilon = 1.0
38 | num_epochs_per_decay = 2.0
39 | num_samples_per_epoch = 17125
40 | moving_average_decay = None
41 |
42 | tf.logging.set_verbosity(tf.logging.DEBUG)
43 | with tf.Graph().as_default():
44 |
45 | # Create global_step.
46 | with tf.device("/device:CPU:0"):
47 | global_step = tf.train.create_global_step()
48 |
49 | ssd = SSDNet()
50 | ssd_anchors = ssd.anchors
51 |
52 | # tfr解析操作放在GPU下有加速，效果不稳定
53 | dataset = \
54 | tfr_data_process.get_split('./TFR_Data',
55 | 'voc2012_*.tfrecord',
56 | num_classes=21,
57 | num_samples=num_samples_per_epoch)
58 |
59 | with tf.device("/device:CPU:0"): # 仅CPU支持队列操作
60 | image, glabels, gbboxes = \
61 | tfr_data_process.tfr_read(dataset)
62 |
63 | image, glabels, gbboxes = \
64 | preprocess_img_tf.preprocess_image(image, glabels, gbboxes, out_shape=(300, 300))
65 |
66 | gclasses, glocalisations, gscores = \
67 | ssd.bboxes_encode(glabels, gbboxes, ssd_anchors)
68 |
69 | batch_shape = [1] + [len(ssd_anchors)] * 3 # (1,f层,f层,f层)
70 | # Training batches and queue.
71 | r = tf.train.batch( # 图片，中心点类别，真实框坐标，得分
72 | util_tf.reshape_list([image, gclasses, glocalisations, gscores]),
73 | batch_size=batch_size,
74 | num_threads=4,
75 | capacity=5 * batch_size)
76 | batch_queue = slim.prefetch_queue.prefetch_queue(
77 | r, # <-----输入格式实际上并不需要调整
78 | capacity=2 * 1)
79 |
80 | # Dequeue batch.
81 | b_image, b_gclasses, b_glocalisations, b_gscores = \
82 | util_tf.reshape_list(batch_queue.dequeue(), batch_shape) # 重整list
83 |
84 | predictions, localisations, logits, end_points = \
85 | ssd.net(b_image, is_training=True, weight_decay=0.00004)
86 |
87 | ssd.losses(logits, localisations,
88 | b_gclasses, b_glocalisations, b_gscores,
89 | match_threshold=.5,
90 | negative_ratio=3,
91 | alpha=1,
92 | label_smoothing=.0)
93 |
94 | update_ops = tf.get_collection(tf.GraphKeys.UPDATE_OPS)
95 | # =================================================================== #
96 | # Configure the moving averages.
97 | # =================================================================== #
98 | if moving_average_decay:
99 | moving_average_variables = slim.get_model_variables()
100 | variable_averages = tf.train.ExponentialMovingAverage(
101 | moving_average_decay, global_step)
102 | else:
103 | moving_average_variables, variable_averages = None, None
104 |
105 | # =================================================================== #
106 | # Configure the optimization procedure.
107 | # =================================================================== #
108 | with tf.device("/device:CPU:0"): # learning_rate节点使用CPU（不明）
109 | decay_steps = int(num_samples_per_epoch / batch_size * num_epochs_per_decay)
110 | learning_rate = tf.train.exponential_decay(0.01,
111 | global_step,
112 | decay_steps,
113 | 0.94, # learning_rate_decay_factor,
114 | staircase=True,
115 | name='exponential_decay_learning_rate')
116 | optimizer = tf.train.AdamOptimizer(
117 | learning_rate,
118 | beta1=adam_beta1,
119 | beta2=adam_beta2,
120 | epsilon=opt_epsilon)
121 | tf.summary.scalar('learning_rate', learning_rate)
122 |
123 | if moving_average_decay:
124 | # Update ops executed locally by trainer.
125 | update_ops.append(variable_averages.apply(moving_average_variables))
126 |
127 | # Variables to train.
128 | trainable_scopes = None
129 | if trainable_scopes is None:
130 | variables_to_train = tf.trainable_variables()
131 | else:
132 | scopes = [scope.strip() for scope in trainable_scopes.split(',')]
133 | variables_to_train = []
134 | for scope in scopes:
135 | variables = tf.get_collection(tf.GraphKeys.TRAINABLE_VARIABLES, scope)
136 | variables_to_train.extend(variables)
137 |
138 | losses = tf.get_collection(tf.GraphKeys.LOSSES)
139 | regularization_losses = tf.get_collection(
140 | tf.GraphKeys.REGULARIZATION_LOSSES)
141 | regularization_loss = tf.add_n(regularization_losses)
142 | loss = tf.add_n(losses)
143 | tf.summary.scalar("loss", loss)
144 | tf.summary.scalar("regularization_loss", regularization_loss)
145 |
146 | grad = optimizer.compute_gradients(loss, var_list=variables_to_train)
147 | grad_updates = optimizer.apply_gradients(grad,
148 | global_step=global_step)
149 | update_ops.append(grad_updates)
150 | # update_op = tf.group(*update_ops)
151 |
152 | with tf.control_dependencies(update_ops):
153 | total_loss = tf.add_n([loss, regularization_loss])
154 | tf.summary.scalar("total_loss", total_loss)
155 |
156 | # =================================================================== #
157 | # Kicks off the training.
158 | # =================================================================== #
159 | gpu_options = tf.GPUOptions(per_process_gpu_memory_fraction=0.8)
160 | config = tf.ConfigProto(log_device_placement=False,
161 | gpu_options=gpu_options)
162 | saver = tf.train.Saver(max_to_keep=5,
163 | keep_checkpoint_every_n_hours=1.0,
164 | write_version=2,
165 | pad_step_number=False)
166 |
167 | if True:
168 | import os
169 | import time
170 |
171 | print('start......')
172 | model_path = './logs'
173 | batch_size = batch_size
174 | with tf.Session(config=config) as sess:
175 | summary = tf.summary.merge_all()
176 | coord = tf.train.Coordinator()
177 | threads = tf.train.start_queue_runners(sess=sess, coord=coord)
178 | writer = tf.summary.FileWriter(model_path, sess.graph)
179 |
180 | init_op = tf.group(tf.global_variables_initializer(),
181 | tf.local_variables_initializer())
182 | init_op.run()
183 | for step in range(max_steps):
184 | start_time = time.time()
185 | loss_value = sess.run(total_loss)
186 | # loss_value, summary_str = sess.run([train_tensor, summary_op])
187 | # writer.add_summary(summary_str, step)
188 |
189 | duration = time.time() - start_time
190 | if step % 10 == 0:
191 | summary_str = sess.run(summary)
192 | writer.add_summary(summary_str, step)
193 |
194 | examples_per_sec = batch_size / duration
195 | sec_per_batch = float(duration)
196 | format_str = "[*] step %d, loss=%.2f (%.1f examples/sec; %.3f sec/batch)"
197 | print(format_str % (step, loss_value, examples_per_sec, sec_per_batch))
198 | # if step % 100 == 0:
199 | # accuracy_step = test_cifar10(sess, training=False)
200 | # acc.append('{:.3f}'.format(accuracy_step))
201 | # print(acc)
202 | if step % 500 == 0 and step != 0:
203 | saver.save(sess, os.path.join(model_path, "ssd_tf.model"), global_step=step)
204 |
205 | coord.request_stop()
206 | coord.join(threads)
207 |
208 |
209 | if __name__ == '__main__':
210 | main()
211 |
212 |
--------------------------------------------------------------------------------
/util_tf.py:
--------------------------------------------------------------------------------
1 | import tensorflow as tf
2 | from tensorflow.python.ops import control_flow_ops
3 |
4 | slim = tf.contrib.slim
5 |
6 | _R_MEAN = 123.68
7 | _G_MEAN = 116.78
8 | _B_MEAN = 103.94
9 |
10 | _RESIZE_SIDE_MIN = 256
11 | _RESIZE_SIDE_MAX = 512
12 |
13 |
14 | def tensor_shape(x, rank=3):
15 | """Returns the dimensions of a tensor.
16 | """
17 | if x.get_shape().is_fully_defined():
18 | return x.get_shape().as_list()
19 | else:
20 | # get_shape返回值，with_rank相当于断言assert，是否rank为指定值
21 | static_shape = x.get_shape().with_rank(rank).as_list()
22 | # tf.shape返回张量，其中num解释为"The length of the dimension `axis`."，axis默认为0
23 | dynamic_shape = tf.unstack(tf.shape(x), num=rank)
24 | # list，有定义的给数字，没有的给tensor
25 | return [s if s is not None else d
26 | for s, d in zip(static_shape, dynamic_shape)]
27 |
28 |
29 | def abs_smooth(x):
30 | """Smoothed absolute function. Useful to compute an L1 smooth error.
31 |
32 | Define as:
33 | x^2 / 2 if abs(x) < 1
34 | abs(x) - 0.5 if abs(x) > 1
35 | We use here a differentiable definition using min(x) and abs(x). Clearly
36 | not optimal, but good enough for our purpose!
37 | """
38 | absx = tf.abs(x)
39 | minx = tf.minimum(absx, 1)
40 | r = 0.5 * ((absx - 1) * minx + absx)
41 | return r
42 |
43 |
44 | def reshape_list(l, shape=None):
45 | """Reshape list of (list): 1D to 2D or the other way around.
46 |
47 | Args:
48 | l: List or List of list.
49 | shape: 1D or 2D shape.
50 | Return
51 | Reshaped list.
52 | """
53 | r = []
54 | if shape is None:
55 | # Flatten everything.
56 | for a in l:
57 | if isinstance(a, (list, tuple)):
58 | r = r + list(a)
59 | else:
60 | r.append(a)
61 | else:
62 | # Reshape to list of list.
63 | i = 0
64 | for s in shape:
65 | if s == 1:
66 | r.append(l[i])
67 | else:
68 | r.append(l[i:i + s])
69 | i += s
70 | return r
71 |
72 |
73 | def resize_image(image, size,
74 | method=tf.image.ResizeMethod.BILINEAR,
75 | align_corners=False):
76 | """Resize an image and bounding boxes.
77 | """
78 | # Resize image.
79 | with tf.name_scope('resize_image'):
80 | height, width, channels = tensor_shape(image)
81 | image = tf.expand_dims(image, 0)
82 | image = tf.image.resize_images(image, size,
83 | method, align_corners)
84 | image = tf.reshape(image, tf.stack([size[0], size[1], channels]))
85 | return image
86 |
87 |
88 | def _check3dimage(image, require_static=True):
89 | """Assert that we are working with properly shaped image.
90 | Args:
91 | image: 3-D Tensor of shape [height, width, channels]
92 | require_static: If `True`, requires that all dimensions of `image` are
93 | known and non-zero.
94 | Raises:
95 | ValueError: if `image.shape` is not a 3-vector.
96 | Returns:
97 | An empty list, if `image` has fully defined dimensions. Otherwise, a list
98 | containing an assert op is returned.
99 | """
100 | try:
101 | image_shape = image.get_shape().with_rank(3)
102 | except ValueError:
103 | raise ValueError("'image' must be three-dimensional.")
104 | if require_static and not image_shape.is_fully_defined():
105 | raise ValueError("'image' must be fully defined.")
106 | if any(x == 0 for x in image_shape):
107 | raise ValueError("all dims of 'image.shape' must be > 0: %s" %
108 | image_shape)
109 | if not image_shape.is_fully_defined():
110 | return [tf.assert_positive(tf.shape(image),
111 | ["all dims of 'image.shape' "
112 | "must be > 0."])]
113 | else:
114 | return []
115 |
116 |
117 | def random_flip_left_right(image, bboxes, seed=None):
118 | """Random flip left-right of an image and its bounding boxes.
119 | """
120 | def flip_bboxes(bboxes):
121 | """Flip bounding boxes coordinates.
122 | """
123 | bboxes = tf.stack([bboxes[:, 0], 1 - bboxes[:, 3],
124 | bboxes[:, 2], 1 - bboxes[:, 1]], axis=-1)
125 | return bboxes
126 |
127 | # Random flip. Tensorflow implementation.
128 | with tf.name_scope('random_flip_left_right'):
129 | image = tf.convert_to_tensor(image, name='image')
130 | _check3dimage(image, require_static=False)
131 | uniform_random = tf.random_uniform([], 0, 1.0, seed=seed)
132 | mirror_cond = tf.less(uniform_random, .5)
133 | # Flip image.
134 | result = tf.cond(mirror_cond,
135 | lambda: tf.reverse_v2(image, [1]),
136 | lambda: image)
137 | # Flip bboxes.
138 | bboxes = tf.cond(mirror_cond,
139 | lambda: flip_bboxes(bboxes),
140 | lambda: bboxes)
141 |
142 | image_shape = image.get_shape()
143 | result.set_shape(image_shape)
144 | return result, bboxes
145 |
146 |
147 | def tf_image_whitened(image, means=(_R_MEAN, _G_MEAN, _B_MEAN)):
148 | """Subtracts the given means from each image channel.
149 | Returns:
150 | the centered image.
151 | """
152 | if image.get_shape().ndims != 3:
153 | raise ValueError('Input must be of size [height, width, C>0]')
154 | num_channels = image.get_shape().as_list()[-1]
155 | if len(means) != num_channels:
156 | raise ValueError('len(means) must match the number of channels')
157 |
158 | mean = tf.constant(means, dtype=image.dtype)
159 | image = image - mean
160 | return image
161 |
162 |
163 | def distort_color(image, color_ordering=0, fast_mode=True, scope=None):
164 | """Distort the color of a Tensor image.
165 | Each color distortion is non-commutative and thus ordering of the color ops
166 | matters. Ideally we would randomly permute the ordering of the color ops.
167 | Rather then adding that level of complication, we select a distinct ordering
168 | of color ops for each preprocessing thread.
169 | Args:
170 | image: 3-D Tensor containing single image in [0, 1].
171 | color_ordering: Python int, a type of distortion (valid values: 0-3).
172 | fast_mode: Avoids slower ops (random_hue and random_contrast)
173 | scope: Optional scope for name_scope.
174 | Returns:
175 | 3-D Tensor color-distorted image on range [0, 1]
176 | Raises:
177 | ValueError: if color_ordering not in [0, 3]
178 | """
179 | with tf.name_scope(scope, 'distort_color', [image]):
180 | if fast_mode:
181 | if color_ordering == 0:
182 | image = tf.image.random_brightness(image, max_delta=32. / 255.)
183 | image = tf.image.random_saturation(image, lower=0.5, upper=1.5)
184 | else:
185 | image = tf.image.random_saturation(image, lower=0.5, upper=1.5)
186 | image = tf.image.random_brightness(image, max_delta=32. / 255.)
187 | else:
188 | if color_ordering == 0:
189 | image = tf.image.random_brightness(image, max_delta=32. / 255.)
190 | image = tf.image.random_saturation(image, lower=0.5, upper=1.5)
191 | image = tf.image.random_hue(image, max_delta=0.2)
192 | image = tf.image.random_contrast(image, lower=0.5, upper=1.5)
193 | elif color_ordering == 1:
194 | image = tf.image.random_saturation(image, lower=0.5, upper=1.5)
195 | image = tf.image.random_brightness(image, max_delta=32. / 255.)
196 | image = tf.image.random_contrast(image, lower=0.5, upper=1.5)
197 | image = tf.image.random_hue(image, max_delta=0.2)
198 | elif color_ordering == 2:
199 | image = tf.image.random_contrast(image, lower=0.5, upper=1.5)
200 | image = tf.image.random_hue(image, max_delta=0.2)
201 | image = tf.image.random_brightness(image, max_delta=32. / 255.)
202 | image = tf.image.random_saturation(image, lower=0.5, upper=1.5)
203 | elif color_ordering == 3:
204 | image = tf.image.random_hue(image, max_delta=0.2)
205 | image = tf.image.random_saturation(image, lower=0.5, upper=1.5)
206 | image = tf.image.random_contrast(image, lower=0.5, upper=1.5)
207 | image = tf.image.random_brightness(image, max_delta=32. / 255.)
208 | else:
209 | raise ValueError('color_ordering must be in [0, 3]')
210 | # The random_* ops do not necessarily clamp.
211 | return tf.clip_by_value(image, 0.0, 1.0)
212 |
213 |
214 | def apply_with_random_selector(x, func, num_cases):
215 | """Computes func(x, sel), with sel sampled from [0...num_cases-1].
216 | Args:
217 | x: input Tensor.
218 | func: Python function to apply.
219 | num_cases: Python int32, number of cases to sample sel from.
220 | Returns:
221 | The result of func(x, sel), where func receives the value of the
222 | selector as a python integer, but sel is sampled dynamically.
223 | """
224 | sel = tf.random_uniform([], maxval=num_cases, dtype=tf.int32)
225 | # Pass the real x only to one of the func calls.
226 | return control_flow_ops.merge([
227 | func(control_flow_ops.switch(x, tf.equal(sel, case))[1], case)
228 | for case in range(num_cases)])[0]
229 |
230 |
231 | def get_init_fn(checkpoint_path,
232 | train_dir,
233 | checkpoint_exclude_scopes,
234 | checkpoint_model_scope,
235 | model_name,
236 | ignore_missing_vars):
237 | """Returns a function run by the chief worker to warm-start the training.
238 | Note that the init_fn is only run when initializing the model during the very
239 | first global step.
240 |
241 | Returns:
242 | An init function run by the supervisor.
243 | """
244 | if checkpoint_path is None:
245 | return None
246 | # Warn the user if a checkpoint exists in the train_dir. Then ignore.
247 | if tf.train.latest_checkpoint(train_dir):
248 | tf.logging.info(
249 | 'Ignoring --checkpoint_path because a checkpoint already exists in %s'
250 | % train_dir)
251 | return None
252 |
253 | exclusions = []
254 | if checkpoint_exclude_scopes:
255 | exclusions = [scope.strip()
256 | for scope in checkpoint_exclude_scopes.split(',')]
257 | variables_to_restore = []
258 | for var in slim.get_model_variables():
259 | excluded = False
260 | for exclusion in exclusions:
261 | if var.op.name.startswith(exclusion):
262 | excluded = True
263 | break
264 | if not excluded:
265 | variables_to_restore.append(var)
266 | # Change model scope if necessary.
267 | if checkpoint_model_scope is not None:
268 | variables_to_restore = \
269 | {var.op.name.replace(model_name,
270 | checkpoint_model_scope): var
271 | for var in variables_to_restore}
272 |
273 | if tf.gfile.IsDirectory(checkpoint_path):
274 | checkpoint_path = tf.train.latest_checkpoint(checkpoint_path)
275 | tf.logging.info('Fine-tuning from %s. Ignoring missing vars: %s' % (checkpoint_path, ignore_missing_vars))
276 |
277 | return slim.assign_from_checkpoint_fn(
278 | checkpoint_path,
279 | variables_to_restore,
280 | ignore_missing_vars=ignore_missing_vars)
281 |
282 |
--------------------------------------------------------------------------------