├── README.md ├── analyze.txt ├── analyze_ipZ.py ├── base.py ├── build.sh ├── build ├── solver.prototxt ├── train_imp.sh └── train_test_SVD.prototext ├── build_ip1 ├── solver.prototxt ├── solver_conv3x3.prototxt ├── train_imp.sh ├── train_imp_conv3x3.sh ├── train_test_SVD_ip1.prototext └── train_test_SVD_ip1_conv3x3.prototxt ├── chart.py ├── chart2.py ├── chart_imp.py ├── convert_mean.py ├── eval_model_ip1.py ├── eval_model_ip2.py ├── evaluate.py ├── improve_ip1_new.py ├── improve_ip2.py ├── improve_ip2_new.py ├── improve_model_ip1.py ├── label.npy ├── mcluster.py ├── mean.npy ├── net.txt ├── noimprove_ip2.py ├── pic ├── acc_time_space.png ├── acc_time_space_ip1.png ├── statistics.png └── statistics_ip1.png ├── proto ├── cifar10_SVD.template ├── cifar10_ip1_SVD.template ├── cifar10_quick.prototxt ├── cifar10_quick_solver_imp.template ├── cifar10_quick_solver_noimp.template ├── cifar10_quick_train_test.prototxt ├── cifar10_quick_train_test_SVD.template ├── solver.template ├── train_imp.sh ├── train_quick_imp.sh ├── train_quick_noimp.sh ├── train_test_SVD.template └── train_test_SVD_ip1.template ├── result ├── label.npy ├── mean.npy ├── net_SVD10.npy ├── net_SVD3.npy ├── net_SVD4.npy ├── net_SVD5.npy ├── net_SVD6.npy ├── net_SVD7.npy ├── net_SVD8.npy ├── net_SVD9.npy ├── net_ip1_SVD25.npy ├── net_ip1_SVD35.npy ├── net_ip1_SVD40.npy ├── net_ip1_SVD52.npy ├── net_ip1_SVD55.npy ├── net_ip1_SVD60.npy ├── net_ip1_SVD62.npy ├── net_ip1_SVD63.npy └── net_normal.npy ├── slides ├── slide.pptx └── slide2.pptx ├── test.py ├── test2.py ├── test_ip1_new.py └── test_new.py /README.md: -------------------------------------------------------------------------------- 1 | # 实现一个压缩的深度神经网络 2 | 3 | **注：请在typora或其它支持Markdown公式语法的阅读器下阅读本文档.** 4 | 5 | ## 动机 6 | 7 | 深度神经网络中, 全连接网络层的参数占据的存储空间比较大, 在深度卷积神经网络中, 全连接层的参数占据了参数存储空间的90%以上.[1] 深度学习算法是计算密集型和存储密集型的[2], 使得它难以部署在拥有有限硬件资源的嵌入式系统上. 为了降低参数所占用的存储空间, 提高预测效率, 使用奇异值分解的方法(SVD)对全连接层的参数矩阵进行压缩, 并研究模型预测准确度与压缩率的关系. 8 | 9 | ## 实验环境 10 | 操作系统: Arch Linux 4.10.8 11 | 深度学习框架: Caffe(Apr 7,2017) 12 | 编程语言: Python 2.7.13 13 | 编译器: g++ (GCC) 6.3.1 20170306 14 | Cuda compilation tools, release 8.0, V8.0.44 15 | 数据库: lmdb 16 | 图像处理库: OpenCV3 17 | 环境配置: 18 | CPU: Intel(r) Core(tm) i7-4702MQ CPU @ 2.20GHZ 19 | GPU: NVIDIA Corporation GK107M [GeForce GT 750M] 20 | 21 | ## 文件说明 22 | 23 | |文件名|说明| 24 | |------|----| 25 | |test.py|压缩IP2全连接层, 可以通过修改SVD_R变量改变压缩参数| 26 | |test2.py|压缩IP1全连接层, 可以通过修改SVD_R变量改变压缩参数| 27 | |improve_model_ip1.py|使用SVD, 聚类，Reduced-Precision方法压缩| 28 | |improve_ip1_new.py|压缩IP1全连接层后进行fine-tune| 29 | |improve_ip2.py|压缩IP2全连接层后以原参数为初值，进行fine-tune| 30 | |improve_ip2_new.py|压缩IP2全连接层后保存caffemodel(不进行fine-tune)| 31 | |noimprove_ip2.py|随机初始化权重，重新训练被压缩ip2层后的网络| 32 | |eval_model_ip1.py|评估压缩ip1层后的性能, 在result文件夹生成对应的*.npy文件| 33 | |eval_model_ip2.py|评估压缩ip2层后的性能, 在result文件夹生成对应的*.npy文件| 34 | |evaluate.py|读取result文件夹下的*.npy文件并进行评估| 35 | |base.py|基本操作，比如proto模板生成, 模型评分| 36 | |build.sh|训练脚本| 37 | |chart.py|显示压缩ip2层的对比结果| 38 | |chart2.py|显示压缩ip1层的对比结果| 39 | |chart_imp.py|绘制改进的压缩方法(retrain)的结果图| 40 | |convert_mean.py|将mean.binaryproto转换为mean.npy| 41 | |mcluster.py|使用K-means对全连接层进行压缩| 42 | |analyze_ipZ.py|分析ipZ层| 43 | |mean.npy|图像均值文件| 44 | |label.npy|图像测试集的标签文件| 45 | 46 | |文件夹|说明| 47 | |------|----| 48 | |build/|SVD压缩ip2后训练网络的文件| 49 | |build_ip1/|SVD压缩ip1后训练网络的文件| 50 | |proto/|proto和proto模板, 自动化脚本模板| 51 | |pic/|文档图片| 52 | |slides/|演示文件，介绍了项目的压缩方法| 53 | 54 | ## 深度神经网络 55 | 56 | - 数据集 57 | 58 | 数据集名称: CIFAR-10 dataset 59 | 60 | 每个样本为: 32 $$\times$$ 32的三通道(RGB)图片 61 | 62 | 分类数: 10 63 | 64 | | 数据集名称 | 总样本数 | 每个分类的样本数 | 65 | | ----- | ----- | -------- | 66 | | 训练集 | 50000 | 5000 | 67 | | 测试集 | 10000 | 1000 | 68 | 69 | - 网络模型 70 | 71 | * 训练网络时的神经网络定义 72 | ``` 73 | cifar10_quick_train_test.prototxt 74 | conv1: data -> conv1 75 | pool1: conv1 -> pool1 76 | relu1: pool1 -> pool1 77 | conv2: pool1 -> conv2 78 | relu2: conv2 -> conv2 79 | pool2: conv2 -> pool2 80 | conv3: pool2 -> conv3 81 | relu3: conv3 -> conv3 82 | pool3: conv3 -> pool3 83 | ip1 : pool3 -> ip1 84 | ip2 : ip1 -> ip2 85 | accuracy[TEST]: ip2, label -> accuracy 86 | loss(softmax with loss): ip2, label -> loss 87 | ``` 88 | 89 | * 训练结束后保存的神经网络的定义 90 | ``` 91 | conv1: data -> conv1 92 | pool1: conv1 -> pool1 93 | relu1: pool1 -> pool1 94 | conv2: pool1 -> conv2 95 | relu2: conv2 -> conv2 96 | pool2: conv2 -> pool2 97 | conv3: pool2 -> conv3 98 | relu3: conv3 -> conv3 99 | pool3: conv3 -> pool3 100 | ip1 : pool3 -> ip1 101 | ip2 : ip1 -> ip2 102 | prob : ip2 -> prob 103 | ``` 104 | 105 | 其中ip指InnerProduct, 即全连接层. 106 | 107 | ## 方法 108 | 109 | - 奇异值(SVD)分解 110 | 111 | 奇异值分解能够提取矩阵的重要特征, 它可以对任何形状的矩阵进行分解. 112 | 113 | $$A=U{\sum}V^T$$ 114 | 115 | - 部分奇异值分解 116 | 117 | 部分奇异值分解可以对矩阵的存储空间进行压缩 118 | 119 | 假设矩阵A的大小为$$m\times n$$, 奇异值分解后得到: 120 | 121 | $$A_{m\times n}=U_{m\times m}{\sum}_{m\times n}{V^T}_{n\times n}$$ 122 | 123 | $${\sum}$$为对角矩阵, 奇异值沿着对角线从大到小排列, 选取前r个奇异值来近似矩阵A, 有: 124 | 125 | $$A_{m\times n}{\approx}U_{m\times r}{\sum}_{r\times r}{V^T}_{r\times n}$$ 126 | 127 | - 使用SVD对神经网络的全连接层进行压缩 128 | 129 | 首先需要训练一遍深度神经网络, 然后使用SVD方法对全连接层的参数矩阵进行压缩. 130 | 131 | 设神经网络的某一层全连接层的参数矩阵为$$W \in R^{m\times n}$$, 偏差向量为$$b \in R^m$$ 132 | 133 | 输入向量: $$x \in R^n$$, 输出向量: $$y \in R^m$$ 134 | 135 | 全连接层输入输出的关系: 136 | 137 | $$y = Wx + b$$ 138 | 139 | 对W进行奇异值分解: 140 | 141 | $$W = U_{m\times m}{\sum}_{m\times n}{V^T}_{n\times n}$$ 142 | 143 | 进行部分奇异值分解: 144 | 145 | $$W_{m\times n}{\approx}U_{m\times r}{\sum}_{r\times r}{V^T}_{r\times n}$$ 146 | 147 | 其中, $$r <= min(m, n)$$ 148 | 149 | 部分奇异值分解后, 原全连接层可以分为三个全连接层或两个全连接层, 但是不同的划分组合的时间复杂度与空间复杂度是不一样的 150 | 151 | 152 | - 不同划分方式对应的时间复杂度与空间复杂度 153 | 154 | | 划分方式 | 层数 | 时间复杂度 | 空间复杂度 | 155 | | ---------------------------------------- | ---- | ---------------- | ---------------- | 156 | | 不压缩 | 1 | $$O(mn)$$ | $$O(mn)$$ | 157 | | $$(U_{m\times r}{\sum}_{r\times r}){V^T}_{r\times n}$$ | 2 | $$O(mr+rn)$$ | $$O(mr+rn)$$ | 158 | | $$U_{m\times r}({\sum}_{r\times r}{V^T}_{r\times n})$$ | 2 | $$O(mr+rn)$$ | $$O(mr+rn)$$ | 159 | | $$U_{m\times r}{\sum}_{r\times r}{V^T}_{r\times n}$$ | 3 | $$O(mr+r^2+rn)$$ | $$O(mr+r^2+rn)$$ | 160 | 161 | 注: 上述表格中, 括号表示将多个矩阵合并为一个矩阵. 时间复杂度指的是一个输入经过原全连接层或压缩后生成的多个全连接层的时间复杂度, 由于参数矩阵比较大, 忽略偏差矩阵所占的空间以及计算时使用的时间. 162 | 163 | 从表格上可以看到, 当全连接层的参数矩阵被拆分为两层时, 其时间复杂度和空间复杂度比拆分成三层的情况低. 将一个矩阵拆分成两层全连接网络有两种方式, 它们的时间复杂度和空间复杂度是一样的, 当$$mr + rn < mn$$时, 即 $$r < \frac{mn}{m+n}$$时, 使用SVD压缩参数矩阵, 在时间复杂度和空间复杂度上有优势. 164 | 165 | 记要压缩的全连接层为$$ip_x$$, 其参数矩阵为W, 偏差向量为b 166 | 167 | 对参数矩阵W进行SVD分解后, 168 | 169 | 令$$Z={\sum}V^T$$, 则$$y = U(Zx) + b$$ 170 | 171 | 原全连接层被分为两个新的且更小的全连接层 172 | 173 | 即: 输入 -> W,b -> 输出 174 | 175 | 变为: 输入 -> Z,0 -> U,b -> 输出 176 | 177 | 全连接层Z的偏差向量全部取0, 全连接层U的偏差向量为b. 178 | 179 | caffe自带的cifar10_quick样例有两个全连接层: 180 | 181 | | 全连接层名称 | 输入特征数 | 输出特征数 | 182 | | ------ | ----- | ----- | 183 | | ip1 | 1024 | 64 | 184 | | ip2 | 64 | 10 | 185 | 186 | ## 结果分析 187 | 188 | - 未压缩全连接层前的预测精度 189 | 190 | caffe自带的cifar10_quick神经网络经过训练后, 准确度达到了0.7511 191 | 192 | - 压缩全连接层ip2 193 | 194 | 使用SVD压缩全连接层ip2的参数矩阵: 195 | 196 | $$W_{m\times n}{\approx}U_{m\times r}{\sum}_{r\times r}{V^T}_{r\times n}$$ 197 | 198 | $$r$$分别取值为从3到10的整数. 引入统计量accuarcy, precision, recall 和 F value 199 | 200 | $$压缩率 = \frac{新参数矩阵存储空间大小}{原参数矩阵存储空间大小}$$ 201 | 202 | 压缩率越小, 代表压缩后的存储空间越小. 203 | 204 | | r | accuracy | mean precision | mean recall | mean F | time | space | compression rate | 205 | | ------------ | -------- | -------------- | ----------- | -------- | ---- | ----- | ---------------- | 206 | | uncompressed | 0.751100 | 0.751777 | 0.751100 | 0.750558 | 640 | 640 | 100.00% | 207 | | 10 | 0.751100 | 0.751777 | 0.751100 | 0.750558 | 740 | 740 | 115.62% | 208 | | 9 | 0.716000 | 0.709909 | 0.716000 | 0.696003 | 666 | 666 | 104.06% | 209 | | 8 | 0.690300 | 0.683415 | 0.690300 | 0.672081 | 592 | 592 | 92.50% | 210 | | 7 | 0.666800 | 0.663032 | 0.666800 | 0.637553 | 518 | 518 | 80.94% | 211 | | 6 | 0.613200 | 0.607457 | 0.613200 | 0.580367 | 444 | 444 | 69.38% | 212 | | 5 | 0.580400 | 0.585117 | 0.580400 | 0.537906 | 370 | 370 | 57.81% | 213 | | 4 | 0.518000 | 0.545117 | 0.518000 | 0.472613 | 296 | 296 | 46.25% | 214 | | 3 | 0.430400 | nan | 0.430400 | nan | 222 | 222 | 34.69% | 215 | 216 | ![](pic/acc_time_space.png) 217 | ![](pic/statistics.png) 218 | 219 | 可以看到, 当$$r >= 9$$时, 准确度没有提升, 压缩率反而增大. 220 | 221 | $$r = 10$$的预测结果和未压缩的神经网络的预测结果是一致的, 因为其保留了SVD分解的完整结果, 然而保存多个矩阵导致空间复杂度和时间复杂度增高. 222 | 223 | 随着r值的降低, 准确度和压缩率都在降低. 224 | 225 | 当r = 3的时候, 已经存在某个分类无法被预测出来. 226 | 227 | 由于被压缩的全连接网络层ip2的参数矩阵大小为$$10\times 64$$, 矩阵占用的存储空间很小, 被SVD方法压缩后容易出现欠拟合的现象, 导致准确度降低, 比较难体现出参数矩阵被压缩的优势. 228 | 229 | ip2的参数矩阵经过SVD分解后, 230 | 231 | $\sum$的对角线元素即奇异值为: 232 | 233 | ``` 234 | [ 1.02686238, 0.86153692, 0.77797216, 0.75643724, 0.7272585, 235 | 0.67449301, 0.62932462, 0.57285255, 0.51419431, 0.45387536] 236 | ``` 237 | 238 | 奇异值之间差异不大, 说明奇异向量的重要性相差不远. 239 | 240 | 当r逐步变小时, 神经网络的准确度降低得比较快. 241 | 242 | - 压缩全连接层ip1 243 | 244 | 我也尝试了保持全连接层ip2不变, 对全连接层ip1($$64\times 1024$$)进行SVD分解, 245 | 246 | ip1的参数矩阵的最大奇异值为3.354, 最小奇异值为2.059, 两者相差也不大. 247 | 248 | 当对ip1进行压缩时, 得到以下结果: 249 | 250 | | r | accuracy | mean precision | mean recall | mean F | time | space | compression rate | 251 | | ------------ | -------- | -------------- | ----------- | -------- | ----- | ----- | ---------------- | 252 | | uncompressed | 0.751100 | 0.751777 | 0.751100 | 0.750558 | 65536 | 65536 | 100.00% | 253 | | 63 | 0.751000 | 0.752306 | 0.751000 | 0.750361 | 68544 | 68544 | 104.59% | 254 | | 62 | 0.749400 | 0.750306 | 0.749400 | 0.748556 | 67456 | 67456 | 102.93% | 255 | | 60 | 0.742500 | 0.743564 | 0.742500 | 0.741240 | 65280 | 65280 | 99.61% | 256 | | 55 | 0.729300 | 0.732731 | 0.729300 | 0.725926 | 59840 | 59840 | 91.31% | 257 | | 52 | 0.727200 | 0.728836 | 0.727200 | 0.724121 | 56576 | 56576 | 86.33% | 258 | | 50 | 0.713900 | 0.718902 | 0.713900 | 0.710930 | 54400 | 54400 | 83.01% | 259 | | 45 | 0.696400 | 0.714023 | 0.696400 | 0.697290 | 48960 | 48960 | 74.71% | 260 | | 40 | 0.673500 | 0.688363 | 0.673500 | 0.663389 | 43520 | 43520 | 66.41% | 261 | | 35 | 0.645400 | 0.671528 | 0.645400 | 0.636024 | 38080 | 38080 | 58.11% | 262 | | 30 | 0.632100 | 0.653558 | 0.632100 | 0.622893 | 32640 | 32640 | 49.80% | 263 | | 25 | 0.609400 | 0.621119 | 0.609400 | 0.599473 | 27200 | 27200 | 41.50% | 264 | | 20 | 0.546700 | 0.562711 | 0.546700 | 0.543171 | 21760 | 21760 | 33.20% | 265 | | 10 | 0.397900 | 0.428503 | 0.397900 | 0.370067 | 10880 | 10880 | 16.60% | 266 | 267 | ![](pic/acc_time_space_ip1.png) 268 | ![](pic/statistics_ip1.png) 269 | 270 | 随着r的减小, 神经网络的准确度在降低. 271 | 272 | 压缩ip1层时, 同样存在r接近min(m,n)时压缩率大于100%的情况. 273 | 274 | 当只对ip1进行SVD压缩时, 压缩率为74.71%时, 准确度为0.696. 275 | 276 | 时间,空间复杂度减少了$$64 \times 1024 \times (100\% - 74.71\%) = 16574$$ 277 | 278 | 而只对ip2进行SVD压缩时, 压缩率为92.50%时, 准确度为0.690. 279 | 280 | 时间,空间复杂度减少了$$10 \times 64 \times (100\% - 92.50\%) = 48$$ 281 | 282 | 同等准确度的情况下, 压缩较大的全连接层, 得到的效果更好. 283 | 284 | ## 改进 285 | 286 | 在上述对全连接层压缩的方法中, 用了两个小的全连接层替代原来的大的全连接层. 287 | 288 | 其中靠近输出的一个小的全连接层的偏差向量为大的全连接层的偏差向量, 而靠近输入的小的全连接层的偏差向量的元素全部为0. 289 | 290 | 偏差向量的元素全部为0浪费了存储空间, 我想到了有两种方法解决这个问题: 291 | 292 | 一种解决方法为删除全0的偏差向量, 另一种方法为利用该偏差向量对神经网络再进行一次修正, 弥补部分SVD分解时带来的损失. 293 | 294 | - 利用全零偏差向量弥补压缩的损失 295 | 296 | 对于某个全连接层, 有$$y = Wx + b$$ 297 | 298 | 进行部分SVD分解后, 得到y的估计$$y' = U_{m\times r}{\sum}_{r\times r}{V^T}_{r\times n}x + b$$ 299 | 300 | 令$$Z = {\sum}_{r\times r}{V^T}_{r\times n}$$, 将参数矩阵为W的全连接层分为两个更小的全连接层. 301 | 302 | 有$$y' = U_{m\times r}(Zx + g) + b$$ 303 | 304 | 其中g是参数矩阵为Z的全连接层的偏差向量 305 | 306 | 对于样本i, 存在误差$$err_i = \frac{1}{2}(y - y')^2 $$ 307 | 308 | 得到$$err_i = \frac{1}{2}[(W - U_{m\times r}Z)x_i + U_{m\times r}g]^2$$ 309 | 310 | 记$$D = W - U_{m\times r}$$, $$D$$不变 311 | 312 | 最小化$$J = \sum_{i = 1}^{m}err_i$$, 有$$g = {U_{m\times r}}^{-1}mean(Dx_i), i = 1,2,3,...,m $$ 313 | 314 | 经验证, 该校正方法计算出的$$g$$值很接近零矩阵, 对模型的准确度提升没有帮助. 315 | 316 | - 校正模型 317 | 318 | 对全连接层进行压缩后, 以原有的模型参数作为初始值, 用训练样本再训练一次模型. 319 | 320 | | r | accuracy | mean precision | mean recall | mean F | time | space | compression rate | 321 | | -------------- | -------- | -------------- | ----------- | -------- | ---- | ----- | ---------------- | 322 | | uncompressed | 0.751100 | 0.751777 | 0.751100 | 0.750558 | 640 | 640 | 100.00% | 323 | | 9(unimproved) | 0.716000 | 0.709909 | 0.716000 | 0.696003 | 666 | 666 | 104.06% | 324 | | 8(retrain4500) | 0.715300 | 0.714202 | 0.715300 | 0.712567 | 592 | 592 | 92.50% | 325 | | 8(improved500) | 0.712700 | 0.717100 | 0.712700 | 0.712591 | 592 | 592 | 92.50% | 326 | | 6(retrain4500) | 0.702100 | 0.704799 | 0.702100 | 0.700917 | 444 | 444 | 69.38% | 327 | | 6(improved500) | 0.693100 | 0.697976 | 0.693100 | 0.691144 | 444 | 444 | 69.38% | 328 | | 8(unimproved) | 0.690300 | 0.683415 | 0.690300 | 0.672081 | 592 | 592 | 92.50% | 329 | | 4(retrain4500) | 0.670500 | 0.672067 | 0.670500 | 0.669481 | 296 | 296 | 46.25% | 330 | | 4(improved500) | 0.657000 | 0.660422 | 0.657000 | 0.657608 | 296 | 296 | 46.25% | 331 | | 6(unimproved) | 0.613200 | 0.607457 | 0.613200 | 0.580367 | 444 | 444 | 69.38% | 332 | | 4(unimproved) | 0.518000 | 0.545117 | 0.518000 | 0.472613 | 296 | 296 | 46.25% | 333 | 334 | 其中, uncompressed代表未压缩的网络 335 | 336 | unimporved代表刚压缩的网络 337 | 338 | improved500代表网络压缩后, 基于当前的模型参数, 用训练样本训练500代; 339 | 340 | retrain4500代表网络压缩后, 重新初始化模型参数, 用训练样本训练4500代. 341 | 342 | 可以看到, improved和retrain模型的准确度相对于unimproved, 都得到了改进. 343 | 344 | 然而重新训练得到的模型的准确度比压缩后微调的模型的准确度更高. 345 | 346 | - 自动压缩全连接层 347 | 348 | 从"结果分析"中的列表可以看到, 在同等准确度的情况下, 压缩规模较大的全连接层能得到更好的压缩效果. 349 | 350 | 在SVD分解中, 奇异值越大, 对应的左奇异向量以及右奇异向量的重要性越强. 将奇异值从大到小排序后, 若较大的奇异值与较小的奇异值相差很远, 取较大的奇异值对应的部分就可以近似原矩阵. 351 | 352 | 对每一层的参数矩阵进行SVD分解, 奇异值从大到小排序后有$$[s_1, s_2, s_3, ..., s_m]$$ 353 | 354 | 若$$\frac{s_i}{s_{i+1}} > threshold$$, 则取r = i, 参数矩阵$$W_{m\times n}{\approx}U_{m\times r}{\sum}_{r\times r}{V^T}_{r\times n}$$ 355 | 356 | - 通过剪枝(pruning), 量化(quantization)和哈夫曼编码(Huffman Coding)压缩深度神经网络 357 | 358 | Song Han, Huizi Mao, William J. Dally. Deep Compression的论文中提到用剪枝, 量化, 哈夫曼编码的方法对深度神经网络进行压缩, 这些方法是在神经网络训练后进行的. 359 | 360 | - 剪枝 361 | 362 | 删除权重小于一定阈值的连接, 对稀疏连接进行再训练. 363 | 364 | - 量化 365 | 366 | 使用K-means算法将不同的权重聚类, 每个聚类共享一个权值, 在网络进行更新时, 将一个聚类内的所有梯度相加, 更新该聚类中的共享权值. 367 | 368 | - 哈夫曼编码 369 | 370 | 对不同的聚类进行哈夫曼编码. 371 | 372 | ## 参考资料 373 | 374 | [1] Zichao Yang,Marcin Moczulski,Misha Denil,Nando de Freitas,Alex Smola,Le Song,Ziyu Wang. Deep Fried Convnets. Arxiv, 1412.7149, 2015 375 | 376 | [2] Song Han, Huizi Mao, William J. Dally. Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding. Arxiv, 1510.00149. 2016. 377 | 378 | [3] LeftNotEasy, 机器学习中的数学(5)-强大的矩阵奇异值分解(SVD)及其应用, http://www.cnblogs.com/LeftNotEasy/archive/2011/01/19/svd-and-applications.html 379 | -------------------------------------------------------------------------------- /analyze.txt: -------------------------------------------------------------------------------- 1 | 分析ip1 SVD6 ipZ参数矩阵 2 | count 6144.000000 3 | mean 0.000260 4 | std 0.063288 5 | min -0.225779 6 | 25% -0.041734 7 | 50% -0.000137 8 | 75% 0.042516 9 | max 0.217526 10 | dtype: float64 11 | -------------------------------------------------------------------------------- /analyze_ipZ.py: -------------------------------------------------------------------------------- 1 | import pandas as pd 2 | import numpy as np 3 | import matplotlib.pyplot as plt 4 | z = np.load("ip1_SVD6_ipZ.npy") 5 | rows, cols = z.shape 6 | li = z.reshape(1,rows * cols).tolist()[0] 7 | s = pd.Series(li) 8 | print (s.describe()) 9 | plt.hist(li, bins = 100) 10 | plt.show() 11 | -------------------------------------------------------------------------------- /base.py: -------------------------------------------------------------------------------- 1 | import os 2 | import numpy as np 3 | 4 | def GetSVDProtoI(r, filename, template): 5 | if not os.path.exists(filename): 6 | fin = open(template, "r") 7 | fout = open(filename, "w") 8 | for line in fin.readlines(): 9 | fout.write(line.replace("$", str(r))) 10 | fin.close() 11 | fout.close() 12 | 13 | return filename 14 | 15 | def BuildFile(rs, filename, template): 16 | if True or not os.path.exists(filename): 17 | fin = open(template, "r") 18 | fout = open(filename, "w") 19 | for line in fin.readlines(): 20 | for fr, to in rs: 21 | line = line.replace(fr, to) 22 | fout.write(line) 23 | fin.close() 24 | fout.close() 25 | 26 | def GetSVDProto(r): 27 | filename = "./proto/SVD/cifar10_SVD%d.prototxt" % r 28 | template = "./proto/cifar10_SVD.template" 29 | return GetSVDProtoI(r, filename, template) 30 | def GetIP1SVDProto(r): 31 | filename = "./proto/SVD/cifar10_ip1_SVD%d.prototxt" % r 32 | template = "./proto/cifar10_ip1_SVD.template" 33 | return GetSVDProtoI(r, filename, template) 34 | 35 | def get_comfusion_matrix(label, pre, k): 36 | # TP, FN 37 | # FP, TN 38 | wlabel = (label == k) 39 | wpre = (pre == k) 40 | TP = np.sum(wlabel & wpre) 41 | FN = np.sum(wlabel & ~wpre) 42 | FP = np.sum(~wlabel & wpre) 43 | TN = np.sum(~wlabel & ~wpre) 44 | return np.matrix([[TP, FN], [FP, TN]]) 45 | 46 | def eval_result_k(label, pre, k): 47 | cm = get_comfusion_matrix(label, pre, k) 48 | print ("Confusion Matrix: TP, FN; FP, TN") 49 | print (cm) 50 | acc = (cm[0,0] + cm[1,1]) * 1.0 / np.sum(cm) 51 | precision = cm[0,0] * 1.0 / np.sum(cm[:, 0]) 52 | recall = cm[0,0] * 1.0 / np.sum(cm[0, :]) 53 | F = 2.0 * recall * precision / (recall + precision) 54 | print ("Accuracy: %f" % acc, "Precision: %f" % precision) 55 | print ("Recall: %f" % recall, "F-measure: %f" % F) 56 | return acc, precision, recall, F 57 | 58 | def eval_result(label, pre, num_kinds): 59 | n = len(label) 60 | s_acc = 0.0 61 | s_precision = 0.0 62 | s_recall = 0.0 63 | s_F = 0.0 64 | for k in range(num_kinds): 65 | print ("Kind: %d" % k) 66 | acc, precision, recall, F = eval_result_k(label, pre, k) 67 | s_acc += acc 68 | s_precision += precision 69 | s_recall += recall 70 | s_F += F 71 | print ("========") 72 | 73 | acc = np.sum(pre == label) * 1.0 / n 74 | mean_acc = s_acc / num_kinds 75 | mean_precision = s_precision / num_kinds 76 | mean_recall = s_recall / num_kinds 77 | mean_F = s_F / num_kinds 78 | print ("Totally:") 79 | print ("Accuracy: %f" % acc) 80 | print ("Mean Accuracy: %f" % (mean_acc)) 81 | print ("Mean Precision: %f" % (mean_precision)) 82 | print ("Mean Recall: %f" % (mean_recall)) 83 | print ("Mean F: %f" % (mean_F)) 84 | print ("%f|%f|%f|%f" % (acc, mean_precision, mean_recall, mean_F)) 85 | return acc, mean_acc, mean_precision, mean_recall, mean_F 86 | -------------------------------------------------------------------------------- /build.sh: -------------------------------------------------------------------------------- 1 | sudo optirun python2 test_ip1_new.py 2 | -------------------------------------------------------------------------------- /build/solver.prototxt: -------------------------------------------------------------------------------- 1 | # reduce the learning rate after 8 epochs (4000 iters) by a factor of 10 2 | 3 | # The train/test net protocol buffer definition 4 | net: "./build/train_test_SVD.prototext" 5 | # test_iter specifies how many forward passes the test should carry out. 6 | # In the case of MNIST, we have test batch size 100 and 100 test iterations, 7 | # covering the full 10,000 testing images. 8 | test_iter: 100 9 | # Carry out testing every 500 training iterations. 10 | test_interval: 500 11 | # The base learning rate, momentum and the weight decay of the network. 12 | base_lr: 0.001 13 | momentum: 0.9 14 | weight_decay: 0.004 15 | # The learning rate policy 16 | lr_policy: "fixed" 17 | # Display every 100 iterations 18 | display: 100 19 | # The maximum number of iterations 20 | max_iter: 20000 21 | # snapshot intermediate results 22 | snapshot: 500 23 | snapshot_format: HDF5 24 | snapshot_prefix: "ip2_SVD4" 25 | # solver mode: CPU or GPU 26 | solver_mode: GPU 27 | -------------------------------------------------------------------------------- /build/train_imp.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env sh 2 | set -e 3 | 4 | TOOLS=/home/luwei1/caffe/build/tools 5 | 6 | $TOOLS/caffe train \ 7 | --solver=./build/solver.prototxt \ 8 | --weights ./build/ip2_SVD4.caffemodel $@ 9 | -------------------------------------------------------------------------------- /build/train_test_SVD.prototext: -------------------------------------------------------------------------------- 1 | name: "CIFAR10_quick" 2 | layer { 3 | name: "cifar" 4 | type: "Data" 5 | top: "data" 6 | top: "label" 7 | include { 8 | phase: TRAIN 9 | } 10 | transform_param { 11 | mean_file: "/home/luwei1/caffe/examples/cifar10/mean.binaryproto" 12 | } 13 | data_param { 14 | source: "/home/luwei1/caffe/examples/cifar10/cifar10_train_lmdb" 15 | batch_size: 100 16 | backend: LMDB 17 | } 18 | } 19 | layer { 20 | name: "cifar" 21 | type: "Data" 22 | top: "data" 23 | top: "label" 24 | include { 25 | phase: TEST 26 | } 27 | transform_param { 28 | mean_file: "/home/luwei1/caffe/examples/cifar10/mean.binaryproto" 29 | } 30 | data_param { 31 | source: "/home/luwei1/caffe/examples/cifar10/cifar10_test_lmdb" 32 | batch_size: 100 33 | backend: LMDB 34 | } 35 | } 36 | layer { 37 | name: "conv1" 38 | type: "Convolution" 39 | bottom: "data" 40 | top: "conv1" 41 | param { 42 | lr_mult: 1 43 | } 44 | param { 45 | lr_mult: 2 46 | } 47 | convolution_param { 48 | num_output: 32 49 | pad: 2 50 | kernel_size: 5 51 | stride: 1 52 | weight_filler { 53 | type: "gaussian" 54 | std: 0.0001 55 | } 56 | bias_filler { 57 | type: "constant" 58 | } 59 | } 60 | } 61 | layer { 62 | name: "pool1" 63 | type: "Pooling" 64 | bottom: "conv1" 65 | top: "pool1" 66 | pooling_param { 67 | pool: MAX 68 | kernel_size: 3 69 | stride: 2 70 | } 71 | } 72 | layer { 73 | name: "relu1" 74 | type: "ReLU" 75 | bottom: "pool1" 76 | top: "pool1" 77 | } 78 | layer { 79 | name: "conv2" 80 | type: "Convolution" 81 | bottom: "pool1" 82 | top: "conv2" 83 | param { 84 | lr_mult: 1 85 | } 86 | param { 87 | lr_mult: 2 88 | } 89 | convolution_param { 90 | num_output: 32 91 | pad: 2 92 | kernel_size: 5 93 | stride: 1 94 | weight_filler { 95 | type: "gaussian" 96 | std: 0.01 97 | } 98 | bias_filler { 99 | type: "constant" 100 | } 101 | } 102 | } 103 | layer { 104 | name: "relu2" 105 | type: "ReLU" 106 | bottom: "conv2" 107 | top: "conv2" 108 | } 109 | layer { 110 | name: "pool2" 111 | type: "Pooling" 112 | bottom: "conv2" 113 | top: "pool2" 114 | pooling_param { 115 | pool: AVE 116 | kernel_size: 3 117 | stride: 2 118 | } 119 | } 120 | layer { 121 | name: "conv3" 122 | type: "Convolution" 123 | bottom: "pool2" 124 | top: "conv3" 125 | param { 126 | lr_mult: 1 127 | } 128 | param { 129 | lr_mult: 2 130 | } 131 | convolution_param { 132 | num_output: 64 133 | pad: 2 134 | kernel_size: 5 135 | stride: 1 136 | weight_filler { 137 | type: "gaussian" 138 | std: 0.01 139 | } 140 | bias_filler { 141 | type: "constant" 142 | } 143 | } 144 | } 145 | layer { 146 | name: "relu3" 147 | type: "ReLU" 148 | bottom: "conv3" 149 | top: "conv3" 150 | } 151 | layer { 152 | name: "pool3" 153 | type: "Pooling" 154 | bottom: "conv3" 155 | top: "pool3" 156 | pooling_param { 157 | pool: AVE 158 | kernel_size: 3 159 | stride: 2 160 | } 161 | } 162 | layer { 163 | name: "ip1" 164 | type: "InnerProduct" 165 | bottom: "pool3" 166 | top: "ip1" 167 | param { 168 | lr_mult: 1 169 | } 170 | param { 171 | lr_mult: 2 172 | } 173 | inner_product_param { 174 | num_output: 64 175 | weight_filler { 176 | type: "gaussian" 177 | std: 0.1 178 | } 179 | bias_filler { 180 | type: "constant" 181 | } 182 | } 183 | } 184 | 185 | layer { 186 | name: "ipZ" 187 | type: "InnerProduct" 188 | bottom: "ip1" 189 | top: "ipZ" 190 | param { 191 | lr_mult: 1 192 | } 193 | param { 194 | lr_mult: 2 195 | } 196 | inner_product_param{ 197 | num_output: 4 198 | 199 | weight_filler { 200 | type: "gaussian" 201 | std: 0.1 202 | } 203 | bias_filler { 204 | type: "constant" 205 | } 206 | } 207 | } 208 | 209 | layer { 210 | name: "ipU" 211 | type: "InnerProduct" 212 | bottom: "ipZ" 213 | top: "ipSVD" 214 | param { 215 | lr_mult: 1 216 | } 217 | param { 218 | lr_mult: 2 219 | } 220 | inner_product_param{ 221 | num_output: 10 222 | 223 | weight_filler { 224 | type: "gaussian" 225 | std: 0.1 226 | } 227 | bias_filler { 228 | type: "constant" 229 | } 230 | } 231 | } 232 | 233 | layer { 234 | name: "accuracy" 235 | type: "Accuracy" 236 | bottom: "ipSVD" 237 | bottom: "label" 238 | top: "accuracy" 239 | include { 240 | phase: TEST 241 | } 242 | } 243 | layer { 244 | name: "loss" 245 | type: "SoftmaxWithLoss" 246 | bottom: "ipSVD" 247 | bottom: "label" 248 | top: "loss" 249 | } 250 | -------------------------------------------------------------------------------- /build_ip1/solver.prototxt: -------------------------------------------------------------------------------- 1 | # reduce the learning rate after 8 epochs (4000 iters) by a factor of 10 2 | 3 | # The train/test net protocol buffer definition 4 | net: "./build_ip1/train_test_SVD_ip1.prototext" 5 | # test_iter specifies how many forward passes the test should carry out. 6 | # In the case of MNIST, we have test batch size 100 and 100 test iterations, 7 | # covering the full 10,000 testing images. 8 | test_iter: 100 9 | # Carry out testing every 500 training iterations. 10 | test_interval: 500 11 | # The base learning rate, momentum and the weight decay of the network. 12 | base_lr: 0.001 13 | momentum: 0.9 14 | weight_decay: 0.004 15 | # The learning rate policy 16 | lr_policy: "fixed" 17 | # Display every 100 iterations 18 | display: 100 19 | # The maximum number of iterations 20 | max_iter: 20000 21 | # snapshot intermediate results 22 | snapshot: 500 23 | snapshot_format: HDF5 24 | snapshot_prefix: "ip1_SVD6" 25 | # solver mode: CPU or GPU 26 | solver_mode: GPU 27 | -------------------------------------------------------------------------------- /build_ip1/solver_conv3x3.prototxt: -------------------------------------------------------------------------------- 1 | # reduce the learning rate after 8 epochs (4000 iters) by a factor of 10 2 | 3 | # The train/test net protocol buffer definition 4 | net: "./build_ip1/train_test_SVD_ip1_conv3x3.prototxt" 5 | # test_iter specifies how many forward passes the test should carry out. 6 | # In the case of MNIST, we have test batch size 100 and 100 test iterations, 7 | # covering the full 10,000 testing images. 8 | test_iter: 100 9 | # Carry out testing every 500 training iterations. 10 | test_interval: 500 11 | # The base learning rate, momentum and the weight decay of the network. 12 | base_lr: 0.001 13 | momentum: 0.9 14 | weight_decay: 0.004 15 | # The learning rate policy 16 | lr_policy: "fixed" 17 | # Display every 100 iterations 18 | display: 100 19 | # The maximum number of iterations 20 | max_iter: 20000 21 | # snapshot intermediate results 22 | snapshot: 500 23 | snapshot_format: HDF5 24 | snapshot_prefix: "ip1_SVD6_conv3x3" 25 | # solver mode: CPU or GPU 26 | solver_mode: GPU 27 | -------------------------------------------------------------------------------- /build_ip1/train_imp.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env sh 2 | set -e 3 | 4 | TOOLS=/home/luwei1/caffe/build/tools 5 | 6 | $TOOLS/caffe train \ 7 | --solver=./build_ip1/solver.prototxt \ 8 | --weights ./build_ip1/ip1_SVD6.caffemodel $@ 9 | -------------------------------------------------------------------------------- /build_ip1/train_imp_conv3x3.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env sh 2 | set -e 3 | 4 | TOOLS=/home/luwei1/caffe/build/tools 5 | 6 | $TOOLS/caffe train \ 7 | --solver=./build_ip1/solver_conv3x3.prototxt \ 8 | --weights ./build_ip1/ip1_SVD6_iter_20000.caffemodel.h5 $@ 9 | -------------------------------------------------------------------------------- /build_ip1/train_test_SVD_ip1.prototext: -------------------------------------------------------------------------------- 1 | name: "CIFAR10_quick" 2 | layer { 3 | name: "cifar" 4 | type: "Data" 5 | top: "data" 6 | top: "label" 7 | include { 8 | phase: TRAIN 9 | } 10 | transform_param { 11 | mean_file: "/home/luwei1/caffe/examples/cifar10/mean.binaryproto" 12 | } 13 | data_param { 14 | source: "/home/luwei1/caffe/examples/cifar10/cifar10_train_lmdb" 15 | batch_size: 100 16 | backend: LMDB 17 | } 18 | } 19 | layer { 20 | name: "cifar" 21 | type: "Data" 22 | top: "data" 23 | top: "label" 24 | include { 25 | phase: TEST 26 | } 27 | transform_param { 28 | mean_file: "/home/luwei1/caffe/examples/cifar10/mean.binaryproto" 29 | } 30 | data_param { 31 | source: "/home/luwei1/caffe/examples/cifar10/cifar10_test_lmdb" 32 | batch_size: 100 33 | backend: LMDB 34 | } 35 | } 36 | layer { 37 | name: "conv1" 38 | type: "Convolution" 39 | bottom: "data" 40 | top: "conv1" 41 | param { 42 | lr_mult: 1 43 | } 44 | param { 45 | lr_mult: 2 46 | } 47 | convolution_param { 48 | num_output: 32 49 | pad: 2 50 | kernel_size: 5 51 | stride: 1 52 | weight_filler { 53 | type: "gaussian" 54 | std: 0.0001 55 | } 56 | bias_filler { 57 | type: "constant" 58 | } 59 | } 60 | } 61 | layer { 62 | name: "pool1" 63 | type: "Pooling" 64 | bottom: "conv1" 65 | top: "pool1" 66 | pooling_param { 67 | pool: MAX 68 | kernel_size: 3 69 | stride: 2 70 | } 71 | } 72 | layer { 73 | name: "relu1" 74 | type: "ReLU" 75 | bottom: "pool1" 76 | top: "pool1" 77 | } 78 | layer { 79 | name: "conv2" 80 | type: "Convolution" 81 | bottom: "pool1" 82 | top: "conv2" 83 | param { 84 | lr_mult: 1 85 | } 86 | param { 87 | lr_mult: 2 88 | } 89 | convolution_param { 90 | num_output: 32 91 | pad: 2 92 | kernel_size: 5 93 | stride: 1 94 | weight_filler { 95 | type: "gaussian" 96 | std: 0.01 97 | } 98 | bias_filler { 99 | type: "constant" 100 | } 101 | } 102 | } 103 | layer { 104 | name: "relu2" 105 | type: "ReLU" 106 | bottom: "conv2" 107 | top: "conv2" 108 | } 109 | layer { 110 | name: "pool2" 111 | type: "Pooling" 112 | bottom: "conv2" 113 | top: "pool2" 114 | pooling_param { 115 | pool: AVE 116 | kernel_size: 3 117 | stride: 2 118 | } 119 | } 120 | layer { 121 | name: "conv3" 122 | type: "Convolution" 123 | bottom: "pool2" 124 | top: "conv3" 125 | param { 126 | lr_mult: 1 127 | } 128 | param { 129 | lr_mult: 2 130 | } 131 | convolution_param { 132 | num_output: 64 133 | pad: 2 134 | kernel_size: 5 135 | stride: 1 136 | weight_filler { 137 | type: "gaussian" 138 | std: 0.01 139 | } 140 | bias_filler { 141 | type: "constant" 142 | } 143 | } 144 | } 145 | layer { 146 | name: "relu3" 147 | type: "ReLU" 148 | bottom: "conv3" 149 | top: "conv3" 150 | } 151 | layer { 152 | name: "pool3" 153 | type: "Pooling" 154 | bottom: "conv3" 155 | top: "pool3" 156 | pooling_param { 157 | pool: AVE 158 | kernel_size: 3 159 | stride: 2 160 | } 161 | } 162 | 163 | 164 | layer { 165 | name: "ipZ" 166 | type: "InnerProduct" 167 | bottom: "pool3" 168 | top: "ipZ" 169 | 170 | param { 171 | lr_mult: 1 172 | } 173 | param { 174 | lr_mult: 2 175 | } 176 | 177 | inner_product_param{ 178 | num_output: 6 179 | 180 | weight_filler { 181 | type: "gaussian" 182 | std: 0.1 183 | } 184 | bias_filler { 185 | type: "constant" 186 | } 187 | } 188 | } 189 | 190 | layer { 191 | name: "ipU" 192 | type: "InnerProduct" 193 | bottom: "ipZ" 194 | top: "ipSVD" 195 | param { 196 | lr_mult: 1 197 | } 198 | param { 199 | lr_mult: 2 200 | } 201 | inner_product_param{ 202 | num_output: 64 203 | weight_filler { 204 | type: "gaussian" 205 | std: 0.1 206 | } 207 | bias_filler { 208 | type: "constant" 209 | } 210 | } 211 | } 212 | 213 | layer { 214 | name: "ip2" 215 | type: "InnerProduct" 216 | bottom: "ipSVD" 217 | top: "ip2" 218 | param { 219 | lr_mult: 1 220 | } 221 | param { 222 | lr_mult: 2 223 | } 224 | inner_product_param { 225 | num_output: 10 226 | weight_filler { 227 | type: "gaussian" 228 | std: 0.1 229 | } 230 | bias_filler { 231 | type: "constant" 232 | } 233 | } 234 | } 235 | layer { 236 | name: "accuracy" 237 | type: "Accuracy" 238 | bottom: "ip2" 239 | bottom: "label" 240 | top: "accuracy" 241 | include { 242 | phase: TEST 243 | } 244 | } 245 | layer { 246 | name: "loss" 247 | type: "SoftmaxWithLoss" 248 | bottom: "ip2" 249 | bottom: "label" 250 | top: "loss" 251 | } 252 | -------------------------------------------------------------------------------- /build_ip1/train_test_SVD_ip1_conv3x3.prototxt: -------------------------------------------------------------------------------- 1 | name: "CIFAR10_quick" 2 | layer { 3 | name: "cifar" 4 | type: "Data" 5 | top: "data" 6 | top: "label" 7 | include { 8 | phase: TRAIN 9 | } 10 | transform_param { 11 | mean_file: "/home/luwei1/caffe/examples/cifar10/mean.binaryproto" 12 | } 13 | data_param { 14 | source: "/home/luwei1/caffe/examples/cifar10/cifar10_train_lmdb" 15 | batch_size: 100 16 | backend: LMDB 17 | } 18 | } 19 | layer { 20 | name: "cifar" 21 | type: "Data" 22 | top: "data" 23 | top: "label" 24 | include { 25 | phase: TEST 26 | } 27 | transform_param { 28 | mean_file: "/home/luwei1/caffe/examples/cifar10/mean.binaryproto" 29 | } 30 | data_param { 31 | source: "/home/luwei1/caffe/examples/cifar10/cifar10_test_lmdb" 32 | batch_size: 100 33 | backend: LMDB 34 | } 35 | } 36 | 37 | layer { 38 | name: "conv1_1" 39 | type: "Convolution" 40 | bottom: "data" 41 | top: "conv1_1" 42 | param { 43 | lr_mult: 1 44 | } 45 | param { 46 | lr_mult: 2 47 | } 48 | convolution_param { 49 | num_output: 32 50 | pad: 1 51 | kernel_size: 3 52 | stride: 1 53 | weight_filler { 54 | type: "gaussian" 55 | std: 0.0001 56 | } 57 | bias_filler { 58 | type: "constant" 59 | } 60 | } 61 | } 62 | 63 | layer { 64 | name: "conv1_2" 65 | type: "Convolution" 66 | bottom: "conv1_1" 67 | top: "conv1" 68 | param { 69 | lr_mult: 1 70 | } 71 | param { 72 | lr_mult: 2 73 | } 74 | convolution_param { 75 | num_output: 32 76 | pad: 1 77 | kernel_size: 3 78 | stride: 1 79 | weight_filler { 80 | type: "gaussian" 81 | std: 0.0001 82 | } 83 | bias_filler { 84 | type: "constant" 85 | } 86 | } 87 | } 88 | 89 | layer { 90 | name: "pool1" 91 | type: "Pooling" 92 | bottom: "conv1" 93 | top: "pool1" 94 | pooling_param { 95 | pool: MAX 96 | kernel_size: 3 97 | stride: 2 98 | } 99 | } 100 | layer { 101 | name: "relu1" 102 | type: "ReLU" 103 | bottom: "pool1" 104 | top: "pool1" 105 | } 106 | 107 | layer { 108 | name: "conv2_1" 109 | type: "Convolution" 110 | bottom: "pool1" 111 | top: "conv2_1" 112 | param { 113 | lr_mult: 1 114 | } 115 | param { 116 | lr_mult: 2 117 | } 118 | convolution_param { 119 | num_output: 32 120 | pad: 1 121 | kernel_size: 3 122 | stride: 1 123 | weight_filler { 124 | type: "gaussian" 125 | std: 0.01 126 | } 127 | bias_filler { 128 | type: "constant" 129 | } 130 | } 131 | } 132 | 133 | layer { 134 | name: "conv2_2" 135 | type: "Convolution" 136 | bottom: "conv2_1" 137 | top: "conv2" 138 | param { 139 | lr_mult: 1 140 | } 141 | param { 142 | lr_mult: 2 143 | } 144 | convolution_param { 145 | num_output: 32 146 | pad: 1 147 | kernel_size: 3 148 | stride: 1 149 | weight_filler { 150 | type: "gaussian" 151 | std: 0.01 152 | } 153 | bias_filler { 154 | type: "constant" 155 | } 156 | } 157 | } 158 | 159 | layer { 160 | name: "relu2" 161 | type: "ReLU" 162 | bottom: "conv2" 163 | top: "conv2" 164 | } 165 | layer { 166 | name: "pool2" 167 | type: "Pooling" 168 | bottom: "conv2" 169 | top: "pool2" 170 | pooling_param { 171 | pool: AVE 172 | kernel_size: 3 173 | stride: 2 174 | } 175 | } 176 | 177 | layer { 178 | name: "conv3_1" 179 | type: "Convolution" 180 | bottom: "pool2" 181 | top: "conv3_1" 182 | param { 183 | lr_mult: 1 184 | } 185 | param { 186 | lr_mult: 2 187 | } 188 | convolution_param { 189 | num_output: 64 190 | pad: 1 191 | kernel_size: 3 192 | stride: 1 193 | weight_filler { 194 | type: "gaussian" 195 | std: 0.01 196 | } 197 | bias_filler { 198 | type: "constant" 199 | } 200 | } 201 | } 202 | 203 | layer { 204 | name: "conv3_2" 205 | type: "Convolution" 206 | bottom: "conv3_1" 207 | top: "conv3" 208 | param { 209 | lr_mult: 1 210 | } 211 | param { 212 | lr_mult: 2 213 | } 214 | convolution_param { 215 | num_output: 64 216 | pad: 1 217 | kernel_size: 3 218 | stride: 1 219 | weight_filler { 220 | type: "gaussian" 221 | std: 0.01 222 | } 223 | bias_filler { 224 | type: "constant" 225 | } 226 | } 227 | } 228 | 229 | layer { 230 | name: "relu3" 231 | type: "ReLU" 232 | bottom: "conv3" 233 | top: "conv3" 234 | } 235 | layer { 236 | name: "pool3" 237 | type: "Pooling" 238 | bottom: "conv3" 239 | top: "pool3" 240 | pooling_param { 241 | pool: AVE 242 | kernel_size: 3 243 | stride: 2 244 | } 245 | } 246 | 247 | 248 | layer { 249 | name: "ipZ" 250 | type: "InnerProduct" 251 | bottom: "pool3" 252 | top: "ipZ" 253 | 254 | param { 255 | lr_mult: 1 256 | } 257 | param { 258 | lr_mult: 2 259 | } 260 | 261 | inner_product_param{ 262 | num_output: 6 263 | 264 | weight_filler { 265 | type: "gaussian" 266 | std: 0.1 267 | } 268 | bias_filler { 269 | type: "constant" 270 | } 271 | } 272 | } 273 | 274 | layer { 275 | name: "ipU" 276 | type: "InnerProduct" 277 | bottom: "ipZ" 278 | top: "ipSVD" 279 | param { 280 | lr_mult: 1 281 | } 282 | param { 283 | lr_mult: 2 284 | } 285 | inner_product_param{ 286 | num_output: 64 287 | weight_filler { 288 | type: "gaussian" 289 | std: 0.1 290 | } 291 | bias_filler { 292 | type: "constant" 293 | } 294 | } 295 | } 296 | 297 | layer { 298 | name: "ip2" 299 | type: "InnerProduct" 300 | bottom: "ipSVD" 301 | top: "ip2" 302 | param { 303 | lr_mult: 1 304 | } 305 | param { 306 | lr_mult: 2 307 | } 308 | inner_product_param { 309 | num_output: 10 310 | weight_filler { 311 | type: "gaussian" 312 | std: 0.1 313 | } 314 | bias_filler { 315 | type: "constant" 316 | } 317 | } 318 | } 319 | layer { 320 | name: "accuracy" 321 | type: "Accuracy" 322 | bottom: "ip2" 323 | bottom: "label" 324 | top: "accuracy" 325 | include { 326 | phase: TEST 327 | } 328 | } 329 | layer { 330 | name: "loss" 331 | type: "SoftmaxWithLoss" 332 | bottom: "ip2" 333 | bottom: "label" 334 | top: "loss" 335 | } 336 | -------------------------------------------------------------------------------- /chart.py: -------------------------------------------------------------------------------- 1 | from base import * 2 | import matplotlib.pyplot as plt 3 | # 显示压缩ip2层的对比结果 4 | 5 | label = np.load("label.npy") 6 | n = len(label) 7 | num_kinds = 10 8 | # ip2 layer: 10 x 64 9 | accs = [] 10 | maccs = [] 11 | mpres = [] 12 | mrecalls = [] 13 | mfs = [] 14 | 15 | # A(nxw) * B(wxm), time:O(nwm) space:O(nw+wm) 16 | # IP2(10 x 64) * x(64x1) 17 | # Time:O(640) Space:O(640) # exclude x 18 | # W = U(10xr) * S(rxr) * VT(rx64) 19 | # U(10xr) * Z(rx64) * x(64 x 1) 20 | # Time: O(r*64 + 10 * r) = O(74r) 21 | # Space: O(10r + 64r) = O(74r) 22 | SOURCE_SIZE = 640 23 | 24 | 25 | t = [i for i in range(3, 11)] 26 | time_complex = [74 * r for r in t] 27 | print ("Time Complexity:") 28 | print (time_complex) 29 | space_complex = [74 * r for r in t] 30 | # 0 - 1 31 | time_complex_n = np.array(time_complex) * 1.0 / max(time_complex) 32 | space_complex_n = np.array(space_complex) * 1.0 / max(space_complex) 33 | for r in t: 34 | name = "result/net_SVD%d.npy" % r 35 | pre = np.load(name) 36 | acc, mean_acc, mean_precision, mean_recall, mean_F = eval_result(label, pre, num_kinds) 37 | accs.append(acc) 38 | maccs.append(mean_acc) 39 | mpres.append(mean_precision) 40 | mrecalls.append(mean_recall) 41 | mfs.append(mean_F) 42 | 43 | acc, mean_acc, mean_precision, mean_recall, mean_F = eval_result(label, np.load("result/net_normal.npy"), num_kinds) 44 | print ("r\taccuracy\tmean_precision\tmean_recall\tmean_F\t\ttime\tspace\tcompression_rate") 45 | print ("source\t%f\t%f\t%f\t%f\t%d\t%d\t%.2f%%" % (acc, mean_precision, mean_recall,mean_F, 640, 640, 100.0)) 46 | for i in range(len(t)-1,-1,-1): 47 | print ("%d\t%f\t%f\t%f\t%f\t%d\t%d\t%.2f%%" % (t[i], accs[i], mpres[i], mrecalls[i],mfs[i], time_complex[i], space_complex[i], space_complex[i] * 100.0 / SOURCE_SIZE)) 48 | 49 | plt.title("accuracy(totally), time and space complexity") 50 | plt.plot(t, accs, "g", label = "accuracy", linestyle="-") 51 | plt.plot(t, time_complex_n, "r", label = "time complexity", linestyle="--") 52 | plt.plot(t, space_complex_n, "k", label = "space complexity", linestyle="-.") 53 | plt.legend(loc = "upper left") 54 | plt.show() 55 | 56 | plt.title("statistics") 57 | plt.plot(t, accs, "r", label = "accuracy", linestyle="-") 58 | #plt.plot(t, maccs, "r", label = "mean_accuracy") 59 | plt.plot(t, mpres, "g", label = "mean_precision", linestyle="--") 60 | plt.plot(t, mrecalls, "b", label = "mean_recall",linestyle="-.") 61 | plt.plot(t, mfs, "y", label = "mean_F", linestyle=":") 62 | plt.legend(loc = "upper left") 63 | 64 | plt.show() 65 | -------------------------------------------------------------------------------- /chart2.py: -------------------------------------------------------------------------------- 1 | from base import * 2 | import matplotlib.pyplot as plt 3 | # 显示压缩ip1层的对比结果 4 | 5 | label = np.load("label.npy") 6 | n = len(label) 7 | num_kinds = 10 8 | # ip1 layer: 64 x 1024 9 | accs = [] 10 | maccs = [] 11 | mpres = [] 12 | mrecalls = [] 13 | mfs = [] 14 | 15 | # A(nxw) * B(wxm), time:O(nwm) space:O(nw+wm) 16 | rows = 64 17 | cols = 1024 18 | SOURCE_SIZE = rows * cols 19 | 20 | 21 | st = [i for i in range(1, 65)] 22 | t = [] 23 | time_complex = [] 24 | space_complex = [] 25 | for r in range(1, min(rows, cols) + 1): 26 | name = "result/net_ip1_SVD%d.npy" % r 27 | if os.path.exists(name): 28 | pre = np.load(name) 29 | acc, mean_acc, mean_precision, mean_recall, mean_F = eval_result(label, pre, num_kinds) 30 | accs.append(acc) 31 | maccs.append(mean_acc) 32 | mpres.append(mean_precision) 33 | mrecalls.append(mean_recall) 34 | mfs.append(mean_F) 35 | t.append(r) 36 | time_complex.append((rows + cols) * r) 37 | space_complex.append((rows + cols) * r) 38 | # 0 - 1 39 | time_complex_n = np.array(time_complex) * 1.0 / max(time_complex) 40 | space_complex_n = np.array(space_complex) * 1.0 / max(space_complex) 41 | 42 | acc, mean_acc, mean_precision, mean_recall, mean_F = eval_result(label, np.load("result/net_normal.npy"), num_kinds) 43 | print ("r|\taccuracy|\tmean_precision|\tmean_recall|\tmean_F|\t|\ttime|\tspace|\tcompression_rate") 44 | print ("---|---|---|---|---|---|---|---|---") 45 | print ("source|\t%f|\t%f|\t%f|\t%f|\t%d|\t%d|\t%.2f%%" % (acc, mean_precision, mean_recall,mean_F, SOURCE_SIZE, SOURCE_SIZE, 100.0)) 46 | for i in range(len(t)-1,-1,-1): 47 | print ("%d|\t%f|\t%f|\t%f|\t%f|\t%d|\t%d|\t%.2f%%" % (t[i], accs[i], mpres[i], mrecalls[i],mfs[i], time_complex[i], space_complex[i], space_complex[i] * 100.0 / SOURCE_SIZE)) 48 | 49 | plt.title("accuracy(totally), time and space complexity") 50 | plt.plot(t, accs, "g", label = "accuracy", linestyle="-") 51 | plt.plot(t, time_complex_n, "r", label = "time complexity", linestyle="--") 52 | plt.plot(t, space_complex_n, "k", label = "space complexity", linestyle="-.") 53 | plt.legend(loc = "upper left") 54 | plt.show() 55 | 56 | plt.title("statistics") 57 | plt.plot(t, accs, "r", label = "accuracy", linestyle="-") 58 | #plt.plot(t, maccs, "r", label = "mean_accuracy") 59 | plt.plot(t, mpres, "g", label = "mean_precision", linestyle="--") 60 | plt.plot(t, mrecalls, "b", label = "mean_recall",linestyle="-.") 61 | plt.plot(t, mfs, "y", label = "mean_F", linestyle=":") 62 | plt.legend(loc = "upper left") 63 | 64 | plt.show() 65 | -------------------------------------------------------------------------------- /chart_imp.py: -------------------------------------------------------------------------------- 1 | from base import * 2 | import matplotlib.pyplot as plt 3 | 4 | label = np.load("label.npy") 5 | n = len(label) 6 | num_kinds = 10 7 | # ip1 layer: 64 x 1024 8 | accs = [] 9 | maccs = [] 10 | mpres = [] 11 | mrecalls = [] 12 | mfs = [] 13 | 14 | # A(nxw) * B(wxm), time:O(nwm) space:O(nw+wm) 15 | rows = 64 16 | cols = 1024 17 | SOURCE_SIZE = rows * cols 18 | 19 | 20 | st = [i for i in range(1, 65)] 21 | t = [] 22 | time_complex = [] 23 | space_complex = [] 24 | for r in range(1, min(rows, cols) + 1): 25 | name = "result/net_SVD%d_imp.npy" % r 26 | if os.path.exists(name): 27 | pre = np.load(name) 28 | acc, mean_acc, mean_precision, mean_recall, mean_F = eval_result(label, pre, num_kinds) 29 | accs.append(acc) 30 | maccs.append(mean_acc) 31 | mpres.append(mean_precision) 32 | mrecalls.append(mean_recall) 33 | mfs.append(mean_F) 34 | t.append(r) 35 | time_complex.append((rows + cols) * r) 36 | space_complex.append((rows + cols) * r) 37 | # 0 - 1 38 | time_complex_n = np.array(time_complex) * 1.0 / max(time_complex) 39 | space_complex_n = np.array(space_complex) * 1.0 / max(space_complex) 40 | 41 | acc, mean_acc, mean_precision, mean_recall, mean_F = eval_result(label, np.load("result/net_normal.npy"), num_kinds) 42 | print ("r|\taccuracy|\tmean_precision|\tmean_recall|\tmean_F|\t|\ttime|\tspace|\tcompression_rate") 43 | print ("---|---|---|---|---|---|---|---|---") 44 | print ("source|\t%f|\t%f|\t%f|\t%f|\t%d|\t%d|\t%.2f%%" % (acc, mean_precision, mean_recall,mean_F, SOURCE_SIZE, SOURCE_SIZE, 100.0)) 45 | for i in range(len(t)-1,-1,-1): 46 | print ("%d|\t%f|\t%f|\t%f|\t%f|\t%d|\t%d|\t%.2f%%" % (t[i], accs[i], mpres[i], mrecalls[i],mfs[i], time_complex[i], space_complex[i], space_complex[i] * 100.0 / SOURCE_SIZE)) 47 | 48 | plt.title("accuracy(totally), time and space complexity") 49 | plt.plot(t, accs, "g", label = "accuracy", linestyle="-") 50 | plt.plot(t, time_complex_n, "r", label = "time complexity", linestyle="--") 51 | plt.plot(t, space_complex_n, "k", label = "space complexity", linestyle="-.") 52 | plt.legend(loc = "upper left") 53 | plt.show() 54 | 55 | plt.title("statistics") 56 | plt.plot(t, accs, "r", label = "accuracy", linestyle="-") 57 | #plt.plot(t, maccs, "r", label = "mean_accuracy") 58 | plt.plot(t, mpres, "g", label = "mean_precision", linestyle="--") 59 | plt.plot(t, mrecalls, "b", label = "mean_recall",linestyle="-.") 60 | plt.plot(t, mfs, "y", label = "mean_F", linestyle=":") 61 | plt.legend(loc = "upper left") 62 | 63 | plt.show() 64 | -------------------------------------------------------------------------------- /convert_mean.py: -------------------------------------------------------------------------------- 1 | import caffe 2 | import sys 3 | import numpy as np 4 | if len(sys.argv) != 3: 5 | print ("convert_mean.py mean.binaryproto mean.npy") 6 | sys.exit() 7 | 8 | blob = caffe.proto.caffe_pb2.BlobProto() 9 | bin_mean = open(sys.argv[1], 'rb').read() 10 | blob.ParseFromString(bin_mean) 11 | arr = np.array(caffe.io.blobproto_to_array(blob)) 12 | npy_mean = arr[0] 13 | print (npy_mean.shape) 14 | np.save(sys.argv[2], npy_mean) 15 | -------------------------------------------------------------------------------- /eval_model_ip1.py: -------------------------------------------------------------------------------- 1 | #coding=utf-8 2 | # test decompress ip1 layer 3 | import caffe 4 | from caffe import layers as L, params as P, to_proto 5 | from caffe.proto import caffe_pb2 6 | import lmdb 7 | import numpy as np 8 | import os 9 | import sys 10 | from numpy import linalg as la 11 | import matplotlib.pyplot as plt 12 | from base import * 13 | 14 | CAFFE_HOME = "/opt/caffe/" 15 | RESULT_DIR = "./result/" 16 | 17 | SVD_R = 6 # 64 x 1024 18 | deploySVD = GetIP1SVDProto(SVD_R) 19 | iter_num = 20000 20 | 21 | train_db = CAFFE_HOME + "examples/cifar10/cifar10_train_lmdb" 22 | test_db = CAFFE_HOME + "examples/cifar10/cifar10_test_lmdb" 23 | mean_proto = CAFFE_HOME + "examples/cifar10/mean.binaryproto" 24 | mean_npy = "./mean.npy" 25 | mean_pic = np.load(mean_npy) 26 | 27 | def read_db(db_name): 28 | lmdb_env = lmdb.open(db_name) 29 | lmdb_txn = lmdb_env.begin() 30 | lmdb_cursor = lmdb_txn.cursor() 31 | datum = caffe.proto.caffe_pb2.Datum() 32 | 33 | X = [] 34 | y = [] 35 | cnts = {} 36 | for key, value in lmdb_cursor: 37 | datum.ParseFromString(value) 38 | label = datum.label 39 | data = caffe.io.datum_to_array(datum) 40 | #data = data.swapaxes(0, 2).swapaxes(0, 1) 41 | X.append(data) 42 | y.append(label) 43 | if label not in cnts: 44 | cnts[label] = 0 45 | cnts[label] += 1 46 | #plt.imshow(data) 47 | #plt.show() 48 | return X, np.array(y), cnts 49 | 50 | testX, testy, cnts = read_db(test_db) 51 | #testX, testy, cnts = read_db(train_db) 52 | print ("#train set: ", len(testX)) 53 | print ("the size of sample:", testX[0].shape) 54 | print ("kinds: ", cnts) 55 | 56 | if not os.path.exists("label.npy"): 57 | np.save("label.npy", testy) 58 | 59 | # 生成配置文件 60 | # CAFFE_HOME 61 | example_dir = CAFFE_HOME + "examples/cifar10/" 62 | build_dir = "./build_ip1/" 63 | 64 | # 加载新的模型 65 | if len(sys.argv) == 1: 66 | print ("NO MODEL :-(") 67 | sys.exit(1) 68 | else: 69 | new_model = sys.argv[1] 70 | 71 | #new_model = "./new/ip1_SVD%d/ip1_SVD%d_iter_%d.caffemodel.h5" % (SVD_R,SVD_R, iter_num) 72 | nn = caffe.Net(deploySVD, new_model, caffe.TEST) 73 | 74 | n = len(testX) 75 | pre = np.zeros(testy.shape) 76 | print ("N = %d" % n) 77 | for i in range(n): 78 | nn.blobs["data"].data[...] = testX[i] - mean_pic 79 | nn.forward() 80 | prob = nn.blobs["prob"].data 81 | pre[i] = prob.argmax() 82 | print ("%d / %d" % (i + 1, n)) 83 | right = np.sum(pre == testy) 84 | print ("Accuracy: %f" % (right * 1.0 / n)) 85 | 86 | 87 | model_name = new_model.split("/")[-1] 88 | np.save(RESULT_DIR + "%s.npy" % (model_name), pre) 89 | -------------------------------------------------------------------------------- /eval_model_ip2.py: -------------------------------------------------------------------------------- 1 | #coding=utf-8 2 | # test decompress ip2 layer 3 | import caffe 4 | from caffe import layers as L, params as P, to_proto 5 | from caffe.proto import caffe_pb2 6 | import lmdb 7 | import numpy as np 8 | import os 9 | import sys 10 | from numpy import linalg as la 11 | import matplotlib.pyplot as plt 12 | from base import * 13 | 14 | CAFFE_HOME = "/opt/caffe/" 15 | RESULT_DIR = "./result/" 16 | 17 | SVD_R = 4 18 | deploySVD = GetSVDProto(SVD_R) 19 | iter_num = 20000 20 | 21 | train_db = CAFFE_HOME + "examples/cifar10/cifar10_train_lmdb" 22 | test_db = CAFFE_HOME + "examples/cifar10/cifar10_test_lmdb" 23 | mean_proto = CAFFE_HOME + "examples/cifar10/mean.binaryproto" 24 | mean_npy = "./mean.npy" 25 | mean_pic = np.load(mean_npy) 26 | 27 | def read_db(db_name): 28 | lmdb_env = lmdb.open(db_name) 29 | lmdb_txn = lmdb_env.begin() 30 | lmdb_cursor = lmdb_txn.cursor() 31 | datum = caffe.proto.caffe_pb2.Datum() 32 | 33 | X = [] 34 | y = [] 35 | cnts = {} 36 | for key, value in lmdb_cursor: 37 | datum.ParseFromString(value) 38 | label = datum.label 39 | data = caffe.io.datum_to_array(datum) 40 | #data = data.swapaxes(0, 2).swapaxes(0, 1) 41 | X.append(data) 42 | y.append(label) 43 | if label not in cnts: 44 | cnts[label] = 0 45 | cnts[label] += 1 46 | #plt.imshow(data) 47 | #plt.show() 48 | return X, np.array(y), cnts 49 | 50 | testX, testy, cnts = read_db(test_db) 51 | #testX, testy, cnts = read_db(train_db) 52 | print ("#train set: ", len(testX)) 53 | print ("the size of sample:", testX[0].shape) 54 | print ("kinds: ", cnts) 55 | 56 | if not os.path.exists("label.npy"): 57 | np.save("label.npy", testy) 58 | 59 | # 生成配置文件 60 | # CAFFE_HOME 61 | example_dir = CAFFE_HOME + "examples/cifar10/" 62 | build_dir = "./build/" 63 | 64 | # 加载新的模型 65 | if len(sys.argv) == 1: 66 | print ("NO MODEL :-(") 67 | sys.exit(1) 68 | else: 69 | new_model = sys.argv[1] 70 | 71 | nn = caffe.Net(deploySVD, new_model, caffe.TEST) 72 | 73 | n = len(testX) 74 | pre = np.zeros(testy.shape) 75 | print ("N = %d" % n) 76 | for i in range(n): 77 | nn.blobs["data"].data[...] = testX[i] - mean_pic 78 | nn.forward() 79 | prob = nn.blobs["prob"].data 80 | pre[i] = prob.argmax() 81 | print ("%d / %d" % (i + 1, n)) 82 | right = np.sum(pre == testy) 83 | print ("Accuracy: %f" % (right * 1.0 / n)) 84 | 85 | 86 | model_name = new_model.split("/")[-1] 87 | np.save(RESULT_DIR + "%s.npy" % (model_name), pre) 88 | -------------------------------------------------------------------------------- /evaluate.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | import sys 3 | from base import * 4 | 5 | if len(sys.argv) == 1: 6 | PREDICT_FILENAME = "result/net_normal.npy" 7 | else: 8 | PREDICT_FILENAME = sys.argv[1] 9 | 10 | print ("Compare with Real Label and %s" % PREDICT_FILENAME) 11 | 12 | num_kinds = 10 13 | pre = np.load(PREDICT_FILENAME) 14 | label = np.load("label.npy") 15 | n = len(label) 16 | 17 | eval_result(label, pre, num_kinds) 18 | -------------------------------------------------------------------------------- /improve_ip1_new.py: -------------------------------------------------------------------------------- 1 | #coding=utf-8 2 | # decompress ip1 layer 3 | import caffe 4 | from caffe import layers as L, params as P, to_proto 5 | from caffe.proto import caffe_pb2 6 | import lmdb 7 | import numpy as np 8 | import os 9 | import sys 10 | from numpy import linalg as la 11 | import matplotlib.pyplot as plt 12 | from base import * 13 | 14 | CAFFE_HOME = "/opt/caffe/" 15 | RESULT_DIR = "./result/" 16 | 17 | SVD_R = 6 # 64 x 1024 18 | deploySVD = GetIP1SVDProto(SVD_R) 19 | USE_WEIGHT = True 20 | 21 | deploy = "./proto/cifar10_quick.prototxt" 22 | caffe_model = CAFFE_HOME + "/examples/cifar10/cifar10_quick_iter_5000.caffemodel.h5" 23 | train_db = CAFFE_HOME + "examples/cifar10/cifar10_train_lmdb" 24 | test_db = CAFFE_HOME + "examples/cifar10/cifar10_test_lmdb" 25 | mean_proto = CAFFE_HOME + "examples/cifar10/mean.binaryproto" 26 | mean_npy = "./mean.npy" 27 | mean_pic = np.load(mean_npy) 28 | 29 | def read_db(db_name): 30 | lmdb_env = lmdb.open(db_name) 31 | lmdb_txn = lmdb_env.begin() 32 | lmdb_cursor = lmdb_txn.cursor() 33 | datum = caffe.proto.caffe_pb2.Datum() 34 | 35 | X = [] 36 | y = [] 37 | cnts = {} 38 | for key, value in lmdb_cursor: 39 | datum.ParseFromString(value) 40 | label = datum.label 41 | data = caffe.io.datum_to_array(datum) 42 | #data = data.swapaxes(0, 2).swapaxes(0, 1) 43 | X.append(data) 44 | y.append(label) 45 | if label not in cnts: 46 | cnts[label] = 0 47 | cnts[label] += 1 48 | #plt.imshow(data) 49 | #plt.show() 50 | return X, np.array(y), cnts 51 | 52 | testX, testy, cnts = read_db(test_db) 53 | #testX, testy, cnts = read_db(train_db) 54 | print ("#train set: ", len(testX)) 55 | print ("the size of sample:", testX[0].shape) 56 | print ("kinds: ", cnts) 57 | 58 | if not os.path.exists("label.npy"): 59 | np.save("label.npy", testy) 60 | 61 | # Load model and network 62 | net = caffe.Net(deploy, caffe_model, caffe.TEST) 63 | for layer_name, param in net.params.items(): 64 | # 0 is weight, 1 is biases 65 | print (layer_name, param[0].data.shape,net.blobs[layer_name].data.shape) 66 | 67 | if SVD_R > 0: 68 | netSVD = caffe.Net(deploySVD, caffe_model, caffe.TEST) 69 | print ("SVD NET:") 70 | for layer_name, param in netSVD.params.items(): 71 | # 0 is weight, 1 is biases 72 | print (layer_name, param[0].data.shape) 73 | 74 | print (type(net.params)) 75 | print (net.params.keys()) 76 | print ("layer ip1:") 77 | print ("WEIGHT:") 78 | print (net.params["ip1"][0].data.shape) 79 | print ("BIASES:") 80 | print (net.params["ip1"][1].data.shape) 81 | 82 | 83 | data, label = L.Data(source = test_db, backend = P.Data.LMDB, batch_size = 100, ntop = 2, mean_file = mean_proto) 84 | 85 | model_file = "model/net_SVD%d.caffemodel" % SVD_R 86 | 87 | if SVD_R > 0: 88 | # SVD 89 | print ("SVD %d" % SVD_R) 90 | u, sigma, vt = la.svd(net.params["ip1"][0].data) 91 | print ("Sigma: ", sigma) 92 | if SVD_R > len(sigma): 93 | print ("SVD_R is too large :-(") 94 | sys.exit() 95 | U = np.matrix(u[:, :SVD_R]) 96 | S = np.matrix(np.diag(sigma[:SVD_R])) 97 | VT = np.matrix(vt[:SVD_R, :]) 98 | print ("IP1", net.params["ip1"][0].data.shape) # 10, 64 99 | print ("U", U.shape) 100 | print ("S", S.shape) 101 | print ("VT", VT.shape) 102 | 103 | # y = Wx + b 104 | # y = U * S * VT * x + b 105 | # y = U * ((S * VT) * x) + b 106 | # y = U * (Z * x) + b 107 | Z = S * VT 108 | 109 | np.copyto(netSVD.params["ipZ"][0].data, Z) 110 | np.copyto(netSVD.params["ipU"][0].data, U) 111 | np.copyto(netSVD.params["ipU"][1].data, net.params["ip1"][1].data) 112 | 113 | nn = netSVD 114 | nn.save(model_file) 115 | 116 | else: 117 | print ("NORMAL") 118 | nn = net 119 | 120 | # 生成配置文件 121 | # CAFFE_HOME 122 | print ("CONFIG") 123 | example_dir = CAFFE_HOME + "examples/cifar10/" 124 | build_dir = "./build_ip1/" 125 | cwd = os.getcwd() 126 | 127 | proto = build_dir + "train_test_SVD_ip1.prototext" 128 | BuildFile([("$", "%d" % SVD_R)], proto, "./proto/train_test_SVD_ip1.template") 129 | 130 | ps = [("$prototxt", proto), ("$snap_pre", "ip1_SVD%d" % (SVD_R))] 131 | BuildFile(ps, build_dir + "solver.prototxt","./proto/solver.template") 132 | model_file = build_dir + "ip1_SVD%d.caffemodel" % (SVD_R) 133 | rp = [("$proto", build_dir + "solver.prototxt"), ("$model", model_file)] 134 | BuildFile(rp, build_dir + "train_imp.sh", "./proto/train_imp.sh") 135 | nn.save(model_file) 136 | 137 | print ("OK") 138 | sys.exit(0) 139 | print ("train..") 140 | os.system("cd %s && sudo optirun sh ./examples/cifar10/train_imp.sh" % CAFFE_HOME) 141 | raw_input("aha") 142 | 143 | # 加载新的模型 144 | new_model = CAFFE_HOME + "examples/cifar10/net_improve_ip2_SVD%d_iter_500.caffemodel.h5" % SVD_R 145 | nn = caffe.Net(deploySVD, new_model, caffe.TEST) 146 | 147 | n = len(testX) 148 | pre = np.zeros(testy.shape) 149 | print ("N = %d" % n) 150 | for i in range(n): 151 | nn.blobs["data"].data[...] = testX[i] - mean_pic 152 | nn.forward() 153 | prob = nn.blobs["prob"].data 154 | pre[i] = prob.argmax() 155 | print ("%d / %d" % (i + 1, n)) 156 | right = np.sum(pre == testy) 157 | print ("Accuracy: %f" % (right * 1.0 / n)) 158 | 159 | 160 | if SVD_R > 0: 161 | np.save(RESULT_DIR + "net_imp_SVD%d.npy" % SVD_R, pre) 162 | else: 163 | np.save(RESULT_DIR + "net_imp_normal.npy", pre) 164 | -------------------------------------------------------------------------------- /improve_ip2.py: -------------------------------------------------------------------------------- 1 | #coding=utf-8 2 | # decompress ip2 layer 3 | import caffe 4 | from caffe import layers as L, params as P, to_proto 5 | from caffe.proto import caffe_pb2 6 | import lmdb 7 | import numpy as np 8 | import os 9 | import sys 10 | from numpy import linalg as la 11 | import matplotlib.pyplot as plt 12 | from base import * 13 | 14 | CAFFE_HOME = "/opt/caffe/" 15 | RESULT_DIR = "./result/" 16 | 17 | SVD_R = 4 18 | deploySVD = GetSVDProto(SVD_R) 19 | USE_WEIGHT = True 20 | 21 | deploy = "./proto/cifar10_quick.prototxt" 22 | caffe_model = CAFFE_HOME + "/examples/cifar10/cifar10_quick_iter_5000.caffemodel.h5" 23 | train_db = CAFFE_HOME + "examples/cifar10/cifar10_train_lmdb" 24 | test_db = CAFFE_HOME + "examples/cifar10/cifar10_test_lmdb" 25 | mean_proto = CAFFE_HOME + "examples/cifar10/mean.binaryproto" 26 | mean_npy = "./mean.npy" 27 | mean_pic = np.load(mean_npy) 28 | 29 | def read_db(db_name): 30 | lmdb_env = lmdb.open(db_name) 31 | lmdb_txn = lmdb_env.begin() 32 | lmdb_cursor = lmdb_txn.cursor() 33 | datum = caffe.proto.caffe_pb2.Datum() 34 | 35 | X = [] 36 | y = [] 37 | cnts = {} 38 | for key, value in lmdb_cursor: 39 | datum.ParseFromString(value) 40 | label = datum.label 41 | data = caffe.io.datum_to_array(datum) 42 | #data = data.swapaxes(0, 2).swapaxes(0, 1) 43 | X.append(data) 44 | y.append(label) 45 | if label not in cnts: 46 | cnts[label] = 0 47 | cnts[label] += 1 48 | #plt.imshow(data) 49 | #plt.show() 50 | return X, np.array(y), cnts 51 | 52 | testX, testy, cnts = read_db(test_db) 53 | #testX, testy, cnts = read_db(train_db) 54 | print ("#train set: ", len(testX)) 55 | print ("the size of sample:", testX[0].shape) 56 | print ("kinds: ", cnts) 57 | 58 | if not os.path.exists("label.npy"): 59 | np.save("label.npy", testy) 60 | 61 | # Load model and network 62 | net = caffe.Net(deploy, caffe_model, caffe.TEST) 63 | for layer_name, param in net.params.items(): 64 | # 0 is weight, 1 is biases 65 | print (layer_name, param[0].data.shape,net.blobs[layer_name].data.shape) 66 | 67 | if SVD_R > 0: 68 | netSVD = caffe.Net(deploySVD, caffe_model, caffe.TEST) 69 | print ("SVD NET:") 70 | for layer_name, param in netSVD.params.items(): 71 | # 0 is weight, 1 is biases 72 | print (layer_name, param[0].data.shape) 73 | 74 | print (type(net.params)) 75 | print (net.params.keys()) 76 | print ("layer ip2:") 77 | print ("WEIGHT:") 78 | print (net.params["ip2"][0].data.shape) 79 | print ("BIASES:") 80 | print (net.params["ip2"][1].data.shape) 81 | 82 | 83 | data, label = L.Data(source = test_db, backend = P.Data.LMDB, batch_size = 100, ntop = 2, mean_file = mean_proto) 84 | 85 | model_file = "model/net_SVD%d.caffemodel" % SVD_R 86 | 87 | if SVD_R > 0: 88 | # SVD 89 | print ("SVD %d" % SVD_R) 90 | u, sigma, vt = la.svd(net.params["ip2"][0].data) 91 | print ("Sigma: ", sigma) 92 | if SVD_R > len(sigma): 93 | print ("SVD_R is too large :-(") 94 | sys.exit() 95 | U = np.matrix(u[:, :SVD_R]) 96 | S = np.matrix(np.diag(sigma[:SVD_R])) 97 | VT = np.matrix(vt[:SVD_R, :]) 98 | print ("IP2", net.params["ip2"][0].data.shape) # 10, 64 99 | print ("U", U.shape) 100 | print ("S", S.shape) 101 | print ("VT", VT.shape) 102 | 103 | # y = Wx + b 104 | # y = U * S * VT * x + b 105 | # y = U * ((S * VT) * x) + b 106 | # y = U * (Z * x) + b 107 | Z = S * VT 108 | 109 | np.copyto(netSVD.params["ipZ"][0].data, Z) 110 | np.copyto(netSVD.params["ipU"][0].data, U) 111 | np.copyto(netSVD.params["ipU"][1].data, net.params["ip2"][1].data) 112 | 113 | nn = netSVD 114 | nn.save(model_file) 115 | 116 | else: 117 | print ("NORMAL") 118 | nn = net 119 | 120 | # 生成配置文件 121 | # CAFFE_HOME 122 | print ("CONFIG") 123 | example_dir = CAFFE_HOME + "examples/cifar10/" 124 | cwd = os.getcwd() 125 | 126 | proto = os.getcwd() + "/tmp/cifar10_quick_train_test.prototext" 127 | BuildFile([("$", "%d" % SVD_R)], proto, "./proto/cifar10_quick_train_test_SVD.template") 128 | 129 | ps = [("$prototxt", proto), ("$snap_pre", "examples/cifar10/net_improve_ip2_SVD%d" % (SVD_R))] 130 | BuildFile(ps, example_dir + "cifar10_quick_solver_imp.prototxt","./proto/cifar10_quick_solver_imp.template") 131 | rp = [("$proto", example_dir + "cifar10_quick_solver_imp.prototxt"), ("$model", "%s/%s" % (cwd, model_file))] 132 | BuildFile(rp, example_dir + "train_imp.sh", "./proto/train_quick_imp.sh") 133 | 134 | print ("train..") 135 | os.system("cd %s && sudo optirun sh ./examples/cifar10/train_imp.sh" % CAFFE_HOME) 136 | raw_input("aha") 137 | 138 | # 加载新的模型 139 | new_model = CAFFE_HOME + "examples/cifar10/net_improve_ip2_SVD%d_iter_500.caffemodel.h5" % SVD_R 140 | nn = caffe.Net(deploySVD, new_model, caffe.TEST) 141 | 142 | n = len(testX) 143 | pre = np.zeros(testy.shape) 144 | print ("N = %d" % n) 145 | for i in range(n): 146 | nn.blobs["data"].data[...] = testX[i] - mean_pic 147 | nn.forward() 148 | prob = nn.blobs["prob"].data 149 | pre[i] = prob.argmax() 150 | print ("%d / %d" % (i + 1, n)) 151 | right = np.sum(pre == testy) 152 | print ("Accuracy: %f" % (right * 1.0 / n)) 153 | 154 | 155 | if SVD_R > 0: 156 | np.save(RESULT_DIR + "net_imp_SVD%d.npy" % SVD_R, pre) 157 | else: 158 | np.save(RESULT_DIR + "net_imp_normal.npy", pre) 159 | -------------------------------------------------------------------------------- /improve_ip2_new.py: -------------------------------------------------------------------------------- 1 | #coding=utf-8 2 | # decompress ip2 layer 3 | import caffe 4 | from caffe import layers as L, params as P, to_proto 5 | from caffe.proto import caffe_pb2 6 | import lmdb 7 | import numpy as np 8 | import os 9 | import sys 10 | from numpy import linalg as la 11 | import matplotlib.pyplot as plt 12 | from base import * 13 | 14 | CAFFE_HOME = "/opt/caffe/" 15 | RESULT_DIR = "./result/" 16 | 17 | SVD_R = 4 18 | deploySVD = GetSVDProto(SVD_R) 19 | USE_WEIGHT = True 20 | 21 | deploy = "./proto/cifar10_quick.prototxt" 22 | caffe_model = CAFFE_HOME + "/examples/cifar10/cifar10_quick_iter_5000.caffemodel.h5" 23 | train_db = CAFFE_HOME + "examples/cifar10/cifar10_train_lmdb" 24 | test_db = CAFFE_HOME + "examples/cifar10/cifar10_test_lmdb" 25 | mean_proto = CAFFE_HOME + "examples/cifar10/mean.binaryproto" 26 | mean_npy = "./mean.npy" 27 | mean_pic = np.load(mean_npy) 28 | 29 | def read_db(db_name): 30 | lmdb_env = lmdb.open(db_name) 31 | lmdb_txn = lmdb_env.begin() 32 | lmdb_cursor = lmdb_txn.cursor() 33 | datum = caffe.proto.caffe_pb2.Datum() 34 | 35 | X = [] 36 | y = [] 37 | cnts = {} 38 | for key, value in lmdb_cursor: 39 | datum.ParseFromString(value) 40 | label = datum.label 41 | data = caffe.io.datum_to_array(datum) 42 | #data = data.swapaxes(0, 2).swapaxes(0, 1) 43 | X.append(data) 44 | y.append(label) 45 | if label not in cnts: 46 | cnts[label] = 0 47 | cnts[label] += 1 48 | #plt.imshow(data) 49 | #plt.show() 50 | return X, np.array(y), cnts 51 | 52 | testX, testy, cnts = read_db(test_db) 53 | #testX, testy, cnts = read_db(train_db) 54 | print ("#train set: ", len(testX)) 55 | print ("the size of sample:", testX[0].shape) 56 | print ("kinds: ", cnts) 57 | 58 | if not os.path.exists("label.npy"): 59 | np.save("label.npy", testy) 60 | 61 | # Load model and network 62 | net = caffe.Net(deploy, caffe_model, caffe.TEST) 63 | print ("=====") 64 | print (net.blobs["data"].data.shape) 65 | print (net.blobs["pool1"].data.shape) 66 | print (net.blobs["pool2"].data.shape) 67 | print (net.blobs["pool3"].data.shape) 68 | for layer_name, param in net.params.items(): 69 | # 0 is weight, 1 is biases 70 | print (layer_name, param[0].data.shape,net.blobs[layer_name].data.shape) 71 | 72 | if SVD_R > 0: 73 | netSVD = caffe.Net(deploySVD, caffe_model, caffe.TEST) 74 | print ("SVD NET:") 75 | for layer_name, param in netSVD.params.items(): 76 | # 0 is weight, 1 is biases 77 | print (layer_name, param[0].data.shape) 78 | 79 | print (type(net.params)) 80 | print (net.params.keys()) 81 | print ("layer ip2:") 82 | print ("WEIGHT:") 83 | print (net.params["ip2"][0].data.shape) 84 | print ("BIASES:") 85 | print (net.params["ip2"][1].data.shape) 86 | 87 | 88 | data, label = L.Data(source = test_db, backend = P.Data.LMDB, batch_size = 100, ntop = 2, mean_file = mean_proto) 89 | 90 | model_file = "model/net_SVD%d.caffemodel" % SVD_R 91 | 92 | if SVD_R > 0: 93 | # SVD 94 | print ("SVD %d" % SVD_R) 95 | u, sigma, vt = la.svd(net.params["ip2"][0].data) 96 | print ("Sigma: ", sigma) 97 | if SVD_R > len(sigma): 98 | print ("SVD_R is too large :-(") 99 | sys.exit() 100 | U = np.matrix(u[:, :SVD_R]) 101 | S = np.matrix(np.diag(sigma[:SVD_R])) 102 | VT = np.matrix(vt[:SVD_R, :]) 103 | print ("IP2", net.params["ip2"][0].data.shape) # 10, 64 104 | print ("U", U.shape) 105 | print ("S", S.shape) 106 | print ("VT", VT.shape) 107 | 108 | # y = Wx + b 109 | # y = U * S * VT * x + b 110 | # y = U * ((S * VT) * x) + b 111 | # y = U * (Z * x) + b 112 | Z = S * VT 113 | 114 | np.copyto(netSVD.params["ipZ"][0].data, Z) 115 | np.copyto(netSVD.params["ipU"][0].data, U) 116 | np.copyto(netSVD.params["ipU"][1].data, net.params["ip2"][1].data) 117 | 118 | nn = netSVD 119 | nn.save(model_file) 120 | 121 | else: 122 | print ("NORMAL") 123 | nn = net 124 | 125 | # 生成配置文件 126 | # CAFFE_HOME 127 | print ("CONFIG") 128 | example_dir = CAFFE_HOME + "examples/cifar10/" 129 | build_dir = "./build/" 130 | cwd = os.getcwd() 131 | 132 | proto = build_dir + "train_test_SVD.prototext" 133 | BuildFile([("$", "%d" % SVD_R)], proto, "./proto/train_test_SVD.template") 134 | 135 | ps = [("$prototxt", proto), ("$snap_pre", "ip2_SVD%d" % (SVD_R))] 136 | BuildFile(ps, build_dir + "solver.prototxt","./proto/solver.template") 137 | model_file = build_dir + "ip2_SVD%d.caffemodel" % (SVD_R) 138 | rp = [("$proto", build_dir + "solver.prototxt"), ("$model", model_file)] 139 | BuildFile(rp, build_dir + "train_imp.sh", "./proto/train_imp.sh") 140 | nn.save(model_file) 141 | 142 | print ("OK") 143 | sys.exit(0) 144 | print ("train..") 145 | os.system("cd %s && sudo optirun sh ./examples/cifar10/train_imp.sh" % CAFFE_HOME) 146 | raw_input("aha") 147 | 148 | # 加载新的模型 149 | new_model = CAFFE_HOME + "examples/cifar10/net_improve_ip2_SVD%d_iter_500.caffemodel.h5" % SVD_R 150 | nn = caffe.Net(deploySVD, new_model, caffe.TEST) 151 | 152 | n = len(testX) 153 | pre = np.zeros(testy.shape) 154 | print ("N = %d" % n) 155 | for i in range(n): 156 | nn.blobs["data"].data[...] = testX[i] - mean_pic 157 | nn.forward() 158 | prob = nn.blobs["prob"].data 159 | pre[i] = prob.argmax() 160 | print ("%d / %d" % (i + 1, n)) 161 | right = np.sum(pre == testy) 162 | print ("Accuracy: %f" % (right * 1.0 / n)) 163 | 164 | 165 | if SVD_R > 0: 166 | np.save(RESULT_DIR + "net_imp_SVD%d.npy" % SVD_R, pre) 167 | else: 168 | np.save(RESULT_DIR + "net_imp_normal.npy", pre) 169 | -------------------------------------------------------------------------------- /improve_model_ip1.py: -------------------------------------------------------------------------------- 1 | #coding=utf-8 2 | # test decompress ip1 layer 3 | import caffe 4 | from caffe import layers as L, params as P, to_proto 5 | from caffe.proto import caffe_pb2 6 | import lmdb 7 | import numpy as np 8 | import os 9 | import sys 10 | from numpy import linalg as la 11 | import matplotlib.pyplot as plt 12 | from base import * 13 | from mcluster import * 14 | 15 | CAFFE_HOME = "/opt/caffe/" 16 | RESULT_DIR = "./result/" 17 | 18 | SVD_R = 6 # 64 x 1024 19 | deploySVD = GetIP1SVDProto(SVD_R) 20 | iter_num = 20000 21 | 22 | train_db = CAFFE_HOME + "examples/cifar10/cifar10_train_lmdb" 23 | test_db = CAFFE_HOME + "examples/cifar10/cifar10_test_lmdb" 24 | mean_proto = CAFFE_HOME + "examples/cifar10/mean.binaryproto" 25 | mean_npy = "./mean.npy" 26 | mean_pic = np.load(mean_npy) 27 | 28 | def read_db(db_name): 29 | lmdb_env = lmdb.open(db_name) 30 | lmdb_txn = lmdb_env.begin() 31 | lmdb_cursor = lmdb_txn.cursor() 32 | datum = caffe.proto.caffe_pb2.Datum() 33 | 34 | X = [] 35 | y = [] 36 | cnts = {} 37 | for key, value in lmdb_cursor: 38 | datum.ParseFromString(value) 39 | label = datum.label 40 | data = caffe.io.datum_to_array(datum) 41 | #data = data.swapaxes(0, 2).swapaxes(0, 1) 42 | X.append(data) 43 | y.append(label) 44 | if label not in cnts: 45 | cnts[label] = 0 46 | cnts[label] += 1 47 | #plt.imshow(data) 48 | #plt.show() 49 | return X, np.array(y), cnts 50 | 51 | testX, testy, cnts = read_db(test_db) 52 | #testX, testy, cnts = read_db(train_db) 53 | print ("#train set: ", len(testX)) 54 | print ("the size of sample:", testX[0].shape) 55 | print ("kinds: ", cnts) 56 | 57 | if not os.path.exists("label.npy"): 58 | np.save("label.npy", testy) 59 | 60 | # 生成配置文件 61 | # CAFFE_HOME 62 | example_dir = CAFFE_HOME + "examples/cifar10/" 63 | build_dir = "./build_ip1/" 64 | 65 | # 加载新的模型 66 | if len(sys.argv) == 1: 67 | print ("NO MODEL :-(") 68 | sys.exit(1) 69 | else: 70 | new_model = sys.argv[1] 71 | 72 | #new_model = "./new/ip1_SVD%d/ip1_SVD%d_iter_%d.caffemodel.h5" % (SVD_R,SVD_R, iter_num) 73 | nn = caffe.Net(deploySVD, new_model, caffe.TEST) 74 | 75 | #np.save("ip1_SVD%d_ipZ.npy" % SVD_R, data) 76 | 77 | Z = nn.params["ipZ"][0].data 78 | Z = get_cluster_mat(Z, 16) 79 | Z = get_round_mat(Z, 4) 80 | nn.params["ipZ"][0].data[...] = Z 81 | 82 | U = nn.params["ipU"][0].data 83 | U = get_cluster_mat(U, 16) 84 | U = get_round_mat(U, 4) 85 | nn.params["ipU"][0].data[...] = U 86 | 87 | ip2 = nn.params["ip2"][0].data 88 | ip2 = get_cluster_mat(ip2, 16) 89 | ip2 = get_round_mat(ip2, 4) 90 | nn.params["ip2"][0].data[...] = ip2 91 | 92 | n = len(testX) 93 | pre = np.zeros(testy.shape) 94 | print ("N = %d" % n) 95 | for i in range(n): 96 | nn.blobs["data"].data[...] = testX[i] - mean_pic 97 | nn.forward() 98 | prob = nn.blobs["prob"].data 99 | pre[i] = prob.argmax() 100 | print ("%d / %d" % (i + 1, n)) 101 | right = np.sum(pre == testy) 102 | print ("Accuracy: %f" % (right * 1.0 / n)) 103 | 104 | 105 | model_name = new_model.split("/")[-1] 106 | np.save(RESULT_DIR + "%s_cluster16_f4_all.npy" % (model_name), pre) 107 | -------------------------------------------------------------------------------- /label.npy: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wkcn/CaffeSVD/ea4a61bf4a53f5c65f2f369242b1e8996e5d91bb/label.npy -------------------------------------------------------------------------------- /mcluster.py: -------------------------------------------------------------------------------- 1 | #coding=utf-8 2 | import numpy as np 3 | from sklearn.cluster import KMeans 4 | 5 | class IP: 6 | def __init__(self, W, k, t = -np.inf): 7 | rows, cols = W.shape 8 | kmeans_model = KMeans(k) 9 | W_flat = W.reshape((rows * cols, 1)) 10 | W_flat[W_flat < t] = 0.0 11 | kmeans_model.fit(W_flat) 12 | self.lb = kmeans_model.labels_ 13 | self.centroids = np.zeros(k) 14 | for i in range(k): 15 | b = (self.lb == i) 16 | mean = np.mean(W_flat[b]) 17 | self.centroids[i] = mean 18 | self.rows = rows 19 | self.cols = cols 20 | self.k = k 21 | def get_matrix(self): 22 | # 为了测试方便, 直接获取压缩后的矩阵 23 | mat = np.zeros(self.rows * self.cols) 24 | for i in range(self.k): 25 | mat[self.lb == i] = self.centroids[i] 26 | return mat.reshape(self.rows, self.cols) 27 | 28 | def get_cluster_mat(W, k, t = -np.inf): 29 | rows, cols = W.shape 30 | kmeans_model = KMeans(k) 31 | W_flat = W.reshape((rows * cols, 1)) 32 | W_flat[W_flat < t] = 0.0 33 | kmeans_model.fit(W_flat) 34 | lb = kmeans_model.labels_ 35 | centroids = np.zeros(k) 36 | for i in range(k): 37 | b = (lb == i) 38 | mean = np.mean(W_flat[b]) 39 | centroids[i] = mean 40 | 41 | mat = np.zeros(rows * cols) 42 | for i in range(k): 43 | mat[lb == i] = centroids[i] 44 | return mat.reshape(rows, cols) 45 | 46 | def get_round_mat(W, k): 47 | return np.round(W, k) 48 | 49 | if __name__ == "__main__": 50 | a = np.matrix("2.09,-0.98,1.48,0.09;0.05,-0.14,-1.08,2.12;-0.91,1.92,0,-1.03;1.87,0,1.53,1.49") 51 | ''' 52 | ip = IP(a, 4) 53 | print (ip.get_matrix()) 54 | ''' 55 | print get_cluster_mat(a, 4) 56 | print get_round_mat(a, 1) 57 | -------------------------------------------------------------------------------- /mean.npy: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wkcn/CaffeSVD/ea4a61bf4a53f5c65f2f369242b1e8996e5d91bb/mean.npy -------------------------------------------------------------------------------- /net.txt: -------------------------------------------------------------------------------- 1 | name, param.shape, out_blobs.shape 2 | ('conv1', (32, 3, 5, 5), (1, 32, 32, 32)) 3 | ('conv2', (32, 32, 5, 5), (1, 32, 16, 16)) 4 | ('conv3', (64, 32, 5, 5), (1, 64, 8, 8)) 5 | ('ip1', (64, 1024), (1, 64)) 6 | ('ip2', (10, 64), (1, 10)) 7 | -------------------------------------------------------------------------------- /noimprove_ip2.py: -------------------------------------------------------------------------------- 1 | #coding=utf-8 2 | # decompress ip2 layer 3 | import caffe 4 | from caffe import layers as L, params as P, to_proto 5 | from caffe.proto import caffe_pb2 6 | import lmdb 7 | import numpy as np 8 | import os 9 | import sys 10 | from numpy import linalg as la 11 | import matplotlib.pyplot as plt 12 | from base import * 13 | 14 | CAFFE_HOME = "/opt/caffe/" 15 | RESULT_DIR = "./result/" 16 | 17 | SVD_R = 8 18 | deploySVD = GetSVDProto(SVD_R) 19 | USE_WEIGHT = True 20 | 21 | deploy = "./proto/cifar10_quick.prototxt" 22 | caffe_model = CAFFE_HOME + "/examples/cifar10/cifar10_quick_iter_5000.caffemodel.h5" 23 | train_db = CAFFE_HOME + "examples/cifar10/cifar10_train_lmdb" 24 | test_db = CAFFE_HOME + "examples/cifar10/cifar10_test_lmdb" 25 | mean_proto = CAFFE_HOME + "examples/cifar10/mean.binaryproto" 26 | mean_npy = "./mean.npy" 27 | mean_pic = np.load(mean_npy) 28 | 29 | def read_db(db_name): 30 | lmdb_env = lmdb.open(db_name) 31 | lmdb_txn = lmdb_env.begin() 32 | lmdb_cursor = lmdb_txn.cursor() 33 | datum = caffe.proto.caffe_pb2.Datum() 34 | 35 | X = [] 36 | y = [] 37 | cnts = {} 38 | for key, value in lmdb_cursor: 39 | datum.ParseFromString(value) 40 | label = datum.label 41 | data = caffe.io.datum_to_array(datum) 42 | #data = data.swapaxes(0, 2).swapaxes(0, 1) 43 | X.append(data) 44 | y.append(label) 45 | if label not in cnts: 46 | cnts[label] = 0 47 | cnts[label] += 1 48 | #plt.imshow(data) 49 | #plt.show() 50 | return X, np.array(y), cnts 51 | 52 | testX, testy, cnts = read_db(test_db) 53 | #testX, testy, cnts = read_db(train_db) 54 | print ("#train set: ", len(testX)) 55 | print ("the size of sample:", testX[0].shape) 56 | print ("kinds: ", cnts) 57 | 58 | if not os.path.exists("label.npy"): 59 | np.save("label.npy", testy) 60 | 61 | # Load model and network 62 | net = caffe.Net(deploy, caffe_model, caffe.TEST) 63 | for layer_name, param in net.params.items(): 64 | # 0 is weight, 1 is biase 65 | print (layer_name, param[0].data.shape,net.blobs[layer_name].data.shape) 66 | 67 | if SVD_R > 0: 68 | netSVD = caffe.Net(deploySVD, caffe_model, caffe.TEST) 69 | print ("SVD NET:") 70 | for layer_name, param in netSVD.params.items(): 71 | # 0 is weight, 1 is biase 72 | print (layer_name, param[0].data.shape) 73 | 74 | print (type(net.params)) 75 | print (net.params.keys()) 76 | print ("layer ip2:") 77 | print ("WEIGHT:") 78 | print (net.params["ip2"][0].data.shape) 79 | print ("BIASES:") 80 | print (net.params["ip2"][1].data.shape) 81 | 82 | 83 | data, label = L.Data(source = test_db, backend = P.Data.LMDB, batch_size = 100, ntop = 2, mean_file = mean_proto) 84 | 85 | model_file = "model/net_SVD%d.caffemodel" % SVD_R 86 | 87 | if SVD_R > 0: 88 | # SVD 89 | print ("SVD %d" % SVD_R) 90 | u, sigma, vt = la.svd(net.params["ip2"][0].data) 91 | print ("Sigma: ", sigma) 92 | if SVD_R > len(sigma): 93 | print ("SVD_R is too large :-(") 94 | sys.exit() 95 | U = np.matrix(u[:, :SVD_R]) 96 | S = np.matrix(np.diag(sigma[:SVD_R])) 97 | VT = np.matrix(vt[:SVD_R, :]) 98 | print ("IP2", net.params["ip2"][0].data.shape) # 10, 64 99 | print ("U", U.shape) 100 | print ("S", S.shape) 101 | print ("VT", VT.shape) 102 | 103 | # y = Wx + b 104 | # y = U * S * VT * x + b 105 | # y = U * ((S * VT) * x) + b 106 | # y = U * (Z * x) + b 107 | Z = S * VT 108 | 109 | np.copyto(netSVD.params["ipZ"][0].data, Z) 110 | np.copyto(netSVD.params["ipU"][0].data, U) 111 | np.copyto(netSVD.params["ipU"][1].data, net.params["ip2"][1].data) 112 | 113 | nn = netSVD 114 | nn.save(model_file) 115 | 116 | else: 117 | print ("NORMAL") 118 | nn = net 119 | 120 | # 生成配置文件 121 | # CAFFE_HOME 122 | print ("CONFIG") 123 | example_dir = CAFFE_HOME + "examples/cifar10/" 124 | cwd = os.getcwd() 125 | 126 | proto = os.getcwd() + "/tmp/cifar10_quick_train_test.prototext" 127 | BuildFile([("$", "%d" % SVD_R)], proto, "./proto/cifar10_quick_train_test_SVD.template") 128 | 129 | ps = [("$prototxt", proto), ("$snap_pre", "examples/cifar10/net_noimprove_ip2_SVD%d" % (SVD_R))] 130 | BuildFile(ps, example_dir + "cifar10_quick_solver_noimp.prototxt","./proto/cifar10_quick_solver_noimp.template") 131 | rp = [("$proto", example_dir + "cifar10_quick_solver_noimp.prototxt")] 132 | BuildFile(rp, example_dir + "train_noimp.sh", "./proto/train_quick_noimp.sh") 133 | 134 | print ("train..") 135 | os.system("cd %s && sudo optirun sh ./examples/cifar10/train_noimp.sh" % CAFFE_HOME) 136 | raw_input("aha") 137 | 138 | # 加载新的模型 139 | new_model = CAFFE_HOME + "examples/cifar10/net_noimprove_ip2_SVD%d_iter_4500.caffemodel.h5" % SVD_R 140 | nn = caffe.Net(deploySVD, new_model, caffe.TEST) 141 | 142 | n = len(testX) 143 | pre = np.zeros(testy.shape) 144 | print ("N = %d" % n) 145 | for i in range(n): 146 | nn.blobs["data"].data[...] = testX[i] - mean_pic 147 | nn.forward() 148 | prob = nn.blobs["prob"].data 149 | pre[i] = prob.argmax() 150 | print ("%d / %d" % (i + 1, n)) 151 | right = np.sum(pre == testy) 152 | print ("Accuracy: %f" % (right * 1.0 / n)) 153 | 154 | 155 | if SVD_R > 0: 156 | np.save(RESULT_DIR + "net_noimp_SVD%d.npy" % SVD_R, pre) 157 | else: 158 | np.save(RESULT_DIR + "net_noimp_normal.npy", pre) 159 | -------------------------------------------------------------------------------- /pic/acc_time_space.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wkcn/CaffeSVD/ea4a61bf4a53f5c65f2f369242b1e8996e5d91bb/pic/acc_time_space.png -------------------------------------------------------------------------------- /pic/acc_time_space_ip1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wkcn/CaffeSVD/ea4a61bf4a53f5c65f2f369242b1e8996e5d91bb/pic/acc_time_space_ip1.png -------------------------------------------------------------------------------- /pic/statistics.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wkcn/CaffeSVD/ea4a61bf4a53f5c65f2f369242b1e8996e5d91bb/pic/statistics.png -------------------------------------------------------------------------------- /pic/statistics_ip1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wkcn/CaffeSVD/ea4a61bf4a53f5c65f2f369242b1e8996e5d91bb/pic/statistics_ip1.png -------------------------------------------------------------------------------- /proto/cifar10_SVD.template: -------------------------------------------------------------------------------- 1 | name: "CIFAR10_quick_test" 2 | layer { 3 | name: "data" 4 | type: "Input" 5 | top: "data" 6 | input_param { shape: { dim: 1 dim: 3 dim: 32 dim: 32 } } 7 | } 8 | layer { 9 | name: "conv1" 10 | type: "Convolution" 11 | bottom: "data" 12 | top: "conv1" 13 | param { 14 | lr_mult: 1 15 | } 16 | param { 17 | lr_mult: 2 18 | } 19 | convolution_param { 20 | num_output: 32 21 | pad: 2 22 | kernel_size: 5 23 | stride: 1 24 | } 25 | } 26 | layer { 27 | name: "pool1" 28 | type: "Pooling" 29 | bottom: "conv1" 30 | top: "pool1" 31 | pooling_param { 32 | pool: MAX 33 | kernel_size: 3 34 | stride: 2 35 | } 36 | } 37 | layer { 38 | name: "relu1" 39 | type: "ReLU" 40 | bottom: "pool1" 41 | top: "pool1" 42 | } 43 | layer { 44 | name: "conv2" 45 | type: "Convolution" 46 | bottom: "pool1" 47 | top: "conv2" 48 | param { 49 | lr_mult: 1 50 | } 51 | param { 52 | lr_mult: 2 53 | } 54 | convolution_param { 55 | num_output: 32 56 | pad: 2 57 | kernel_size: 5 58 | stride: 1 59 | } 60 | } 61 | layer { 62 | name: "relu2" 63 | type: "ReLU" 64 | bottom: "conv2" 65 | top: "conv2" 66 | } 67 | layer { 68 | name: "pool2" 69 | type: "Pooling" 70 | bottom: "conv2" 71 | top: "pool2" 72 | pooling_param { 73 | pool: AVE 74 | kernel_size: 3 75 | stride: 2 76 | } 77 | } 78 | layer { 79 | name: "conv3" 80 | type: "Convolution" 81 | bottom: "pool2" 82 | top: "conv3" 83 | param { 84 | lr_mult: 1 85 | } 86 | param { 87 | lr_mult: 2 88 | } 89 | convolution_param { 90 | num_output: 64 91 | pad: 2 92 | kernel_size: 5 93 | stride: 1 94 | } 95 | } 96 | layer { 97 | name: "relu3" 98 | type: "ReLU" 99 | bottom: "conv3" 100 | top: "conv3" 101 | } 102 | layer { 103 | name: "pool3" 104 | type: "Pooling" 105 | bottom: "conv3" 106 | top: "pool3" 107 | pooling_param { 108 | pool: AVE 109 | kernel_size: 3 110 | stride: 2 111 | } 112 | } 113 | layer { 114 | name: "ip1" 115 | type: "InnerProduct" 116 | bottom: "pool3" 117 | top: "ip1" 118 | param { 119 | lr_mult: 1 120 | } 121 | param { 122 | lr_mult: 2 123 | } 124 | inner_product_param { 125 | num_output: 64 126 | } 127 | } 128 | 129 | layer { 130 | name: "ipZ" 131 | type: "InnerProduct" 132 | bottom: "ip1" 133 | top: "ipZ" 134 | inner_product_param{ 135 | num_output: $ 136 | } 137 | } 138 | 139 | layer { 140 | name: "ipU" 141 | type: "InnerProduct" 142 | bottom: "ipZ" 143 | top: "ipSVD" 144 | inner_product_param{ 145 | num_output: 10 146 | } 147 | } 148 | 149 | layer { 150 | name: "prob" 151 | type: "Softmax" 152 | bottom: "ipSVD" 153 | top: "prob" 154 | } 155 | -------------------------------------------------------------------------------- /proto/cifar10_ip1_SVD.template: -------------------------------------------------------------------------------- 1 | name: "CIFAR10_quick_test" 2 | layer { 3 | name: "data" 4 | type: "Input" 5 | top: "data" 6 | input_param { shape: { dim: 1 dim: 3 dim: 32 dim: 32 } } 7 | } 8 | layer { 9 | name: "conv1" 10 | type: "Convolution" 11 | bottom: "data" 12 | top: "conv1" 13 | param { 14 | lr_mult: 1 15 | } 16 | param { 17 | lr_mult: 2 18 | } 19 | convolution_param { 20 | num_output: 32 21 | pad: 2 22 | kernel_size: 5 23 | stride: 1 24 | } 25 | } 26 | layer { 27 | name: "pool1" 28 | type: "Pooling" 29 | bottom: "conv1" 30 | top: "pool1" 31 | pooling_param { 32 | pool: MAX 33 | kernel_size: 3 34 | stride: 2 35 | } 36 | } 37 | layer { 38 | name: "relu1" 39 | type: "ReLU" 40 | bottom: "pool1" 41 | top: "pool1" 42 | } 43 | layer { 44 | name: "conv2" 45 | type: "Convolution" 46 | bottom: "pool1" 47 | top: "conv2" 48 | param { 49 | lr_mult: 1 50 | } 51 | param { 52 | lr_mult: 2 53 | } 54 | convolution_param { 55 | num_output: 32 56 | pad: 2 57 | kernel_size: 5 58 | stride: 1 59 | } 60 | } 61 | layer { 62 | name: "relu2" 63 | type: "ReLU" 64 | bottom: "conv2" 65 | top: "conv2" 66 | } 67 | layer { 68 | name: "pool2" 69 | type: "Pooling" 70 | bottom: "conv2" 71 | top: "pool2" 72 | pooling_param { 73 | pool: AVE 74 | kernel_size: 3 75 | stride: 2 76 | } 77 | } 78 | layer { 79 | name: "conv3" 80 | type: "Convolution" 81 | bottom: "pool2" 82 | top: "conv3" 83 | param { 84 | lr_mult: 1 85 | } 86 | param { 87 | lr_mult: 2 88 | } 89 | convolution_param { 90 | num_output: 64 91 | pad: 2 92 | kernel_size: 5 93 | stride: 1 94 | } 95 | } 96 | layer { 97 | name: "relu3" 98 | type: "ReLU" 99 | bottom: "conv3" 100 | top: "conv3" 101 | } 102 | layer { 103 | name: "pool3" 104 | type: "Pooling" 105 | bottom: "conv3" 106 | top: "pool3" 107 | pooling_param { 108 | pool: AVE 109 | kernel_size: 3 110 | stride: 2 111 | } 112 | } 113 | 114 | layer { 115 | name: "ipZ" 116 | type: "InnerProduct" 117 | bottom: "pool3" 118 | top: "ipZ" 119 | inner_product_param{ 120 | num_output: $ 121 | } 122 | } 123 | 124 | layer { 125 | name: "ipU" 126 | type: "InnerProduct" 127 | bottom: "ipZ" 128 | top: "ipSVD" 129 | inner_product_param{ 130 | num_output: 64 131 | } 132 | } 133 | 134 | layer { 135 | name: "ip2" 136 | type: "InnerProduct" 137 | bottom: "ipSVD" 138 | top: "ip2" 139 | param { 140 | lr_mult: 1 141 | } 142 | param { 143 | lr_mult: 2 144 | } 145 | inner_product_param { 146 | num_output: 10 147 | } 148 | } 149 | 150 | layer { 151 | name: "prob" 152 | type: "Softmax" 153 | bottom: "ip2" 154 | top: "prob" 155 | } 156 | -------------------------------------------------------------------------------- /proto/cifar10_quick.prototxt: -------------------------------------------------------------------------------- 1 | name: "CIFAR10_quick_test" 2 | layer { 3 | name: "data" 4 | type: "Input" 5 | top: "data" 6 | input_param { shape: { dim: 1 dim: 3 dim: 32 dim: 32 } } 7 | } 8 | layer { 9 | name: "conv1" 10 | type: "Convolution" 11 | bottom: "data" 12 | top: "conv1" 13 | param { 14 | lr_mult: 1 15 | } 16 | param { 17 | lr_mult: 2 18 | } 19 | convolution_param { 20 | num_output: 32 21 | pad: 2 22 | kernel_size: 5 23 | stride: 1 24 | } 25 | } 26 | layer { 27 | name: "pool1" 28 | type: "Pooling" 29 | bottom: "conv1" 30 | top: "pool1" 31 | pooling_param { 32 | pool: MAX 33 | kernel_size: 3 34 | stride: 2 35 | } 36 | } 37 | layer { 38 | name: "relu1" 39 | type: "ReLU" 40 | bottom: "pool1" 41 | top: "pool1" 42 | } 43 | layer { 44 | name: "conv2" 45 | type: "Convolution" 46 | bottom: "pool1" 47 | top: "conv2" 48 | param { 49 | lr_mult: 1 50 | } 51 | param { 52 | lr_mult: 2 53 | } 54 | convolution_param { 55 | num_output: 32 56 | pad: 2 57 | kernel_size: 5 58 | stride: 1 59 | } 60 | } 61 | layer { 62 | name: "relu2" 63 | type: "ReLU" 64 | bottom: "conv2" 65 | top: "conv2" 66 | } 67 | layer { 68 | name: "pool2" 69 | type: "Pooling" 70 | bottom: "conv2" 71 | top: "pool2" 72 | pooling_param { 73 | pool: AVE 74 | kernel_size: 3 75 | stride: 2 76 | } 77 | } 78 | layer { 79 | name: "conv3" 80 | type: "Convolution" 81 | bottom: "pool2" 82 | top: "conv3" 83 | param { 84 | lr_mult: 1 85 | } 86 | param { 87 | lr_mult: 2 88 | } 89 | convolution_param { 90 | num_output: 64 91 | pad: 2 92 | kernel_size: 5 93 | stride: 1 94 | } 95 | } 96 | layer { 97 | name: "relu3" 98 | type: "ReLU" 99 | bottom: "conv3" 100 | top: "conv3" 101 | } 102 | layer { 103 | name: "pool3" 104 | type: "Pooling" 105 | bottom: "conv3" 106 | top: "pool3" 107 | pooling_param { 108 | pool: AVE 109 | kernel_size: 3 110 | stride: 2 111 | } 112 | } 113 | layer { 114 | name: "ip1" 115 | type: "InnerProduct" 116 | bottom: "pool3" 117 | top: "ip1" 118 | param { 119 | lr_mult: 1 120 | } 121 | param { 122 | lr_mult: 2 123 | } 124 | inner_product_param { 125 | num_output: 64 126 | } 127 | } 128 | layer { 129 | name: "ip2" 130 | type: "InnerProduct" 131 | bottom: "ip1" 132 | top: "ip2" 133 | param { 134 | lr_mult: 1 135 | } 136 | param { 137 | lr_mult: 2 138 | } 139 | inner_product_param { 140 | num_output: 10 141 | } 142 | } 143 | layer { 144 | name: "prob" 145 | type: "Softmax" 146 | bottom: "ip2" 147 | top: "prob" 148 | } 149 | -------------------------------------------------------------------------------- /proto/cifar10_quick_solver_imp.template: -------------------------------------------------------------------------------- 1 | # reduce the learning rate after 8 epochs (4000 iters) by a factor of 10 2 | 3 | # The train/test net protocol buffer definition 4 | net: "$prototxt" 5 | # test_iter specifies how many forward passes the test should carry out. 6 | # In the case of MNIST, we have test batch size 100 and 100 test iterations, 7 | # covering the full 10,000 testing images. 8 | test_iter: 100 9 | # Carry out testing every 500 training iterations. 10 | test_interval: 500 11 | # The base learning rate, momentum and the weight decay of the network. 12 | base_lr: 0.001 13 | momentum: 0.9 14 | weight_decay: 0.004 15 | # The learning rate policy 16 | lr_policy: "fixed" 17 | # Display every 100 iterations 18 | display: 100 19 | # The maximum number of iterations 20 | max_iter: 20000 21 | # snapshot intermediate results 22 | snapshot: 500 23 | snapshot_format: HDF5 24 | snapshot_prefix: "$snap_pre" 25 | # solver mode: CPU or GPU 26 | solver_mode: GPU 27 | -------------------------------------------------------------------------------- /proto/cifar10_quick_solver_noimp.template: -------------------------------------------------------------------------------- 1 | # reduce the learning rate after 8 epochs (4000 iters) by a factor of 10 2 | 3 | # The train/test net protocol buffer definition 4 | net: "$prototxt" 5 | # test_iter specifies how many forward passes the test should carry out. 6 | # In the case of MNIST, we have test batch size 100 and 100 test iterations, 7 | # covering the full 10,000 testing images. 8 | test_iter: 100 9 | # Carry out testing every 500 training iterations. 10 | test_interval: 500 11 | # The base learning rate, momentum and the weight decay of the network. 12 | base_lr: 0.001 13 | momentum: 0.9 14 | weight_decay: 0.004 15 | # The learning rate policy 16 | lr_policy: "fixed" 17 | # Display every 100 iterations 18 | display: 100 19 | # The maximum number of iterations 20 | max_iter: 4500 21 | # snapshot intermediate results 22 | snapshot: 4500 23 | snapshot_format: HDF5 24 | snapshot_prefix: "$snap_pre" 25 | # solver mode: CPU or GPU 26 | solver_mode: GPU 27 | -------------------------------------------------------------------------------- /proto/cifar10_quick_train_test.prototxt: -------------------------------------------------------------------------------- 1 | name: "CIFAR10_quick" 2 | layer { 3 | name: "cifar" 4 | type: "Data" 5 | top: "data" 6 | top: "label" 7 | include { 8 | phase: TRAIN 9 | } 10 | transform_param { 11 | mean_file: "examples/cifar10/mean.binaryproto" 12 | } 13 | data_param { 14 | source: "examples/cifar10/cifar10_train_lmdb" 15 | batch_size: 100 16 | backend: LMDB 17 | } 18 | } 19 | layer { 20 | name: "cifar" 21 | type: "Data" 22 | top: "data" 23 | top: "label" 24 | include { 25 | phase: TEST 26 | } 27 | transform_param { 28 | mean_file: "examples/cifar10/mean.binaryproto" 29 | } 30 | data_param { 31 | source: "examples/cifar10/cifar10_test_lmdb" 32 | batch_size: 100 33 | backend: LMDB 34 | } 35 | } 36 | layer { 37 | name: "conv1" 38 | type: "Convolution" 39 | bottom: "data" 40 | top: "conv1" 41 | param { 42 | lr_mult: 1 43 | } 44 | param { 45 | lr_mult: 2 46 | } 47 | convolution_param { 48 | num_output: 32 49 | pad: 2 50 | kernel_size: 5 51 | stride: 1 52 | weight_filler { 53 | type: "gaussian" 54 | std: 0.0001 55 | } 56 | bias_filler { 57 | type: "constant" 58 | } 59 | } 60 | } 61 | layer { 62 | name: "pool1" 63 | type: "Pooling" 64 | bottom: "conv1" 65 | top: "pool1" 66 | pooling_param { 67 | pool: MAX 68 | kernel_size: 3 69 | stride: 2 70 | } 71 | } 72 | layer { 73 | name: "relu1" 74 | type: "ReLU" 75 | bottom: "pool1" 76 | top: "pool1" 77 | } 78 | layer { 79 | name: "conv2" 80 | type: "Convolution" 81 | bottom: "pool1" 82 | top: "conv2" 83 | param { 84 | lr_mult: 1 85 | } 86 | param { 87 | lr_mult: 2 88 | } 89 | convolution_param { 90 | num_output: 32 91 | pad: 2 92 | kernel_size: 5 93 | stride: 1 94 | weight_filler { 95 | type: "gaussian" 96 | std: 0.01 97 | } 98 | bias_filler { 99 | type: "constant" 100 | } 101 | } 102 | } 103 | layer { 104 | name: "relu2" 105 | type: "ReLU" 106 | bottom: "conv2" 107 | top: "conv2" 108 | } 109 | layer { 110 | name: "pool2" 111 | type: "Pooling" 112 | bottom: "conv2" 113 | top: "pool2" 114 | pooling_param { 115 | pool: AVE 116 | kernel_size: 3 117 | stride: 2 118 | } 119 | } 120 | layer { 121 | name: "conv3" 122 | type: "Convolution" 123 | bottom: "pool2" 124 | top: "conv3" 125 | param { 126 | lr_mult: 1 127 | } 128 | param { 129 | lr_mult: 2 130 | } 131 | convolution_param { 132 | num_output: 64 133 | pad: 2 134 | kernel_size: 5 135 | stride: 1 136 | weight_filler { 137 | type: "gaussian" 138 | std: 0.01 139 | } 140 | bias_filler { 141 | type: "constant" 142 | } 143 | } 144 | } 145 | layer { 146 | name: "relu3" 147 | type: "ReLU" 148 | bottom: "conv3" 149 | top: "conv3" 150 | } 151 | layer { 152 | name: "pool3" 153 | type: "Pooling" 154 | bottom: "conv3" 155 | top: "pool3" 156 | pooling_param { 157 | pool: AVE 158 | kernel_size: 3 159 | stride: 2 160 | } 161 | } 162 | layer { 163 | name: "ip1" 164 | type: "InnerProduct" 165 | bottom: "pool3" 166 | top: "ip1" 167 | param { 168 | lr_mult: 1 169 | } 170 | param { 171 | lr_mult: 2 172 | } 173 | inner_product_param { 174 | num_output: 64 175 | weight_filler { 176 | type: "gaussian" 177 | std: 0.1 178 | } 179 | bias_filler { 180 | type: "constant" 181 | } 182 | } 183 | } 184 | layer { 185 | name: "ip2" 186 | type: "InnerProduct" 187 | bottom: "ip1" 188 | top: "ip2" 189 | param { 190 | lr_mult: 1 191 | } 192 | param { 193 | lr_mult: 2 194 | } 195 | inner_product_param { 196 | num_output: 10 197 | weight_filler { 198 | type: "gaussian" 199 | std: 0.1 200 | } 201 | bias_filler { 202 | type: "constant" 203 | } 204 | } 205 | } 206 | layer { 207 | name: "accuracy" 208 | type: "Accuracy" 209 | bottom: "ip2" 210 | bottom: "label" 211 | top: "accuracy" 212 | include { 213 | phase: TEST 214 | } 215 | } 216 | layer { 217 | name: "loss" 218 | type: "SoftmaxWithLoss" 219 | bottom: "ip2" 220 | bottom: "label" 221 | top: "loss" 222 | } 223 | -------------------------------------------------------------------------------- /proto/cifar10_quick_train_test_SVD.template: -------------------------------------------------------------------------------- 1 | name: "CIFAR10_quick" 2 | layer { 3 | name: "cifar" 4 | type: "Data" 5 | top: "data" 6 | top: "label" 7 | include { 8 | phase: TRAIN 9 | } 10 | transform_param { 11 | mean_file: "examples/cifar10/mean.binaryproto" 12 | } 13 | data_param { 14 | source: "examples/cifar10/cifar10_train_lmdb" 15 | batch_size: 100 16 | backend: LMDB 17 | } 18 | } 19 | layer { 20 | name: "cifar" 21 | type: "Data" 22 | top: "data" 23 | top: "label" 24 | include { 25 | phase: TEST 26 | } 27 | transform_param { 28 | mean_file: "examples/cifar10/mean.binaryproto" 29 | } 30 | data_param { 31 | source: "examples/cifar10/cifar10_test_lmdb" 32 | batch_size: 100 33 | backend: LMDB 34 | } 35 | } 36 | layer { 37 | name: "conv1" 38 | type: "Convolution" 39 | bottom: "data" 40 | top: "conv1" 41 | param { 42 | lr_mult: 1 43 | } 44 | param { 45 | lr_mult: 2 46 | } 47 | convolution_param { 48 | num_output: 32 49 | pad: 2 50 | kernel_size: 5 51 | stride: 1 52 | weight_filler { 53 | type: "gaussian" 54 | std: 0.0001 55 | } 56 | bias_filler { 57 | type: "constant" 58 | } 59 | } 60 | } 61 | layer { 62 | name: "pool1" 63 | type: "Pooling" 64 | bottom: "conv1" 65 | top: "pool1" 66 | pooling_param { 67 | pool: MAX 68 | kernel_size: 3 69 | stride: 2 70 | } 71 | } 72 | layer { 73 | name: "relu1" 74 | type: "ReLU" 75 | bottom: "pool1" 76 | top: "pool1" 77 | } 78 | layer { 79 | name: "conv2" 80 | type: "Convolution" 81 | bottom: "pool1" 82 | top: "conv2" 83 | param { 84 | lr_mult: 1 85 | } 86 | param { 87 | lr_mult: 2 88 | } 89 | convolution_param { 90 | num_output: 32 91 | pad: 2 92 | kernel_size: 5 93 | stride: 1 94 | weight_filler { 95 | type: "gaussian" 96 | std: 0.01 97 | } 98 | bias_filler { 99 | type: "constant" 100 | } 101 | } 102 | } 103 | layer { 104 | name: "relu2" 105 | type: "ReLU" 106 | bottom: "conv2" 107 | top: "conv2" 108 | } 109 | layer { 110 | name: "pool2" 111 | type: "Pooling" 112 | bottom: "conv2" 113 | top: "pool2" 114 | pooling_param { 115 | pool: AVE 116 | kernel_size: 3 117 | stride: 2 118 | } 119 | } 120 | layer { 121 | name: "conv3" 122 | type: "Convolution" 123 | bottom: "pool2" 124 | top: "conv3" 125 | param { 126 | lr_mult: 1 127 | } 128 | param { 129 | lr_mult: 2 130 | } 131 | convolution_param { 132 | num_output: 64 133 | pad: 2 134 | kernel_size: 5 135 | stride: 1 136 | weight_filler { 137 | type: "gaussian" 138 | std: 0.01 139 | } 140 | bias_filler { 141 | type: "constant" 142 | } 143 | } 144 | } 145 | layer { 146 | name: "relu3" 147 | type: "ReLU" 148 | bottom: "conv3" 149 | top: "conv3" 150 | } 151 | layer { 152 | name: "pool3" 153 | type: "Pooling" 154 | bottom: "conv3" 155 | top: "pool3" 156 | pooling_param { 157 | pool: AVE 158 | kernel_size: 3 159 | stride: 2 160 | } 161 | } 162 | layer { 163 | name: "ip1" 164 | type: "InnerProduct" 165 | bottom: "pool3" 166 | top: "ip1" 167 | param { 168 | lr_mult: 1 169 | } 170 | param { 171 | lr_mult: 2 172 | } 173 | inner_product_param { 174 | num_output: 64 175 | weight_filler { 176 | type: "gaussian" 177 | std: 0.1 178 | } 179 | bias_filler { 180 | type: "constant" 181 | } 182 | } 183 | } 184 | 185 | layer { 186 | name: "ipZ" 187 | type: "InnerProduct" 188 | bottom: "ip1" 189 | top: "ipZ" 190 | param { 191 | lr_mult: 1 192 | } 193 | param { 194 | lr_mult: 2 195 | } 196 | inner_product_param{ 197 | num_output: $ 198 | 199 | weight_filler { 200 | type: "gaussian" 201 | std: 0.1 202 | } 203 | bias_filler { 204 | type: "constant" 205 | } 206 | } 207 | } 208 | 209 | layer { 210 | name: "ipU" 211 | type: "InnerProduct" 212 | bottom: "ipZ" 213 | top: "ipSVD" 214 | param { 215 | lr_mult: 1 216 | } 217 | param { 218 | lr_mult: 2 219 | } 220 | inner_product_param{ 221 | num_output: 10 222 | 223 | weight_filler { 224 | type: "gaussian" 225 | std: 0.1 226 | } 227 | bias_filler { 228 | type: "constant" 229 | } 230 | } 231 | } 232 | 233 | layer { 234 | name: "accuracy" 235 | type: "Accuracy" 236 | bottom: "ipSVD" 237 | bottom: "label" 238 | top: "accuracy" 239 | include { 240 | phase: TEST 241 | } 242 | } 243 | layer { 244 | name: "loss" 245 | type: "SoftmaxWithLoss" 246 | bottom: "ipSVD" 247 | bottom: "label" 248 | top: "loss" 249 | } 250 | -------------------------------------------------------------------------------- /proto/solver.template: -------------------------------------------------------------------------------- 1 | # reduce the learning rate after 8 epochs (4000 iters) by a factor of 10 2 | 3 | # The train/test net protocol buffer definition 4 | net: "$prototxt" 5 | # test_iter specifies how many forward passes the test should carry out. 6 | # In the case of MNIST, we have test batch size 100 and 100 test iterations, 7 | # covering the full 10,000 testing images. 8 | test_iter: 100 9 | # Carry out testing every 500 training iterations. 10 | test_interval: 500 11 | # The base learning rate, momentum and the weight decay of the network. 12 | base_lr: 0.001 13 | momentum: 0.9 14 | weight_decay: 0.004 15 | # The learning rate policy 16 | lr_policy: "fixed" 17 | # Display every 100 iterations 18 | display: 100 19 | # The maximum number of iterations 20 | max_iter: 20000 21 | # snapshot intermediate results 22 | snapshot: 500 23 | snapshot_format: HDF5 24 | snapshot_prefix: "$snap_pre" 25 | # solver mode: CPU or GPU 26 | solver_mode: GPU 27 | -------------------------------------------------------------------------------- /proto/train_imp.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env sh 2 | set -e 3 | 4 | TOOLS=/home/luwei1/caffe/build/tools 5 | 6 | $TOOLS/caffe train \ 7 | --solver=$proto \ 8 | --weights $model $@ 9 | -------------------------------------------------------------------------------- /proto/train_quick_imp.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env sh 2 | set -e 3 | 4 | TOOLS=./build/tools 5 | 6 | $TOOLS/caffe train \ 7 | --solver=$proto \ 8 | --weights $model $@ 9 | -------------------------------------------------------------------------------- /proto/train_quick_noimp.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env sh 2 | set -e 3 | 4 | TOOLS=./build/tools 5 | 6 | $TOOLS/caffe train \ 7 | --solver=$proto $@ 8 | -------------------------------------------------------------------------------- /proto/train_test_SVD.template: -------------------------------------------------------------------------------- 1 | name: "CIFAR10_quick" 2 | layer { 3 | name: "cifar" 4 | type: "Data" 5 | top: "data" 6 | top: "label" 7 | include { 8 | phase: TRAIN 9 | } 10 | transform_param { 11 | mean_file: "/home/luwei1/caffe/examples/cifar10/mean.binaryproto" 12 | } 13 | data_param { 14 | source: "/home/luwei1/caffe/examples/cifar10/cifar10_train_lmdb" 15 | batch_size: 100 16 | backend: LMDB 17 | } 18 | } 19 | layer { 20 | name: "cifar" 21 | type: "Data" 22 | top: "data" 23 | top: "label" 24 | include { 25 | phase: TEST 26 | } 27 | transform_param { 28 | mean_file: "/home/luwei1/caffe/examples/cifar10/mean.binaryproto" 29 | } 30 | data_param { 31 | source: "/home/luwei1/caffe/examples/cifar10/cifar10_test_lmdb" 32 | batch_size: 100 33 | backend: LMDB 34 | } 35 | } 36 | layer { 37 | name: "conv1" 38 | type: "Convolution" 39 | bottom: "data" 40 | top: "conv1" 41 | param { 42 | lr_mult: 1 43 | } 44 | param { 45 | lr_mult: 2 46 | } 47 | convolution_param { 48 | num_output: 32 49 | pad: 2 50 | kernel_size: 5 51 | stride: 1 52 | weight_filler { 53 | type: "gaussian" 54 | std: 0.0001 55 | } 56 | bias_filler { 57 | type: "constant" 58 | } 59 | } 60 | } 61 | layer { 62 | name: "pool1" 63 | type: "Pooling" 64 | bottom: "conv1" 65 | top: "pool1" 66 | pooling_param { 67 | pool: MAX 68 | kernel_size: 3 69 | stride: 2 70 | } 71 | } 72 | layer { 73 | name: "relu1" 74 | type: "ReLU" 75 | bottom: "pool1" 76 | top: "pool1" 77 | } 78 | layer { 79 | name: "conv2" 80 | type: "Convolution" 81 | bottom: "pool1" 82 | top: "conv2" 83 | param { 84 | lr_mult: 1 85 | } 86 | param { 87 | lr_mult: 2 88 | } 89 | convolution_param { 90 | num_output: 32 91 | pad: 2 92 | kernel_size: 5 93 | stride: 1 94 | weight_filler { 95 | type: "gaussian" 96 | std: 0.01 97 | } 98 | bias_filler { 99 | type: "constant" 100 | } 101 | } 102 | } 103 | layer { 104 | name: "relu2" 105 | type: "ReLU" 106 | bottom: "conv2" 107 | top: "conv2" 108 | } 109 | layer { 110 | name: "pool2" 111 | type: "Pooling" 112 | bottom: "conv2" 113 | top: "pool2" 114 | pooling_param { 115 | pool: AVE 116 | kernel_size: 3 117 | stride: 2 118 | } 119 | } 120 | layer { 121 | name: "conv3" 122 | type: "Convolution" 123 | bottom: "pool2" 124 | top: "conv3" 125 | param { 126 | lr_mult: 1 127 | } 128 | param { 129 | lr_mult: 2 130 | } 131 | convolution_param { 132 | num_output: 64 133 | pad: 2 134 | kernel_size: 5 135 | stride: 1 136 | weight_filler { 137 | type: "gaussian" 138 | std: 0.01 139 | } 140 | bias_filler { 141 | type: "constant" 142 | } 143 | } 144 | } 145 | layer { 146 | name: "relu3" 147 | type: "ReLU" 148 | bottom: "conv3" 149 | top: "conv3" 150 | } 151 | layer { 152 | name: "pool3" 153 | type: "Pooling" 154 | bottom: "conv3" 155 | top: "pool3" 156 | pooling_param { 157 | pool: AVE 158 | kernel_size: 3 159 | stride: 2 160 | } 161 | } 162 | layer { 163 | name: "ip1" 164 | type: "InnerProduct" 165 | bottom: "pool3" 166 | top: "ip1" 167 | param { 168 | lr_mult: 1 169 | } 170 | param { 171 | lr_mult: 2 172 | } 173 | inner_product_param { 174 | num_output: 64 175 | weight_filler { 176 | type: "gaussian" 177 | std: 0.1 178 | } 179 | bias_filler { 180 | type: "constant" 181 | } 182 | } 183 | } 184 | 185 | layer { 186 | name: "ipZ" 187 | type: "InnerProduct" 188 | bottom: "ip1" 189 | top: "ipZ" 190 | param { 191 | lr_mult: 1 192 | } 193 | param { 194 | lr_mult: 2 195 | } 196 | inner_product_param{ 197 | num_output: $ 198 | 199 | weight_filler { 200 | type: "gaussian" 201 | std: 0.1 202 | } 203 | bias_filler { 204 | type: "constant" 205 | } 206 | } 207 | } 208 | 209 | layer { 210 | name: "ipU" 211 | type: "InnerProduct" 212 | bottom: "ipZ" 213 | top: "ipSVD" 214 | param { 215 | lr_mult: 1 216 | } 217 | param { 218 | lr_mult: 2 219 | } 220 | inner_product_param{ 221 | num_output: 10 222 | 223 | weight_filler { 224 | type: "gaussian" 225 | std: 0.1 226 | } 227 | bias_filler { 228 | type: "constant" 229 | } 230 | } 231 | } 232 | 233 | layer { 234 | name: "accuracy" 235 | type: "Accuracy" 236 | bottom: "ipSVD" 237 | bottom: "label" 238 | top: "accuracy" 239 | include { 240 | phase: TEST 241 | } 242 | } 243 | layer { 244 | name: "loss" 245 | type: "SoftmaxWithLoss" 246 | bottom: "ipSVD" 247 | bottom: "label" 248 | top: "loss" 249 | } 250 | -------------------------------------------------------------------------------- /proto/train_test_SVD_ip1.template: -------------------------------------------------------------------------------- 1 | name: "CIFAR10_quick" 2 | layer { 3 | name: "cifar" 4 | type: "Data" 5 | top: "data" 6 | top: "label" 7 | include { 8 | phase: TRAIN 9 | } 10 | transform_param { 11 | mean_file: "/home/luwei1/caffe/examples/cifar10/mean.binaryproto" 12 | } 13 | data_param { 14 | source: "/home/luwei1/caffe/examples/cifar10/cifar10_train_lmdb" 15 | batch_size: 100 16 | backend: LMDB 17 | } 18 | } 19 | layer { 20 | name: "cifar" 21 | type: "Data" 22 | top: "data" 23 | top: "label" 24 | include { 25 | phase: TEST 26 | } 27 | transform_param { 28 | mean_file: "/home/luwei1/caffe/examples/cifar10/mean.binaryproto" 29 | } 30 | data_param { 31 | source: "/home/luwei1/caffe/examples/cifar10/cifar10_test_lmdb" 32 | batch_size: 100 33 | backend: LMDB 34 | } 35 | } 36 | layer { 37 | name: "conv1" 38 | type: "Convolution" 39 | bottom: "data" 40 | top: "conv1" 41 | param { 42 | lr_mult: 1 43 | } 44 | param { 45 | lr_mult: 2 46 | } 47 | convolution_param { 48 | num_output: 32 49 | pad: 2 50 | kernel_size: 5 51 | stride: 1 52 | weight_filler { 53 | type: "gaussian" 54 | std: 0.0001 55 | } 56 | bias_filler { 57 | type: "constant" 58 | } 59 | } 60 | } 61 | layer { 62 | name: "pool1" 63 | type: "Pooling" 64 | bottom: "conv1" 65 | top: "pool1" 66 | pooling_param { 67 | pool: MAX 68 | kernel_size: 3 69 | stride: 2 70 | } 71 | } 72 | layer { 73 | name: "relu1" 74 | type: "ReLU" 75 | bottom: "pool1" 76 | top: "pool1" 77 | } 78 | layer { 79 | name: "conv2" 80 | type: "Convolution" 81 | bottom: "pool1" 82 | top: "conv2" 83 | param { 84 | lr_mult: 1 85 | } 86 | param { 87 | lr_mult: 2 88 | } 89 | convolution_param { 90 | num_output: 32 91 | pad: 2 92 | kernel_size: 5 93 | stride: 1 94 | weight_filler { 95 | type: "gaussian" 96 | std: 0.01 97 | } 98 | bias_filler { 99 | type: "constant" 100 | } 101 | } 102 | } 103 | layer { 104 | name: "relu2" 105 | type: "ReLU" 106 | bottom: "conv2" 107 | top: "conv2" 108 | } 109 | layer { 110 | name: "pool2" 111 | type: "Pooling" 112 | bottom: "conv2" 113 | top: "pool2" 114 | pooling_param { 115 | pool: AVE 116 | kernel_size: 3 117 | stride: 2 118 | } 119 | } 120 | layer { 121 | name: "conv3" 122 | type: "Convolution" 123 | bottom: "pool2" 124 | top: "conv3" 125 | param { 126 | lr_mult: 1 127 | } 128 | param { 129 | lr_mult: 2 130 | } 131 | convolution_param { 132 | num_output: 64 133 | pad: 2 134 | kernel_size: 5 135 | stride: 1 136 | weight_filler { 137 | type: "gaussian" 138 | std: 0.01 139 | } 140 | bias_filler { 141 | type: "constant" 142 | } 143 | } 144 | } 145 | layer { 146 | name: "relu3" 147 | type: "ReLU" 148 | bottom: "conv3" 149 | top: "conv3" 150 | } 151 | layer { 152 | name: "pool3" 153 | type: "Pooling" 154 | bottom: "conv3" 155 | top: "pool3" 156 | pooling_param { 157 | pool: AVE 158 | kernel_size: 3 159 | stride: 2 160 | } 161 | } 162 | 163 | 164 | layer { 165 | name: "ipZ" 166 | type: "InnerProduct" 167 | bottom: "pool3" 168 | top: "ipZ" 169 | 170 | param { 171 | lr_mult: 1 172 | } 173 | param { 174 | lr_mult: 2 175 | } 176 | 177 | inner_product_param{ 178 | num_output: $ 179 | 180 | weight_filler { 181 | type: "gaussian" 182 | std: 0.1 183 | } 184 | bias_filler { 185 | type: "constant" 186 | } 187 | } 188 | } 189 | 190 | layer { 191 | name: "ipU" 192 | type: "InnerProduct" 193 | bottom: "ipZ" 194 | top: "ipSVD" 195 | param { 196 | lr_mult: 1 197 | } 198 | param { 199 | lr_mult: 2 200 | } 201 | inner_product_param{ 202 | num_output: 64 203 | weight_filler { 204 | type: "gaussian" 205 | std: 0.1 206 | } 207 | bias_filler { 208 | type: "constant" 209 | } 210 | } 211 | } 212 | 213 | layer { 214 | name: "ip2" 215 | type: "InnerProduct" 216 | bottom: "ipSVD" 217 | top: "ip2" 218 | param { 219 | lr_mult: 1 220 | } 221 | param { 222 | lr_mult: 2 223 | } 224 | inner_product_param { 225 | num_output: 10 226 | weight_filler { 227 | type: "gaussian" 228 | std: 0.1 229 | } 230 | bias_filler { 231 | type: "constant" 232 | } 233 | } 234 | } 235 | layer { 236 | name: "accuracy" 237 | type: "Accuracy" 238 | bottom: "ip2" 239 | bottom: "label" 240 | top: "accuracy" 241 | include { 242 | phase: TEST 243 | } 244 | } 245 | layer { 246 | name: "loss" 247 | type: "SoftmaxWithLoss" 248 | bottom: "ip2" 249 | bottom: "label" 250 | top: "loss" 251 | } 252 | -------------------------------------------------------------------------------- /result/label.npy: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wkcn/CaffeSVD/ea4a61bf4a53f5c65f2f369242b1e8996e5d91bb/result/label.npy -------------------------------------------------------------------------------- /result/mean.npy: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wkcn/CaffeSVD/ea4a61bf4a53f5c65f2f369242b1e8996e5d91bb/result/mean.npy -------------------------------------------------------------------------------- /result/net_SVD10.npy: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wkcn/CaffeSVD/ea4a61bf4a53f5c65f2f369242b1e8996e5d91bb/result/net_SVD10.npy -------------------------------------------------------------------------------- /result/net_SVD3.npy: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wkcn/CaffeSVD/ea4a61bf4a53f5c65f2f369242b1e8996e5d91bb/result/net_SVD3.npy -------------------------------------------------------------------------------- /result/net_SVD4.npy: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wkcn/CaffeSVD/ea4a61bf4a53f5c65f2f369242b1e8996e5d91bb/result/net_SVD4.npy -------------------------------------------------------------------------------- /result/net_SVD5.npy: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wkcn/CaffeSVD/ea4a61bf4a53f5c65f2f369242b1e8996e5d91bb/result/net_SVD5.npy -------------------------------------------------------------------------------- /result/net_SVD6.npy: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wkcn/CaffeSVD/ea4a61bf4a53f5c65f2f369242b1e8996e5d91bb/result/net_SVD6.npy -------------------------------------------------------------------------------- /result/net_SVD7.npy: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wkcn/CaffeSVD/ea4a61bf4a53f5c65f2f369242b1e8996e5d91bb/result/net_SVD7.npy -------------------------------------------------------------------------------- /result/net_SVD8.npy: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wkcn/CaffeSVD/ea4a61bf4a53f5c65f2f369242b1e8996e5d91bb/result/net_SVD8.npy -------------------------------------------------------------------------------- /result/net_SVD9.npy: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wkcn/CaffeSVD/ea4a61bf4a53f5c65f2f369242b1e8996e5d91bb/result/net_SVD9.npy -------------------------------------------------------------------------------- /result/net_ip1_SVD25.npy: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wkcn/CaffeSVD/ea4a61bf4a53f5c65f2f369242b1e8996e5d91bb/result/net_ip1_SVD25.npy -------------------------------------------------------------------------------- /result/net_ip1_SVD35.npy: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wkcn/CaffeSVD/ea4a61bf4a53f5c65f2f369242b1e8996e5d91bb/result/net_ip1_SVD35.npy -------------------------------------------------------------------------------- /result/net_ip1_SVD40.npy: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wkcn/CaffeSVD/ea4a61bf4a53f5c65f2f369242b1e8996e5d91bb/result/net_ip1_SVD40.npy -------------------------------------------------------------------------------- /result/net_ip1_SVD52.npy: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wkcn/CaffeSVD/ea4a61bf4a53f5c65f2f369242b1e8996e5d91bb/result/net_ip1_SVD52.npy -------------------------------------------------------------------------------- /result/net_ip1_SVD55.npy: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wkcn/CaffeSVD/ea4a61bf4a53f5c65f2f369242b1e8996e5d91bb/result/net_ip1_SVD55.npy -------------------------------------------------------------------------------- /result/net_ip1_SVD60.npy: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wkcn/CaffeSVD/ea4a61bf4a53f5c65f2f369242b1e8996e5d91bb/result/net_ip1_SVD60.npy -------------------------------------------------------------------------------- /result/net_ip1_SVD62.npy: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wkcn/CaffeSVD/ea4a61bf4a53f5c65f2f369242b1e8996e5d91bb/result/net_ip1_SVD62.npy -------------------------------------------------------------------------------- /result/net_ip1_SVD63.npy: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wkcn/CaffeSVD/ea4a61bf4a53f5c65f2f369242b1e8996e5d91bb/result/net_ip1_SVD63.npy -------------------------------------------------------------------------------- /result/net_normal.npy: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wkcn/CaffeSVD/ea4a61bf4a53f5c65f2f369242b1e8996e5d91bb/result/net_normal.npy -------------------------------------------------------------------------------- /slides/slide.pptx: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wkcn/CaffeSVD/ea4a61bf4a53f5c65f2f369242b1e8996e5d91bb/slides/slide.pptx -------------------------------------------------------------------------------- /slides/slide2.pptx: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wkcn/CaffeSVD/ea4a61bf4a53f5c65f2f369242b1e8996e5d91bb/slides/slide2.pptx -------------------------------------------------------------------------------- /test.py: -------------------------------------------------------------------------------- 1 | #coding=utf-8 2 | # decompress ip2 layer 3 | import caffe 4 | from caffe import layers as L, params as P, to_proto 5 | from caffe.proto import caffe_pb2 6 | import lmdb 7 | import numpy as np 8 | import os 9 | import sys 10 | from numpy import linalg as la 11 | import matplotlib.pyplot as plt 12 | from base import * 13 | 14 | CAFFE_HOME = "/opt/caffe/" 15 | RESULT_DIR = "./result/" 16 | 17 | SVD_R = 6 18 | deploySVD = GetSVDProto(SVD_R) 19 | 20 | deploy = "./proto/cifar10_quick.prototxt" 21 | caffe_model = CAFFE_HOME + "/examples/cifar10/cifar10_quick_iter_5000.caffemodel.h5" 22 | train_db = CAFFE_HOME + "examples/cifar10/cifar10_train_lmdb" 23 | test_db = CAFFE_HOME + "examples/cifar10/cifar10_test_lmdb" 24 | mean_proto = CAFFE_HOME + "examples/cifar10/mean.binaryproto" 25 | mean_npy = "./mean.npy" 26 | mean_pic = np.load(mean_npy) 27 | 28 | def read_db(db_name): 29 | lmdb_env = lmdb.open(db_name) 30 | lmdb_txn = lmdb_env.begin() 31 | lmdb_cursor = lmdb_txn.cursor() 32 | datum = caffe.proto.caffe_pb2.Datum() 33 | 34 | X = [] 35 | y = [] 36 | cnts = {} 37 | for key, value in lmdb_cursor: 38 | datum.ParseFromString(value) 39 | label = datum.label 40 | data = caffe.io.datum_to_array(datum) 41 | #data = data.swapaxes(0, 2).swapaxes(0, 1) 42 | X.append(data) 43 | y.append(label) 44 | if label not in cnts: 45 | cnts[label] = 0 46 | cnts[label] += 1 47 | #plt.imshow(data) 48 | #plt.show() 49 | return X, np.array(y), cnts 50 | 51 | testX, testy, cnts = read_db(test_db) 52 | #testX, testy, cnts = read_db(train_db) 53 | print ("#train set: ", len(testX)) 54 | print ("the size of sample:", testX[0].shape) 55 | print ("kinds: ", cnts) 56 | 57 | if not os.path.exists("label.npy"): 58 | np.save("label.npy", testy) 59 | 60 | # Load model and network 61 | net = caffe.Net(deploy, caffe_model, caffe.TEST) 62 | for layer_name, param in net.params.items(): 63 | # 0 is weight, 1 is biases 64 | print (layer_name, param[0].data.shape,net.blobs[layer_name].data.shape) 65 | 66 | if SVD_R > 0: 67 | netSVD = caffe.Net(deploySVD, caffe_model, caffe.TEST) 68 | print ("SVD NET:") 69 | for layer_name, param in netSVD.params.items(): 70 | # 0 is weight, 1 is biases 71 | print (layer_name, param[0].data.shape) 72 | 73 | print (type(net.params)) 74 | print (net.params.keys()) 75 | print ("layer ip2:") 76 | print ("WEIGHT:") 77 | print (net.params["ip2"][0].data.shape) 78 | print ("BIASES:") 79 | print (net.params["ip2"][1].data.shape) 80 | 81 | 82 | data, label = L.Data(source = test_db, backend = P.Data.LMDB, batch_size = 100, ntop = 2, mean_file = mean_proto) 83 | 84 | 85 | if SVD_R > 0: 86 | # SVD 87 | print ("SVD %d" % SVD_R) 88 | u, sigma, vt = la.svd(net.params["ip2"][0].data) 89 | print ("Sigma: ", sigma) 90 | if SVD_R > len(sigma): 91 | print ("SVD_R is too large :-(") 92 | sys.exit() 93 | U = np.matrix(u[:, :SVD_R]) 94 | S = np.matrix(np.diag(sigma[:SVD_R])) 95 | VT = np.matrix(vt[:SVD_R, :]) 96 | print ("IP2", net.params["ip2"][0].data.shape) # 10, 64 97 | print ("U", U.shape) 98 | print ("S", S.shape) 99 | print ("VT", VT.shape) 100 | 101 | # y = Wx + b 102 | # y = U * S * VT * x + b 103 | # y = U * ((S * VT) * x) + b 104 | # y = U * (Z * x) + b 105 | Z = S * VT 106 | 107 | np.copyto(netSVD.params["ipZ"][0].data, Z) 108 | np.copyto(netSVD.params["ipU"][0].data, U) 109 | np.copyto(netSVD.params["ipU"][1].data, net.params["ip2"][1].data) 110 | 111 | nn = netSVD 112 | #nn.save("net_SVD%d.caffemodel" % SVD_R) 113 | 114 | else: 115 | print ("NORMAL") 116 | nn = net 117 | 118 | 119 | n = len(testX) 120 | pre = np.zeros(testy.shape) 121 | print ("N = %d" % n) 122 | for i in range(n): 123 | nn.blobs["data"].data[...] = testX[i] - mean_pic 124 | nn.forward() 125 | prob = nn.blobs["prob"].data 126 | pre[i] = prob.argmax() 127 | print ("%d / %d" % (i + 1, n)) 128 | right = np.sum(pre == testy) 129 | print ("Accuracy: %f" % (right * 1.0 / n)) 130 | 131 | 132 | if SVD_R > 0: 133 | np.save(RESULT_DIR + "net_SVD%d.npy" % SVD_R, pre) 134 | else: 135 | np.save(RESULT_DIR + "net_normal.npy", pre) 136 | -------------------------------------------------------------------------------- /test2.py: -------------------------------------------------------------------------------- 1 | #coding=utf-8 2 | # decompress ip1 layer 3 | import caffe 4 | from caffe import layers as L, params as P, to_proto 5 | from caffe.proto import caffe_pb2 6 | import lmdb 7 | import numpy as np 8 | import os 9 | import sys 10 | from numpy import linalg as la 11 | import matplotlib.pyplot as plt 12 | from base import * 13 | 14 | CAFFE_HOME = "/opt/caffe/" 15 | RESULT_DIR = "./result/" 16 | 17 | SVD_R = 50 18 | deploySVD = GetIP1SVDProto(SVD_R) 19 | 20 | deploy = "./proto/cifar10_quick.prototxt" 21 | caffe_model = CAFFE_HOME + "/examples/cifar10/cifar10_quick_iter_5000.caffemodel.h5" 22 | train_db = CAFFE_HOME + "examples/cifar10/cifar10_train_lmdb" 23 | test_db = CAFFE_HOME + "examples/cifar10/cifar10_test_lmdb" 24 | mean_proto = CAFFE_HOME + "examples/cifar10/mean.binaryproto" 25 | mean_npy = "./mean.npy" 26 | mean_pic = np.load(mean_npy) 27 | 28 | def read_db(db_name): 29 | lmdb_env = lmdb.open(db_name) 30 | lmdb_txn = lmdb_env.begin() 31 | lmdb_cursor = lmdb_txn.cursor() 32 | datum = caffe.proto.caffe_pb2.Datum() 33 | 34 | X = [] 35 | y = [] 36 | for key, value in lmdb_cursor: 37 | datum.ParseFromString(value) 38 | label = datum.label 39 | data = caffe.io.datum_to_array(datum) 40 | #data = data.swapaxes(0, 2).swapaxes(0, 1) 41 | X.append(data) 42 | y.append(label) 43 | #plt.imshow(data) 44 | #plt.show() 45 | return X, np.array(y) 46 | 47 | testX, testy = read_db(test_db) 48 | 49 | if not os.path.exists("label.npy"): 50 | np.save("label.npy", testy) 51 | 52 | # Load model and network 53 | net = caffe.Net(deploy, caffe_model, caffe.TEST) 54 | for layer_name, param in net.params.items(): 55 | # 0 is weight, 1 is biases 56 | print (layer_name, param[0].data.shape) 57 | if SVD_R > 0: 58 | netSVD = caffe.Net(deploySVD, caffe_model, caffe.TEST) 59 | print ("SVD NET:") 60 | for layer_name, param in netSVD.params.items(): 61 | # 0 is weight, 1 is biases 62 | print (layer_name, param[0].data.shape) 63 | 64 | print (type(net.params)) 65 | print (net.params.keys()) 66 | print ("layer ip2:") 67 | print ("WEIGHT:") 68 | print (net.params["ip2"][0].data.shape) 69 | print ("BIASES:") 70 | print (net.params["ip2"][1].data.shape) 71 | 72 | 73 | data, label = L.Data(source = test_db, backend = P.Data.LMDB, batch_size = 100, ntop = 2, mean_file = mean_proto) 74 | 75 | 76 | if SVD_R > 0: 77 | # SVD 78 | print ("SVD %d" % SVD_R) 79 | u, sigma, vt = la.svd(net.params["ip1"][0].data) 80 | print ("Sigma: ", sigma) 81 | if SVD_R > len(sigma): 82 | print ("SVD_R is too large :-(") 83 | sys.exit() 84 | U = np.matrix(u[:, :SVD_R]) 85 | S = np.matrix(np.diag(sigma[:SVD_R])) 86 | VT = np.matrix(vt[:SVD_R, :]) 87 | print ("IP2", net.params["ip1"][0].data.shape) # 10, 64 88 | print ("U", U.shape) 89 | print ("S", S.shape) 90 | print ("VT", VT.shape) 91 | 92 | # y = Wx + b 93 | # y = U * S * VT * x + b 94 | # y = U * ((S * VT) * x) + b 95 | # y = U * (Z * x) + b 96 | Z = S * VT 97 | 98 | np.copyto(netSVD.params["ipZ"][0].data, Z) 99 | np.copyto(netSVD.params["ipU"][0].data, U) 100 | np.copyto(netSVD.params["ipU"][1].data, net.params["ip1"][1].data) 101 | 102 | nn = netSVD 103 | else: 104 | print ("NORMAL") 105 | nn = net 106 | 107 | 108 | n = len(testX) 109 | pre = np.zeros(testy.shape) 110 | print ("N = %d" % n) 111 | for i in range(n): 112 | nn.blobs["data"].data[...] = testX[i] - mean_pic 113 | nn.forward() 114 | prob = nn.blobs["prob"].data 115 | pre[i] = prob.argmax() 116 | print ("%d / %d" % (i + 1, n)) 117 | right = np.sum(pre == testy) 118 | print ("Accuracy: %f" % (right * 1.0 / n)) 119 | 120 | 121 | if SVD_R > 0: 122 | np.save(RESULT_DIR + "net_ip1_SVD%d.npy" % SVD_R, pre) 123 | else: 124 | np.save(RESULT_DIR + "net_ip1_normal.npy", pre) 125 | -------------------------------------------------------------------------------- /test_ip1_new.py: -------------------------------------------------------------------------------- 1 | #coding=utf-8 2 | # test decompress ip1 layer 3 | import caffe 4 | from caffe import layers as L, params as P, to_proto 5 | from caffe.proto import caffe_pb2 6 | import lmdb 7 | import numpy as np 8 | import os 9 | import sys 10 | from numpy import linalg as la 11 | import matplotlib.pyplot as plt 12 | from base import * 13 | 14 | CAFFE_HOME = "/opt/caffe/" 15 | RESULT_DIR = "./result/" 16 | 17 | SVD_R = 6 # 64 x 1024 18 | deploySVD = GetIP1SVDProto(SVD_R) 19 | iter_num = 19000 20 | 21 | train_db = CAFFE_HOME + "examples/cifar10/cifar10_train_lmdb" 22 | test_db = CAFFE_HOME + "examples/cifar10/cifar10_test_lmdb" 23 | mean_proto = CAFFE_HOME + "examples/cifar10/mean.binaryproto" 24 | mean_npy = "./mean.npy" 25 | mean_pic = np.load(mean_npy) 26 | 27 | def read_db(db_name): 28 | lmdb_env = lmdb.open(db_name) 29 | lmdb_txn = lmdb_env.begin() 30 | lmdb_cursor = lmdb_txn.cursor() 31 | datum = caffe.proto.caffe_pb2.Datum() 32 | 33 | X = [] 34 | y = [] 35 | cnts = {} 36 | for key, value in lmdb_cursor: 37 | datum.ParseFromString(value) 38 | label = datum.label 39 | data = caffe.io.datum_to_array(datum) 40 | #data = data.swapaxes(0, 2).swapaxes(0, 1) 41 | X.append(data) 42 | y.append(label) 43 | if label not in cnts: 44 | cnts[label] = 0 45 | cnts[label] += 1 46 | #plt.imshow(data) 47 | #plt.show() 48 | return X, np.array(y), cnts 49 | 50 | testX, testy, cnts = read_db(test_db) 51 | #testX, testy, cnts = read_db(train_db) 52 | print ("#train set: ", len(testX)) 53 | print ("the size of sample:", testX[0].shape) 54 | print ("kinds: ", cnts) 55 | 56 | if not os.path.exists("label.npy"): 57 | np.save("label.npy", testy) 58 | 59 | # 生成配置文件 60 | # CAFFE_HOME 61 | example_dir = CAFFE_HOME + "examples/cifar10/" 62 | build_dir = "./build_ip1/" 63 | 64 | # 加载新的模型 65 | new_model = "./new/ip1_SVD%d/ip1_SVD%d_iter_%d.caffemodel.h5" % (SVD_R,SVD_R, iter_num) 66 | nn = caffe.Net(deploySVD, new_model, caffe.TEST) 67 | 68 | n = len(testX) 69 | pre = np.zeros(testy.shape) 70 | print ("N = %d" % n) 71 | for i in range(n): 72 | nn.blobs["data"].data[...] = testX[i] - mean_pic 73 | nn.forward() 74 | prob = nn.blobs["prob"].data 75 | pre[i] = prob.argmax() 76 | print ("%d / %d" % (i + 1, n)) 77 | right = np.sum(pre == testy) 78 | print ("Accuracy: %f" % (right * 1.0 / n)) 79 | 80 | 81 | np.save(RESULT_DIR + "ip1_SVD%d_iter%d_ft.npy" % (SVD_R, iter_num), pre) 82 | -------------------------------------------------------------------------------- /test_new.py: -------------------------------------------------------------------------------- 1 | #coding=utf-8 2 | # decompress ip2 layer 3 | import caffe 4 | from caffe import layers as L, params as P, to_proto 5 | from caffe.proto import caffe_pb2 6 | import lmdb 7 | import numpy as np 8 | import os 9 | import sys 10 | from numpy import linalg as la 11 | import matplotlib.pyplot as plt 12 | from base import * 13 | 14 | CAFFE_HOME = "/opt/caffe/" 15 | RESULT_DIR = "./result/" 16 | 17 | SVD_R = 0 18 | deploySVD = GetSVDProto(SVD_R) 19 | 20 | deploy = "./proto/cifar10_quick.prototxt" 21 | caffe_model = "./new/cifar10_quick/source/cifar10_quick_iter_20000.caffemodel.h5" 22 | train_db = CAFFE_HOME + "examples/cifar10/cifar10_train_lmdb" 23 | test_db = CAFFE_HOME + "examples/cifar10/cifar10_test_lmdb" 24 | mean_proto = CAFFE_HOME + "examples/cifar10/mean.binaryproto" 25 | mean_npy = "./mean.npy" 26 | mean_pic = np.load(mean_npy) 27 | 28 | def read_db(db_name): 29 | lmdb_env = lmdb.open(db_name) 30 | lmdb_txn = lmdb_env.begin() 31 | lmdb_cursor = lmdb_txn.cursor() 32 | datum = caffe.proto.caffe_pb2.Datum() 33 | 34 | X = [] 35 | y = [] 36 | cnts = {} 37 | for key, value in lmdb_cursor: 38 | datum.ParseFromString(value) 39 | label = datum.label 40 | data = caffe.io.datum_to_array(datum) 41 | #data = data.swapaxes(0, 2).swapaxes(0, 1) 42 | X.append(data) 43 | y.append(label) 44 | if label not in cnts: 45 | cnts[label] = 0 46 | cnts[label] += 1 47 | #plt.imshow(data) 48 | #plt.show() 49 | return X, np.array(y), cnts 50 | 51 | testX, testy, cnts = read_db(test_db) 52 | #testX, testy, cnts = read_db(train_db) 53 | print ("#train set: ", len(testX)) 54 | print ("the size of sample:", testX[0].shape) 55 | print ("kinds: ", cnts) 56 | 57 | if not os.path.exists("label.npy"): 58 | np.save("label.npy", testy) 59 | 60 | # Load model and network 61 | net = caffe.Net(deploy, caffe_model, caffe.TEST) 62 | for layer_name, param in net.params.items(): 63 | # 0 is weight, 1 is biases 64 | print (layer_name, param[0].data.shape,net.blobs[layer_name].data.shape) 65 | 66 | if SVD_R > 0: 67 | netSVD = caffe.Net(deploySVD, caffe_model, caffe.TEST) 68 | print ("SVD NET:") 69 | for layer_name, param in netSVD.params.items(): 70 | # 0 is weight, 1 is biases 71 | print (layer_name, param[0].data.shape) 72 | 73 | print (type(net.params)) 74 | print (net.params.keys()) 75 | print ("layer ip2:") 76 | print ("WEIGHT:") 77 | print (net.params["ip2"][0].data.shape) 78 | print ("BIASES:") 79 | print (net.params["ip2"][1].data.shape) 80 | 81 | 82 | data, label = L.Data(source = test_db, backend = P.Data.LMDB, batch_size = 100, ntop = 2, mean_file = mean_proto) 83 | 84 | 85 | if SVD_R > 0: 86 | # SVD 87 | print ("SVD %d" % SVD_R) 88 | u, sigma, vt = la.svd(net.params["ip2"][0].data) 89 | print ("Sigma: ", sigma) 90 | if SVD_R > len(sigma): 91 | print ("SVD_R is too large :-(") 92 | sys.exit() 93 | U = np.matrix(u[:, :SVD_R]) 94 | S = np.matrix(np.diag(sigma[:SVD_R])) 95 | VT = np.matrix(vt[:SVD_R, :]) 96 | print ("IP2", net.params["ip2"][0].data.shape) # 10, 64 97 | print ("U", U.shape) 98 | print ("S", S.shape) 99 | print ("VT", VT.shape) 100 | 101 | # y = Wx + b 102 | # y = U * S * VT * x + b 103 | # y = U * ((S * VT) * x) + b 104 | # y = U * (Z * x) + b 105 | Z = S * VT 106 | 107 | np.copyto(netSVD.params["ipZ"][0].data, Z) 108 | np.copyto(netSVD.params["ipU"][0].data, U) 109 | np.copyto(netSVD.params["ipU"][1].data, net.params["ip2"][1].data) 110 | 111 | nn = netSVD 112 | #nn.save("net_SVD%d.caffemodel" % SVD_R) 113 | 114 | else: 115 | print ("NORMAL") 116 | nn = net 117 | 118 | 119 | n = len(testX) 120 | pre = np.zeros(testy.shape) 121 | print ("N = %d" % n) 122 | for i in range(n): 123 | nn.blobs["data"].data[...] = testX[i] - mean_pic 124 | nn.forward() 125 | prob = nn.blobs["prob"].data 126 | pre[i] = prob.argmax() 127 | print ("%d / %d" % (i + 1, n)) 128 | right = np.sum(pre == testy) 129 | print ("Accuracy: %f" % (right * 1.0 / n)) 130 | 131 | 132 | if SVD_R > 0: 133 | np.save(RESULT_DIR + "net_SVD%d.npy" % SVD_R, pre) 134 | else: 135 | np.save(RESULT_DIR + "net_normal_20000.npy", pre) 136 | --------------------------------------------------------------------------------