├── ImgInWebsite ├── __init__.py ├── __init__.pyc ├── imgs.txt ├── items.py ├── pipelines.py ├── settings.py ├── settings.pyc └── spiders │ ├── ImgSpider.py │ ├── __init__.py │ └── __init__.pyc ├── README.md └── scrapy.cfg /ImgInWebsite/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wly2014/ImageSpider/97cdaff203b407d6a9a9e103e8c74fe525f1dda6/ImgInWebsite/__init__.py -------------------------------------------------------------------------------- /ImgInWebsite/__init__.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wly2014/ImageSpider/97cdaff203b407d6a9a9e103e8c74fe525f1dda6/ImgInWebsite/__init__.pyc -------------------------------------------------------------------------------- /ImgInWebsite/imgs.txt: -------------------------------------------------------------------------------- 1 | http://cimg2.res.meizu.com/www/201611/1478251988566.jpg 2 | http://cimg2.res.meizu.com/www/201611/1478252014158.jpg 3 | http://cimg2.res.meizu.com/www/201611/1478484390321.jpg 4 | http://cimg2.res.meizu.com/www/201611/1478484390323.jpg 5 | http://cimg2.res.meizu.com/www/201611/1478170715515.jpg 6 | http://cimg2.res.meizu.com/www/201611/1478170715512.jpg 7 | http://cimg2.res.meizu.com/www/201611/1478167571669.jpg 8 | http://cimg2.res.meizu.com/www/201611/1478167578803.jpg 9 | http://cimg2.res.meizu.com/www/201611/1478488348706.jpg 10 | http://cimg2.res.meizu.com/www/201611/1478488348708.jpg 11 | http://cimg2.res.meizu.com/www/201610/1477906687677.jpg 12 | http://cimg2.res.meizu.com/www/201610/1477906695021.jpg 13 | http://www3.res.meizu.com/static/cn/_partial/header/images/pro6s_e769573.png 14 | http://www3.res.meizu.com/static/cn/_partial/header/images/pro6s@2x_99bafaa.png 15 | http://www3.res.meizu.com/static/cn/_partial/header/images/pro6_d31bab6.png 16 | http://www3.res.meizu.com/static/cn/_partial/header/images/pro6@2x_d2edd30.png 17 | http://www3.res.meizu.com/static/cn/_partial/header/images/pro5_e0970c9.png 18 | http://www3.res.meizu.com/static/cn/_partial/header/images/pro5@2x_84bf375.png 19 | http://www3.res.meizu.com/static/cn/_partial/header/images/mx5_d0aca4c.png 20 | http://www3.res.meizu.com/static/cn/_partial/header/images/mx5@2x_06f9706.png 21 | http://www3.res.meizu.com/static/cn/_partial/header/images/mx6_23e5fc6.png 22 | http://www3.res.meizu.com/static/cn/_partial/header/images/mx6@2x_1512beb.png 23 | http://www3.res.meizu.com/static/cn/_partial/header/images/mx4pro_55776f8.png 24 | http://www3.res.meizu.com/static/cn/_partial/header/images/mx4pro@2x_ec54581.png 25 | http://www3.res.meizu.com/static/cn/_partial/header/images/mx4_07bfbac.png 26 | http://www3.res.meizu.com/static/cn/_partial/header/images/mx4@2x_fa67ebb.png 27 | http://www3.res.meizu.com/static/cn/_partial/header/images/m3note_d4c10cf.png 28 | http://www3.res.meizu.com/static/cn/_partial/header/images/m3note@2x_fd8f6e7.png 29 | http://www3.res.meizu.com/static/cn/_partial/header/images/meilanmetal_ffc5a14.png 30 | http://www3.res.meizu.com/static/cn/_partial/header/images/meilanmetal@2x_c1d10e3.png 31 | http://www3.res.meizu.com/static/cn/_partial/header/images/m2note_2c82398.png 32 | http://www3.res.meizu.com/static/cn/_partial/header/images/m2note@2x_1a87fe1.png 33 | http://www3.res.meizu.com/static/cn/_partial/header/images/meilannote_0f6dded.png 34 | http://www3.res.meizu.com/static/cn/_partial/header/images/meilannote@2x_b187893.png 35 | http://www3.res.meizu.com/static/cn/_partial/header/images/m5_45957e3.png 36 | http://www3.res.meizu.com/static/cn/_partial/header/images/m5@2x_23b8aad.png 37 | http://www3.res.meizu.com/static/cn/_partial/header/images/m3e_a922cdc.png 38 | http://www3.res.meizu.com/static/cn/_partial/header/images/m3e@2x_7095bb2.png 39 | http://www3.res.meizu.com/static/cn/_partial/header/images/m3s_7cff047.png 40 | http://www3.res.meizu.com/static/cn/_partial/header/images/m3s@2x_e2379c2.png 41 | http://www3.res.meizu.com/static/cn/_partial/header/images/m3_fa5c797.png 42 | http://www3.res.meizu.com/static/cn/_partial/header/images/m3@2x_01ec0cf.png 43 | http://www3.res.meizu.com/static/cn/_partial/header/images/m2_8280688.png 44 | http://www3.res.meizu.com/static/cn/_partial/header/images/m2@2x_99198cc.png 45 | http://www3.res.meizu.com/static/cn/_partial/header/images/meilan_a18cc7a.png 46 | http://www3.res.meizu.com/static/cn/_partial/header/images/meilan@2x_4683105.png 47 | http://www3.res.meizu.com/static/cn/_partial/header/images/router_1280efe.png 48 | http://www3.res.meizu.com/static/cn/_partial/header/images/router@2x_87c1710.png 49 | http://www3.res.meizu.com/static/cn/_partial/header/images/router_s_82d6bf4.png 50 | http://www3.res.meizu.com/static/cn/_partial/header/images/router_s@2x_a8564e2.png 51 | http://www3.res.meizu.com/static/cn/_partial/header/images/m8_87170a0.png 52 | http://www3.res.meizu.com/static/cn/_partial/header/images/m8@2x_52e6a4f.png 53 | http://www3.res.meizu.com/static/cn/_partial/header/images/m10_15b9f13.png 54 | http://www3.res.meizu.com/static/cn/_partial/header/images/m10@2x_788831f.png 55 | http://www3.res.meizu.com/static/cn/_partial/header/images/hd50_0ef1eab.png 56 | http://www3.res.meizu.com/static/cn/_partial/header/images/hd50@2x_e7c3538.png 57 | http://www3.res.meizu.com/static/cn/_partial/header/images/ep51_7b728d8.png 58 | http://www3.res.meizu.com/static/cn/_partial/header/images/ep51@2x_b795648.png 59 | http://www3.res.meizu.com/static/cn/_partial/header/images/ep2x_d112722.png 60 | http://www3.res.meizu.com/static/cn/_partial/header/images/ep2x@2x_0c902f3.png 61 | http://www3.res.meizu.com/static/cn/_partial/header/images/loopjacket_003383b.png 62 | http://www3.res.meizu.com/static/cn/_partial/header/images/loopjacket@2x_5c0de55.png 63 | http://www3.res.meizu.com/static/cn/_partial/header/images/mmax_d7883c0.png 64 | http://www3.res.meizu.com/static/cn/_partial/header/images/mmax@2x_0449ec0.png 65 | http://www3.res.meizu.com/static/cn/_partial/header/images/u10_3e53f09.png 66 | http://www3.res.meizu.com/static/cn/_partial/header/images/u10@2x_800bf1d.png 67 | http://www3.res.meizu.com/static/cn/_partial/header/images/u20_5f69f11.png 68 | http://www3.res.meizu.com/static/cn/_partial/header/images/u20@2x_7534854.png 69 | http://www3.res.meizu.com/static/cn/_partial/header/images/backpack_b762aa8.png 70 | http://www3.res.meizu.com/static/cn/_partial/header/images/backpack@2x_0e6e09a.png 71 | http://www3.res.meizu.com/static/cn/_partial/header/images/musiccard_77b8fdf.png 72 | http://www3.res.meizu.com/static/cn/_partial/header/images/musiccard@2x_2fb411b.png 73 | http://www3.res.meizu.com/static/cn/_partial/header/images/tvbox_cd026fb.png 74 | http://www3.res.meizu.com/static/cn/_partial/header/images/tvbox@2x_7e79186.png 75 | http://cimg2.res.meizu.com/www/201607/1467873105514.jpg 76 | http://cimg2.res.meizu.com/www/201607/1467873105519.jpg 77 | http://cimg2.res.meizu.com/www/201605/1462760369902.jpg 78 | http://cimg2.res.meizu.com/www/201605/1462760369898.jpg 79 | http://cimg2.res.meizu.com/www/201610/1476091013254.jpg 80 | http://cimg2.res.meizu.com/www/201610/1476091091349.jpg 81 | http://www.meizu.com/static/_partial/footer/icp2.jpg 82 | http://www.meizu.com/static/_partial/footer/com_licence.jpg 83 | http://www3.res.meizu.com/static/cn/index/images/about_1_4fdaece.jpg 84 | http://www3.res.meizu.com/static/cn/index/images/about_2_6554945.jpg 85 | http://www3.res.meizu.com/static/cn/index/images/about_3_2431f25.jpg 86 | http://www3.res.meizu.com/static/cn/index/images/enter1_7748b8e.png 87 | http://www3.res.meizu.com/static/cn/index/images/enter2_3f42260.png 88 | http://www3.res.meizu.com/static/cn/index/images/enter3_270f7c3.png 89 | http://cimg2.res.meizu.com/www/201607/1468810424596.png 90 | http://www3.res.meizu.com/static/cn/accessory/loopjacket/images/flyme1_65c7bc0.png 91 | http://www3.res.meizu.com/static/cn/accessory/loopjacket/images/flyme2_189052e.png 92 | http://www3.res.meizu.com/static/cn/accessory/loopjacket/images/flyme3_4871566.png 93 | http://www3.res.meizu.com/static/cn/accessory/backpack/images/album1_57631d9.jpg 94 | http://www3.res.meizu.com/static/cn/accessory/backpack/images/album2_a275ece.jpg 95 | http://www3.res.meizu.com/static/cn/accessory/backpack/images/album3_2cdd607.jpg 96 | http://www3.res.meizu.com/static/cn/accessory/hd50/summary/images/cross_shadow_b6cfed0.jpg 97 | http://www3.res.meizu.com/static/cn/accessory/hd50/summary/images/cross_white_d2d07af.png 98 | http://www3.res.meizu.com/static/cn/accessory/hd50/summary/images/cross_black2_00dbb89.png 99 | http://www3.res.meizu.com/static/cn/accessory/hd50/summary/images/cross_black1_7df2568.png 100 | http://www3.res.meizu.com/static/cn/accessory/hd50/summary/images/chart_63b4754.png 101 | http://www3.res.meizu.com/static/cn/accessory/hd50/summary/images/chart@2x_3f8432e.png 102 | http://www3.res.meizu.com/static/cn/accessory/hd50/summary/images/chart_blue_709c9de.png 103 | http://www3.res.meizu.com/static/cn/accessory/hd50/summary/images/chart_blue@2x_aec2826.png 104 | http://www3.res.meizu.com/static/cn/accessory/hd50/summary/images/chart_red_73ef05b.png 105 | http://www3.res.meizu.com/static/cn/accessory/hd50/summary/images/chart_red@2x_76aa133.png 106 | http://www3.res.meizu.com/static/cn/accessory/hd50/summary/images/line_98a9ccc.png 107 | http://www3.res.meizu.com/static/cn/accessory/hd50/summary/images/headphone_seq_1a02c63.png 108 | http://www3.res.meizu.com/static/cn/accessory/hd50/summary/images/gallery_next_d5109c7.png 109 | http://www3.res.meizu.com/static/cn/smart/router_s/summary/images/banner_05d4f22.jpg 110 | http://www3.res.meizu.com/static/cn/smart/router_s/summary/images/simple_7dc2106.jpg 111 | http://www3.res.meizu.com/static/cn/smart/router_s/summary/images/band_chart_line_c3fb07c.png 112 | http://www3.res.meizu.com/static/cn/smart/router_s/summary/images/signal_bg_790f4b2.jpg 113 | http://www3.res.meizu.com/static/cn/smart/router_s/summary/images/signal_router_b27cb94.png 114 | http://www3.res.meizu.com/static/cn/smart/router_s/summary/images/cpuram_bg_7b89644.jpg 115 | http://www3.res.meizu.com/static/cn/smart/router_s/summary/images/power_6a5de93.jpg 116 | http://www3.res.meizu.com/static/cn/smart/router_s/summary/images/client_icon_1_3148877.jpg 117 | http://www3.res.meizu.com/static/cn/smart/router_s/summary/images/client_icon_2_b9eadd0.jpg 118 | http://www3.res.meizu.com/static/cn/smart/router_s/summary/images/client_icon_3_cc6580d.jpg 119 | http://www3.res.meizu.com/static/cn/smart/router_s/summary/images/usb_2968ca5.jpg 120 | http://www3.res.meizu.com/static/cn/smart/router_s/summary/images/setup_2_1ed4f82.jpg 121 | http://www3.res.meizu.com/static/cn/accessory/ep51/images/comfortable01_dfda6ad.png 122 | http://www3.res.meizu.com/static/cn/products/mx5/summary/images/banner1_e9c4c77.png 123 | http://www3.res.meizu.com/static/cn/products/mx5/summary/images/banner2_bc31862.png 124 | http://www3.res.meizu.com/static/cn/products/mx5/summary/images/tech1_25e6592.png 125 | http://www3.res.meizu.com/static/cn/products/mx5/summary/images/tech2_236f981.png 126 | http://www3.res.meizu.com/static/cn/products/mx5/summary/images/power_4ee0f6a.png 127 | http://www3.res.meizu.com/static/cn/products/mx5/summary/images/screen1_cf9dd60.png 128 | http://www3.res.meizu.com/static/cn/products/mx5/summary/images/screen2_fbf9efa.png 129 | http://www3.res.meizu.com/static/cn/products/mx5/summary/images/card_459bc92.png 130 | http://www3.res.meizu.com/static/cn/products/mx5/summary/images/lens1_26577bc.png 131 | http://www3.res.meizu.com/static/cn/products/pro5/summary/images/perfect_big_bg_c88db6e.jpg 132 | http://www3.res.meizu.com/static/cn/products/pro5/summary/images/perfect_big_bg_active_dec3517.jpg 133 | http://www3.res.meizu.com/static/cn/products/pro5/summary/images/snowmou2_911bab1.jpg 134 | http://www3.res.meizu.com/static/cn/products/pro5/summary/images/tech_25dc306.jpg 135 | http://www3.res.meizu.com/static/cn/products/pro5/summary/images/triangle_27b5158.png 136 | http://www3.res.meizu.com/static/cn/products/pro5/summary/images/triangle@2x_e6637e6.png 137 | http://www3.res.meizu.com/static/cn/products/pro5/summary/images/mtouch_61200ff.jpg 138 | http://www3.res.meizu.com/static/cn/products/pro5/summary/images/doublephone_b482dd2.png 139 | http://www3.res.meizu.com/static/cn/products/pro5/summary/images/papa1_91337f2.jpg 140 | http://www3.res.meizu.com/static/cn/products/pro5/summary/images/bertty_mcharge2_110d7e1.png 141 | http://www3.res.meizu.com/static/cn/products/pro5/summary/images/bertty_mchange3_fb92c25.png 142 | http://www3.res.meizu.com/static/cn/products/mx5/summary/images/lens2_e0c5c0a.png 143 | http://www3.res.meizu.com/static/cn/products/mx5/summary/images/lens3_2905da8.png 144 | http://www3.res.meizu.com/static/cn/products/mx5/summary/images/lens4_f6bf500.png 145 | http://www3.res.meizu.com/static/cn/products/mx5/summary/images/lens5_9059032.png 146 | http://www3.res.meizu.com/static/cn/products/mx5/summary/images/lens6_34168ad.png 147 | http://www3.res.meizu.com/static/cn/products/mx5/summary/images/lens7_584d047.png 148 | http://www3.res.meizu.com/static/cn/products/mx5/summary/images/mcharge_71f4f0b.png 149 | http://www3.res.meizu.com/static/cn/products/mx5/summary/images/mcharge1_49c8e25.png 150 | http://www3.res.meizu.com/static/cn/products/mx5/summary/images/mtouch_e0cae20.png 151 | http://www3.res.meizu.com/static/cn/products/mx5/summary/images/flyme_bf3079d.png 152 | http://www3.res.meizu.com/static/cn/products/mx5/summary/images/next_63055d9.png 153 | http://www3.res.meizu.com/static/cn/products/pro5/summary/images/mc_usb_2722ce0.png 154 | http://www3.res.meizu.com/static/cn/products/pro5/summary/images/pro_service_1b7c53d.jpg 155 | http://www3.res.meizu.com/static/cn/products/pro5/summary/images/next_952f1c4.png 156 | http://www3.res.meizu.com/static/cn/products/m3s/summary/images/phone_1_c494516.png 157 | http://www3.res.meizu.com/static/cn/products/m3s/summary/images/phone_2_09ae158.png 158 | http://www3.res.meizu.com/static/cn/products/m3s/summary/images/metal_4_1893bd5.png 159 | http://www3.res.meizu.com/static/cn/products/m3s/summary/images/metal_3_1ec56c3.png 160 | http://www3.res.meizu.com/static/cn/products/m3s/summary/images/metal_2_79e286e.png 161 | http://www3.res.meizu.com/static/cn/products/m3s/summary/images/metal_1_55f08c8.png 162 | http://www3.res.meizu.com/static/cn/products/m3s/summary/images/screen_2_bc288db.png 163 | http://www3.res.meizu.com/static/cn/products/m3s/summary/images/screen_1_2445f91.png 164 | http://www3.res.meizu.com/static/cn/products/m3s/summary/images/cpu_8_cb0e847.png 165 | http://www3.res.meizu.com/static/cn/products/m3s/summary/images/cpu_7_3185bc3.png 166 | http://www3.res.meizu.com/static/cn/products/m3s/summary/images/cpu_6_7a791ab.png 167 | http://www3.res.meizu.com/static/cn/products/m3s/summary/images/cpu_5_084a50a.png 168 | http://www3.res.meizu.com/static/cn/products/m3s/summary/images/cpu_4_3c0c31b.png 169 | http://www3.res.meizu.com/static/cn/products/u10/summary/images/slogan_2808d4f.jpg 170 | http://www3.res.meizu.com/static/cn/products/u10/summary/images/screen_f28c2de.png 171 | http://www3.res.meizu.com/static/cn/products/u10/summary/images/glass_9d04968.jpg 172 | http://www3.res.meizu.com/static/cn/products/u10/summary/images/camera_db03aa0.jpg 173 | http://www3.res.meizu.com/static/cn/products/u10/summary/images/battery_8d9b8d3.jpg 174 | http://www3.res.meizu.com/static/cn/products/pro6/summary/images/light_phone_up_4aee9d9.png 175 | http://www3.res.meizu.com/static/cn/products/pro6/summary/images/light_phone_down_3dbd4df.png 176 | http://www3.res.meizu.com/static/cn/products/pro6/summary/images/light_phone_shadow_mid_472ccc5.png 177 | http://www3.res.meizu.com/static/cn/products/pro6/summary/images/light_phone_shadow_f3f362d.png 178 | http://www3.res.meizu.com/static/cn/products/pro6/summary/images/learn_more_icon_bc57565.png 179 | http://www3.res.meizu.com/static/cn/products/m3s/summary/images/cpu_3_bf673f9.png 180 | http://www3.res.meizu.com/static/cn/products/m3s/summary/images/cpu_2_79b7519.png 181 | http://www3.res.meizu.com/static/cn/products/m3s/summary/images/cpu_1_eb44ab8.png 182 | http://www3.res.meizu.com/static/cn/products/m3s/summary/images/network_2_e912fdc.png 183 | http://www3.res.meizu.com/static/cn/products/m3s/summary/images/network_3_8ab2ae8.png 184 | http://www3.res.meizu.com/static/cn/products/m3s/summary/images/network_1_43a13ef.png 185 | http://www3.res.meizu.com/static/cn/products/m3s/summary/images/system_6_91039d8.png 186 | http://www3.res.meizu.com/static/cn/products/m3s/summary/images/system_5_2ed1948.png 187 | http://www3.res.meizu.com/static/cn/products/m3s/summary/images/system_7_cf4c6dd.png 188 | http://www3.res.meizu.com/static/cn/products/pro6/summary/images/mpress_glass_3542734.jpg 189 | http://www3.res.meizu.com/static/cn/products/pro6/summary/images/mpress_signal_88a4634.jpg 190 | http://www3.res.meizu.com/static/cn/products/pro6/summary/images/mpress_metal_59ac726.png 191 | http://www3.res.meizu.com/static/cn/products/pro6/summary/images/mpress_phone_56e337d.png 192 | http://www3.res.meizu.com/static/cn/products/pro6/summary/images/mpress_phone_shadow_ff5d346.jpg 193 | http://www3.res.meizu.com/static/cn/products/pro6/summary/images/process_phone_01dd2e5.png 194 | http://www3.res.meizu.com/static/cn/products/pro6/summary/images/process_phone_pic_top_354c363.png 195 | http://www3.res.meizu.com/static/cn/products/pro6/summary/images/process_phone_pic_bottom_502cdb3.jpg 196 | http://www3.res.meizu.com/static/cn/products/pro6/summary/images/amoled_logo_468cef3.jpg 197 | http://www3.res.meizu.com/static/cn/products/pro6/summary/images/amoled_phone_0bb6235.png 198 | http://www3.res.meizu.com/static/cn/products/pro6/summary/images/amoled_shadow_e0e90c4.jpg 199 | http://www3.res.meizu.com/static/cn/products/pro6/camera/images/camera_phone.hd_efab5d1.png 200 | http://www3.res.meizu.com/static/cn/products/pro6/camera/images/camera_l_1_9a7b4e9.png 201 | http://www3.res.meizu.com/static/cn/products/pro6/camera/images/camera_l_2_f16948c.png 202 | http://www3.res.meizu.com/static/cn/products/pro6/camera/images/camera_l_3_b95bb3e.png 203 | http://www3.res.meizu.com/static/cn/products/pro6/camera/images/camera_s_1_411aed6.png 204 | http://www3.res.meizu.com/static/cn/products/pro6/camera/images/camera_s_2_9df5957.png 205 | http://www3.res.meizu.com/static/cn/products/pro6/camera/images/camera_s_3_ab9f758.png 206 | http://www3.res.meizu.com/static/cn/products/pro6/camera/images/camera_s_4_d2fb2e2.png 207 | http://www3.res.meizu.com/static/cn/products/pro6/camera/images/camera_s_5_d3a6d85.png 208 | http://www3.res.meizu.com/static/cn/products/pro6/summary/images/mtouch_95dc782.jpg 209 | http://www3.res.meizu.com/static/cn/products/pro6/summary/images/craft_phone_7cd20fb.jpg 210 | http://www3.res.meizu.com/static/cn/products/pro6/summary/images/hifi_headset_shadow_2f60cad.jpg 211 | http://www3.res.meizu.com/static/cn/products/pro6/summary/images/hifi_phone_0229f9e.png 212 | http://www3.res.meizu.com/static/cn/products/pro6/summary/images/hifi_phone_shadow_c463d9e.png 213 | http://www.meizu.com/static/products/pro6/summary/images/msound_icon_1.png 214 | http://www.meizu.com/static/products/pro6/summary/images/msound_icon_2.png 215 | http://www3.res.meizu.com/static/cn/products/pro6/summary/images/msound_phone_f6dbe96.png 216 | http://www3.res.meizu.com/static/cn/products/pro6/summary/images/network_yidong_4f5d1bc.png 217 | http://www3.res.meizu.com/static/cn/products/pro6/summary/images/network_dianxin_6ed79e6.png 218 | http://www3.res.meizu.com/static/cn/products/pro6/summary/images/network_liantong_ac1685d.png 219 | http://www3.res.meizu.com/static/cn/products/pro6/summary/images/typec_phone_4d90fe5.jpg 220 | http://www3.res.meizu.com/static/cn/products/pro6/summary/images/mcharge_phone_206553e.jpg 221 | http://www3.res.meizu.com/static/cn/products/m5/summary/images/screen_c0032e3.jpg 222 | http://www3.res.meizu.com/static/cn/products/m5/summary/images/cpu_e5b9202.jpg 223 | http://www3.res.meizu.com/static/cn/products/mx6/summary/images/screen_a1198b8.jpg 224 | http://www3.res.meizu.com/static/cn/products/mx6/summary/images/cpu_phone_8fc79d3.png 225 | http://www3.res.meizu.com/static/cn/products/mx6/summary/images/cpu_phone_shadow_cb46ccc.jpg 226 | http://www3.res.meizu.com/static/cn/products/mx6/summary/images/camera_phone_2aaae44.jpg 227 | http://www3.res.meizu.com/static/cn/products/mx6/summary/images/camera_phone_shadow_1ed670d.jpg 228 | http://www3.res.meizu.com/static/cn/products/mx6/summary/images/zhifubao_4ca770a.png 229 | http://www3.res.meizu.com/static/cn/products/mx6/summary/images/weixin_acde025.png 230 | http://www3.res.meizu.com/static/cn/global/images/subscribe_10b6977.jpg 231 | http://www3.res.meizu.com/static/cn/global/images/subscribed_1463c80.jpg 232 | http://www3.res.meizu.com/static/cn/products/mx6/summary/images/pay_phone_fingerprint_8396bbe.jpg 233 | http://www3.res.meizu.com/static/cn/products/mx6/summary/images/pay_phone_ac41c50.jpg 234 | http://www3.res.meizu.com/static/cn/accessory/hd50/gallery/images/016_c693ce0.jpg 235 | http://www3.res.meizu.com/static/cn/accessory/hd50/gallery/images/enjoy_girl_44073b4.jpg 236 | http://www3.res.meizu.com/static/cn/accessory/hd50/gallery/images/doubleline_6784956.png 237 | http://www3.res.meizu.com/static/cn/accessory/hd50/gallery/images/black_headphone_f368e34.jpg 238 | http://www3.res.meizu.com/static/cn/accessory/hd50/gallery/images/black_single_5b9f16c.jpg 239 | http://www3.res.meizu.com/static/cn/accessory/hd50/gallery/images/slide_girl_6247789.jpg 240 | http://www3.res.meizu.com/static/cn/accessory/hd50/gallery/images/cool_boy_b170fc2.jpg 241 | http://www3.res.meizu.com/static/cn/accessory/musiccard/images/process_36d8405.jpg 242 | http://www3.res.meizu.com/static/cn/products/pro5/hifi/images/line1_2b0fbe5.png 243 | http://www3.res.meizu.com/static/cn/products/pro5/hifi/images/line2_7a16067.png 244 | http://www3.res.meizu.com/static/cn/products/pro5/hifi/images/solo_1d8c060.jpg 245 | http://www3.res.meizu.com/static/cn/products/pro5/hifi/images/balance_0709d45.jpg 246 | http://www3.res.meizu.com/static/cn/products/pro5/hifi/images/next_1e53401.jpg 247 | http://www3.res.meizu.com/static/cn/products/pro5/accessory/images/banner_dcc4f00.jpg 248 | http://www3.res.meizu.com/static/cn/products/pro5/accessory/images/ep31_right_3a9e60f.jpg 249 | http://www3.res.meizu.com/static/cn/products/pro5/accessory/images/next_78bdd8c.jpg 250 | http://www3.res.meizu.com/static/cn/products/pro5/performance/images/pro5_game_car_8448a85.jpg 251 | http://www3.res.meizu.com/static/cn/products/pro5/performance/images/pro5_game_nba_d3f9d5f.jpg 252 | http://www3.res.meizu.com/static/cn/products/pro5/performance/images/pro5_game_mc4_96888b6.jpg 253 | http://www3.res.meizu.com/static/cn/products/pro5/performance/images/pro5_game_spider_af8053f.jpg 254 | http://www3.res.meizu.com/static/cn/products/pro5/performance/images/ufs_cc794a7.jpg 255 | http://www3.res.meizu.com/static/cn/products/pro5/performance/images/next_5170e41.jpg 256 | http://www3.res.meizu.com/static/cn/products/u10/spec/images/interface_55081eb.png 257 | http://www3.res.meizu.com/static/cn/products/mx5/mtouch/images/next_63055d9.png 258 | http://www3.res.meizu.com/static/cn/products/pro6/3dpress/images/press_01152d5.jpg 259 | http://www3.res.meizu.com/static/cn/products/mx5/camera/images/banner_923a749.png 260 | http://www3.res.meizu.com/static/cn/products/mx5/spec/images/interface_56ff020.jpg 261 | http://www3.res.meizu.com/static/cn/products/pro6/3dpress/images/email_44e72fe.jpg 262 | http://www3.res.meizu.com/static/cn/products/pro6/3dpress/images/email2_9720c35.jpg 263 | http://www3.res.meizu.com/static/cn/products/pro6/3dpress/images/web_6c2e572.jpg 264 | http://www3.res.meizu.com/static/cn/products/pro6/3dpress/images/web2_2f6ee97.jpg 265 | http://www3.res.meizu.com/static/cn/products/pro6/3dpress/images/msg_eefbb40.jpg 266 | http://www3.res.meizu.com/static/cn/products/pro6/3dpress/images/msg2_7f32d1d.jpg 267 | http://www3.res.meizu.com/static/cn/products/mx5/camera/images/lens1_0c7136f.png 268 | http://www3.res.meizu.com/static/cn/products/mx5/camera/images/lens2_e0c5c0a.png 269 | http://www3.res.meizu.com/static/cn/products/mx5/camera/images/lens3_2905da8.png 270 | http://www3.res.meizu.com/static/cn/products/mx5/camera/images/lens4_f6bf500.png 271 | http://www3.res.meizu.com/static/cn/products/mx5/camera/images/lens5_9059032.png 272 | http://www3.res.meizu.com/static/cn/products/mx5/camera/images/lens6_34168ad.png 273 | http://www3.res.meizu.com/static/cn/products/mx5/camera/images/lens7_584d047.png 274 | http://www3.res.meizu.com/static/cn/products/mx5/camera/images/flashlight_84688c3.jpg 275 | http://www3.res.meizu.com/static/cn/products/mx5/camera/images/front_e25c7a0.png 276 | http://www3.res.meizu.com/static/cn/products/mx5/camera/images/next_63055d9.png 277 | http://www3.res.meizu.com/static/cn/products/mx5/design/images/banner1_24607b5.png 278 | http://www3.res.meizu.com/static/cn/products/pro6/flyme/images/banner_phone_shadow_44ab28f.jpg 279 | http://www3.res.meizu.com/static/cn/products/mx5/craft/images/1_020c838.jpg 280 | http://www3.res.meizu.com/static/cn/products/mx5/spec/images/sim_card_0a05af0.jpg 281 | http://www3.res.meizu.com/static/cn/products/mx5/spec/images/sim_step1_528b6a6.jpg 282 | http://www3.res.meizu.com/static/cn/products/mx5/spec/images/sim_step2_44c8b00.jpg 283 | http://www3.res.meizu.com/static/cn/products/mx5/spec/images/sim_step3_9e8fdde.jpg 284 | http://www3.res.meizu.com/static/cn/products/mx5/spec/images/sim_step1_single_80f113c.jpg 285 | http://www3.res.meizu.com/static/cn/products/mx5/spec/images/sim_step2_single_d3612db.jpg 286 | http://www3.res.meizu.com/static/cn/products/mx5/spec/images/solfware_8f41008.jpg 287 | http://www3.res.meizu.com/static/cn/products/mx5/design/images/banner2_d98d8ae.png 288 | http://www3.res.meizu.com/static/cn/products/mx5/design/images/T_8598dab.jpg 289 | http://www3.res.meizu.com/static/cn/products/mx5/design/images/light_86924d8.png 290 | http://www3.res.meizu.com/static/cn/products/mx5/design/images/thin_8e4a8f9.png 291 | http://www3.res.meizu.com/static/cn/products/mx5/design/images/color_8218bdf.jpg 292 | http://www3.res.meizu.com/static/cn/products/mx5/design/images/dd_1781f42.jpg 293 | http://www3.res.meizu.com/static/cn/products/mx5/design/images/ff_9467726.jpg 294 | http://www3.res.meizu.com/static/cn/products/mx5/design/images/screw_90cda59.jpg 295 | http://www3.res.meizu.com/static/cn/products/mx5/design/images/next_63055d9.png 296 | http://www3.res.meizu.com/static/cn/products/pro6/flyme/images/banner_phone_ef733b1.png 297 | http://www3.res.meizu.com/static/cn/products/pro6/flyme/images/tender_phone_left_28a4d79.jpg 298 | http://www3.res.meizu.com/static/cn/products/pro6/flyme/images/tender_phone_right_870535e.png 299 | http://www3.res.meizu.com/static/cn/products/pro6/flyme/images/weather_logo_70ee86a.png 300 | http://www3.res.meizu.com/static/cn/products/pro6/flyme/images/list_1_1_c7449ef.jpg 301 | http://www3.res.meizu.com/static/cn/products/pro6/flyme/images/list_1_2_ddc99bb.jpg 302 | http://www3.res.meizu.com/static/cn/products/pro6/flyme/images/list_1_3_aa283f3.jpg 303 | http://www3.res.meizu.com/static/cn/products/pro6/flyme/images/effi_phone_56f9e56.png 304 | http://www3.res.meizu.com/static/cn/products/pro6/flyme/images/effi_phone_shadow_54145a3.jpg 305 | http://www3.res.meizu.com/static/cn/products/pro6/flyme/images/task_phone_1_4226e0d.jpg 306 | http://www3.res.meizu.com/static/cn/products/pro6/flyme/images/task_phone_2_7963731.jpg 307 | http://www3.res.meizu.com/static/cn/products/pro6/flyme/images/task_phone_3_1764955.jpg 308 | http://www3.res.meizu.com/static/cn/products/pro6/flyme/images/task_phone_4_56ab795.jpg 309 | http://www3.res.meizu.com/static/cn/products/pro6/flyme/images/interation_phone_812bd39.jpg 310 | http://www3.res.meizu.com/static/cn/products/pro6/flyme/images/mpress_phone_1_b94061e.jpg 311 | http://www3.res.meizu.com/static/cn/products/pro6/flyme/images/mpress_phone_2_2664467.jpg 312 | http://www3.res.meizu.com/static/cn/products/pro6/flyme/images/mpress_phone_3_2decc36.jpg 313 | http://www3.res.meizu.com/static/cn/products/pro6/flyme/images/mpress_phone_4_5dccab6.jpg 314 | http://www3.res.meizu.com/static/cn/products/pro6/flyme/images/safe_phone_35bba8b.png 315 | http://www3.res.meizu.com/static/cn/products/pro6/flyme/images/safe_phone_shadow_992bd38.png 316 | http://www3.res.meizu.com/static/cn/products/pro6/flyme/images/find_phone_1_314eec6.jpg 317 | http://www3.res.meizu.com/static/cn/products/pro6/flyme/images/find_phone_2_9b6b165.jpg 318 | http://www3.res.meizu.com/static/cn/products/pro6/flyme/images/find_phone_3_64314d2.jpg 319 | http://www3.res.meizu.com/static/cn/products/pro6/flyme/images/unique_phone_221ee74.png 320 | http://www3.res.meizu.com/static/cn/products/pro6/flyme/images/theme_phone_1_46f42d8.jpg 321 | http://www3.res.meizu.com/static/cn/products/pro6/flyme/images/theme_phone_2_f9f3c02.jpg 322 | http://www3.res.meizu.com/static/cn/products/pro6/flyme/images/theme_phone_3_a497a5b.jpg 323 | http://www3.res.meizu.com/static/cn/products/pro6/flyme/images/cloud_phone_right_d27238c.jpg 324 | http://www3.res.meizu.com/static/cn/products/pro6/flyme/images/cloud_phone_left_5a59f4b.png 325 | http://www3.res.meizu.com/static/cn/products/mx5/craft/images/3_5c43313.jpg 326 | http://www3.res.meizu.com/static/cn/products/mx5/craft/images/4_1f5f1f3.jpg 327 | http://www3.res.meizu.com/static/cn/products/mx5/craft/images/5_46fe689.jpg 328 | http://www3.res.meizu.com/static/cn/products/mx5/craft/images/6_54111b7.jpg 329 | http://www3.res.meizu.com/static/cn/products/mx5/craft/images/next_63055d9.png 330 | http://www3.res.meizu.com/static/cn/products/mx5/display/images/unlikeliness_8145af1.jpg 331 | http://www3.res.meizu.com/static/cn/products/pro6s/summary/images/mtouch_6653db0.jpg 332 | http://www3.res.meizu.com/static/cn/products/pro6s/summary/images/hifi_headset_shadow_2f60cad.jpg 333 | http://www3.res.meizu.com/static/cn/products/pro6s/summary/images/hifi_phone_0229f9e.png 334 | http://www3.res.meizu.com/static/cn/products/pro6s/summary/images/hifi_phone_shadow_c463d9e.png 335 | http://www3.res.meizu.com/static/cn/products/pro6s/summary/images/msound_phone_f6dbe96.png 336 | http://www3.res.meizu.com/static/cn/products/pro6s/summary/images/network_yidong_4f5d1bc.png 337 | http://www3.res.meizu.com/static/cn/products/pro6s/summary/images/network_dianxin_6ed79e6.png 338 | http://www3.res.meizu.com/static/cn/products/pro6s/summary/images/network_liantong_ac1685d.png 339 | http://www3.res.meizu.com/static/cn/products/pro6s/summary/images/typec_phone_4d90fe5.jpg 340 | http://www3.res.meizu.com/static/cn/products/pro6s/summary/images/mcharge_phone_a452979.jpg 341 | http://www3.res.meizu.com/static/cn/accessory/ep2x/images/ep2x_f221952.png 342 | -------------------------------------------------------------------------------- /ImgInWebsite/items.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | 3 | # Define here the models for your scraped items 4 | # 5 | # See documentation in: 6 | # http://doc.scrapy.org/en/latest/topics/items.html 7 | 8 | import scrapy 9 | 10 | 11 | class ImginwebsiteItem(scrapy.Item): 12 | # define the fields for your item here like: 13 | # name = scrapy.Field() 14 | pass 15 | 16 | class ImgItem(scrapy.Item): 17 | img_url=scrapy.Field() -------------------------------------------------------------------------------- /ImgInWebsite/pipelines.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | 3 | # Define your item pipelines here 4 | # 5 | # Don't forget to add your pipeline to the ITEM_PIPELINES setting 6 | # See: http://doc.scrapy.org/en/latest/topics/item-pipeline.html 7 | 8 | 9 | class ImginwebsitePipeline(object): 10 | def process_item(self, item, spider): 11 | return item 12 | 13 | class ImgPipeline(object): 14 | def __init__(self): 15 | self.file=open('imgs.txt','wb') 16 | 17 | def process_item(self, item, spider): 18 | url = dict(item)['img_url'] + "\n" 19 | self.file.write(url) 20 | return item -------------------------------------------------------------------------------- /ImgInWebsite/settings.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | 3 | # Scrapy settings for ImgInWebsite project 4 | # 5 | # For simplicity, this file contains only settings considered important or 6 | # commonly used. You can find more settings consulting the documentation: 7 | # 8 | # http://doc.scrapy.org/en/latest/topics/settings.html 9 | # http://scrapy.readthedocs.org/en/latest/topics/downloader-middleware.html 10 | # http://scrapy.readthedocs.org/en/latest/topics/spider-middleware.html 11 | 12 | BOT_NAME = 'ImgInWebsite' 13 | 14 | SPIDER_MODULES = ['ImgInWebsite.spiders'] 15 | NEWSPIDER_MODULE = 'ImgInWebsite.spiders' 16 | 17 | # 设置LOG 18 | LOG_LEVEL='ERROR' 19 | 20 | # Crawl responsibly by identifying yourself (and your website) on the user-agent 21 | #USER_AGENT = 'ImgInWebsite (+http://www.yourdomain.com)' 22 | 23 | # Obey robots.txt rules 24 | ROBOTSTXT_OBEY = True 25 | 26 | # Configure maximum concurrent requests performed by Scrapy (default: 16) 27 | #CONCURRENT_REQUESTS = 32 28 | 29 | # Configure a delay for requests for the same website (default: 0) 30 | # See http://scrapy.readthedocs.org/en/latest/topics/settings.html#download-delay 31 | # See also autothrottle settings and docs 32 | #DOWNLOAD_DELAY = 3 33 | # The download delay setting will honor only one of: 34 | #CONCURRENT_REQUESTS_PER_DOMAIN = 16 35 | #CONCURRENT_REQUESTS_PER_IP = 16 36 | 37 | # Disable cookies (enabled by default) 38 | #COOKIES_ENABLED = False 39 | 40 | # Disable Telnet Console (enabled by default) 41 | #TELNETCONSOLE_ENABLED = False 42 | 43 | # Override the default request headers: 44 | #DEFAULT_REQUEST_HEADERS = { 45 | # 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8', 46 | # 'Accept-Language': 'en', 47 | #} 48 | 49 | # Enable or disable spider middlewares 50 | # See http://scrapy.readthedocs.org/en/latest/topics/spider-middleware.html 51 | #SPIDER_MIDDLEWARES = { 52 | # 'ImgInWebsite.middlewares.MyCustomSpiderMiddleware': 543, 53 | #} 54 | 55 | # Enable or disable downloader middlewares 56 | # See http://scrapy.readthedocs.org/en/latest/topics/downloader-middleware.html 57 | #DOWNLOADER_MIDDLEWARES = { 58 | # 'ImgInWebsite.middlewares.MyCustomDownloaderMiddleware': 543, 59 | #} 60 | 61 | # Enable or disable extensions 62 | # See http://scrapy.readthedocs.org/en/latest/topics/extensions.html 63 | #EXTENSIONS = { 64 | # 'scrapy.extensions.telnet.TelnetConsole': None, 65 | #} 66 | 67 | # Configure item pipelines 68 | # See http://scrapy.readthedocs.org/en/latest/topics/item-pipeline.html 69 | ITEM_PIPELINES = { 70 | 'ImgInWebsite.pipelines.ImgPipeline': 300, 71 | } 72 | 73 | # Enable and configure the AutoThrottle extension (disabled by default) 74 | # See http://doc.scrapy.org/en/latest/topics/autothrottle.html 75 | #AUTOTHROTTLE_ENABLED = True 76 | # The initial download delay 77 | #AUTOTHROTTLE_START_DELAY = 5 78 | # The maximum download delay to be set in case of high latencies 79 | #AUTOTHROTTLE_MAX_DELAY = 60 80 | # The average number of requests Scrapy should be sending in parallel to 81 | # each remote server 82 | #AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0 83 | # Enable showing throttling stats for every response received: 84 | #AUTOTHROTTLE_DEBUG = False 85 | 86 | # Enable and configure HTTP caching (disabled by default) 87 | # See http://scrapy.readthedocs.org/en/latest/topics/downloader-middleware.html#httpcache-middleware-settings 88 | #HTTPCACHE_ENABLED = True 89 | #HTTPCACHE_EXPIRATION_SECS = 0 90 | #HTTPCACHE_DIR = 'httpcache' 91 | #HTTPCACHE_IGNORE_HTTP_CODES = [] 92 | #HTTPCACHE_STORAGE = 'scrapy.extensions.httpcache.FilesystemCacheStorage' 93 | -------------------------------------------------------------------------------- /ImgInWebsite/settings.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wly2014/ImageSpider/97cdaff203b407d6a9a9e103e8c74fe525f1dda6/ImgInWebsite/settings.pyc -------------------------------------------------------------------------------- /ImgInWebsite/spiders/ImgSpider.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | import scrapy 3 | import re 4 | from scrapy.http import Request 5 | from ImgInWebsite.items import ImgItem 6 | 7 | class ImgSpider(scrapy.Spider): 8 | name = "ImgSpider" 9 | allowed_domains = ["www.meizu.com"] 10 | start_urls = ['http://www.meizu.com/'] 11 | 12 | URL=[] 13 | IMG=[] 14 | 15 | def parse(self, response): 16 | allUrl=response.url 17 | print allUrl 18 | # 爬取图片连接 19 | s=allUrl.split('/') 20 | root='/'.join(s[0:3]) 21 | # 下面的正则会匹配多个括号 22 | pattern=re.compile(r'((((https|http):\/\/)|\/)[0-9a-zA-Z\/\.@-_%]*?\.(jpg|png))') 23 | items=pattern.findall(response.text) 24 | for item in items: 25 | img=item[0] 26 | if img in self.IMG: 27 | continue 28 | else: 29 | self.IMG.append(img) 30 | 31 | if img.startswith('/'): 32 | img=root+img 33 | elif img.startswith('http') or img.startswith('www'): 34 | img=img 35 | else: 36 | # 当前url加上img 37 | # 严格还要判断allUrl是不是以'/'结尾 38 | if allUrl.endswith('/'): 39 | img=allUrl+img 40 | else: 41 | img='/'.join(allUrl.split('/')[0:-1])+'/'+img 42 | imgItem=ImgItem() 43 | imgItem['img_url']=img 44 | yield imgItem 45 | 46 | # 爬取继续爬取的网址 47 | urls=response.xpath('//a/@href').extract() 48 | 49 | for url in urls: 50 | if url in self.URL: 51 | continue 52 | else: 53 | self.URL.append(url) 54 | # 组合成完整的url 55 | if url.startswith('/'): 56 | url='/'.join(allUrl.split('/')[0:3])+url 57 | elif url.startswith('http') or url.startswith('www'): 58 | url=url 59 | else: 60 | # 相对路径 61 | # 还要判断allUrl是不是以'/'结尾 62 | if allUrl.endswith('/'): 63 | url=allUrl+url 64 | else: 65 | url='/'.join(allUrl.split('/')[0:-1])+'/'+url 66 | 67 | yield Request(url, callback=self.parse) -------------------------------------------------------------------------------- /ImgInWebsite/spiders/__init__.py: -------------------------------------------------------------------------------- 1 | # This package will contain the spiders of your Scrapy project 2 | # 3 | # Please refer to the documentation for information on how to create and manage 4 | # your spiders. 5 | -------------------------------------------------------------------------------- /ImgInWebsite/spiders/__init__.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wly2014/ImageSpider/97cdaff203b407d6a9a9e103e8c74fe525f1dda6/ImgInWebsite/spiders/__init__.pyc -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # ImageSpider 2 | 3 | ## 如何使用 4 | 5 | 1. 将`ImageSpider\ImgInWebsite\spiders\ImgSpider.py`文件中的`allowed_domains = ["www.meizu.com"]`和`start_urls = ['http://www.meizu.com/']`中的地址换成自己想要爬取图片网站的地址。 6 | 2. 运行命令`scrapy crawl ImgSpider` 7 | 3. 本地生成一个`imgs.txt`文件，其中有全部图片的下载地址 8 | 9 | ## 更多 10 | 11 | [使用scrapy爬取网站上的所有图片](http://blog.csdn.net/u014271114/article/details/53080447) -------------------------------------------------------------------------------- /scrapy.cfg: -------------------------------------------------------------------------------- 1 | # Automatically created by: scrapy startproject 2 | # 3 | # For more information about the [deploy] section see: 4 | # https://scrapyd.readthedocs.org/en/latest/deploy.html 5 | 6 | [settings] 7 | default = ImgInWebsite.settings 8 | 9 | [deploy] 10 | #url = http://localhost:6800/ 11 | project = ImgInWebsite 12 | --------------------------------------------------------------------------------