├── README.md ├── Read-Please ├── _SUCCESS ├── mapper.py ├── output2 ├── _SUCCESS └── part-00000 ├── part-00000 ├── reducer.py └── running switch.txt /README.md: -------------------------------------------------------------------------------- 1 | # hadoop_map_reduce 2 | Here we have some example for working on hadoop, map reduce. 3 | -------------------------------------------------------------------------------- /Read-Please: -------------------------------------------------------------------------------- 1 | agar bakhshi ke dar mapper va bakhshi k dar reducer comment shode, az comment dar biarim va jay gozin konim khoroji count shode va behtari darim!!!! -------------------------------------------------------------------------------- /_SUCCESS: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/AISoltani/hadoop_map_reduce/f2a4f1466d9fc6dd4b0fd184716ff323077d3230/_SUCCESS -------------------------------------------------------------------------------- /mapper.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python 2 | # -*-coding:utf-8 -* 3 | from __future__ import print_function 4 | import sys 5 | 6 | for line in sys.stdin: 7 | print(line) 8 | 9 | 10 | # for line in sys.stdin: 11 | # # print(line) 12 | 13 | 14 | # try: 15 | # vid,uploader,age,category,other = line.split("\t",4) 16 | # except: 17 | # continue 18 | # print(category,uploader,other, sep="\t") -------------------------------------------------------------------------------- /output2/_SUCCESS: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/AISoltani/hadoop_map_reduce/f2a4f1466d9fc6dd4b0fd184716ff323077d3230/output2/_SUCCESS -------------------------------------------------------------------------------- /output2/part-00000: -------------------------------------------------------------------------------- 1 | News & Politics 1 2 | Comedy 1 3 | Sports 1 4 | Entertainment 1 5 | Film & Animation 1 6 | Sports 2 7 | Comedy 1 8 | Howto & DIY 1 9 | Entertainment 1 10 | Sports 1 11 | Pets & Animals 1 12 | Entertainment 2 13 | Film & Animation 1 14 | People & Blogs 1 15 | Entertainment 1 16 | Sports 1 17 | Film & Animation 2 18 | Gadgets & Games 1 19 | Music 1 20 | Entertainment 1 21 | Comedy 1 22 | Sports 2 23 | Entertainment 2 24 | Sports 1 25 | Pets & Animals 1 26 | Entertainment 1 27 | People & Blogs 1 28 | UNA 1 29 | Sports 1 30 | Comedy 2 31 | People & Blogs 1 32 | Sports 3 33 | Entertainment 2 34 | Music 1 35 | Comedy 1 36 | Music 1 37 | People & Blogs 1 38 | Entertainment 2 39 | Comedy 1 40 | People & Blogs 2 41 | Comedy 1 42 | Sports 1 43 | Travel & Places 1 44 | Sports 1 45 | Entertainment 1 46 | Comedy 1 47 | Entertainment 1 48 | Sports 1 49 | Comedy 1 50 | News & Politics 1 51 | Entertainment 1 52 | Music 1 53 | News & Politics 1 54 | Sports 1 55 | Gadgets & Games 1 56 | Sports 1 57 | Comedy 1 58 | Entertainment 1 59 | People & Blogs 1 60 | Gadgets & Games 1 61 | Music 1 62 | Sports 1 63 | Entertainment 1 64 | Sports 1 65 | People & Blogs 1 66 | Pets & Animals 1 67 | Music 1 68 | Sports 1 69 | People & Blogs 1 70 | Music 1 71 | News & Politics 1 72 | Entertainment 1 73 | Film & Animation 1 74 | Entertainment 1 75 | Film & Animation 1 76 | Music 1 77 | Sports 2 78 | Film & Animation 1 79 | Sports 1 80 | Entertainment 1 81 | Gadgets & Games 1 82 | Music 1 83 | People & Blogs 1 84 | Comedy 1 85 | Music 1 86 | Comedy 1 87 | Entertainment 1 88 | Comedy 1 89 | People & Blogs 2 90 | Howto & DIY 1 91 | People & Blogs 3 92 | Comedy 1 93 | Entertainment 1 94 | People & Blogs 1 95 | Music 1 96 | Sports 2 97 | Howto & DIY 1 98 | Travel & Places 1 99 | Music 1 100 | News & Politics 1 101 | People & Blogs 1 102 | Howto & DIY 1 103 | Film & Animation 1 104 | News & Politics 2 105 | Film & Animation 1 106 | People & Blogs 1 107 | Music 1 108 | News & Politics 1 109 | People & Blogs 1 110 | Sports 1 111 | Travel & Places 1 112 | Pets & Animals 1 113 | Entertainment 1 114 | Sports 3 115 | People & Blogs 1 116 | Sports 3 117 | Entertainment 1 118 | Music 1 119 | Film & Animation 1 120 | People & Blogs 1 121 | Comedy 1 122 | Music 1 123 | People & Blogs 1 124 | Music 1 125 | People & Blogs 1 126 | Entertainment 1 127 | Film & Animation 2 128 | Music 1 129 | Gadgets & Games 1 130 | Sports 2 131 | Music 1 132 | Film & Animation 1 133 | News & Politics 1 134 | Sports 1 135 | Film & Animation 1 136 | Comedy 1 137 | Sports 1 138 | Entertainment 1 139 | Music 1 140 | People & Blogs 1 141 | Sports 1 142 | Entertainment 1 143 | Comedy 1 144 | Music 2 145 | Sports 2 146 | Music 2 147 | Comedy 1 148 | Sports 2 149 | Music 1 150 | Gadgets & Games 1 151 | Pets & Animals 1 152 | Comedy 2 153 | Film & Animation 1 154 | Autos & Vehicles 1 155 | News & Politics 1 156 | Comedy 1 157 | Gadgets & Games 1 158 | Autos & Vehicles 1 159 | Travel & Places 1 160 | People & Blogs 1 161 | Music 2 162 | People & Blogs 2 163 | Film & Animation 1 164 | Sports 1 165 | People & Blogs 1 166 | News & Politics 1 167 | Comedy 1 168 | Film & Animation 1 169 | Entertainment 1 170 | People & Blogs 1 171 | Entertainment 1 172 | People & Blogs 1 173 | Entertainment 1 174 | Music 1 175 | People & Blogs 1 176 | News & Politics 1 177 | Gadgets & Games 1 178 | Sports 1 179 | Comedy 1 180 | Music 1 181 | Entertainment 1 182 | News & Politics 1 183 | Comedy 1 184 | Entertainment 1 185 | Music 1 186 | Sports 1 187 | Film & Animation 1 188 | Music 1 189 | Comedy 1 190 | People & Blogs 3 191 | Film & Animation 1 192 | People & Blogs 1 193 | Film & Animation 1 194 | Music 1 195 | Film & Animation 1 196 | Entertainment 2 197 | Gadgets & Games 1 198 | Entertainment 1 199 | People & Blogs 1 200 | Entertainment 2 201 | People & Blogs 1 202 | Entertainment 1 203 | Film & Animation 1 204 | Entertainment 3 205 | Comedy 1 206 | Music 1 207 | Sports 1 208 | People & Blogs 1 209 | News & Politics 1 210 | Entertainment 1 211 | People & Blogs 1 212 | Comedy 1 213 | Autos & Vehicles 1 214 | People & Blogs 1 215 | Entertainment 1 216 | Howto & DIY 1 217 | Music 1 218 | Film & Animation 1 219 | People & Blogs 1 220 | News & Politics 1 221 | Entertainment 1 222 | Music 1 223 | People & Blogs 1 224 | Sports 1 225 | Entertainment 2 226 | Film & Animation 1 227 | Sports 1 228 | Comedy 1 229 | Entertainment 2 230 | People & Blogs 2 231 | Sports 1 232 | News & Politics 1 233 | Sports 1 234 | Film & Animation 1 235 | Entertainment 1 236 | Sports 1 237 | Comedy 1 238 | Pets & Animals 1 239 | Music 1 240 | Comedy 2 241 | Music 2 242 | People & Blogs 1 243 | Entertainment 1 244 | Music 1 245 | Gadgets & Games 1 246 | News & Politics 1 247 | Film & Animation 1 248 | Entertainment 1 249 | News & Politics 1 250 | Entertainment 1 251 | People & Blogs 3 252 | Entertainment 2 253 | Film & Animation 1 254 | People & Blogs 2 255 | Film & Animation 1 256 | News & Politics 1 257 | People & Blogs 1 258 | Entertainment 1 259 | Music 1 260 | Sports 1 261 | Entertainment 2 262 | Music 2 263 | Film & Animation 1 264 | Comedy 1 265 | Entertainment 1 266 | Travel & Places 1 267 | Entertainment 1 268 | Film & Animation 1 269 | Howto & DIY 1 270 | Sports 1 271 | Gadgets & Games 1 272 | Music 1 273 | Sports 1 274 | Film & Animation 1 275 | People & Blogs 1 276 | Entertainment 1 277 | People & Blogs 1 278 | Entertainment 2 279 | News & Politics 1 280 | Film & Animation 1 281 | Entertainment 2 282 | News & Politics 1 283 | Film & Animation 1 284 | People & Blogs 1 285 | Comedy 1 286 | Sports 1 287 | Entertainment 1 288 | People & Blogs 1 289 | Gadgets & Games 1 290 | Comedy 1 291 | Entertainment 1 292 | News & Politics 1 293 | People & Blogs 1 294 | Sports 1 295 | Entertainment 1 296 | Music 1 297 | People & Blogs 1 298 | Comedy 2 299 | Entertainment 1 300 | Gadgets & Games 2 301 | Comedy 1 302 | News & Politics 1 303 | -------------------------------------------------------------------------------- /part-00000: -------------------------------------------------------------------------------- 1 | UNA 1 2 | Autos & Vehicles 3 3 | Comedy 33 4 | Entertainment 45 5 | Film & Animation 22 6 | Gadgets & Games 13 7 | Howto & DIY 3 8 | Music 39 9 | News & Politics 21 10 | People & Blogs 42 11 | Pets & Animals 6 12 | Sports 46 13 | Travel & Places 4 14 | -------------------------------------------------------------------------------- /reducer.py: -------------------------------------------------------------------------------- 1 | from __future__ import print_function 2 | import sys 3 | 4 | last_category = None 5 | last_uploader=None 6 | for line in sys.stdin: 7 | try: 8 | #category, uploader, others = line.split("\t") 9 | video_id, uploader, age, category, others = line.split("\t", 4) 10 | if category == last_category: 11 | if uploader!=last_uploader: 12 | total_uploader += 1 13 | last_uploader = uploader 14 | 15 | else: 16 | if last_category : 17 | print(last_category,total_uploader, sep="\t") 18 | last_category = category 19 | last_uploader = uploader 20 | total_uploader = 1 21 | except: 22 | continue 23 | if last_category: 24 | print(last_category,total_uploader, sep="\t") 25 | 26 | -------------------------------------------------------------------------------- /running switch.txt: -------------------------------------------------------------------------------- 1 | yarn $HADOOP_EMPTY_CONFIG jar $HADOOP_STREAMING \ 2 | -D stream.num.map.output.key.fields=2 \ 3 | -D mapreduce.partition.keycomparator.options="-k2,2 -k1,1" \ 4 | -files mapper.py,reducer.py \ 5 | -mapper 'python3 mapper.py' \ 6 | -reducer 'python3 reducer.py' \ 7 | -input /user/linuxlite/youtube/0.txt \ 8 | -output /user/linuxlite/output --------------------------------------------------------------------------------