├── README.adoc ├── aggregatorservice ├── .classpath ├── .gitignore ├── .project ├── .settings │ ├── org.eclipse.core.resources.prefs │ ├── org.eclipse.jdt.core.prefs │ └── org.eclipse.m2e.core.prefs ├── pom.xml └── src │ └── main │ ├── java │ └── io │ │ └── pivotal │ │ └── data │ │ └── aggregatorservice │ │ ├── App.java │ │ ├── controller │ │ └── AggregatorServiceController.java │ │ ├── model │ │ ├── DriverIncorrectData.java │ │ ├── DriverRevenueData.java │ │ └── RouteData.java │ │ └── service │ │ └── AggregatorService.java │ └── resources │ └── application.yml ├── eventservice ├── .classpath ├── .project ├── .settings │ ├── org.eclipse.core.resources.prefs │ ├── org.eclipse.jdt.core.prefs │ └── org.eclipse.m2e.core.prefs ├── pom.xml └── src │ └── main │ ├── java │ └── io │ │ └── pivotal │ │ └── data │ │ └── eventservice │ │ ├── App.java │ │ ├── controllers │ │ └── EventMetricsController.java │ │ └── services │ │ └── EventsService.java │ └── resources │ ├── application.yml │ └── redis-bean.xml ├── images ├── app_microservices.png ├── circuit_breaker.pdf ├── data_microservices.png ├── problemstmt.jpg ├── service_registry.pdf ├── solution_architecture_1.png ├── stream_definition.png ├── streaming_img1.png ├── streaming_img2.png └── streaming_img3.png ├── manifest-aggservice.yml ├── manifest-eventservice.yml ├── manifest-scdf.yml ├── manifest-taxiservice.yml ├── pom.xml ├── rxjava-taxi-processor ├── pom.xml ├── src │ ├── main │ │ ├── java │ │ │ └── demo │ │ │ │ ├── RxJavaApplication.java │ │ │ │ ├── RxJavaTransformer.java │ │ │ │ └── model │ │ │ │ ├── EventData.java │ │ │ │ ├── EventDataCollection.java │ │ │ │ └── FreeTaxiData.java │ │ └── resources │ │ │ └── application.yml │ └── test │ │ └── java │ │ └── demo │ │ └── ModuleApplicationTests.java └── staticdir │ ├── Staticfile │ └── manifest.yml ├── taxiservice ├── .classpath ├── .gitignore ├── .project ├── .settings │ ├── org.eclipse.core.resources.prefs │ ├── org.eclipse.jdt.core.prefs │ └── org.eclipse.m2e.core.prefs ├── pom.xml └── src │ └── main │ ├── java │ └── io │ │ └── pivotal │ │ └── data │ │ └── taxiservice │ │ ├── App.java │ │ ├── controller │ │ └── TaxiServiceController.java │ │ ├── models │ │ └── RouteData.java │ │ └── services │ │ └── TaxiService.java │ └── resources │ ├── application.yml │ └── redis-bean.xml └── webapp_php ├── .bp-config └── options.json ├── Pivotal-Logo.jpg ├── app_microservices.png ├── data_microservices.png ├── freetaxiesandroutes.html ├── getTaxiData.php ├── index.html ├── manifest.yml ├── problemstmt.jpg ├── sampledata.html ├── solarchitect.png └── taxidata.html /README.adoc: -------------------------------------------------------------------------------- 1 | = Large Scale Data Processing using Data and Application Microservices 2 | 3 | === Introduction 4 | 5 | This demo aims to solve problem that appeared in DEBS challenge 2015 6 | 7 | http://www.debs2015.org/call-grand-challenge.html 8 | 9 | We use Pivotal Stack to solve the problem. Particularly, following products are used. 10 | 11 | 1. Spring Cloud Dataflow 12 | 2. Spring Cloud Streams (RxJava) 13 | 3. Pivotal Cloud Foundry 14 | 4. Redis 15 | 5. RabbitMQ 16 | 8. Spring Cloud Services (for Microservices driven architecture) 17 | 7. Spring Boot, Google Charts and PHP based application 18 | 19 | Apps are built using Microservices pattern and utilizes Spring Cloud Suite (config server, service registry and circuit breaker) for a resilient, scale out architecture. 20 | 21 | === Problem Statement 22 | 23 | Stream of taxi data comes in real time for new york region. Region is to be divided into areas, each measuring 300mx300m. Following questions needs to be solved in real time 24 | 25 | 26 | image::images/problemstmt.jpg[] 27 | 28 | 29 | Question 1 - For every 10 seconds, find out the top 10 routes where taxies are plying the most from one area to another area. 30 | 31 | Question 2 - For every 10 seconds, find out the top 3 routes (areas) 32 | 33 | Question 3 - For every 10 seconds, find out free taxies (show only 50 in the map) available in the region 34 | 35 | Question 4 - For every 10 seconds, find out how much data is incorrectly reported by taxi drivers 36 | 37 | Question 5 - For every 10 seconds, find out how much time does your product takes to compute the above 4 things 38 | 39 | === Solution Architecture 40 | 41 | There are two parts to this problem. 42 | 43 | Part 1 deals with extracting data from the source (in real time), process the dtata in 10 seconds window, and then move the results to a database. Solving this problem largely involves creating data microservices and deploying them via Spring Cloud Dataflow on Pivotal Cloud Foundry. 44 | 45 | Part 2 deals with creating applications that extract data from a database, process the data and return via API calls. These applications need to be resilient, fault-tolerant, scalable and therefore uses Spring Cloud Services (config server, service registry and circuit breaker) which are deployed on Pivotal Cloud Foundry. All applications run on Pivotal Cloud Foundry as well. 46 | 47 | See the below diagram for solving this problem. We would use FTP to retrieve data from FTP server. 48 | 49 | 50 | image::images/solution_architecture_1.png[] 51 | 52 | === Requirements 53 | 1. Need 12GB dedicated memory for PCFDev VM on your laptop 54 | 2. VirtualBox on your laptop 55 | 3. redis-cli on your laptop 56 | 4. FTP server (on your laptop) or some machine accessible by PCFDev 57 | 58 | === Part 1 - Building Data Microservices 59 | 60 | image::images/data_microservices.png[] 61 | 62 | As shown in the image above, we have following data microservices 63 | 64 | 1. FTP "Source" Application - FTP source application is deployed via SCDF on PCF. This out-of-the-box microservice's job is to retrieve files from FTP server and then post each line (as a message) to the next in-line microservice. 65 | 66 | 2. Custom Transformer "Processor" Application - This microservice is a custom processor using RxJava. It has a concept of a moving window. A window is created which captures all the messages for 10 seconds. Once 10 seconds are over, it processes all the data collected and answers above questions. Finally the derived output is sent to the next in-line microservice. 67 | 68 | 3. Splitter "Processor" Application - This out-of-the-box microservice's receives input from custom processors and splits messages based on keys. It then moves the message to the next in-line microservice. 69 | 70 | 4. Redis "Sink" Application - This microservice takes the incoming message, connects to the redis database and push the data with an assigned key. 71 | 72 | Why this architecture? 73 | 74 | 1. Notice, how we have moved the computation to application layer. 75 | 2. Because of this, computation can now be distributed by scaling individual applications. 76 | 3. Pivotal Cloud Foundry manages all these applications, so if any application goes down, it would bring it back. 77 | 4. You don't have to worry about the underlying messaging layer. It could be kafka or rabbit or redis or gemfire. 78 | 5. You could have data affinity using SCDF's partitioning logic. 79 | 6. Since the logic resides in applications, you could test these applications thoroughly and reuse them in your other architectures. 80 | 81 | ==== Step 1. Install PCFDev 82 | Run following commands on your laptop. Make sure you have stopped majority of applications to free up memory. 83 | ---- 84 | $ cf dev start -s redis,rabbitmq,mysql,scs -m 12000 85 | Using existing image. 86 | Starting VM... 87 | Provisioning VM... 88 | Waiting for services to start... 89 | 50 out of 50 running 90 | is now running. 91 | To begin using PCF Dev, please run: 92 | cf login -a https://api.local.pcfdev.io --skip-ssl-validation 93 | Admin user => Email: admin / Password: admin 94 | Regular user => Email: user / Password: pass 95 | ---- 96 | Make sure you replace http url below, if it is different in your environment 97 | ---- 98 | $ cf login -a https://api.local.pcfdev.io --skip-ssl-validation -u admin -p admin -o pcfdev-org 99 | $ cf create-space SCDF 100 | $ cf target -s SCDF 101 | ---- 102 | 103 | We need to update java buildpack to 3.8.1+ version. Download buildpack here 104 | 105 | https://github.com/cloudfoundry/java-buildpack/releases/download/v3.8.1/java-buildpack-offline-v3.8.1.zip 106 | 107 | ---- 108 | $ cf update-buildpack java_buildpack -p --enable -i 1 109 | ---- 110 | 111 | ==== Step 2. Install Spring Cloud Dataflow (SCDF) 112 | Download SCDF for cloud foundry and dataflow shell jar files here 113 | 114 | SCDF for cloud foundry - http://repo.spring.io/release/org/springframework/cloud/spring-cloud-dataflow-server-cloudfoundry/1.0.0.RELEASE/spring-cloud-dataflow-server-cloudfoundry-1.0.0.RELEASE.jar 115 | 116 | SCDF client - http://repo.spring.io/release/org/springframework/cloud/spring-cloud-dataflow-shell/1.0.0.RELEASE/spring-cloud-dataflow-shell-1.0.0.RELEASE.jar 117 | 118 | Run following commands on shell prompt 119 | ---- 120 | $ git clone https://github.com/kgshukla/Realtime-Streaming-DataMicroservices-AppMicroservices.git 121 | $ cd Realtime-Streaming-DataMicroservices-AppMicroservices/ 122 | $ cf create-service p-redis shared-vm redis 123 | $ cf create-service p-rabbitmq standard rabbit 124 | ---- 125 | 126 | Check if SPRING_CLOUD_DEPLOYER_CLOUDFOUNDRY_URL and SPRING_CLOUD_DEPLOYER_CLOUDFOUNDRY_DOMAIN values are ok in your environment in manifest-scdf.yml file. Change the URL and domainname if it is differnt in yours. 127 | Replace below "path" to the cloud foundry jar file and run 128 | ---- 129 | $ cf push -f manifest-scdf.yml --no-start -p -k 1500M 130 | $ cf start dataflow-server 131 | ---- 132 | 133 | ==== Step 3. Compile rxjava-taxi-processor Processor Application 134 | Run following commands - 135 | ---- 136 | $ cd rxjava-taxi-processor 137 | $ mvn package 138 | $ cd staticdir 139 | $ cp ../target/spring-cloud-stream-rxjava-taxi-rabbit-1.1.1.RELEASE-exec.jar . 140 | $ cf push 141 | ---- 142 | 143 | Goto http://rxjavataxi-jar.local.pcfdev.io/ and copy the URL of the JAR File. We need it later to upload this jar file 144 | 145 | ==== Step 4. Start SCDF Client 146 | Run following commands. Make sure you replace url of dataflow-server application if it is different - 147 | ---- 148 | $ java -jar spring-cloud-dataflow-shell-1.0.0.RELEASE.jar 149 | server-unknown:>dataflow config server http://dataflow-server.local.pcfdev.io 150 | dataflow:>app import --uri http://bit.ly/stream-applications-rabbit-maven 151 | ---- 152 | Test a sample stream. We would deploy time "source" application that will output time every 1 second. The ouput would go to log "sink" application that would print whatever output it gets into its syslog. 153 | 154 | On dataflow prompt, run following - 155 | ---- 156 | dataflow:>stream create --name test --definition "time | log" --deploy 157 | ---- 158 | 159 | Wait for deployment. It might take some time to complete. Once completed run the following commands on shell prompt - 160 | 161 | ---- 162 | $ cf apps 163 | Getting apps in org pcfdev-org / space SCDF as admin... 164 | OK 165 | 166 | name requested state instances memory disk urls 167 | dataflow-server started 1/1 2G 1G dataflow-server.local2.pcfdev.io 168 | rxjavataxi-jar started 1/1 300M 512M rxjavataxi-jar.local2.pcfdev.io 169 | dataflow-lithaemic-thew-test-log started 1/1 400M 1G dataflow-lithaemic-thew-test-log.local2.pcfdev.io 170 | dataflow-lithaemic-thew-test-time started 1/1 400M 1G dataflow-lithaemic-thew-test-time.local2.pcfdev.io 171 | 172 | $ cf logs dataflow-lithaemic-thew-test-log 173 | Connected, tailing logs for app dataflow-lithaemic-thew-test-log in org pcfdev-org / space SCDF as admin... 174 | 175 | 2016-10-03T09:39:28.35+0800 [APP/0] OUT 2016-10-03 01:39:28.353 INFO 21 --- [est.time.test-1] log.sink : 10/03/16 01:39:28 176 | 2016-10-03T09:39:29.35+0800 [APP/0] OUT 2016-10-03 01:39:29.357 INFO 21 --- [est.time.test-1] log.sink : 10/03/16 01:39:29 177 | 2016-10-03T09:39:30.38+0800 [APP/0] OUT 2016-10-03 01:39:30.379 INFO 21 --- [est.time.test-1] log.sink : 10/03/16 01:39:30 178 | ---- 179 | 180 | Destroy the stream now 181 | ---- 182 | dataflow:>stream destroy --name test 183 | ---- 184 | Register rxjava-taxi "Processor" application. Change the URL appropriately if it different in your PCFDev env. 185 | ---- 186 | dataflow:> app register --name rxjava-taxi --type processor --uri http://rxjavataxi-jar.local.pcfdev.io/spring-cloud-stream-rxjava-taxi-rabbit-1.1.1.RELEASE-exec.jar 187 | 188 | dataflow:> app info --id processor:rxjava-taxi 189 | ---- 190 | 191 | ==== Step 5. Create and Run our Stream 192 | 193 | Create Redis instance and an account (key). This is where we are going to push our processed data. Note down the account details. We would need it later during our stream definition. 194 | ---- 195 | $ cf create-service p-redis shared-vm redis-taxi 196 | $ cf create-service-key redis-taxi redis-taxi-key 197 | $ cf service-key redis-taxi redis-taxi-key 198 | Getting key redis-taxi-key for service instance redis-taxi as admin... 199 | 200 | { 201 | "host": "redis.local.pcfdev.io", 202 | "password": "ed4c3e65-d092-4e3a-a37d-07e4589f7f86", 203 | "port": 40829 204 | } 205 | ---- 206 | Download the data file that we would be streaming here. It is around 50MB in size and put under an empty directory. I've used /tmp/SCDF to store this file. 207 | 208 | http://taxi-data-ks.cfapps.io/sorted_data.csv 209 | 210 | Create 4 Streams for our Taxi demo. These streams could be visualized better here. 211 | 212 | image::images/stream_definition.png[] 213 | 214 | Find out IP address of your laptop. Do not use "localhost" in the --host property of ftp below 215 | Replace passsword, host, remoteDir for FTP definition as well as password, host and port for redis-pubsub application below 216 | ---- 217 | dataflow:> stream create --name TAXISTREAM_1 --definition "ftp --remote-dir=/tmp/SCDF --password=***** --host=192.168.177.1 --username=shuklk2 --auto-create-local-dir=true --filename-pattern=* --mode=lines --client-mode=PASSIVE --local-dir=/tmp/rxjava | rxjava-taxi | splitter --expression=#jsonPath(payload,'$.processingInfoString') | redis-pubsub --key=processingInfo_LATEST_DATA --password=f5fa6412-4417-4fab-b488-14e8a6454a29 --host=redis.local.pcfdev.io --port=35458" 218 | ---- 219 | Create second stream, which gets data out of TAXISTREAM_1's rxjava-taxi's output. Replace password, host and port information for redis-pubsub appropriately. Notice splitter here is extracting data for "freetaxiesListString" key - 220 | ---- 221 | dataflow:> stream create --name TAXISTREAM_2 --definition ":TAXISTREAM_1.rxjava-taxi > splitter --expression=#jsonPath(payload,'$.freetaxiesListString') | redis-pubsub --key=freeTaxies_LATEST_DATA --password=ed4c3e65-d092-4e3a-a37d-07e4589f7f86 --host=redis.local2.pcfdev.io --port=40829" 222 | ---- 223 | Create third stream, which gets data out of TAXISTREAM_1's rxjava-taxi's output. Replace password, host and port information for redis-pubsub appropriately. Notice splitter here is extracting data for "top10routesString" key - 224 | ---- 225 | dataflow:> stream create --name TAXISTREAM_3 --definition ":TAXISTREAM_1.rxjava-taxi > splitter --expression=#jsonPath(payload,'$.top10routesString') | redis-pubsub --key=top10Routes_LATEST_DATA --password=ed4c3e65-d092-4e3a-a37d-07e4589f7f86 --host=redis.local2.pcfdev.io --port=40829" 226 | ---- 227 | Lastly create a stream that is just printing data received from TAXISTREAM_1's rxjava-taxi to syslog 228 | ---- 229 | dataflow:> stream create --name TAXISTREAM_4 --definition ":TAXISTREAM_1.rxjava-taxi > log" 230 | ---- 231 | 232 | Once these are created, you need to deploy them one by one 233 | ---- 234 | dataflow:> stream deploy --name TAXISTREAM_4 235 | dataflow:> stream deploy --name TAXISTREAM_3 236 | dataflow:> stream deploy --name TAXISTREAM_2 237 | dataflow:> stream deploy --name TAXISTREAM_1 238 | ---- 239 | What just happened here? 240 | Spring Cloud Dataflow instruct PCF to create applications and bind them to messaging layer (which is rabbit in this case. See above where we created a rabbit instance. PCF now manages the entire application lifecyle - ie app scaling, fault-tolerance, messaging layer resiliency, application self-healing. 241 | 242 | Notice that you never specified here that which exchange each of the application should bind to get the message from rabbit. SCDF handles this during application deployment. 243 | 244 | Make sure, all are deployed by checking their "status" column 245 | ---- 246 | dataflow:> stream list 247 | ---- 248 | Goto shell prompt, make sure all applications status is "started" and get the name of the "log" application 249 | ---- 250 | $ cf apps; 251 | Getting apps in org pcfdev-org / space SCDF as admin... 252 | OK 253 | 254 | name requested state instances memory disk urls 255 | dataflow-server started 1/1 2G 1.5G dataflow-server.local.pcfdev.io 256 | rxjavataxi-jar started 1/1 300M 512M rxjavataxi-jar.local.pcfdev.io 257 | dataflow-nonconvective-sabotage-TAXISTREAM_4-log started 1/1 400M 1G dataflow-nonconvective-sabotage-TAXISTREAM_4-log.local.pcfdev.io 258 | dataflow-nonconvective-sabotage-TAXISTREAM_3-redis-pubsub started 1/1 400M 1G dataflow-nonconvective-sabotage-TAXISTREAM_3-redis-pubsub.local.pcfdev.io 259 | dataflow-nonconvective-sabotage-TAXISTREAM_3-splitter started 1/1 400M 1G dataflow-nonconvective-sabotage-TAXISTREAM_3-splitter.local.pcfdev.io 260 | dataflow-nonconvective-sabotage-TAXISTREAM_2-redis-pubsub started 1/1 400M 1G dataflow-nonconvective-sabotage-TAXISTREAM_2-redis-pubsub.local.pcfdev.io 261 | dataflow-nonconvective-sabotage-TAXISTREAM_2-splitter started 1/1 400M 1G dataflow-nonconvective-sabotage-TAXISTREAM_2-splitter.local.pcfdev.io 262 | dataflow-nonconvective-sabotage-TAXISTREAM_1-redis-pubsub started 1/1 400M 1G dataflow-nonconvective-sabotage-TAXISTREAM_1-redis-pubsub.local.pcfdev.io 263 | dataflow-nonconvective-sabotage-TAXISTREAM_1-splitter started 1/1 400M 1G dataflow-nonconvective-sabotage-TAXISTREAM_1-splitter.local.pcfdev.io 264 | dataflow-nonconvective-sabotage-TAXISTREAM_1-rxjava-taxi started 1/1 400M 1G dataflow-nonconvective-sabotage-TAXISTREAM_1-rxjava-taxi.local.pcfdev.io 265 | dataflow-nonconvective-sabotage-TAXISTREAM_1-ftp started 1/1 400M 1G dataflow-nonconvective-sabotage-TAXISTREAM_1-ftp.local.pcfdev.io 266 | ---- 267 | Just print the logs, and you would notice that the output changes after every 10 seconds. Whatever be the output from rxjava-taxi application, it is displayed in syslog for the log application 268 | ---- 269 | $ cf logs dataflow-nonconvective-sabotage-TAXISTREAM_4-log 270 | 2016-10-03T11:56:22.47+0800 [APP/0] OUT 2016-10-03 03:56:22.469 INFO 18 --- [.TAXISTREAM_4-1] log.sink : {"processingInfoString":"1173380|28374|616|172|","freetaxiesListString":"2789D8398CBD60E51BF7D4BC78F3D7A9,40.799335,-73.935265|27A66351B6F3872FBF696EB66FFB983C,40.761326,-73.986885|2AF58A5DD84CB1BDF128F20C7D7D24B2,40.749386,-73.992012|2DA69B9659AF1B087A3BC7D5FEF3B2B7,40.682144,-74.000290|2DE23D4516D572D83A6D8F0CD7DF13F7,40.772694,-73.952477|2E34B3CAEB763EA98BAA700EFFC78E3D,40.851303,-73.934013|2E8F02FAEDF24344D4DF461081F69719,40.737156,-73.983330|2FCB26B1A3EE7E5B8B457D74A5CEAEB8,40.786713,-73.954552|32697E6BE565538CAEB0B6421C1F6813,40.782051,-73.975670|338AA5E5CCF2B215438CE6EB2069D8F0,40.723927,-73.992561|33C5CA859B7EB35E11E63A777670DBEB,40.710972,-74.008858|36EDC01D57A6489E2DEE50734ECB327D,40.767937,-73.956024|38114212AD7C2127DE737863C465656D,40.734066,-73.989166|385FC5F391764A5E2439617893CD939D,40.788887,-73.952011|389B87D436761A3DB881F7DFCD141535,40.772522,-73.958641|39367BFD20B0D0B4EE51F070A555BF98,40.769001,-73.951973|3AB9079A089EE33D7FA7259B482770B4,40.784130,-73.977638|3B0D32434F9E18BC4040C1E6C79F4240,40.768196,-73.955727|3B6AE3CF05F34ADC91DC68D20F2EB913,40.768940,-73.985359|3B7509562B5CAF1713351D0E0C393739,40.710789,-74.005951|3C1712C40B216B3D1B658BE671C4AC1F,40.731506,-74.000946|3CACE6A20EB462544D4F0F3DA1303EDC,40.776089,-73.955498|3F390CB15E5448BDD56F97A21C564327,40.743092,-73.919724|40276740E070731293B69358A15550BC,40.671528,-73.987862|402C4787A75C94AD08745DCA8DE9014A,40.761875,-73.983124|40C58619FCA4B6298338F0D9EFB252ED,40.748688,-73.952225|4241EAF272062B8DB62C39351EDBA25A,40.710678,-74.009232|43EE741BA1397FECCE21070A51723179,40.751156,-73.990891|45E4FB0397F8D1F087B629C024ED263E,40.712536,-73.991379|479D0A450FE2536023A99C619635D570,40.760925,-73.990623|497948F23936CDE084CDE55FEC259412,40.699760,-73.939880|49A0A4465DC7BA419D2C96A642DE1471,40.768486,-73.924644|49B8CFC71F0ED39C7B3F68F603DB9D05,40.707893,-74.003677|4A18D911DE22F561086D76262E6AE42E,40.686440,-73.974586|4BA6D1DB19341443B959A08C41489864,40.722054,-73.996529|4CF1844ACD94BB5936B1774FC8B671C2,40.777954,-73.947968|4D24F4D8EF35878595044A52B098DFD2,40.729916,-73.957588|4E9A475A4114E192E07BE354E36C1B60,40.709850,-73.991943|4FDF7467A2038D09DC1089EA72CFBAD2,40.744904,-73.976212|520630AB0295AEF2B625E52300F46513,40.766411,-73.983650|54E0A7E84EAFDF6D0C70DC8E8A272FD9,40.739216,-73.982986|54EC3864DD7CB3DAEB7FF9D200EA82D4,40.754921,-73.921211|582CAF46446FA03320E5178E7BEC1E86,40.774971,-73.953064|59A0C16E586FBDD9CBA755393BC8B279,40.808048,-73.946213|5D1461EE35FACA225F7241668F963419,40.731785,-73.982170|5D2AF934389358F121B59D6DD4A33DCA,40.725376,-73.940636|5D506A80496D56D8E4F6BAA159C76DA5,40.737877,-74.009598|60B87DBDF025AE348F8286CAFE999F2C,40.676746,-73.963562|6133F218393DE520E73324B1822E0E25,40.736462,-73.987099|6204D2988E75007ADDC410D3AD59CD01,40.753407,-74.026077|","top10routesString":"{\"route\":\"c:78.82_to_c:78.82\",\"numTrips\":\"120\"}|{\"route\":\"c:78.82_to_c:78.83\",\"numTrips\":\"108\"}|{\"route\":\"c:78.82_to_c:78.81\",\"numTrips\":\"92\"}|{\"route\":\"c:78.83_to_c:78.82\",\"numTrips\":\"90\"}|{\"route\":\"c:78.82_to_c:79.81\",\"numTrips\":\"86\"}|{\"route\":\"c:78.83_to_c:78.83\",\"numTrips\":\"80\"}|{\"route\":\"c:79.80_to_c:79.80\",\"numTrips\":\"76\"}|{\"route\":\"c:78.83_to_c:79.81\",\"numTrips\":\"73\"}|{\"route\":\"c:78.81_to_c:78.81\",\"numTrips\":\"71\"}|{\"route\":\"c:78.82_to_c:77.81\",\"numTrips\":\"69\"}|"} 271 | ---- 272 | 273 | Once completed, you could undeploy all the streams, other than TAXISTREAM_1 - 274 | ---- 275 | dataflow:>stream undeploy --name TAXISTREAM_4 276 | dataflow:>stream undeploy --name TAXISTREAM_3 277 | dataflow:>stream undeploy --name TAXISTREAM_2 278 | ---- 279 | 280 | 281 | === Part 2 - Building Application Microservices 282 | 283 | Completing Part 1 is necessary before progressing with Part 2. Reason being we want some data to be available in redis-server. We cannot expose our database, in this case Redis, and it's schema to everyone. Instead we would be creating application microservices and expose APIs. There are few things we need to understand here - 284 | 285 | 1. We need to register all our microservices to a centralized service registery. Why? So that anyone who wants to use APIs register his own microservice to service registry and get access to all other registered microservices instantly. 286 | 287 | 2. We need to make sure that we give some control to users who are going to use our microservice. If our microservice goes down, then we want them to take some default action. Think about try catch block in java. The difference is that user might not want to execute code in try block if he knows before hand that the microservice is down. We would use circuit-breakers here. 288 | 289 | 3. We would externalize all our variables to an external configuration server and register all our microservice with it. When our microservices are booting up, they would talk to configuration server and get the all relevant variables for the microservice and the relevant environment they are getting provisioned into (think profiles in spring). One clear example here is that we want to provide redis database information via configuration server. 290 | 291 | All the above mentioned services are part of Spring Cloud Services. They are provided out of the box in Pivotal Cloud Foundry. 292 | 293 | Below diagram shows how we are architecting the solution. Eventservice and TaxiService registers to Configuration Server and Service Registry while Aggservice registers to all three. 294 | 295 | image::images/app_microservices.png[] 296 | 297 | ==== Step 0 - Clone the repository again 298 | ---- 299 | $ git clone https://github.com/kgshukla/Realtime-Streaming-DataMicroservices-AppMicroservices.git 300 | $ cd Realtime-Streaming-DataMicroservices-AppMicroservices/ 301 | ---- 302 | 303 | ==== Step 1 - Create Spring Cloud Services Instances 304 | 305 | Fork following repository from github so that you could provide your own configuration information - https://github.com/kgshukla/iot-taxi-config-repo.git 306 | 307 | Perform following to fork the repository 308 | 309 | 1. Go to https://github.com/kgshukla/iot-taxi-config-repo.git 310 | 311 | 2. Click on "Fork" button on the top-right corner 312 | 313 | 3. Clone the forked repository on your laptop using below command. Replace "xxxx" with your github username 314 | 315 | ---- 316 | $ git clone https://github.com/xxxx/iot-taxi-config-repo.git 317 | ---- 318 | 319 | Run following command on terminal to get redis database information 320 | ---- 321 | $ cf service-key redis-taxi redis-taxi-key 322 | Getting key redis-taxi-key for service instance redis-taxi as admin... 323 | 324 | { 325 | "host": "redis.local.pcfdev.io", 326 | "password": "ed4c3e65-d092-4e3a-a37d-07e4589f7f86", 327 | "port": 40829 328 | } 329 | ---- 330 | 331 | Open application.yml file in your forked repository and replace redis_host, redis_port and redis_password with the above information. Once done, run the following commands to commit the changes in your github repository 332 | ---- 333 | $ git add application.yml 334 | $ git commit -m "changing redis host, port and password information 335 | $ git push origin master 336 | ---- 337 | 338 | Now, we would create configuration server, service registry and circuit-breaker instances in our PCF Dev. Replace "xxxx" with your github username. 339 | ---- 340 | $ cf create-service p-config-server standard config-server -c '{"git": { "uri": "https://github.com/kgshukla/iot-taxi-config-repo.git" } }' 341 | Creating service instance config-server in org pcfdev-org / space SCDF as admin... 342 | OK 343 | 344 | Create in progress. Use 'cf services' or 'cf service config-server' to check operation status. 345 | ---- 346 | 347 | Create service registry and circuit-breaker as well 348 | ---- 349 | $ cf create-service p-service-registry standard service-registry 350 | $ cf create-service p-circuit-breaker-dashboard standard circuit-breaker 351 | ---- 352 | 353 | Make sure that these services are up and running before continuing. Run following command to know the status of "last operation" column. It should be "Create Succeded". If not, then wait until all services are in succeeded status. 354 | ---- 355 | $ cf services 356 | Getting services in org pcfdev-org / space SCDF as admin... 357 | OK 358 | 359 | name service plan bound apps last operation 360 | redis p-redis shared-vm dataflow-server create succeeded 361 | rabbit p-rabbitmq standard dataflow-server create succeeded 362 | redis-taxi p-redis shared-vm create succeeded 363 | config-server p-config-server standard create succeeded 364 | service-registry p-service-registry standard create succeeded 365 | circuit-breaker p-circuit-breaker-dashboard standard create succeeded 366 | ---- 367 | 368 | Now, what we are going to do is a "hack" and is never recommended in actual PCF setup. Reason being that, we are running PCF on our laptop and it does not have so much memory that we could run "everything". So we are going to tune down "memory" for instances created for our last three services. Run following commands - 369 | ---- 370 | $ cf target -o p-spring-cloud-services -s instances 371 | $ cf apps 372 | Getting apps in org p-spring-cloud-services / space instances as admin... 373 | OK 374 | 375 | name requested state instances memory disk urls 376 | config-2ace9b60-f896-4cb7-bf07-00db58d1f3eb started 1/1 1G 512M config-2ace9b60-f896-4cb7-bf07-00db58d1f3eb.local.pcfdev.io 377 | eureka-86b2a02f-73cf-4359-b17e-5c22eed4055a started 1/1 1G 512M eureka-86b2a02f-73cf-4359-b17e-5c22eed4055a.local.pcfdev.io 378 | hystrix-793fdd96-1294-49fe-a3e8-de9dd26f3d91 started 1/1 1G 512M hystrix-793fdd96-1294-49fe-a3e8-de9dd26f3d91.local.pcfdev.io 379 | turbine-793fdd96-1294-49fe-a3e8-de9dd26f3d91 started 1/1 1G 512M turbine-793fdd96-1294-49fe-a3e8-de9dd26f3d91.local.pcfdev.io 380 | ---- 381 | 382 | Now we are going to scale each application to 512MB of memory. Replace each app instance name with yours 383 | ---- 384 | $ cf scale eureka-86b2a02f-73cf-4359-b17e-5c22eed4055a -m 512M 385 | This will cause the app to restart. Are you sure you want to scale eureka-86b2a02f-73cf-4359-b17e-5c22eed4055a?> y 386 | 387 | $ cf scale hystrix-793fdd96-1294-49fe-a3e8-de9dd26f3d91 -m 512M 388 | This will cause the app to restart. Are you sure you want to scale hystrix-793fdd96-1294-49fe-a3e8-de9dd26f3d91?> y 389 | 390 | $ cf scale turbine-793fdd96-1294-49fe-a3e8-de9dd26f3d91 -m 512M 391 | This will cause the app to restart. Are you sure you want to scale turbine-793fdd96-1294-49fe-a3e8-de9dd26f3d91?> y 392 | 393 | $ cf scale config-2ace9b60-f896-4cb7-bf07-00db58d1f3eb -m 512M 394 | This will cause the app to restart. Are you sure you want to scale config-2ace9b60-f896-4cb7-bf07-00db58d1f3eb?> y 395 | 396 | ---- 397 | 398 | ==== Step 2 - Compile code and deploy microservices 399 | 400 | Compile code as follows - 401 | ---- 402 | $ cd Realtime-Streaming-DataMicroservices-AppMicroservices/ 403 | $ mvn package -DskipTests 404 | ---- 405 | 406 | ==== Step 3 - Push all your microservices 407 | 408 | Alright, now is the time to push your applications one by one. We will first push taxiservice. Make sure CF_TARGET value is coorect. You could find target via running "$ cf target" on your terminal. 409 | ---- 410 | $ cat manifest-taxiservice.yml 411 | --- 412 | instances: 1 413 | memory: 400M 414 | applications: 415 | - name: taxiservice-iot 416 | host: taxiservice-iot 417 | path: taxiservice/target/taxiservice-1.0-SNAPSHOT.jar 418 | services: 419 | - config-server 420 | - service-registry 421 | env: 422 | SPRING_PROFILES_ACTIVE: pcf 423 | CF_TARGET: https://api.local.pcfdev.io 424 | ---- 425 | Notice that this application is binded to two services config-server and service-registry. This indicates that this application is going to talk to config-server to get all relevant env variables (which includes redis host, password and port) and it is going to register itself in service-registry. 426 | 427 | ---- 428 | $ cf target -o pcfdev-org -s SCDF 429 | $ cf push -f manifest-taxiservice.yml 430 | .... 431 | .... 432 | Showing health and status for app taxiservice-iot in org pcfdev-org / space SCDF as admin... 433 | OK 434 | 435 | requested state: started 436 | instances: 1/1 437 | usage: 400M x 1 instances 438 | urls: taxiservice-iot.local.pcfdev.io 439 | last uploaded: Sat Oct 8 10:07:30 UTC 2016 440 | stack: cflinuxfs2 441 | buildpack: java-buildpack=v3.8.1-offline-https://github.com/cloudfoundry/java-buildpack.git#29c79f2 java-main open-jdk-like-jre=1.8.0_91-unlimited-crypto open-jdk-like-memory-calculator=2.0.2_RELEASE spring-auto-reconfiguration=1.10.0_RELEASE 442 | ---- 443 | 444 | Once taxiservice is deployed, copy the url (as mentioned in the output above), open a browser and type following - http://taxiservice-iot.local.pcfdev.io/routes/top10routes. It should show you the top 10 routes. This indicates that your taxiservice has received redis credentials properly from config-server and able to show information. Now we push eventservice. Make sure you check CF_TARGET value is set fine in manifest-eventservice.yml file. 445 | 446 | ---- 447 | $ cf push -f manifest-eventservice.yml 448 | ---- 449 | 450 | Once it is deployed, open your PCFDev web browser. Use following link - https://console.local.pcfdev.io/2/ and use admin/admin credentials. Follow this link:images/service_registry.pdf[file] to know where to click in the browser. 451 | 452 | You would notice that both taxiservice and eventservice have registered to service-registry. We would now make aggregator service to use them via application name, rather than a fixed URL. 453 | 454 | Open "aggregatorservice/src/main/java/io/pivotal/data/aggregatorservice/service/AggregatorService.java" and notice following code - 455 | 456 | ---- 457 | @HystrixCommand(fallbackMethod = "fallbackEventCalls") 458 | public long getTotalEvents() { 459 | return restTemplate.getForObject("https://EVENTSERVICE-IOT-V1/events/total", Long.class); 460 | } 461 | 462 | @HystrixCommand(fallbackMethod = "fallbackEventCalls") 463 | public long getMissedEvents() { 464 | return restTemplate.getForObject("https://EVENTSERVICE-IOT-V1/events/missed", Long.class); 465 | } 466 | ---- 467 | Two things are important here - 468 | 1. agg service is calling EventService just by the name it has registered in service-registry, ie EVENTSERVICE-IOT-V1, it is not doing a hard coded URL binding to the service. 469 | 2. If for some reason this call fails, "fallbackEventCalls" method will kick in by circuit breaker and return whatever is coded in the method until circuit breaker figures out that eventservice is up and running again. 470 | 471 | Let's push aggservice now. Make sure CF_TARGET value in manifest-aggservice.yml file is set properly. 472 | ---- 473 | $ cf push -f manifest-aggservice.yml 474 | 475 | create a user provided service. Make sure you use correct url of aggservice as provided in your environment. This will be used by webapp. 476 | 477 | $cf create-user-provided-service agg_service_iot -p '{"AGGSERVICE_URL":"aggservice-iot.local.pcfdev.io"}' 478 | Creating user provided service agg_service_iot in org pcfdev-org / space SCDF as admin... 479 | OK 480 | ---- 481 | 482 | Now push the final webapp 483 | ---- 484 | $ cd webapp_php 485 | $ cf push -b php_buildpack 486 | ... 487 | ... 488 | Showing health and status for app iot-taxi in org pcfdev-org / space SCDF as admin... 489 | OK 490 | 491 | requested state: started 492 | instances: 1/1 493 | usage: 356M x 1 instances 494 | urls: iot-taxi.local.pcfdev.io 495 | last uploaded: Sat Oct 8 10:33:27 UTC 2016 496 | stack: cflinuxfs2 497 | buildpack: php_buildpack 498 | ---- 499 | 500 | Copy http://iot-taxi.local.pcfdev.io and open in web browser. You must be able to see all data now. Click on "Event Processing" and "Free Taxies" button to view the data. 501 | 502 | image::images/streaming_img1.png[] 503 | 504 | image::images/streaming_img3.png[] 505 | 506 | ==== Step 4 - Try out different use cases 507 | 508 | 1. Stop eventservice and see what your UI shows. It should show data that is returned in "fallbackEventCalls" method. 509 | 510 | ---- 511 | $ cf stop eventservice-iot 512 | ---- 513 | 514 | Also, circuit breaker would come into action and it would show red color as shown in following link:images/circuit_breaker.pdf[file]. 515 | 516 | You should see something like this 517 | 518 | image::images/streaming_img2.png[] 519 | 520 | 521 | 2. Scale eventservice to 2 instances and see how service-registry registers both instances. Goto service-registry UI to see the registration 522 | 523 | ---- 524 | $ cf start eventservice-iot 525 | $ cf scale eventservice-iot -i 2 526 | ---- 527 | 528 | 3. Start realtime streaming one more time. Move the downloaded file "sorted_data.csv" to "sorted_data2.csv". I had downloaded this to /tmp/SCDF directory. Since your TAXISTREAM_1 is still active and constantly monitoring the source directory, any new file added to the directory would start processing it automatically as it is added. You could see data movement under "Event Processing" button on your web app. 529 | 530 | ---- 531 | $ cd /tmp/SCDF 532 | $ mv sorted_data.csv sorted_data2.csv 533 | ---- 534 | 535 | This completes the workshop. 536 | -------------------------------------------------------------------------------- /aggregatorservice/.classpath: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | -------------------------------------------------------------------------------- /aggregatorservice/.gitignore: -------------------------------------------------------------------------------- 1 | /target/ 2 | -------------------------------------------------------------------------------- /aggregatorservice/.project: -------------------------------------------------------------------------------- 1 | 2 | 3 | aggregatorservice 4 | 5 | 6 | 7 | 8 | 9 | org.eclipse.jdt.core.javabuilder 10 | 11 | 12 | 13 | 14 | org.eclipse.m2e.core.maven2Builder 15 | 16 | 17 | 18 | 19 | org.springframework.ide.eclipse.core.springbuilder 20 | 21 | 22 | 23 | 24 | 25 | org.springframework.ide.eclipse.core.springnature 26 | org.eclipse.jdt.core.javanature 27 | org.eclipse.m2e.core.maven2Nature 28 | 29 | 30 | -------------------------------------------------------------------------------- /aggregatorservice/.settings/org.eclipse.core.resources.prefs: -------------------------------------------------------------------------------- 1 | eclipse.preferences.version=1 2 | encoding//src/main/java=UTF-8 3 | encoding//src/main/resources=UTF-8 4 | encoding//src/test/java=UTF-8 5 | encoding//src/test/resources=UTF-8 6 | encoding/=UTF-8 7 | -------------------------------------------------------------------------------- /aggregatorservice/.settings/org.eclipse.jdt.core.prefs: -------------------------------------------------------------------------------- 1 | eclipse.preferences.version=1 2 | org.eclipse.jdt.core.compiler.codegen.targetPlatform=1.8 3 | org.eclipse.jdt.core.compiler.compliance=1.8 4 | org.eclipse.jdt.core.compiler.problem.forbiddenReference=warning 5 | org.eclipse.jdt.core.compiler.source=1.8 6 | -------------------------------------------------------------------------------- /aggregatorservice/.settings/org.eclipse.m2e.core.prefs: -------------------------------------------------------------------------------- 1 | activeProfiles= 2 | eclipse.preferences.version=1 3 | resolveWorkspaceProjects=true 4 | version=1 5 | -------------------------------------------------------------------------------- /aggregatorservice/pom.xml: -------------------------------------------------------------------------------- 1 | 2 | 4.0.0 3 | 4 | io.pivotal.spring.cloud 5 | spring-cloud-services-starter-parent 6 | 1.0.1.RELEASE 7 | 8 | 9 | 10 | io.pivotal.data.taxi.iot 11 | aggregatorservice 12 | 1.0-SNAPSHOT 13 | 14 | 15 | 16 | io.pivotal.spring.cloud 17 | spring-cloud-services-starter-config-client 18 | 19 | 20 | io.pivotal.spring.cloud 21 | spring-cloud-services-starter-service-registry 22 | 23 | 24 | io.pivotal.spring.cloud 25 | spring-cloud-services-starter-circuit-breaker 26 | 27 | 28 | org.springframework.boot 29 | spring-boot-starter-web 30 | 31 | 32 | org.springframework.boot 33 | spring-boot-starter-test 34 | 35 | 36 | 37 | com.google.guava 38 | guava 39 | 19.0-rc1 40 | 41 | 42 | org.springframework.cloud 43 | spring-cloud-starter 44 | 1.0.1.RELEASE 45 | 46 | 47 | org.springframework.cloud 48 | spring-cloud-starter-ribbon 49 | 1.0.2.RELEASE 50 | 51 | 52 | org.springframework.boot 53 | spring-boot-starter-actuator 54 | 55 | 56 | 57 | UTF-8 58 | 1.8 59 | 1.8 60 | 61 | 62 | 63 | 64 | 65 | org.springframework.boot 66 | spring-boot-maven-plugin 67 | 68 | 69 | 70 | 71 | 72 | 73 | spring-release 74 | https://repo.spring.io/libs-release 75 | 76 | 77 | 78 | 79 | -------------------------------------------------------------------------------- /aggregatorservice/src/main/java/io/pivotal/data/aggregatorservice/App.java: -------------------------------------------------------------------------------- 1 | /** 2 | * 3 | */ 4 | package io.pivotal.data.aggregatorservice; 5 | 6 | import org.springframework.boot.SpringApplication; 7 | import org.springframework.boot.autoconfigure.SpringBootApplication; 8 | import org.springframework.cloud.client.circuitbreaker.EnableCircuitBreaker; 9 | import org.springframework.cloud.client.discovery.EnableDiscoveryClient; 10 | 11 | /** 12 | * @author shuklk2 13 | * 14 | */ 15 | @SpringBootApplication 16 | @EnableDiscoveryClient 17 | @EnableCircuitBreaker 18 | public class App { 19 | 20 | /** 21 | * @param args 22 | */ 23 | public static void main(String[] args) { 24 | SpringApplication.run(App.class, args); 25 | 26 | } 27 | 28 | } 29 | -------------------------------------------------------------------------------- /aggregatorservice/src/main/java/io/pivotal/data/aggregatorservice/controller/AggregatorServiceController.java: -------------------------------------------------------------------------------- 1 | /** 2 | * 3 | */ 4 | package io.pivotal.data.aggregatorservice.controller; 5 | 6 | import io.pivotal.data.aggregatorservice.model.DriverIncorrectData; 7 | import io.pivotal.data.aggregatorservice.model.DriverRevenueData; 8 | import io.pivotal.data.aggregatorservice.model.RouteData; 9 | import io.pivotal.data.aggregatorservice.service.AggregatorService; 10 | 11 | import java.util.List; 12 | 13 | import org.springframework.beans.factory.annotation.Autowired; 14 | import org.springframework.web.bind.annotation.RequestMapping; 15 | import org.springframework.web.bind.annotation.RequestMethod; 16 | import org.springframework.web.bind.annotation.RestController; 17 | 18 | /** 19 | * @author shuklk2 20 | * 21 | */ 22 | @RestController 23 | public class AggregatorServiceController { 24 | 25 | @Autowired 26 | private AggregatorService service; 27 | 28 | @RequestMapping(value = "/totalevents", method = RequestMethod.GET) 29 | public long getTotalEvents() { 30 | return service.getTotalEvents(); 31 | } 32 | 33 | @RequestMapping(value = "/missedevents", method = RequestMethod.GET) 34 | public long getMissedEvents() { 35 | return service.getMissedEvents(); 36 | } 37 | 38 | @RequestMapping(value = "/correctevents", method = RequestMethod.GET) 39 | public long getCorrectEvents() { 40 | return service.getCorrectEvents(); 41 | } 42 | 43 | @RequestMapping(value = "/processtime", method = RequestMethod.GET) 44 | public long processTime() { 45 | return service.processTime(); 46 | } 47 | 48 | @RequestMapping(value = "/freetaxies", method = RequestMethod.GET) 49 | public List> getFreeTaxiesList() { 50 | return service.getFreeTaxiesList(); 51 | } 52 | 53 | @RequestMapping(value = "/top10routes", method = RequestMethod.GET) 54 | public List getTop10Routes() { 55 | return service.getTop10Routes(); 56 | } 57 | } 58 | -------------------------------------------------------------------------------- /aggregatorservice/src/main/java/io/pivotal/data/aggregatorservice/model/DriverIncorrectData.java: -------------------------------------------------------------------------------- 1 | /** 2 | * 3 | */ 4 | package io.pivotal.data.aggregatorservice.model; 5 | 6 | /** 7 | * @author shuklk2 8 | * 9 | */ 10 | public class DriverIncorrectData { 11 | 12 | private String drivernum; 13 | private int numoferrors; 14 | public String getDrivernum() { 15 | return drivernum; 16 | } 17 | public void setDrivernum(String drivernum) { 18 | this.drivernum = drivernum; 19 | } 20 | public int getNumoferrors() { 21 | return numoferrors; 22 | } 23 | public void setNumoferrors(int numoferrors) { 24 | this.numoferrors = numoferrors; 25 | } 26 | 27 | 28 | } -------------------------------------------------------------------------------- /aggregatorservice/src/main/java/io/pivotal/data/aggregatorservice/model/DriverRevenueData.java: -------------------------------------------------------------------------------- 1 | /** 2 | * 3 | */ 4 | package io.pivotal.data.aggregatorservice.model; 5 | 6 | /** 7 | * @author shuklk2 8 | * 9 | */ 10 | public class DriverRevenueData { 11 | 12 | private String drivernum; 13 | private int earnedrev; 14 | public String getDrivernum() { 15 | return drivernum; 16 | } 17 | public void setDrivernum(String drivernum) { 18 | this.drivernum = drivernum; 19 | } 20 | public int getEarnedrev() { 21 | return earnedrev; 22 | } 23 | public void setEarnedrev(int earnedrev) { 24 | this.earnedrev = earnedrev; 25 | } 26 | 27 | } 28 | -------------------------------------------------------------------------------- /aggregatorservice/src/main/java/io/pivotal/data/aggregatorservice/model/RouteData.java: -------------------------------------------------------------------------------- 1 | /** 2 | * 3 | */ 4 | package io.pivotal.data.aggregatorservice.model; 5 | 6 | /** 7 | * @author shuklk2 8 | * 9 | */ 10 | public class RouteData { 11 | 12 | private String route; 13 | private String numtrips; 14 | private String starttime; 15 | private String endtime; 16 | 17 | public RouteData() {} 18 | 19 | public RouteData (String route, String numtrips, String starttime, String endtime) { 20 | this.route = route; 21 | this.numtrips = numtrips; 22 | this.starttime = starttime; 23 | this.endtime = endtime; 24 | } 25 | 26 | public String getRoute() { 27 | return route; 28 | } 29 | public void setRoute(String route) { 30 | this.route = route; 31 | } 32 | public String getNumtrips() { 33 | return numtrips; 34 | } 35 | public void setNumtrips(String numtrips) { 36 | this.numtrips = numtrips; 37 | } 38 | public String getStarttime() { 39 | return starttime; 40 | } 41 | public void setStarttime(String starttime) { 42 | this.starttime = starttime; 43 | } 44 | public String getEndtime() { 45 | return endtime; 46 | } 47 | public void setEndtime(String endtime) { 48 | this.endtime = endtime; 49 | } 50 | } 51 | -------------------------------------------------------------------------------- /aggregatorservice/src/main/java/io/pivotal/data/aggregatorservice/service/AggregatorService.java: -------------------------------------------------------------------------------- 1 | /** 2 | * 3 | */ 4 | package io.pivotal.data.aggregatorservice.service; 5 | 6 | import io.pivotal.data.aggregatorservice.model.DriverIncorrectData; 7 | import io.pivotal.data.aggregatorservice.model.DriverRevenueData; 8 | import io.pivotal.data.aggregatorservice.model.RouteData; 9 | 10 | import java.util.ArrayList; 11 | import java.util.List; 12 | 13 | import org.springframework.beans.factory.annotation.Autowired; 14 | import org.springframework.cloud.client.loadbalancer.LoadBalanced; 15 | import org.springframework.context.annotation.Bean; 16 | import org.springframework.context.annotation.Primary; 17 | import org.springframework.stereotype.Service; 18 | import org.springframework.web.client.RestTemplate; 19 | 20 | import com.netflix.hystrix.contrib.javanica.annotation.HystrixCommand; 21 | 22 | /** 23 | * @author shuklk2 24 | * 25 | */ 26 | @Service 27 | public class AggregatorService { 28 | 29 | /* 30 | @Primary 31 | @Bean 32 | RestTemplate restTemplate() { 33 | return new RestTemplate(); 34 | } 35 | */ 36 | 37 | @Autowired 38 | RestTemplate restTemplate; 39 | 40 | @HystrixCommand(fallbackMethod = "fallbackEventCalls") 41 | public long getTotalEvents() { 42 | return restTemplate.getForObject("https://EVENTSERVICE-IOT-V1/events/total", Long.class); 43 | } 44 | 45 | @HystrixCommand(fallbackMethod = "fallbackEventCalls") 46 | public long getMissedEvents() { 47 | return restTemplate.getForObject("https://EVENTSERVICE-IOT-V1/events/missed", Long.class); 48 | } 49 | 50 | @HystrixCommand(fallbackMethod = "fallbackEventCalls") 51 | public long getCorrectEvents() { 52 | return restTemplate.getForObject("https://EVENTSERVICE-IOT-V1/events/correct", Long.class); 53 | } 54 | 55 | @HystrixCommand(fallbackMethod = "fallbackEventCalls") 56 | public long processTime() { 57 | return restTemplate.getForObject("https://EVENTSERVICE-IOT-V1/events/proctime", Long.class); 58 | } 59 | 60 | @HystrixCommand(fallbackMethod = "fallbackfreeTaxiCall") 61 | public List> getFreeTaxiesList() { 62 | return restTemplate.getForObject("https://TAXISERVICE-IOT-V1/freetaxies/list", List.class); 63 | } 64 | 65 | @HystrixCommand(fallbackMethod = "fallbackRoutesCall") 66 | public List getTop10Routes() { 67 | return restTemplate.getForObject("https://TAXISERVICE-IOT-V1/routes/top10routes", List.class); 68 | } 69 | 70 | public List getTop10EarningDrivers() { 71 | return restTemplate.getForObject("https://ANALYTICSERVICE-IOT-V1/analytics/top10earningdrivers", List.class); 72 | } 73 | 74 | public List getTop10ErringDrivers() { 75 | return restTemplate.getForObject("https://ANALYTICSERVICE-IOT-V1/analytics/top10erringdrivers", List.class); 76 | } 77 | 78 | private List fallbackRoutesCall() { 79 | return new ArrayList(); 80 | } 81 | 82 | private List> fallbackfreeTaxiCall() { 83 | return new ArrayList>(); 84 | } 85 | 86 | private long fallbackEventCalls() { 87 | return 0L; 88 | } 89 | 90 | private List fallbackAnalyticsEarningDriverCall() { 91 | return new ArrayList(); 92 | } 93 | 94 | private List fallbackAnalyticsErringDriverCall() { 95 | return new ArrayList(); 96 | } 97 | } 98 | -------------------------------------------------------------------------------- /aggregatorservice/src/main/resources/application.yml: -------------------------------------------------------------------------------- 1 | spring: 2 | application: 3 | name: aggservice-iot-v1 4 | 5 | security: 6 | basic: 7 | enabled: false 8 | 9 | management: 10 | security: 11 | enabled: false 12 | 13 | ribbon: 14 | ReadTimeout: 50000 15 | ConnectTimeout: 3000 16 | 17 | --- 18 | 19 | spring: 20 | profiles: pcf 21 | 22 | eureka: 23 | client: 24 | serviceUrl: 25 | defaultZone: ${vcap.services.service-registry.credentials.uri:http://127.0.0.1:8761}/eureka/ 26 | instance: 27 | hostname: ${vcap.application.uris[0]} 28 | nonSecurePort: 80 29 | metadataMap: 30 | instanceId: ${vcap.application.instance_id:${spring.application.name}:${server.port:8080}} 31 | 32 | -------------------------------------------------------------------------------- /eventservice/.classpath: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | -------------------------------------------------------------------------------- /eventservice/.project: -------------------------------------------------------------------------------- 1 | 2 | 3 | eventservice 4 | 5 | 6 | 7 | 8 | 9 | org.eclipse.jdt.core.javabuilder 10 | 11 | 12 | 13 | 14 | org.eclipse.m2e.core.maven2Builder 15 | 16 | 17 | 18 | 19 | org.springframework.ide.eclipse.core.springbuilder 20 | 21 | 22 | 23 | 24 | 25 | org.springframework.ide.eclipse.core.springnature 26 | org.eclipse.jdt.core.javanature 27 | org.eclipse.m2e.core.maven2Nature 28 | 29 | 30 | -------------------------------------------------------------------------------- /eventservice/.settings/org.eclipse.core.resources.prefs: -------------------------------------------------------------------------------- 1 | eclipse.preferences.version=1 2 | encoding//src/main/java=UTF-8 3 | encoding//src/test/java=UTF-8 4 | encoding/=UTF-8 5 | -------------------------------------------------------------------------------- /eventservice/.settings/org.eclipse.jdt.core.prefs: -------------------------------------------------------------------------------- 1 | eclipse.preferences.version=1 2 | org.eclipse.jdt.core.compiler.codegen.targetPlatform=1.8 3 | org.eclipse.jdt.core.compiler.compliance=1.8 4 | org.eclipse.jdt.core.compiler.problem.forbiddenReference=warning 5 | org.eclipse.jdt.core.compiler.source=1.8 6 | -------------------------------------------------------------------------------- /eventservice/.settings/org.eclipse.m2e.core.prefs: -------------------------------------------------------------------------------- 1 | activeProfiles= 2 | eclipse.preferences.version=1 3 | resolveWorkspaceProjects=true 4 | version=1 5 | -------------------------------------------------------------------------------- /eventservice/pom.xml: -------------------------------------------------------------------------------- 1 | 2 | 5 | 4.0.0 6 | 7 | io.pivotal.data.taxi.iot 8 | eventservice 9 | 1.0-SNAPSHOT 10 | 11 | 12 | UTF-8 13 | 1.8 14 | 1.8 15 | 16 | 17 | 18 | io.pivotal.spring.cloud 19 | spring-cloud-services-starter-parent 20 | 1.0.1.RELEASE 21 | 22 | 23 | 24 | 25 | 26 | io.pivotal.spring.cloud 27 | spring-cloud-services-starter-config-client 28 | 29 | 30 | io.pivotal.spring.cloud 31 | spring-cloud-services-starter-service-registry 32 | 33 | 34 | org.springframework.boot 35 | spring-boot-starter-web 36 | 37 | 38 | org.springframework.boot 39 | spring-boot-starter-data-redis 40 | 1.4.1.RELEASE 41 | 42 | 43 | com.google.guava 44 | guava 45 | 19.0-rc1 46 | 47 | 48 | org.springframework.cloud 49 | spring-cloud-starter 50 | 1.0.1.RELEASE 51 | 52 | 53 | org.springframework.cloud 54 | spring-cloud-starter-eureka 55 | 1.0.2.RELEASE 56 | 57 | 58 | org.springframework.boot 59 | spring-boot-starter-test 60 | 61 | 62 | org.springframework.boot 63 | spring-boot-starter-actuator 64 | 65 | 66 | org.springframework.cloud 67 | spring-cloud-starter-bus-amqp 68 | 69 | 70 | 71 | 72 | 73 | 74 | org.springframework.boot 75 | spring-boot-maven-plugin 76 | 77 | 78 | 79 | 80 | 81 | 82 | spring-release 83 | https://repo.spring.io/libs-release 84 | 85 | 86 | 87 | 88 | -------------------------------------------------------------------------------- /eventservice/src/main/java/io/pivotal/data/eventservice/App.java: -------------------------------------------------------------------------------- 1 | package io.pivotal.data.eventservice; 2 | 3 | 4 | import org.springframework.boot.SpringApplication; 5 | import org.springframework.boot.autoconfigure.SpringBootApplication; 6 | import org.springframework.cloud.client.discovery.EnableDiscoveryClient; 7 | import org.springframework.context.annotation.ImportResource; 8 | import org.springframework.context.annotation.Profile; 9 | import org.springframework.test.context.ActiveProfiles; 10 | 11 | /** 12 | * Created by cq on 17/9/15. 13 | */ 14 | @SpringBootApplication 15 | @ActiveProfiles 16 | @EnableDiscoveryClient 17 | public class App { 18 | 19 | public static void main(String[] args) { 20 | SpringApplication.run(App.class, args); 21 | } 22 | } 23 | -------------------------------------------------------------------------------- /eventservice/src/main/java/io/pivotal/data/eventservice/controllers/EventMetricsController.java: -------------------------------------------------------------------------------- 1 | package io.pivotal.data.eventservice.controllers; 2 | 3 | import io.pivotal.data.eventservice.services.EventsService; 4 | 5 | import org.springframework.beans.factory.annotation.Autowired; 6 | import org.springframework.beans.factory.annotation.Value; 7 | import org.springframework.http.MediaType; 8 | import org.springframework.web.bind.annotation.RequestMapping; 9 | import org.springframework.web.bind.annotation.RequestMethod; 10 | import org.springframework.web.bind.annotation.RestController; 11 | 12 | /** 13 | * Created by cq on 17/9/15. 14 | */ 15 | @RestController 16 | public class EventMetricsController { 17 | 18 | @Autowired 19 | private EventsService eventsService; 20 | 21 | @RequestMapping(value = "/events/test", method = RequestMethod.GET) 22 | public String test(){ 23 | return eventsService.test(); 24 | } 25 | 26 | @RequestMapping(value = "/events/total", method = RequestMethod.GET) 27 | public long getTotalEvents(){ 28 | return eventsService.totalOfEvents(); 29 | } 30 | 31 | @RequestMapping(value = "/events/missed", method = RequestMethod.GET) 32 | public long getMissedEvents(){ 33 | return eventsService.totalMissedEvents(); 34 | } 35 | 36 | @RequestMapping(value = "/events/correct", method = RequestMethod.GET) 37 | public long getCorrectEvents(){ 38 | return eventsService.totalCorrectEvents(); 39 | } 40 | 41 | @RequestMapping(value = "/events/proctime", method = RequestMethod.GET, produces = MediaType.APPLICATION_JSON_VALUE) 42 | public int processTime(){ 43 | return eventsService.latestWPT(); 44 | } 45 | 46 | @RequestMapping(value = "/events/kill", method = RequestMethod.GET) 47 | public String kill() { 48 | System.exit(1); 49 | return ""; 50 | } 51 | 52 | @RequestMapping(value = "/events/load", method = RequestMethod.GET) 53 | public String load() { 54 | System.exit(1); 55 | return ""; 56 | } 57 | 58 | @RequestMapping(value = "/events/instance", method = RequestMethod.GET) 59 | public String instance() { 60 | 61 | return ""; 62 | } 63 | 64 | } 65 | -------------------------------------------------------------------------------- /eventservice/src/main/java/io/pivotal/data/eventservice/services/EventsService.java: -------------------------------------------------------------------------------- 1 | package io.pivotal.data.eventservice.services; 2 | 3 | import org.springframework.beans.factory.annotation.Autowired; 4 | import org.springframework.stereotype.Component; 5 | import org.springframework.data.redis.core.StringRedisTemplate; 6 | 7 | import java.util.List; 8 | import java.util.StringTokenizer; 9 | 10 | /** 11 | * Created by cq on 17/9/15. 12 | */ 13 | @Component 14 | public class EventsService { 15 | 16 | @Autowired 17 | private StringRedisTemplate redisTemplate; 18 | 19 | private String totalEvents; 20 | private String missedEventsinLast10secs; 21 | private String correctEventsinLast10secs; 22 | private String processingTimeforLast10secs; 23 | private final String key = "processingInfo_LATEST_DATA"; 24 | 25 | private void getData() { 26 | System.out.println("getData() called"); 27 | long size = redisTemplate.opsForList().size(key); 28 | if (size > 0) { 29 | System.out.println("Size of list == "+size); 30 | String processingInfo = redisTemplate.opsForList().index(key, size-1); 31 | if (processingInfo != null) { 32 | StringTokenizer stk = new StringTokenizer(processingInfo, "|"); 33 | if (stk.hasMoreTokens() && stk != null) { 34 | totalEvents = stk.nextToken(); 35 | correctEventsinLast10secs = stk.nextToken(); 36 | missedEventsinLast10secs = stk.nextToken(); 37 | processingTimeforLast10secs = stk.nextToken(); 38 | System.out.println("totalEvents ="+totalEvents); 39 | } 40 | } 41 | } 42 | 43 | // keep only 4 elements in the key 44 | if (size > 4) { 45 | while (size != 4) { 46 | redisTemplate.opsForList().leftPop(key); 47 | size = size - 1; 48 | } 49 | } 50 | } 51 | 52 | public long totalOfEvents(){ 53 | System.out.println("totalOfEvents called:"); 54 | if (totalEvents == null) 55 | getData(); 56 | 57 | if (totalEvents == null) 58 | return 0; 59 | else { 60 | long totalEvts = Long.valueOf(totalEvents); 61 | // let it recompute if another call is made to totalOfEvents 62 | totalEvents = null; 63 | return totalEvts; 64 | } 65 | } 66 | 67 | public long totalMissedEvents(){ 68 | if (missedEventsinLast10secs == null) 69 | getData(); 70 | 71 | if (missedEventsinLast10secs == null) 72 | return 0; 73 | else { 74 | long missedEvents = Long.valueOf(missedEventsinLast10secs); 75 | // let it recompute if another call is made to totalMissedEvents 76 | missedEventsinLast10secs = null; 77 | return missedEvents; 78 | } 79 | } 80 | 81 | public long totalCorrectEvents(){ 82 | if (correctEventsinLast10secs == null) 83 | getData(); 84 | 85 | if(correctEventsinLast10secs == null) 86 | return 0; 87 | else { 88 | long correctEvents = Long.valueOf(correctEventsinLast10secs); 89 | // let it recompute if another call is made to totalCorrectEvents 90 | correctEventsinLast10secs = null; 91 | return correctEvents; 92 | } 93 | } 94 | 95 | /** 96 | * Gets the latest window processing time 97 | * @return 98 | */ 99 | public int latestWPT(){ 100 | if (processingTimeforLast10secs == null) 101 | getData(); 102 | 103 | if (processingTimeforLast10secs == null) 104 | return 0; 105 | else { 106 | int wpt = Integer.valueOf(processingTimeforLast10secs); 107 | // let it recompute if another call is made to latestWPT 108 | processingTimeforLast10secs = null; 109 | return wpt; 110 | } 111 | } 112 | 113 | public String test() { 114 | redisTemplate.opsForValue().set("LATEST_VALUE", String.valueOf(System.currentTimeMillis())); 115 | return redisTemplate.opsForValue().get("LATEST_VALUE"); 116 | } 117 | } 118 | -------------------------------------------------------------------------------- /eventservice/src/main/resources/application.yml: -------------------------------------------------------------------------------- 1 | spring: 2 | application: 3 | name: eventservice-iot-v1 4 | cloud: 5 | services: 6 | registrationMethod: route 7 | 8 | security: 9 | basic: 10 | enabled: false 11 | 12 | management: 13 | security: 14 | enabled: false 15 | 16 | --- 17 | 18 | spring: 19 | profiles: pcf 20 | 21 | redis: 22 | host: ${redis_host} 23 | port: ${redis_port} 24 | password: ${redis_password} 25 | 26 | eureka: 27 | client: 28 | serviceUrl: 29 | defaultZone: ${vcap.services.service-registry.credentials.uri:http://127.0.0.1:8761}/eureka/ 30 | instance: 31 | hostname: ${vcap.application.uris[0]} 32 | nonSecurePort: 80 33 | metadataMap: 34 | instanceId: ${vcap.application.instance_id:${spring.application.name}:${server.port:8080}} 35 | 36 | --- 37 | 38 | spring: 39 | profiles: local 40 | 41 | redis: 42 | host: localhost 43 | port: 6379 44 | password: changeme 45 | 46 | -------------------------------------------------------------------------------- /eventservice/src/main/resources/redis-bean.xml: -------------------------------------------------------------------------------- 1 | 2 | 6 | 7 | 11 | 12 | 14 | 15 | 17 | 18 | 19 | -------------------------------------------------------------------------------- /images/app_microservices.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kgshukla/Realtime-Streaming-DataMicroservices-AppMicroservices/a50eff54f489ebf1b006ae186a3959c56d4849f3/images/app_microservices.png -------------------------------------------------------------------------------- /images/circuit_breaker.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kgshukla/Realtime-Streaming-DataMicroservices-AppMicroservices/a50eff54f489ebf1b006ae186a3959c56d4849f3/images/circuit_breaker.pdf -------------------------------------------------------------------------------- /images/data_microservices.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kgshukla/Realtime-Streaming-DataMicroservices-AppMicroservices/a50eff54f489ebf1b006ae186a3959c56d4849f3/images/data_microservices.png -------------------------------------------------------------------------------- /images/problemstmt.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kgshukla/Realtime-Streaming-DataMicroservices-AppMicroservices/a50eff54f489ebf1b006ae186a3959c56d4849f3/images/problemstmt.jpg -------------------------------------------------------------------------------- /images/service_registry.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kgshukla/Realtime-Streaming-DataMicroservices-AppMicroservices/a50eff54f489ebf1b006ae186a3959c56d4849f3/images/service_registry.pdf -------------------------------------------------------------------------------- /images/solution_architecture_1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kgshukla/Realtime-Streaming-DataMicroservices-AppMicroservices/a50eff54f489ebf1b006ae186a3959c56d4849f3/images/solution_architecture_1.png -------------------------------------------------------------------------------- /images/stream_definition.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kgshukla/Realtime-Streaming-DataMicroservices-AppMicroservices/a50eff54f489ebf1b006ae186a3959c56d4849f3/images/stream_definition.png -------------------------------------------------------------------------------- /images/streaming_img1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kgshukla/Realtime-Streaming-DataMicroservices-AppMicroservices/a50eff54f489ebf1b006ae186a3959c56d4849f3/images/streaming_img1.png -------------------------------------------------------------------------------- /images/streaming_img2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kgshukla/Realtime-Streaming-DataMicroservices-AppMicroservices/a50eff54f489ebf1b006ae186a3959c56d4849f3/images/streaming_img2.png -------------------------------------------------------------------------------- /images/streaming_img3.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kgshukla/Realtime-Streaming-DataMicroservices-AppMicroservices/a50eff54f489ebf1b006ae186a3959c56d4849f3/images/streaming_img3.png -------------------------------------------------------------------------------- /manifest-aggservice.yml: -------------------------------------------------------------------------------- 1 | --- 2 | instances: 1 3 | memory: 400M 4 | applications: 5 | - name: aggservice-iot 6 | host: aggservice-iot 7 | path: aggregatorservice/target/aggregatorservice-1.0-SNAPSHOT.jar 8 | services: 9 | - service-registry 10 | - circuit-breaker 11 | env: 12 | SPRING_PROFILES_ACTIVE: pcf 13 | CF_TARGET: https://api.local.pcfdev.io 14 | -------------------------------------------------------------------------------- /manifest-eventservice.yml: -------------------------------------------------------------------------------- 1 | --- 2 | instances: 1 3 | memory: 400M 4 | applications: 5 | - name: eventservice-iot 6 | host: eventservice-iot 7 | path: eventservice/target/eventservice-1.0-SNAPSHOT.jar 8 | services: 9 | - config-server 10 | - service-registry 11 | env: 12 | SPRING_PROFILES_ACTIVE: pcf 13 | CF_TARGET: https://api.local.pcfdev.io 14 | -------------------------------------------------------------------------------- /manifest-scdf.yml: -------------------------------------------------------------------------------- 1 | --- 2 | instances: 1 3 | memory: 2048M 4 | applications: 5 | - name: dataflow-server 6 | host: dataflow-server 7 | services: 8 | - redis 9 | - rabbit 10 | env: 11 | SPRING_CLOUD_DEPLOYER_CLOUDFOUNDRY_URL: https://api.local.pcfdev.io 12 | SPRING_CLOUD_DEPLOYER_CLOUDFOUNDRY_ORG: pcfdev-org 13 | SPRING_CLOUD_DEPLOYER_CLOUDFOUNDRY_SPACE: SCDF 14 | SPRING_CLOUD_DEPLOYER_CLOUDFOUNDRY_DOMAIN: local.pcfdev.io 15 | SPRING_CLOUD_DEPLOYER_CLOUDFOUNDRY_USERNAME: admin 16 | SPRING_CLOUD_DEPLOYER_CLOUDFOUNDRY_PASSWORD: admin 17 | SPRING_CLOUD_DEPLOYER_CLOUDFOUNDRY_SKIP_SSL_VALIDATION: true 18 | SPRING_CLOUD_DEPLOYER_CLOUDFOUNDRY_STREAM_SERVICES: rabbit 19 | MAVEN_REMOTE_REPOSITORIES_REPO1_URL: https://repo.spring.io/libs-snapshot 20 | SPRING_CLOUD_DEPLOYER_CLOUDFOUNDRY_DISK: 512 21 | SPRING_CLOUD_DEPLOYER_CLOUDFOUNDRY_STREAM_BUILDPACK: java_buildpack 22 | spring.cloud.deployer.cloudfoundry.stream.memory: 400 23 | spring.cloud.dataflow.features.tasks-enabled: true 24 | spring.cloud.dataflow.features.streams-enabled: true 25 | -------------------------------------------------------------------------------- /manifest-taxiservice.yml: -------------------------------------------------------------------------------- 1 | --- 2 | instances: 1 3 | memory: 400M 4 | applications: 5 | - name: taxiservice-iot 6 | host: taxiservice-iot 7 | path: taxiservice/target/taxiservice-1.0-SNAPSHOT.jar 8 | services: 9 | - config-server 10 | - service-registry 11 | env: 12 | SPRING_PROFILES_ACTIVE: pcf 13 | CF_TARGET: https://api.local.pcfdev.io 14 | -------------------------------------------------------------------------------- /pom.xml: -------------------------------------------------------------------------------- 1 | 5 | 4.0.0 6 | 7 | io.pivotal.data 8 | taxi-iot 9 | pom 10 | 1.0.0-SNAPSHOT 11 | Taxi IOT Cloud-Native Application 12 | 13 | 14 | eventservice 15 | taxiservice 16 | aggregatorservice 17 | rxjava-taxi-processor 18 | 19 | 20 | 21 | org.springframework.cloud 22 | spring-cloud-starter-parent 23 | 1.0.1.RELEASE 24 | 25 | 26 | 27 | 28 | 29 | UTF-8 30 | 1.8 31 | 32 | 33 | 34 | 35 | org.springframework.boot 36 | spring-boot-starter-web 37 | 38 | 39 | org.springframework.boot 40 | spring-boot-starter-actuator 41 | 42 | 43 | org.springframework.boot 44 | spring-boot-starter-test 45 | test 46 | 47 | 48 | 49 | 50 | 51 | 52 | org.springframework.boot 53 | spring-boot-maven-plugin 54 | 55 | 56 | 57 | 58 | -------------------------------------------------------------------------------- /rxjava-taxi-processor/pom.xml: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4.0.0 4 | 5 | spring-cloud-stream-rxjava-taxi-rabbit 6 | jar 7 | 8 | spring-cloud-stream-rxjava-taxi 9 | Demo project for RxJava module 10 | 11 | 12 | org.springframework.cloud 13 | spring-cloud-build 14 | 1.1.1.RELEASE 15 | 16 | 17 | 18 | UTF-8 19 | demo.RxJavaApplication 20 | 1.0.2.RELEASE 21 | 1.8 22 | 23 | 24 | 25 | 26 | org.springframework.cloud 27 | spring-cloud-stream-rxjava 28 | ${spring-cloud-stream.version} 29 | 30 | 31 | org.springframework.cloud 32 | spring-cloud-stream-binder-rabbit 33 | ${spring-cloud-stream.version} 34 | 35 | 36 | org.springframework.boot 37 | spring-boot-starter-redis 38 | 39 | 40 | org.springframework.boot 41 | spring-boot-configuration-processor 42 | true 43 | 44 | 45 | 46 | org.springframework.boot 47 | spring-boot-starter-test 48 | test 49 | 50 | 51 | 52 | 53 | 54 | 55 | org.springframework.boot 56 | spring-boot-maven-plugin 57 | 58 | exec 59 | 60 | 61 | 62 | 63 | 64 | 65 | 66 | org.eclipse.m2e 67 | lifecycle-mapping 68 | 1.0.0 69 | 70 | 71 | 72 | 73 | 74 | 75 | org.codehaus.mojo 76 | 77 | 78 | exec-maven-plugin 79 | 80 | 81 | [1.5.0,) 82 | 83 | 84 | exec 85 | 86 | 87 | 88 | 89 | 90 | 91 | 92 | 93 | 94 | 95 | 96 | 97 | 98 | 99 | 100 | -------------------------------------------------------------------------------- /rxjava-taxi-processor/src/main/java/demo/RxJavaApplication.java: -------------------------------------------------------------------------------- 1 | /* 2 | * Copyright 2015 the original author or authors. 3 | * 4 | * Licensed under the Apache License, Version 2.0 (the "License"); 5 | * you may not use this file except in compliance with the License. 6 | * You may obtain a copy of the License at 7 | * 8 | * http://www.apache.org/licenses/LICENSE-2.0 9 | * 10 | * Unless required by applicable law or agreed to in writing, software 11 | * distributed under the License is distributed on an "AS IS" BASIS, 12 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 13 | * See the License for the specific language governing permissions and 14 | * limitations under the License. 15 | */ 16 | 17 | package demo; 18 | 19 | import org.springframework.boot.SpringApplication; 20 | import org.springframework.boot.autoconfigure.SpringBootApplication; 21 | import org.springframework.context.annotation.ComponentScan; 22 | 23 | /** 24 | * @author Ilayaperumal Gopinathan 25 | */ 26 | @SpringBootApplication 27 | public class RxJavaApplication { 28 | 29 | public static void main(String[] args) { 30 | SpringApplication.run(RxJavaApplication.class, args); 31 | } 32 | 33 | } 34 | -------------------------------------------------------------------------------- /rxjava-taxi-processor/src/main/java/demo/RxJavaTransformer.java: -------------------------------------------------------------------------------- 1 | /* 2 | * Copyright 2015 the original author or authors. 3 | * 4 | * Licensed under the Apache License, Version 2.0 (the "License"); 5 | * you may not use this file except in compliance with the License. 6 | * You may obtain a copy of the License at 7 | * 8 | * http://www.apache.org/licenses/LICENSE-2.0 9 | * 10 | * Unless required by applicable law or agreed to in writing, software 11 | * distributed under the License is distributed on an "AS IS" BASIS, 12 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 13 | * See the License for the specific language governing permissions and 14 | * limitations under the License. 15 | */ 16 | 17 | package demo; 18 | 19 | import java.util.ArrayList; 20 | import java.util.Arrays; 21 | import java.util.Collections; 22 | import java.util.HashMap; 23 | import java.util.List; 24 | import java.util.Map; 25 | 26 | import org.slf4j.Logger; 27 | import org.slf4j.LoggerFactory; 28 | import org.springframework.cloud.stream.annotation.rxjava.EnableRxJavaProcessor; 29 | import org.springframework.cloud.stream.annotation.rxjava.RxJavaProcessor; 30 | import org.springframework.context.annotation.Bean; 31 | 32 | import demo.model.EventData; 33 | import demo.model.FreeTaxiData; 34 | import demo.model.EventDataCollection; 35 | 36 | import java.util.concurrent.TimeUnit; 37 | import com.fasterxml.jackson.databind.ObjectMapper; 38 | 39 | @EnableRxJavaProcessor 40 | public class RxJavaTransformer { 41 | 42 | private static Logger logger = LoggerFactory.getLogger(RxJavaTransformer.class); 43 | 44 | // this variable captures how many events have been processed thus far 45 | // this information should not be stored here and rather be stored outside in a cache 46 | // so that when we scale this app, all instances could get the data. 47 | private static int totalEventsProcessed = 0; 48 | 49 | @Bean 50 | public RxJavaProcessor processor() { 51 | return inputStream -> inputStream.map(data -> { 52 | return data; 53 | }).buffer(10, 10, TimeUnit.SECONDS).map(data -> processData(data)); 54 | 55 | } 56 | 57 | private static String processData(List data) { 58 | 59 | if (data == null || data.size() == 0) 60 | return ""; 61 | 62 | 63 | //start time to find out much time it takes to process data 64 | Long processingstarttime = System.currentTimeMillis(); 65 | 66 | // filter the stream based on number of parameters. It should be 17 67 | List filtereddata = filterData(data); 68 | 69 | // get the cell start and end information for each event and store it as EventData 70 | List transformedData = transformData(filtereddata); 71 | 72 | // get list of all free taxies 73 | List freeTaxiesList = get50FreeTaxies(transformedData); 74 | 75 | // reduced transformed data based on "startendid" key which denotes the starting square and ending square 76 | List eventsDataCollectionList = reducebyCount(transformedData); 77 | 78 | // Find top 10 areas where taxies are traveling the most 79 | List top10areas = getTop10Areas(eventsDataCollectionList); 80 | 81 | // Find data related to processing time 82 | List processingInfoList = getProcessingInfo (processingstarttime, eventsDataCollectionList, data); 83 | 84 | totalEventsProcessed = totalEventsProcessed + data.size(); 85 | 86 | // returned all required data as json 87 | ReturnedData obj = new ReturnedData(processingInfoList, freeTaxiesList, top10areas, totalEventsProcessed); 88 | ObjectMapper mapper = new ObjectMapper(); 89 | String jsonInString = ""; 90 | try { 91 | jsonInString = mapper.writeValueAsString(obj); 92 | } catch (Exception e) { e.printStackTrace(); } 93 | return jsonInString; 94 | } 95 | 96 | 97 | private static class ReturnedData { 98 | String processingInfoString = ""; 99 | String freetaxiesListString = ""; 100 | String top10routesString = ""; 101 | 102 | ReturnedData(List processingInfo, List freetaxiesList, List top10areas, int totalEventsProcessed) { 103 | processingInfoString = String.valueOf(totalEventsProcessed).concat("|"); 104 | for (String e : processingInfo) { 105 | processingInfoString = processingInfoString.concat(e).concat("|"); 106 | } 107 | 108 | for (FreeTaxiData e : freetaxiesList) { 109 | freetaxiesListString = freetaxiesListString.concat(e.toString()).concat("|"); 110 | } 111 | 112 | for (RouteData e : top10areas) { 113 | top10routesString = top10routesString.concat(e.toString()).concat("|"); 114 | } 115 | } 116 | 117 | public String getProcessingInfoString() { 118 | return processingInfoString; 119 | } 120 | 121 | public String getFreetaxiesListString() { 122 | return freetaxiesListString; 123 | } 124 | 125 | public String getTop10routesString() { 126 | return top10routesString; 127 | } 128 | } 129 | 130 | 131 | private static List getProcessingInfo(Long processingstarttime, 132 | List eventsDataCollectionList, List rawData) { 133 | List processingInfoList = new ArrayList(); 134 | Long currentTime = System.currentTimeMillis(); 135 | Long delayinms = currentTime - Long.valueOf(processingstarttime).longValue(); 136 | 137 | int totalProcessedEvents = 0; 138 | for (EventDataCollection edc : eventsDataCollectionList) { 139 | totalProcessedEvents = totalProcessedEvents + edc.getEventDataList().size(); 140 | } 141 | 142 | processingInfoList.add(String.valueOf(totalProcessedEvents)); 143 | processingInfoList.add(String.valueOf(rawData.size()-totalProcessedEvents)); 144 | processingInfoList.add(String.valueOf(delayinms)); 145 | return processingInfoList; 146 | } 147 | 148 | private static List getTop10Areas( 149 | List eventsDataCollectionList) { 150 | List top10areas = new ArrayList(); 151 | 152 | // minimum is 10 or eventsDataCollectionList size 153 | int j = (eventsDataCollectionList.size() < 10) ? eventsDataCollectionList.size() : 10; 154 | 155 | int collsize = eventsDataCollectionList.size(); 156 | 157 | for (int i = 1; i <= j; i++) { 158 | // since the list is ordered in ascending, we need to get last one first 159 | EventDataCollection edc = eventsDataCollectionList.get(collsize-i); 160 | RouteData rd = new RouteData(edc.getStartendid(), String.valueOf(edc.getEventDataList().size())); 161 | top10areas.add(rd); 162 | } 163 | return top10areas; 164 | } 165 | 166 | private static class RouteData { 167 | private String route; 168 | private String numTrips; 169 | 170 | public RouteData(String route, String numTrips) { 171 | this.route = route; 172 | this.numTrips = numTrips; 173 | } 174 | 175 | public String getRoute() { 176 | return this.route; 177 | } 178 | 179 | public String getNumTrips() { 180 | return this.numTrips; 181 | } 182 | 183 | public String toString() { 184 | ObjectMapper mapper = new ObjectMapper(); 185 | String returnedString = ""; 186 | try { 187 | returnedString = mapper.writeValueAsString(this); 188 | } catch (Exception e) { 189 | e.printStackTrace(); 190 | } 191 | return returnedString; 192 | } 193 | } 194 | 195 | private static List reducebyCount( 196 | List transformedData) { 197 | List eventDataCollection = new ArrayList (); 198 | Map mapData = new HashMap(); 199 | 200 | for (EventData event : transformedData) { 201 | String startEndId = event.getStartendId(); 202 | if (mapData.containsKey(startEndId)) { 203 | EventDataCollection edc = mapData.get(startEndId); 204 | edc.addEventData(event); 205 | } else { 206 | EventDataCollection edc = new EventDataCollection(startEndId); 207 | edc.addEventData(event); 208 | mapData.put(startEndId, edc); 209 | } 210 | } 211 | List reducedEventDataCollection = new ArrayList (mapData.values()); 212 | Collections.sort(reducedEventDataCollection); 213 | return reducedEventDataCollection; 214 | 215 | } 216 | 217 | private static List get50FreeTaxies(List transformedData) { 218 | List freetaxiesList = new ArrayList(); 219 | int i = 0; 220 | for(EventData entry : transformedData) { 221 | if (i<50) { 222 | freetaxiesList.add(new FreeTaxiData(entry)); 223 | i = i+1; 224 | } 225 | } 226 | 227 | return freetaxiesList; 228 | } 229 | 230 | private static List transformData(List filtereddata) { 231 | 232 | List transformedData = new ArrayList(); 233 | 234 | for (String d : filtereddata) { 235 | List entryList = Arrays.asList(d.split(",")); 236 | String medallion = entryList.get(0); 237 | String pickupTime = entryList.get(2); 238 | String dropoffTime = entryList.get(3); 239 | String tripDistance = entryList.get(5); 240 | String pickuplong = entryList.get(6); 241 | String pickuplat = entryList.get(7); 242 | String dropofflong = entryList.get(8); 243 | String dropofflat = entryList.get(9); 244 | String startCellid = getCellId(pickuplong, pickuplat); 245 | String destCellid = getCellId(dropofflong, dropofflat); 246 | //String cellId = "wrong"; 247 | if ((null != startCellid) && (null != destCellid)) { 248 | String cellId = startCellid.concat("_to_").concat(destCellid); 249 | EventData eventData = new EventData(medallion, pickupTime, dropoffTime, tripDistance, 250 | pickuplong, pickuplat, dropofflong, dropofflat, startCellid, destCellid); 251 | eventData.setProcessingTimeStart(String.valueOf(System.currentTimeMillis())); 252 | transformedData.add(eventData); 253 | } 254 | } 255 | return transformedData; 256 | } 257 | 258 | private static List filterData(List data) { 259 | List filteredList = new ArrayList(); 260 | List wrongDataList = new ArrayList(); 261 | 262 | for (String d : data) { 263 | List entryList = Arrays.asList(d.split(",")); 264 | if(entryList.size() == 17) { 265 | // check for amount integrity - TODO 266 | filteredList.add(d); 267 | } 268 | else { 269 | logger.info("Wrong event : " + d); 270 | wrongDataList.add(d); 271 | //accuCount.add(1); 272 | //totalStreamCount.add(1); 273 | } 274 | } 275 | return filteredList; 276 | } 277 | 278 | private static String getCellId(String pickuplong, String pickuplat) { 279 | double initialLong = Double.valueOf("-74.913585"); 280 | double initialLat = Double.valueOf("41.474937"); 281 | 282 | int horizontalDist = distance(initialLat, initialLong, initialLat, Double.valueOf(pickuplong)); 283 | if(horizontalDist < 0 || horizontalDist > 300) { 284 | // cannot be negative 285 | return null; 286 | } 287 | 288 | int verticalDist = distance(initialLat, initialLong, Double.valueOf(pickuplat), initialLong); 289 | 290 | if(verticalDist < 0 || verticalDist > 300) { 291 | return null; 292 | } 293 | return "c:".concat(String.valueOf(horizontalDist)).concat(".").concat(String.valueOf(verticalDist)); 294 | } 295 | 296 | private static int distance(double lat1, double lon1, double lat2, double lon2) { 297 | double theta = lon1 - lon2; 298 | double dist = Math.sin(deg2rad(lat1)) * Math.sin(deg2rad(lat2)) + Math.cos(deg2rad(lat1)) * Math.cos(deg2rad(lat2)) * Math.cos(deg2rad(theta)); 299 | dist = Math.acos(dist); 300 | dist = rad2deg(dist); 301 | dist = dist * 60 * 1.1515; 302 | // distance in meters 303 | dist = dist * 1.609344 * 1000; 304 | return (Double.valueOf(dist/1000).intValue()+1); 305 | } 306 | 307 | private static double deg2rad(double deg) { 308 | return (deg * Math.PI / 180.0); 309 | } 310 | 311 | private static double rad2deg(double rad) { 312 | return (rad * 180 / Math.PI); 313 | } 314 | 315 | 316 | private static String combine(List data) { 317 | String result = ""; 318 | for (String d : data) { 319 | result = result.concat(":").concat(d); 320 | } 321 | return result; 322 | } 323 | 324 | private static Double avg(List data) { 325 | double sum = 0; 326 | double count = 0; 327 | for(String d : data) { 328 | count++; 329 | sum += Double.valueOf(d); 330 | } 331 | return sum/count; 332 | } 333 | 334 | } 335 | -------------------------------------------------------------------------------- /rxjava-taxi-processor/src/main/java/demo/model/EventData.java: -------------------------------------------------------------------------------- 1 | package demo.model; 2 | 3 | public class EventData { 4 | 5 | String medallion; 6 | String pickupTime; 7 | String dropoffTime; 8 | String tripDistance; 9 | String pickuplong; 10 | String pickuplat; 11 | String dropofflong; 12 | String dropofflat; 13 | String startCellid; 14 | String destCellid; 15 | String processingTimeStart; 16 | String startendId; 17 | 18 | public EventData(String medallion, String pickupTime, String dropoffTime, 19 | String tripDistance, String pickuplong, String pickuplat, 20 | String dropofflong, String dropofflat, String startCellid, 21 | String destCellid) { 22 | super(); 23 | this.medallion = medallion; 24 | this.pickupTime = pickupTime; 25 | this.dropoffTime = dropoffTime; 26 | this.tripDistance = tripDistance; 27 | this.pickuplong = pickuplong; 28 | this.pickuplat = pickuplat; 29 | this.dropofflong = dropofflong; 30 | this.dropofflat = dropofflat; 31 | this.startCellid = startCellid; 32 | this.destCellid = destCellid; 33 | this.startendId = startCellid.concat("_to_").concat(destCellid); 34 | } 35 | 36 | 37 | 38 | public String getStartendId() { 39 | return startendId; 40 | } 41 | 42 | 43 | 44 | public String getProcessingTimeStart() { 45 | return processingTimeStart; 46 | } 47 | 48 | public void setProcessingTimeStart(String processingTimeStart) { 49 | this.processingTimeStart = processingTimeStart; 50 | } 51 | 52 | public String getMedallion() { 53 | return medallion; 54 | } 55 | 56 | public void setMedallion(String medallion) { 57 | this.medallion = medallion; 58 | } 59 | 60 | public String getPickupTime() { 61 | return pickupTime; 62 | } 63 | 64 | public void setPickupTime(String pickupTime) { 65 | this.pickupTime = pickupTime; 66 | } 67 | 68 | public String getDropoffTime() { 69 | return dropoffTime; 70 | } 71 | 72 | public void setDropoffTime(String dropoffTime) { 73 | this.dropoffTime = dropoffTime; 74 | } 75 | 76 | public String getTripDistance() { 77 | return tripDistance; 78 | } 79 | 80 | public void setTripDistance(String tripDistance) { 81 | this.tripDistance = tripDistance; 82 | } 83 | 84 | public String getPickuplong() { 85 | return pickuplong; 86 | } 87 | 88 | public void setPickuplong(String pickuplong) { 89 | this.pickuplong = pickuplong; 90 | } 91 | 92 | public String getPickuplat() { 93 | return pickuplat; 94 | } 95 | 96 | public void setPickuplat(String pickuplat) { 97 | this.pickuplat = pickuplat; 98 | } 99 | 100 | public String getDropofflong() { 101 | return dropofflong; 102 | } 103 | 104 | public void setDropofflong(String dropofflong) { 105 | this.dropofflong = dropofflong; 106 | } 107 | 108 | public String getDropofflat() { 109 | return dropofflat; 110 | } 111 | 112 | public void setDropofflat(String dropofflat) { 113 | this.dropofflat = dropofflat; 114 | } 115 | 116 | public String getStartCellid() { 117 | return startCellid; 118 | } 119 | 120 | public void setStartCellid(String startCellid) { 121 | this.startCellid = startCellid; 122 | } 123 | 124 | public String getDestCellid() { 125 | return destCellid; 126 | } 127 | 128 | public void setDestCellid(String destCellid) { 129 | this.destCellid = destCellid; 130 | } 131 | 132 | } 133 | -------------------------------------------------------------------------------- /rxjava-taxi-processor/src/main/java/demo/model/EventDataCollection.java: -------------------------------------------------------------------------------- 1 | package demo.model; 2 | 3 | import java.util.ArrayList; 4 | import java.util.Comparator; 5 | import java.util.List; 6 | 7 | public class EventDataCollection implements Comparable { 8 | 9 | String startendid; 10 | List eventdatalist; 11 | 12 | public EventDataCollection (String startendid) { 13 | this.startendid = startendid; 14 | eventdatalist = new ArrayList(); 15 | } 16 | 17 | public void addEventData(EventData eventdata) { 18 | this.eventdatalist.add(eventdata); 19 | } 20 | 21 | public List getEventDataList() { 22 | return eventdatalist; 23 | } 24 | 25 | public String getStartendid() { 26 | return startendid; 27 | } 28 | 29 | @Override 30 | public int compareTo(EventDataCollection eventDataCollection2) { 31 | return Comparators.DEFAULT.compare(this.eventdatalist, eventDataCollection2.getEventDataList()); 32 | } 33 | 34 | public static class Comparators { 35 | public static Comparator> DEFAULT = new Comparator>() { 36 | @Override 37 | public int compare(List o1, List o2) { 38 | if (o1.size() > o2.size()) 39 | return 1; 40 | else if (o1.size() < o2.size()) 41 | return -1; 42 | else 43 | return 0; 44 | } 45 | }; 46 | } 47 | } 48 | 49 | -------------------------------------------------------------------------------- /rxjava-taxi-processor/src/main/java/demo/model/FreeTaxiData.java: -------------------------------------------------------------------------------- 1 | package demo.model; 2 | 3 | public class FreeTaxiData { 4 | String medallion; 5 | String dropofflong; 6 | String dropofflat; 7 | 8 | public FreeTaxiData (EventData eventdata) { 9 | this.medallion = eventdata.getMedallion(); 10 | this.dropofflong = eventdata.getDropofflong(); 11 | this.dropofflat = eventdata.getDropofflat(); 12 | } 13 | 14 | public String toString() { 15 | return this.medallion+","+this.dropofflat+","+this.dropofflong; 16 | } 17 | 18 | public String getMedallion() { 19 | return medallion; 20 | } 21 | 22 | public String getDropofflong() { 23 | return dropofflong; 24 | } 25 | 26 | public String getDropofflat() { 27 | return dropofflat; 28 | } 29 | } 30 | -------------------------------------------------------------------------------- /rxjava-taxi-processor/src/main/resources/application.yml: -------------------------------------------------------------------------------- 1 | server: 2 | port: 8082 3 | spring: 4 | cloud: 5 | stream: 6 | bindings: 7 | output: 8 | destination: xformed 9 | input: 10 | destination: testtock 11 | -------------------------------------------------------------------------------- /rxjava-taxi-processor/src/test/java/demo/ModuleApplicationTests.java: -------------------------------------------------------------------------------- 1 | /* 2 | * Copyright 2015 the original author or authors. 3 | * 4 | * Licensed under the Apache License, Version 2.0 (the "License"); 5 | * you may not use this file except in compliance with the License. 6 | * You may obtain a copy of the License at 7 | * 8 | * http://www.apache.org/licenses/LICENSE-2.0 9 | * 10 | * Unless required by applicable law or agreed to in writing, software 11 | * distributed under the License is distributed on an "AS IS" BASIS, 12 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 13 | * See the License for the specific language governing permissions and 14 | * limitations under the License. 15 | */ 16 | 17 | package demo; 18 | 19 | import org.junit.Test; 20 | import org.junit.runner.RunWith; 21 | import org.springframework.test.annotation.DirtiesContext; 22 | import org.springframework.test.context.web.WebAppConfiguration; 23 | import org.springframework.boot.test.SpringApplicationConfiguration; 24 | import org.springframework.test.context.junit4.SpringJUnit4ClassRunner; 25 | 26 | @RunWith(SpringJUnit4ClassRunner.class) 27 | @SpringApplicationConfiguration(classes = RxJavaApplication.class) 28 | @WebAppConfiguration 29 | @DirtiesContext 30 | public class ModuleApplicationTests { 31 | 32 | @Test 33 | public void contextLoads() { 34 | } 35 | 36 | } 37 | -------------------------------------------------------------------------------- /rxjava-taxi-processor/staticdir/Staticfile: -------------------------------------------------------------------------------- 1 | directory: visible 2 | -------------------------------------------------------------------------------- /rxjava-taxi-processor/staticdir/manifest.yml: -------------------------------------------------------------------------------- 1 | applications: 2 | - name: rxjavataxi-jar 3 | memory: 300M 4 | instances: 1 5 | host: rxjavataxi-jar 6 | -------------------------------------------------------------------------------- /taxiservice/.classpath: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | -------------------------------------------------------------------------------- /taxiservice/.gitignore: -------------------------------------------------------------------------------- 1 | /target/ 2 | -------------------------------------------------------------------------------- /taxiservice/.project: -------------------------------------------------------------------------------- 1 | 2 | 3 | taxirouteservice 4 | 5 | 6 | 7 | 8 | 9 | org.eclipse.jdt.core.javabuilder 10 | 11 | 12 | 13 | 14 | org.eclipse.m2e.core.maven2Builder 15 | 16 | 17 | 18 | 19 | org.springframework.ide.eclipse.core.springbuilder 20 | 21 | 22 | 23 | 24 | 25 | org.springframework.ide.eclipse.core.springnature 26 | org.eclipse.jdt.core.javanature 27 | org.eclipse.m2e.core.maven2Nature 28 | 29 | 30 | -------------------------------------------------------------------------------- /taxiservice/.settings/org.eclipse.core.resources.prefs: -------------------------------------------------------------------------------- 1 | eclipse.preferences.version=1 2 | encoding//src/main/java=UTF-8 3 | encoding//src/main/resources=UTF-8 4 | encoding//src/test/java=UTF-8 5 | encoding//src/test/resources=UTF-8 6 | encoding/=UTF-8 7 | -------------------------------------------------------------------------------- /taxiservice/.settings/org.eclipse.jdt.core.prefs: -------------------------------------------------------------------------------- 1 | eclipse.preferences.version=1 2 | org.eclipse.jdt.core.compiler.codegen.targetPlatform=1.8 3 | org.eclipse.jdt.core.compiler.compliance=1.8 4 | org.eclipse.jdt.core.compiler.problem.forbiddenReference=warning 5 | org.eclipse.jdt.core.compiler.source=1.8 6 | -------------------------------------------------------------------------------- /taxiservice/.settings/org.eclipse.m2e.core.prefs: -------------------------------------------------------------------------------- 1 | activeProfiles= 2 | eclipse.preferences.version=1 3 | resolveWorkspaceProjects=true 4 | version=1 5 | -------------------------------------------------------------------------------- /taxiservice/pom.xml: -------------------------------------------------------------------------------- 1 | 2 | 4.0.0 3 | 4 | io.pivotal.spring.cloud 5 | spring-cloud-services-starter-parent 6 | 1.0.1.RELEASE 7 | 8 | 9 | taxiservice 10 | io.pivotal.data.taxi.iot 11 | 1.0-SNAPSHOT 12 | 13 | 14 | io.pivotal.spring.cloud 15 | spring-cloud-services-starter-config-client 16 | 17 | 18 | io.pivotal.spring.cloud 19 | spring-cloud-services-starter-service-registry 20 | 21 | 22 | org.springframework.boot 23 | spring-boot-starter-web 24 | 25 | 26 | org.springframework.boot 27 | spring-boot-starter-test 28 | 29 | 30 | org.springframework.boot 31 | spring-boot-starter-data-redis 32 | 1.4.1.RELEASE 33 | 34 | 35 | com.google.guava 36 | guava 37 | 19.0-rc1 38 | 39 | 40 | org.springframework.cloud 41 | spring-cloud-starter 42 | 1.0.1.RELEASE 43 | 44 | 45 | org.springframework.boot 46 | spring-boot-starter-actuator 47 | 48 | 49 | 50 | UTF-8 51 | 1.8 52 | 1.8 53 | 1.8 54 | 55 | 56 | 57 | 58 | 59 | 60 | org.springframework.boot 61 | spring-boot-maven-plugin 62 | 63 | 64 | 65 | 66 | 67 | 68 | spring-release 69 | https://repo.spring.io/libs-release 70 | 71 | 72 | 73 | 74 | -------------------------------------------------------------------------------- /taxiservice/src/main/java/io/pivotal/data/taxiservice/App.java: -------------------------------------------------------------------------------- 1 | /** 2 | * 3 | */ 4 | package io.pivotal.data.taxiservice; 5 | 6 | import org.springframework.boot.SpringApplication; 7 | import org.springframework.boot.autoconfigure.SpringBootApplication; 8 | import org.springframework.cloud.client.discovery.EnableDiscoveryClient; 9 | import org.springframework.context.annotation.Bean; 10 | import org.springframework.context.annotation.ImportResource; 11 | import org.springframework.context.annotation.Profile; 12 | import org.springframework.test.context.ActiveProfiles; 13 | 14 | 15 | /** 16 | * @author shuklk2 17 | * 18 | */ 19 | @SpringBootApplication 20 | @ActiveProfiles 21 | @EnableDiscoveryClient 22 | public class App { 23 | 24 | /** 25 | * @param args 26 | */ 27 | public static void main(String[] args) { 28 | SpringApplication.run(App.class, args); 29 | } 30 | 31 | } 32 | -------------------------------------------------------------------------------- /taxiservice/src/main/java/io/pivotal/data/taxiservice/controller/TaxiServiceController.java: -------------------------------------------------------------------------------- 1 | /** 2 | * 3 | */ 4 | package io.pivotal.data.taxiservice.controller; 5 | 6 | import java.util.List; 7 | 8 | import io.pivotal.data.taxiservice.models.RouteData; 9 | import io.pivotal.data.taxiservice.services.TaxiService; 10 | 11 | import org.springframework.beans.factory.annotation.Autowired; 12 | import org.springframework.http.MediaType; 13 | import org.springframework.web.bind.annotation.RequestMapping; 14 | import org.springframework.web.bind.annotation.RequestMethod; 15 | import org.springframework.web.bind.annotation.RestController; 16 | 17 | /** 18 | * @author shuklk2 19 | * 20 | */ 21 | 22 | @RestController 23 | public class TaxiServiceController { 24 | 25 | @Autowired 26 | private TaxiService service; 27 | 28 | @RequestMapping(value = "/routes/top10routes", method = RequestMethod.GET, produces = MediaType.APPLICATION_JSON_VALUE) 29 | public List> getTop10Routes() { 30 | return service.getTop10Routes(); 31 | } 32 | 33 | @RequestMapping(value = "/freetaxies/list", method = RequestMethod.GET, produces = MediaType.APPLICATION_JSON_VALUE) 34 | public List> getFreeTaxiesList() { 35 | return service.getFreeTaxiesList(); 36 | } 37 | 38 | @RequestMapping(value = "/kill", method = RequestMethod.GET) 39 | public String kill() { 40 | System.exit(1); 41 | return ""; 42 | } 43 | } 44 | -------------------------------------------------------------------------------- /taxiservice/src/main/java/io/pivotal/data/taxiservice/models/RouteData.java: -------------------------------------------------------------------------------- 1 | /** 2 | * 3 | */ 4 | package io.pivotal.data.taxiservice.models; 5 | 6 | /** 7 | * @author shuklk2 8 | * 9 | */ 10 | public class RouteData { 11 | 12 | private String route; 13 | private String numTrips; 14 | 15 | public RouteData() {} 16 | 17 | public RouteData (String route, String numTrips) { 18 | this.route = route; 19 | this.numTrips = numTrips; 20 | } 21 | 22 | public String getRoute() { 23 | return route; 24 | } 25 | 26 | public void setRoute(String route) { 27 | this.route = route; 28 | } 29 | 30 | public String getNumTrips() { 31 | return numTrips; 32 | } 33 | 34 | public void setNumTrips(String numTrips) { 35 | this.numTrips = numTrips; 36 | } 37 | } 38 | -------------------------------------------------------------------------------- /taxiservice/src/main/java/io/pivotal/data/taxiservice/services/TaxiService.java: -------------------------------------------------------------------------------- 1 | /** 2 | * 3 | */ 4 | package io.pivotal.data.taxiservice.services; 5 | 6 | import java.util.ArrayList; 7 | import java.util.List; 8 | import java.util.StringTokenizer; 9 | import com.fasterxml.jackson.databind.ObjectMapper; 10 | import com.fasterxml.jackson.core.type.TypeReference; 11 | import io.pivotal.data.taxiservice.models.RouteData; 12 | 13 | import org.springframework.data.redis.core.StringRedisTemplate; 14 | import org.springframework.beans.factory.annotation.Autowired; 15 | import org.springframework.stereotype.Component; 16 | 17 | /** 18 | * @author shuklk2 19 | * 20 | */ 21 | @Component 22 | public class TaxiService { 23 | 24 | @Autowired 25 | private StringRedisTemplate redisTemplate; 26 | private final String top10RoutesKey = "top10Routes_LATEST_DATA"; 27 | private final String freeTaxiesKey = "freeTaxies_LATEST_DATA"; 28 | 29 | public List> getTop10Routes() { 30 | List> rdlist = new ArrayList>(); 31 | 32 | long size = redisTemplate.opsForList().size(top10RoutesKey); 33 | if (size > 0) { 34 | String top10routes = redisTemplate.opsForList().index(top10RoutesKey, size-1); 35 | if (top10routes != null) { 36 | StringTokenizer stk = new StringTokenizer(top10routes, "|"); 37 | ObjectMapper mapper = new ObjectMapper(); 38 | if (stk != null) { 39 | while (stk.hasMoreTokens()) { 40 | String e = stk.nextToken(); 41 | System.out.println("data == "+e); 42 | RouteData rd = new RouteData(); 43 | try { 44 | rd = mapper.readValue(e, RouteData.class); 45 | System.out.println("Route: "+rd.getRoute()); 46 | System.out.println("NumTrips: "+rd.getNumTrips()); 47 | } catch (Exception ex) { 48 | ex.printStackTrace(); 49 | } 50 | 51 | //String route = list.get(0); 52 | //String numtrips = list.get(1); 53 | //StringTokenizer stk2 = new StringTokenizer(e, ","); 54 | //String route = stk2.nextToken(); 55 | //String numtrips = stk2.nextToken(); 56 | //List rd = new ArrayList(); 57 | //rd.add(route); 58 | //rd.add(numtrips); 59 | //RouteData rd = new RouteData(route, numtrips); 60 | List data = new ArrayList(); 61 | data.add(rd.getRoute()); 62 | data.add(rd.getNumTrips()); 63 | rdlist.add(data); 64 | } 65 | } 66 | } 67 | } 68 | 69 | // keep only 4 elements in the key 70 | if (size > 4) { 71 | while (size != 4) { 72 | redisTemplate.opsForList().leftPop(top10RoutesKey); 73 | size = size -1; 74 | } 75 | } 76 | return rdlist; 77 | } 78 | 79 | public List> getFreeTaxiesList() { 80 | List> freetaxiesList = new ArrayList>(); 81 | 82 | long size = redisTemplate.opsForList().size(freeTaxiesKey); 83 | if (size > 0) { 84 | String freetaxies = redisTemplate.opsForList().index(freeTaxiesKey, size -1); 85 | if (freetaxies != null) { 86 | StringTokenizer stk = new StringTokenizer(freetaxies, "|"); 87 | while (stk.hasMoreTokens()) { 88 | String e = stk.nextToken(); 89 | StringTokenizer stk2 = new StringTokenizer(e, ","); 90 | String medallion = stk2.nextToken(); 91 | String dropofflat = stk2.nextToken(); 92 | String dropofflong = stk2.nextToken(); 93 | List freetaxi = new ArrayList(); 94 | freetaxi.add(medallion); 95 | freetaxi.add(dropofflat); 96 | freetaxi.add(dropofflong); 97 | freetaxiesList.add(freetaxi); 98 | } 99 | } 100 | } 101 | 102 | // keep only 4 elements in the key 103 | if (size > 4) { 104 | while (size != 4) { 105 | redisTemplate.opsForList().leftPop(freeTaxiesKey); 106 | size = size -1; 107 | } 108 | } 109 | return freetaxiesList; 110 | } 111 | } 112 | -------------------------------------------------------------------------------- /taxiservice/src/main/resources/application.yml: -------------------------------------------------------------------------------- 1 | spring: 2 | application: 3 | name: taxiservice-iot-v1 4 | 5 | security: 6 | basic: 7 | enabled: false 8 | 9 | management: 10 | security: 11 | enabled: false 12 | 13 | --- 14 | 15 | spring: 16 | profiles: pcf 17 | redis: 18 | host: ${redis_host} 19 | port: ${redis_port} 20 | password: ${redis_password} 21 | 22 | 23 | 24 | eureka: 25 | client: 26 | serviceUrl: 27 | defaultZone: ${vcap.services.service-registry.credentials.uri:http://127.0.0.1:8761}/eureka/ 28 | instance: 29 | hostname: ${vcap.application.uris[0]} 30 | nonSecurePort: 80 31 | metadataMap: 32 | instanceId: ${vcap.application.instance_id:${spring.application.name}:${server.port:8080}} 33 | 34 | --- 35 | 36 | spring: 37 | profiles: local 38 | redis: 39 | host: localhost 40 | port: 6379 41 | password: changeme 42 | 43 | -------------------------------------------------------------------------------- /taxiservice/src/main/resources/redis-bean.xml: -------------------------------------------------------------------------------- 1 | 2 | 6 | 7 | 11 | 12 | 14 | 15 | 17 | 18 | 19 | -------------------------------------------------------------------------------- /webapp_php/.bp-config/options.json: -------------------------------------------------------------------------------- 1 | { 2 | "PHP_EXTENSIONS": ["mbstring", "curl", "pgsql"] 3 | } 4 | -------------------------------------------------------------------------------- /webapp_php/Pivotal-Logo.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kgshukla/Realtime-Streaming-DataMicroservices-AppMicroservices/a50eff54f489ebf1b006ae186a3959c56d4849f3/webapp_php/Pivotal-Logo.jpg -------------------------------------------------------------------------------- /webapp_php/app_microservices.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kgshukla/Realtime-Streaming-DataMicroservices-AppMicroservices/a50eff54f489ebf1b006ae186a3959c56d4849f3/webapp_php/app_microservices.png -------------------------------------------------------------------------------- /webapp_php/data_microservices.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kgshukla/Realtime-Streaming-DataMicroservices-AppMicroservices/a50eff54f489ebf1b006ae186a3959c56d4849f3/webapp_php/data_microservices.png -------------------------------------------------------------------------------- /webapp_php/freetaxiesandroutes.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 72 | 73 | 79 | 80 | 81 | 82 | 83 | 84 | 85 | 86 | 89 | 96 | 97 |
87 | 88 | 90 |
91 |
92 |

Real Time Streaming Demo

93 |
94 |
95 |
98 | 99 |
100 |
101 |
102 |
103 |
104 |
105 |
106 | This demo showcases power of PCF, Spring Cloud Dataflow, RxJava, Spring Cloud Streams and Redis. 107 |
    108 |
  • Spring Cloud Dataflow receives a stream of taxi data. Sample data can be viewed here.
    • 109 |
  • Goal of the demo is to quickly evaluate top 10 busiest area based on taxi pickup and dropoff information received from streams.
    • 110 |
  • Each area is a square 1kmx1km.
    • 111 |
  • Data needs to be refreshed every 10 seconds.
    • 112 |
  • Data is not clean. Drivers could give incorrect information like wrong fare, forgot to enter pickup location, wrong latitude/longitude etc
    • 113 |
114 | Below you could see number of streams being processed, top 10 areas and details. ALL IN REAL TIME. 115 | 116 | 117 |
118 |
119 |
120 | 121 | 122 | 125 | 128 | 131 | 132 |
123 | 124 | 126 | 127 | 129 | 130 |
133 | 134 | 135 |
136 |
137 |
138 | Free Taxies and their Location (only top 50) 139 |
140 | 141 | 142 | 145 | 146 |
143 |
144 |
147 |
148 |
149 | Top 10 Areas where taxies are running 150 | 151 | 152 | 155 | 156 |
153 |
154 |
157 |
158 |
159 | 160 | 161 | -------------------------------------------------------------------------------- /webapp_php/getTaxiData.php: -------------------------------------------------------------------------------- 1 | {'user-provided'}[0]->{'credentials'}->{'AGGSERVICE_URL'}; 5 | 6 | if (isset($_POST["type"])) { 7 | if ($_POST["type"] == "TABLE") { 8 | 9 | $ch = curl_init(); 10 | curl_setopt($ch, CURLOPT_URL, $aggservice_url."/top10routes"); 11 | curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); 12 | $output = curl_exec($ch); 13 | curl_close($ch); 14 | 15 | $table = array(); 16 | $table['cols'] = array( 17 | array('id' => "", 'label' => 'Route', 'pattern' => "", 'type' => 'string'), 18 | array('id' => "", 'label' => '#Trips', 'pattern' => "", 'type' => 'string') 19 | #array('id' => "", 'label' => 'Pickup Time', 'pattern' => "", 'type' => 'string'), 20 | #array('id' => "", 'label' => 'Dropoff Time', 'pattern' => "", 'type' => 'string') 21 | ); 22 | 23 | $rows = array(); 24 | foreach(json_decode($output) as $nt) { 25 | $temp = array(); 26 | $temp[] = array('v' => $nt[0], 'f' =>NULL); 27 | $temp[] = array('v' => $nt[1], 'f' =>NULL); 28 | #$temp[] = array('v' => $nt[2], 'f' =>NULL); 29 | #$temp[] = array('v' => $nt[3], 'f' =>NULL); 30 | $rows[] = array('c' => $temp); 31 | } 32 | 33 | $table['rows']=$rows; 34 | syslog(LOG_WARNING, "data"); 35 | echo json_encode($table); 36 | } else if ($_POST["type"] == "PROCESSTABLE") { 37 | $ch = curl_init(); 38 | curl_setopt($ch, CURLOPT_URL, $gemfire_url."ProcessData/LATEST_DATA"); 39 | curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); 40 | $output1 = curl_exec($ch); 41 | curl_close($ch); 42 | 43 | $table1 = array(); 44 | $table1['cols'] = array( 45 | array('id' => "", 'label' => '#Events Processed', 'pattern' => "", 'type' => 'string'), 46 | array('id' => "", 'label' => '#Events Lacking Data', 'pattern' => "", 'type' => 'string'), 47 | array('id' => "", 'label' => 'Per Batch Spark Processing Time (ms)', 'pattern' => "", 'type' => 'string'), 48 | ); 49 | 50 | $rows1 = array(); 51 | $decodedOutput = json_decode($output1); 52 | $temp1 = array(); 53 | $temp1[] = array('v' => $decodedOutput[0], 'f' => NULL); 54 | $temp1[] = array('v' => $decodedOutput[1], 'f' => NULL); 55 | $temp1[] = array('v' => $decodedOutput[2], 'f' => NULL); 56 | $rows1[] = array('c' => $temp1); 57 | 58 | $table1['rows']=$rows1; 59 | 60 | echo json_encode($table1); 61 | } else if ($_POST["type"] == "STREAMSCHART") { 62 | $ch = curl_init(); 63 | curl_setopt($ch, CURLOPT_URL, $aggservice_url."/totalevents"); 64 | curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); 65 | $output1 = curl_exec($ch); 66 | curl_setopt($ch, CURLOPT_URL, $aggservice_url."/missedevents"); 67 | curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); 68 | $output2 = curl_exec($ch); 69 | curl_close($ch); 70 | 71 | $table1 = array(); 72 | $table1['cols'] = array( 73 | array('id' => "", 'label' => 'Time', 'pattern' => "", 'type' => 'string'), 74 | array('id' => "", 'label' => '#Events Processed', 'pattern' => "", 'type' => 'number'), 75 | array('id' => "", 'label' => '#Events Lacking Data', 'pattern' => "", 'type' => 'number') 76 | ); 77 | 78 | $rows1 = array(); 79 | $temp1 = array(); 80 | $temp1[] = array('v' => date("G:i:s"), 'f' => NULL); 81 | $temp1[] = array('v' => (int)$output1, 'f' => NULL); 82 | $temp1[] = array('v' => (int)$output2, 'f' => NULL); 83 | $rows1[] = array('c' => $temp1); 84 | 85 | $table1['rows']=$rows1; 86 | 87 | echo json_encode($table1); 88 | } else if ($_POST["type"] == "SPARKPROCCHART") { 89 | $ch = curl_init(); 90 | curl_setopt($ch, CURLOPT_URL, $aggservice_url."/processtime"); 91 | curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); 92 | $output1 = curl_exec($ch); 93 | curl_close($ch); 94 | 95 | $table1 = array(); 96 | $table1['cols'] = array( 97 | array('id' => "", 'label' => 'Time', 'pattern' => "", 'type' => 'string'), 98 | array('id' => "", 'label' => 'Spark Processing Time (ms)', 'pattern' => "", 'type' => 'number'), 99 | ); 100 | 101 | $rows1 = array(); 102 | $temp1 = array(); 103 | $temp1[] = array('v' => date("G:i:s"), 'f' => NULL); 104 | $temp1[] = array('v' => (int)$output1, 'f' => NULL); 105 | $rows1[] = array('c' => $temp1); 106 | 107 | $table1['rows']=$rows1; 108 | 109 | echo json_encode($table1); 110 | } else if ($_POST["type"] == "DATAQUALITY") { 111 | $ch = curl_init(); 112 | curl_setopt($ch, CURLOPT_URL, $aggservice_url."/correctevents"); 113 | curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); 114 | $output1 = curl_exec($ch); 115 | curl_close($ch); 116 | 117 | $ch2 = curl_init(); 118 | curl_setopt($ch2, CURLOPT_URL, $aggservice_url."/missedevents"); 119 | curl_setopt($ch2, CURLOPT_RETURNTRANSFER, 1); 120 | $output2 = curl_exec($ch2); 121 | curl_close($ch2); 122 | 123 | $table1 = array(); 124 | $table1['cols'] = array( 125 | array('id' => "", 'label' => 'Time', 'pattern' => "", 'type' => 'string'), 126 | array('id' => "", 'label' => 'Correct Data', 'pattern' => "", 'type' => 'number'), 127 | array('id' => "", 'label' => 'Missed Data', 'pattern' => "", 'type' => 'number'), 128 | ); 129 | 130 | $rows1 = array(); 131 | $temp1 = array(); 132 | $temp1[] = array('v' => date("G:i:s"), 'f' => NULL); 133 | $temp1[] = array('v' => (int)$output1, 'f' => NULL); 134 | $temp1[] = array('v' => (int)$output2, 'f' => NULL); 135 | $rows1[] = array('c' => $temp1); 136 | 137 | $table1['rows']=$rows1; 138 | 139 | echo json_encode($table1); 140 | } 141 | else if ($_POST["type"] == "MAPDATA") { 142 | $ch = curl_init(); 143 | curl_setopt($ch, CURLOPT_URL, $gemfire_url."/RouteData/LATEST_DATA?limit=6"); 144 | curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); 145 | $output2 = curl_exec($ch); 146 | curl_close($ch); 147 | 148 | $table2 = array(); 149 | $table2['cols'] = array( 150 | array('id' => "", 'label' => 'Lat', 'pattern' => "", 'type' => 'number'), 151 | array('id' => "", 'label' => 'Long', 'pattern' => "", 'type' => 'number'), 152 | array('id' => "", 'label' => 'Location', 'pattern' => "", 'type' => 'string'), 153 | array('id' => "", 'label' => 'Marker', 'pattern' => "", 'type' => 'string'), 154 | ); 155 | $rows2 = array(); 156 | $color = array ('blue1', 'blue2', 'green1', 'green2', 'pink1', 'pink2'); 157 | $i = 0; 158 | foreach(json_decode($output2) as $nt2) { 159 | if($i < 6) { 160 | $temp2 = array(); 161 | $temp2[] = array('v' => $nt2[0], 'f' =>NULL); 162 | $temp2[] = array('v' => $nt2[1], 'f' =>NULL); 163 | $temp2[] = array('v' => $nt2[2], 'f' =>NULL); 164 | $temp2[] = array('v' => $color[$i], 'f' =>NULL); 165 | $rows2[] = array('c' => $temp2); 166 | $i = $i+1; 167 | } 168 | } 169 | $table2['rows']=$rows2; 170 | echo json_encode($table2); 171 | } else if ($_POST["type"] == "FREETAXIDATA") { 172 | $ch = curl_init(); 173 | curl_setopt($ch, CURLOPT_URL, $aggservice_url."/freetaxies"); 174 | curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); 175 | $output3 = curl_exec($ch); 176 | curl_close($ch); 177 | 178 | $table3 = array(); 179 | $table3['cols'] = array( 180 | array('id' => "", 'label' => 'Lat', 'pattern' => "", 'type' => 'number'), 181 | array('id' => "", 'label' => 'Long', 'pattern' => "", 'type' => 'number'), 182 | array('id' => "", 'label' => 'Taxi Number', 'pattern' => "", 'type' => 'string') 183 | ); 184 | $rows3 = array(); 185 | foreach(json_decode($output3) as $nt3) { 186 | $temp3 = array(); 187 | $temp3[] = array('v' => $nt3[1], 'f' =>NULL); 188 | $temp3[] = array('v' => $nt3[2], 'f' =>NULL); 189 | $temp3[] = array('v' => $nt3[0], 'f' =>NULL); 190 | $rows3[] = array('c' => $temp3); 191 | } 192 | $table3['rows']=$rows3; 193 | echo json_encode($table3); 194 | } else { 195 | echo ""; 196 | } 197 | } 198 | ?> 199 | -------------------------------------------------------------------------------- /webapp_php/index.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 12 | 19 | 20 |
10 | 11 | 13 |
14 |
15 |

Real Time Streaming Demo

16 |
17 |
18 |
21 | 22 |
23 |
24 |
25 |
26 |
27 |
28 |
29 | This demo showcases power of PCF, Spring Cloud Dataflow, Spring Cloud Service and Redis. 30 |
    31 |
  • Spring Cloud Dataflow receives a stream of taxi data. Sample data can be viewed here.
    • 32 |
  • Goal of the demo is to quickly evaluate top 10 busiest area based on taxi pickup and dropoff information received from streams.
    • 33 |
  • Each area is a square 1kmx1km.
    • 34 |
  • Data needs to be refreshed every 10 seconds.
    • 35 |
  • Data is not clean. Drivers could give incorrect information like wrong fare, forgot to enter pickup location, wrong latitude/longitude etc
    • 36 |
37 | Below you could see number of streams being processed, top 10 areas and details. ALL IN REAL TIME. 38 | 39 | 40 |
41 |
42 |
43 | 44 | 45 | 48 | 51 | 54 | 55 |
46 | 47 | 49 | 50 | 52 | 53 |
56 |
57 |
58 | 59 |
60 |
61 |
62 | Complete Solution Architecture 63 |
64 | 65 |
66 |
67 |
68 | Solution Architecture for Data Microservices 69 |
70 | 71 |
72 |
73 |
74 | Solution Architecture for Application Microservices 75 |
76 | 77 |
78 |
79 | 80 | 81 | -------------------------------------------------------------------------------- /webapp_php/manifest.yml: -------------------------------------------------------------------------------- 1 | --- 2 | instances: 1 3 | memory: 356M 4 | applications: 5 | - name: iot-taxi 6 | host: iot-taxi 7 | services: 8 | - agg_service_iot 9 | -------------------------------------------------------------------------------- /webapp_php/problemstmt.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kgshukla/Realtime-Streaming-DataMicroservices-AppMicroservices/a50eff54f489ebf1b006ae186a3959c56d4849f3/webapp_php/problemstmt.jpg -------------------------------------------------------------------------------- /webapp_php/sampledata.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 
 4 | Fields
 5 | ------
 6 |  1. taxi identifier
 7 |  2. taxi license 
 8 |  3. pickup datetime
 9 |  4. dropoff datetime
10 |  5. trip time in seconds
11 |  6. trip distance 
12 |  7. pickup longitude
13 |  8. pickup latitude
14 |  9. dropoff longitude
15 | 10. dropoff latitude
16 | 11. payment type
17 | 12. fare amount
18 | 13. surcharge
19 | 14. mta tax
20 | 15. tip amount
21 | 16. tolls amount
22 | 17. total amount
23 | 
24 | 07290D3599E7A0D62097A346EFCC1FB5,E7750A37CAB07D0DFF0AF7E3573AC141,2013-01-01 00:00:00,2013-01-01 00:02:00,120,0.44,-73.956528,40.716976,-73.962440,40.715008,CSH,3.50,0.50,0.50,0.00,0.00,4.50
25 | 
26 | 22D70BF00EEB0ADC83BA8177BB861991,3FF2709163DE7036FCAA4E5A3324E4BF,2013-01-01 00:02:00,2013-01-01 00:02:00,0,0.00,0.000000,0.000000,0.000000,0.000000,CSH,27.00,0.00,0.50,0.00,0.00,27.50
27 | 
28 | 0EC22AAF491A8BD91F279350C2B010FD,778C92B26AE78A9EBDF96B49C67E4007,2013-01-01 00:01:00,2013-01-01 00:03:00,120,0.71,-73.973145,40.752827,-73.965897,40.760445,CSH,4.00,0.50,0.50,0.00,0.00,5.00
29 | 
30 | 1390FB380189DF6BBFDA4DC847CAD14F,BE317B986700F63C43438482792C8654,2013-01-01 00:01:00,2013-01-01 00:03:00,120,0.48,-74.004173,40.720947,-74.003838,40.726189,CSH,4.00,0.50,0.50,0.00,0.00,5.00
31 | 
32 | 3B4129883A1D05BE89F2C929DE136281,7077F9FD5AD649AEACA4746B2537E3FA,2013-01-01 00:01:00,2013-01-01 00:03:00,120,0.61,-73.987373,40.724861,-73.983772,40.730995,CRD,4.00,0.50,0.50,0.00,0.00,5.00
33 | 
34 | 5FAA7F69213D26A42FA435CA9511A4FF,00B7691D86D96AEBD21DD9E138F90840,2013-01-01 00:02:00,2013-01-01 00:03:00,60,0.00,0.000000,0.000000,0.000000,0.000000,CRD,2.50,0.50,0.50,0.25,0.00,3.75
35 | 
36 | DFBFA82ECA8F7059B89C3E8B93DAA377,CF8604E72D83840FBA1978C2D2FC9CDB,2013-01-01 00:02:00,2013-01-01 00:03:00,60,0.39,-73.981544,40.781475,-73.979439,40.784386,CRD,3.00,0.50,0.50,0.70,0.00,4.70
37 | 
38 | 1E5F4C1CAE7AB3D06ABBDDD4D9DE7FA6,E0B2F618053518F24790C7FD0264E302,2013-01-01 00:03:00,2013-01-01 00:04:00,60,0.00,-73.993973,40.751266,0.000000,0.000000,CSH,2.50,0.50,0.50,0.00,0.00,3.50
39 | 
40 | 468244D1361B8A3EB8D206CC394BC9E9,BB899DFEA9CC964B50C540A1D685CCFB,2013-01-01 00:00:00,2013-01-01 00:04:00,240,1.71,-73.955383,40.779728,-73.967758,40.760326,CSH,6.50,0.50,0.50,0.00,0.00,7.50
41 | 
42 | 5F78CC6D4ECD0541B765FECE17075B6F,B7567F5BFD558C665D23B18451FE1FD1,2013-01-01 00:00:00,2013-01-01 00:04:00,240,1.21,-73.973000,40.793140,-73.981453,40.778465,CRD,6.00,0.50,0.50,1.30,0.00,8.30
43 | 
44 |
45 | 46 | 47 | -------------------------------------------------------------------------------- /webapp_php/solarchitect.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kgshukla/Realtime-Streaming-DataMicroservices-AppMicroservices/a50eff54f489ebf1b006ae186a3959c56d4849f3/webapp_php/solarchitect.png -------------------------------------------------------------------------------- /webapp_php/taxidata.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 167 | 173 | 174 | 175 | 176 | 177 | 178 | 179 | 180 | 183 | 190 | 191 |
181 | 182 | 184 |
185 |
186 |

Real Time Streaming Demo

187 |
188 |
189 |
192 | 193 |
194 |
195 |
196 |
197 |
198 |
199 |
200 | This demo showcases power of PCF, Spring Cloud Dataflow, Spring Cloud Service and Redis. 201 |
    202 |
  • Spring Cloud Dataflow receives a stream of taxi data. Sample data can be viewed here.
    • 203 |
  • Goal of the demo is to quickly evaluate top 10 busiest area based on taxi pickup and dropoff information received from streams.
    • 204 |
  • Each area is a square 1kmx1km.
    • 205 |
  • Data needs to be refreshed every 10 seconds.
    • 206 |
  • Data is not clean. Drivers could give incorrect information like wrong fare, forgot to enter pickup location, wrong latitude/longitude etc
    • 207 |
208 | Below you could see number of streams being processed, top 10 areas and details. ALL IN REAL TIME. 209 | 210 | 211 |
212 |
213 |
214 | 215 | 216 | 219 | 222 | 225 | 226 |
217 | 218 | 220 | 221 | 223 | 224 |
227 | 228 | 229 |
230 |
231 | 232 | 233 | 236 | 237 | 238 | 241 | 242 | 243 | 246 | 247 |
234 |
235 |
239 |
240 |
244 |
245 |
248 |
249 | 250 | 251 | --------------------------------------------------------------------------------