scrolling bar mockup

├── HorizontalBar.html ├── LICENSE ├── Links.md ├── MLData.htm ├── README.md ├── analysis.md ├── analystOptions.htm ├── client ├── .compilerc ├── .eslintrc ├── .gitignore ├── README.md ├── package.json └── src │ ├── app.js │ ├── index.css │ ├── index.html │ ├── index.js │ └── partials │ ├── chart.htm │ ├── json.htm │ └── stats.htm ├── collaboration.md ├── components.md ├── cover.jpg ├── dashboard.md ├── directives └── ng-prettyjson.js ├── elements.htm ├── exports.md ├── index.css ├── index.js ├── layout-columns.htm ├── layout-rows.htm ├── presenting.md ├── processing.md ├── screenshots ├── elements.png └── machinelearning-analystoptions.png └── sources.md /HorizontalBar.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | scrolling bar mockup 5 | 6 | 19 | 20 | 21 |

Experimenting with bar charts

22 | 23 | Bar charts (aka histograms) are one of the most common charts in use. I've seen a lot of things added to them. 24 | 25 |

added series with clors that designate the series
bar stacks for divisions of the total
threshold lines or backgrounds that designate something significant if a bar is above or below it
bar decorations that represent its data or how it's changed

31 | 32 | What I haven't seen much of is : 33 |

working with the width of the bar
having different horizontal sections
color brightness or other variations that have meaning beyond distinct data set
selective movement

39 | 40 | The improvements Im setting out to make are: 41 |

adjust for streaming data where needed
show most relevant data most prominently
show historical comparison data but not in a way that takes up the same amount of room
accomplish the abive with an economy of space

47 | 48 | My 1st exploration into this is to layout different timescales on the same chart, but give them variable widths and colors based on relevance 49 | the seconds adjust every second and each other time scale section adjusts as that timeframe passes. By putting them all ont he same line they are much easier to compare. Because values for a day are going to be much larger than values for a second everything is normalized to average per second. In general this works well. It's easy to both compare and distinguish the different time scales. 50 | 51 |

fixed width bars based on age/relevance, showing 7 days, 24 hours, 60 minutes, 60 seconds

52 |

53 |

54 |

55 |

56 |

57 |

58 | 59 | 60 | What I don't like about the above is that the width of the chart is based on the number of elements which will be largely based on how much history there is. While this might be accurate from some peoples reading of the chart it would not look very good when it comes to layout and alignments. In some cases it will overflow when there is not enough space, in other cases it would look choppy whene there isnt much data. Overall the variable widths of several metrics would look offputting. 61 | 62 | So in the next version there are distinct percentages of the overall width given to each timescale. however many records there are within that timescale will fill that space. This will definitely allow for better alignment overall. the tradeoff is that the bar widths will no longer reflect the age/relevance. I think it still works with the diminishing brightness and smaler sections given. 63 | 64 |
65 | 66 |

fixed width sections, sections are smaller for older records

67 |

68 |

69 |

70 |

71 |

72 |

Days

73 |

74 |

75 |

76 |

77 |

78 |

Hours

79 |

80 |

81 |

82 |

83 |

84 |

Minutes

85 |

86 |

87 |

88 |

89 |

90 |

Seconds

91 |

92 |

93 | 94 | 95 | The other decorations can always be added to this concept. As appropriate and helpful. 96 | 97 |

98 | 99 | 100 | 101 | 102 | 103 | 104 | 141 | 142 | 143 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2016 SuddenDevelopment 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /Links.md: -------------------------------------------------------------------------------- 1 | ## Concepts 2 | - https://en.wikipedia.org/wiki/Data_visualization data visualization 3 | - https://en.wikipedia.org/wiki/Voronoi_diagram calculated geometry algorithm 4 | - https://en.wikipedia.org/wiki/Hexagon hexagons are special 5 | - http://www.redblobgames.com/grids/hexagons/ details on implementing hexagons 6 | - https://en.wikipedia.org/wiki/Bloom_filter bloom filter 7 | - https://medium.com/@mslima/the-book-of-circles-4b511a5bcffc#.10wjn5hu6 circles 8 | - https://en.wikipedia.org/wiki/Predictive_modelling predictive modeling 9 | 10 | ## Technologies 11 | 12 | - https://nodejs.org/en/ server side javascript 13 | - https://angularjs.org/ web client framework 14 | - https://github.com/reactjs anaother popular web client framework 15 | - https://material.angularjs.org/latest/ apply Google material design philosphy on Angular 16 | - http://electron.atom.io/ build native apps from web technologies 17 | - https://runkit.com/home node.js online sandbox 18 | - https://github.com/Netflix/atlas/wiki - Realtime DB for Operational Intelligence 19 | - http://spark.apache.org/ popular solution for highly scalable processing 20 | 21 | ## Data sources 22 | 23 | - https://aws.amazon.com/datasets/ 24 | - https://blockchain.info/api/api_websocket Blockchain streaming data 25 | - https://www.mediawiki.org/wiki/API:Recent_changes_stream Wikimedia network changes 26 | - https://www.honeynet.org/node/960 streaming honeypot events 27 | 28 | ## Connecting 29 | 30 | - https://github.com/primus/primus mutliple client side library support 31 | - http://orientdb.com/docs/last/Live-Query.html streaming data feature 32 | - https://github.com/cayasso/mongo-oplog library to stream a mongodb oplog 33 | - http://papaparse.com/ stream a file 34 | - https://github.com/nevill/zongji stream a mysql binlog 35 | - https://github.com/SocketCluster/socketcluster distributed data streaming with channels 36 | - https://www.pubnub.com/ commercial service for streaming data 37 | - https://pusher.com/ commercial service for streaming data 38 | - https://www.elastic.co/products/beats the collection and streaming components for a very popular dstrbuted log system 39 | - https://aws.amazon.com/kinesis/ a cloud streaming framework by Amazon 40 | - https://kafka.apache.org/ streaming technology for very large scale implementations 41 | - https://osquery.io/ turn OS information into queryable data 42 | 43 | ## Processing 44 | 45 | - http://nytlabs.com/streamtools/ stream processing tools with workflow visualization 46 | - https://trackingjs.com/ image processing 47 | - http://rpominov.github.io/kefir/ stream processing library 48 | - http://reactivex.io/ stream processing library 49 | - https://momentjs.com/ transform between dat formats 50 | - http://numeraljs.com/ transform between number formats 51 | - https://github.com/SuddenDevelopment/json-collection-decorator JSON defined rules for processing 52 | 53 | ## Presenting 54 | 55 | - https://github.com/yaronn/blessed-contrib console dashboard visualization framework 56 | - http://stack.gl/ one of many webGL libraries for more experimental visualizations 57 | - https://threejs.org/ one of the more popular webgl libraries 58 | - https://github.com/regl-project/regl webgl library 59 | - http://fontawesome.io/ great icon library 60 | - https://fonts.google.com/ popular soution for fonts 61 | - https://github.com/darul75/ng-prettyjson show syntax highlighted json data in Angular 62 | - http://airbnb.design/building-a-visual-language/ a guide to developing a visual language by AirBnB 63 | - http://www.datavizcatalogue.com/ selection of data viz techniques 64 | 65 | ## Visualizing 66 | 67 | - https://www.pubnub.com/developers/eon/ real time library for client visualizations 68 | - https://d3js.org/ The most popular visualization library for client development 69 | - http://christopheviau.com/d3list/gallery.html A great gallery of D3 visualizations 70 | - http://www.aviz.fr/~bbach/timecurves/ an interesting approach to representing progress 71 | - https://github.com/Netflix/Hystrix/wiki/Dashboard A robust monitoring component that is streaming data compatible 72 | - http://grafana.org/ graph library 73 | - https://github.com/artzub/blackhole.js fancy relationship visualization, good for streaming because it has a focus on changes over time 74 | - https://github.com/uber/deck.gl map visualization library 75 | - https://github.com/datacratic/data-projector experimental 3d network visualization 76 | - https://github.com/INRIA/VisualSedimentation visualization method to show accumulated changes over time 77 | - https://github.com/anvaka/ngraph.pixel#demo graph library on webgl 78 | - http://techblog.netflix.com/2016/08/vizceral-open-source.html graph library 79 | - http://epochjs.github.io/epoch/ streaming friendly charts 80 | - https://preziotte.com/partymode/ music visualizations 81 | - https://github.com/cytoscape/cytoscape.js one of the most popular and feature rich realtionship visualization libraries 82 | - http://flowingdata.com/2017/01/24/one-dataset-visualized-25-ways/ illustrating the choices of how to best visualize a dataset 83 | - https://hal-enac.archives-ouvertes.fr/hal-01021607/document paper on node-network simplification 84 | - https://googlecreativelab.github.io/anypixel/ experimental displays 85 | - https://www.kineviz.com/ a company focused on exploring difficult data visualization challenges. 86 | - https://bl.ocks.org/emeeks/9bc7e3f505b25908a26fd8045656b490 example of treemaps with history 87 | - https://github.com/airbnb/superset airBnB open source framework for visualization using Python 88 | - https://github.com/ordina-jworks/microservices-dashboard dashboard for visualing microservices ecosystems 89 | 90 | ## Machine Learning 91 | 92 | - https://www.tensorflow.org/ general purpose machine learning framework 93 | - http://scikit-learn.org/ Python machine learning library 94 | - https://github.com/karpathy/convnetjs deep learning in the browser or node.js 95 | - https://github.com/Yomguithereal/talisman collection of machine learning capabilities on node.js 96 | - https://github.com/NaturalNode/natural popular natural language processing library for node.js 97 | - https://cloud.google.com/ml/ machine learning in Google cloud 98 | - https://aws.amazon.com/machine-learning/ machine learning in aws 99 | - https://azure.microsoft.com/en-us/services/machine-learning/ Microsoft Azure machine learning 100 | - http://yerevann.com/a-guide-to-deep-learning/ machine learning and deep learning tutorials 101 | - http://www.skrasser.com/publications/pdf/krasser2015val.pdf detailed description of machine learning applied to malware detection 102 | - http://distill.pub/ explanations with interactive examples on machine learning concepts 103 | 104 | ## Actions 105 | 106 | - https://api.slack.com/incoming-webhooks send messages to slack 107 | - https://developer.atlassian.com/hipchat/getting-started send messages to hipchat 108 | 109 | ## Streaming Data Clients 110 | - https://planetos.com/powerboard/ 111 | - https://www.alooma.com/blog/kafka-realtime-visualization a streaming data client for Kafka 112 | - http://ohm.ai 113 | - https://www.appdynamics.com/product/application-performance-management/ complete solution for microservices architecture 114 | 115 | ## Inspiration 116 | 117 | - https://initialstate.com/ streaming data visualization for Internet of Things 118 | - https://lab.interactivethings.com/galaxy-of-covers/ a beautiful illustration of song and their covers as solar systems 119 | - https://mapzen.com/blog/tron-v2-visual-scale/ visual element of tron on a streaming data map 120 | - https://www.washingtonpost.com/graphics/national/police-shootings/ interactive infographic on a hot topic 121 | - http://www.darkhorseanalytics.com/portfolio-all/ professional visualizations by a specific firm 122 | - http://dataviz.com.au/ Australian data visualization group 123 | - https://www.kiln.digital/ a collection of examples of presentations intended to convince an audience 124 | - http://d3.artzub.com/wbca/ another map visualization with a lot of coordinated elements 125 | - https://mapbox.github.io/webgl-wind/demo/ map with particle movement to show directions and paths 126 | - https://www.robscanlon.com/github-wargames/ intentionally retro visualization of live data 127 | - http://www.fudgie.org/ end to end streaming data visualization for log files 128 | - https://github.com/NarrativeScience/Log.io complete and free product for watching streaming log files 129 | - https://torflow.uncharted.software/ map of TOR traffic with relationsips 130 | - https://github.com/mbtaviz/mbtaviz.github.io/ impressive multi visualization on transportation data 131 | - http://vallandingham.me/openvis_tweets/ an interesting approach to frequency visualization using tweets 132 | - http://vallandingham.me/speaker_web/ combines frequency and relationships 133 | - https://www.behance.net/gallery/11508419/Falcon-Tactical-HUD a promotional prototype by Brian Staats for Crowdstrike 134 | - http://setosa.io/#/ collection of data visualizations 135 | - http://nulldesign.jp/metrogram3d/ 3d streaming data visualization 136 | - https://vimeo.com/groups/vast2013/videos/80829138 streaming network visualization concept 137 | 138 | ## Tutorials and Guides on Data Visualization 139 | 140 | - https://venngage.com/blog/data-visualization/ great guidelines with examples 141 | 142 | ## Other lists 143 | 144 | - http://dataviz.tools/ a great list of non streaming specific data visualization 145 | - http://treevis.net lots of tree and treemap visualizations 146 | - https://github.com/josephmisiti/awesome-machine-learning machine learning resources 147 | -------------------------------------------------------------------------------- /MLData.htm: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 |

11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Visualizing-Streaming-Data 2 | 3 | ![](cover.jpg) 4 | 5 | This is intended to be a companion to the book here: 6 |
https://www.amazon.com/Visualizing-Streaming-Data-Interactive-analysis/dp/1492031852/ 7 |
Organized by chapter / subject. 8 |
if you have an issue getting something to work or want to suggest a new link / example please create an issue. 9 |
I'm also always looking for interesting places to apply some of these techniques. Let me know what your project is. 10 | 11 | ## Sources 12 | 13 | ### Links 14 | - https://aws.amazon.com/datasets/ 15 | - https://blockchain.info/api/api_websocket Blockchain streaming data 16 | - https://www.mediawiki.org/wiki/API:Recent_changes_stream Wikimedia network changes 17 | - https://www.honeynet.org/node/960 streaming honeypot events 18 | 19 | ### Code: [examples](./sources.md) 20 | 21 | ## Streaming 22 | 23 | ### Links 24 | - https://github.com/primus/primus mutliple client side library support 25 | - http://orientdb.com/docs/last/Live-Query.html streaming data feature 26 | - https://github.com/cayasso/mongo-oplog library to stream a mongodb oplog 27 | - http://papaparse.com/ stream a file 28 | - https://github.com/nevill/zongji stream a mysql binlog 29 | - https://github.com/SocketCluster/socketcluster distributed data streaming with channels 30 | - https://www.pubnub.com/ commercial service for streaming data 31 | - https://pusher.com/ commercial service for streaming data 32 | - https://www.elastic.co/products/beats the collection and streaming components for a very popular dstrbuted log system 33 | - https://aws.amazon.com/kinesis/ a cloud streaming framework by Amazon 34 | - https://kafka.apache.org/ streaming technology for very large scale implementations 35 | - https://osquery.io/ turn OS information into queryable data 36 | 37 | ## Processing 38 | 39 | ### Links 40 | - http://nytlabs.com/streamtools/ stream processing tools with workflow visualization 41 | - https://trackingjs.com/ image processing 42 | - http://rpominov.github.io/kefir/ stream processing library 43 | - http://reactivex.io/ stream processing library 44 | - https://momentjs.com/ transform between dat formats 45 | - http://numeraljs.com/ transform between number formats 46 | - https://github.com/SuddenDevelopment/json-collection-decorator JSON defined rules for processing 47 | 48 | ### Code: [examples](./processing.md) 49 | 50 | ## Client 51 | 52 | ### Links 53 | - 54 | - https://planetos.com/powerboard/ 55 | - https://www.alooma.com/blog/kafka-realtime-visualization a streaming data client for Kafka 56 | - http://ohm.ai 57 | - https://www.appdynamics.com/product/application-performance-management/ complete solution for microservices architecture 58 | 59 | ### Code: [examples](./client) 60 | 61 | ## Presenting 62 | 63 | ### Links 64 | - https://github.com/yaronn/blessed-contrib console dashboard visualization framework 65 | - http://stack.gl/ one of many webGL libraries for more experimental visualizations 66 | - https://threejs.org/ one of the more popular webgl libraries 67 | - https://github.com/regl-project/regl webgl library 68 | - http://fontawesome.io/ great icon library 69 | - https://fonts.google.com/ popular soution for fonts 70 | - https://github.com/darul75/ng-prettyjson show syntax highlighted json data in Angular 71 | - http://airbnb.design/building-a-visual-language/ a guide to developing a visual language by AirBnB 72 | - http://www.datavizcatalogue.com/ selection of data viz techniques 73 | - https://www.pubnub.com/developers/eon/ real time library for client visualizations 74 | - https://d3js.org/ The most popular visualization library for client development 75 | - http://christopheviau.com/d3list/gallery.html A great gallery of D3 visualizations 76 | - http://www.aviz.fr/~bbach/timecurves/ an interesting approach to representing progress 77 | - https://github.com/Netflix/Hystrix/wiki/Dashboard A robust monitoring component that is streaming data compatible 78 | - http://grafana.org/ graph library 79 | - https://github.com/artzub/blackhole.js fancy relationship visualization, good for streaming because it has a focus on changes over time 80 | - https://github.com/uber/deck.gl map visualization library 81 | - https://github.com/datacratic/data-projector experimental 3d network visualization 82 | - https://github.com/INRIA/VisualSedimentation visualization method to show accumulated changes over time 83 | - https://github.com/anvaka/ngraph.pixel#demo graph library on webgl 84 | - http://techblog.netflix.com/2016/08/vizceral-open-source.html graph library 85 | - http://epochjs.github.io/epoch/ streaming friendly charts 86 | - https://preziotte.com/partymode/ music visualizations 87 | - https://github.com/cytoscape/cytoscape.js one of the most popular and feature rich realtionship visualization libraries 88 | - http://flowingdata.com/2017/01/24/one-dataset-visualized-25-ways/ illustrating the choices of how to best visualize a dataset 89 | - https://hal-enac.archives-ouvertes.fr/hal-01021607/document paper on node-network simplification 90 | - https://googlecreativelab.github.io/anypixel/ experimental displays 91 | - https://www.kineviz.com/ a company focused on exploring difficult data visualization challenges. 92 | - https://bl.ocks.org/emeeks/9bc7e3f505b25908a26fd8045656b490 example of treemaps with history 93 | - https://github.com/airbnb/superset airBnB open source framework for visualization using Python 94 | - https://github.com/ordina-jworks/microservices-dashboard dashboard for visualing microservices ecosystems 95 | 96 | ### Code: [examples](./presenting.md) 97 | 98 | ## Components 99 | 100 | ### Links 101 | 102 | ### Code: [examples](./components.md) 103 | 104 | ## Analysis 105 | 106 | ### Links 107 | 108 | ### Code: [examples](./analysis.md) 109 | 110 | ## Workflow 111 | 112 | ### Links 113 | 114 | 115 | ## Dashboard 116 | 117 | ### Links 118 | 119 | ### Code: [examples](./dashboard.md) 120 | 121 | ## Machine Learning 122 | 123 | ### Links 124 | - https://www.tensorflow.org/ general purpose machine learning framework 125 | - http://scikit-learn.org/ Python machine learning library 126 | - https://github.com/karpathy/convnetjs deep learning in the browser or node.js 127 | - https://github.com/Yomguithereal/talisman collection of machine learning capabilities on node.js 128 | - https://github.com/NaturalNode/natural popular natural language processing library for node.js 129 | - https://cloud.google.com/ml/ machine learning in Google cloud 130 | - https://aws.amazon.com/machine-learning/ machine learning in aws 131 | - https://azure.microsoft.com/en-us/services/machine-learning/ Microsoft Azure machine learning 132 | - http://yerevann.com/a-guide-to-deep-learning/ machine learning and deep learning tutorials 133 | - http://www.skrasser.com/publications/pdf/krasser2015val.pdf detailed description of machine learning applied to malware detection 134 | - http://distill.pub/ explanations with interactive examples on machine learning concepts 135 | 136 | 137 | ## Collaboration 138 | 139 | ### Links 140 | - https://api.slack.com/incoming-webhooks send messages to slack 141 | - https://developer.atlassian.com/hipchat/getting-started send messages to hipchat 142 | 143 | ### Code: [examples](./collaboration.md) 144 | 145 | ## Exports 146 | 147 | ### Links 148 | 149 | ### Code: [examples](./exports.md) 150 | 151 | ## Use Cases 152 | 153 | ### Links 154 | - https://initialstate.com/ streaming data visualization for Internet of Things 155 | - https://lab.interactivethings.com/galaxy-of-covers/ a beautiful illustration of song and their covers as solar systems 156 | - https://mapzen.com/blog/tron-v2-visual-scale/ visual element of tron on a streaming data map 157 | - https://www.washingtonpost.com/graphics/national/police-shootings/ interactive infographic on a hot topic 158 | - http://www.darkhorseanalytics.com/portfolio-all/ professional visualizations by a specific firm 159 | - http://dataviz.com.au/ Australian data visualization group 160 | - https://www.kiln.digital/ a collection of examples of presentations intended to convince an audience 161 | - http://d3.artzub.com/wbca/ another map visualization with a lot of coordinated elements 162 | - https://mapbox.github.io/webgl-wind/demo/ map with particle movement to show directions and paths 163 | - https://www.robscanlon.com/github-wargames/ intentionally retro visualization of live data 164 | - http://www.fudgie.org/ end to end streaming data visualization for log files 165 | - https://github.com/NarrativeScience/Log.io complete and free product for watching streaming log files 166 | - https://torflow.uncharted.software/ map of TOR traffic with relationsips 167 | - https://github.com/mbtaviz/mbtaviz.github.io/ impressive multi visualization on transportation data 168 | - http://vallandingham.me/openvis_tweets/ an interesting approach to frequency visualization using tweets 169 | - http://vallandingham.me/speaker_web/ combines frequency and relationships 170 | - https://www.behance.net/gallery/11508419/Falcon-Tactical-HUD a promotional prototype by Brian Staats for Crowdstrike 171 | - http://setosa.io/#/ collection of data visualizations 172 | - http://nulldesign.jp/metrogram3d/ 3d streaming data visualization 173 | - https://vimeo.com/groups/vast2013/videos/80829138 streaming network visualization concept 174 | 175 | ### Continuing Research: 176 | - http://dataviz.tools/ a great list of non streaming specific data visualization 177 | - http://treevis.net lots of tree and treemap visualizations 178 | - https://github.com/josephmisiti/awesome-machine-learning machine learning resources 179 | 180 | ### Added after Version 1 publication: 181 | - http://ethviewer.live/ : impressive / experimental visualization of ethereum blockchain 182 | - https://github.com/cubedro/eth-netstats 183 | - http://www.r2d3.us/visual-intro-to-machine-learning-part-1/ 184 | - http://www.r2d3.us/visual-intro-to-machine-learning-part-2/ 185 | -------------------------------------------------------------------------------- /analysis.md: -------------------------------------------------------------------------------- 1 | # Analysis Code Examples 2 | 3 | ### Detecting unexpected data fields 4 | ```javascript 5 | var objTemplate={ ip:['ip'], md5:['md5'] } 6 | var fnCompareSchema=function(objMsg){ 7 | var arrKeys=Object.keys(objMsg); 8 | for(var i=0;i 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 |

12 |

13 |

Data to examine here

14 |

15 |

21 |

22 |

Dynamically shown Form to explain here

23 |

24 | 25 | Category 26 | 27 | 28 | 29 | Why 30 | 31 | 32 |

33 |

34 |

35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 | 47 | 48 | -------------------------------------------------------------------------------- /client/.compilerc: -------------------------------------------------------------------------------- 1 | { 2 | "env": { 3 | "development": { 4 | "application/javascript": { 5 | "presets": [ 6 | [ 7 | "env", 8 | { 9 | "targets": { 10 | "electron": 1.8 11 | } 12 | } 13 | ], 14 | "react" 15 | ], 16 | "plugins": [ 17 | "transform-async-to-generator" 18 | ], 19 | "sourceMaps": "inline" 20 | } 21 | }, 22 | "production": { 23 | "application/javascript": { 24 | "presets": [ 25 | [ 26 | "env", 27 | { 28 | "targets": { 29 | "electron": 1.8 30 | } 31 | } 32 | ], 33 | "react" 34 | ], 35 | "plugins": [ 36 | "transform-async-to-generator" 37 | ], 38 | "sourceMaps": "none" 39 | } 40 | } 41 | } 42 | } -------------------------------------------------------------------------------- /client/.eslintrc: -------------------------------------------------------------------------------- 1 | { 2 | "extends": "eslint-config-airbnb", 3 | "rules": { 4 | "import/extensions": 0, 5 | "import/no-extraneous-dependencies": 0, 6 | "import/no-unresolved": [2, { "ignore": ["electron"] }], 7 | "linebreak-style": 0 8 | } 9 | } 10 | -------------------------------------------------------------------------------- /client/.gitignore: -------------------------------------------------------------------------------- 1 | node_modules 2 | out 3 | -------------------------------------------------------------------------------- /client/README.md: -------------------------------------------------------------------------------- 1 | # Streaming Data Visualization Prototyping Client 2 | 3 | ### Getting Started 4 | 5 | ```bash 6 | git clone https://github.com/SuddenDevelopment/Visualizing-Streaming-Data.git 7 | cd visualizing-streaming-data/client 8 | npm install 9 | electron-forge start 10 | ``` 11 | -------------------------------------------------------------------------------- /client/package.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "client", 3 | "productName": "client", 4 | "version": "1.0.0", 5 | "description": "My Electron application description", 6 | "main": "src/index.js", 7 | "scripts": { 8 | "start": "electron-forge start", 9 | "package": "electron-forge package", 10 | "make": "electron-forge make", 11 | "lint": "eslint src" 12 | }, 13 | "keywords": [], 14 | "author": "anthony", 15 | "license": "MIT", 16 | "config": { 17 | "forge": { 18 | "make_targets": { 19 | "win32": [ 20 | "squirrel" 21 | ], 22 | "darwin": [ 23 | "zip" 24 | ], 25 | "linux": [ 26 | "deb", 27 | "rpm" 28 | ] 29 | }, 30 | "electronPackagerConfig": { 31 | "packageManager": "npm" 32 | }, 33 | "electronWinstallerConfig": { 34 | "name": "client" 35 | }, 36 | "electronInstallerDebian": {}, 37 | "electronInstallerRedhat": {}, 38 | "github_repository": { 39 | "owner": "", 40 | "name": "" 41 | }, 42 | "windowsStoreConfig": { 43 | "packageName": "", 44 | "name": "client" 45 | } 46 | } 47 | }, 48 | "dependencies": { 49 | "angular": "^1.6.9", 50 | "angular-animate": "^1.6.9", 51 | "angular-aria": "^1.6.9", 52 | "angular-material": "^1.1.7", 53 | "angular-sanitize": "^1.6.9", 54 | "cytoscape": "^3.2.8", 55 | "cytoscape-cola": "^2.1.0", 56 | "electron-compile": "^6.4.2", 57 | "faker": "^4.1.0", 58 | "lodash": "^4.17.5", 59 | "ng-prettyjson": "^0.2.0" 60 | }, 61 | "devDependencies": { 62 | "babel-plugin-transform-async-to-generator": "^6.24.1", 63 | "babel-preset-env": "^1.6.1", 64 | "babel-preset-react": "^6.24.1", 65 | "electron-prebuilt-compile": "1.8.2", 66 | "eslint": "^3.19.0", 67 | "eslint-config-airbnb": "^15.1.0", 68 | "eslint-plugin-import": "^2.8.0", 69 | "eslint-plugin-jsx-a11y": "^5.1.1", 70 | "eslint-plugin-react": "^7.6.1" 71 | } 72 | } 73 | -------------------------------------------------------------------------------- /client/src/app.js: -------------------------------------------------------------------------------- 1 | //faker is to generate data for testing, not needed if you have a stream of data to connect to 2 | const libBS = require('faker'); 3 | //lodash is a great utility library for common functions native js doesnt do or do well 4 | const _ = require('lodash'); 5 | 6 | //necessary angular stuff, material is angular-matrial, ngprettjson formats json data to be readable 7 | var app = angular.module('appMain', ['ngMaterial','ngPrettyJson']).config(function($mdThemingProvider) { 8 | $mdThemingProvider.theme('default').dark(); 9 | }); 10 | app.controller('controllerMain', function($scope) { 11 | // $scope variables are what the template reads / watches 12 | $scope.title='Demo Client'; $scope.col2=false; $scope.stats={"total":0, "mac":0,"win":0}; 13 | $scope.arrData=[]; 14 | 15 | //$scope functions are the functions uysed by the template, they dont all have to be in $scope 16 | $scope.fnStartStream=function(){ 17 | $scope.title='Streaming' 18 | //this will create random data to simulate, comment this out and use a real streaming protocol instead once configured 19 | var objStream=setInterval(function(){ 20 | var objMsg={ 21 | "user":libBS.internet.userName(), 22 | "ip": libBS.internet.ip(), 23 | "agent": libBS.internet.userAgent(), 24 | "job": libBS.name.jobType() 25 | 26 | }; 27 | $scope.fnOnMessage(objMsg); 28 | }, 1000); 29 | }; 30 | $scope.fnOnMessage=function(objMsg){ 31 | //This is a function to be called on every message 32 | //console.log(objMsg); 33 | if($scope.objButtons.filter.enabled===true){ 34 | // modify object, null if it shouldnt shown at all 35 | $scope.fnFilter(objMsg); 36 | } 37 | if($scope.objButtons.transform.enabled===true){ 38 | // modify object, trasform 39 | $scope.fnTransform(objMsg); 40 | } 41 | if($scope.objButtons.stats.enabled===true){ 42 | // add stat counts 43 | $scope.fnStats(objMsg); 44 | } 45 | //add the data to the object that is being rendered 46 | //unshift adds it to the beginning, push adds to the end. makes a difference on how you want to display and buffer 47 | if(objMsg !== null){ 48 | //filtering might set it to null dont need to add those 49 | $scope.arrData.unshift(objMsg); 50 | } 51 | //this is a quirky thing with Angular to force it to re evaluate items watched. 52 | $scope.$evalAsync(); 53 | }; 54 | $scope.fnFilter=function(objMsg){ 55 | //set what needs to be filtered. In the demo case we decide we don't need ip field or any record where user starts with A 56 | if(objMsg.user.substring(0,1)==='A'){ objMsg=null } 57 | else{ objMsg.ip=undefined; } 58 | }; 59 | $scope.fnTransform=function(objMsg){ 60 | //add a field and populate it based on values in another one 61 | var strSystem='Windows'; 62 | if(objMsg.agent.indexOf('Macintosh') > -1){ strSystem='Mac'; } 63 | objMsg.system=strSystem; 64 | }; 65 | $scope.fnStatsInit=function(){ 66 | $scope.col2=true; 67 | } 68 | $scope.fnStats=function(objMsg){ 69 | //keeping stats on the fly is very important in a high volume situation, you dont want to have to loop through an array every second to get a stat 70 | if( objMsg.system==='Mac' ){ $scope.stats.mac++; } 71 | if( objMsg.system==='Windows' ){ $scope.stats.win++; } 72 | $scope.stats.total++; 73 | }; 74 | $scope.bgProgress=function(intVal,intMax){ 75 | var o={color:'#373b41',bgcolor:'#1d1f21'}; 76 | return { 'background': 'linear-gradient(90deg, '+ o.color +' '+ ((intVal/intMax)*100).toFixed(2) +'%, '+ o.bgcolor +' '+ (1-(intVal/intMax)*100).toFixed(2) +'%)' }; 77 | }; 78 | //demo navigation buttons 79 | $scope.objButtons={ 80 | "stream":{"label":'start stream',"enabled":false,"icon":'fa-play',"fn":$scope.fnStartStream}, 81 | "filter":{"label":'filter',"enabled":false,"icon":'fa-filter',"fn":$scope.fnGraph}, 82 | "transform":{"label":'transform',"enabled":false,"icon":'fa-cogs',"fn":$scope.fnGraph}, 83 | "highlight":{"label":'highlight',"enabled":false,"icon":'fa-lightbulb',"fn":$scope.fnGraph}, 84 | "stats":{"label":'stats',"enabled":false,"icon":'fa-percent',"fn":$scope.fnStatsInit}, 85 | "chart":{"label":'chart',"enabled":false,"icon":'fa-chart-bar',"fn":$scope.fnGraph}, 86 | "graph":{"label":'graph',"enabled":false,"icon":'fa-share-alt',"fn":$scope.fnGraph} 87 | }; 88 | 89 | }); -------------------------------------------------------------------------------- /client/src/index.css: -------------------------------------------------------------------------------- 1 | .Mac{ background:#555; } 2 | .Windows{ background:#f66; } 3 | .record{ overflow:hidden;} 4 | .btn-enabled {color:#66f;} -------------------------------------------------------------------------------- /client/src/index.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | Visualizing Streaming Data Sandbox Client 6 | 7 | 8 | 9 | 10 | 11 | 12 |

13 | 16 |

17 |

18 |

19 |

20 |

21 |

22 |

23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | -------------------------------------------------------------------------------- /client/src/index.js: -------------------------------------------------------------------------------- 1 | import { app, BrowserWindow } from 'electron'; 2 | 3 | // Keep a global reference of the window object, if you don't, the window will 4 | // be closed automatically when the JavaScript object is garbage collected. 5 | let mainWindow; 6 | 7 | const createWindow = () => { 8 | // Create the browser window. 9 | mainWindow = new BrowserWindow({ 10 | width: 800, 11 | height: 600, 12 | }); 13 | 14 | // and load the index.html of the app. 15 | mainWindow.loadURL(`file://${__dirname}/index.html`); 16 | 17 | // Open the DevTools. 18 | mainWindow.webContents.openDevTools(); 19 | 20 | // Emitted when the window is closed. 21 | mainWindow.on('closed', () => { 22 | // Dereference the window object, usually you would store windows 23 | // in an array if your app supports multi windows, this is the time 24 | // when you should delete the corresponding element. 25 | mainWindow = null; 26 | }); 27 | }; 28 | 29 | // This method will be called when Electron has finished 30 | // initialization and is ready to create browser windows. 31 | // Some APIs can only be used after this event occurs. 32 | app.on('ready', createWindow); 33 | 34 | // Quit when all windows are closed. 35 | app.on('window-all-closed', () => { 36 | // On OS X it is common for applications and their menu bar 37 | // to stay active until the user quits explicitly with Cmd + Q 38 | if (process.platform !== 'darwin') { 39 | app.quit(); 40 | } 41 | }); 42 | 43 | app.on('activate', () => { 44 | // On OS X it's common to re-create a window in the app when the 45 | // dock icon is clicked and there are no other windows open. 46 | if (mainWindow === null) { 47 | createWindow(); 48 | } 49 | }); 50 | 51 | // In this file you can include the rest of your app's specific main process 52 | // code. You can also put them in separate files and import them here. 53 | -------------------------------------------------------------------------------- /client/src/partials/chart.htm: -------------------------------------------------------------------------------- 1 |

2 |

{{strStat}} :

3 |

-------------------------------------------------------------------------------- /client/src/partials/json.htm: -------------------------------------------------------------------------------- 1 |

2 |

3 |

-------------------------------------------------------------------------------- /client/src/partials/stats.htm: -------------------------------------------------------------------------------- 1 |

2 | {{strStat}} : {{intStat}} 3 |

-------------------------------------------------------------------------------- /collaboration.md: -------------------------------------------------------------------------------- 1 | # Collaboration Code Examples 2 | 3 | ### Example API call in Angular 1.x 4 | ```javascript 5 | var objConfig={ 6 | 'url': 7 | "https://www.threatcrowd.org/searchApi/v2/ip/report/?ip="+strIp 8 | }; 9 | 10 | $http(objConfig).then( 11 | function fnSuccess(objResponse){ 12 | //do this with what's returned 13 | 14 | }, 15 | function fnError(objResponse){} 16 | ); 17 | ``` 18 | -------------------------------------------------------------------------------- /components.md: -------------------------------------------------------------------------------- 1 | # Components Chapter Code Examples 2 | 3 | ### keeping simple streaming stats 4 | ```javascript 5 | var intMin=null, intMax=null, intTotal=0, intCount=0; 6 | var objUniqueValues={}; 7 | var fnOnMessage=function(objMsg){ 8 | //assuming objMsg.value is a number 9 | if(objMsg.value > intMax || intMax===null){ 10 | intMax=objMsg.value; 11 | } 12 | if(objMsg.value < intMin || intMin===null){ 13 | intMin=objMsg.value; 14 | } 15 | intTotal+=objMsg.value; 16 | intCount++; 17 | if(typeof objUniqueValues[objMsg.value] === 'undefined'){ 18 | objUniqueValues[objMsg.value]=1; 19 | }else{ 20 | objUniqueValues[objMsg.value]=++; 21 | } 22 | } 23 | ``` 24 | 25 | ### Example of keeping track of time windows 26 | ```javascript 27 | var updateWindows = function(arrData, objStat){ 28 | //console.log(arrData,objStat); 29 | //does a new bucket need to be created? 30 | //var intNow=Date.now(); //shouldnt be needed, can use ls 31 | if(objStat.windows.current.fs < objStat.ls-60000 ){ 32 | //take current bucket, snapshot it to history 33 | objStat.windows.minute.push(objStat.windows.current); 34 | //re-init current bucket 35 | objStat.windows.current = 36 | _.defaults({},objDefaults[objStat.type]()); 37 | if(objStat.windows.hasOwnProperty('hour') 38 | && objStat.windows.ts_hour < objStat.ls-360000){ 39 | //loop through minutes and drop off anything older than an hour 40 | objStat.windows.minute = 41 | _.filterOld(objStat.windows.minute, 'fs', 360000); 42 | //then take the remaining ones to aggregate into an hour 43 | objStat.windows.hour.push( 44 | aggStats[objStat.type](objStat.windows.minute) 45 | ); 46 | objStat.windows.ts_hour = objStat.ls; 47 | } 48 | if(objStat.windows.hasOwnProperty('day') 49 | && objStat.windows.ts_hour < objStat.ls-86400000){ 50 | //loop through minutes and drop off anything older than an hour 51 | objStat.windows.hour = 52 | _.filterOld(objStat.windows.hour, 'fs', 86400000); 53 | //then take the remaining ones to aggregate into an hour 54 | objStat.windows.day.push( 55 | aggStats[objStat.type](objStat.windows.hour) 56 | ); 57 | objStat.windows.ts_day = objStat.ls; 58 | } 59 | } 60 | //process current 61 | objStat.windows.current = 62 | updateStats[objStat.type](arrData,objStat.windows.current); 63 | return objStat; 64 | }; 65 | ``` 66 | 67 | ### Example Streaming Data population for multiple time increment horizontal bar chart 68 | ```javascript 69 | //init the arrays 70 | var arrDays=[], arrHours=[], arrMinutes=[], arrSeconds=[]; 71 | var iSeconds=0,iMinutes=0,iHours=0,iDays=0; 72 | 73 | //small utility function to get the sum for an array 74 | var fnSum=function(arrIn){ 75 | var intSum=0; 76 | for(var i=0;i59){ 84 | //add minute 85 | var intSum=fnSum(arrSeconds); 86 | arrMinutes.push(intSum/60); 87 | iSeconds=0; 88 | if(iMinutes>59){ 89 | //add hour 90 | var intSum=fnSum(arrSeconds); 91 | arrHours.push(intSum/3600); 92 | iMinutes=0; 93 | if(iHours>23){ 94 | //add day 95 | var intSum=fnSum(arrSeconds); 96 | //divide by number of seconds to normalize to average/second 97 | arrDays.push(intSum/86400); 98 | iHours=0; 99 | } 100 | iHours++; 101 | } 102 | iMinutes++; 103 | } 104 | iSeconds++; 105 | } 106 | ``` 107 | 108 | ### Example Angular HTML to display multiple inline simple charts 109 | ```html 110 |

111 |

112 |

113 |

116 |

117 |

118 |

Days

119 |

120 |

121 |

122 |

126 |

127 |

128 |

Hours

129 |

130 |

131 |

132 |

136 |

137 |

138 |

Minutes

139 |

140 |

141 |

142 |

146 |

147 |

148 |

Seconds

149 |

150 |

151 | ``` 152 | -------------------------------------------------------------------------------- /cover.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SuddenDevelopment/Visualizing-Streaming-Data/3fc1d008d0784f1e89a088981ad6972959f579e2/cover.jpg -------------------------------------------------------------------------------- /dashboard.md: -------------------------------------------------------------------------------- 1 | # Dashboard Code Examples 2 | 3 | ### Columns Example in Angular 4 | ```html 5 |

6 |

column1

7 |

column2

8 |

9 | 10 |

11 |

12 |

13 | ``` 14 | 15 | ### Rows Example in Angular 16 | ```html 17 |

18 |

row1

19 |

row2

20 |

21 | 22 |

23 |

24 |

25 | ``` 26 | 27 | ### Recursive function to create hierarchical containers in javascript 28 | ```javascript 29 | /* 30 | Example data structure, with infinite depth 31 | { 32 | id:'parent id' 33 | ,kids:[ 34 | {id:'kid1'} 35 | ,{id:'kid2',kids:[ 36 | {id:'grandkid'} 37 | ]} 38 | ] 39 | } 40 | */ 41 | 42 | var fnCreateLayout=function(objConfig){ 43 | //recursive function, calls itself for each child and so do they 44 | self.last_updated=Date.now(); 45 | //find the current item 46 | var objElement = document.getElementById(objConfig.id); 47 | //determine layout direction, alternating rows and columns 48 | if( typeof objConfig.layout === 'undefined'){ 49 | if( objElement.clientWidth > objElement.clientHeight ) 50 | { objConfig.layout='dashboard-columns'; } 51 | if( objElement.clientWidth < objElement.clientHeight ) 52 | { objConfig.layout='dashboard-rows'; } 53 | } 54 | //add a subcontainer for kids 55 | var objKids = document.createElement('div'); 56 | objKids.setAttribute('class','dashboard-kids '+objConfig.layout); 57 | objElement.appendChild(objKids); 58 | //add the kids 59 | for(var i=0; i 87 |

88 | column {{objCol.id}} 89 |

90 | 91 | ``` 92 | 93 | -------------------------------------------------------------------------------- /directives/ng-prettyjson.js: -------------------------------------------------------------------------------- 1 | (function(angular) { 2 | 'use strict'; 3 | 4 | angular.module('ngPrettyJson', []) 5 | .directive('prettyJson', ['$compile', '$templateCache', 'ngPrettyJsonFunctions', 6 | function (compile, templateCache, ngPrettyJsonFunctions) { 7 | 8 | var isDefined = angular.isDefined; 9 | 10 | return { 11 | restrict: 'AE', 12 | scope: { 13 | prettyJson: '=', 14 | tags: '=' 15 | }, 16 | template: '

', 17 | replace: true, 18 | link: function (scope, elm, attrs) { 19 | var currentValue = {}, editor = null, clonedElement = null,i=0,ii=0; 20 | 21 | scope.id = attrs.id || 'prettyjson'; 22 | // compile template 23 | var e = compile(templateCache.get('ng-prettyjson/ng-prettyjson-panel.tmpl.html'))(scope, function(clonedElement, scope) { 24 | scope.tmplElt = clonedElement; 25 | }); 26 | 27 | elm.removeAttr("id"); 28 | //put the new item in the container 29 | elm.append(e); 30 | //console.log(scope.tags); 31 | // prefer the "json" attribute over the "prettyJson" one. 32 | // the value on the scope might not be defined yet, so look at the markup. 33 | var exp = 'prettyJson', 34 | highlight = function highlight(value) { 35 | //get the css classes to add 36 | var strClasses=''; 37 | if(typeof value !== 'undefined' && typeof value._styles !== 'undefined' && value._styles.constructor === Array && value._styles.length > 0){ 38 | //convert from an array in the object to a space separated string 39 | strClasses=value._styles.join(' '); 40 | //apply to the div 41 | //console.log(strClasses); 42 | scope.tmplElt.addClass(strClasses); 43 | } 44 | //look for recordque to add 45 | //TODO: this shoudl call recordque instead of manually embedding it 46 | 47 | var htmlStyle = ''; 48 | if(typeof value !== 'undefined' && typeof value._score !== 'undefined'){ 49 | htmlStyle = 'background: linear-gradient(90deg, rgb(55, 59, 65) '+((value._score/100)*100).toFixed(2)+'%, rgb(29, 31, 33) '+(1-(value._score/100)*100).toFixed(2)+'%);'; 50 | //console.log(htmlStyle); 51 | } 52 | 53 | //add icons for tags that are defined 54 | var htmlIcons = ''; 55 | if(typeof value !== 'undefined' && typeof value._tags !== 'undefined' && scope.tags !== 'undefined'){ 56 | //loop through item tags 57 | for(i=0;i'; 61 | } 62 | } 63 | 64 | var html = ngPrettyJsonFunctions.syntaxHighlight(value) || ""; 65 | //add the brackets highlighting 66 | 67 | html = html 68 | .replace(/\{/g, "{") 69 | .replace(/\}/g, "}") 70 | .replace(/\[/g, "[") 71 | .replace(/\]/g, "]") 72 | .replace(/\,/g, ","); 73 | 74 | //s there a _recordque to render 75 | if(htmlIcons!==''){ html = html + htmlIcons; } 76 | //add in the style property based on _score for backfround fill 77 | scope.tmplElt.find('pre').attr('style',htmlStyle); 78 | //return isDefined(value) ? scope.tmplElt.find('pre').html(html) : scope.tmplElt.find('pre').empty(); 79 | return scope.tmplElt.find('pre').html(html); 80 | }, 81 | objWatch; 82 | 83 | objWatch = scope.$watch(exp, highlight, false); 84 | } 85 | }; 86 | }]) 87 | // mostly we want to just expose this stuff for unit testing; if it's within the directive's 88 | // function scope, we can't get to it. 89 | .factory('ngPrettyJsonFunctions', function ngPrettyJsonFunctions() { 90 | 91 | // cache some regular expressions 92 | var rx = { 93 | entities: /((&)|(<)|(>))/g, 94 | json: /"(\\u[a-zA-Z0-9]{4}|\\[^u]|[^\\"])*"(\s*:)?|\b(true|false|(null))\b|-?\d+(?:\.\d*)?(?:[eE][+\-]?\d+)?/g 95 | }; 96 | 97 | // mapping of chars to entities 98 | var entities = ['&','<','>']; 99 | 100 | // lookup of positional regex matches in rx.json to CSS classes 101 | var classes = ['number','string','key','boolean','null']; 102 | 103 | var fnReplace = function(k,v){ 104 | //properties to filter out 105 | //TODO abstract these to be inputs into the directive 106 | if(k==='_score' || k==='_id' || k==='_column' || k==='_styles'|| k==='_image' || k==='_event' || k==='_text' || k==='_recordque' || k==='$$hashKey'){ return undefined; }else{return v;} 107 | }; 108 | 109 | /** 110 | * @description Used by {@link makeEntities} and {@link markup}. Expects all arguments 111 | * to be nonempty strings. 112 | * @private 113 | * @returns {number} Index of last nonempty string argument in list of arguments. 114 | */ 115 | var reverseCoalesce = function reverseCoalesce() { 116 | var i = arguments.length - 2; 117 | do { 118 | i--; 119 | } while (!arguments[i]); 120 | return i; 121 | }; 122 | 123 | /** 124 | * @description Callback to String.prototype.replace(); marks up JSON string 125 | * @param {string|number} match Any one of the below, or if none of the below, it's a number 126 | * @returns {string} Marked-up JSON string 127 | */ 128 | var markup = function markup(match) { 129 | var idx; 130 | // the final two arguments are the length, and the entire string itself; 131 | // we don't care about those. 132 | if (arguments.length < 7) { 133 | throw new Error('markup() must be called from String.prototype.replace()'); 134 | } 135 | idx = reverseCoalesce.apply(null, arguments); 136 | return '' + match + ''; 137 | }; 138 | 139 | /** 140 | * @description Finds chars in string to turn into entities for HTML output. 141 | * @returns {string} Entity-ized string 142 | */ 143 | var makeEntities = function makeEntities() { 144 | var idx; 145 | if (arguments.length < 5) { 146 | throw new Error('makeEntities() must be called from String.prototype.replace()'); 147 | } 148 | idx = reverseCoalesce.apply(null, arguments); 149 | return entities[idx - 2]; 150 | }; 151 | 152 | /** 153 | * @description Does some regex matching to sanitize for HTML and finally returns a bunch of 154 | * syntax-highlighted markup. 155 | * @param {*} json Something to be output as pretty-printed JSON 156 | * @returns {string|undefined} If we could convert to JSON, you get markup as a string, otherwise 157 | * no return value for you. 158 | */ 159 | var syntaxHighlight = function syntaxHighlight(json) { 160 | if (!angular.isString(json)) 161 | json = JSON.stringify(json, fnReplace, 2); 162 | if (angular.isDefined(json)) { 163 | return json.replace(rx.entities, makeEntities) 164 | .replace(rx.json, markup); 165 | } 166 | }; 167 | 168 | return { 169 | syntaxHighlight: syntaxHighlight, 170 | makeEntities: makeEntities, 171 | markup: markup, 172 | rx: rx 173 | }; 174 | }); 175 | 176 | })(window.angular); 177 | (function(angular) { 178 | 'use strict'; 179 | 180 | angular.module('ngPrettyJson') 181 | .run(['$templateCache', function ($templateCache) { 182 | $templateCache.put('ng-prettyjson/ng-prettyjson-panel.tmpl.html', 183 | '

' + 184 | '

' + 185 | '

'); 186 | }]); 187 | 188 | })(window.angular); 189 | -------------------------------------------------------------------------------- /elements.htm: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 |

13 |

14 |

15 |

16 |

17 |

18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | -------------------------------------------------------------------------------- /exports.md: -------------------------------------------------------------------------------- 1 | # Exports Code Examples 2 | 3 | ```javascript 4 | Convert a JSON object to a string 5 | //assuming objConfig is the json object you want to convert 6 | var strConfig = JSON.stringify(objConfig); 7 | 8 | //the reverse process, when required 9 | var objConfig=JSON.parse(strConfig); 10 | ``` 11 | 12 | ### JSON Data Arrays Export 13 | ```javascript 14 | var arrData=[]; 15 | 16 | var fnOnMessage=function(objMsg){ 17 | arrData.unshift(objMsg); 18 | } 19 | 20 | var fnExportData=function(){ 21 | var a = document.createElement("a"); 22 | var strData=JSON.stringify(arrData); 23 | var file = new Blob([strData], {type: 'text/plain'}); 24 | a.href = URL.createObjectURL(file); 25 | a.download = 'export.json'; 26 | a.click(); 27 | } 28 | ``` 29 | -------------------------------------------------------------------------------- /index.css: -------------------------------------------------------------------------------- 1 | /* 2 | 3 | base00: "ffffff" 4 | base01: "e1e1e2" 5 | base02: "c4c3c5" 6 | base03: "a7a5a8" 7 | base04: "89878b" 8 | base05: "6c696e" 9 | base06: "4f4b51" 10 | base07: "322d34" 11 | base08: "d8137f" 12 | base09: "d65407" 13 | base0A: "dc8a0e" 14 | base0B: "17ad98" 15 | base0C: "149bda" 16 | base0D: "775dff" 17 | base0E: "aa17e6" 18 | base0F: "e013d0" 19 | 20 | $scope.objClasses={ 21 | alignment:['left','right','center','left','right'] 22 | ,borders:['solid','double','dotted','dashed','groove'] 23 | ,color:['red','orange','yellow','green','blue'] 24 | ,size:['normal','small','large','tiny','xl'] 25 | ,icons:[] 26 | ,fonts:['sans','serif','mono','normal','normal'] 27 | ,background:['bgred','bgorange','bgyellow','bggreen','bgblue'] 28 | ,thickness:['thin','thick','xthick','thick-left','thick-sides'] 29 | ,spacing:['none','barely','equal','double'] 30 | ,shape:['normal','round'] 31 | }; 32 | 33 | */ 34 | 35 | .border{border: 1px solid #666;} 36 | 37 | pre.pretty-json{ overflow-x:hidden; margin:5px;} 38 | pre.pretty-json span.key{ color: #d8137f; } 39 | pre.pretty-json span.string{ color: #322d34; } 40 | pre.pretty-json span.number{ color: #d65407; } 41 | pre.pretty-json span.sep{ color: #a7a5a8; } 42 | pre.pretty-json span.boolean{ color: #775dff; } 43 | pre.pretty-json span.null{ color: #aa17e6; } 44 | 45 | .right{text-align:right;border-top:1px solid #000;} 46 | .center{text-align:center;border:1px solid #000;} 47 | 48 | .double{border:1px double #000;} 49 | .dotted{border:1px dotted #000;} 50 | .dashed{border:1px dashed #000;} 51 | .groove{border:1px groove #000;} 52 | 53 | .small{font-size: .75em;} 54 | .large{font-size:1.5em;} 55 | .xl{font-size: 2em;} 56 | .tiny{font-size:.5em;} 57 | 58 | .thick{border:3px solid #000;} 59 | .xthick{border:5px solid #000;} 60 | .thick-left{border-left:10px solid #000;} 61 | .thick-sides{border-left:10px solid #000;border-right:10px solid #000;} 62 | 63 | .none{border:1px solid #000;margin:0;padding:0;} 64 | .barely{border:1px solid #000;margin:5px;padding:5px;} 65 | .equal{border:1px solid #000;margin:10px;padding:10px;} 66 | 67 | .round{border:1px solid #000;border-radius: 30px;} 68 | .round-side{border:1px solid #000;border-top-right-radius: 30px;border-bottom-right-radius: 30px;} 69 | .round-top{border:1px solid #000;border-top-left-radius: 30px;border-top-right-radius: 30px;} 70 | 71 | .pre.sans{font-family:Arial, Helvetica, sans-serif;} 72 | .pre.serif{font-family:Georgia, serif;} 73 | .pre.mono{font-family:"Courier New", Courier, monospace;} 74 | 75 | .bgred{background-color:#fdd;} 76 | .bgyellow{background-color:#ffd;} 77 | .bggreen{background-color:#dfd;} -------------------------------------------------------------------------------- /index.js: -------------------------------------------------------------------------------- 1 | // 2 | /* 3 | 4 | https://tswiki.atlassian.net/wiki/display/LABS/Useful+SQL+Queries 5 | 6 | */ 7 | 8 | var app=angular.module('mockups', ['ngMaterial','ngPrettyJson']); 9 | app.controller('main',function($scope,$http){ 10 | $scope.objML={ 11 | classification:'Suspicious' 12 | ,confidence:79 13 | ,model:'K Means Clustering' 14 | ,factors:['asn','geo','whois','risk','history','dns','activity','honeypots','network'] 15 | ,topFactors:[ 16 | {factor:'risk',influence:23} 17 | ,{factor:'history',influence:21} 18 | ,{factor:'dns',influence:12} 19 | ,{factor:'honeypots',influence:7} 20 | ] 21 | } 22 | 23 | $scope.arrRows=[ 24 | ['alignment','borders','color','size','icons'] 25 | ,['fonts','background','thickness','spacing','shape'] 26 | ]; 27 | 28 | $scope.objClasses={ 29 | alignment:['left','right','center','left','right'] 30 | ,borders:['solid','double','dotted','dashed','groove'] 31 | ,color:['red','orange','yellow','green','blue'] 32 | ,size:['normal','small','large','tiny','xl'] 33 | ,icons:[] 34 | ,fonts:['sans','serif','mono','normal','normal'] 35 | ,background:['bgred','bgorange','bgyellow','bggreen','bgblue'] 36 | ,thickness:['thin','thick','xthick','thick-left','thick-sides'] 37 | ,spacing:['normal','none','barely','equal','double'] 38 | ,shape:['normal','round','round-side','round-top'] 39 | }; 40 | 41 | $scope.layouts=['rows','columns','radar','adjacency','position']; 42 | 43 | $scope.arrItems=[ 44 | {"place": { 45 | "country_code": "BR" 46 | },"_tags":["bell"], 47 | "text": "Relendo estes RPGs que estavam perdidos na minha estante. Quem gostaria de jogar uma sessão… https://t.co/BJXQci7YB6" 48 | },{ 49 | "place": { 50 | "country_code": "PH" 51 | },"_tags":["bomb"], 52 | "text": "@kittyhalei24 ahahahahahhahaah grabe ka hard!" 53 | },{ 54 | "place": { 55 | "country_code": "US" 56 | },"_tags":["bug"], 57 | "text": "😪 https://t.co/sRqjsoH6GV" 58 | },{ 59 | "place": { 60 | "country_code": "VE" 61 | },"_tags":["trash"], 62 | "text": "Yo te quiero tener" 63 | } 64 | ]; 65 | 66 | $scope.borders=['dotted','dashed','double','groove']; 67 | 68 | }); -------------------------------------------------------------------------------- /layout-columns.htm: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 |

11 |

12 |

Raw Data

13 |

Stats

14 |

Data

15 |

16 |

17 |

Processed Data

18 |

Stats

19 |

Data

20 |

21 |

22 |

Augmented Data

23 |

Stats

24 |

Data

25 |

26 |

27 |

Analyst Queue

28 |

Stats

29 |

Data

30 |

31 |

32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | -------------------------------------------------------------------------------- /layout-rows.htm: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 |

11 |

12 |

Raw Data

13 |

Stats

14 |

Data

15 |

16 |

17 |

Processed Data

18 |

Stats

19 |

Data

20 |

21 |

22 |

Augmented Data

23 |

Stats

24 |

Data

25 |

26 |

27 |

Analyst Queue

28 |

Stats

29 |

Data

30 |

31 |

32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | -------------------------------------------------------------------------------- /presenting.md: -------------------------------------------------------------------------------- 1 | # Presenting Data 2 | 3 | ### Batch everything in a timeframe, then process. 4 | ```javascript 5 | // a time reference is needed to keep track of batches. 6 | var intBatchStartTime=Date.now(); 7 | // an array to hold the batches 8 | var arrBatch=[]; 9 | //a function to be called on events 10 | var fnOnEvent=function(objEvent){ 11 | var intNow=Date.now(); 12 | //always add the event to the batch 13 | arrBatch.push(objEvent); 14 | if(intBatchStartTime+1000 0) 88 | { 89 | //reduce the severity and recalc the ttl 90 | arrObjects[i].severity--; 91 | arrObjects[i].ttl=objMsg.severity*60000; 92 | }else{ 93 | fnRemove(arrObjects[i]); 94 | } 95 | } 96 | } 97 | ``` 98 | 99 | -------------------------------------------------------------------------------- /processing.md: -------------------------------------------------------------------------------- 1 | # Processing 2 | 3 | ### Processing in batches of 1 second. 4 | ```javascript 5 | var arrBatch=[]; 6 | var intTimeStart=Date.now(); 7 | var intLimit=1000; 8 | var fnOnMessage=function(objMsg){ 9 | var intNow=Date.now(); 10 | arrBatch.unshift(objMsg); 11 | if(intNow>intTimeStart+intLimit){ 12 | intTimeStart=intNow; 13 | fnProcessBatch(); 14 | } 15 | } 16 | var fnProcessBatch=function(){ 17 | for(var i=0;i intMax){ intMax=intData; } 84 | if(intData < intMin){ intMin=intData; } 85 | intMean = intSum/intCount; 86 | ``` 87 | 88 | ### Example sampling logic 89 | ```javascript 90 | var intSample=10; 91 | var intCount=0; 92 | //call the sampling function every time 93 | var fnSample=function(objMsg){ 94 | intCount++; 95 | if( intCount%intSample===0){ 96 | //the nth count, fire the function to process 97 | fnOnMessage(objMsg); 98 | } 99 | }; 100 | ``` 101 | 102 | ### Example throttle logic 103 | ```javascript 104 | var intCap=1000; 105 | var intCount=0; 106 | var intStart=Date.now(); 107 | //call the throttling function every time 108 | var fnThrottle=function(objMsg){ 109 | var intNow=Date.now(); 110 | //see if it's in a new time increment 111 | if(intNow>intStart+1000){ intStart=intNow; intCount=0; } 112 | else{ intCount++; } 113 | if( intCount < intCap){ 114 | //only run if under the cap per timeperiod 115 | fnOnMessage(objMsg); 116 | } 117 | }; 118 | ``` 119 | 120 | -------------------------------------------------------------------------------- /screenshots/elements.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SuddenDevelopment/Visualizing-Streaming-Data/3fc1d008d0784f1e89a088981ad6972959f579e2/screenshots/elements.png -------------------------------------------------------------------------------- /screenshots/machinelearning-analystoptions.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SuddenDevelopment/Visualizing-Streaming-Data/3fc1d008d0784f1e89a088981ad6972959f579e2/screenshots/machinelearning-analystoptions.png -------------------------------------------------------------------------------- /sources.md: -------------------------------------------------------------------------------- 1 | ### Split stream example, simple round robin 2 | 3 | ```javascript 4 | 5 | //set stream count 6 | var intStreams=4; 7 | //current stream 8 | var intCurrentStream=1; 9 | var fnOnMessage=function(objMsg){ 10 | //assuming this function will send to the stream specified 11 | fnSendMessage(intCurrentStream,objMsg); 12 | //update the stream pointer 13 | if(intCurrentStream === intStreams){ 14 | //reset 15 | intCurrentStream=1; 16 | }else{ 17 | //increment 18 | intCurrentStream++; 19 | } 20 | } 21 | 22 | ``` 23 | ### Stream sort / map example 24 | ```javascript 25 | 26 | var objMap={ 27 | "production":1, 28 | "development":2, 29 | "staging":3, 30 | "certification":4 31 | }; 32 | var fnOnMessage(objMsg){ 33 | //assuming this function will send to the stream specified 34 | //map the environment data by the map 35 | fnSendMessage(objMap[objMsg.environment],objMsg); 36 | } 37 | 38 | ``` 39 | ### Sort by schema example 40 | ```javascript 41 | 42 | //properties to look for in an object 43 | var objMap={ 44 | "ip":1, 45 | "url":2, 46 | "sha256":3, 47 | "regex":4 48 | }; 49 | var arrFields=Object.keys(objMap); 50 | var fnOnMessage(objMsg){ 51 | var fFound=false; 52 | var intStream=0; 53 | for(var i=0;i

Experimenting with bar charts

fixed width bars based on age/relevance, showing 7 days, 24 hours, 60 minutes, 60 seconds

fixed width sections, sections are smaller for older records

{{ title }}