├── .gitignore ├── .rspec ├── .travis.yml ├── Gemfile ├── README.md ├── Rakefile ├── _config.yml ├── bin ├── console ├── setup └── snowly ├── config.ru ├── lib ├── schemas │ └── snowplow_protocol.json ├── snowly.rb └── snowly │ ├── app │ ├── collector.rb │ └── views │ │ ├── index.erb │ │ └── js.erb │ ├── each_validator.rb │ ├── extensions │ └── custom_dependencies.rb │ ├── protocol_schema_finder.rb │ ├── request.rb │ ├── schema_cache.rb │ ├── transformer.rb │ ├── validator.rb │ ├── validators │ └── self_desc.rb │ └── version.rb ├── snowly.gemspec └── spec ├── fixtures ├── snowly │ ├── context_test_0 │ │ └── jsonschema │ │ │ └── 1-0-0 │ ├── context_test_1 │ │ └── jsonschema │ │ │ └── 1-0-0 │ └── event_test │ │ └── jsonschema │ │ └── 1-0-0 ├── snowplow_context.json └── snowplow_ue.json ├── protocol_resolver └── snowplow_protocol.json ├── snowly ├── app │ └── collector_spec.rb ├── integration │ └── validator_tracker_spec.rb └── unit │ ├── each_validator_spec.rb │ ├── protocol_schema_finder_spec.rb │ ├── request_spec.rb │ ├── schema_cache_spec.rb │ ├── transformer_spec.rb │ └── validator_spec.rb ├── snowly_spec.rb ├── spec_helper.rb └── support ├── emitter.rb ├── event_assignments.rb ├── server_starter.rb └── tracker_patch.rb /.gitignore: -------------------------------------------------------------------------------- 1 | /.bundle/ 2 | /.yardoc 3 | /Gemfile.lock 4 | /_yardoc/ 5 | /coverage/ 6 | /doc/ 7 | /pkg/ 8 | /spec/reports/ 9 | /tmp/ 10 | /personal 11 | *.gem 12 | -------------------------------------------------------------------------------- /.rspec: -------------------------------------------------------------------------------- 1 | --format documentation 2 | --color 3 | -------------------------------------------------------------------------------- /.travis.yml: -------------------------------------------------------------------------------- 1 | language: ruby 2 | rvm: 3 | - 2.1.4 4 | before_install: gem install bundler -v 1.11.2 5 | -------------------------------------------------------------------------------- /Gemfile: -------------------------------------------------------------------------------- 1 | source 'https://rubygems.org' 2 | 3 | # Specify your gem's dependencies in snowly.gemspec 4 | gemspec 5 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Snowly - Snowplow Request Validator 2 | 3 | [![Deploy](https://www.herokucdn.com/deploy/button.svg)](https://github.com/angelim/snowly-heroku) 4 | 5 | Debug your Snowplow implementation locally, without resorting to Snowplow's ETL tasks. It's like Facebook's URL Linter, but for Snowplow. 6 | 7 | Snowly is a minimal [Collector](https://github.com/snowplow/snowplow/wiki/Setting-up-a-collector) implementation intended to run on your development environment. It comes with a comprehensive validation engine, that will point out any schema requirement violations. You can easily validate your event requests before having to emit them to a cloudfront, closure or scala collector. 8 | 9 | ### Motivation 10 | 11 | Snowplow has an excellent toolset, but the first implementation stages can be hard. To run Snowplow properly you have to set up a lot of external dependencies like AWS permissions, Cloudfront distributions and EMR jobs. If you're tweaking the snowplow model to fit your needs or using trackers that don't enforce every requirement, you'll find yourself waiting for the ETL jobs to run in order to validate every implementation change. 12 | 13 | ### Who will get the most from Snowly 14 | 15 | - Teams that need to extend the snowplow model with custom contexts or unstructured events. 16 | - Applications that are constantly evolving their schemas and rules. 17 | - Developers trying out Snowplow before commiting to it. 18 | 19 | ### Features 20 | 21 | With Snowly you can use [Json Schemas](http://spacetelescope.github.io/understanding-json-schema/) to define more expressive event requirements. Aside from assuring that you're fully compatible with the snowplow protocol, you can go even further and extend it with a set of more specific rules. Snowly emulates both cloudfront and closure collectors and will handle its differences automatically. 22 | 23 | Use cases: 24 | 25 | - Validate custom contexts or unstructured event types and required fields. 26 | - Restrict values for any field, like using a custom dictionary for the structured event action field. 27 | - Define requirements based on the content of another field: If __event action__ is 'viewed_product', then __event property__ is required. 28 | 29 | ## Installation 30 | 31 | ```bash 32 | gem install snowly 33 | ``` 34 | That will copy a `snowly` executable to your system. 35 | 36 | ### Development Iglu Resolver Path 37 | 38 | If you still don't know anything about [Snowplow's Iglu Resolvers](https://github.com/snowplow/iglu), don't worry. It's pretty straightforward. 39 | Snowly must be able to find your custom context and unstructured event schemas, so you have to set up a local path to store them. You can also choose to use an [external resolver](https://github.com/snowplow/iglu/wiki/Static-repo-setup) pointing to an URL. 40 | 41 | For a local setup, store your schemas under any path accessible by your user(eg: ~/schemas). The only catch is that you must comply with snowplow's naming conventions for your json schemas. Snowplow References:[[1]](https://github.com/snowplow/snowplow/wiki/snowplow-tracker-protocol#custom-contexts),[[2]](https://github.com/snowplow/snowplow/wiki/snowplow-tracker-protocol#310-custom-unstructured-event-tracking) 42 | 43 | You must export an environment variable to make Snowly aware of that path. Add it to your .bash_profile or equivalent. 44 | ```bash 45 | # A local path is the recommended approach, as its easier to evolve your schemas 46 | # without the hassle of setting up an actual resolver. 47 | export DEVELOPMENT_IGLU_RESOLVER_PATH=~/schema 48 | 49 | # or host on a Static Website on Amazon Web Services, for instance. 50 | export DEVELOPMENT_IGLU_RESOLVER_PATH=http://my_resolver_bucket.s3-website-us-east-1.amazonaws.com 51 | ``` 52 | 53 | Example: 54 | ```bash 55 | # create a user context 56 | mkdir -p ~/schemas/com.my_company/hero_user/jsonschema 57 | touch ~/schemas/com.my_company/hero_user/jsonschema/1-0-0 58 | 59 | # create a viewed product unstructured event 60 | mkdir -p ~/schemas/com.my_company/viewed_product/jsonschema 61 | touch ~/schemas/com.my_company/viewed_product/jsonschema/1-0-0 62 | ``` 63 | 64 | `1-0-0` is the actual json schema file. You will find examples just ahead. 65 | 66 | ## Usage 67 | 68 | Just use `snowly` to start and `snowly -K` to stop. Where allowed, a browser window will open showing the collector's address. Use `snowly --help` for other options. 69 | 70 | ### Output 71 | 72 | When Snowly finds something wrong, it renders a parsed array of requests along with its errors. 73 | 74 | When everything is ok, Snowly delivers the default Snowplow pixel, unless you're using the debug mode. 75 | 76 | If you can't investigate the request's response, you can start Snowly in the foreground and in __Debug Mode__ to output the response to __STDOUT__. 77 | `snowly -d -F` 78 | 79 | Example: 80 | `http://0.0.0.0:5678/i?&e=pv&page=Root%20README&url=http%3A%2F%2Fgithub.com%2Fsnowplow%2Fsnowplow&aid=snowplow&p=i&tv=no-js-0.1.0&eid=ev-id-1` 81 | ```json 82 | [ 83 | { 84 | "event_id": "ev-id-1", 85 | "errors": [ 86 | "The property '#/platform' value \"i\" did not match one of the following values: web, mob, pc, srv, tv, cnsl, iot in schema snowplow_protocol.json", 87 | "The property '#/' did not contain a required property of 'useragent' in schema snowplow_protocol.json" 88 | ], 89 | "content": { 90 | "event": "pv", 91 | "page_title": "Root README", 92 | "page_url": "http://github.com/snowplow/snowplow", 93 | "app_id": "snowplow", 94 | "platform": "i", 95 | "v_tracker": "no-js-0.1.0", 96 | "event_id": "ev-id-1" 97 | } 98 | } 99 | ] 100 | ``` 101 | 102 | If you're using the closure collector and can't see your requests firing up right away, try [manually flushing](https://github.com/snowplow/snowplow/wiki/Ruby-Tracker#54-manual-flushing) or change your emitter's buffer_size(number of events before flusing) to a lower value. 103 | 104 | In debug mode Snowly always renders the parsed contents of your requests. If you're using the javascript tracker, use the __post__ option to be able to read the response in your browser inspector. The js tracker implementation for __get__ requests works by changing an image src, so the inspector hides the response. 105 | 106 | ## JSON Schemas 107 | 108 | JSON Schema is a powerful tool for validating the structure of JSON data. I recommend reading this excellent [Guide](http://spacetelescope.github.io/understanding-json-schema/) from Michael Droettboom to understand all of its capabilities, but you can start with the examples bellow. 109 | 110 | Example: 111 | 112 | A user context. Well... Not just any user can get there. 113 | 114 | __Note that this is not valid json because of the comments.__ 115 | ```ruby 116 | # ~/schemas/com.my_company/hero_user/jsonschema/1-0-0 117 | { 118 | # Your schema will also be checked against the Snowplow Self-Desc Schema requirements. 119 | "$schema": "http://iglucentral.com/schemas/com.snowplowanalytics.self-desc/schema/jsonschema/1-0-0#", 120 | "id": "com.my_company/hero_user/jsonschema/1-0-0", # Give your schema an id for better validation output 121 | "description": "My first Hero Context", 122 | "self": { 123 | "vendor": "com.my_company", 124 | "name": "hero_user", 125 | "format": "jsonschema", 126 | "version": "1-0-0" 127 | }, 128 | 129 | "type": "object", 130 | "properties": { 131 | "name": { 132 | "type": "string", 133 | "maxLength": 100 # The hero's name can't be larger than 100 chars 134 | }, 135 | "special_powers": { 136 | "type": "array", 137 | "minItems": 2, # This is not just any hero. He must have at least two special powers. 138 | "uniqueItems": true 139 | }, 140 | "age": { 141 | "type": "integer", # Strings are not allowed. 142 | "minimum": 15, # The Powerpuff Girls aren't allowed 143 | "maximum": 100 # Wolverine is out 144 | }, 145 | "cape_color": { 146 | "type": "string", 147 | "enum": ["red", "blue", "black"] # Vision is not welcome 148 | }, 149 | "is_avenger": { 150 | "type": "boolean" 151 | }, 152 | "rating": { 153 | "type": "number" # Allows for float values 154 | }, 155 | "address": { # cascading objects with their own validation rules. 156 | "type": "object", 157 | "properties": { 158 | "street_name": { 159 | "type": "string" 160 | }, 161 | "number": { 162 | "type": "integer" 163 | } 164 | } 165 | } 166 | }, 167 | "required": ["name", "age"], # Name and Age must always be present 168 | "custom_dependencies": { 169 | "cape_color": { "name": "superman" } # If the hero's #name is 'superman', #cape_color has to be present. 170 | }, 171 | "additionalProperties": false # No other unspecified attributes are allowed. 172 | } 173 | ``` 174 | 175 | ### Extending Snowplow's Protocol 176 | 177 | Although the Snowplow's protocol isn't originally defined as a JSON schema, it doesn't hurt to do so and take advantage of all its perks. It's also here for the sake of consistency, right? 178 | 179 | By expressing the protocol in a JSON schema you can extend it to fit your particular needs and enforce domain rules that otherwise wouldn't be available. [Take a look](https://github.com/angelim/snowly/blob/master/lib/schemas/snowplow_protocol.json) at the default schema, derived from the rules specified on the [canonical model](https://github.com/snowplow/snowplow/wiki/canonical-event-model). 180 | 181 | Whenever possible, Snowly will output column names mapped from query string parameters. When two parameters can map to the same content(eg. regular and base64 versions), a common intuitive name is used(eg. contexts and unstruct_event). 182 | 183 | You can override the protocol schema by placing it anywhere inside your Local Resolver Path. As of now, the whole file has to be replaced: 184 | 185 | __It's important to name the file as `snowplow_protocol.json`.__ 186 | 187 | One example of useful extensions. 188 | ```ruby 189 | ... 190 | "se_action": { 191 | "type": "string", 192 | "enum": ["product_view", "add_to_cart", "product_zoom"] # Only these values are allowed for an structured event action. 193 | } 194 | 195 | "custom_dependencies": { 196 | "true_tstamp": {"platform": "mob"} # You must submit the true timestamp when the platform is set to "mob". 197 | { 198 | ... 199 | ``` 200 | 201 | ## Development 202 | 203 | After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment. 204 | 205 | To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org). 206 | 207 | ## Contributing 208 | 209 | Bug reports and pull requests are welcome on GitHub at https://github.com/angelim/snowly. 210 | 211 | -------------------------------------------------------------------------------- /Rakefile: -------------------------------------------------------------------------------- 1 | require "bundler/gem_tasks" 2 | require "rspec/core/rake_task" 3 | 4 | RSpec::Core::RakeTask.new(:spec) 5 | 6 | task :default => :spec 7 | -------------------------------------------------------------------------------- /_config.yml: -------------------------------------------------------------------------------- 1 | theme: jekyll-theme-slate -------------------------------------------------------------------------------- /bin/console: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env ruby 2 | 3 | require "bundler/setup" 4 | require "snowly" 5 | 6 | # You can add fixtures and/or initialization code here to make experimenting 7 | # with your gem easier. You can also use a different console, if you like. 8 | 9 | # (If you use this, don't forget to add pry to your Gemfile!) 10 | # require "pry" 11 | # Pry.start 12 | 13 | require "irb" 14 | IRB.start 15 | -------------------------------------------------------------------------------- /bin/setup: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | set -euo pipefail 3 | IFS=$'\n\t' 4 | set -vx 5 | 6 | bundle install 7 | 8 | # Do any other automated setup that you need to do here 9 | -------------------------------------------------------------------------------- /bin/snowly: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env ruby 2 | # /snowly 3 | require "bundler/setup" 4 | require 'snowly' 5 | require 'snowly/app/collector' 6 | require 'vegas' 7 | Snowly.debug_mode = true if ARGV.index{ |arg| arg == '-d' or arg == '--debug' } 8 | Vegas::Runner.new(Snowly::App::Collector, 'collector') -------------------------------------------------------------------------------- /config.ru: -------------------------------------------------------------------------------- 1 | require 'rubygems' 2 | require 'bundler' 3 | Bundler.setup 4 | require 'snowly' 5 | require 'snowly/app/collector' 6 | run Snowly::App::Collector -------------------------------------------------------------------------------- /lib/schemas/snowplow_protocol.json: -------------------------------------------------------------------------------- 1 | { 2 | "$schema": "http://json-schema.org/draft-04/schema#", 3 | "id": "snowplow_protocol.json", 4 | "description": "Representation of Snowplow Protocol in JSON Schema format for validation", 5 | "type": "object", 6 | "properties": { 7 | "name_tracker": { 8 | "type": "string", 9 | "maxLength": 128 10 | }, 11 | "event_vendor": { 12 | "type": "string", 13 | "maxLength": 1000 14 | }, 15 | "app_id": { 16 | "type": "string", 17 | "maxLength": 255 18 | }, 19 | "platform": { 20 | "type": "string", 21 | "enum": ["web", "mob", "pc", "srv", "tv", "cnsl", "iot"], 22 | "maxLength": 255 23 | }, 24 | "dvce_created_tstamp": { 25 | "type": "integer" 26 | }, 27 | "dvce_sent_tstamp": { 28 | "type": "integer" 29 | }, 30 | "true_tstamp": { 31 | "type": "integer" 32 | }, 33 | "os_timezone": { 34 | "type": "string", 35 | "maxLength": 255 36 | }, 37 | "event": { 38 | "type": "string", 39 | "enum": ["se", "ev", "ue", "ad", "tr", "ti", "pv", "pp"] 40 | }, 41 | "txn_id": { 42 | "type": "integer" 43 | }, 44 | "event_id": { 45 | "type": "string", 46 | "maxLength": 36 47 | }, 48 | "v_tracker": { 49 | "type": "string", 50 | "maxLength": 100 51 | }, 52 | "domain_userid": { 53 | "type": "string", 54 | "maxLength": 36 55 | }, 56 | "network_userid": { 57 | "type": "string", 58 | "maxLength": 38 59 | }, 60 | "user_id": { 61 | "type": "string", 62 | "maxLength": 255 63 | }, 64 | "domain_sessionidx": { 65 | "type": "integer" 66 | }, 67 | "domain_sessionid": { 68 | "type": "string", 69 | "maxLength": 36 70 | }, 71 | "user_ipaddress": { 72 | "type": "string", 73 | "maxLength": 45 74 | }, 75 | "screen_res_width_x_height": { 76 | "type": "string", 77 | "pattern": "^[0-9]+x[0-9]+$" 78 | }, 79 | "page_url": { 80 | "type": "string", 81 | "maxLength": 4096 82 | }, 83 | "useragent": { 84 | "type": "string", 85 | "maxLength": 1000 86 | }, 87 | "page_title": { 88 | "type": "string", 89 | "maxLength": 2000 90 | }, 91 | "page_referer": { 92 | "type": "string", 93 | "maxLength": 4096 94 | }, 95 | "user_fingerprint": { 96 | "type": "integer" 97 | }, 98 | "br_cookies": { 99 | "type": "string", 100 | "enum": ["1", "0"] 101 | }, 102 | "br_lang": { 103 | "type": "string", 104 | "maxLength": 255 105 | }, 106 | "br_features_pdf": { 107 | "type": "string", 108 | "enum": ["1", "0"] 109 | }, 110 | "br_features_quicktime": { 111 | "type": "string", 112 | "enum": ["1", "0"] 113 | }, 114 | "br_features_realplayer": { 115 | "type": "string", 116 | "enum": ["1", "0"] 117 | }, 118 | "br_features_windowsmedia": { 119 | "type": "string", 120 | "enum": ["1", "0"] 121 | }, 122 | "br_features_director": { 123 | "type": "string", 124 | "enum": ["1", "0"] 125 | }, 126 | "br_features_flash": { 127 | "type": "string", 128 | "enum": ["1", "0"] 129 | }, 130 | "br_features_java": { 131 | "type": "string", 132 | "enum": ["1", "0"] 133 | }, 134 | "br_features_gears": { 135 | "type": "string", 136 | "enum": ["1", "0"] 137 | }, 138 | "br_features_silverlight": { 139 | "type": "string", 140 | "enum": ["1", "0"] 141 | }, 142 | "br_colordepth": { 143 | "type": "integer" 144 | }, 145 | "doc_width_x_height": { 146 | "type": "string", 147 | "pattern": "^[0-9]+x[0-9]+$" 148 | }, 149 | "doc_charset": { 150 | "type": "string", 151 | "maxLength": 128 152 | }, 153 | "browser_viewport_width_x_height": { 154 | "type": "string", 155 | "pattern": "^[0-9]+x[0-9]+$" 156 | }, 157 | "mac_address": { 158 | "type": "string", 159 | "maxLength": 36 160 | }, 161 | "pp_xoffset_min": { 162 | "type": "integer" 163 | }, 164 | "pp_xoffset_max": { 165 | "type": "integer" 166 | }, 167 | "pp_yoffset_min": { 168 | "type": "integer" 169 | }, 170 | "pp_yoffset_max": { 171 | "type": "integer" 172 | }, 173 | "tr_orderid": { 174 | "type": "string", 175 | "maxLength": 255 176 | }, 177 | "tr_affiliation": { 178 | "type": "string", 179 | "maxLength": 255 180 | }, 181 | "tr_total": { 182 | "type": "number" 183 | }, 184 | "tr_tax": { 185 | "type": "number" 186 | }, 187 | "tr_shipping": { 188 | "type": "number" 189 | }, 190 | "tr_city": { 191 | "type": "string", 192 | "maxLength": 255 193 | }, 194 | "tr_state": { 195 | "type": "string", 196 | "maxLength": 255 197 | }, 198 | "tr_country": { 199 | "type": "string", 200 | "maxLength": 255 201 | }, 202 | "tr_currency": { 203 | "type": "string", 204 | "maxLength": 255 205 | }, 206 | "ti_orderid": { 207 | "type": "string", 208 | "maxLength": 255 209 | }, 210 | "ti_sku": { 211 | "type": "string", 212 | "maxLength": 255 213 | }, 214 | "ti_name": { 215 | "type": "string", 216 | "maxLength": 255 217 | }, 218 | "ti_category": { 219 | "type": "string", 220 | "maxLength": 255 221 | }, 222 | "ti_price": { 223 | "type": "number" 224 | }, 225 | "ti_quantity": { 226 | "type": "integer" 227 | }, 228 | "ti_currency": { 229 | "type": "string", 230 | "maxLength": 255 231 | }, 232 | "se_category": { 233 | "type": "string", 234 | "maxLength": 255 235 | }, 236 | "se_action": { 237 | "type": "string", 238 | "maxLength": 255 239 | }, 240 | "se_label": { 241 | "type": "string", 242 | "maxLength": 255 243 | }, 244 | "se_property": { 245 | "type": "string", 246 | "maxLength": 255 247 | }, 248 | "se_value": { 249 | "type": "number" 250 | } 251 | }, 252 | "required": ["app_id", "platform", "event", "event_id", "v_tracker"], 253 | "custom_dependencies": { 254 | "se_category": { "event": "se" }, 255 | "se_action": {"event": "se" }, 256 | "tr_orderid": { "event": "tr" }, 257 | "tr_total": { "event": "tr" }, 258 | "ti_orderid": { "event": "ti" }, 259 | "ti_sku": { "event": "ti" }, 260 | "ti_quantity": { "event": "ti" }, 261 | "ti_price": { "event": "ti" }, 262 | "unstruct_event": { "event": "ue"}, 263 | "page_url": { "platform": "web" } 264 | } 265 | } -------------------------------------------------------------------------------- /lib/snowly.rb: -------------------------------------------------------------------------------- 1 | require 'active_support/core_ext' 2 | require "pry" 3 | require 'json' 4 | require 'rack' 5 | require 'json-schema' 6 | require 'snowly/validators/self_desc' 7 | require "snowly/version" 8 | require 'snowly/each_validator' 9 | require 'snowly/validator' 10 | require 'snowly/schema_cache' 11 | 12 | module Snowly 13 | mattr_accessor :development_iglu_resolver_path, :debug_mode, :logger 14 | 15 | @@development_iglu_resolver_path = ENV['DEVELOPMENT_IGLU_RESOLVER_PATH'] 16 | @@debug_mode = false 17 | @@logger = Logger.new(STDOUT) 18 | 19 | def self.config 20 | yield self 21 | end 22 | end 23 | 24 | -------------------------------------------------------------------------------- /lib/snowly/app/collector.rb: -------------------------------------------------------------------------------- 1 | require 'thin' 2 | require 'erb' 3 | require 'base64' 4 | require 'sinatra' 5 | require 'sinatra/reloader' if development? 6 | 7 | module Snowly 8 | module App 9 | class Collector < Sinatra::Base 10 | set :server, 'thin' 11 | GIF = Base64.decode64('R0lGODlhAQABAPAAAAAAAAAAACH5BAEAAAAALAAAAAABAAEAAAICRAEAOw==') 12 | configure :development do 13 | register Sinatra::Reloader 14 | end 15 | 16 | def handle_response(validator) 17 | content_type :json 18 | if validator.validate 19 | status 200 20 | if params[:debug] || Snowly.debug_mode 21 | content = validator.as_hash 22 | Snowly.logger.info content 23 | body(content.to_json) 24 | else 25 | content_type 'image/gif' 26 | Snowly::App::Collector::GIF 27 | end 28 | else 29 | status 422 30 | content = validator.as_hash 31 | Snowly.logger.error content 32 | body (content.to_json) 33 | end 34 | end 35 | 36 | get '/' do 37 | @resolved_schemas = if resolver = Snowly.development_iglu_resolver_path 38 | Dir[File.join(resolver,"/**/*")].select{ |e| File.file? e } 39 | else 40 | nil 41 | end 42 | erb :index 43 | end 44 | 45 | get '/i' do 46 | validator = Snowly::Validator.new request.query_string 47 | handle_response(validator) 48 | end 49 | 50 | get '/js' do 51 | erb :js 52 | end 53 | 54 | post '/com.snowplowanalytics.snowplow/tp2' do 55 | response.headers['Allow'] = 'HEAD,GET,PUT,POST,DELETE,OPTIONS' 56 | response.headers['Access-Control-Allow-Headers'] = 'X-Requested-With, X-HTTP-Method-Override, Content-Type, Cache-Control, Accept' 57 | response.headers['Access-Control-Allow-Credentials'] = 'true' 58 | response.headers['Access-Control-Allow-Origin'] = env['HTTP_ORIGIN'] || '*' 59 | payload = JSON.parse request.body.read 60 | validator = Snowly::Validator.new payload, batch: true 61 | handle_response(validator) 62 | end 63 | 64 | options '*' do 65 | response.headers['Allow'] = 'HEAD,GET,PUT,POST,DELETE,OPTIONS' 66 | response.headers['Access-Control-Allow-Headers'] = 'X-Requested-With, X-HTTP-Method-Override, Content-Type, Cache-Control, Accept' 67 | response.headers['Access-Control-Allow-Credentials'] = 'true' 68 | response.headers['Access-Control-Allow-Origin'] = env['HTTP_ORIGIN'] || '*' 69 | 200 70 | end 71 | end 72 | end 73 | end 74 | -------------------------------------------------------------------------------- /lib/snowly/app/views/index.erb: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | Snowly - Snowplow Request Validator 5 | 6 | 7 | 8 | 9 |
10 |
11 | 17 |

Snowly - Snowplow Request Validator

18 |
19 | 20 |
21 |

Test your snowplow implementation locally!

22 |

Snowly is a minimal collector implementation intended to validate your event tracking requests before emitting them to cloudfront or a closure collector.

23 |

When Snowly finds something wrong, it renders the parsed request along with its errors.

24 |

If everything is ok, Snowly delivers the default Snowplow pixel, unless you're using the debug mode.

25 |

Point your collector URL to <%= request.env['HTTP_HOST'] %> and have fun!

26 |

27 | See it working! 28 | Event with errors! 29 |

30 | <% unless Snowly.development_iglu_resolver_path %> 31 | 32 | <% end %> 33 |

34 | Use snowly -K to stop the collector. 35 |

36 |
37 | 38 |
39 |
40 |
41 |
42 | Current Configuration 43 |
version <%= Snowly::VERSION %>
44 |
45 | 46 | 47 | 48 | 49 | 50 | 51 | 52 | 53 | 54 | 55 | 56 | 57 | 58 | 59 | 60 | 61 | 62 | 63 | 64 | 65 | 66 |
ConfigurationValueDescription
Debug Mode<%= Snowly.debug_mode %>Renders parsed request instead of a pixel. Defaults to false
DEVELOPMENT_IGLU_RESOLVER_PATH<%= Snowly.development_iglu_resolver_path %>Local path for contexts and unstructured event schemas.
67 |
68 |
69 |
70 |
71 |
72 |
73 |
Resolved Schemas
74 |
75 | <% if @resolved_schemas and not @resolved_schemas == [] %> 76 |
    77 | <% @resolved_schemas.each do |r| %> 78 |
  • <%= r %>
    • 79 | <% end %> 80 |
81 | <% else %> 82 |

No resolved schemas

83 | <% end %> 84 |
85 |
86 |
87 |
88 |

Local Iglu Resolver

89 |

90 | Snowly must be able to find your custom context and unstructured event schemas. 91 | Just like the Resolver you may have already configured for the official ETL tools, Snowly needs a 92 | local path to find your custom schemas. You can store them under any path(eg: ~/schemas) 93 | Inside that folder you must create a resolver compatible structure: 94 | ~/schemas/com.yoursite/my_context/jsonschema/1-0-0
95 | ~/schemas/com.yoursite/my_event/jsonschema/1-0-0
96 | 1-0-0 is the file holding the schema. 97 |

98 |

99 | When you emmit events, use the schema path from the Resolver path
100 | { schema: 'iglu:com.yoursite/my_context/jsonschema/1-0-0', data: !some_schema_data! } 101 |

102 |

103 | Be sure to give your schemas an id, so Snowly can output more helpful validation error messages. 104 |

105 |
106 |
107 |
108 | 109 | -------------------------------------------------------------------------------- /lib/snowly/app/views/js.erb: -------------------------------------------------------------------------------- 1 | 3 | 15 | 16 | 17 | Small asynchronous website/webapp examples for snowplow.js 18 | 19 | 20 | 21 | 50 | 51 | 52 | 53 | 131 | 132 | 133 | 134 |

Small_asynchronous_examples_for_snowplow.js

135 | 136 |

Warning: if your browser's Do Not Track feature is enabled and respectDoNotTrack is enabled, all tracking will be prevented.

137 |

If you are viewing the page using a file URL, you must edit the script URL in the Snowplow tag to include an http scheme. Otherwise a file scheme will be inferred and the page will attempt to load sp.js from the local filesystem..

138 | 139 |

Press the buttons below to trigger individual tracking events:
140 |
141 |
142 |
143 |
144 | 145 |

146 | Link 147 |
148 | Ignored link 149 | 150 | 151 | 152 | -------------------------------------------------------------------------------- /lib/snowly/each_validator.rb: -------------------------------------------------------------------------------- 1 | # Performs the validation for the root attributes and associated contexts and unstructured events. 2 | require 'snowly/request' 3 | require 'snowly/protocol_schema_finder' 4 | require 'snowly/extensions/custom_dependencies' 5 | 6 | module Snowly 7 | class EachValidator 8 | attr_reader :request, :errors 9 | 10 | def initialize(payload) 11 | @request = Request.new payload 12 | @errors = [] 13 | end 14 | 15 | # If request is valid 16 | # @return [true, false] if valid 17 | def valid? 18 | @errors == [] 19 | end 20 | 21 | # Entry point for validation. 22 | def validate 23 | validate_root 24 | validate_associated 25 | valid? 26 | end 27 | 28 | def protocol_schema 29 | @protocol_schema ||= ProtocolSchemaFinder.new.schema 30 | end 31 | 32 | def as_hash 33 | { event_id: request.as_hash['event_id'], errors: errors, content: request.as_hash } 34 | end 35 | 36 | private 37 | 38 | # @return [Hash] all contexts content and schema definitions 39 | def associated_contexts 40 | load_contexts request.as_hash['contexts'] 41 | end 42 | 43 | # @return [Hash] all unstructured events content and schema definitions 44 | def associated_unstruct_event 45 | load_unstruct_event request.as_hash['unstruct_event'] 46 | end 47 | 48 | # @return [Array] all associated content 49 | def associated_elements 50 | (Array(associated_contexts) + Array(associated_unstruct_event)).compact 51 | end 52 | 53 | # Performs initial validation for associated contexts and loads their contents and definitions. 54 | # @return [Array] 55 | def load_contexts(hash) 56 | return unless hash 57 | response = [] 58 | unless hash['data'] 59 | @errors << "All custom contexts must be contain a `data` element" and return 60 | end 61 | schema = SchemaCache.instance[hash['schema']] 62 | response << { content: hash['data'], definition: schema, schema_name: hash['schema'] } 63 | unless hash['data'].is_a? Array 64 | @errors << "All custom contexts must be wrapped in an Array" and return 65 | end 66 | hash['data'].each do |data_item| 67 | schema = SchemaCache.instance[data_item['schema']] 68 | response << { content: data_item['data'], definition: schema, schema_name: data_item['schema'] } 69 | end 70 | response 71 | end 72 | 73 | def register_missing_schema(name) 74 | @errors << "#{ name } wasn't found in any resolvers." 75 | end 76 | 77 | # Performs initial validation for associated unstructured events and loads their contents and definitions. 78 | # @return [Array] 79 | def load_unstruct_event(hash) 80 | return unless hash 81 | response = [] 82 | unless hash['data'] 83 | @errors << "All custom unstruct event must be contain a `data` element" and return 84 | end 85 | outer_data = hash['data'] 86 | inner_data = outer_data['data'] 87 | response << { content: outer_data, definition: SchemaCache.instance[hash['schema']], schema_name: hash['schema'] } 88 | response << { content: inner_data, definition: SchemaCache.instance[outer_data['schema']], schema_name: outer_data['schema'] } 89 | response 90 | end 91 | 92 | # Validates associated contexts and unstructured events 93 | def validate_associated 94 | return unless associated_elements 95 | missing_schemas, valid_elements = associated_elements.partition{ |el| el[:definition].blank? } 96 | missing_schemas.each { |element| register_missing_schema(element[:schema_name]) } 97 | valid_elements.each do |element| 98 | this_error = JSON::Validator.fully_validate JSON.parse(element[:definition]), element[:content] 99 | @errors += this_error if this_error.count > 0 100 | end 101 | end 102 | 103 | # Validates root attributes for the events table 104 | def validate_root 105 | this_error = JSON::Validator.fully_validate protocol_schema, request.as_hash 106 | @errors += this_error if this_error.count > 0 107 | end 108 | end 109 | end -------------------------------------------------------------------------------- /lib/snowly/extensions/custom_dependencies.rb: -------------------------------------------------------------------------------- 1 | # Extended validation to require custom dependencies 2 | # This validation allows the schema designer to define requirements based on other attribute content. 3 | # @example Require apartment number if address type is `apartment` 4 | # { 5 | # 'address': { 'type': string' }, 6 | # 'adress_type': { 'type': 'string', 'enum': ['house', 'apartment'] }, 7 | # 'apartment_number': { 'type': 'number'}, 8 | # 'custom_dependencies': { 9 | # apartment_number: { 'address_type': 'apartment' } 10 | # } 11 | # } 12 | # In this example if the address type is 'house', apartment_number if not required. 13 | # It only becomes a requirement if address type is set to 'apartment'. 14 | require 'json-schema/attribute' 15 | 16 | class CustomDependenciesAttribute < JSON::Schema::Attribute 17 | def self.validate(current_schema, data, fragments, processor, validator, options = {}) 18 | return unless data.is_a?(Hash) 19 | current_schema.schema['custom_dependencies'].each do |property, dependency_value| 20 | next unless accept_value?(dependency_value) 21 | case dependency_value 22 | when Array 23 | dependency_value.each do |dependency_hash| 24 | validate_dependency(current_schema, data, property, dependency_hash, fragments, processor, self, options) 25 | end 26 | when Hash 27 | validate_dependency(current_schema, data, property, dependency_value, fragments, processor, self, options) 28 | end 29 | end 30 | end 31 | 32 | def self.validate_dependency(schema, data, property, dependency_hash, fragments, processor, attribute, options) 33 | key, value = Array(dependency_hash).flatten 34 | return unless data[key.to_s] == value.to_s 35 | return if data.has_key?(property.to_s) && not(data[property.to_s].blank?) 36 | message = "The property '#{build_fragment(fragments)}' did not contain a required property of '#{property}' when property '#{key}' is '#{value}'" 37 | validation_error(processor, message, fragments, schema, attribute, options[:record_errors]) 38 | end 39 | 40 | def self.accept_value?(value) 41 | value.is_a?(Array) || value.is_a?(Hash) 42 | end 43 | end 44 | 45 | # Registers custom dependencies for Draft-4. This used when evaluating the protocol schema 46 | # snowly/schemas/snowplow_protocol.json 47 | class RootExtendedSchema < JSON::Schema::Validator 48 | def initialize 49 | super 50 | extend_schema_definition("http://json-schema.org/draft-04/schema#") 51 | @attributes["custom_dependencies"] = CustomDependenciesAttribute 52 | @uri = URI.parse("http://json-schema.org/draft-04/schema") 53 | end 54 | JSON::Validator.register_validator(self.new) 55 | end 56 | 57 | # Registers custom dependencies for Snowplot Self-Describing schema. Contexts and Unstructure events derive from it. 58 | class DescExtendedSchema < JSON::Schema::Validator 59 | def initialize 60 | super 61 | extend_schema_definition("http://iglucentral.com/schemas/com.snowplowanalytics.self-desc/schema/jsonschema/1-0-0#") 62 | @attributes["custom_dependencies"] = CustomDependenciesAttribute 63 | @uri = URI.parse("http://iglucentral.com/schemas/com.snowplowanalytics.self-desc/schema/jsonschema/1-0-0#") 64 | end 65 | JSON::Validator.register_validator(self.new) 66 | end 67 | -------------------------------------------------------------------------------- /lib/snowly/protocol_schema_finder.rb: -------------------------------------------------------------------------------- 1 | module Snowly 2 | class ProtocolSchemaFinder 3 | PROTOCOL_FILE_NAME = 'snowplow_protocol.json' 4 | attr_reader :schema 5 | 6 | def initialize(custom_schema = nil) 7 | @custom_schema = custom_schema 8 | @schema = load_protocol_schema 9 | end 10 | 11 | def find_protocol_schema 12 | return @custom_schema if @custom_schema 13 | if resolver && alternative_protocol_schema 14 | alternative_protocol_schema 15 | else 16 | File.expand_path("../../schemas/#{PROTOCOL_FILE_NAME}", __FILE__) 17 | end 18 | end 19 | 20 | def resolver 21 | Snowly.development_iglu_resolver_path 22 | end 23 | 24 | def alternative_protocol_schema 25 | Dir[File.join(resolver,"/**/*")].select{ |f| File.basename(f) == PROTOCOL_FILE_NAME }[0] 26 | end 27 | 28 | # Loads the protocol schema created to describe snowplow events table attributes 29 | # @return [Hash] parsed schema 30 | def load_protocol_schema 31 | JSON.parse File.read(find_protocol_schema) 32 | end 33 | end 34 | end -------------------------------------------------------------------------------- /lib/snowly/request.rb: -------------------------------------------------------------------------------- 1 | require 'snowly/transformer' 2 | module Snowly 3 | class Request 4 | 5 | attr_reader :parsed_payload 6 | 7 | def initialize(payload) 8 | @parsed_payload = payload.is_a?(String) ? parse_query(payload) : payload 9 | end 10 | 11 | # Retuns request as json, after transforming parameters into column names 12 | # @return [String] encoded JSON 13 | def as_json 14 | @json ||= as_hash.to_json 15 | end 16 | 17 | # Retuns request as hash, after transforming parameters into column names 18 | # @return [Hash] 19 | def as_hash 20 | @hash ||= Transformer.transform(parsed_payload) 21 | end 22 | 23 | # Returns query parameters as hash 24 | # @return [Hash] 25 | def parse_query(query_string) 26 | @parsed_query ||= Rack::Utils.parse_nested_query(query_string) 27 | end 28 | 29 | end 30 | end -------------------------------------------------------------------------------- /lib/snowly/schema_cache.rb: -------------------------------------------------------------------------------- 1 | # Caches schemas found during validation so they don't have to be 2 | # retrieved a second time. Also uses the resolvers convert the iglu: location to an actual address (local or remote)/ 3 | require 'singleton' 4 | module Snowly 5 | class SchemaCache 6 | include Singleton 7 | SNOWPLOW_IGLU_RESOLVER = 'http://iglucentral.com/schemas/' 8 | @@schema_cache = {} 9 | 10 | # Provides easy access to the schema cache based on its registered key 11 | # @param location [String] Location provided in the schema 12 | # @return [String] Json for schema 13 | def [](location) 14 | @@schema_cache[location] || save_in_cache(location) 15 | end 16 | 17 | # Resets the schema cache 18 | def reset_cache 19 | @@schema_cache = {} 20 | end 21 | 22 | # Accessor to the global cache 23 | def cache 24 | @@schema_cache 25 | end 26 | 27 | private 28 | 29 | def external?(location) 30 | location.match(/^(http|https):\/\//) 31 | end 32 | 33 | # Translate an iglu address to an actual local or remote location 34 | # @param location [String] 35 | # @param resolver [String] local or remote path to look for the schema 36 | # @return [String] Schema's actual location 37 | def resolve(location, resolver) 38 | path = location.sub(/^iglu\:/, '') 39 | File.join resolver, path 40 | end 41 | 42 | # Caches the schema content under its original location name 43 | # @param location [String] 44 | # @return [String] schema content 45 | def save_in_cache(location) 46 | content = begin 47 | full_path = resolve(location, (Snowly.development_iglu_resolver_path || SNOWPLOW_IGLU_RESOLVER) ) 48 | external?(full_path) ? Net::HTTP.get(URI(full_path)) : File.read(full_path) 49 | rescue 50 | Snowly.logger.warn "Could't locate #{location} in development resolver. Attemping IgluCentral Server..." 51 | full_path = resolve(location, SNOWPLOW_IGLU_RESOLVER) 52 | begin 53 | result = Net::HTTP.get(URI(full_path)) 54 | JSON.load(result) && result 55 | rescue 56 | Snowly.logger.error "#{location} schema is not available in any resolver" 57 | return nil 58 | end 59 | end 60 | @@schema_cache[location] = content 61 | end 62 | end 63 | end -------------------------------------------------------------------------------- /lib/snowly/transformer.rb: -------------------------------------------------------------------------------- 1 | # Maps query string parameters to column names to provide more helpful references on validations. 2 | module Snowly 3 | module Transformer 4 | module_function 5 | # Boolean fields are mapped to string because the tracker sends '1' or '0' in the query string. 6 | # The actual conversion to boolean happens during the enrichment phase. 7 | MAP = { 8 | "e" => { field: "event", type: 'string' }, 9 | "ip" => { field: "user_ipaddress", type: "string" }, 10 | "aid" => { field: "app_id", type: "string" }, 11 | "p" => { field: "platform", type: "string" }, 12 | "tid" => { field: "txn_id", type: "integer" }, 13 | "uid" => { field: "user_id", type: "string" }, 14 | "duid" => { field: "domain_userid", type: "string" }, 15 | "nuid" => { field: "network_userid", type: "string" }, 16 | "ua" => { field: "useragent", type: "string" }, 17 | "fp" => { field: "user_fingerprint", type: "integer" }, 18 | "vid" => { field: "domain_sessionidx", type: "integer" }, 19 | "sid" => { field: "domain_sessionid", type: "string" }, 20 | "dtm" => { field: "dvce_created_tstamp", type: "integer" }, 21 | "ttm" => { field: "true_tstamp", type: "integer" }, 22 | "stm" => { field: "dvce_sent_tstamp", type: "integer" }, 23 | "tna" => { field: "name_tracker", type: "string" }, 24 | "tv" => { field: "v_tracker", type: "string" }, 25 | "cv" => { field: "v_collector", type: "string" }, 26 | "lang" => { field: "br_lang", type: "string" }, 27 | "f_pdf" => { field: "br_features_pdf", type: "string" }, 28 | "f_fla" => { field: "br_features_flash", type: "string" }, 29 | "f_java" => { field: "br_features_java", type: "string" }, 30 | "f_dir" => { field: "br_features_director", type: "string" }, 31 | "f_qt" => { field: "br_features_quicktime", type: "string" }, 32 | "f_realp" => { field: "br_features_realplayer", type: "string" }, 33 | "f_wma" => { field: "br_features_windowsmedia", type: "string" }, 34 | "f_gears" => { field: "br_features_gears", type: "string" }, 35 | "f_ag" => { field: "br_features_silverlight", type: "string" }, 36 | "cookie" => { field: "br_cookies", type: "string" }, 37 | "res" => { field: "screen_res_width_x_height", type: "string" }, 38 | "cd" => { field: "br_colordepth", type: "integer" }, 39 | "tz" => { field: "os_timezone", type: "string" }, 40 | "refr" => { field: "page_referrer", type: "string" }, 41 | "url" => { field: "page_url", type: "string" }, 42 | "page" => { field: "page_title", type: "string" }, 43 | "cs" => { field: "doc_charset", type: "string" }, 44 | "ds" => { field: "doc_width_x_height", type: "string" }, 45 | "vp" => { field: "browser_viewport_width_x_height", type: "string" }, 46 | "eid" => { field: "event_id", type: "string" }, 47 | "co" => { field: "contexts", type: "json" }, 48 | "cx" => { field: "contexts", type: "base64" }, 49 | "ev_ca" => { field: "se_category", type: "string" }, 50 | "ev_ac" => { field: "se_action", type: "string" }, 51 | "ev_la" => { field: "se_label", type: "string" }, 52 | "ev_pr" => { field: "se_property", type: "string" }, 53 | "ev_va" => { field: "se_value", type: "string" }, 54 | "se_ca" => { field: "se_category", type: "string" }, 55 | "se_ac" => { field: "se_action", type: "string" }, 56 | "se_la" => { field: "se_label", type: "string" }, 57 | "se_pr" => { field: "se_property", type: "string" }, 58 | "se_va" => { field: "se_value", type: "number" }, 59 | "ue_pr" => { field: "unstruct_event", type: "json" }, 60 | "ue_px" => { field: "unstruct_event", type: "base64" }, 61 | "tr_id" => { field: "tr_orderid", type: "string" }, 62 | "tr_af" => { field: "tr_affiliation", type: "string" }, 63 | "tr_tt" => { field: "tr_total", type: "number" }, 64 | "tr_tx" => { field: "tr_tax", type: "number" }, 65 | "tr_sh" => { field: "tr_shipping", type: "number" }, 66 | "tr_ci" => { field: "tr_city", type: "string" }, 67 | "tr_st" => { field: "tr_state", type: "string" }, 68 | "tr_co" => { field: "tr_country", type: "string" }, 69 | "ti_id" => { field: "ti_orderid", type: "string" }, 70 | "ti_sk" => { field: "ti_sku", type: "string" }, 71 | "ti_na" => { field: "ti_name", type: "string" }, 72 | "ti_nm" => { field: "ti_name", type: "string" }, 73 | "ti_ca" => { field: "ti_category", type: "string" }, 74 | "ti_pr" => { field: "ti_price", type: "number" }, 75 | "ti_qu" => { field: "ti_quantity", type: "integer" }, 76 | "pp_mix" => { field: "pp_xoffset_min", type: "integer" }, 77 | "pp_max" => { field: "pp_xoffset_max", type: "integer" }, 78 | "pp_miy" => { field: "pp_yoffset_min", type: "integer" }, 79 | "pp_may" => { field: "pp_yoffset_max", type: "integer" }, 80 | "tr_cu" => { field: "tr_currency", type: "string" }, 81 | "ti_cu" => { field: "ti_currency", type: "integer" } 82 | } 83 | 84 | # Transforms the request params into column names 85 | # @param parsed_query [Hash] hash using parameter names for keys 86 | # @return [Hash] hash using column names for keys 87 | def transform(parsed_query) 88 | parsed_query.inject({}) do |all, (key, value)| 89 | if node = MAP[key] 90 | field = node[:field] 91 | all[field] = convert(value, node[:type]) 92 | end 93 | all 94 | end 95 | end 96 | 97 | # Tries to cast or parse each value so they can be properly validated by json-schema 98 | # If the casting fails, leaves the value as string and it will be caught by the valication 99 | # @param value [String] 100 | # @param type [String] the intended param type 101 | def convert(value, type) 102 | begin 103 | case type 104 | when 'json' then JSON.parse(value) 105 | when 'base64' then JSON.parse(Base64.decode64(value)) 106 | when 'integer' then Integer(value) 107 | when 'number' then Float(value) 108 | else 109 | value.to_s 110 | end 111 | rescue ArgumentError 112 | value.to_s 113 | end 114 | end 115 | 116 | end 117 | end -------------------------------------------------------------------------------- /lib/snowly/validator.rb: -------------------------------------------------------------------------------- 1 | require 'snowly/each_validator' 2 | module Snowly 3 | class Validator 4 | attr_reader :validators 5 | 6 | def initialize(payload, batch: false) 7 | @validators = if batch 8 | payload['data'].map do |req| 9 | EachValidator.new(req) 10 | end 11 | else 12 | [ EachValidator.new(payload) ] 13 | end 14 | end 15 | 16 | def validate 17 | validators.each(&:validate) 18 | valid? 19 | end 20 | 21 | def valid? 22 | validators.all? { |v| v.valid? } 23 | end 24 | 25 | def as_hash 26 | validators.map(&:as_hash) 27 | end 28 | end 29 | end -------------------------------------------------------------------------------- /lib/snowly/validators/self_desc.rb: -------------------------------------------------------------------------------- 1 | # Register a validator for the self-describing schema. 2 | # This is required to allow extended validations being attached to it. 3 | require 'json-schema/validators/draft4' 4 | module JSON 5 | class Schema 6 | class SelfDesc < Draft4 7 | URL = "http://iglucentral.com/schemas/com.snowplowanalytics.self-desc/schema/jsonschema/1-0-0#" 8 | def initialize 9 | super 10 | @uri = JSON::Util::URI.parse(URL) 11 | end 12 | 13 | JSON::Validator.register_validator(self.new) 14 | end 15 | end 16 | end -------------------------------------------------------------------------------- /lib/snowly/version.rb: -------------------------------------------------------------------------------- 1 | module Snowly 2 | VERSION = "0.2.4" 3 | end 4 | -------------------------------------------------------------------------------- /snowly.gemspec: -------------------------------------------------------------------------------- 1 | # coding: utf-8 2 | lib = File.expand_path('../lib', __FILE__) 3 | $LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib) 4 | require 'snowly/version' 5 | 6 | Gem::Specification.new do |spec| 7 | spec.name = "snowly" 8 | spec.version = Snowly::VERSION 9 | spec.authors = ["Alexandre Angelim"] 10 | spec.email = ["angelim@angelim.com.br"] 11 | 12 | spec.summary = %q{Snowplow Request Validator} 13 | spec.description = %q{Snowly is a minimal collector implementation intended to validate your event tracking requests before emitting them to cloudfront or a closure collector.} 14 | spec.homepage = "https://github.com/angelim/snowly" 15 | spec.license = "MIT" 16 | 17 | spec.files = `git ls-files -z`.split("\x0").reject { |f| f.match(%r{^(test|spec|features)/}) } 18 | spec.bindir = "bin" 19 | spec.executables << 'snowly' 20 | spec.require_paths = ["lib"] 21 | 22 | 23 | spec.add_dependency 'json-schema', '~> 2.6' 24 | spec.add_dependency 'rack', '~> 1.6' 25 | spec.add_dependency 'activesupport', "~> 3.0" 26 | spec.add_dependency 'sinatra', '~> 1.4' 27 | spec.add_dependency 'sinatra-contrib', '~> 1.4' 28 | spec.add_dependency 'vegas', '~> 0.1' 29 | spec.add_dependency 'thin', '~> 1.7' 30 | spec.add_dependency 'pry-byebug', '~> 3.3' 31 | 32 | spec.add_development_dependency 'bundler', '~> 1.11' 33 | spec.add_development_dependency 'rake', '~> 10.0' 34 | spec.add_development_dependency 'rspec', '~> 3.0' 35 | spec.add_development_dependency 'snowplow-tracker', '~> 0.5' 36 | spec.add_development_dependency 'webmock', '~> 2.0' 37 | spec.add_development_dependency "shotgun", '~> 0.9' 38 | end 39 | -------------------------------------------------------------------------------- /spec/fixtures/snowly/context_test_0/jsonschema/1-0-0: -------------------------------------------------------------------------------- 1 | { 2 | "$schema": "http://iglucentral.com/schemas/com.snowplowanalytics.self-desc/schema/jsonschema/1-0-0#", 3 | "id": "test_context", 4 | "description": "Test context", 5 | "self": { 6 | "vendor": "snowly", 7 | "name": "test_context", 8 | "format": "jsonschema", 9 | "version": "1-0-0" 10 | }, 11 | 12 | "type": "object", 13 | "properties": { 14 | "name": { 15 | "type": "string", 16 | "maxLength": 10 17 | }, 18 | "age": { 19 | "type": "integer", 20 | "maximum": 100 21 | } 22 | }, 23 | "required": ["name", "age"], 24 | "additionalProperties": false 25 | } 26 | -------------------------------------------------------------------------------- /spec/fixtures/snowly/context_test_1/jsonschema/1-0-0: -------------------------------------------------------------------------------- 1 | { 2 | "$schema": "http://iglucentral.com/schemas/com.snowplowanalytics.self-desc/schema/jsonschema/1-0-0#", 3 | "id": "test_context_1", 4 | "description": "Context 1", 5 | "self": { 6 | "vendor": "snowly", 7 | "name": "test_context", 8 | "format": "jsonschema", 9 | "version": "1-0-1" 10 | }, 11 | 12 | "type": "object", 13 | "properties": { 14 | "street": { 15 | "type": "string", 16 | "maxLength": 10 17 | }, 18 | "number": { 19 | "type": "integer", 20 | "maximum": 100 21 | } 22 | }, 23 | "custom_dependencies": { 24 | "number": { "street": "home" } 25 | }, 26 | "additionalProperties": false 27 | } 28 | -------------------------------------------------------------------------------- /spec/fixtures/snowly/event_test/jsonschema/1-0-0: -------------------------------------------------------------------------------- 1 | { 2 | "$schema": "http://iglucentral.com/schemas/com.snowplowanalytics.self-desc/schema/jsonschema/1-0-0#", 3 | "id": "unstruct_event", 4 | "description": "Unstructured Event Test", 5 | "self": { 6 | "vendor": "snowly", 7 | "name": "test_ue", 8 | "format": "jsonschema", 9 | "version": "1-0-0" 10 | }, 11 | "type": "object", 12 | "properties": { 13 | "category": { 14 | "type": "string", 15 | "maxLength": 50 16 | }, 17 | "name": { 18 | "type": "string", 19 | "maxLength": 50 20 | }, 21 | "elapsed_time": { 22 | "type": "integer", 23 | "maximum": 2147483647 24 | }, 25 | "object_id": { 26 | "type": "string", 27 | "maxLength": 50 28 | }, 29 | "string_property_name": { 30 | "type": "string", 31 | "maxLength": 20 32 | }, 33 | "string_property_value": { 34 | "type": "string", 35 | "maxLength": 255 36 | }, 37 | "number_property_name": { 38 | "type": "string", 39 | "maxLength": 20 40 | }, 41 | "number_property_value": { 42 | "type": "number" 43 | } 44 | }, 45 | "additionalProperties": false 46 | } 47 | 48 | -------------------------------------------------------------------------------- /spec/fixtures/snowplow_context.json: -------------------------------------------------------------------------------- 1 | { 2 | "$schema": "http://iglucentral.com/schemas/com.snowplowanalytics.self-desc/schema/jsonschema/1-0-0#", 3 | "description": "Schema for custom contexts", 4 | "self": { 5 | "vendor": "com.snowplowanalytics.snowplow", 6 | "name": "contexts", 7 | "format": "jsonschema", 8 | "version": "1-0-1" 9 | }, 10 | 11 | "type": "array", 12 | 13 | "items": { 14 | 15 | "type": "object", 16 | 17 | "properties": { 18 | 19 | "schema": { 20 | "type": "string", 21 | "pattern": "^iglu:[a-zA-Z0-9-_.]+/[a-zA-Z0-9-_]+/[a-zA-Z0-9-_]+/[0-9]+-[0-9]+-[0-9]+$" 22 | }, 23 | 24 | "data": {} 25 | }, 26 | 27 | "required": ["schema", "data"], 28 | "additionalProperties": false 29 | } 30 | } 31 | -------------------------------------------------------------------------------- /spec/fixtures/snowplow_ue.json: -------------------------------------------------------------------------------- 1 | { 2 | "$schema": "http://iglucentral.com/schemas/com.snowplowanalytics.self-desc/schema/jsonschema/1-0-0#", 3 | "description": "Schema for a Snowplow unstructured event", 4 | "self": { 5 | "vendor": "com.snowplowanalytics.snowplow", 6 | "name": "unstruct_event", 7 | "format": "jsonschema", 8 | "version": "1-0-0" 9 | }, 10 | 11 | "type": "object", 12 | 13 | "properties": { 14 | 15 | "schema": { 16 | "type": "string", 17 | "pattern": "^iglu:[a-zA-Z0-9-_.]+/[a-zA-Z0-9-_]+/[a-zA-Z0-9-_]+/[0-9]+-[0-9]+-[0-9]+$" 18 | }, 19 | 20 | "data": {} 21 | }, 22 | 23 | "required": ["schema", "data"], 24 | "additionalProperties": false 25 | } 26 | -------------------------------------------------------------------------------- /spec/protocol_resolver/snowplow_protocol.json: -------------------------------------------------------------------------------- 1 | { 2 | "$schema": "http://json-schema.org/draft-04/schema#", 3 | "id": "snowplow_protocol.json", 4 | "description": "Alternative Snowplow Protocol", 5 | "type": "object", 6 | "properties": { 7 | "name_tracker": { 8 | "type": "string", 9 | "maxLength": 1 10 | }, 11 | "event_vendor": { 12 | "type": "string", 13 | "maxLength": 1000 14 | }, 15 | "app_id": { 16 | "type": "string", 17 | "maxLength": 255 18 | }, 19 | "platform": { 20 | "type": "string", 21 | "enum": ["pc"], 22 | "maxLength": 255 23 | }, 24 | "dvce_created_tstamp": { 25 | "type": "integer" 26 | }, 27 | "dvce_sent_tstamp": { 28 | "type": "integer" 29 | }, 30 | "true_tstamp": { 31 | "type": "integer" 32 | }, 33 | "os_timezone": { 34 | "type": "string", 35 | "maxLength": 255 36 | }, 37 | "event": { 38 | "type": "string", 39 | "enum": ["se", "ev", "ue", "ad", "tr", "ti", "pv", "pp"] 40 | }, 41 | "txn_id": { 42 | "type": "integer" 43 | }, 44 | "event_id": { 45 | "type": "string", 46 | "maxLength": 36 47 | }, 48 | "v_tracker": { 49 | "type": "string", 50 | "maxLength": 100 51 | }, 52 | "domain_userid": { 53 | "type": "string", 54 | "maxLength": 36 55 | }, 56 | "network_userid": { 57 | "type": "string", 58 | "maxLength": 38 59 | }, 60 | "user_id": { 61 | "type": "string", 62 | "maxLength": 255 63 | }, 64 | "domain_sessionidx": { 65 | "type": "integer" 66 | }, 67 | "domain_sessionid": { 68 | "type": "string", 69 | "maxLength": 36 70 | }, 71 | "user_ipaddress": { 72 | "type": "string", 73 | "maxLength": 45 74 | }, 75 | "screen_res_width_x_height": { 76 | "type": "string", 77 | "pattern": "^[0-9]+x[0-9]+$" 78 | }, 79 | "page_url": { 80 | "type": "string", 81 | "maxLength": 4096 82 | }, 83 | "useragent": { 84 | "type": "string", 85 | "maxLength": 1000 86 | }, 87 | "page_title": { 88 | "type": "string", 89 | "maxLength": 2000 90 | }, 91 | "page_referer": { 92 | "type": "string", 93 | "maxLength": 4096 94 | }, 95 | "user_fingerprint": { 96 | "type": "integer" 97 | }, 98 | "br_cookies": { 99 | "type": "string", 100 | "enum": ["1", "0"] 101 | }, 102 | "br_lang": { 103 | "type": "string", 104 | "maxLength": 255 105 | }, 106 | "br_features_pdf": { 107 | "type": "string", 108 | "enum": ["1", "0"] 109 | }, 110 | "br_features_quicktime": { 111 | "type": "string", 112 | "enum": ["1", "0"] 113 | }, 114 | "br_features_realplayer": { 115 | "type": "string", 116 | "enum": ["1", "0"] 117 | }, 118 | "br_features_windowsmedia": { 119 | "type": "string", 120 | "enum": ["1", "0"] 121 | }, 122 | "br_features_director": { 123 | "type": "string", 124 | "enum": ["1", "0"] 125 | }, 126 | "br_features_flash": { 127 | "type": "string", 128 | "enum": ["1", "0"] 129 | }, 130 | "br_features_java": { 131 | "type": "string", 132 | "enum": ["1", "0"] 133 | }, 134 | "br_features_gears": { 135 | "type": "string", 136 | "enum": ["1", "0"] 137 | }, 138 | "br_features_silverlight": { 139 | "type": "string", 140 | "enum": ["1", "0"] 141 | }, 142 | "br_colordepth": { 143 | "type": "integer" 144 | }, 145 | "doc_width_x_height": { 146 | "type": "string", 147 | "pattern": "^[0-9]+x[0-9]+$" 148 | }, 149 | "doc_charset": { 150 | "type": "string", 151 | "maxLength": 128 152 | }, 153 | "browser_viewport_width_x_height": { 154 | "type": "string", 155 | "pattern": "^[0-9]+x[0-9]+$" 156 | }, 157 | "mac_address": { 158 | "type": "string", 159 | "maxLength": 36 160 | }, 161 | "pp_xoffset_min": { 162 | "type": "integer" 163 | }, 164 | "pp_xoffset_max": { 165 | "type": "integer" 166 | }, 167 | "pp_yoffset_min": { 168 | "type": "integer" 169 | }, 170 | "pp_yoffset_max": { 171 | "type": "integer" 172 | }, 173 | "tr_orderid": { 174 | "type": "string", 175 | "maxLength": 255 176 | }, 177 | "tr_affiliation": { 178 | "type": "string", 179 | "maxLength": 255 180 | }, 181 | "tr_total": { 182 | "type": "number" 183 | }, 184 | "tr_tax": { 185 | "type": "number" 186 | }, 187 | "tr_shipping": { 188 | "type": "number" 189 | }, 190 | "tr_city": { 191 | "type": "string", 192 | "maxLength": 255 193 | }, 194 | "tr_state": { 195 | "type": "string", 196 | "maxLength": 255 197 | }, 198 | "tr_country": { 199 | "type": "string", 200 | "maxLength": 255 201 | }, 202 | "tr_currency": { 203 | "type": "string", 204 | "maxLength": 255 205 | }, 206 | "ti_orderid": { 207 | "type": "string", 208 | "maxLength": 255 209 | }, 210 | "ti_sku": { 211 | "type": "string", 212 | "maxLength": 255 213 | }, 214 | "ti_name": { 215 | "type": "string", 216 | "maxLength": 255 217 | }, 218 | "ti_category": { 219 | "type": "string", 220 | "maxLength": 255 221 | }, 222 | "ti_price": { 223 | "type": "number" 224 | }, 225 | "ti_quantity": { 226 | "type": "integer" 227 | }, 228 | "ti_currency": { 229 | "type": "string", 230 | "maxLength": 255 231 | }, 232 | "se_category": { 233 | "type": "string", 234 | "maxLength": 255 235 | }, 236 | "se_action": { 237 | "type": "string", 238 | "maxLength": 255 239 | }, 240 | "se_label": { 241 | "type": "string", 242 | "maxLength": 255 243 | }, 244 | "se_property": { 245 | "type": "string", 246 | "maxLength": 255 247 | }, 248 | "se_value": { 249 | "type": "number" 250 | } 251 | }, 252 | "required": ["app_id", "platform", "event", "event_id", "v_tracker", "useragent"], 253 | "custom_dependencies": { 254 | "se_category": { "event": "se" }, 255 | "se_action": {"event": "se" }, 256 | "tr_orderid": { "event": "tr" }, 257 | "tr_total": { "event": "tr" }, 258 | "ti_orderid": { "event": "ti" }, 259 | "ti_sku": { "event": "ti" }, 260 | "ti_quantity": { "event": "ti" }, 261 | "ti_price": { "event": "ti" }, 262 | "unstruct_event": { "event": "ue"}, 263 | "page_url": { "platform": "web" } 264 | } 265 | } -------------------------------------------------------------------------------- /spec/snowly/app/collector_spec.rb: -------------------------------------------------------------------------------- 1 | require 'spec_helper' 2 | 3 | describe "Collector" do 4 | describe 'GET #index' do 5 | it "should allow accessing the home page" do 6 | get '/' 7 | expect(last_response).to be_ok 8 | end 9 | 10 | it 'has showcase links' do 11 | get '/' 12 | expect(last_response.body).to include("/i?") 13 | end 14 | context 'when the local resolver path is set' do 15 | before { Snowly.development_iglu_resolver_path = 'myresolverpath' } 16 | it 'shows local iglu resolver' do 17 | get '/' 18 | expect(last_response.body).to include("myresolverpath") 19 | end 20 | end 21 | context 'when no resolver has been set' do 22 | before { Snowly.development_iglu_resolver_path = nil } 23 | it 'shows warning' do 24 | get '/' 25 | expect(last_response.body).to include("The Local Iglu Resolver Path is missing") 26 | end 27 | end 28 | context 'when no schemas were resolved' do 29 | before { Snowly.development_iglu_resolver_path = nil } 30 | it 'shows empty resolver message' do 31 | get '/' 32 | expect(last_response.body).to include("No resolved schemas") 33 | end 34 | end 35 | context 'when there are resolved schemas' do 36 | before { Snowly.development_iglu_resolver_path = File.expand_path("../../../fixtures", __FILE__) } 37 | it 'shows schemas in local resolver' do 38 | get '/' 39 | expect(last_response.body).to include("fixtures/snowly/context_test_0/jsonschema/1-0-0") 40 | end 41 | end 42 | end 43 | describe 'GET /i' do 44 | context 'with a valid request' do 45 | let(:valid_request) { '/i?&e=pv&page=Root%20README&url=http%3A%2F%2Fgithub.com%2Fsnowplow%2Fsnowplow&aid=snowplow&p=web&tv=no-js-0.1.0&ua=firefox&&eid=u2i3' } 46 | it 'responds with 200' do 47 | get valid_request 48 | expect(last_response).to be_ok 49 | end 50 | context 'when in production mode' do 51 | it 'responds with image content type' do 52 | get valid_request 53 | expect(last_response.content_type).to eq 'image/gif' 54 | end 55 | end 56 | context 'when in debug mode' do 57 | before { Snowly.debug_mode = true } 58 | it 'responds with json content type' do 59 | get valid_request 60 | expect(last_response.content_type).to eq 'application/json' 61 | end 62 | end 63 | end 64 | context 'with an invalid request' do 65 | before { Snowly.debug_mode = false } 66 | let(:invalid_request) { '/i?&e=pv&page=Root%20README&url=http%3A%2F%2Fgithub.com%2Fsnowplow%2Fsnowplow&aid=snowplow&p=i&tv=no-js-0.1.0' } 67 | it 'responds with 500' do 68 | get invalid_request 69 | expect(last_response).not_to be_ok 70 | end 71 | it 'renders errors' do 72 | get invalid_request 73 | expect(last_response.body).to include("errors") 74 | end 75 | it 'always responds with json content type' do 76 | get invalid_request 77 | expect(last_response.content_type).to eq 'application/json' 78 | end 79 | end 80 | end 81 | describe 'POST /com.snowplowanalytics.snowplow/tp2' do 82 | let(:url) { '/com.snowplowanalytics.snowplow/tp2' } 83 | context 'with a valid request' do 84 | let(:valid_params) { {"data" => [{ "e"=>"pv", "page"=>"Root README", "url"=>"http://github.com/snowplow/snowplow", "aid"=>"snowplow", "p"=>"web", "tv"=>"no-js-0.1.0", "ua"=>"firefox", "eid"=>"u2i3" }] } } 85 | it 'responds with 200' do 86 | post url, JSON.dump(valid_params) 87 | expect(last_response).to be_ok 88 | end 89 | end 90 | context 'with an invalid request' do 91 | let(:invalid_params) { { "data" => [{ "e"=>"pv", "page"=>"Root README", "url"=>"http://github.com/snowplow/snowplow", "aid"=>"snowplow", "p"=>"web", "tv"=>"no-js-0.1.0" }] } } 92 | it 'responds with 500' do 93 | post url, invalid_params 94 | expect(last_response).not_to be_ok 95 | end 96 | it 'renders errors' do 97 | post url, JSON.dump(invalid_params) 98 | expect(last_response.body).to include("errors") 99 | end 100 | end 101 | end 102 | end -------------------------------------------------------------------------------- /spec/snowly/integration/validator_tracker_spec.rb: -------------------------------------------------------------------------------- 1 | require 'spec_helper' 2 | describe 'Tracker Validation' do 3 | attr_reader :emitter 4 | 5 | before { emitter.reset_responses! } 6 | 7 | let(:snowplow_subject) do 8 | @snow_subject ||= begin 9 | SnowplowTracker::Subject.new.tap do |snow_subject| 10 | snow_subject.set_platform 'mob' 11 | snow_subject.set_user_id '1' 12 | snow_subject.set_useragent 'user agent' 13 | end 14 | end 15 | end 16 | 17 | let(:tracker) do 18 | SnowplowTracker::Tracker.new(emitter.emitter, snowplow_subject, nil, 'tracker-1') 19 | end 20 | 21 | with_event_definitions do 22 | context 'with two events' do 23 | before(:all) do 24 | @emitter ||= Snowly::Emitter.new(emitter_type: 'closure') 25 | end 26 | context 'when both are valid' do 27 | let(:se1) { Snowly::Transformer.transform(valid_se) } 28 | let(:se2) { Snowly::Transformer.transform(invalid_se) } 29 | it 'returns 200' do 30 | tracker.track_struct_event(*valid_se.values) 31 | tracker.track_struct_event(*valid_se.values) 32 | tracker.flush 33 | expect(response.code).to eq '200' 34 | end 35 | it 'returns empty errors' do 36 | tracker.track_struct_event(*valid_se.values) 37 | tracker.track_struct_event(*valid_se.values) 38 | tracker.flush 39 | expect(JSON.load(response.body).first['errors']).to be_empty 40 | expect(JSON.load(response.body).last['errors']).to be_empty 41 | end 42 | end 43 | context 'when one is invalid' do 44 | it 'returns 422' do 45 | tracker.track_struct_event(*valid_se.values) 46 | tracker.track_struct_event(*invalid_se.values) 47 | tracker.flush 48 | expect(response.code).to eq '422' 49 | end 50 | it 'returns errors' do 51 | tracker.track_struct_event(*valid_se.values) 52 | tracker.track_struct_event(*invalid_se.values) 53 | tracker.flush 54 | expect(JSON.load(response.body).first['errors']).to be_empty 55 | expect(JSON.load(response.body).last['errors'].count).to eq 2 56 | end 57 | end 58 | end 59 | 60 | %w(cloudfront closure).each do |emitter_type| 61 | let(:response) { emitter.responses.first } 62 | let(:body) { JSON.load(response.body).first['content'] } 63 | let(:errors) { JSON.load(response.body).first['errors'] } 64 | 65 | context "with emitter: #{emitter_type}" do 66 | before(:all) do 67 | @emitter ||= Snowly::Emitter.new(emitter_type: emitter_type) 68 | end 69 | context 'with valid structured event request' do 70 | let(:translated_se) { Snowly::Transformer.transform(valid_se) } 71 | 72 | it 'returns 200 for valid structured event request' do 73 | tracker.track_struct_event(*valid_se.values) 74 | tracker.flush 75 | expect(response.code).to eq '200' 76 | end 77 | it 'returns valid content' do 78 | tracker.track_struct_event(*valid_se.values) 79 | tracker.flush 80 | expect(body).to include translated_se 81 | end 82 | end 83 | 84 | context 'with invalid structured event' do 85 | let(:translated_se) { Snowly::Transformer.transform(invalid_se) } 86 | it 'returns 422 for invalid structured event request' do 87 | tracker.track_struct_event(*invalid_se.values) 88 | tracker.flush 89 | expect(response.code).to eq '422' 90 | end 91 | it 'returns errors' do 92 | tracker.track_struct_event(*invalid_se.values) 93 | tracker.flush 94 | expect(errors.count).to eq 2 95 | end 96 | end 97 | context 'with context' do 98 | context 'and context is valid' do 99 | before { stub_request(:get, context_url).to_return(body: context_content, status: 200) } 100 | let(:valid_se_co) { valid_se.merge(co: [valid_co_object]) } 101 | it 'returns 200' do 102 | tracker.track_struct_event(*valid_se_co.values) 103 | tracker.flush 104 | expect(response.code).to eq '200' 105 | end 106 | end 107 | context 'and context is invalid' do 108 | let(:invalid_se_co) { valid_se.merge(co: [invalid_co_object]) } 109 | it 'returns errors' do 110 | tracker.track_struct_event(*invalid_se_co.values) 111 | tracker.flush 112 | expect(errors.count).to eq 2 113 | end 114 | end 115 | end 116 | context 'with unstructured event' do 117 | before { stub_request(:get, context_url).to_return(body: context_content, status: 200) } 118 | context 'when valid' do 119 | it 'returns 200' do 120 | tracker.track_unstruct_event(valid_ue_object) 121 | tracker.flush 122 | expect(response.code).to eq '200' 123 | end 124 | end 125 | context 'when invalid' do 126 | it 'returns errors' do 127 | tracker.track_unstruct_event(invalid_ue_object) 128 | tracker.flush 129 | expect(errors.count).to eq 1 130 | end 131 | end 132 | end 133 | end 134 | end 135 | end 136 | 137 | end -------------------------------------------------------------------------------- /spec/snowly/unit/each_validator_spec.rb: -------------------------------------------------------------------------------- 1 | require 'yaml' 2 | require 'rack/utils' 3 | require 'spec_helper' 4 | 5 | describe Snowly::EachValidator do 6 | def to_query(hash) 7 | Rack::Utils.build_nested_query(hash) 8 | end 9 | before { Snowly.development_iglu_resolver_path = File.expand_path("../../../fixtures", __FILE__)+"/" } 10 | before { Snowly::SchemaCache.instance.reset_cache } 11 | 12 | let(:validator) { Snowly::EachValidator.new to_query(hash) } 13 | 14 | with_event_definitions do 15 | context 'with mininum required attributes' do 16 | let(:hash) { valid_root.merge(e: 'ad') } 17 | it 'returns true' do 18 | expect(validator.validate).to be true 19 | end 20 | it 'does not set errors' do 21 | validator.validate 22 | expect(validator.errors).to eq [] 23 | end 24 | context 'and an alternative more restrictive protocol schema' do 25 | let(:custom_schema) { Snowly::ProtocolSchemaFinder.new(alternative_protocol_schema).schema } 26 | before do 27 | allow_any_instance_of(Snowly::EachValidator) 28 | .to receive(:protocol_schema) 29 | .and_return(custom_schema) 30 | end 31 | it 'set errors' do 32 | validator.validate 33 | puts validator.errors 34 | expect(validator.errors.count).to eq 2 35 | end 36 | end 37 | end 38 | context 'with wrong type for attribute' do 39 | let(:hash) { valid_root.merge(e: 'ad', tid: 'none') } 40 | it 'sets error' do 41 | validator.validate 42 | puts validator.errors 43 | expect(validator.errors.count).to eq 1 44 | end 45 | end 46 | describe 'CustomDependency' do 47 | context 'with missing custom dependency' do 48 | let(:hash) { valid_root.merge(invalid_se) } 49 | it 'sets error' do 50 | validator.validate 51 | puts validator.errors 52 | expect(validator.errors.count).to eq 2 53 | end 54 | end 55 | end 56 | context 'with missing dependency' do 57 | let(:hash) { valid_root.merge(valid_se).tap{|n| n.delete(:p)} } 58 | it 'sets error' do 59 | validator.validate 60 | puts validator.errors 61 | expect(validator.errors.count).to eq 1 62 | end 63 | end 64 | context 'with context' do 65 | before { stub_request(:get, context_url).to_return(body: context_content, status: 200) } 66 | context 'and context is base64 and valid' do 67 | let(:hash) { valid_root.merge(e: 'ad', cx: valid_base64_co) } 68 | it 'returns true' do 69 | expect(validator.validate).to be true 70 | end 71 | end 72 | context 'and context is urlsafe_base64 and valid' do 73 | let(:hash) { valid_root.merge(e: 'ad', cx: valid_urlsafe_base64_co) } 74 | it 'returns true' do 75 | expect(validator.validate).to be true 76 | end 77 | end 78 | context 'and context is not an array' do 79 | let(:hash) { valid_root.merge(e: 'ad', co: not_array_co) } 80 | it 'returns false' do 81 | expect(validator.validate).to be false 82 | end 83 | end 84 | context 'and context is valid' do 85 | let(:hash) { valid_root.merge(e: 'ad', co: valid_co) } 86 | it 'returns true' do 87 | expect(validator.validate).to be true 88 | end 89 | it 'does not set errors' do 90 | validator.validate 91 | expect(validator.errors).to eq [] 92 | end 93 | end 94 | context 'and context is invalid' do 95 | let(:hash) { valid_root.merge(e: 'ad', co: invalid_co) } 96 | it 'sets error' do 97 | validator.validate 98 | puts validator.errors 99 | expect(validator.errors.count).to eq 1 100 | end 101 | end 102 | end 103 | context 'with multiple contexts' do 104 | before { stub_request(:get, context_url).to_return(body: context_content, status: 200) } 105 | context "and they're both valid" do 106 | let(:hash) { valid_root.merge(e: 'ad', co: valid_co_multiple) } 107 | it 'returns true' do 108 | expect(validator.validate).to be true 109 | end 110 | end 111 | context 'and one is invalid' do 112 | let(:hash) { valid_root.merge(e: 'ad', co: invalid_co_multiple) } 113 | it 'returns true' do 114 | validator.validate 115 | puts validator.errors 116 | expect(validator.errors.count).to eq 1 117 | end 118 | end 119 | end 120 | context 'with unstruct event' do 121 | before { stub_request(:get, ue_url).to_return(body: ue_content, status: 200) } 122 | context 'and event is valid' do 123 | let(:hash) { valid_root.merge(e: 'ue', ue_pr: valid_ue) } 124 | it 'returns true' do 125 | expect(validator.validate).to be true 126 | end 127 | end 128 | context 'and event is base64 and valid' do 129 | let(:hash) { valid_root.merge(e: 'ue', ue_px: valid_base64_ue) } 130 | it 'returns true' do 131 | expect(validator.validate).to be true 132 | end 133 | end 134 | context 'when event is invalid' do 135 | let(:hash) { valid_root.merge(e: 'ue', ue_pr: invalid_ue) } 136 | it 'returns true' do 137 | validator.validate 138 | puts validator.errors 139 | expect(validator.errors.count).to eq 1 140 | end 141 | end 142 | end 143 | context 'with unstruct event and context' do 144 | before { stub_request(:get, context_url).to_return(body: context_content, status: 200) } 145 | before { stub_request(:get, ue_url).to_return(body: ue_content, status: 200) } 146 | context 'and event is valid' do 147 | let(:hash) { valid_root.merge(e: 'ue', ue_pr: valid_ue, co: valid_co) } 148 | it 'returns true' do 149 | expect(validator.validate).to be true 150 | end 151 | end 152 | context 'and context is invalid' do 153 | let(:hash) { valid_root.merge(e: 'ue', ue_pr: valid_ue, co: invalid_co) } 154 | it 'sets error' do 155 | validator.validate 156 | puts validator.errors 157 | expect(validator.errors.count).to eq 1 158 | end 159 | end 160 | context 'and event is invalid' do 161 | let(:hash) { valid_root.merge(e: 'ue', ue_pr: invalid_ue, co: valid_co) } 162 | it 'returns true' do 163 | validator.validate 164 | puts validator.errors 165 | expect(validator.errors.count).to eq 1 166 | end 167 | end 168 | context 'and event and context are invalid' do 169 | let(:hash) { valid_root.merge(e: 'ue', ue_pr: invalid_ue, co: invalid_co) } 170 | it 'returns true' do 171 | validator.validate 172 | puts validator.errors 173 | expect(validator.errors.count).to eq 2 174 | end 175 | end 176 | end 177 | end 178 | end -------------------------------------------------------------------------------- /spec/snowly/unit/protocol_schema_finder_spec.rb: -------------------------------------------------------------------------------- 1 | require 'spec_helper' 2 | 3 | describe Snowly::ProtocolSchemaFinder do 4 | let(:default_description) { 'Representation of Snowplow Protocol in JSON Schema format for validation' } 5 | 6 | context 'with default protocol schema' do 7 | let(:schema_description) { Snowly::ProtocolSchemaFinder.new.schema['description'] } 8 | it 'returns the correct schema' do 9 | expect(schema_description).to eq default_description 10 | end 11 | end 12 | 13 | context 'with custom protocol schema' do 14 | let(:alternative_description) { 'Alternative Snowplow Protocol' } 15 | let(:alternative_schema_path) { File.expand_path('../../../protocol_resolver/snowplow_protocol.json', __FILE__) } 16 | 17 | context 'when schema is given' do 18 | let(:schema_description) { Snowly::ProtocolSchemaFinder.new(alternative_schema_path).schema['description'] } 19 | it 'returns the correct schema' do 20 | expect(schema_description).to eq alternative_description 21 | end 22 | end 23 | 24 | context 'when schema is in resolver path' do 25 | before do 26 | allow(Snowly).to receive(:development_iglu_resolver_path).and_return(File.expand_path("../../../protocol_resolver", __FILE__)+"/") 27 | end 28 | let(:schema_description) { Snowly::ProtocolSchemaFinder.new.schema['description'] } 29 | it 'returns the correct schema' do 30 | expect(schema_description).to eq alternative_description 31 | end 32 | end 33 | end 34 | end -------------------------------------------------------------------------------- /spec/snowly/unit/request_spec.rb: -------------------------------------------------------------------------------- 1 | require 'spec_helper' 2 | 3 | describe Snowly::Request do 4 | let(:query) { "e=se&aid=app&tna=1.0"} 5 | let(:parsed_hash) { {"e" => 'se', 'aid' => 'app', 'tna' => '1.0'} } 6 | let(:hash) { {"event" => "se", "app_id" => "app", "name_tracker" => "1.0" } } 7 | let(:json) { hash.to_json } 8 | let(:get_request) { Snowly::Request.new query } 9 | let(:post_request) { Snowly::Request.new(parsed_hash) } 10 | 11 | describe '#as_json' do 12 | context 'with querystring' do 13 | it 'returns transformed JSON' do 14 | expect(get_request.as_json).to eq json 15 | end 16 | end 17 | context 'with post payload' do 18 | it 'returns transformed JSON', :focus do 19 | expect(post_request.as_json).to eq json 20 | end 21 | end 22 | end 23 | 24 | describe '#as_hash' do 25 | context 'with querystring' do 26 | it 'returns transformed hash' do 27 | expect(get_request.as_hash).to eq hash 28 | end 29 | end 30 | context 'with post payload' do 31 | it 'returns transformed hash' do 32 | expect(post_request.as_hash).to eq hash 33 | end 34 | end 35 | end 36 | 37 | describe '#parse_query' do 38 | it 'returns hash from query string' do 39 | parsed_payload = get_request.parse_query(query) 40 | expect(parsed_payload).to eq parsed_hash 41 | end 42 | end 43 | end -------------------------------------------------------------------------------- /spec/snowly/unit/schema_cache_spec.rb: -------------------------------------------------------------------------------- 1 | require 'spec_helper' 2 | 3 | describe Snowly::SchemaCache do 4 | it 'is singleton' do 5 | expect(Snowly::SchemaCache.instance).to eq Snowly::SchemaCache.instance 6 | end 7 | 8 | let(:url) { "http://iglucentral.com/schemas/com.snowplowanalytics.snowplow/contexts/jsonschema/1-0-0" } 9 | let(:location) { 'iglu:com.snowplowanalytics.snowplow/contexts/jsonschema/1-0-0'} 10 | let(:file_content) { File.read(File.expand_path('../../../fixtures/snowly/context_test_0/jsonschema/1-0-0', __FILE__)) } 11 | 12 | context 'without development resolver' do 13 | before { Snowly.development_iglu_resolver_path = nil } 14 | before { Snowly::SchemaCache.instance.reset_cache } 15 | context 'when already loaded' do 16 | before do 17 | stub_request(:get, url).to_return(body: file_content, status: 200) 18 | Snowly::SchemaCache.instance[location] 19 | end 20 | it 'does not save on cache' do 21 | expect(Snowly::SchemaCache.instance).not_to receive(:save_in_cache).with(location) 22 | Snowly::SchemaCache.instance[location] 23 | end 24 | it 'loads schema from cache' do 25 | expect(Snowly::SchemaCache.instance[location]).to eq file_content 26 | end 27 | end 28 | context 'when not yet loaded' do 29 | it 'saves on cache' do 30 | stub_request(:get, url).to_return(body: file_content, status: 200) 31 | expect(Snowly::SchemaCache.instance).to receive(:save_in_cache).with(location) 32 | Snowly::SchemaCache.instance[location] 33 | end 34 | it 'loads and returns schema' do 35 | stub_request(:get, url).to_return(body: file_content, status: 200) 36 | expect(Snowly::SchemaCache.instance[location]).to eq file_content 37 | end 38 | end 39 | end 40 | 41 | context 'with development resolver' do 42 | before { Snowly.development_iglu_resolver_path = File.expand_path("../../../fixtures", __FILE__)+"/" } 43 | before { Snowly::SchemaCache.instance.reset_cache } 44 | let(:location) { 'iglu:snowly/context_test_0/jsonschema/1-0-0'} 45 | context 'when schema is found' do 46 | context 'when already loaded' do 47 | before { Snowly::SchemaCache.instance[location] } 48 | it 'does not save on cache' do 49 | expect(Snowly::SchemaCache.instance).not_to receive(:save_in_cache).with(location) 50 | Snowly::SchemaCache.instance[location] 51 | end 52 | it 'loads schema from cache' do 53 | expect(Snowly::SchemaCache.instance[location]).to eq file_content 54 | end 55 | end 56 | context 'when not yet loaded' do 57 | it 'saves on cache' do 58 | expect(Snowly::SchemaCache.instance).to receive(:save_in_cache).with(location) 59 | Snowly::SchemaCache.instance[location] 60 | end 61 | it 'loads and returns schema' do 62 | expect(Snowly::SchemaCache.instance[location]).to eq file_content 63 | end 64 | end 65 | context 'when using external resolver' do 66 | before { Snowly.development_iglu_resolver_path = "http://snowly"} 67 | let(:url) { "http://snowly/context_test_0/jsonschema/1-0-0" } 68 | it 'saves on cache' do 69 | stub_request(:get, url).to_return(body: file_content, status: 200) 70 | expect(Snowly::SchemaCache.instance).to receive(:save_in_cache).with(location) 71 | Snowly::SchemaCache.instance[location] 72 | end 73 | end 74 | end 75 | context 'when schema is not in development resolver' do 76 | let(:location) { 'iglu:snowly/context_test_0/jsonschema/2-0-0'} 77 | it 'warns about not finding schema in local resolver', :focus do 78 | stub_request(:get, url).to_return(body: file_content, status: 200) 79 | expect(Snowly.logger).to receive(:warn) 80 | Snowly::SchemaCache.instance[location] 81 | end 82 | it 'loads schema from iglucentral' do 83 | expect(Snowly::SchemaCache.instance).to receive(:save_in_cache).with(location) 84 | Snowly::SchemaCache.instance[location] 85 | end 86 | end 87 | context 'when schema is not in any resolver' do 88 | let(:location) { 'iglu:snowly/context_test_0/jsonschema/2-0-0'} 89 | it 'logs an error', :focus do 90 | stub_request(:get, url).to_return(body: 'not json', status: 200) 91 | expect(Snowly.logger).to receive(:error) 92 | Snowly::SchemaCache.instance[location] 93 | end 94 | it 'sets schema cache to nil' do 95 | expect(Snowly::SchemaCache.instance[location]).to be_nil 96 | end 97 | end 98 | end 99 | end 100 | 101 | -------------------------------------------------------------------------------- /spec/snowly/unit/transformer_spec.rb: -------------------------------------------------------------------------------- 1 | require 'spec_helper' 2 | describe Snowly::Transformer do 3 | let(:source) do 4 | { 5 | "e" => "e", 6 | "ip" => "ip", 7 | "aid" => "aid", 8 | "p" => "p", 9 | "tid" => "tid", 10 | "uid" => "uid", 11 | "duid" => "duid", 12 | "nuid" => "nuid", 13 | "ua" => "ua", 14 | "fp" => "fp", 15 | "vid" => "vid", 16 | "sid" => "sid", 17 | "dtm" => "dtm", 18 | "ttm" => "ttm", 19 | "stm" => "stm", 20 | "tna" => "tna", 21 | "tv" => "tv", 22 | "cv" => "cv", 23 | "lang" => "lang", 24 | "f_pdf" => "f_pdf", 25 | "f_fla" => "f_fla", 26 | "f_java" => "f_java", 27 | "f_dir" => "f_dir", 28 | "f_qt" => "f_qt", 29 | "f_realp" => "f_realp", 30 | "f_wma" => "f_wma", 31 | "f_gears" => "f_gears", 32 | "f_ag" => "f_ag", 33 | "cookie" => "cookie", 34 | "res" => "res", 35 | "cd" => "cd", 36 | "tz" => "tz", 37 | "refr" => "refr", 38 | "url" => "url", 39 | "page" => "page", 40 | "cs" => "cs", 41 | "ds" => "ds", 42 | "vp" => "vp", 43 | "eid" => "eid", 44 | "ev_ca" => "ev_ca", 45 | "ev_ac" => "ev_ac", 46 | "ev_la" => "ev_la", 47 | "ev_pr" => "ev_pr", 48 | "ev_va" => "ev_va", 49 | "se_ca" => "se_ca", 50 | "se_ac" => "se_ac", 51 | "se_la" => "se_la", 52 | "se_pr" => "se_pr", 53 | "se_va" => "se_va", 54 | "tr_id" => "tr_id", 55 | "tr_af" => "tr_af", 56 | "tr_tt" => "tr_tt", 57 | "tr_tx" => "tr_tx", 58 | "tr_sh" => "tr_sh", 59 | "tr_ci" => "tr_ci", 60 | "tr_st" => "tr_st", 61 | "tr_co" => "tr_co", 62 | "ti_id" => "ti_id", 63 | "ti_sk" => "ti_sk", 64 | "ti_na" => "ti_na", 65 | "ti_nm" => "ti_nm", 66 | "ti_ca" => "ti_ca", 67 | "ti_pr" => "ti_pr", 68 | "ti_qu" => "ti_qu", 69 | "pp_mix" => "pp_mix", 70 | "pp_max" => "pp_max", 71 | "pp_miy" => "pp_miy", 72 | "pp_may" => "pp_may", 73 | "tr_cu" => "tr_cu", 74 | "ti_cu" => "ti_cu" 75 | } 76 | end 77 | let(:unencoded) do 78 | source.merge('co' => { name:'co' }.to_json, "ue_pr" => { name:'ue_pr' }.to_json) 79 | end 80 | let(:encoded) do 81 | source.merge('cx' => Base64.encode64({ name:'cx' }.to_json), "ue_px" => Base64.encode64({ name:'ue_px' }.to_json)) 82 | end 83 | let(:translated) do 84 | { 85 | "event" => "e", 86 | "user_ipaddress" => "ip", 87 | "app_id" => "aid", 88 | "platform" => "p", 89 | "txn_id" => "tid", 90 | "user_id" => "uid", 91 | "domain_userid" => "duid", 92 | "network_userid" => "nuid", 93 | "useragent" => "ua", 94 | "user_fingerprint" => "fp", 95 | "domain_sessionidx" => "vid", 96 | "domain_sessionid" => "sid", 97 | "dvce_created_tstamp" => "dtm", 98 | "true_tstamp" => "ttm", 99 | "dvce_sent_tstamp" => "stm", 100 | "name_tracker" => "tna", 101 | "v_tracker" => "tv", 102 | "v_collector" => "cv", 103 | "br_lang" => "lang", 104 | "br_features_pdf" => "f_pdf", 105 | "br_features_flash" => "f_fla", 106 | "br_features_java" => "f_java", 107 | "br_features_director" => "f_dir", 108 | "br_features_quicktime" => "f_qt", 109 | "br_features_realplayer" => "f_realp", 110 | "br_features_windowsmedia" => "f_wma", 111 | "br_features_gears" => "f_gears", 112 | "br_features_silverlight" => "f_ag", 113 | "br_cookies" => "cookie", 114 | "screen_res_width_x_height" => "res", 115 | "br_colordepth" => "cd", 116 | "os_timezone" => "tz", 117 | "page_referrer" => "refr", 118 | "page_url" => "url", 119 | "page_title" => "page", 120 | "doc_charset" => "cs", 121 | "doc_width_x_height" => "ds", 122 | "browser_viewport_width_x_height" => "vp", 123 | "event_id" => "eid", 124 | "se_category" => "ev_ca", 125 | "se_action" => "ev_ac", 126 | "se_label" => "ev_la", 127 | "se_property" => "ev_pr", 128 | "se_value" => "ev_va", 129 | "se_category" => "se_ca", 130 | "se_action" => "se_ac", 131 | "se_label" => "se_la", 132 | "se_property" => "se_pr", 133 | "se_value" => "se_va", 134 | "tr_orderid" => "tr_id", 135 | "tr_affiliation" => "tr_af", 136 | "tr_total" => "tr_tt", 137 | "tr_tax" => "tr_tx", 138 | "tr_shipping" => "tr_sh", 139 | "tr_city" => "tr_ci", 140 | "tr_state" => "tr_st", 141 | "tr_country" => "tr_co", 142 | "ti_orderid" => "ti_id", 143 | "ti_sku" => "ti_sk", 144 | "ti_name" => "ti_na", 145 | "ti_name" => "ti_nm", 146 | "ti_category" => "ti_ca", 147 | "ti_price" => "ti_pr", 148 | "ti_quantity" => "ti_qu", 149 | "pp_xoffset_min" => "pp_mix", 150 | "pp_xoffset_max" => "pp_max", 151 | "pp_yoffset_min" => "pp_miy", 152 | "pp_yoffset_max" => "pp_may", 153 | "tr_currency" => "tr_cu", 154 | "ti_currency" => "ti_cu" 155 | } 156 | end 157 | let(:tunencoded) do 158 | translated.merge("contexts" => {'name' => 'co'}, "unstruct_event" => {'name' => 'ue_pr'}) 159 | end 160 | let(:tencoded) do 161 | translated.merge("contexts" => {'name' => 'cx'}, "unstruct_event" => {'name' => 'ue_px'}) 162 | end 163 | 164 | describe '.translate' do 165 | it 'maps query string keys to snowplow fields' do 166 | expect(subject.transform(source)).to eq translated 167 | end 168 | context 'with unencoded context and unstructured event' do 169 | it 'maps query string keys to unencoded contexts and ue' do 170 | expect(subject.transform(unencoded)).to eq tunencoded 171 | end 172 | end 173 | context 'with encoded context and unstructured event' do 174 | it 'maps query string keys to encoded contexts and ue' do 175 | expect(subject.transform(encoded)).to eq tencoded 176 | end 177 | end 178 | 179 | context 'with unknown keys' do 180 | let(:source_with_unknown) { source.merge("unknown" => "unknown")} 181 | it 'ignores unknown keys' do 182 | expect(subject.transform(source_with_unknown)).to eq translated 183 | end 184 | end 185 | end 186 | describe '.convert' do 187 | context 'with string type' do 188 | let(:result) { subject.convert('valid', 'string') } 189 | it 'returns string' do 190 | expect(result).to be_a String 191 | end 192 | end 193 | context 'with integer type' do 194 | context 'and value is integer' do 195 | let(:result) { subject.convert('1', 'integer') } 196 | it 'returns integer' do 197 | expect(result).to eq 1 198 | end 199 | end 200 | context 'and value is not integer' do 201 | let(:result) { subject.convert('abc', 'integer') } 202 | it 'returns string' do 203 | expect(result).to eq 'abc' 204 | end 205 | end 206 | end 207 | context 'with float type' do 208 | let(:result) { subject.convert('1.99', 'number') } 209 | context 'and value is float' do 210 | it 'returns float' do 211 | expect(result).to eq 1.99 212 | end 213 | end 214 | context 'and value is not float' do 215 | let(:result) { subject.convert('abc', 'number') } 216 | it 'returns string' do 217 | expect(result).to eq 'abc' 218 | end 219 | end 220 | end 221 | end 222 | end 223 | -------------------------------------------------------------------------------- /spec/snowly/unit/validator_spec.rb: -------------------------------------------------------------------------------- 1 | require 'spec_helper' 2 | 3 | describe Snowly::Validator do 4 | let(:invalid_request) { { "e"=>"pv", "page"=>"Root README", "url"=>"http://github.com/snowplow/snowplow", "aid"=>"snowplow", "p"=>"web", "tv"=>"no-js-0.1.0" } } 5 | let(:valid_request) { { "e"=>"pv", "page"=>"Root README", "url"=>"http://github.com/snowplow/snowplow", "aid"=>"snowplow", "p"=>"web", "tv"=>"no-js-0.1.0", "ua"=>"firefox", "eid"=>"u2i3" } } 6 | let(:valid_params) { {"data" => [valid_request, valid_request] } } 7 | let(:invalid_params) { {"data" => [valid_request, invalid_request] } } 8 | describe '#validate' do 9 | it 'calls validate for each request' do 10 | batch = Snowly::Validator.new(valid_params, batch: true) 11 | batch.validators.each do |validator| 12 | expect(validator).to receive(:validate) 13 | end 14 | batch.validate 15 | end 16 | it 'returns whether all requests are valid' do 17 | expect(Snowly::Validator.new(valid_params, batch: true).validate).to be true 18 | end 19 | end 20 | describe 'valid?' do 21 | context 'when all requests are valid' do 22 | it 'returns true' do 23 | batch = Snowly::Validator.new(valid_params, batch: true) 24 | batch.validate 25 | expect(batch).to be_valid 26 | end 27 | end 28 | context 'when one of the requests is invalid' do 29 | it 'returns false' do 30 | batch = Snowly::Validator.new(invalid_params, batch: true) 31 | batch.validate 32 | expect(batch).not_to be_valid 33 | end 34 | end 35 | end 36 | end 37 | -------------------------------------------------------------------------------- /spec/snowly_spec.rb: -------------------------------------------------------------------------------- 1 | require 'spec_helper' 2 | 3 | describe Snowly do 4 | it 'has a version number' do 5 | expect(Snowly::VERSION).not_to be nil 6 | end 7 | end 8 | -------------------------------------------------------------------------------- /spec/spec_helper.rb: -------------------------------------------------------------------------------- 1 | # Snowplow uses questionable regex to validate the iglu location as it doesn't scape - 2 | # Setting verbose to nil supresses a lot of warnings emmited by the Regexp class. 3 | $VERBOSE = nil 4 | 5 | $LOAD_PATH.unshift File.expand_path('../../lib', __FILE__) 6 | require 'snowly' 7 | require 'snowly/app/collector' 8 | require 'snowplow-tracker' 9 | require 'pry' 10 | require 'webmock/rspec' 11 | require 'rack/test' 12 | 13 | Dir[File.join(Dir.pwd,"spec/support/**/*.rb")].each {|f| require f} 14 | WebMock.allow_net_connect! 15 | TestServer.new.start_sinatra_server 16 | 17 | module RSpecMixin 18 | include Rack::Test::Methods 19 | def app() Snowly::App::Collector end 20 | end 21 | 22 | RSpec.configure do |config| 23 | config.include RSpecMixin 24 | config.extend EventAssignments 25 | end -------------------------------------------------------------------------------- /spec/support/emitter.rb: -------------------------------------------------------------------------------- 1 | module Snowly 2 | class EmitterError < StandardError 3 | attr_reader :failed_events 4 | 5 | def initialize(failed_events) 6 | @failed_events = failed_events 7 | end 8 | end 9 | class Emitter 10 | delegate :responses, :reset_responses!, to: :emitter 11 | # include Singleton 12 | COLLECTOR_URL = 'localhost:4567' 13 | 14 | attr_reader :emitter 15 | 16 | def initialize(emitter_type: 'cloudfront', buffer_size: 2, thread_count: 1) 17 | @emitter_type = emitter_type 18 | @buffer_size = buffer_size 19 | @thread_count = thread_count 20 | @emitter = emitter_klass.new(COLLECTOR_URL, emitter_options) 21 | end 22 | 23 | def emitter_options 24 | { 25 | protocol: 'http', 26 | method: http_method, 27 | buffer_size: buffer_size, 28 | thread_count: thread_count, 29 | on_success: lambda { |success_count| 30 | handle_on_success(success_count) 31 | }, 32 | on_failure: lambda { |success_count, failed_events| 33 | handle_on_failure(success_count, failed_events) 34 | } 35 | } 36 | end 37 | 38 | def self.should_handle_failure=(value) 39 | @@should_handle_failure = value 40 | end 41 | 42 | def emitter_klass 43 | cloudfront? ? SnowplowTracker::Emitter : SnowplowTracker::AsyncEmitter 44 | end 45 | 46 | def cloudfront? 47 | @emitter_type == 'cloudfront' 48 | end 49 | 50 | def buffer_size 51 | cloudfront? ? 0 : @buffer_size 52 | end 53 | 54 | def thread_count 55 | cloudfront? ? 1 : @thread_count 56 | end 57 | 58 | def http_method 59 | cloudfront? ? 'get' : 'post' 60 | end 61 | 62 | private 63 | 64 | def handle_on_failure(success_count, failed_events) 65 | puts 'failed' 66 | end 67 | 68 | def handle_on_success(success_count) 69 | puts 'success' 70 | end 71 | end 72 | end 73 | 74 | SnowplowTracker::LOGGER.level = Logger::DEBUG if Snowly.debug_mode -------------------------------------------------------------------------------- /spec/support/event_assignments.rb: -------------------------------------------------------------------------------- 1 | module EventAssignments 2 | def with_event_definitions(&block) 3 | context "importing event definitions" do 4 | 5 | let(:context_url) { "http://iglucentral.com/schemas/com.snowplowanalytics.snowplow/contexts/jsonschema/1-0-0" } 6 | let(:ue_url) { "http://iglucentral.com/schemas/com.snowplowanalytics.snowplow/unstruct_event/jsonschema/1-0-0" } 7 | let(:context_location) { 'iglu:com.snowplowanalytics.snowplow/contexts/jsonschema/1-0-0'} 8 | let(:ue_location) { 'iglu:com.snowplowanalytics.snowplow/unstruct_event/jsonschema/1-0-0'} 9 | let(:context_content) { File.read(File.expand_path('../../fixtures/snowplow_context.json', __FILE__)) } 10 | let(:ue_content) { File.read(File.expand_path('../../fixtures/snowplow_ue.json', __FILE__)) } 11 | let(:alternative_protocol_schema) { File.expand_path('../../protocol_resolver/snowplow_protocol.json', __FILE__) } 12 | 13 | let(:valid_root) do 14 | { 15 | uid: 1, 16 | aid: 'app', 17 | tna: '1.0', 18 | dtm: Time.now.to_i, 19 | e: 'se', 20 | ua: 'user agent', 21 | p: 'mob', 22 | eid: 'eventid', 23 | tv: 'tracker-1' 24 | } 25 | end 26 | 27 | let(:valid_se) do 28 | { 29 | 'se_ca' => 'web', 30 | 'se_ac' => 'click', 31 | 'se_la' => 'label', 32 | 'se_pr' => 'property', 33 | 'se_va' => 1 34 | } 35 | end 36 | 37 | let(:invalid_se) do 38 | valid_se.merge('se_ca' => '', 'se_ac' => '') 39 | end 40 | 41 | let(:valid_co) do 42 | { 43 | schema: 'iglu:com.snowplowanalytics.snowplow/contexts/jsonschema/1-0-0', 44 | data: [ valid_co_data ] 45 | }.to_json 46 | end 47 | 48 | 49 | 50 | let(:invalid_co) do 51 | { 52 | schema: 'iglu:com.snowplowanalytics.snowplow/contexts/jsonschema/1-0-0', 53 | data: [ valid_co_data.deep_merge(data: {age: 1000}) ] 54 | }.to_json 55 | end 56 | 57 | let(:not_array_co) do 58 | { 59 | schema: 'iglu:com.snowplowanalytics.snowplow/contexts/jsonschema/1-0-0', 60 | data: valid_co_data 61 | }.to_json 62 | end 63 | 64 | let(:valid_co_data) do 65 | { 66 | schema: 'iglu:snowly/context_test_0/jsonschema/1-0-0', 67 | data: { 68 | name: 'name', 69 | age: 10 70 | } 71 | } 72 | end 73 | 74 | let(:valid_co_object) do 75 | SnowplowTracker::SelfDescribingJson.new(valid_co_data[:schema],valid_co_data[:data]) 76 | end 77 | 78 | let(:invalid_co_object) do 79 | SnowplowTracker::SelfDescribingJson.new(valid_co_data[:schema], {age: 1000}) 80 | end 81 | 82 | let(:valid_co_data_1) do 83 | { 84 | schema: 'iglu:snowly/context_test_1/jsonschema/1-0-0', 85 | data: { 86 | street: 'street' 87 | } 88 | } 89 | end 90 | 91 | let(:valid_co_multiple) do 92 | { 93 | schema: 'iglu:com.snowplowanalytics.snowplow/contexts/jsonschema/1-0-0', 94 | data: [ valid_co_data, valid_co_data_1] 95 | }.to_json 96 | end 97 | 98 | let(:invalid_co_multiple) do 99 | { 100 | schema: 'iglu:com.snowplowanalytics.snowplow/contexts/jsonschema/1-0-0', 101 | data: [ valid_co_data, valid_co_data_1.deep_merge(data: {street: 'home'}) ] 102 | }.to_json 103 | end 104 | 105 | let(:non_array_cx) { Base64.strict_encode64(not_array_co) } 106 | let(:valid_base64_co) { Base64.strict_encode64(valid_co) } 107 | let(:valid_urlsafe_base64_co) { Base64.urlsafe_encode64(valid_co) } 108 | let(:valid_ue_data) do 109 | { 110 | schema: "iglu:snowly/event_test/jsonschema/1-0-0", 111 | data: { 112 | category: 'reading', 113 | name: 'view', 114 | elapsed_time: 10, 115 | object_id: "oid", 116 | number_property_name: 'nprop', 117 | number_property_value: 1, 118 | string_property_name: 'sprop', 119 | string_property_value: 'sval' 120 | } 121 | } 122 | end 123 | 124 | let(:valid_ue_object) do 125 | SnowplowTracker::SelfDescribingJson.new(valid_ue_data[:schema], valid_ue_data[:data]) 126 | end 127 | let(:invalid_ue_object) do 128 | SnowplowTracker::SelfDescribingJson.new(valid_ue_data[:schema], valid_ue_data[:data].merge(elapsed_time: 'none')) 129 | end 130 | 131 | let(:valid_ue) do 132 | { 133 | schema: 'iglu:com.snowplowanalytics.snowplow/unstruct_event/jsonschema/1-0-0', 134 | data: valid_ue_data 135 | }.to_json 136 | end 137 | 138 | let(:invalid_ue) do 139 | { 140 | schema: 'iglu:com.snowplowanalytics.snowplow/unstruct_event/jsonschema/1-0-0', 141 | data: valid_ue_data.deep_merge(data: { elapsed_time: 'none'} ) 142 | }.to_json 143 | end 144 | 145 | let(:valid_base64_ue) { Base64.strict_encode64(valid_ue) } 146 | 147 | instance_eval &block 148 | end 149 | 150 | end 151 | 152 | end -------------------------------------------------------------------------------- /spec/support/server_starter.rb: -------------------------------------------------------------------------------- 1 | class TestServer 2 | 3 | def initialize(collector_url = nil) 4 | @collector_url = collector_url || "http://localhost:4567" 5 | end 6 | 7 | def sinatra_startup_timeout 8 | @sinatra_startup_timeout || 15 9 | end 10 | 11 | def wait_until_sinatra_starts 12 | (sinatra_startup_timeout * 10).times do 13 | break if sinatra_running? 14 | sleep(0.1) 15 | end 16 | raise Timeout::Error, "Sinatra failed to start after #{sinatra_startup_timeout} seconds" unless sinatra_running? 17 | end 18 | 19 | def sinatra_running? 20 | begin 21 | ping_uri = URI.parse(@collector_url) 22 | Net::HTTP.get(ping_uri) 23 | true 24 | rescue 25 | false 26 | end 27 | end 28 | 29 | def start_sinatra_server 30 | Snowly.development_iglu_resolver_path = File.expand_path("../../fixtures", __FILE__)+"/" 31 | WebMock.allow_net_connect! 32 | unless sinatra_running? 33 | pid = fork do 34 | Snowly.debug_mode = true 35 | Snowly::App::Collector.run! 36 | end 37 | 38 | at_exit do 39 | WebMock.disable_net_connect! 40 | Process.kill("TERM", pid) 41 | end 42 | 43 | wait_until_sinatra_starts 44 | end 45 | end 46 | end -------------------------------------------------------------------------------- /spec/support/tracker_patch.rb: -------------------------------------------------------------------------------- 1 | SnowplowTracker::Emitter.class_eval do 2 | attr_reader :responses 3 | alias_method :original_http_get, :http_get 4 | alias_method :original_http_post, :http_post 5 | 6 | def reset_responses! 7 | @responses = [] 8 | end 9 | 10 | def http_get(*args) 11 | original_http_get(*args).tap do |response| 12 | @responses ||= [] 13 | @responses << response 14 | end 15 | end 16 | def http_post(*args) 17 | response = original_http_post(*args).tap do |response| 18 | @responses ||= [] 19 | @responses << response 20 | end 21 | end 22 | end --------------------------------------------------------------------------------