├── .gitignore ├── .rspec ├── .travis.yml ├── CHANGELOG.md ├── CODE_OF_CONDUCT.md ├── Gemfile ├── LICENSE.txt ├── README.md ├── Rakefile ├── bin ├── console └── setup ├── lib ├── ruby_px.rb └── ruby_px │ ├── dataset.rb │ └── dataset │ └── data.rb ├── ruby_px.gemspec └── spec ├── fixtures ├── ine-fenomenos-demograficos-2014-alava-23001.px └── ine-padron-2014.px ├── ruby_px ├── dataset │ └── data_spec.rb ├── dataset_fenomenos_demograficos_spec.rb ├── dataset_padron_spec.rb └── remote_dataset_spec.rb └── spec_helper.rb /.gitignore: -------------------------------------------------------------------------------- 1 | /.bundle/ 2 | /.yardoc 3 | /Gemfile.lock 4 | /_yardoc/ 5 | /coverage/ 6 | /doc/ 7 | /pkg/ 8 | /spec/reports/ 9 | /tmp/ 10 | .byebug_history 11 | .ruby-version 12 | -------------------------------------------------------------------------------- /.rspec: -------------------------------------------------------------------------------- 1 | --format documentation 2 | --color 3 | -------------------------------------------------------------------------------- /.travis.yml: -------------------------------------------------------------------------------- 1 | language: ruby 2 | rvm: 3 | - 2.7.1 4 | - 2.6.3 5 | before_install: gem install bundler 6 | -------------------------------------------------------------------------------- /CHANGELOG.md: -------------------------------------------------------------------------------- 1 | # CHANGELOG 2 | 3 | ## 0.8.0 (2021-03-04) 4 | 5 | - Fix wrong DATA key error [#10](https://github.com/PopulateTools/ruby_px/pull/10) 6 | 7 | ## 0.7.0 (2021-03-04) 8 | 9 | - Performance in large datasets [#9](https://github.com/PopulateTools/ruby_px/pull/9) 10 | 11 | ## 0.6.0 (2020-04-28) 12 | 13 | - Nice badge with the current version 14 | - Added Changelog! 15 | - Updated dependencies 16 | - Fix multilingual VALUES and equals signs in values [#7](https://github.com/PopulateTools/ruby_px/pull/7) 17 | - Use Rubocop to check syntax 18 | 19 | -------------------------------------------------------------------------------- /CODE_OF_CONDUCT.md: -------------------------------------------------------------------------------- 1 | # Contributor Code of Conduct 2 | 3 | As contributors and maintainers of this project, we pledge to respect all people who contribute through reporting issues, posting feature requests, updating documentation, submitting pull requests or patches, and other activities. 4 | 5 | We are committed to making participation in this project a harassment-free experience for everyone, regardless of level of experience, gender, gender identity and expression, sexual orientation, disability, personal appearance, body size, race, ethnicity, age, or religion. 6 | 7 | Examples of unacceptable behavior by participants include the use of sexual language or imagery, derogatory comments or personal attacks, trolling, public or private harassment, insults, or other unprofessional conduct. 8 | 9 | Project maintainers have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct. Project maintainers who do not follow the Code of Conduct may be removed from the project team. 10 | 11 | Instances of abusive, harassing, or otherwise unacceptable behavior may be reported by opening an issue or contacting one or more of the project maintainers. 12 | 13 | This Code of Conduct is adapted from the [Contributor Covenant](http://contributor-covenant.org), version 1.0.0, available at [http://contributor-covenant.org/version/1/0/0/](http://contributor-covenant.org/version/1/0/0/) 14 | -------------------------------------------------------------------------------- /Gemfile: -------------------------------------------------------------------------------- 1 | # frozen_string_literal: true 2 | 3 | source 'https://rubygems.org' 4 | 5 | # Specify your gem's dependencies in ruby_px.gemspec 6 | gemspec 7 | -------------------------------------------------------------------------------- /LICENSE.txt: -------------------------------------------------------------------------------- 1 | The MIT License (MIT) 2 | 3 | Copyright (c) 2015 Fernando Blat 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in 13 | all copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN 21 | THE SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # RubyPx 2 | 3 | [![Gem Version](https://badge.fury.io/rb/ruby_px.svg)](https://badge.fury.io/rb/ruby_px) 4 | [![Build Status](https://travis-ci.org/PopulateTools/ruby_px.svg?branch=master)](https://travis-ci.org/PopulateTools/ruby_px) 5 | 6 | Work with PC-Axis files using Ruby. 7 | 8 | ## Motivation 9 | 10 | The Spanish Statistics Institute ([INE](http://www.ine.es/welcome.shtml)) favourite format to publish 11 | the data is PC-Axis, a semi-plain-text format which is kind of difficult to work with unless you 12 | have a PC with Windows installed in. 13 | 14 | There is a library in R called [pxR](https://github.com/cran/pxR) from [@gilbellosta](https://twitter.com/gilbellosta), but I don't know more alternatives in other programming languages. 15 | 16 | ## Installation 17 | 18 | Add this line to your application's Gemfile: 19 | 20 | ```ruby 21 | gem 'ruby_px' 22 | ``` 23 | 24 | And then execute: 25 | 26 | $ bundle 27 | 28 | Or install it yourself as: 29 | 30 | $ gem install ruby_px 31 | 32 | ## Usage 33 | 34 | ```ruby 35 | # Load a dataset 36 | dataset = RubyPx::Dataset.new 'spec/fixtures/ine-padron-2014.px' 37 | 38 | # Query some metadata 39 | dataset.title 40 | => "Población por sexo, municipios y edad (año a año)." 41 | 42 | dataset.units 43 | => "personas" 44 | 45 | dataset.source 46 | => "Instituto Nacional de Estadística" 47 | 48 | dataset.contact 49 | => "INE E-mail:www.ine.es/infoine. Internet: www.ine.es. Tel: +34 91 583 91 00 Fax: +34 91 583 91 58" 50 | 51 | dataset.last_updated 52 | => "05/10/99" # Really, INE? 53 | 54 | dataset.creation_date 55 | => "20141201" 56 | 57 | # Obtain the headings and the stubs 58 | dataset.headings 59 | => ["edad (año a año)"] 60 | 61 | dataset.stubs 62 | => ["sexo", "municipios"] 63 | 64 | # Obtain the dimensions of the dataset 65 | dataset.dimensions 66 | => ["sexo", "edad (año a año)", "municipios"] 67 | 68 | # Get the list of values of a dimension 69 | dataset.dimension('sexo') 70 | => ["Ambos sexos", "Hombres", "Mujeres"] 71 | 72 | # Query the data using the method #data 73 | # You can query the data in two ways: 74 | # 1 - providing all the dimensions, so you'll obtain a single value 75 | dataset.data('edad (año a año)' => 'Total', 'sexo' => 'Ambos sexos', 'municipios' => '28079-Madrid') 76 | => "3165235" 77 | 78 | # 2 - providing all the dimensions except one, so you'll obtain an array 79 | dataset.data('edad (año a año)' => 'Total', 'sexo' => 'Ambos sexos') 80 | => [....] # an array with the population of all places for all ages and both sexs 81 | 82 | ``` 83 | 84 | ## TODO 85 | 86 | - Refactor 87 | - Test the gem with more files 88 | - Speed-up the parsing time 89 | 90 | 91 | ## Development 92 | 93 | After checking out the repository, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment. 94 | 95 | To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org). 96 | 97 | ## Contributing 98 | 99 | Bug reports and pull requests are welcome on GitHub at https://github.com/PopulateTools/ruby_px. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [Contributor Covenant](contributor-covenant.org) code of conduct. 100 | 101 | ## Thanks 102 | 103 | Thank you [Xavier Badosa](https://twitter.com/badosa) for inspiring me with [json-stat.org](http://json-stat.org/) and the API of 104 | the Javascript library [json-stat.com](http://json-stat.com/). 105 | 106 | ## License 107 | 108 | The gem is available as open source under the terms of the [MIT License](http://opensource.org/licenses/MIT). 109 | 110 | -------------------------------------------------------------------------------- /Rakefile: -------------------------------------------------------------------------------- 1 | # frozen_string_literal: true 2 | 3 | require 'bundler/gem_tasks' 4 | require 'rspec/core/rake_task' 5 | 6 | RSpec::Core::RakeTask.new(:spec) 7 | 8 | task default: :spec 9 | -------------------------------------------------------------------------------- /bin/console: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env ruby 2 | # frozen_string_literal: true 3 | 4 | require 'bundler/setup' 5 | require_relative '../lib/ruby_px' 6 | 7 | # You can add fixtures and/or initialization code here to make experimenting 8 | # with your gem easier. You can also use a different console, if you like. 9 | 10 | # (If you use this, don't forget to add pry to your Gemfile!) 11 | # require "pry" 12 | # Pry.start 13 | 14 | require 'irb' 15 | IRB.start 16 | -------------------------------------------------------------------------------- /bin/setup: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | set -euo pipefail 3 | IFS=$'\n\t' 4 | 5 | bundle install 6 | 7 | # Do any other automated setup that you need to do here 8 | -------------------------------------------------------------------------------- /lib/ruby_px.rb: -------------------------------------------------------------------------------- 1 | # frozen_string_literal: true 2 | 3 | require 'active_support/all' 4 | 5 | module RubyPx 6 | require 'ruby_px/dataset' 7 | end 8 | -------------------------------------------------------------------------------- /lib/ruby_px/dataset.rb: -------------------------------------------------------------------------------- 1 | # frozen_string_literal: true 2 | 3 | require 'open-uri' 4 | 5 | module RubyPx 6 | class Dataset 7 | require 'ruby_px/dataset/data' 8 | 9 | attr_reader :headings, :stubs 10 | 11 | METADATA_RECORDS = %w[TITLE UNITS SOURCE CONTACT LAST-UPDATED CREATION-DATE].freeze 12 | HEADING_RECORD = 'HEADING' 13 | STUB_RECORD = 'STUB' 14 | 15 | def initialize(resource_uri) 16 | @metadata = {} 17 | @headings = [] 18 | @stubs = [] 19 | @values = {} 20 | @data = Data.new 21 | 22 | parse_resource(resource_uri) 23 | end 24 | 25 | def title 26 | @metadata['TITLE'] 27 | end 28 | 29 | def units 30 | @metadata['UNITS'] 31 | end 32 | 33 | def source 34 | @metadata['SOURCE'] 35 | end 36 | 37 | def contact 38 | @metadata['CONTACT'] 39 | end 40 | 41 | def last_updated 42 | @metadata['LAST-UPDATED'] 43 | end 44 | 45 | def creation_date 46 | @metadata['CREATION-DATE'] 47 | end 48 | 49 | def dimension(name) 50 | @values[name] || raise("Missing dimension #{name}") 51 | end 52 | 53 | def dimensions 54 | @values.keys 55 | end 56 | 57 | def data(options) 58 | # Validate parameters 59 | options.each do |k, v| 60 | unless dimension(k).include?(v) 61 | raise "Invalid value #{v} for dimension #{k}" 62 | end 63 | end 64 | 65 | # Return a single value 66 | # 2D array[ i*M + j] 67 | # 3D array[ i*(N*M) + j*M + k ] 68 | # 4D array[ i*(N*M*R) + j*M*R + k*R + l] 69 | if options.length == dimensions.length 70 | offset = 0 71 | 72 | # positions are i, j, k 73 | positions = (stubs + headings).map do |dimension_name| 74 | dimension(dimension_name).index(options[dimension_name]) 75 | end 76 | 77 | # dimension_sizes are from all dimensions except the first one 78 | dimension_sizes = (stubs + headings)[1..-1].map do |dimension_name| 79 | dimension(dimension_name).length 80 | end 81 | 82 | positions.each_with_index do |p, i| 83 | d = dimension_sizes[i..-1].reduce(&:*) 84 | offset += (d ? p * d : p) 85 | end 86 | 87 | @data.at(offset) 88 | 89 | # Return an array of options 90 | elsif options.length == dimensions.length - 1 91 | result = [] 92 | 93 | missing_dimension = (dimensions - options.keys).first 94 | dimension(missing_dimension).each do |dimension_value| 95 | result << data(options.merge(missing_dimension => dimension_value)) 96 | end 97 | 98 | result 99 | else 100 | raise 'Not implented yet, sorry' 101 | end 102 | end 103 | 104 | def inspect 105 | "#<#{self.class.name}:#{object_id}>" 106 | end 107 | 108 | private 109 | 110 | def parse_resource(resource_uri) 111 | open(resource_uri).each_line do |line| 112 | parse_line(line.chomp) 113 | end 114 | 115 | true 116 | end 117 | 118 | def parse_line(line) 119 | @line = line.force_encoding('utf-8').encode('utf-8') 120 | 121 | if @current_record.nil? 122 | key, value = line.split('=', 2) 123 | set_current_record(key) 124 | else 125 | value = line 126 | end 127 | 128 | return if @current_record.nil? || value.nil? 129 | 130 | if @type == :data 131 | value = value.split(/[\ ;,\t]/).delete_if(&:blank?).each(&:strip) 132 | 133 | add_value_to_bucket(bucket, value) unless value == [';'] 134 | else 135 | # First format: "\"20141201\";" 136 | if value =~ /\A\"([^"]+)\";\z/ 137 | value = value.match(/\A\"([^"]+)\";\z/)[1] 138 | add_value_to_bucket(bucket, value.strip) 139 | 140 | # Second format: "Ambos sexos","Hombres","Mujeres"; 141 | elsif value =~ /\"([^"]+)\",?/ 142 | value = value.split(/\"([^"]+)\",?;?/).delete_if(&:blank?).each(&:strip) 143 | add_value_to_bucket(bucket, value) 144 | end 145 | end 146 | 147 | # If we see a ; at the end of the line, close out the record so we 148 | # expect a new record. 149 | @current_record = nil if line[-1..-1] == ';' 150 | end 151 | 152 | def set_current_record(key) 153 | @current_record = if METADATA_RECORDS.include?(key) 154 | @type = :metadata 155 | key 156 | elsif key == HEADING_RECORD 157 | @type = :headings 158 | key 159 | elsif key == STUB_RECORD 160 | @type = :stubs 161 | key 162 | elsif key =~ /\AVALUES/ && key !~ /\[\w\w\]/ 163 | @type = :values 164 | key.match(/\"([^"]+)\"/)[1] 165 | elsif key =~ /\ADATA\z/ 166 | @type = :data 167 | key 168 | end 169 | end 170 | 171 | def bucket 172 | instance_variable_get("@#{@type}") 173 | end 174 | 175 | def add_value_to_bucket(bucket, value) 176 | if @type == :data 177 | @data.concat(value) 178 | elsif @type == :headings || @type == :stubs 179 | bucket << value 180 | bucket.flatten! 181 | elsif bucket.is_a?(Hash) 182 | if value.is_a?(Array) 183 | value = value.map(&:strip) 184 | elsif value.is_a?(String) 185 | value.strip! 186 | end 187 | if bucket[@current_record].nil? 188 | value = Array.wrap(value) if @type == :values 189 | bucket[@current_record] = value 190 | else 191 | bucket[@current_record].concat([value]) 192 | bucket[@current_record].flatten! 193 | end 194 | end 195 | end 196 | end 197 | end 198 | -------------------------------------------------------------------------------- /lib/ruby_px/dataset/data.rb: -------------------------------------------------------------------------------- 1 | module RubyPx 2 | class Dataset 3 | class Data 4 | 5 | CHUNK_SIZE = 5_000 6 | attr_accessor :current_chunk_index 7 | 8 | def initialize 9 | @current_chunk_index = 0 10 | end 11 | 12 | def at index 13 | chunk_index = index/CHUNK_SIZE 14 | index_inside_chunk = index%CHUNK_SIZE 15 | 16 | get_chunk(chunk_index)[index_inside_chunk] 17 | end 18 | 19 | def concat array 20 | current_chunk.concat(array) 21 | if current_chunk.size > CHUNK_SIZE 22 | excess = current_chunk.pop(current_chunk.size-CHUNK_SIZE) 23 | self.current_chunk_index += 1 24 | concat(excess) 25 | end 26 | end 27 | 28 | def indexes_count 29 | self.current_chunk_index+1 30 | end 31 | 32 | private 33 | 34 | 35 | def current_chunk 36 | current = instance_variable_get("@chunk_#{self.current_chunk_index}") 37 | return current if current 38 | 39 | instance_variable_set("@chunk_#{self.current_chunk_index}", []) 40 | end 41 | 42 | def get_chunk chunk_index 43 | instance_variable_get("@chunk_#{chunk_index}") 44 | end 45 | 46 | end 47 | end 48 | end 49 | -------------------------------------------------------------------------------- /ruby_px.gemspec: -------------------------------------------------------------------------------- 1 | # frozen_string_literal: true 2 | 3 | lib = File.expand_path('lib', __dir__) 4 | $LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib) 5 | 6 | Gem::Specification.new do |spec| 7 | spec.name = 'ruby_px' 8 | spec.version = '0.8.0' 9 | spec.authors = ['Fernando Blat'] 10 | spec.email = ['fernando@blat.es'] 11 | 12 | spec.summary = 'Read PC-Axis files using Ruby' 13 | spec.description = 'Read PC-Axis files using Ruby' 14 | spec.homepage = 'https://github.com/PopulateTools/ruby_px' 15 | spec.license = 'MIT' 16 | 17 | # Prevent pushing this gem to RubyGems.org by setting 'allowed_push_host', or 18 | # delete this section to allow pushing this gem to any host. 19 | if spec.respond_to?(:metadata) 20 | spec.metadata['allowed_push_host'] = 'https://rubygems.org' 21 | else 22 | raise 'RubyGems 2.0 or newer is required to protect against public gem pushes.' 23 | end 24 | 25 | spec.files = `git ls-files -z`.split("\x0").reject { |f| f.match(%r{^(test|spec|features)/}) } 26 | spec.bindir = 'exe' 27 | spec.executables = spec.files.grep(%r{^exe/}) { |f| File.basename(f) } 28 | spec.require_paths = ['lib'] 29 | 30 | spec.add_runtime_dependency 'activesupport', '>= 6.0' 31 | 32 | spec.add_development_dependency 'bundler' 33 | spec.add_development_dependency 'rake', '~> 13.0' 34 | spec.add_development_dependency 'rspec', '~> 3.9' 35 | end 36 | -------------------------------------------------------------------------------- /spec/fixtures/ine-fenomenos-demograficos-2014-alava-23001.px: -------------------------------------------------------------------------------- 1 | AXIS-VERSION="2006"; 2 | CREATION-DATE="20161124"; 3 | CHARSET="ANSI"; 4 | SUBJECT-AREA="Fenómenos demográficos. Año 2014"; 5 | SUBJECT-CODE="null"; 6 | MATRIX="23001"; 7 | TITLE="Araba/Álava por municipios y fenómeno demográfico ."; 8 | CONTENTS="Araba/Álava"; 9 | CODEPAGE="iso-8859-15"; 10 | DESCRIPTION=""; 11 | COPYRIGHT=YES; 12 | DECIMALS=0; 13 | SHOWDECIMALS=0; 14 | STUB="Municipios"; 15 | HEADING="Fenómeno demográfico","Periodo"; 16 | VALUES("Municipios")="01001 Alegría-Dulantzi","01002 Amurrio","01049 Añana","01003 Aramaio", 17 | "01006 Armiñón","01037 Arraia-Maeztu","01008 Arrazua-Ubarrundia","01004 Artziniega", 18 | "01009 Asparrena","01010 Ayala/Aiara","01011 Baños de Ebro/Mañueta","01013 Barrundia", 19 | "01014 Berantevilla","01016 Bernedo","01017 Campezo/Kanpezu","01021 Elburgo/Burgelu", 20 | "01022 Elciego","01023 Elvillar/Bilar","01046 Erriberagoitia/Ribera Alta", 21 | "01056 Harana/Valle de Arana","01901 Iruña Oka/Iruña de Oca","01027 Iruraiz-Gauna", 22 | "01019 Kripan","01020 Kuartango","01028 Labastida/Bastida","01030 Lagrán", 23 | "01031 Laguardia","01032 Lanciego/Lantziego","01902 Lantarón", 24 | "01033 Lapuebla de Labarca","01036 Laudio/Llodio","01058 Legutio","01034 Leza", 25 | "01039 Moreda de Álava/Moreda Araba","01041 Navaridas","01042 Okondo", 26 | "01043 Oyón-Oion","01044 Peñacerrada-Urizaharra","01047 Ribera Baja/Erribera Beitia", 27 | "01051 Salvatierra/Agurain","01052 Samaniego","01053 San Millán/Donemiliaga", 28 | "01054 Urkabustaiz","01055 Valdegovía/Gaubea","01057 Villabuena de Álava/Eskuernaga", 29 | "01059 Vitoria-Gasteiz","01060 Yécora/Iekora","01061 Zalduondo","01062 Zambrana", 30 | "01018 Zigoitia","01063 Zuia"; 31 | VALUES("Fenómeno demográfico")="nacidos vivos por residencia materna", 32 | "muertes fetales tardías por residencia materna", 33 | "matrimonios por el lugar en que han fijado residencia", 34 | "fallecidos por el lugar de residencia","crecimiento vegetativo"; 35 | VALUES("Periodo")="periodopx"; 36 | CODES("Municipios")="01001","01002","01049","01003","01006","01037","01008","01004","01009","01010","01011", 37 | "01013","01014","01016","01017","01021","01022","01023","01046","01056","01901","01027", 38 | "01019","01020","01028","01030","01031","01032","01902","01033","01036","01058","01034", 39 | "01039","01041","01042","01043","01044","01047","01051","01052","01053","01054","01055", 40 | "01057","01059","01060","01061","01062","01018","01063"; 41 | MAP("Municipios")="muni0109_esp"; 42 | UNITS="fenómenos demográficos"; 43 | SOURCE="Instituto Nacional de Estadística"; 44 | DATASYMBOL1=":"; 45 | DATA= 46 | 42.0 0.0 10.0 23.0 19.0 99.0 0.0 48.0 75.0 24.0 0.0 0.0 0.0 3.0 -3.0 20.0 0.0 6.0 9.0 11.0 2.0 0.0 0.0 5.0 -3.0 9.0 0.0 2.0 5.0 4.0 7.0 0.0 5.0 7.0 0.0 16.0 0.0 7.0 13.0 3.0 12.0 0.0 4.0 9.0 3.0 31.0 0.0 13.0 22.0 9.0 47 | 2.0 0.0 0.0 4.0 -2.0 2.0 0.0 2.0 5.0 -3.0 4.0 0.0 1.0 7.0 -3.0 4.0 0.0 2.0 8.0 -4.0 8.0 0.0 1.0 13.0 -5.0 10.0 0.0 1.0 3.0 7.0 7.0 0.0 5.0 10.0 -3.0 2.0 0.0 0.0 4.0 -2.0 5.0 0.0 1.0 4.0 1.0 0.0 0.0 2.0 6.0 -6.0 48 | 28.0 0.0 11.0 10.0 18.0 1.0 0.0 3.0 5.0 -4.0 0.0 0.0 0.0 2.0 -2.0 1.0 0.0 4.0 4.0 -3.0 11.0 0.0 2.0 7.0 4.0 1.0 0.0 2.0 4.0 -3.0 18.0 0.0 3.0 21.0 -3.0 5.0 0.0 3.0 7.0 -2.0 5.0 0.0 1.0 9.0 -4.0 14.0 0.0 4.0 3.0 11.0 49 | 140.0 2.0 61.0 179.0 -39.0 21.0 0.0 6.0 11.0 10.0 2.0 0.0 1.0 2.0 0.0 1.0 0.0 0.0 7.0 -6.0 1.0 0.0 0.0 3.0 -2.0 9.0 1.0 6.0 7.0 2.0 40.0 0.0 9.0 22.0 18.0 3.0 0.0 0.0 5.0 -2.0 8.0 0.0 4.0 8.0 0.0 68.0 1.0 32.0 30.0 38.0 50 | 0.0 0.0 0.0 5.0 -5.0 6.0 0.0 0.0 7.0 -1.0 17.0 0.0 4.0 8.0 9.0 5.0 0.0 0.0 14.0 -9.0 2.0 0.0 0.0 7.0 -5.0 2554.0 9.0 938.0 1818.0 736.0 2.0 0.0 0.0 3.0 -1.0 2.0 0.0 0.0 2.0 0.0 2.0 0.0 0.0 6.0 -4.0 17.0 0.0 6.0 7.0 10.0 51 | 11.0 0.0 6.0 18.0 -7.0; 52 | -------------------------------------------------------------------------------- /spec/ruby_px/dataset/data_spec.rb: -------------------------------------------------------------------------------- 1 | # frozen_string_literal: true 2 | 3 | require 'spec_helper' 4 | 5 | describe RubyPx::Dataset::Data do 6 | 7 | let(:subject) { described_class.new } 8 | 9 | describe '#concat and #at' do 10 | it 'should store the values and return them ' do 11 | subject.concat (0..12000).to_a 12 | subject.concat (12001..32000).to_a 13 | 14 | expect(subject.at(12)).to eq 12 15 | expect(subject.at(3475)).to eq 3475 16 | expect(subject.at(14223)).to eq 14223 17 | expect(subject.at(23987)).to eq 23987 18 | end 19 | end 20 | end 21 | -------------------------------------------------------------------------------- /spec/ruby_px/dataset_fenomenos_demograficos_spec.rb: -------------------------------------------------------------------------------- 1 | # frozen_string_literal: true 2 | 3 | require 'spec_helper' 4 | 5 | describe RubyPx::Dataset do 6 | before(:all) do 7 | @subject = described_class.new 'spec/fixtures/ine-fenomenos-demograficos-2014-alava-23001.px' 8 | end 9 | 10 | let(:subject) { @subject } 11 | 12 | context "ine-fenomenos-demograficos-2014-alava-23001.px" do 13 | describe '#headings' do 14 | it 'should return the list of headings described in the file' do 15 | expect(subject.headings).to eq(['Fenómeno demográfico', 'Periodo']) 16 | end 17 | end 18 | 19 | describe '#stubs' do 20 | it 'should return the list of stubs described in the file' do 21 | expect(subject.stubs).to eq(['Municipios']) 22 | end 23 | end 24 | 25 | describe 'metadata methods' do 26 | it 'should return the title' do 27 | expect(subject.title).to eq('Araba/Álava por municipios y fenómeno demográfico .') 28 | end 29 | 30 | it 'should return the units' do 31 | expect(subject.units).to eq('fenómenos demográficos') 32 | end 33 | 34 | it 'should return the source' do 35 | expect(subject.source).to eq('Instituto Nacional de Estadística') 36 | end 37 | 38 | it 'should return the creation_date' do 39 | expect(subject.creation_date).to eq('20161124') 40 | end 41 | end 42 | 43 | describe '#dimension' do 44 | it 'should return all the values of a dimension' do 45 | expect(subject.dimension('Fenómeno demográfico')).to include('nacidos vivos por residencia materna') 46 | expect(subject.dimension('Fenómeno demográfico')).to include('fallecidos por el lugar de residencia') 47 | expect(subject.dimension('Fenómeno demográfico')).to include('crecimiento vegetativo') 48 | 49 | expect(subject.dimension('Periodo')).to eq ['periodopx'] 50 | end 51 | 52 | it 'should return the number of values' do 53 | expect(subject.dimension('Fenómeno demográfico').length).to eq(5) 54 | expect(subject.dimension('Periodo').length).to eq(1) 55 | expect(subject.dimension('Municipios').length).to eq(51) 56 | end 57 | 58 | it 'should return an error if the dimension does not exist' do 59 | expect do 60 | subject.dimension('foo').values 61 | end.to raise_error('Missing dimension foo') 62 | end 63 | end 64 | 65 | describe '#dimensions' do 66 | it 'should return an array with all the dimensions of the dataset' do 67 | expect(subject.dimensions).to eq(['Municipios', 'Fenómeno demográfico', 'Periodo']) 68 | end 69 | end 70 | 71 | describe '#data' do 72 | it 'should raise an error if a dimension value does not exist' do 73 | expect do 74 | subject.data('Fenómeno demográfico' => 'nacidos', 'Periodo' => 'periodopx', 'Municipios' => '01030 Lagrán') 75 | end.to raise_error('Invalid value nacidos for dimension Fenómeno demográfico') 76 | end 77 | 78 | it 'should raise an error if a dimension does not exist' do 79 | expect do 80 | subject.data('foo' => 'Total', 'Periodo' => 'periodopx', 'Municipios' => '01030 Lagrán') 81 | end.to raise_error('Missing dimension foo') 82 | end 83 | 84 | it 'should return data when all the dimensions are provided' do 85 | expect(subject.data('Fenómeno demográfico' => 'nacidos vivos por residencia materna', 'Periodo' => 'periodopx', 'Municipios' => '01030 Lagrán')).to eq('1.0') 86 | expect(subject.data('Fenómeno demográfico' => 'muertes fetales tardías por residencia materna', 'Periodo' => 'periodopx', 'Municipios' => '01030 Lagrán')).to eq('0.0') 87 | expect(subject.data('Fenómeno demográfico' => 'matrimonios por el lugar en que han fijado residencia', 'Periodo' => 'periodopx', 'Municipios' => '01030 Lagrán')).to eq('2.0') 88 | expect(subject.data('Fenómeno demográfico' => 'fallecidos por el lugar de residencia', 'Periodo' => 'periodopx', 'Municipios' => '01030 Lagrán')).to eq('4.0') 89 | expect(subject.data('Fenómeno demográfico' => 'crecimiento vegetativo', 'Periodo' => 'periodopx', 'Municipios' => '01030 Lagrán')).to eq('-3.0') 90 | 91 | expect(subject.data('Fenómeno demográfico' => 'nacidos vivos por residencia materna', 'Periodo' => 'periodopx', 'Municipios' => '01063 Zuia')).to eq('11.0') 92 | expect(subject.data('Fenómeno demográfico' => 'muertes fetales tardías por residencia materna', 'Periodo' => 'periodopx', 'Municipios' => '01063 Zuia')).to eq('0.0') 93 | expect(subject.data('Fenómeno demográfico' => 'matrimonios por el lugar en que han fijado residencia', 'Periodo' => 'periodopx', 'Municipios' => '01063 Zuia')).to eq('6.0') 94 | expect(subject.data('Fenómeno demográfico' => 'fallecidos por el lugar de residencia', 'Periodo' => 'periodopx', 'Municipios' => '01063 Zuia')).to eq('18.0') 95 | expect(subject.data('Fenómeno demográfico' => 'crecimiento vegetativo', 'Periodo' => 'periodopx', 'Municipios' => '01063 Zuia')).to eq('-7.0') 96 | end 97 | 98 | it 'should return an array of data when all the dimensions are provided except 1' do 99 | result = subject.data('Periodo' => 'periodopx', 'Fenómeno demográfico' => 'nacidos vivos por residencia materna') 100 | expect(result.first).to eq('42.0') 101 | expect(result.last).to eq('11.0') 102 | expect(result.length).to eq(51) 103 | end 104 | 105 | it 'should return an error if more than one dimension is expected in the result' do 106 | expect do 107 | subject.data('Fenómeno demográfico' => 'crecimiento vegetativo') 108 | end.to raise_error('Not implented yet, sorry') 109 | end 110 | end 111 | end 112 | end 113 | -------------------------------------------------------------------------------- /spec/ruby_px/dataset_padron_spec.rb: -------------------------------------------------------------------------------- 1 | # frozen_string_literal: true 2 | 3 | require 'spec_helper' 4 | 5 | describe RubyPx::Dataset do 6 | before(:all) do 7 | @subject = described_class.new 'spec/fixtures/ine-padron-2014.px' 8 | end 9 | 10 | let(:subject) { @subject } 11 | 12 | 13 | context "ine-padron-2014.px" do 14 | describe '#headings' do 15 | it 'should return the list of headings described in the file' do 16 | expect(subject.headings).to eq(['edad (año a año)']) 17 | end 18 | end 19 | 20 | describe '#stubs' do 21 | it 'should return the list of stubs described in the file' do 22 | expect(subject.stubs).to eq(%w[sexo municipios]) 23 | end 24 | end 25 | 26 | describe 'metadata methods' do 27 | it 'should return the title' do 28 | expect(subject.title).to eq('Población por sexo, municipios y edad (año a año).') 29 | end 30 | 31 | it 'should return the units' do 32 | expect(subject.units).to eq('personas') 33 | end 34 | 35 | it 'should return the source' do 36 | expect(subject.source).to eq('Instituto Nacional de Estadística') 37 | end 38 | 39 | it 'should return the contact' do 40 | expect(subject.contact).to eq('INE E-mail:www.ine.es/infoine. Internet: www.ine.es. Tel: +34 91 583 91 00 Fax: +34 91 583 91 58') 41 | end 42 | 43 | it 'should return the last_updated' do 44 | expect(subject.last_updated).to eq('05/10/99') 45 | end 46 | 47 | it 'should return the creation_date' do 48 | expect(subject.creation_date).to eq('20141201') 49 | end 50 | end 51 | 52 | describe '#dimension' do 53 | it 'should return all the values of a dimension' do 54 | expect(subject.dimension('edad (año a año)')).to include('Total') 55 | expect(subject.dimension('edad (año a año)')).to include('100 y más') 56 | end 57 | 58 | it 'should return the number of values' do 59 | expect(subject.dimension('edad (año a año)').length).to eq(102) 60 | end 61 | 62 | it 'should return an error if the dimension does not exist' do 63 | expect do 64 | subject.dimension('foo').values 65 | end.to raise_error('Missing dimension foo') 66 | end 67 | end 68 | 69 | describe '#dimensions' do 70 | it 'should return an array with all the dimensions of the dataset' do 71 | expect(subject.dimensions).to eq(['sexo', 'edad (año a año)', 'municipios']) 72 | end 73 | end 74 | 75 | describe '#data' do 76 | it 'should raise an error if a dimension value does not exist' do 77 | expect do 78 | subject.data('edad (año a año)' => 'Totalxxx', 'sexo' => 'Ambos sexos', 'municipios' => '28079-Madrid') 79 | end.to raise_error('Invalid value Totalxxx for dimension edad (año a año)') 80 | end 81 | 82 | it 'should raise an error if a dimension does not exist' do 83 | expect do 84 | subject.data('foo' => 'Total', 'sexo' => 'Ambos sexos', 'municipios' => '28079-Madrid') 85 | end.to raise_error('Missing dimension foo') 86 | end 87 | 88 | it 'should return data when all the dimensions are provided' do 89 | expect(subject.data('edad (año a año)' => 'Total', 'sexo' => 'Ambos sexos', 'municipios' => '28079-Madrid')).to eq('3165235') 90 | expect(subject.data('edad (año a año)' => 'Total', 'sexo' => 'Hombres', 'municipios' => '28079-Madrid')).to eq('1472990') 91 | expect(subject.data('edad (año a año)' => 'Total', 'sexo' => 'Mujeres', 'municipios' => '28079-Madrid')).to eq('1692245') 92 | end 93 | 94 | it 'should return an array of data when all the dimensions are provided except 1' do 95 | result = subject.data('edad (año a año)' => 'Total', 'sexo' => 'Ambos sexos') 96 | expect(result.first).to eq('46771341') 97 | expect(result[3899]).to eq('3165235') 98 | expect(result.length).to eq(8118) 99 | end 100 | 101 | it 'should return an error if more than one dimension is expected in the result' do 102 | expect do 103 | subject.data('edad (año a año)' => 'Total') 104 | end.to raise_error('Not implented yet, sorry') 105 | end 106 | end 107 | end 108 | end 109 | -------------------------------------------------------------------------------- /spec/ruby_px/remote_dataset_spec.rb: -------------------------------------------------------------------------------- 1 | # frozen_string_literal: true 2 | 3 | require 'spec_helper' 4 | 5 | describe RubyPx::Dataset do 6 | let(:subject) { described_class.new 'http://populate-data.s3.amazonaws.com/ruby_px/f1.px' } 7 | 8 | describe '#headings' do 9 | it 'should return the list of headings described in the file' do 10 | expect(subject.headings).to eq(['Fenómeno demográfico']) 11 | end 12 | end 13 | end 14 | -------------------------------------------------------------------------------- /spec/spec_helper.rb: -------------------------------------------------------------------------------- 1 | # frozen_string_literal: true 2 | 3 | $LOAD_PATH.unshift File.expand_path('../lib', __dir__) 4 | require 'ruby_px' 5 | --------------------------------------------------------------------------------