├── .rspec ├── Gemfile ├── Gemfile.lock ├── LICENSE ├── README.md ├── dl_experiment.gemspec ├── lib ├── dl_experiment.rb └── experiment.rb └── spec ├── experiment_spec.rb └── spec_helper.rb /.rspec: -------------------------------------------------------------------------------- 1 | --require spec_helper 2 | -------------------------------------------------------------------------------- /Gemfile: -------------------------------------------------------------------------------- 1 | # frozen_string_literal: true 2 | 3 | source "https://rubygems.org" 4 | 5 | 6 | gem "rspec", "~> 3.9" 7 | -------------------------------------------------------------------------------- /Gemfile.lock: -------------------------------------------------------------------------------- 1 | GEM 2 | remote: https://rubygems.org/ 3 | specs: 4 | diff-lcs (1.3) 5 | rspec (3.9.0) 6 | rspec-core (~> 3.9.0) 7 | rspec-expectations (~> 3.9.0) 8 | rspec-mocks (~> 3.9.0) 9 | rspec-core (3.9.0) 10 | rspec-support (~> 3.9.0) 11 | rspec-expectations (3.9.0) 12 | diff-lcs (>= 1.2.0, < 2.0) 13 | rspec-support (~> 3.9.0) 14 | rspec-mocks (3.9.0) 15 | diff-lcs (>= 1.2.0, < 2.0) 16 | rspec-support (~> 3.9.0) 17 | rspec-support (3.9.0) 18 | 19 | PLATFORMS 20 | ruby 21 | 22 | DEPENDENCIES 23 | rspec (~> 3.9) 24 | 25 | BUNDLED WITH 26 | 2.0.2 27 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2018 Doctolib 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # dl_experiment 2 | 3 | A lightweight scientist-like framework to refactor critical paths. 4 | 5 | ## Requirements 6 | 7 | - Ruby 2.3+ 8 | 9 | ## Usage 10 | 11 | Just drop this line in your Gemfile: 12 | 13 | ```rb 14 | gem 'dl_experiment' 15 | ``` 16 | 17 | Let's consider the following method we'd like to refactor: 18 | 19 | ```rb 20 | def allows?(user) 21 | model.check_user?(user).valid? 22 | end 23 | ``` 24 | 25 | If we'd like to use cancancan, we could add the following experiment: 26 | 27 | ```rb 28 | def allows?(user) 29 | Experiment.protocol('cancancan') do |e| 30 | e.legacy { model.check_user?(user).valid? } # old way 31 | e.experiment { user.can?(:read, model) } # new way 32 | end 33 | end 34 | ``` 35 | 36 | Your code will still return the same thing (or raise the same exception), but, from now on, assuming you are using rails, your test suite will fail if there is a single time where both implementations are not returning the same thing. 37 | 38 | More interesting: you can trigger a code block when there is a difference and log it the way you'd like: 39 | 40 | ```rb 41 | def allows?(user) 42 | Experiment.protocol('cancancan') do |e| 43 | e.legacy { model.check_user?(user).valid? } # old way 44 | e.experiment { user.can?(:read, model) } # new way 45 | 46 | e.on_diff do |legacy, experiment| 47 | Rails.logger.warn( 48 | "[Experiment][User:#{user}] Results not equals: " + 49 | "#{legacy.value} != #{experiment.value}" 50 | ) 51 | end 52 | end 53 | end 54 | ``` 55 | 56 | This will allow you to compare, even in production, some code implementation. 57 | 58 | *Warning:* Be careful with side effects. You don't want to create twice the same data in your database in production. Don't experiment on non-functional code. 59 | 60 | ## Motivation 61 | 62 | Sometimes, you'd like to change some code you don't fully understand and that is not fully covered by your tests. 63 | 64 | This tool, like scientist (the framework from github), is made to help you do that, but with a smaller integration cost. 65 | 66 | ## Feature set 67 | 68 | 69 | ## Runnings tests 70 | 71 | ```bash 72 | bundle 73 | rspec 74 | ``` 75 | 76 | ## Authors 77 | 78 | - [Alexandre Ignjatovic](https://github.com/bankair) 79 | 80 | ## License 81 | 82 | [MIT](https://github.com/doctolib/dl_experiment/blob/master/LICENSE) © [Doctolib](https://github.com/doctolib/) 83 | 84 | ## Additional resources 85 | 86 | Alternatives: 87 | 88 | - https://github.com/github/scientist 89 | - https://github.com/testdouble/suture 90 | 91 | -------------------------------------------------------------------------------- /dl_experiment.gemspec: -------------------------------------------------------------------------------- 1 | Gem::Specification.new do |s| 2 | s.name = 'dl_experiment' 3 | s.version = '0.0.1' 4 | s.date = '2019-12-11' 5 | s.summary = "A mini scientist-like framework" 6 | s.description = "A framework meant to help you test (in production or in tests) the impact of an iso functional change" 7 | s.authors = ["Alexandre Ignjatovic"] 8 | s.email = 'alexandre.ignjatovic@doctolib.com' 9 | s.files = ["lib/dl_experiment.rb", "lib/experiment.rb"] 10 | s.homepage = 11 | 'https://rubygems.org/gems/hola' 12 | s.license = 'MIT' 13 | end 14 | -------------------------------------------------------------------------------- /lib/dl_experiment.rb: -------------------------------------------------------------------------------- 1 | require 'experiment' 2 | 3 | -------------------------------------------------------------------------------- /lib/experiment.rb: -------------------------------------------------------------------------------- 1 | # frozen_string_literal: true 2 | 3 | class Experiment 4 | class << self 5 | def protocol(name) 6 | name = String(name) 7 | raise('Please provide an experiment name') if name.empty? 8 | experiment = Experiment.new(name) 9 | yield(experiment) 10 | experiment.raise_on_diff! if rails_test_mode? 11 | experiment.result 12 | end 13 | 14 | def rails_test_mode? 15 | defined?(Rails) && Rails.env.test? 16 | end 17 | end 18 | 19 | class Result 20 | attr_accessor :value, :error 21 | def initialize(value: nil, error: nil) 22 | self.value = value 23 | self.error = error 24 | end 25 | end 26 | 27 | DEFAULT_COMPARISON = ->(legacy, experiment) { legacy == experiment } 28 | DEFAULT_ENABLER = -> { true } 29 | 30 | def initialize(name) 31 | @name = name 32 | @compare_with = DEFAULT_COMPARISON 33 | @enable = DEFAULT_ENABLER 34 | end 35 | 36 | def legacy(&block) 37 | raise 'Missing block' unless block 38 | @legacy = block 39 | self 40 | end 41 | 42 | def experiment(&block) 43 | raise 'Missing block' unless block 44 | @experiment = block 45 | self 46 | end 47 | 48 | def compare_with(&block) 49 | raise 'Missing block' unless block 50 | @compare_with = block 51 | self 52 | end 53 | 54 | def enable(&block) 55 | raise 'Missing block' unless block 56 | @enable = block 57 | self 58 | end 59 | 60 | def on_diff(&block) 61 | raise 'Missing block' unless block 62 | @on_diff = block 63 | self 64 | end 65 | 66 | def result 67 | raise 'Please call the legacy helper in your protocol block' unless @legacy 68 | raise 'Please call the experiment helper in your protocol block' unless @experiment 69 | legacy_result = exec(@legacy) 70 | return forward(legacy_result) unless self.class.rails_test_mode? || @enable.call 71 | experiment_result = exec(@experiment) 72 | if @on_diff 73 | if legacy_result.error.class != experiment_result.error.class || 74 | legacy_result.error&.message != experiment_result.error&.message || 75 | !@compare_with.call(legacy_result.value, experiment_result.value) 76 | @on_diff.call(legacy_result, experiment_result) 77 | end 78 | end 79 | forward(legacy_result) 80 | end 81 | 82 | def raise_on_diff! 83 | on_diff do |legacy_result, experiment_result| 84 | raise ExperimentError, 85 | "Experiment: #{@name}; "\ 86 | "Legacy result: #{legacy_result.inspect}; "\ 87 | "Experiment result: #{experiment_result.inspect}" 88 | end 89 | end 90 | 91 | private 92 | 93 | def forward(result) 94 | raise result.error if result.error 95 | result.value 96 | end 97 | 98 | def exec(block) 99 | Result.new(value: block.call) 100 | rescue StandardError => error 101 | Result.new(error: error) 102 | end 103 | 104 | class ExperimentError < StandardError; end 105 | end 106 | -------------------------------------------------------------------------------- /spec/experiment_spec.rb: -------------------------------------------------------------------------------- 1 | require 'experiment' 2 | 3 | RSpec.describe Experiment do 4 | let (:failing_experiment) do 5 | Experiment.protocol('failing') do |e| 6 | e.legacy { :legacy } 7 | e.experiment { :experiment } 8 | end 9 | end 10 | 11 | let (:successful_experiment) do 12 | Experiment.protocol('successful') do |e| 13 | e.legacy { :same_result } 14 | e.experiment { :same_result } 15 | end 16 | end 17 | 18 | context 'when in Rails test mode' do 19 | before { stub_const('Rails', double(env: double(test?: true))) } 20 | 21 | it { expect{failing_experiment}.to raise_error(Experiment::ExperimentError) } 22 | end 23 | 24 | context 'when not in test mode' do 25 | it { expect { Experiment.protocol(nil) }.to raise_error(/Please provide an experiment name/) } 26 | 27 | it { expect(failing_experiment).to eq(:legacy) } 28 | 29 | it { expect(successful_experiment).to eq(:same_result) } 30 | 31 | it 'raise when #raise_on_diff! is called and legacy != experiment' do 32 | expect do 33 | Experiment.protocol('fake experiment') do |exp| 34 | exp.legacy { :legacy } 35 | exp.experiment { :experiment } 36 | exp.raise_on_diff! 37 | end 38 | end.to raise_error(Experiment::ExperimentError) 39 | end 40 | 41 | it 'raise the legacy exception' do 42 | expect do 43 | Experiment.protocol('fake experiment') do |exp| 44 | exp.legacy { raise 'legacy' } 45 | exp.experiment { raise 'experiment' } 46 | end 47 | end.to raise_error(/legacy/) 48 | end 49 | 50 | it 'detect when exception messages are different' do 51 | on_diff_called = false 52 | expect do 53 | Experiment.protocol('fake experiment') do |exp| 54 | exp.legacy { raise 'legacy' } 55 | exp.experiment { raise 'experiment' } 56 | exp.on_diff { on_diff_called = true } 57 | end 58 | end.to raise_error(/legacy/) 59 | expect(on_diff_called).to be(true) 60 | end 61 | 62 | 63 | 64 | it 'call on_diff block when values are not equal' do 65 | legacy = double 66 | experiment = double 67 | on_diff_called = false 68 | Experiment.protocol('fake experiment') do |exp| 69 | exp.legacy { legacy } 70 | exp.experiment { experiment } 71 | exp.on_diff do |legacy_result, experiment_result| 72 | expect(legacy).to eq(legacy_result.value) 73 | expect(experiment).to eq(experiment_result.value) 74 | on_diff_called = true 75 | end 76 | end 77 | expect(on_diff_called).to be(true) 78 | end 79 | 80 | it 'call on_diff block when an exception is raised' do 81 | on_diff_called = false 82 | Experiment.protocol('fake experiment') do |exp| 83 | exp.legacy { :legacy } 84 | exp.experiment { raise 'experiment' } 85 | exp.on_diff do |legacy_result, experiment_result| 86 | expect(:legacy).to eq(legacy_result.value) 87 | expect(legacy_result.error).to be_nil 88 | expect(experiment_result.value).to be_nil 89 | expect('experiment').to eq(experiment_result.error.message) 90 | expect(RuntimeError).to eq(experiment_result.error.class) 91 | on_diff_called = true 92 | end 93 | end 94 | expect(on_diff_called).to be(true) 95 | end 96 | 97 | it 'use compare_with block when provided' do 98 | compare_with_block_called = false 99 | on_diff_called = false 100 | Experiment.protocol('fake experiment') do |exp| 101 | exp.legacy { :legacy } 102 | exp.experiment { :experiment } 103 | exp.compare_with do |legacy, experiment| 104 | compare_with_block_called = true 105 | expect(:legacy).to eq(legacy) 106 | expect(:experiment).to eq(experiment) 107 | true 108 | end 109 | exp.on_diff { on_diff_called = true } 110 | end 111 | expect(compare_with_block_called).to be_truthy 112 | expect(on_diff_called).to be_falsey 113 | compare_with_block_called = false 114 | on_diff_called = false 115 | Experiment.protocol('fake experiment') do |exp| 116 | exp.legacy { :legacy } 117 | exp.experiment { :experiment } 118 | exp.compare_with do |_legacy, _experiment| 119 | compare_with_block_called = true 120 | false 121 | end 122 | exp.on_diff { on_diff_called = true } 123 | end 124 | expect(compare_with_block_called).to be_truthy 125 | expect(on_diff_called).to be_truthy 126 | end 127 | 128 | 129 | 130 | end 131 | end 132 | -------------------------------------------------------------------------------- /spec/spec_helper.rb: -------------------------------------------------------------------------------- 1 | # This file was generated by the `rspec --init` command. Conventionally, all 2 | # specs live under a `spec` directory, which RSpec adds to the `$LOAD_PATH`. 3 | # The generated `.rspec` file contains `--require spec_helper` which will cause 4 | # this file to always be loaded, without a need to explicitly require it in any 5 | # files. 6 | # 7 | # Given that it is always loaded, you are encouraged to keep this file as 8 | # light-weight as possible. Requiring heavyweight dependencies from this file 9 | # will add to the boot time of your test suite on EVERY test run, even for an 10 | # individual file that may not need all of that loaded. Instead, consider making 11 | # a separate helper file that requires the additional dependencies and performs 12 | # the additional setup, and require it from the spec files that actually need 13 | # it. 14 | # 15 | # See http://rubydoc.info/gems/rspec-core/RSpec/Core/Configuration 16 | RSpec.configure do |config| 17 | # rspec-expectations config goes here. You can use an alternate 18 | # assertion/expectation library such as wrong or the stdlib/minitest 19 | # assertions if you prefer. 20 | config.expect_with :rspec do |expectations| 21 | # This option will default to `true` in RSpec 4. It makes the `description` 22 | # and `failure_message` of custom matchers include text for helper methods 23 | # defined using `chain`, e.g.: 24 | # be_bigger_than(2).and_smaller_than(4).description 25 | # # => "be bigger than 2 and smaller than 4" 26 | # ...rather than: 27 | # # => "be bigger than 2" 28 | expectations.include_chain_clauses_in_custom_matcher_descriptions = true 29 | end 30 | 31 | # rspec-mocks config goes here. You can use an alternate test double 32 | # library (such as bogus or mocha) by changing the `mock_with` option here. 33 | config.mock_with :rspec do |mocks| 34 | # Prevents you from mocking or stubbing a method that does not exist on 35 | # a real object. This is generally recommended, and will default to 36 | # `true` in RSpec 4. 37 | mocks.verify_partial_doubles = true 38 | end 39 | 40 | # This option will default to `:apply_to_host_groups` in RSpec 4 (and will 41 | # have no way to turn it off -- the option exists only for backwards 42 | # compatibility in RSpec 3). It causes shared context metadata to be 43 | # inherited by the metadata hash of host groups and examples, rather than 44 | # triggering implicit auto-inclusion in groups with matching metadata. 45 | config.shared_context_metadata_behavior = :apply_to_host_groups 46 | 47 | # The settings below are suggested to provide a good initial experience 48 | # with RSpec, but feel free to customize to your heart's content. 49 | =begin 50 | # This allows you to limit a spec run to individual examples or groups 51 | # you care about by tagging them with `:focus` metadata. When nothing 52 | # is tagged with `:focus`, all examples get run. RSpec also provides 53 | # aliases for `it`, `describe`, and `context` that include `:focus` 54 | # metadata: `fit`, `fdescribe` and `fcontext`, respectively. 55 | config.filter_run_when_matching :focus 56 | 57 | # Allows RSpec to persist some state between runs in order to support 58 | # the `--only-failures` and `--next-failure` CLI options. We recommend 59 | # you configure your source control system to ignore this file. 60 | config.example_status_persistence_file_path = "spec/examples.txt" 61 | 62 | # Limits the available syntax to the non-monkey patched syntax that is 63 | # recommended. For more details, see: 64 | # - http://rspec.info/blog/2012/06/rspecs-new-expectation-syntax/ 65 | # - http://www.teaisaweso.me/blog/2013/05/27/rspecs-new-message-expectation-syntax/ 66 | # - http://rspec.info/blog/2014/05/notable-changes-in-rspec-3/#zero-monkey-patching-mode 67 | config.disable_monkey_patching! 68 | 69 | # This setting enables warnings. It's recommended, but in some cases may 70 | # be too noisy due to issues in dependencies. 71 | config.warnings = true 72 | 73 | # Many RSpec users commonly either run the entire suite or an individual 74 | # file, and it's useful to allow more verbose output when running an 75 | # individual spec file. 76 | if config.files_to_run.one? 77 | # Use the documentation formatter for detailed output, 78 | # unless a formatter has already been configured 79 | # (e.g. via a command-line flag). 80 | config.default_formatter = "doc" 81 | end 82 | 83 | # Print the 10 slowest examples and example groups at the 84 | # end of the spec run, to help surface which specs are running 85 | # particularly slow. 86 | config.profile_examples = 10 87 | 88 | # Run specs in random order to surface order dependencies. If you find an 89 | # order dependency and want to debug it, you can fix the order by providing 90 | # the seed, which is printed after each run. 91 | # --seed 1234 92 | config.order = :random 93 | 94 | # Seed global randomization in this process using the `--seed` CLI option. 95 | # Setting this allows you to use `--seed` to deterministically reproduce 96 | # test failures related to randomization by passing the same `--seed` value 97 | # as the one that triggered the failure. 98 | Kernel.srand config.seed 99 | =end 100 | end 101 | --------------------------------------------------------------------------------