├── .gitignore ├── .travis.yml ├── CODE_OF_CONDUCT.md ├── Gemfile ├── LICENSE.txt ├── README.md ├── Rakefile ├── bin ├── console └── setup ├── env_mem.gemspec ├── exe ├── env_mem └── gc_stat_dump.txt ├── lib ├── env_mem.rb └── env_mem │ └── version.rb ├── sample └── discourse_long_parallel_run.txt └── test ├── env_mem_test.rb └── test_helper.rb /.gitignore: -------------------------------------------------------------------------------- 1 | /.bundle/ 2 | /.yardoc 3 | /Gemfile.lock 4 | /_yardoc/ 5 | /coverage/ 6 | /doc/ 7 | /pkg/ 8 | /spec/reports/ 9 | /tmp/ 10 | -------------------------------------------------------------------------------- /.travis.yml: -------------------------------------------------------------------------------- 1 | sudo: false 2 | language: ruby 3 | rvm: 4 | - 2.3.1 5 | before_install: gem install bundler -v 1.14.6 6 | -------------------------------------------------------------------------------- /CODE_OF_CONDUCT.md: -------------------------------------------------------------------------------- 1 | # Contributor Covenant Code of Conduct 2 | 3 | ## Our Pledge 4 | 5 | In the interest of fostering an open and welcoming environment, we as 6 | contributors and maintainers pledge to making participation in our project and 7 | our community a harassment-free experience for everyone, regardless of age, body 8 | size, disability, ethnicity, gender identity and expression, level of experience, 9 | nationality, personal appearance, race, religion, or sexual identity and 10 | orientation. 11 | 12 | ## Our Standards 13 | 14 | Examples of behavior that contributes to creating a positive environment 15 | include: 16 | 17 | * Using welcoming and inclusive language 18 | * Being respectful of differing viewpoints and experiences 19 | * Gracefully accepting constructive criticism 20 | * Focusing on what is best for the community 21 | * Showing empathy towards other community members 22 | 23 | Examples of unacceptable behavior by participants include: 24 | 25 | * The use of sexualized language or imagery and unwelcome sexual attention or 26 | advances 27 | * Trolling, insulting/derogatory comments, and personal or political attacks 28 | * Public or private harassment 29 | * Publishing others' private information, such as a physical or electronic 30 | address, without explicit permission 31 | * Other conduct which could reasonably be considered inappropriate in a 32 | professional setting 33 | 34 | ## Our Responsibilities 35 | 36 | Project maintainers are responsible for clarifying the standards of acceptable 37 | behavior and are expected to take appropriate and fair corrective action in 38 | response to any instances of unacceptable behavior. 39 | 40 | Project maintainers have the right and responsibility to remove, edit, or 41 | reject comments, commits, code, wiki edits, issues, and other contributions 42 | that are not aligned to this Code of Conduct, or to ban temporarily or 43 | permanently any contributor for other behaviors that they deem inappropriate, 44 | threatening, offensive, or harmful. 45 | 46 | ## Scope 47 | 48 | This Code of Conduct applies both within project spaces and in public spaces 49 | when an individual is representing the project or its community. Examples of 50 | representing a project or community include using an official project e-mail 51 | address, posting via an official social media account, or acting as an appointed 52 | representative at an online or offline event. Representation of a project may be 53 | further defined and clarified by project maintainers. 54 | 55 | ## Enforcement 56 | 57 | Instances of abusive, harassing, or otherwise unacceptable behavior may be 58 | reported by contacting the project team at the.codefolio.guy@gmail.com. All 59 | complaints will be reviewed and investigated and will result in a response that 60 | is deemed necessary and appropriate to the circumstances. The project team is 61 | obligated to maintain confidentiality with regard to the reporter of an incident. 62 | Further details of specific enforcement policies may be posted separately. 63 | 64 | Project maintainers who do not follow or enforce the Code of Conduct in good 65 | faith may face temporary or permanent repercussions as determined by other 66 | members of the project's leadership. 67 | 68 | ## Attribution 69 | 70 | This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4, 71 | available at [http://contributor-covenant.org/version/1/4][version] 72 | 73 | [homepage]: http://contributor-covenant.org 74 | [version]: http://contributor-covenant.org/version/1/4/ 75 | -------------------------------------------------------------------------------- /Gemfile: -------------------------------------------------------------------------------- 1 | source 'https://rubygems.org' 2 | 3 | # Specify your gem's dependencies in env_mem.gemspec 4 | gemspec 5 | -------------------------------------------------------------------------------- /LICENSE.txt: -------------------------------------------------------------------------------- 1 | The MIT License (MIT) 2 | 3 | Copyright (c) 2018 Noah Gibbs 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in 13 | all copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN 21 | THE SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # EnvMem 2 | 3 | Ever read a web page about how to set your Ruby memory environment 4 | variables and thought, "but how do I know that's right for my app?" 5 | EnvMem is here to help you out. 6 | 7 | Specifically, if you have a long-running or high-memory Ruby process 8 | (server, batch, etc) then your process will do more garbage collecting 9 | than is necessary in getting up to its long-term size. You can save a 10 | bit of time and processor by setting its environment variables close 11 | to their steady-state or end-of-process values. 12 | 13 | This is the same thing you do when you set Ruby environment variables 14 | to more Rails-friendly, batch-friendly or your-server-friendly values 15 | from a web page. It's just that this way you can make sure it's a good 16 | match for your app, specifically. 17 | 18 | EnvMem generates a small, simple shellscript to set your environment 19 | variable values. To use it, just source the script before running your 20 | application. You can manually tweak it later if you like, or remove 21 | variables you don't want to set for some reason - such as OLDMALLOC 22 | values if you've compiled a Ruby without it, for instance. 23 | 24 | ## Installation 25 | 26 | Add this line to your application's Gemfile: 27 | 28 | ```ruby 29 | gem 'env_mem' 30 | ``` 31 | 32 | And then execute: 33 | 34 | $ bundle 35 | 36 | Or install it yourself as: 37 | 38 | $ gem install env_mem 39 | 40 | ## Usage 41 | 42 | EnvMem needs a dump of GC.stat values from your application in the 43 | configuration you want to match. If you have a long-running Rails 44 | server, that means after it has processed a bunch of HTTP requests. If 45 | you're using EnvMem to configure your batch script, that probably 46 | means dumping GC.stat after you've finished your batch work and your 47 | job's memory configuration is nice and stable. 48 | 49 | Since you'll need the GC.stat values from the process, you'll need to 50 | dump them. First, here's how to do it *without* EnvMem: 51 | 52 | ~~~ ruby 53 | File.open("gc_stat_dump.txt", "w") { |f| f.write GC.stat.inspect } 54 | ~~~ 55 | 56 | You can use EnvMem itself to dump GC.stat, but then you're using it at 57 | runtime. Here's how: 58 | 59 | ~~~ ruby 60 | require 'env_mem' 61 | EnvMem.dump_to_file("gc_stat_dump.txt") 62 | ~~~ 63 | 64 | To create the environment script from the stat dump, translate from one filename to another: 65 | 66 | ~~~ ruby 67 | $ env_mem gc_stat_dump.txt > env_wrapper.sh 68 | ~~~ 69 | 70 | Keep in mind that your application may change over time, and so it may 71 | need different memory settings. A simple way to handle that is to run 72 | your app *without* any Ruby memory environment variables set and then 73 | dump GC.stat again and regenerate them. 74 | 75 | ### Long-Running Servers and Other Challenges 76 | 77 | There isn't always an obvious way to get statistics at the start and 78 | the end of the process. Start is usually easy, but end can be a 79 | challenge. Here's something I've tried with a large Rails server that 80 | has worked okay: 81 | 82 | ~~~ ruby 83 | File.open("gc_stats_#{Process.pid}_start.txt", "w") { |f| f.print GC.stats.inspect } 84 | at_exit { 85 | File.open("gc_stats_#{Process.pid}_stop.txt", "w") { |f| f.print GC.stats.inspect } 86 | } 87 | ~~~ 88 | 89 | The "at_exit" block is saying that before the process exits, it should 90 | stop and write out the GC stats again. Doing this during teardown 91 | means you won't necessarily have an accurate count of how many active 92 | objects are currently sitting around... But most of your statistics 93 | will work great. 94 | 95 | You can get a tiny bit of extra accuracy by instead adding a 96 | controller action to dump the GC stats while the Rails server is still 97 | fully active. But for most purposes, this will do just fine. 98 | 99 | ### What the Variables Mean 100 | 101 | Ruby has two obvious thresholds, "malloc" and "oldmalloc", that keep 102 | going up. The "malloc" limit is so that Ruby garbage-collects 103 | regularly every so many bytes allocated. The "oldmalloc" limit is to 104 | garbage collect as (its estimate of) the old-generation size in bytes 105 | increases. 106 | 107 | Ordinarily a Ruby process will increase in size asymptotically, 108 | approaching its "full size." This is common for things like server 109 | processes that add and retain long-term memory (e.g. classes, caches) 110 | while adding a much smaller amount of per-request memory that gets 111 | garbage collected soon after the request is finished. 112 | 113 | After each time the limit causes a major garbage collection (e.g. the 114 | total allocated size crosses the "malloc" limit), that limit is raised 115 | by a configurable "growth factor". For instance, with the default 116 | RUBY\_GC\_MALLOC\_LIMIT\_GROWTH\_FACTOR of 1.4, the malloc limit will 117 | get 40% bigger each time. With a growth factor of 1.6, it would get 118 | 60% bigger. There can also be a LIMIT_MAX variable, so that the limit 119 | grows by the smaller of the growth factor or the limit max. For 120 | instance, with a growth factor of 1.6 and a limit max of 100,000, Ruby 121 | would grow its malloc limit by 60% each time until 60% was bigger than 122 | 100,000, and then it would grow by 100,000 each time. 123 | 124 | The malloc (but not oldmalloc) limit will also slowly decrease back 125 | toward what you specify, by around 2% with each garbage 126 | collection. But not below the specified limit. The 2% is a "magic 127 | number" and not configurable. 128 | 129 | Slots are slightly different than the malloc and oldmalloc limits - 130 | slots are fully managed by Ruby itself, while Ruby uses a system 131 | allocator to managed the malloc and oldmalloc systems. 132 | 133 | With slots, Ruby starts with RUBY\_GC\_HEAP\_INIT\_SLOTS of them 134 | allocated. Slots also have a growth factor 135 | (RUBY\_GC\_HEAP\_GROWTH\_FACTOR) and a maximum growth 136 | (RUBY\_GC\_HEAP\_GROWTH\_MAX\_SLOTS). But Ruby will only use them if 137 | you don't set ratios of free slots (see below.) By default, Ruby will 138 | aim for 40% of slots free, allocating more to reach this ratio. By 139 | default it will free pages of slots when at least 65% of its slots are free. 140 | 141 | Here is a list of the variables in question: 142 | 143 | * RUBY\_GC\_HEAP\_INIT\_SLOTS - initial number of slots 144 | * RUBY\_GC\_HEAP\_FREE\_SLOTS - minimum free slots allowable after GC 145 | * RUBY\_GC\_HEAP\_GROWTH\_FACTOR - growth factor for slots 146 | * RUBY\_GC\_HEAP\_GROWTH\_MAX\_SLOTS - maximum slots to add at one time 147 | * RUBY\_GC\_HEAP\_FREE\_SLOTS\_MIN\_RATIO - allocate additional slots when below this ratio 148 | * RUBY\_GC\_HEAP\_FREE\_SLOTS\_MAX\_RATIO - free pages of slots when above this ratio 149 | * RUBY\_GC\_HEAP\_FREE\_SLOTS\_GOAL\_RATIO - allocate slots to get to this ratio free (if 0.0, use the growth factor) 150 | 151 | * RUBY\_GC\_HEAP\_OLDOBJECT\_LIMIT\_FACTOR - do a major GC when the 152 | number of old objects is above this factor times the old objects 153 | after the *last* major GC. 154 | 155 | * RUBY\_GC\_MALLOC\_LIMIT 156 | * RUBY\_GC\_MALLOC\_LIMIT\_MAX 157 | * RUBY\_GC\_MALLOC\_LIMIT\_GROWTH\_FACTOR 158 | 159 | * RUBY\_GC\_OLDMALLOC\_LIMIT 160 | * RUBY\_GC\_OLDMALLOC\_LIMIT\_MAX 161 | * RUBY\_GC\_OLDMALLOC\_LIMIT\_GROWTH\_FACTOR 162 | 163 | ### Limitations 164 | 165 | First: EnvMem assumes you're using the latest Ruby or something close 166 | to it. If you want to improve speed and memory use in Ruby and you're 167 | running an old version, fix that first. 168 | 169 | There are a *lot* of things you can do with the Ruby environment 170 | variables, and many different applications with different needs. Right 171 | now, EnvMem tries to do a bit to help you. But there's always room for 172 | more. 173 | 174 | (You can view these as limitations in EnvMem. You can also view them 175 | as places *you* can begin optimization. Both are correct.) 176 | 177 | For instance: 178 | 179 | EnvMem doesn't try to preserve environment variable settings from 180 | before you ran it. If you changed any of the "growth factors," for 181 | instance, EnvMem won't currently change them. You may also want to 182 | reduce the growth factors for a fully mature application, or set some 183 | of the LIMIT\_MAX environment variables so that your app can't bloat as 184 | quickly. EnvMem won't do that for you either since it's so 185 | application-specific what "reasonable" behavior is. 186 | 187 | EnvMem also tries not to assert anything about the balance of old- and 188 | new-generation objects. In a tightly-optimized application you'd 189 | expect old objects to dominate, while an application that generates a 190 | lot of transient garbage may need different settings. It's possible to 191 | balance MALLOC\_LIMIT settings with OLDMALLOC\_LIMIT settings to 192 | affect this, but EnvMem doesn't try to. 193 | 194 | Similarly, you may want a much smaller FREE\_SLOTS ratio with a more 195 | mature, more tightly-tuned application. EnvMem doesn't look at this, 196 | either. 197 | 198 | ## Development 199 | 200 | After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake test` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment. 201 | 202 | To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org). 203 | 204 | ## Contributing 205 | 206 | Bug reports and pull requests are welcome on GitHub at https://github.com/noahgibbs/env_mem. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [Contributor Covenant](http://contributor-covenant.org) code of conduct. 207 | 208 | 209 | ## License 210 | 211 | The gem is available as open source under the terms of the [MIT License](http://opensource.org/licenses/MIT). 212 | 213 | -------------------------------------------------------------------------------- /Rakefile: -------------------------------------------------------------------------------- 1 | require "bundler/gem_tasks" 2 | require "rake/testtask" 3 | 4 | Rake::TestTask.new(:test) do |t| 5 | t.libs << "test" 6 | t.libs << "lib" 7 | t.test_files = FileList['test/**/*_test.rb'] 8 | end 9 | 10 | task :default => :test 11 | -------------------------------------------------------------------------------- /bin/console: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env ruby 2 | 3 | require "bundler/setup" 4 | require "env_mem" 5 | 6 | # You can add fixtures and/or initialization code here to make experimenting 7 | # with your gem easier. You can also use a different console, if you like. 8 | 9 | # (If you use this, don't forget to add pry to your Gemfile!) 10 | # require "pry" 11 | # Pry.start 12 | 13 | require "irb" 14 | IRB.start(__FILE__) 15 | -------------------------------------------------------------------------------- /bin/setup: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | set -euo pipefail 3 | IFS=$'\n\t' 4 | set -vx 5 | 6 | bundle install 7 | 8 | # Do any other automated setup that you need to do here 9 | -------------------------------------------------------------------------------- /env_mem.gemspec: -------------------------------------------------------------------------------- 1 | # coding: utf-8 2 | lib = File.expand_path('../lib', __FILE__) 3 | $LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib) 4 | require 'env_mem/version' 5 | 6 | Gem::Specification.new do |spec| 7 | spec.name = "env_mem" 8 | spec.version = EnvMem::VERSION 9 | spec.authors = ["Noah Gibbs"] 10 | spec.email = ["the.codefolio.guy@gmail.com"] 11 | 12 | spec.summary = %q{EnvMem takes a GC.stat dump and produce a memory-optimized script for your large Ruby app.} 13 | spec.description = %q{EnvMem allows you to dump your GC.stat information from a long-running server, then produce a simple shellscript to set up your Ruby memory environment variables to match that configuration. This improves startup time slightly and allows you finer-grain control over your memory setup. If you've ever found Ruby memory environment variables on a web page and used them to try to speed up your app, this is a better approach.} 14 | spec.homepage = "https://github.com/noahgibbs/env_mem" 15 | spec.license = "MIT" 16 | 17 | spec.files = `git ls-files -z`.split("\x0").reject do |f| 18 | f.match(%r{^(test|spec|features)/}) 19 | end 20 | spec.bindir = "exe" 21 | spec.executables = spec.files.grep(%r{^exe/}) { |f| File.basename(f) } 22 | spec.require_paths = ["lib"] 23 | 24 | spec.add_development_dependency "bundler", "~> 1.14" 25 | spec.add_development_dependency "rake", "~> 10.0" 26 | spec.add_development_dependency "minitest", "~> 5.0" 27 | end 28 | -------------------------------------------------------------------------------- /exe/env_mem: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env ruby 2 | 3 | if ARGV.size < 1 4 | STDERR.puts "You must supply a filename for the GC.stat dump!" 5 | exit -1 6 | end 7 | 8 | devel_dir = File.join(__dir__, "..", "lib") 9 | if File.exist? devel_dir 10 | $LOAD_PATH.unshift devel_dir 11 | end 12 | require "env_mem" 13 | 14 | out = EnvMem.gc_stat_to_shell File.read(ARGV[0]).chomp 15 | 16 | puts out 17 | -------------------------------------------------------------------------------- /exe/gc_stat_dump.txt: -------------------------------------------------------------------------------- 1 | {:count=>11, :heap_allocated_pages=>132, :heap_sorted_length=>133, :heap_allocatable_pages=>0, :heap_available_slots=>53801, :heap_live_slots=>49948, :heap_free_slots=>3853, :heap_final_slots=>0, :heap_marked_slots=>21045, :heap_swept_slots=>26874, :heap_eden_pages=>123, :heap_tomb_pages=>9, :total_allocated_pages=>132, :total_freed_pages=>0, :total_allocated_objects=>191166, :total_freed_objects=>141218, :malloc_increase_bytes=>4240, :malloc_increase_bytes_limit=>16777216, :minor_gc_count=>8, :major_gc_count=>3, :remembered_wb_unprotected_objects=>201, :remembered_wb_unprotected_objects_limit=>378, :old_objects=>20381, :old_objects_limit=>36098, :oldmalloc_increase_bytes=>463088, :oldmalloc_increase_bytes_limit=>16777216} -------------------------------------------------------------------------------- /lib/env_mem.rb: -------------------------------------------------------------------------------- 1 | require "env_mem/version" 2 | 3 | module EnvMem 4 | extend self 5 | 6 | def dump_to_file(filename) 7 | File.open(filename, "w") { |f| f.write GC.stat.inspect } 8 | end 9 | 10 | def gc_stat_to_shell(stats) 11 | stats_hash = {} 12 | stats.scan(/:([a-zA-Z_]+)\s*=>\s*([0-9]+)/).each { |key, val| stats_hash[key] = val.to_i } 13 | 14 | <127, :heap_allocated_pages=>3206, :heap_sorted_length=>3206, :heap_allocatable_pages=>0, :heap_available_slots=>1306745, :heap_live_slots=>1306012, :heap_free_slots=>733, :heap_final_slots=>0, :heap_marked_slots=>793684, :heap_eden_pages=>3206, :heap_tomb_pages=>0, :total_allocated_pages=>3206, :total_freed_pages=>0, :total_allocated_objects=>36083815, :total_freed_objects=>34777803, :malloc_increase_bytes=>2643440, :malloc_increase_bytes_limit=>30330547, :minor_gc_count=>113, :major_gc_count=>14, :remembered_wb_unprotected_objects=>82721, :remembered_wb_unprotected_objects_limit=>165276, :old_objects=>707426, :old_objects_limit=>1411534, :oldmalloc_increase_bytes=>18853704, :oldmalloc_increase_bytes_limit=>53381097} 2 | -------------------------------------------------------------------------------- /test/env_mem_test.rb: -------------------------------------------------------------------------------- 1 | require 'test_helper' 2 | 3 | class EnvMemTest < Minitest::Test 4 | def test_that_it_has_a_version_number 5 | refute_nil ::EnvMem::VERSION 6 | end 7 | 8 | def test_it_does_something_useful 9 | assert false 10 | end 11 | end 12 | -------------------------------------------------------------------------------- /test/test_helper.rb: -------------------------------------------------------------------------------- 1 | $LOAD_PATH.unshift File.expand_path('../../lib', __FILE__) 2 | require 'env_mem' 3 | 4 | require 'minitest/autorun' 5 | --------------------------------------------------------------------------------