├── .gitignore ├── Gemfile ├── LICENSE.txt ├── README.md ├── Rakefile ├── lib ├── puma_doctor.rb └── puma_doctor │ ├── capistrano.rb │ ├── daemon_template.rb.erb │ ├── doctor.rb │ ├── logger.rb │ └── version.rb └── puma_doctor.gemspec /.gitignore: -------------------------------------------------------------------------------- 1 | *.gem 2 | *.rbc 3 | .bundle 4 | .config 5 | .yardoc 6 | Gemfile.lock 7 | InstalledFiles 8 | _yardoc 9 | coverage 10 | doc/ 11 | lib/bundler/man 12 | pkg 13 | rdoc 14 | spec/reports 15 | test/tmp 16 | test/version_tmp 17 | tmp 18 | -------------------------------------------------------------------------------- /Gemfile: -------------------------------------------------------------------------------- 1 | source 'https://rubygems.org' 2 | 3 | # Specify your gem's dependencies in puma_doctor.gemspec 4 | gemspec 5 | -------------------------------------------------------------------------------- /LICENSE.txt: -------------------------------------------------------------------------------- 1 | Copyright (c) 2015 Alex Krasynskyi 2 | 3 | MIT License 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining 6 | a copy of this software and associated documentation files (the 7 | "Software"), to deal in the Software without restriction, including 8 | without limitation the rights to use, copy, modify, merge, publish, 9 | distribute, sublicense, and/or sell copies of the Software, and to 10 | permit persons to whom the Software is furnished to do so, subject to 11 | the following conditions: 12 | 13 | The above copyright notice and this permission notice shall be 14 | included in all copies or substantial portions of the Software. 15 | 16 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, 17 | EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF 18 | MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND 19 | NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE 20 | LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION 21 | OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION 22 | WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 23 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # PumaDoctor 2 | 3 | Inspired by ( https://github.com/schneems/puma_worker_killer ). Idea is to run 4 | separate process as a daemon to measure puma memory and restart worker when memory 5 | threshold reached. 6 | 7 | ## Installation 8 | 9 | Add this line to your application's Gemfile: 10 | 11 | gem 'puma_doctor' 12 | 13 | And then execute: 14 | 15 | $ bundle 16 | 17 | Or install it yourself as: 18 | 19 | $ gem install puma_doctor 20 | 21 | ## Usage 22 | 23 | ### Running from ruby code. 24 | To run from your code: 25 | 26 | PumaDoctor.start(frequency: 60, memory_threshold: 2000, puma_pid: 99999) 27 | 28 | This is not very useful in production since it blocks execution, but you can play 29 | around with options locally. Available options with defaults are: 30 | 31 | frequency: 60 # Interval in seconds 32 | puma_pid_file: 'puma.pid' # Location of puma pid file 33 | memory_threshold: 4000 # Amount in MB 34 | log_file: 'puma_doctor.log' # Name and location of log file 35 | 36 | ### Running as a daemon. 37 | 38 | To run as daemon you can create file with content below(Ex.: doctor.rb) 39 | 40 | require 'puma_doctor' 41 | require 'daemons' 42 | 43 | pid_dir = '../' # Path to directory to store pid. 44 | Daemons.run_proc('puma_doctor', { dir: pid_dir }) do 45 | PumaDoctor.start(frequency: 60, memory_threshold: 1000) 46 | end 47 | 48 | Then control it with(for more details visit https://github.com/thuehlinger/daemons): 49 | 50 | bundle exec ruby doctor.rb start 51 | bundle exec ruby doctor.rb stop 52 | bundle exec ruby doctor.rb restart 53 | 54 | ### Using with capistrano. 55 | 56 | Probably the easiest way to run `puma_doctor` in production is to use `capistrano`. Require script in `Capfile`: 57 | 58 | require 'puma_doctor/capistrano' 59 | 60 | This will add hook to start/restart daemon on `after deploy:finished`. If you want to start/stop from capistrano manually - this tasks are available: 61 | 62 | cap puma_doctor:check # Check if config file exixts on server 63 | cap puma_doctor:config # Config daemon 64 | cap puma_doctor:restart # Restart daemon 65 | cap puma_doctor:start # Start daemon 66 | cap puma_doctor:stop # Stop daemon 67 | 68 | Available options with defaults: 69 | 70 | set :puma_doctor_pid, -> { File.join(shared_path, 'tmp', 'pids', 'puma_doctor.pid') } 71 | set :puma_doctor_frequency, 30 #seconds 72 | set :puma_doctor_memory_threshold, 4000 #mb 73 | set :puma_doctor_daemon_file, -> { File.join(shared_path, 'puma_doctor_daemon.rb') } 74 | set :puma_doctor_log_file, -> { File.join(shared_path, 'log', 'puma_doctor.log') } 75 | set :puma_pid, -> { File.join(shared_path, 'tmp', 'pids', 'puma.pid') } 76 | 77 | 78 | ### Logging 79 | 80 | You can always see what `puma_doctor` is doing by reading logs. 81 | 82 | ## TODO 83 | 84 | Test 85 | 86 | ## Contributing 87 | 88 | 1. Fork it ( http://github.com/spilin/puma_doctor/fork ) 89 | 2. Create your feature branch (`git checkout -b my-new-feature`) 90 | 3. Commit your changes (`git commit -am 'Add some feature'`) 91 | 4. Push to the branch (`git push origin my-new-feature`) 92 | 5. Create new Pull Request 93 | -------------------------------------------------------------------------------- /Rakefile: -------------------------------------------------------------------------------- 1 | require "bundler/gem_tasks" 2 | -------------------------------------------------------------------------------- /lib/puma_doctor.rb: -------------------------------------------------------------------------------- 1 | require "puma_doctor/version" 2 | require "get_process_mem" 3 | 4 | require 'puma_doctor/doctor' 5 | require 'puma_doctor/logger' 6 | 7 | module PumaDoctor 8 | extend self 9 | 10 | attr_accessor :frequency, :pid_file, :puma_pid, :puma_pid_file, :memory_threshold, :log_file 11 | attr_reader :logger 12 | self.frequency = 60 # seconds 13 | self.pid_file = 'puma_doctor.pid' 14 | self.puma_pid_file = 'puma.pid' 15 | self.memory_threshold = 4000 # mb 16 | self.log_file = 'puma_doctor.log' 17 | 18 | def start(options = {}) 19 | @logger = ::PumaDoctor::Logger.new(log_file: options[:log_file] || self.log_file, log_level: options[:log_level]) 20 | @logger.log_start 21 | doctor = Doctor.new(default_options.merge(options).merge(logger: @logger)) 22 | loop do 23 | doctor.examine 24 | sleep(options[:frequency] || self.frequency) 25 | end 26 | end 27 | 28 | def default_options 29 | { 30 | memory_threshold: self.memory_threshold, 31 | puma_pid_file: self.puma_pid_file, 32 | puma_pid: self.puma_pid 33 | } 34 | end 35 | 36 | end 37 | -------------------------------------------------------------------------------- /lib/puma_doctor/capistrano.rb: -------------------------------------------------------------------------------- 1 | namespace :load do 2 | task :defaults do 3 | set :puma_doctor_pid, -> { File.join(shared_path, 'tmp', 'pids', 'puma_doctor.pid') } 4 | set :puma_doctor_frequency, 30 #seconds 5 | set :puma_doctor_memory_threshold, 4000 #mb 6 | set :puma_doctor_daemon_file, -> { File.join(shared_path, 'puma_doctor_daemon.rb') } 7 | set :puma_doctor_log_file, -> { File.join(shared_path, 'log', 'puma_doctor.log') } 8 | set :puma_pid, -> { File.join(shared_path, 'tmp', 'pids', 'puma.pid') } 9 | end 10 | end 11 | 12 | namespace :puma_doctor do 13 | desc 'Config daemon. Generate and send puma_doctor.rb' 14 | task :config do 15 | on roles(:app), in: :sequence, wait: 5 do 16 | config 17 | end 18 | end 19 | 20 | desc 'Start daemon' 21 | task :start do 22 | on roles(:app), in: :sequence, wait: 5 do 23 | within release_path do 24 | config 25 | execute :bundle, :exec, :ruby, fetch(:puma_doctor_daemon_file), 'start' 26 | end 27 | end 28 | end 29 | 30 | desc 'Stop daemon' 31 | task :stop do 32 | on roles(:app), in: :sequence, wait: 5 do 33 | within release_path do 34 | config 35 | execute :bundle, :exec, :ruby, fetch(:puma_doctor_daemon_file), 'stop' 36 | end 37 | end 38 | end 39 | 40 | desc 'Restart daemon' 41 | task :restart do 42 | on roles(:app), in: :sequence, wait: 5 do 43 | within release_path do 44 | config 45 | execute :bundle, :exec, :ruby, fetch(:puma_doctor_daemon_file), 'restart' 46 | end 47 | end 48 | end 49 | 50 | desc 'Check if config file exixts on server. If not - create and upload one.' 51 | task :check do 52 | on roles(:app), in: :sequence, wait: 5 do 53 | config unless test "[ -f #{fetch(:puma_doctor_daemon_file)} ]" 54 | end 55 | end 56 | 57 | after 'deploy:finished', 'puma_doctor:restart' 58 | 59 | def config 60 | path = File.expand_path("../daemon_template.rb.erb", __FILE__) 61 | if File.file?(path) 62 | erb = File.read(path) 63 | upload! StringIO.new(ERB.new(erb).result(binding)), fetch(:puma_doctor_daemon_file) 64 | end 65 | end 66 | end 67 | -------------------------------------------------------------------------------- /lib/puma_doctor/daemon_template.rb.erb: -------------------------------------------------------------------------------- 1 | require 'puma_doctor' 2 | require 'daemons' 3 | 4 | Daemons.run_proc('puma_doctor', { dir: '<%= File.dirname(fetch(:puma_doctor_pid)) %>' }) do 5 | options = { 6 | frequency: <%= fetch(:puma_doctor_frequency) %>, 7 | memory_threshold: <%= fetch(:puma_doctor_memory_threshold) %>, 8 | puma_pid_file: '<%= fetch(:puma_pid) %>', 9 | log_file: '<%= fetch(:puma_doctor_log_file) %>' 10 | } 11 | PumaDoctor.start(options) 12 | end 13 | -------------------------------------------------------------------------------- /lib/puma_doctor/doctor.rb: -------------------------------------------------------------------------------- 1 | module PumaDoctor 2 | class Doctor 3 | def initialize(options = {}) 4 | @memory_threshold = options[:memory_threshold] 5 | @puma_pid_file = options[:puma_pid_file] 6 | @puma_pid = options[:puma_pid] && options[:puma_pid].to_i 7 | @logger = options[:logger] 8 | end 9 | 10 | def examine 11 | @master_pid = get_master_pid(@master_pid) 12 | return if @master_pid.nil? 13 | workers = get_workers(@master_pid) # worker pids with size, last one is the largest one 14 | used_memory = workers.inject(0) {|memo, v| memo += v.last } + GetProcessMem.new(@master_pid).mb 15 | logger.info "[Puma Doctor] Total memory used: #{used_memory} mb. Workers online: #{workers.size}" 16 | if used_memory > @memory_threshold 17 | kill_largest_worker(workers) 18 | end 19 | end 20 | 21 | private 22 | 23 | def get_master_pid(current_puma_pid) 24 | if current_puma_pid && process_is_running?(current_puma_pid) 25 | current_puma_pid 26 | elsif current_puma_pid && (@puma_pid_file.nil? || !File.exists?(@puma_pid_file)) 27 | logger.warn "[Puma Doctor] Master pid is no longer represents running process. 28 | Reload failed because pid file is not set or invalid(File: '#{@puma_pid_file}')" 29 | nil 30 | elsif current_puma_pid && (current_puma_pid = File.read(@puma_pid_file).to_i) && process_is_running?(current_puma_pid) 31 | logger.warn "[Puma Doctor] Master pid is no longer represents running process. Successfully Reloaded pid file." 32 | current_puma_pid 33 | elsif @puma_pid && process_is_running?(@puma_pid) 34 | current_puma_pid = @puma_pid 35 | elsif @puma_pid_file && File.exists?(@puma_pid_file) && process_is_running?(current_puma_pid = File.read(@puma_pid_file).to_i) 36 | current_puma_pid 37 | else 38 | logger.warn "[Puma Doctor] Puma master pidfile is not found" 39 | nil 40 | end 41 | end 42 | 43 | def get_workers(puma_pid) 44 | `pgrep -P #{puma_pid} -d ','`.split(',').compact.map do |pid| 45 | [pid.to_i, GetProcessMem.new(pid).mb] 46 | end 47 | end 48 | 49 | def kill_largest_worker(workers) 50 | pid, memory_used = workers.max_by {|a| a[1]} 51 | Process.kill('TERM', pid) 52 | logger.info "[Puma Doctor] Doctor killed worker(#{pid}).It was using #{memory_used} mb. Workers online: #{workers.size - 1}" 53 | end 54 | 55 | def process_is_running?(pid) 56 | Process.getpgid(pid) 57 | true 58 | rescue Errno::ESRCH 59 | false 60 | end 61 | 62 | def logger 63 | @logger 64 | end 65 | 66 | end 67 | end 68 | -------------------------------------------------------------------------------- /lib/puma_doctor/logger.rb: -------------------------------------------------------------------------------- 1 | require 'logger' 2 | 3 | module PumaDoctor 4 | class Logger 5 | def initialize(options = {}) 6 | @logger = ::Logger.new(options[:log_file]) 7 | @logger.level = options[:log_level] || ::Logger::INFO 8 | end 9 | 10 | def info(text) 11 | @logger.info(text) 12 | end 13 | 14 | def warn(text) 15 | @logger.warn(text) 16 | end 17 | 18 | def log_start 19 | @logger.info "[Puma Doctor] Starting..." 20 | end 21 | 22 | end 23 | end 24 | -------------------------------------------------------------------------------- /lib/puma_doctor/version.rb: -------------------------------------------------------------------------------- 1 | module PumaDoctor 2 | VERSION = "0.0.2" 3 | end 4 | -------------------------------------------------------------------------------- /puma_doctor.gemspec: -------------------------------------------------------------------------------- 1 | # coding: utf-8 2 | lib = File.expand_path('../lib', __FILE__) 3 | $LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib) 4 | require 'puma_doctor/version' 5 | 6 | Gem::Specification.new do |spec| 7 | spec.name = "puma_doctor" 8 | spec.version = PumaDoctor::VERSION 9 | spec.authors = ["Alex Krasynskyi"] 10 | spec.email = ["lyoshakr@gmail.com"] 11 | spec.summary = %q{Process to keep your puma workers healthy.} 12 | spec.description = %q{Kills largest worker. Runs seperate daemon, managed with sidekiq.} 13 | spec.homepage = "https://github.com/spilin/puma_doctor" 14 | spec.license = "MIT" 15 | 16 | spec.files = `git ls-files`.split($/) 17 | spec.executables = [] 18 | spec.test_files = spec.files.grep(%r{^(test|spec|features)/}) 19 | spec.require_paths = ["lib"] 20 | 21 | spec.add_dependency "daemons" 22 | spec.add_dependency "get_process_mem" 23 | 24 | spec.add_development_dependency "bundler", "~> 1.5" 25 | spec.add_development_dependency "rake" 26 | end 27 | --------------------------------------------------------------------------------