├── README.markdown ├── lib ├── sim_cache.rb └── sim_cache │ ├── log_replayer.rb │ ├── mock_cache.rb │ └── replay_statistics_methods.rb └── spec ├── fixtures ├── cache_util.png ├── miss_percentage.png ├── predicted_vs_actual.png ├── sample_out.log └── test.log ├── log_replayer_spec.rb ├── mock_cache_spec.rb └── spec_helper.rb /README.markdown: -------------------------------------------------------------------------------- 1 | # SimCache - A tool for investigating cache performance given observed log data 2 | 3 | Modern web applications depend heavily on caching strategies to reduce load on primary databases and to boost performance of slow-running or heavily-utilized features or queries. 4 | 5 | When evaluating different caching strategies (e.g., Memcache vs. Redis vs. something else), it is often important to understand the expected load that the cache will be expected to bear. During this planning, several key questions must often be answered: 6 | 7 | * What read and write throughput must the cache sustain? 8 | * How large should our cache be to deliver a given level of performance? 9 | * How will cache performance degrade with increased traffic? 10 | * How long does it take to saturate the cache? 11 | * What will be the steady-state performance of our cache? 12 | 13 | Answering such questions may help inform decisions about which caching strategy is appropriate, and how costly such a strategy would be. 14 | 15 | ## Why SimCache? 16 | 17 | SimCache is a simple tool replays a log file against an idealized LRU cache of a fixed size (specified by the user). Using this log data, SimCache will create a timeseries report with the following metrics: 18 | 19 | 1. Requests / second 20 | 2. Cache Hits / second 21 | 3. Cache Misses / second 22 | 4. Hit Percentage 23 | 5. Total keys stored in cache 24 | 6. Percentage of cache used 25 | 7. Aggregate percentage of hits 26 | 27 | # Usage 28 | 29 | SimCache consists of two parts: a simulated LRU cache with a user-specified size, and a replay script that plays a log file against the idealized cache. 30 | 31 | Please note that most LRU caches (e.g., Memcache) do not behave as a perfect LRU cache. Therefore, the results obtained from SimCache will not perfectly match reality perfectly. However, the results are accurate enough to provide key characteristics about cache performance. 32 | 33 | ## The Log File 34 | 35 | The log file must be in the following format. The bracketed time stamp designates when an object was accessed; the second column is any string that uniquely identifies the object requested from the cache. For an example, see `spec/fixtures/test.log` (excerpted here): 36 | 37 | [Feb 28 01:34:31] post_44164631_js 38 | [Feb 28 01:34:59] post_44164631_js 39 | [Feb 28 01:52:36] post_11080788_html 40 | [Feb 28 02:09:28] post_44427139_html 41 | [Feb 28 02:26:08] post_19142226_js 42 | [Feb 28 02:45:01] post_29425455_js 43 | [Feb 28 03:01:27] post_20254178_js 44 | [Feb 28 03:20:21] post_44359317_js 45 | 46 | ## Replaying Log Files 47 | 48 | To start a replay, simply do the following: 49 | 50 | require 'sim_cache' 51 | SimCache::LogReplayer.new( 52 | :log_file => "/path/to/logfile.log", 53 | :report_file => "/path/to/report.log, 54 | :cache_options => {:max_keys => 6_000_000} 55 | ).replay! 56 | 57 | ## Report Output 58 | 59 | The report that is generated is in the following format (excerpted from `spec/fixtures/sample_out.log`): 60 | 61 | [ Time] Unix Time req/s hits/s misses/sec % hit - TotKeys %Cache %Hits 62 | -------------------------------------------------------------------------------------------------------- 63 | [ 02-28 01:34] 1298885640 0.0333 0.0167 0.0167 0.50 - 1.00e+00 10.00 0.00 64 | [ 02-28 01:52] 1298886720 0.0167 0.0000 0.0167 0.00 - 2.00e+00 20.00 33.33 65 | [ 02-28 02:09] 1298887740 0.0167 0.0000 0.0167 0.00 - 3.00e+00 30.00 25.00 66 | [ 02-28 02:26] 1298888760 0.0167 0.0000 0.0167 0.00 - 4.00e+00 40.00 20.00 67 | [ 02-28 02:45] 1298889900 0.0167 0.0000 0.0167 0.00 - 5.00e+00 50.00 16.67 68 | 69 | ## Plotting Results 70 | 71 | Gnuplot may be used to plot the data in the report file immediately. As an example, the following Gnuplot command will plot the miss percentage for four different sizes of cache: 72 | 73 | plot 'log_0' u 4:(100*($7/$5)) w l, 'log_1' u 4:(100*($7/$5)) w l, 'log_2' u 4:(100*($7/$5)) w l, 'log_3' u 4:(100*($7/$5)) w l 74 | 75 | Graph of results: 76 | 77 | ![Cache Miss Percentage](https://github.com/vincentchu/sim_cache/blob/master/spec/fixtures/miss_percentage.png?raw=true) 78 | 79 | Similarly, the following: 80 | 81 | plot 'log_0' u 4:($11) w l, 'log_1' u 4:($11) w l, 'log_2' u 4:($11) w l, 'log_3' u 4:($11) w l 82 | 83 | will plot the cache utilization percentage over time: 84 | 85 | ![Cache Utilization](https://github.com/vincentchu/sim_cache/blob/master/spec/fixtures/cache_util.png?raw=true) 86 | 87 | We have used SimCache at Posterous to analyze our cache performance. Using the data from our logs, we used SimCache to appropriately size our cache. This is how the actual results stack up against prediction. At time = 0, the larger cache was put into place, leading to a spike in miss rate. As the cache filled, the actual miss rate matched the predicted miss rate quite well (1 == baseline). 88 | 89 | ![Prediction vs. Actual](https://github.com/vincentchu/sim_cache/blob/master/spec/fixtures/predicted_vs_actual.png?raw=true) 90 | 91 | # Requirements 92 | 93 | SimCache uses Redis as a backing store and uses the redis gem (v2.0.5). So you'll have to install that. 94 | 95 | # Speed 96 | 97 | Informally, SimCache is pretty fast. On my 2.4 GHz Macbook Pro with 8 GB RAM I was able to replay ~1 million log entries in just over 4 minutes while doing normal activities (e.g., browsing the web, watching YouTube). Your mileage may vary. If you have a large sample of data (e.g., tens of millions of rows or more), it might be best to take a subsample. 98 | 99 | However, be careful how you choose the subsample; instead of randomly selecting log entries, it's probably better to take *all* log entries for a given sample of keys. Doing so will ensure that the hit / miss rates are representative of the entire sample. 100 | 101 | # Contact 102 | 103 | SimCache was created by Vincent Chu (vince [at] posterous.com) and is used at Posterous. -------------------------------------------------------------------------------- /lib/sim_cache.rb: -------------------------------------------------------------------------------- 1 | require 'rubygems' 2 | require 'redis' 3 | require 'time' 4 | 5 | %w(mock_cache replay_statistics_methods log_replayer).each {|f| require File.join(File.dirname(__FILE__), "sim_cache", f)} 6 | -------------------------------------------------------------------------------- /lib/sim_cache/log_replayer.rb: -------------------------------------------------------------------------------- 1 | module SimCache 2 | class LogReplayer 3 | 4 | include ReplayStatisticsMethods 5 | 6 | attr_reader :log_file, :report_file, :mock_cache 7 | 8 | def initialize(opts) 9 | @log_file = opts[:log_file] 10 | @report_file = opts[:report_file] 11 | @mock_cache = MockCache.new( opts[:cache_options] ) 12 | end 13 | 14 | def replay! 15 | open_files! 16 | replay_logs! 17 | ensure 18 | close_files! 19 | end 20 | 21 | private 22 | 23 | def replay_logs! 24 | reset_minute_counters! 25 | output_report_header 26 | 27 | n_lines = 0 28 | while(log_item = read_next_logline) 29 | next if log_item.nil? 30 | process_log_item( log_item ) 31 | 32 | puts "[#{Time.now}] Processed #{n_lines} lines" if (n_lines%10000 == 0) 33 | n_lines += 1 34 | end 35 | output_report 36 | 37 | true 38 | end 39 | 40 | def process_log_item( log_item ) 41 | log_item[:cache_hit] = begin 42 | !!@mock_cache.get( log_item[:key] ) 43 | rescue MockCache::KeyNotFound => ex 44 | false 45 | end 46 | 47 | increment_minute_counters(log_item) 48 | end 49 | 50 | def output_report_header 51 | vals = [ 52 | "Time", 53 | "Unix Time", 54 | "req/s", 55 | "hits/s", 56 | "misses/sec", 57 | "% hit", 58 | "TotKeys", 59 | "%Cache", 60 | "%Hits" 61 | ] 62 | @report_file_handle.puts sprintf("[%12s] %11s %14s %14s %14s %6s - %8s %6s %6s", *vals) 63 | @report_file_handle.puts "-" * 104 64 | end 65 | 66 | def output_report 67 | @minute_counter.keys.sort.each do |index| 68 | output_report_line(index) 69 | end 70 | end 71 | 72 | def output_report_line(index) 73 | 74 | vals = [ 75 | Time.at(index).strftime("%m-%d %H:%M"), 76 | index, 77 | requests_per_second(index), 78 | hits_per_second(index), 79 | misses_per_second(index), 80 | hit_percentage(index), 81 | total_keys_stored(index), 82 | 100 * total_cache_utilization(index), 83 | 100 * total_cache_hit_rate(index) 84 | ] 85 | 86 | @report_file_handle.puts sprintf("[%12s] %11d %14.4f %14.4f %14.4f %6.2f - %8.2e %6.2f %6.2f", *vals) 87 | end 88 | 89 | def read_next_logline 90 | line = begin 91 | @log_file_handle.readline 92 | rescue EOFError => ex 93 | nil 94 | end 95 | 96 | (line =~ /\[(.*?)\] (.*?)$/) ? {:time => Time.parse($1), :key => $2} : nil 97 | end 98 | 99 | def open_files! 100 | @log_file_handle = File.open(@log_file, "r") 101 | @report_file_handle = File.open(@report_file, "w") 102 | end 103 | 104 | def close_files! 105 | @log_file_handle.close 106 | @report_file_handle.close 107 | end 108 | 109 | end 110 | end -------------------------------------------------------------------------------- /lib/sim_cache/mock_cache.rb: -------------------------------------------------------------------------------- 1 | module SimCache 2 | class MockCache 3 | 4 | DEFAULT_MAX_KEYS = 100_000 5 | DEFAULT_CACHE_NAME = "cache" 6 | attr_reader :redis, :cache_name, :max_keys, :hits, :misses, :num_keys 7 | 8 | KeyNotFound = Class.new(StandardError) 9 | 10 | def initialize(opts = {}) 11 | @max_keys = (opts[:max_keys] || DEFAULT_MAX_KEYS) 12 | @cache_name = (opts[:cache_name] || DEFAULT_CACHE_NAME) 13 | init_redis!(opts) 14 | init_counters! 15 | end 16 | 17 | def get(key) 18 | retval = @redis.client.call(:zadd, cache_name, -(Time.now.to_f*1000000).to_i, key) 19 | @num_keys += 1 if (retval == 1) 20 | prune_cache! 21 | 22 | if (retval == 1) 23 | @misses += 1 24 | raise KeyNotFound 25 | else 26 | @hits += 1 27 | end 28 | end 29 | 30 | def rank_for_key( key ) 31 | @redis.zrank(cache_name, key) 32 | end 33 | 34 | def percent_utilization 35 | self.num_keys.to_f / self.max_keys 36 | end 37 | 38 | def hit_rate 39 | hits.to_f / (hits + misses) 40 | end 41 | 42 | def miss_rate 43 | misses.to_f / (hits + misses) 44 | end 45 | 46 | private 47 | 48 | def init_counters! 49 | @misses = 0 50 | @hits = 0 51 | @num_keys = 0 52 | end 53 | 54 | def init_redis!(opts) 55 | ## Just use default for redis 56 | @redis_host = opts[:redis_host] || "localhost" 57 | @redis_port = opts[:redis_port] || 6379 58 | 59 | @redis = Redis.new(:host => @redis_host, :port => @redis_port) 60 | end 61 | 62 | def prune_cache! 63 | if (num_keys > max_keys) 64 | redis.zremrangebyrank(cache_name, max_keys, 2*max_keys) 65 | @num_keys = max_keys 66 | end 67 | end 68 | end 69 | end -------------------------------------------------------------------------------- /lib/sim_cache/replay_statistics_methods.rb: -------------------------------------------------------------------------------- 1 | module SimCache 2 | module ReplayStatisticsMethods 3 | 4 | def increment_minute_counters(log_item) 5 | index = round_time(log_item[:time]).to_i 6 | 7 | unless @minute_counter[index] 8 | @minute_counter[index] = {:hits => 0, :misses => 0, :cache_total_keys => 0, :cache_utilization => 0, :cache_hit_rate => 0} 9 | @minute_counter[index][:cache_total_keys] = @mock_cache.num_keys 10 | @minute_counter[index][:cache_utilization] = @mock_cache.percent_utilization 11 | @minute_counter[index][:cache_hit_rate] = @mock_cache.hit_rate 12 | end 13 | 14 | if log_item[:cache_hit] 15 | @minute_counter[index][:hits] += 1 16 | else 17 | @minute_counter[index][:misses] += 1 18 | end 19 | end 20 | 21 | def round_time(time) 22 | Time.at(time.to_i - time.to_i%60) 23 | end 24 | 25 | def misses_per_second(index) 26 | @minute_counter[index][:misses].to_f / 60 27 | end 28 | 29 | def hits_per_second(index) 30 | @minute_counter[index][:hits].to_f / 60 31 | end 32 | 33 | def requests_per_second(index) 34 | misses_per_second(index) + hits_per_second(index) 35 | end 36 | 37 | def total_requests(index) 38 | @minute_counter[index][:hits] + @minute_counter[index][:misses] 39 | end 40 | 41 | def hit_percentage(index) 42 | @minute_counter[index][:hits].to_f / total_requests(index) 43 | end 44 | 45 | def miss_percentage(index) 46 | @minute_counter[ndex][:misses].to_f / total_requests(index) 47 | end 48 | 49 | def total_keys_stored(index) 50 | @minute_counter[index][:cache_total_keys] 51 | end 52 | 53 | def total_cache_utilization(index) 54 | @minute_counter[index][:cache_utilization] 55 | end 56 | 57 | def total_cache_hit_rate(index) 58 | @minute_counter[index][:cache_hit_rate] 59 | end 60 | 61 | def reset_minute_counters! 62 | @minute_counter = {} 63 | end 64 | end 65 | end -------------------------------------------------------------------------------- /spec/fixtures/cache_util.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/vincentchu/sim_cache/cb6ab7b52344d94d8a5efa3ac459450b7aa0de1d/spec/fixtures/cache_util.png -------------------------------------------------------------------------------- /spec/fixtures/miss_percentage.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/vincentchu/sim_cache/cb6ab7b52344d94d8a5efa3ac459450b7aa0de1d/spec/fixtures/miss_percentage.png -------------------------------------------------------------------------------- /spec/fixtures/predicted_vs_actual.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/vincentchu/sim_cache/cb6ab7b52344d94d8a5efa3ac459450b7aa0de1d/spec/fixtures/predicted_vs_actual.png -------------------------------------------------------------------------------- /spec/fixtures/sample_out.log: -------------------------------------------------------------------------------- 1 | [ Time] Unix Time req/s hits/s misses/sec % hit - TotKeys %Cache %Hits 2 | -------------------------------------------------------------------------------------------------------- 3 | [ 02-28 01:34] 1298885640 0.0333 0.0167 0.0167 0.50 - 1.00e+00 10.00 0.00 4 | [ 02-28 01:52] 1298886720 0.0167 0.0000 0.0167 0.00 - 2.00e+00 20.00 33.33 5 | [ 02-28 02:09] 1298887740 0.0167 0.0000 0.0167 0.00 - 3.00e+00 30.00 25.00 6 | [ 02-28 02:26] 1298888760 0.0167 0.0000 0.0167 0.00 - 4.00e+00 40.00 20.00 7 | [ 02-28 02:45] 1298889900 0.0167 0.0000 0.0167 0.00 - 5.00e+00 50.00 16.67 8 | [ 02-28 03:01] 1298890860 0.0167 0.0000 0.0167 0.00 - 6.00e+00 60.00 14.29 9 | [ 02-28 03:20] 1298892000 0.0167 0.0000 0.0167 0.00 - 7.00e+00 70.00 12.50 10 | [ 02-28 03:38] 1298893080 0.0167 0.0000 0.0167 0.00 - 8.00e+00 80.00 11.11 11 | [ 02-28 03:58] 1298894280 0.0167 0.0000 0.0167 0.00 - 9.00e+00 90.00 10.00 12 | [ 02-28 04:17] 1298895420 0.0167 0.0000 0.0167 0.00 - 1.00e+01 100.00 9.09 13 | [ 02-28 04:35] 1298896500 0.0167 0.0000 0.0167 0.00 - 1.00e+01 100.00 8.33 14 | [ 02-28 04:54] 1298897640 0.0167 0.0000 0.0167 0.00 - 1.00e+01 100.00 7.69 15 | [ 02-28 05:11] 1298898660 0.0167 0.0000 0.0167 0.00 - 1.00e+01 100.00 7.14 16 | [ 02-28 05:30] 1298899800 0.0167 0.0000 0.0167 0.00 - 1.00e+01 100.00 6.67 17 | [ 02-28 05:47] 1298900820 0.0167 0.0000 0.0167 0.00 - 1.00e+01 100.00 6.25 18 | [ 02-28 06:05] 1298901900 0.0167 0.0000 0.0167 0.00 - 1.00e+01 100.00 5.88 19 | [ 02-28 06:21] 1298902860 0.0167 0.0000 0.0167 0.00 - 1.00e+01 100.00 5.56 20 | [ 02-28 06:37] 1298903820 0.0167 0.0000 0.0167 0.00 - 1.00e+01 100.00 5.26 21 | [ 02-28 06:54] 1298904840 0.0167 0.0000 0.0167 0.00 - 1.00e+01 100.00 5.00 22 | [ 02-28 07:11] 1298905860 0.0167 0.0000 0.0167 0.00 - 1.00e+01 100.00 4.76 23 | [ 02-28 07:31] 1298907060 0.0167 0.0000 0.0167 0.00 - 1.00e+01 100.00 4.55 24 | [ 02-28 07:48] 1298908080 0.0167 0.0000 0.0167 0.00 - 1.00e+01 100.00 4.35 25 | [ 02-28 08:06] 1298909160 0.0167 0.0000 0.0167 0.00 - 1.00e+01 100.00 4.17 26 | [ 02-28 08:24] 1298910240 0.0167 0.0000 0.0167 0.00 - 1.00e+01 100.00 4.00 27 | [ 02-28 08:42] 1298911320 0.0167 0.0000 0.0167 0.00 - 1.00e+01 100.00 3.85 28 | [ 02-28 09:00] 1298912400 0.0167 0.0000 0.0167 0.00 - 1.00e+01 100.00 3.70 29 | [ 02-28 09:18] 1298913480 0.0167 0.0000 0.0167 0.00 - 1.00e+01 100.00 3.57 30 | [ 02-28 09:37] 1298914620 0.0167 0.0000 0.0167 0.00 - 1.00e+01 100.00 3.45 31 | [ 02-28 09:53] 1298915580 0.0167 0.0000 0.0167 0.00 - 1.00e+01 100.00 3.33 32 | [ 02-28 10:10] 1298916600 0.0167 0.0000 0.0167 0.00 - 1.00e+01 100.00 3.23 33 | [ 02-28 10:26] 1298917560 0.0167 0.0000 0.0167 0.00 - 1.00e+01 100.00 3.12 34 | [ 02-28 10:45] 1298918700 0.0167 0.0000 0.0167 0.00 - 1.00e+01 100.00 3.03 35 | [ 02-28 11:03] 1298919780 0.0167 0.0000 0.0167 0.00 - 1.00e+01 100.00 2.94 36 | [ 02-28 11:22] 1298920920 0.0167 0.0000 0.0167 0.00 - 1.00e+01 100.00 2.86 37 | [ 02-28 11:41] 1298922060 0.0167 0.0000 0.0167 0.00 - 1.00e+01 100.00 2.78 38 | [ 02-28 12:00] 1298923200 0.0167 0.0000 0.0167 0.00 - 1.00e+01 100.00 2.70 39 | [ 02-28 12:19] 1298924340 0.0167 0.0000 0.0167 0.00 - 1.00e+01 100.00 2.63 40 | [ 02-28 12:35] 1298925300 0.0167 0.0000 0.0167 0.00 - 1.00e+01 100.00 2.56 41 | [ 02-28 12:52] 1298926320 0.0167 0.0000 0.0167 0.00 - 1.00e+01 100.00 2.50 42 | [ 02-28 13:09] 1298927340 0.0167 0.0000 0.0167 0.00 - 1.00e+01 100.00 2.44 43 | [ 02-28 13:26] 1298928360 0.0167 0.0000 0.0167 0.00 - 1.00e+01 100.00 2.38 44 | [ 02-28 13:43] 1298929380 0.0167 0.0000 0.0167 0.00 - 1.00e+01 100.00 2.33 45 | [ 02-28 13:59] 1298930340 0.0167 0.0000 0.0167 0.00 - 1.00e+01 100.00 2.27 46 | [ 02-28 14:14] 1298931240 0.0167 0.0000 0.0167 0.00 - 1.00e+01 100.00 2.22 47 | [ 02-28 14:31] 1298932260 0.0167 0.0000 0.0167 0.00 - 1.00e+01 100.00 2.17 48 | [ 02-28 14:51] 1298933460 0.0167 0.0000 0.0167 0.00 - 1.00e+01 100.00 2.13 49 | [ 02-28 15:11] 1298934660 0.0167 0.0000 0.0167 0.00 - 1.00e+01 100.00 2.08 50 | [ 02-28 15:31] 1298935860 0.0167 0.0000 0.0167 0.00 - 1.00e+01 100.00 2.04 51 | [ 02-28 15:49] 1298936940 0.0167 0.0000 0.0167 0.00 - 1.00e+01 100.00 2.00 52 | [ 02-28 16:07] 1298938020 0.0167 0.0000 0.0167 0.00 - 1.00e+01 100.00 1.96 53 | [ 02-28 16:25] 1298939100 0.0167 0.0000 0.0167 0.00 - 1.00e+01 100.00 1.92 54 | [ 02-28 16:44] 1298940240 0.0167 0.0000 0.0167 0.00 - 1.00e+01 100.00 1.89 55 | [ 02-28 17:04] 1298941440 0.0167 0.0000 0.0167 0.00 - 1.00e+01 100.00 1.85 56 | [ 02-28 17:22] 1298942520 0.0167 0.0000 0.0167 0.00 - 1.00e+01 100.00 1.82 57 | [ 02-28 17:42] 1298943720 0.0167 0.0000 0.0167 0.00 - 1.00e+01 100.00 1.79 58 | [ 02-28 18:01] 1298944860 0.0167 0.0000 0.0167 0.00 - 1.00e+01 100.00 1.75 59 | [ 02-28 18:20] 1298946000 0.0167 0.0000 0.0167 0.00 - 1.00e+01 100.00 1.72 60 | [ 02-28 18:41] 1298947260 0.0167 0.0000 0.0167 0.00 - 1.00e+01 100.00 1.69 61 | [ 02-28 19:01] 1298948460 0.0167 0.0000 0.0167 0.00 - 1.00e+01 100.00 1.67 62 | [ 02-28 19:19] 1298949540 0.0167 0.0000 0.0167 0.00 - 1.00e+01 100.00 1.64 63 | [ 02-28 19:41] 1298950860 0.0167 0.0000 0.0167 0.00 - 1.00e+01 100.00 1.61 64 | [ 02-28 20:03] 1298952180 0.0167 0.0000 0.0167 0.00 - 1.00e+01 100.00 1.59 65 | [ 02-28 20:21] 1298953260 0.0167 0.0000 0.0167 0.00 - 1.00e+01 100.00 1.56 66 | [ 02-28 20:39] 1298954340 0.0167 0.0000 0.0167 0.00 - 1.00e+01 100.00 1.54 67 | [ 02-28 20:58] 1298955480 0.0167 0.0000 0.0167 0.00 - 1.00e+01 100.00 1.52 68 | [ 02-28 21:20] 1298956800 0.0167 0.0000 0.0167 0.00 - 1.00e+01 100.00 1.49 69 | [ 02-28 21:42] 1298958120 0.0167 0.0000 0.0167 0.00 - 1.00e+01 100.00 1.47 70 | [ 02-28 22:03] 1298959380 0.0167 0.0000 0.0167 0.00 - 1.00e+01 100.00 1.45 71 | [ 02-28 22:25] 1298960700 0.0167 0.0000 0.0167 0.00 - 1.00e+01 100.00 1.43 72 | [ 02-28 22:48] 1298962080 0.0167 0.0000 0.0167 0.00 - 1.00e+01 100.00 1.41 73 | [ 02-28 23:12] 1298963520 0.0167 0.0000 0.0167 0.00 - 1.00e+01 100.00 1.39 74 | [ 02-28 23:33] 1298964780 0.0167 0.0000 0.0167 0.00 - 1.00e+01 100.00 1.37 75 | [ 02-28 23:54] 1298966040 0.0167 0.0000 0.0167 0.00 - 1.00e+01 100.00 1.35 76 | [ 03-01 00:12] 1298967120 0.0167 0.0000 0.0167 0.00 - 1.00e+01 100.00 1.33 77 | [ 03-01 00:26] 1298967960 0.0167 0.0000 0.0167 0.00 - 1.00e+01 100.00 1.32 78 | -------------------------------------------------------------------------------- /spec/fixtures/test.log: -------------------------------------------------------------------------------- 1 | [Feb 28 01:34:31] post_44164631_js 2 | [Feb 28 01:34:59] post_44164631_js 3 | [Feb 28 01:52:36] post_11080788_html 4 | [Feb 28 02:09:28] post_44427139_html 5 | [Feb 28 02:26:08] post_19142226_js 6 | [Feb 28 02:45:01] post_29425455_js 7 | [Feb 28 03:01:27] post_20254178_js 8 | [Feb 28 03:20:21] post_44359317_js 9 | [Feb 28 03:38:38] post_30461313_html 10 | [Feb 28 03:58:07] post_11132355_html 11 | [Feb 28 04:17:19] post_42756810_html 12 | [Feb 28 04:35:16] post_44097667_html 13 | [Feb 28 04:54:18] post_5841079_html 14 | [Feb 28 05:11:41] post_34436257_js 15 | [Feb 28 05:30:37] post_44413528_html 16 | [Feb 28 05:47:35] post_590946_js 17 | [Feb 28 06:05:41] post_43040891_js 18 | [Feb 28 06:21:57] post_35951609_html 19 | [Feb 28 06:37:22] post_44371357_js 20 | [Feb 28 06:54:12] post_44370352_js 21 | [Feb 28 07:11:59] post_43931894_js 22 | [Feb 28 07:31:26] post_44451227_html 23 | [Feb 28 07:48:50] post_43447312_js 24 | [Feb 28 08:06:17] post_41772081_js 25 | [Feb 28 08:24:41] post_39465275_html 26 | [Feb 28 08:42:23] post_21393090_html 27 | [Feb 28 09:00:03] post_22422432_html 28 | [Feb 28 09:18:36] post_38864066_html 29 | [Feb 28 09:37:09] post_44464850_body 30 | [Feb 28 09:53:24] post_44416933_js 31 | [Feb 28 10:10:15] post_33857865_html 32 | [Feb 28 10:26:36] post_43106709_js 33 | [Feb 28 10:45:03] post_44011761_js 34 | [Feb 28 11:03:38] post_7514064_js 35 | [Feb 28 11:22:09] post_44441467_html 36 | [Feb 28 11:41:43] post_26268595_js 37 | [Feb 28 12:00:38] post_34540193_js 38 | [Feb 28 12:19:05] post_43983905_js 39 | [Feb 28 12:35:49] post_25949337_html 40 | [Feb 28 12:52:41] post_37018444_js 41 | [Feb 28 13:09:23] post_44311078_html 42 | [Feb 28 13:26:43] post_44281932_html 43 | [Feb 28 13:43:47] post_43797442_js 44 | [Feb 28 13:59:43] post_21531863_html 45 | [Feb 28 14:14:36] post_35045483_html 46 | [Feb 28 14:31:32] post_18763179_html 47 | [Feb 28 14:51:22] post_18291504_html 48 | [Feb 28 15:11:02] post_25396589_html 49 | [Feb 28 15:31:16] post_22020349_html 50 | [Feb 28 15:49:40] post_44483029_js 51 | [Feb 28 16:07:22] post_27390667_js 52 | [Feb 28 16:25:49] post_44402219_js 53 | [Feb 28 16:44:35] post_44397413_html 54 | [Feb 28 17:04:42] post_18140672_js 55 | [Feb 28 17:22:27] post_12613685_js 56 | [Feb 28 17:42:13] post_31699730_html 57 | [Feb 28 18:01:49] post_44017330_html 58 | [Feb 28 18:20:11] post_44257513_js 59 | [Feb 28 18:41:13] post_42009796_js 60 | [Feb 28 19:01:12] post_11683538_html 61 | [Feb 28 19:19:56] post_1115141_html 62 | [Feb 28 19:41:32] post_21455392_js 63 | [Feb 28 20:03:21] post_44362212_html 64 | [Feb 28 20:21:40] post_21437632_js 65 | [Feb 28 20:39:57] post_44489038_js 66 | [Feb 28 20:58:37] post_41747796_html 67 | [Feb 28 21:20:01] post_44087638_js 68 | [Feb 28 21:42:35] post_2240726_js 69 | [Feb 28 22:03:56] post_40861206_html 70 | [Feb 28 22:25:11] post_4163737_html 71 | [Feb 28 22:48:19] post_16676849_html 72 | [Feb 28 23:12:46] post_44097667_js 73 | [Feb 28 23:33:52] post_9672620_js 74 | [Feb 28 23:54:12] post_44510680_nojs 75 | [Mar 01 00:12:24] post_3604185_html 76 | [Mar 01 00:26:47] post_44526454_js 77 | -------------------------------------------------------------------------------- /spec/log_replayer_spec.rb: -------------------------------------------------------------------------------- 1 | require 'spec_helper' 2 | 3 | describe SimCache::LogReplayer do 4 | 5 | before(:all) do 6 | @log_replayer = SimCache::LogReplayer.new( 7 | :log_file => test_log, 8 | :report_file => report_log, 9 | :cache_options => {:max_keys => 10} 10 | ) 11 | end 12 | 13 | before(:each) { @log_replayer.mock_cache.redis.flushdb } 14 | 15 | it "should relay the log" do 16 | @log_replayer.replay! 17 | end 18 | end -------------------------------------------------------------------------------- /spec/mock_cache_spec.rb: -------------------------------------------------------------------------------- 1 | require 'spec_helper' 2 | 3 | describe SimCache::MockCache do 4 | 5 | before(:all) { @mock_cache = SimCache::MockCache.new(:max_keys => 5) } 6 | before(:each) do 7 | @mock_cache.redis.flushdb 8 | @mock_cache.send(:init_counters!) 9 | end 10 | 11 | it "should have a method for getting" do 12 | @mock_cache.should respond_to(:get) 13 | end 14 | 15 | it "should instantiate a redis connection" do 16 | @mock_cache.redis.class.should == Redis 17 | end 18 | 19 | it "should raise an error if a key isn't found" do 20 | lambda { @mock_cache.get("unknown_key") }.should raise_error(SimCache::MockCache::KeyNotFound) 21 | end 22 | 23 | it "should act as an LRU-cache (i.e., more recent items have a higher rank)" do 24 | %w(key1 key2).each {|k| @mock_cache.get(k) rescue nil} 25 | %w(key1 key2).each_with_index {|k, i| @mock_cache.rank_for_key(k).should == 1-i} 26 | end 27 | 28 | it "should evict keys if the limit of the cache is exceeded" do 29 | (1..5).each {|i| @mock_cache.get("key_#{i}") rescue nil} 30 | @mock_cache.get("key_1") 31 | @mock_cache.get("key_6") rescue nil 32 | 33 | @mock_cache.rank_for_key("key_6").should == 0 34 | @mock_cache.rank_for_key("key_1").should == 1 35 | @mock_cache.rank_for_key("key_2").should be_nil 36 | @mock_cache.num_keys.should == 5 37 | end 38 | 39 | it "should tell you how many keys are in the cache" do 40 | (1..5).each do |i| 41 | @mock_cache.get("key_#{i}") rescue nil 42 | @mock_cache.num_keys.should == i 43 | end 44 | end 45 | 46 | it "should have basic methods for cache statistics" do 47 | @mock_cache.send(:init_counters!) 48 | 4.times { @mock_cache.get("rand_key") rescue nil } 49 | 50 | @mock_cache.misses.should == 1 51 | @mock_cache.hits.should == 3 52 | @mock_cache.hit_rate.should == 0.75 53 | @mock_cache.miss_rate.should == 0.25 54 | @mock_cache.num_keys.should == 1 55 | @mock_cache.percent_utilization.should == 0.2 56 | end 57 | end -------------------------------------------------------------------------------- /spec/spec_helper.rb: -------------------------------------------------------------------------------- 1 | $LOAD_PATH.unshift(File.dirname(__FILE__)) 2 | $LOAD_PATH.unshift(File.join(File.dirname(__FILE__), '..', 'lib')) 3 | require 'sim_cache' 4 | 5 | def test_log 6 | File.join(File.dirname(__FILE__), "fixtures/test.log") 7 | end 8 | 9 | def report_log 10 | File.join(File.dirname(__FILE__), "fixtures/sample_out.log") 11 | end 12 | 13 | # awk '{print $1, $2, $3, "p_"$16"_"$10}' test_log | awk -F: '{print $2":"$3":"$4}' | awk '{print "["$1, $2, $3"]", $4}' --------------------------------------------------------------------------------