trivial benchmark
3.0.0An easy to use benchmarking system.
About Trivial-Benchmark
Frequently I want to do a quick benchmark comparison of my functions. time is nice to get some data, but it's limited to a single run so there isn't really much of a statistical value in it. Trivial-Benchmark runs a block of code many times and outputs some statistical data for it. On SBCL this includes the data from time, for all other implementations just the REAL- and RUN-TIME data.
How-To
We assume that there's a local or global nickname for org.shirakumo.trivial-benchmark called tb. You can activate the global nickname with (org.shirakumo.trivial-benchmark:add-nickname).
For basic throwaway benchmarking, the with-timing macro should suffice:
(tb:with-timing (1000)
2 | (+ 1 1))
However, you can also do more complex timing using your own timer and with-sampling. The former creates a new timer object (with an optional list of metrics to sample) and the latter collects one sample for each metric of the timer for the duration of the body forms.
(defvar *timer* (make-instance 'tb:timer))
3 |
4 | (tb:with-sampling (*timer*)
5 | (+ 1 1))
6 |
7 | (tb:with-sampling (*timer*)
8 | (expt 10 100))
9 |
10 | (tb:report *timer*)
11 |
12 | (tb:reset *timer*)
13 |
14 | (tb:report *timer*)
System Information
Definition Index
-
ORG.SHIRAKUMO.TRIVIAL-BENCHMARK
- TB
- TRIVIAL-BENCHMARK
No documentation provided.-
EXTERNAL SPECIAL-VARIABLE *DEFAULT-COMPUTATIONS*
Which computations to make by default. 15 | 16 | See COMPUTE
-
EXTERNAL SPECIAL-VARIABLE *DEFAULT-METRICS*
Which metrics to include by default. 17 | 18 | If NIL, all sampled metrics are included. 19 | 20 | See FORMAT-TIMER-STATS
-
EXTERNAL SPECIAL-VARIABLE *DEFAULT-SAMPLERS*
Which sampler types to use by default. 21 | 22 | See SAMPLER 23 | See WITH-SAMPLING
-
EXTERNAL CLASS BYTES-CONSED
No documentation provided. -
EXTERNAL CLASS CYCLE-COUNTER
No documentation provided. -
EXTERNAL CLASS EVAL-CALLS
No documentation provided. -
EXTERNAL CLASS GC-RUN-TIME
No documentation provided. -
EXTERNAL CLASS REAL-TIME
No documentation provided. -
EXTERNAL CLASS RUN-TIME
No documentation provided. -
EXTERNAL CLASS SAMPLER
Representation of a sample measuring method. 24 | 25 | See VARIABLES 26 | See WRAP-MEASUREMENT-FORM 27 | See COMMIT-SAMPLES-FORM 28 | See DEFINE-SAMPLER 29 | See WITH-SAMPLING
-
EXTERNAL CLASS SB-TIME
No documentation provided. -
EXTERNAL CLASS SYSTEM-INFO
No documentation provided. -
EXTERNAL CLASS TIMER
Encompasses a set of samples. 30 | 31 | A timer can be re-used to sample in individual steps. 32 | 33 | See FORMAT-TIMER-STATS 34 | See SAMPLES 35 | See METRIC-TYPES 36 | See REPORT-TO 37 | See RESET 38 | See WITH-SAMPLING 39 | See WITH-TIMING
-
EXTERNAL FUNCTION ADD-NICKNAME
Adds the nikcnames TRIVIAL-BENCHMARK and TB to the package. 40 | 41 | This is for convenience.
-
EXTERNAL FUNCTION FORMAT-TIMER-STATS
- STREAM
- TIMER
- &REST
- ARGS
- &KEY
- COMPUTATIONS
- METRICS
- &ALLOW-OTHER-KEYS
Print a table of the samples contained in the timer. 42 | 43 | STREAM should be a stream to print to. 44 | TIMER should be a TIMER instance. 45 | COMPUTATIONS should be the derived statistical properties to compute 46 | METRICS should be the metrics to include. If NIL, uses all metrics the 47 | timer recorded samples for. 48 | Other arguments are passed on to PRINT-TABLE 49 | 50 | See TIMER (type) 51 | See PRINT-TABLE 52 | See *DEFAULT-COMPUTATIONS* 53 | See *DEFAULT-METRICS*
-
EXTERNAL FUNCTION PRINT-TABLE
- TABLE
- &KEY
- STREAM
- PADDING
- FORMAT
Print a table of values 54 | 55 | TABLE should be a list of lists in row-major order. Each row must have 56 | the same number of elements. 57 | 58 | STREAM is the stream to print the table to. 59 | 60 | PADDING is the amount of space to insert to the left and right of 61 | every cell. 62 | 63 | FORMAT is the format to print the table in. Can be one of: 64 | :PRINC --- justs PRINCs the table instead 65 | :MINIMAL --- prints the table as small as it can 66 | :FANCY --- prints the table with box drawing glyphs
-
EXTERNAL FUNCTION REPORT
- THING
- &REST
- ARGS
- &KEY
- STREAM
- &ALLOW-OTHER-KEYS
Prints a report on the given thing. 67 | 68 | This is a wrapper around REPORT-TO. 69 | 70 | See REPORT-TO
-
EXTERNAL GENERIC-FUNCTION COMMIT-SAMPLES-FORM
- SAMPLER
- COMMIT-FN
Emit a form to commit measured samples 71 | 72 | Returns a new form. 73 | The returned form is emitted into a lexical environment where 74 | COMMIT-FN is bound to a macro with the following behaviour: 75 | 76 | (commit-fn metric sample ...) 77 | 78 | Each pair of METRIC and SAMPLE are recorded in the timer 79 | METRIC should be a symbol naming the metric to store the sample 80 | under, and SAMPLE should be a REAL number to record. The sample is 81 | coerced to a DOUBLE-FLOAT automatically. 82 | 83 | See WITH-SAMPLING 84 | See SAMPLER (type)
-
EXTERNAL GENERIC-FUNCTION COMPUTE
- THING
- SAMPLES
Compute a derived statistic from a vector of samples. 85 | 86 | You may add additional methods to this, though the following 87 | computation types are provided by default: 88 | 89 | :COUNT 90 | :SAMPLES --- The number of samples 91 | :TOTAL --- The sum of all samples 92 | :MINIMUM --- The smallest sample 93 | :MAXIMUM --- The biggest sample 94 | :MEDIAN --- The median of all samples 95 | :AVERAGE --- The average of all samples 96 | :DEVIATION --- The standard deviation based on the average 97 | 98 | See *DEFAULT-COMPUTATIONS*
-
EXTERNAL GENERIC-FUNCTION METRIC-TYPES
- TIMER
Returns a list of metric types included in the timer. 99 | 100 | See TIMER (type)
-
EXTERNAL GENERIC-FUNCTION REPORT-TO
- STREAM
- THING
- &KEY
- COMPUTATIONS
- &ALLOW-OTHER-KEYS
Print a report about THING to STREAM. 101 | 102 | STREAM may be a stream, T, or NIL, according to FORMAT's output 103 | designators. 104 | 105 | COMPUTATIONS may be a list of statistical properties to report on a 106 | vector of samples. 107 | 108 | Additional arguments are passed on to PRINT-TABLE 109 | 110 | See TIMER (type) 111 | See FORMAT-TIMER-STATS 112 | See PRINT-TABLE
-
EXTERNAL GENERIC-FUNCTION RESET
- TIMER
Resets the timer and clears all samples. 113 | 114 | See TIMER (type)
-
EXTERNAL GENERIC-FUNCTION SAMPLES
- TIMER
- METRIC
Accesses a sample vector for the given metric. 115 | 116 | The returned vector is adjustable and has a fill-pointer. 117 | If no vector for the given metric existed before, it is created for 118 | you. 119 | 120 | See TIMER (type)
-
EXTERNAL GENERIC-FUNCTION VARIABLES
- SAMPLER
Return a list of variable specs the sampler needs. 121 | 122 | Each variable spec should be a list of the following elements: 123 | SYMBOL --- The symbol the variable is bound to 124 | DEFAULT --- The default value the variable is initialised to 125 | TYPE --- The type to declare the variable as 126 | 127 | See WITH-SAMPLING 128 | See SAMPLER (type)
-
EXTERNAL GENERIC-FUNCTION WRAP-MEASUREMENT-FORM
- SAMPLER
- FORM
Wrap FORM in a form to record sampling data. 129 | 130 | Returns the new form. 131 | The returned form is emitted into a lexical environment where 132 | variables according to the SAMPLER's VARIBALES specs are bound. 133 | 134 | See WITH-SAMPLING 135 | See SAMPLER (type)
-
EXTERNAL MACRO DEFINE-DELTA-SAMPLER
- NAME
- &BODY
- SAMPLE-POINT-FORMS
Convenience macro to define a sampler type with a single delta metric. 136 | 137 | NAME can either be the name of the sampler, or a list of the name and 138 | a normalisation factor by which the delta is divided to convert it to 139 | a proper sample point. 140 | 141 | SAMPLE-POINT-FORMS should be one or more forms whose ultimate return 142 | value evaluates to some REAL number. The forms are emitted *twice*, 143 | the first time before the execution to be measured, and the second 144 | time after. The sample is computed based on the difference of the two 145 | return values. 146 | 147 | See DEFINE-SAMPLER 148 | See SAMPLER (type)
-
EXTERNAL MACRO DEFINE-SAMPLER
- NAME
- VARS
- &BODY
- FORMS
Convenience macro to define a new sampler type. 149 | 150 | NAME should be the name of the new sampler class. 151 | VARS should be a list of variable specs that the sampler will use to 152 | store data during a sampling step. Each spec can be either a symbol 153 | naming the variable, or a variable spec list. 154 | FORMS should match the following structures: 155 | 156 | (:MEASURE (FORM-VAR) BODY...) 157 | (:COMMIT (COMMIT-FN-VAR) BODY...) 158 | 159 | Which are converted to methods for WRAP-MEASUREMENT-FORM and 160 | COMMIT-SAMPLES-FORM respectively. Every variable in VARS is bound 161 | during the body of either, so that you can conveniently emit it into 162 | the resulting forms. 163 | 164 | See VARIABLES 165 | See WRAP-MEASUREMENT-FORM 166 | See COMMIT-SAMPLES-FORM 167 | See SAMPLER (type) 168 | See DEFINE-DELTA-SAMPLER
-
EXTERNAL MACRO WITH-SAMPLING
- TIMER-FORM
- &REST
- SAMPLERS
- &BODY
- FORMS
Records sampling information about FORMS into a timer. 169 | 170 | TIMER-FORM should evaluate to a TIMER instance, into which the 171 | sampling data will be committed. 172 | 173 | SAMPLERS should be a list of names of SAMPLER types that will measure 174 | the FORMS. If none are given, *DEFAULT-SAMPLERS* are used. 175 | 176 | The values of FORMS are returned. 177 | 178 | During expansion of this macro each sampler in SAMPLERS is 179 | instantiated and variable bindings according to their VARIABLES specs 180 | are emitted. Then, the FORMS are wrapped according to each sampler's 181 | WRAP-MEASUREMENT-FORM return value. The order here can matter, and the 182 | first sampler specified will be the "innermost" wrapper. Finally, 183 | each sampler's COMMIT-SAMPLES-FORM is emitted in order to record the 184 | sampling data into the timer. 185 | 186 | See *DEFAULT-SAMPLERS* 187 | See SAMPLER (type) 188 | See TIMER (type) 189 | See VARIABLES 190 | See WRAP-MEASUREMENT-FORM 191 | See COMMIT-SAMPLES-FORM
-
EXTERNAL MACRO WITH-TIMING
- N
- &REST
- REPORT-ARGS
- &KEY
- TIMER
- SAMPLERS
- &BODY
- FORMS
Convenience macro to do a one-off timing. 192 | 193 | Creates a timer instance according to TIMER, runs WITH-SAMPLING N 194 | times, and then REPORTs the timer using the additional REPORT-ARGS. 195 | 196 | See TIMER (type) 197 | See WITH-SAMPLING 198 | See REPORT