├── Makefile ├── README.md ├── analyze.R ├── autoexp.py ├── autoexp_consumer.py ├── autoexp_download_output.py ├── autoexp_pool.py ├── autoexp_producer.py ├── config.py ├── histsummary-slides.emf ├── histsummary-slides.pdf ├── histsummary.pdf ├── paper.pdf ├── paper.tex ├── slides.pptx └── stats.Rnw /Makefile: -------------------------------------------------------------------------------- 1 | all: 2 | R CMD Sweave stats.Rnw 3 | pdflatex paper 4 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | Overview 2 | ======== 3 | 4 | This project is a set of skeleton scripts for running experiments, 5 | analyzing them, and reporting the results in an academic paper. 6 | Specifically, the scripts: 7 | 8 | 1. Automatically runs experiments using a script and uploads the 9 | results to a google doc and/or rabbitmq queue. 10 | 11 | 2. Downloads the results into [R][R] 12 | 13 | 3. Generates statistics and figures 14 | 15 | 4. Incorporates the statistics and figures into [LaTeX][LaTeX] with 16 | [Sweave][Sweave] 17 | 18 | This process has a number of benefits for academic paper writing: 19 | 20 | * You can watch your experiments progress by opening google docs and 21 | watching the new data come in. This can often give you an early 22 | indication of the effect, if any, your most recent change will have. 23 | 24 | * You can run experiments up to the deadline. Got last minute 25 | improved numbers? Simply run `make` and regenerate the paper. Did 26 | your last run hurt your numbers? Simply revert your google doc to 27 | the last good version. 28 | 29 | * Years later, you will always be able to find your experimental data. 30 | Need to make a new figure? No problem. Need to share your data? 31 | Just share the google doc, and optionally share your R script for 32 | analyzing it. 33 | 34 | Quick Start 35 | =========== 36 | 37 | * Add google docs spreadsheet key and login information to 38 | `config.py`. 39 | 40 | * Run `python autoexp_pool.py` 41 | 42 | * Edit `analyze.R` to use your own google docs spreadsheet (you can 43 | get this url from `File` -> `Publish to the web`). 44 | 45 | * Run `make` 46 | 47 | Details 48 | ======= 49 | 50 | The scripts in this project run a fictitious experiment that measures 51 | how long it takes to type my name (Ed) and the name of my co-author 52 | (Thanassis). Thanassis helped me write some of these scripts. 53 | 54 | The main experiment script is `autoexp.py`. It is written in python, 55 | and is primarily set up to time external commands. The current 56 | version "measures" the time it takes to type a name by generating a 57 | random number. However, it should be easily adapted to real 58 | experiments. Before it can be used, you must put your google account 59 | information and the google docs spreadsheet key in the `config.py` 60 | file. You can find a spreadsheet's key by looking at its url. You can 61 | run the experiment using multiple cores on a single machine by running 62 | `autoexp_pool.py`. Alternatively, you can run the experiment on 63 | multiple machines with the help of a [RabbitMQ][RabbitMQ] server by 64 | running `autoexp_producer.py` on any machine, and then running 65 | `autoexp_consumer.py` on each worker machine. In either case, the 66 | scripts should add a new table called `paper` to the google docs 67 | spreadsheet. It should look like [this][example-spreadsheet]. If you 68 | look at the spreadsheet while the script is running, you should be 69 | able to see each row being added to the spreadsheet. This is more 70 | useful for experiments that take hours or days to run. Alternatively, 71 | for experiments that produce outputs too large for a spreadsheet, you 72 | can save the results in a [RabbitMQ][RabbitMQ] queue by setting 73 | `use_google=False` and `output_rabbitmq=True` in `config.py`. You can 74 | then download the results into a csv file by running 75 | `autoexp_download_output.py`. 76 | 77 | `analyze.R` is an R script that analyzes the uploaded data. It reads 78 | the experiment data directly from google docs, counts the number of 79 | samples, computes the mean time to "type" both Ed and Thanassis, and 80 | then produces two figures. You will probably want to edit this script 81 | so that it uses your own google docs csv file instead of mine. I 82 | often start analyzing experimental data by opening R and running 83 | `source("analyze.R")` to download the experimental data. There are 84 | many tools inside of R for exploratory data analysis, but I personally 85 | prefer visualization using the [ggplot2][ggplot2] package. 86 | 87 | Finally, the results from `analyze.R` can also be incorporated into a 88 | LaTeX paper. This is done using the [Sweave][Sweave] file 89 | `Stats.Rnw`, which creates commands for each statistic in R that needs 90 | to be referenced in the paper. An example LaTeX paper is in 91 | `paper.tex`. The final step is to use `Makefile` to process the 92 | [Sweave][Sweave] file and run LaTeX. The final result should produce a 93 | file similar to `paper.pdf`. 94 | 95 | It is also possible to generate nice figures for inclusion in 96 | Powerpoint slides. To produce `histsummary-slides.emf`, which is 97 | suitable for inclusion in Windows Powerpoint, convert 98 | `histsummary-slides.svg` to EMF format using Inkscape on Windows. An 99 | example can be seen in `slides.pptx`. Note that Mac Powerpoint cannot 100 | view EMF files properly. For Mac Powerpoint, you can simply use 101 | histsummary-slides.pdf, which is generated by `analyze.R`. 102 | 103 | [example-spreadsheet]: https://docs.google.com/spreadsheet/ccc?key=0Au4zXzOoce8JdGFjZ0JBVTIxRmgzeEpZN0VFRVktb0E&usp=sharing 104 | [ggplot2]: http://ggplot2.org/ 105 | [LaTeX]: http://www.latex-project.org/ 106 | [R]: http://www.r-project.org 107 | [Sweave]: http://www.stat.uni-muenchen.de/~leisch/Sweave/ 108 | [RabbitMQ]: http://www.rabbitmq.com/ 109 | -------------------------------------------------------------------------------- /analyze.R: -------------------------------------------------------------------------------- 1 | #!/usr/bin/Rscript 2 | # Example R script to analyze fake typing time data. The mean times 3 | # for both words are computed. Two figures are also produced. 4 | 5 | library(ggplot2) 6 | library(reshape) 7 | library(plyr) 8 | library(grid) 9 | require(RCurl) 10 | 11 | theme_set(theme_bw()) 12 | theme_slides <- theme_bw(base_size = 28) + 13 | theme(axis.title.x = element_text(vjust = -1), 14 | axis.title.y = element_text(vjust = 0.35), 15 | plot.margin = unit(c(.5, .5, 1, .5), "cm"), 16 | legend.text = element_text(size = 18)) 17 | 18 | ## From http://www.cookbook-r.com/Manipulating_data/Summarizing_data/#using-summaryby 19 | ## Summarizes data. 20 | ## Gives count, mean, standard deviation, standard error of the mean, and confidence interval (default 95%). 21 | ## data: a data frame. 22 | ## measurevar: the name of a column that contains the variable to be summariezed 23 | ## groupvars: a vector containing names of columns that contain grouping variables 24 | ## na.rm: a boolean that indicates whether to ignore NA's 25 | ## conf.interval: the percent range of the confidence interval (default is 95%) 26 | summarySE <- function(data=NULL, measurevar, groupvars=NULL, na.rm=FALSE, 27 | conf.interval=.95, .drop=TRUE) { 28 | require(plyr) 29 | 30 | # New version of length which can handle NA's: if na.rm==T, don't count them 31 | length2 <- function (x, na.rm=FALSE) { 32 | if (na.rm) sum(!is.na(x)) 33 | else length(x) 34 | } 35 | 36 | # This does the summary. For each group's data frame, return a vector with 37 | # N, mean, and sd 38 | datac <- ddply(data, groupvars, .drop=.drop, 39 | .fun = function(xx, col) { 40 | c(N = length2(xx[[col]], na.rm=na.rm), 41 | mean = mean (xx[[col]], na.rm=na.rm), 42 | sd = sd (xx[[col]], na.rm=na.rm) 43 | ) 44 | }, 45 | measurevar 46 | ) 47 | 48 | # Rename the "mean" column 49 | datac <- rename(datac, c("mean" = measurevar)) 50 | 51 | datac$se <- datac$sd / sqrt(datac$N) # Calculate standard error of the mean 52 | 53 | # Confidence interval multiplier for standard error 54 | # Calculate t-statistic for confidence interval: 55 | # e.g., if conf.interval is .95, use .975 (above/below), and use df=N-1 56 | ciMult <- qt(conf.interval/2 + .5, datac$N-1) 57 | datac$ci <- datac$se * ciMult 58 | 59 | return(datac) 60 | } 61 | 62 | ## Read in data 63 | 64 | if (!exists("d")) { 65 | 66 | print("Downloading..."); 67 | 68 | d <- read.csv(textConnection(getURL("https://docs.google.com/spreadsheet/pub?key=0Au4zXzOoce8JdGFjZ0JBVTIxRmgzeEpZN0VFRVktb0E&single=true&gid=3&output=csv")), na.strings = c("-1")) 69 | 70 | d$name <- factor(d$name, levels=c("ed", "thanassis")) 71 | 72 | print("Factorizing") 73 | 74 | md = melt(d, measure.vars=c("time")) 75 | 76 | } 77 | 78 | print("Downloaded csv data"); 79 | 80 | dsummary <- summarySE(d, "time", c("name")) 81 | 82 | g = qplot(name, time, data=d, geom="boxplot", xlab="Name", ylab="Time (seconds)") 83 | ggsave(file = "box.pdf", width=6.6, height=2.2, scale=1.5) 84 | 85 | g = qplot(time, data=d, fill = name) 86 | ggsave(file = "hist.pdf", width=6.6, height=2.2, scale=1.5) 87 | 88 | g = qplot(name, time, geom=c("bar"), fill=name, data=dsummary) + geom_errorbar(aes(ymin=time-ci, ymax=time+ci)) 89 | ggsave(file = "histsummary.pdf", width=3.16, height=1.75, scale=1.5) 90 | 91 | g + theme_slides 92 | ggsave(file = "histsummary-slides.pdf", width=11.5, height=7.5, scale=0.9) 93 | ggsave(file = "histsummary-slides.svg", width=11.5, height=7.5, scale=0.9) 94 | 95 | # Stats 96 | numsamples = nrow(d) 97 | 98 | edmean = round(mean(d[d$name == "ed",]$time), digits=2) 99 | thanassismean = round(mean(d[d$name == "thanassis",]$time), digits=2) 100 | -------------------------------------------------------------------------------- /autoexp.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python 2 | # Example experiment script that generates random data representing 3 | # the time it takes to type a particular word. The results are 4 | # uploaded to a google docs spreadsheet. 5 | 6 | # Authors: Ed Schwartz and Thanassis Avgerinos 7 | 8 | from config import * 9 | from subprocess import Popen, PIPE 10 | import csv 11 | import os 12 | import random 13 | import string 14 | import subprocess 15 | import sys 16 | import time 17 | 18 | # Redo entries already in sheet? 19 | redo = False 20 | 21 | trials = 20 22 | names = ["ed", "thanassis"] 23 | inputs = reduce(list.__add__, map(lambda n: map(lambda num: {"name": n, "num": num}, xrange(trials)), names)) 24 | #print inputs 25 | 26 | # Input columns 27 | ids = ["name", "num"] 28 | # Measurement (output) columns 29 | measured = ["time"] 30 | 31 | client = None 32 | 33 | def login(): 34 | 35 | global client 36 | if client is None: 37 | # Change this to the name of the worksheet you want to use 38 | dbname="paper" 39 | 40 | import gdata.spreadsheet.text_db 41 | client = gdata.spreadsheet.text_db.DatabaseClient(username=user, password=password) 42 | 43 | global db 44 | db = client.GetDatabases(spreadsheet_key=key)[0] 45 | global tables 46 | tables = db.GetTables(name=dbname) 47 | global table 48 | if len(tables) == 1: 49 | table = tables[0] 50 | else: 51 | table = db.CreateTable(dbname, ids + measured) 52 | 53 | def setup(): 54 | if use_google: 55 | login() 56 | 57 | def timeit(cmd): 58 | stime = time.time() 59 | p = subprocess.Popen(cmd, shell=True, stdout=PIPE, stderr=PIPE) 60 | stdout, stderr = p.communicate() 61 | duration = time.time() - stime 62 | return_code = p.returncode 63 | 64 | if return_code != 0: 65 | duration = -1 66 | 67 | return duration, stderr 68 | 69 | def quote_if_needed(x): 70 | x = str(x) 71 | if x.isdigit(): 72 | return x 73 | else: 74 | return "\"" + x + "\"" 75 | 76 | def run_experiment(inputs): 77 | 78 | # Check for existing rows 79 | query_strs = map(lambda column: column + " == " + quote_if_needed(inputs[column]), ids) 80 | query_str = string.join(query_strs, " and ") 81 | #print query_str 82 | 83 | if use_google: 84 | login() 85 | records = table.FindRecords(query_str) 86 | #print records 87 | 88 | if redo: 89 | for row in records: 90 | row.Delete() 91 | go = True 92 | else: 93 | go = len(records) == 0 94 | else: 95 | # Always go when not using the spreadsheet db 96 | go = True 97 | 98 | if go: 99 | runtime, out = timeit("sleep " + str(random.normalvariate(len(inputs["name"]), 1.0))) 100 | 101 | measurements = {"time": runtime} 102 | 103 | # Add input columns 104 | m = map(lambda column: (column, str(inputs[column])), ids) 105 | # Add measurement columns 106 | m = m + map(lambda column: (column, str(measurements[column])), measured) 107 | d = dict(m) 108 | 109 | return d 110 | 111 | else: 112 | # Don't make Google too mad. 113 | print "Skipping", inputs 114 | sys.stdout.flush() 115 | time.sleep(1) 116 | return None 117 | 118 | def process_results(d, channel=None): 119 | 120 | if use_google: 121 | login() 122 | 123 | ## Try a couple times to add the data 124 | for i in xrange(10): 125 | try: 126 | print "adding", d 127 | table.AddRecord(d) 128 | break 129 | except: 130 | print "Unexpected error:", sys.exc_info()[0] 131 | time.sleep(i*10) 132 | 133 | if output_rabbitmq: 134 | import json 135 | import pika 136 | 137 | channel.queue_declare(queue='autoexp_output_queue', durable=True) 138 | channel.basic_publish(exchange='', 139 | routing_key='autoexp_output_queue', 140 | body=json.dumps(d), 141 | properties=pika.BasicProperties( 142 | delivery_mode = 2, # make message persistent 143 | )) 144 | 145 | 146 | def run_and_process(i, channel=None): 147 | x = run_experiment(i) 148 | if not x is None: 149 | process_results(x, channel) 150 | -------------------------------------------------------------------------------- /autoexp_consumer.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | import argparse 3 | import autoexp 4 | import json 5 | import pika 6 | import sys 7 | import time 8 | 9 | import logging 10 | logging.basicConfig(level=logging.INFO) 11 | 12 | parser = argparse.ArgumentParser(description='Run experiments from RabbitMQ server.') 13 | parser.add_argument('--server', dest='server', action='store', 14 | default='localhost', 15 | help='address of the RabbitMQ server (default: localhost)') 16 | args = parser.parse_args() 17 | 18 | def all_done(): 19 | logging.info("Queue empty") 20 | connection.close() 21 | sys.exit(0) 22 | 23 | connection = pika.BlockingConnection(pika.ConnectionParameters( 24 | host=args.server)) 25 | channel = connection.channel() 26 | 27 | channel.queue_declare(queue='autoexp_queue', durable=True) 28 | logging.info('Getting messages.') 29 | 30 | def callback(ch, method, properties, body): 31 | try: 32 | logging.debug("Received %r" % (body,)) 33 | input=json.loads(body) 34 | autoexp.run_and_process(input, channel) 35 | logging.debug("Done with %r" % (body,)) 36 | ch.basic_ack(delivery_tag = method.delivery_tag) 37 | except: 38 | print sys.exc_info()[0] 39 | 40 | channel.basic_qos(prefetch_count=1) 41 | while True: 42 | method_frame, header_frame, body = channel.basic_get(queue = 'autoexp_queue') 43 | #print method_frame, header_frame, body 44 | if method_frame is not None and method_frame.NAME == 'Basic.GetOk': 45 | callback(channel, method_frame, header_frame, body) 46 | else: 47 | all_done() 48 | -------------------------------------------------------------------------------- /autoexp_download_output.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | import argparse 3 | import autoexp 4 | import csv 5 | import json 6 | import pika 7 | import sys 8 | 9 | import logging 10 | logging.basicConfig(level=logging.INFO) 11 | 12 | parser = argparse.ArgumentParser(description='Download results from RabbitMQ server.') 13 | parser.add_argument('--server', dest='server', action='store', 14 | default='localhost', 15 | help='address of the RabbitMQ server (default: localhost)') 16 | args = parser.parse_args() 17 | 18 | connection = pika.BlockingConnection(pika.ConnectionParameters( 19 | host=args.server)) 20 | channel = connection.channel() 21 | 22 | channel.queue_declare(queue='autoexp_output_queue', durable=True) 23 | 24 | writer = csv.DictWriter(sys.stdout, fieldnames=autoexp.ids + autoexp.measured) 25 | writer.writeheader() 26 | 27 | def all_done(): 28 | logging.info("Finished") 29 | connection.close() 30 | sys.exit(0) 31 | 32 | def callback(ch, method, properties, body): 33 | try: 34 | logging.debug("Received %r" % (body,)) 35 | output=json.loads(body) 36 | writer.writerow(output) 37 | logging.debug("Done with %r" % (output,)) 38 | ch.basic_ack(delivery_tag = method.delivery_tag) 39 | except: 40 | print sys.exc_info()[0] 41 | 42 | channel.basic_qos(prefetch_count=1) 43 | while True: 44 | method_frame, header_frame, body = channel.basic_get(queue = 'autoexp_output_queue') 45 | if method_frame is not None and method_frame.NAME == 'Basic.GetOk': 46 | callback(channel, method_frame, header_frame, body) 47 | else: 48 | all_done() 49 | -------------------------------------------------------------------------------- /autoexp_pool.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | from multiprocessing import Pool 4 | import autoexp 5 | import sys 6 | 7 | # Make sure the database exists to avoid race conditions 8 | autoexp.setup() 9 | 10 | pool = Pool() 11 | 12 | # By specifying a timeout, keyboard interrupts are processed. 13 | # See http://stackoverflow.com/a/1408476/670527 14 | pool.map_async(autoexp.run_and_process, autoexp.inputs).get(sys.maxint) 15 | 16 | -------------------------------------------------------------------------------- /autoexp_producer.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | import autoexp 4 | import argparse 5 | import json 6 | import pika 7 | import sys 8 | import logging 9 | logging.basicConfig(level=logging.INFO) 10 | 11 | parser = argparse.ArgumentParser(description='Set up experiments in RabbitMQ server.') 12 | parser.add_argument('--server', dest='server', action='store', 13 | default='localhost', 14 | help='address of the RabbitMQ server (default: localhost)') 15 | args = parser.parse_args() 16 | 17 | connection = pika.BlockingConnection(pika.ConnectionParameters( 18 | host=args.server)) 19 | channel = connection.channel() 20 | 21 | channel.queue_declare(queue='autoexp_queue', durable=True) 22 | 23 | # Make sure the database exists to avoid race conditions 24 | autoexp.setup() 25 | 26 | for input in autoexp.inputs: 27 | channel.basic_publish(exchange='', 28 | routing_key='autoexp_queue', 29 | body=json.dumps(input), 30 | properties=pika.BasicProperties( 31 | delivery_mode = 2, # make message persistent 32 | )) 33 | 34 | connection.close() 35 | 36 | logging.info("Producer has filled the queue") 37 | -------------------------------------------------------------------------------- /config.py: -------------------------------------------------------------------------------- 1 | # output to google doc 2 | use_google=True 3 | # key of your google docs spreadsheet 4 | key="0Au4zXzOoce8JdGFjZ0JBVTIxRmgzeEpZN0VFRVktb0E" 5 | # google id 6 | user='user@google.com' 7 | # google password 8 | password='password' 9 | 10 | # output to rabbitmq 11 | output_rabbitmq=False 12 | -------------------------------------------------------------------------------- /histsummary-slides.emf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/edmcman/experiment-scripts/9dac471f4f366624176821bb45cdcf1dea7d90fb/histsummary-slides.emf -------------------------------------------------------------------------------- /histsummary-slides.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/edmcman/experiment-scripts/9dac471f4f366624176821bb45cdcf1dea7d90fb/histsummary-slides.pdf -------------------------------------------------------------------------------- /histsummary.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/edmcman/experiment-scripts/9dac471f4f366624176821bb45cdcf1dea7d90fb/histsummary.pdf -------------------------------------------------------------------------------- /paper.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/edmcman/experiment-scripts/9dac471f4f366624176821bb45cdcf1dea7d90fb/paper.pdf -------------------------------------------------------------------------------- /paper.tex: -------------------------------------------------------------------------------- 1 | % Example LaTeX file that uses statistics produced by stats.Rnw 2 | 3 | \documentclass{article} 4 | %\usepackage{fullpage} 5 | \usepackage{graphicx} 6 | \usepackage{xspace} 7 | 8 | \author{Edward J. Schwartz \and Thanassis Avgerinos} 9 | \title{Ed or Thanassis: Which takes longer to type?} 10 | 11 | \input{stats} 12 | 13 | \begin{document} 14 | \maketitle 15 | 16 | In this paper, we analyze which name takes longer to type: Ed or 17 | Thanassis. We performed \numsamples samples in total to determine 18 | which takes longer. Our experiments show that Ed takes \edmean 19 | seconds to type on average, compared to Thanassis, which takes 20 | \thanassismean seconds on average. This can be seen in 21 | Figure~\ref{fig:hist}. 22 | 23 | \begin{figure} 24 | \includegraphics[width=\textwidth]{histsummary.pdf} 25 | \caption{The sample mean time it takes to type Ed and Thanassis in 26 | \numsamples samples. Error bars show 95\% confidence interval.} 27 | \label{fig:hist} 28 | \end{figure} 29 | 30 | \end{document} 31 | 32 | %%% Local Variables: 33 | %%% mode: latex 34 | %%% TeX-master: t 35 | %%% End: 36 | -------------------------------------------------------------------------------- /slides.pptx: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/edmcman/experiment-scripts/9dac471f4f366624176821bb45cdcf1dea7d90fb/slides.pptx -------------------------------------------------------------------------------- /stats.Rnw: -------------------------------------------------------------------------------- 1 | % Example sweave file that gathers statistics from analyze.R 2 | 3 | <>= 4 | source("analyze.R") 5 | @ 6 | 7 | \newcommand{\numsamples}{% 8 | \Sexpr{numsamples}\xspace% 9 | } 10 | 11 | \newcommand{\edmean}{% 12 | \Sexpr{edmean}\xspace% 13 | } 14 | 15 | \newcommand{\thanassismean}{% 16 | \Sexpr{thanassismean}\xspace% 17 | } 18 | --------------------------------------------------------------------------------