├── .gitignore ├── .gitmodules ├── README.md ├── crate-index.js ├── crater-db.js ├── crater-util.js ├── docker └── Dockerfile ├── mocha.opts ├── monitor.js ├── notes.md ├── package.json ├── print-report.js ├── reports.js ├── rs ├── Cargo.lock ├── Cargo.toml ├── crater-api │ ├── Cargo.toml │ └── lib.rs ├── crater-bus │ ├── Cargo.toml │ └── lib.rs ├── crater-cli │ └── main.rs ├── crater-db │ ├── Cargo.toml │ └── lib.rs ├── crater-engine │ ├── Cargo.toml │ └── lib.rs ├── crater-web │ └── main.rs ├── static │ └── index.html └── taskcluster │ ├── Cargo.toml │ └── lib.rs ├── run-crater-task.sh ├── rust-dist.js ├── schedule-tasks.js ├── scheduler.js ├── test.js └── test ├── dist ├── 2015-03-03 │ └── channel-rust-beta └── index.json └── versions └── toml ├── 0.1.18 └── 0.1.7 /.gitignore: -------------------------------------------------------------------------------- 1 | *~ 2 | cache 3 | node_modules 4 | tc-credentials.json 5 | pulse-credentials.json 6 | pg-credentials.json 7 | rs/target/* 8 | crater-web-config.json 9 | crater-cli-config.json -------------------------------------------------------------------------------- /.gitmodules: -------------------------------------------------------------------------------- 1 | [submodule "test/crates.io-index"] 2 | path = test/crates.io-index 3 | url = git://github.com/rust-lang/crates.io-index 4 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Crater - The Rust crate tester 2 | 3 | This is a tool for testing builds of the Rust compiler against massive 4 | numbers of Rust crates. 5 | 6 | It is in a very rough state and the official deployment is 7 | presently accessible by invitation only. 8 | 9 | # Getting started 10 | 11 | Clone the repo and cd to the 'rs' directory. 12 | 13 | ```sh 14 | $ git clone https://github.com/brson/taskcluster-crater 15 | $ cd taskcluster-crater/rs 16 | ``` 17 | 18 | In this directory create crater-cli-config.json. It looks like this: 19 | 20 | ```json 21 | { 22 | "server_url": "https://crater.rust-lang.org", 23 | "username": "your_username", 24 | "auth_token": "your_token" 25 | } 26 | ``` 27 | 28 | Get your credentials from brson. 29 | 30 | Test your configuration: 31 | 32 | ```sh 33 | $ cargo run --bin crater-cli self-test 34 | ``` 35 | 36 | If it says 'self-test succeeded' you've authenticated. Now you can 37 | cause some havok. 38 | 39 | There are three steps to running crater: 40 | 41 | 1. Build two custom toolchains (optional). 42 | 2. Build crates against two toolchains. 43 | 3. Run a 'comparison' report. 44 | 45 | ## Step 1 - build two custom toolchains 46 | 47 | Skip this step if you are just testing a compiler from an official 48 | release channel. 49 | 50 | To test a revision of the compiler that has not yet been merged into 51 | master you will need to upload the branch to GitHub, then ask crater 52 | to build it for you. You will need two commit shas: the most recent 53 | commit on master that you are working off of, and the 'merge-base' 54 | with upstream master (the most recent commit your branch shares with 55 | master). Record these somewhere; you'll need them later. In the 56 | remaining examples these will be refered to as `$SHA1` and `$SHA2`, 57 | and the repo address as `$REPO`. 58 | 59 | Build the toolchains: 60 | 61 | ```sh 62 | $ cargo run --bin crater-cli custom-build $REPO $SHA1 63 | $ cargo run --bin crater-cli custom-build $REPO $SHA2 64 | ``` 65 | 66 | These will launch the builds, which will take an hour or two to 67 | complete. Each of the above commands will print an 'inspector link', a 68 | link to the TaskCluster page for that build. Check back on these links 69 | periodically until they are both finished. It will take 1-2 hours. 70 | 71 | Once these builds are done you can start testing crates. 72 | 73 | Note: if you are just testing official builds then identify the 74 | toolchains using multirust-style 'toolchain specs', 75 | e.g. 'nightly-2015-06-06'. Always test compilers from the archives and 76 | not straight from the release channel (e.g. 'nightly') since the 77 | release channel compilers change regularly and the results won't make 78 | sense down the road if we do historical analysis. 79 | 80 | # Step 2 - build lots of crates 81 | 82 | As before, we're going to ask Crater to run some builds, then wait 83 | a few hours while those builds complete. 84 | 85 | ```sh 86 | $ cargo run --bin crater-cli crate-build $SHA1 87 | $ cargo run --bin crater-cli crate-build $SHA2 88 | ``` 89 | 90 | Both of these commands will print a ton of inspector links. You'll 91 | probably just want to ignore them since they are not worth monitoring 92 | individually. Instead, just wait two hours, then proceed to step 3. 93 | 94 | This command will take a long time to complete, displaying no 95 | progress. Give it a few minutes. 96 | 97 | You might also watch the [status page for the TaskCluster AWS 98 | provisioner][prov], waiting for the number of builds on the 'crater' 99 | workers to drop back to zero before proceeding to step 3. 100 | 101 | [prov]: https://tools.taskcluster.net/aws-provisioner/ 102 | 103 | # Step 3 - run the report 104 | 105 | Now you can ask for a 'comparison' report: 106 | 107 | ```sh 108 | $ cargo run --bin crater-cli report comparison $SHA1 $SHA2 109 | ``` 110 | 111 | It will report statuses for some number of crates. Knowing whether the 112 | report is 'done' and the crates have built correctly is not simple - 113 | generally, if I see the number of known statuses is in the right 114 | ballpark and the number of unknown statuses is minimal (maybe 30-40) 115 | then I consider the coverage sufficient. 116 | 117 | If the numbers look wrong then either the builds are not finished, 118 | something went wrong internally to the cobbled-together distributed 119 | system that is Crater, or you've issued one of the commands 120 | incorrectly. 121 | 122 | If the numbers are weird then you might ask for a report on a single 123 | toolchain at a time, which can tell you if you got the commands for 124 | one or the other incorrect: 125 | 126 | ```sh 127 | $ cargo run --bin crater-cli report toolchain $SHA1 128 | ``` 129 | 130 | OK, that's all I can tell you for now. Good luck. Sorry it's so rough. 131 | 132 | # Older docs 133 | 134 | This is a collection of node.js tools for testing large numbers of 135 | Rust crates against arbitrary builds of Rust in parallel. 136 | 137 | It currently consistents of a variety of modules for accessing 138 | services, scheduling builds, monitoring status, analysis and 139 | reporting, as well as several command-line utilities for interacting 140 | with the system. 141 | 142 | **Note: currently Crater is unusable without a local installation and 143 | a number of credentials. Eventually it will be deployed somewhere 144 | with a more convenient interface.** 145 | 146 | Crater has a number of service dependencies, that make it difficult to 147 | set up: 148 | 149 | * [TaskCluster] for coordinating builds, and specifically Mozilla's 150 | instance of TaskCluster. Requires credentials. 151 | * [Pulse], Mozilla's AMQP service, used by TaskCluster. Requires 152 | credentials. 153 | * [The crates.io index]. A git repo containing metadata about 154 | registered crate revisions for crates.io. 155 | * The crates.io API. For downloading metadata 156 | * The Rust distribution S3 bucket. For downloading crates and builds. 157 | * A PostgreSQL database for storing results (Amazon RDS). Requires 158 | credentials. 159 | 160 | [TaskCluster]: http://docs.taskcluster.net/ 161 | [Pulse]: https://pulse.mozilla.org/ 162 | [The crates.io index]: https://github.com/rust-lang/crates.io-index/ 163 | 164 | # Modules 165 | 166 | * `crate-index.js` - Functions relating to crates.io and the crates. 167 | * `rust-dist.js` - Access to Rust release channels. 168 | * `crater-db.js` - Domain specific storage abstractions over a SQL 169 | database. 170 | * `reports.js` - Report generation. 171 | * `scheduler.js` - Logic for scheduling builds. 172 | * `monitor.js` - Deamon that monitors the pulse queue for events. 173 | * `schedule-tasks.js` - CLI tool for scheduling builds. 174 | * `print-report.js` - CLI tool for creating reports. 175 | * `crate-util.js` - Common stuff. 176 | * `test.js` - Unit tests. 177 | 178 | # CLI tools 179 | 180 | Scheduling a test the 20 most popular crates againast a specific toolchain: 181 | 182 | $ nodejs schedule-tasks.js nightly-2015-03-01 --top 20 --most-recent-only 183 | 184 | Running the result monitoring and storage service: 185 | 186 | $ nodejs monitor.js 187 | 188 | monitor.js will store the results in a database for later analysis. 189 | 190 | Running reports: 191 | 192 | $ nodejs print-report.js comparison nightly-2015-03-01 nightly-2015-03-02 193 | 194 | # Credentials 195 | 196 | The files "tc-credentials.json", "pulse-credentials.json", and "pg-credentials.json" must 197 | be in the current directory. 198 | 199 | tc-credentials.json looks like yon: 200 | 201 | ``` 202 | { 203 | "clientId": "...", 204 | "accessToken": "...", 205 | "certificate": { ... } 206 | } 207 | ``` 208 | 209 | The values can be obtained from https://auth.taskcluster.net/. 210 | 211 | pulse-credentials.json looks like yon: 212 | 213 | ``` 214 | { 215 | "username": "...", 216 | "password": "..." 217 | } 218 | ``` 219 | 220 | The values can be obtained from https://pulse.mozilla.org. 221 | 222 | ## PostgreSQL setup 223 | 224 | monitor.js and test.js needs a PostgreSQL user, which can be set up 225 | with 226 | 227 | sudo -u postgres createuser $USER 228 | 229 | You'll need a test database and a production database. 230 | 231 | sudo -u postgres createdb crater-test -O $USER 232 | sudo -u postgres createdb crater -O $USER 233 | 234 | And create a password for the user 235 | 236 | psql -c "\password" -U $USER -d crater 237 | 238 | The credentials need to be in `pg-credentials.js`: 239 | 240 | ``` 241 | { 242 | "username": "...", 243 | "password": "...", 244 | "host": "...", 245 | "port": 5432 246 | } 247 | ``` 248 | 249 | # Testing 250 | 251 | $ npm test 252 | 253 | You'll need to have a 'crater-test' database configured locally, 254 | for user 'crater-test' with password 'crater-test'. 255 | 256 | # Creating the docker image 257 | 258 | ``` 259 | $ docker build -t brson/crater:1 . 260 | $ docker push brson/crater 261 | ``` 262 | 263 | # Future work 264 | 265 | * Use task graphs that mirror the crate dependency structure. 266 | * Custom builds 267 | * Only create tasks for build_results we don't have yet, unless --all is passed 268 | * REST service 269 | * HTML frontend 270 | * CLI <-> REST frontend 271 | * Use customized docker containers to avoid huge dls 272 | * Fix urls in pop report 273 | * Crate DAGs with stability coloring 274 | * Add analysis of feature usage 275 | * Toolchain build status 276 | * non-crates.io projects, focus on prod users 277 | * https://github.com/witheve/Eve/ 278 | * nightly breakage dashboard 279 | 280 | 11:53 if we had a massive dashboard showing, for every crate in crates.io (and eventually other sources): 281 | 11:54 - most recent stable release 282 | 11:54 - most recent nightly release 283 | 11:54 - when was it last tested 284 | -------------------------------------------------------------------------------- /crate-index.js: -------------------------------------------------------------------------------- 1 | 'use strict'; 2 | 3 | var debug = require('debug')(__filename.slice(__dirname.length + 1)); 4 | var Promise = require('promise'); 5 | var _fs = require('graceful-fs'); // Need to parse many files at once 6 | var fs = { 7 | existsSync: _fs.existsSync, 8 | readFile: Promise.denodeify(_fs.readFile), 9 | writeFile: Promise.denodeify(_fs.writeFile), 10 | }; 11 | var walk = require('walkdir'); 12 | var path = require('path'); 13 | var semver = require('semver'); 14 | var util = require('./crater-util'); 15 | var assert = require('assert'); 16 | 17 | var localIndexName = "crate-index" 18 | var crateCacheName = "crate-cache" 19 | var sourceCacheName = "source-cache" 20 | 21 | /** 22 | * Ensures the index is fresh and metadata for all crates is cached. 23 | */ 24 | function updateCaches(config) { 25 | return cloneIndex(config); 26 | } 27 | 28 | /** 29 | * Ensure that the crate-index repository is either present or created 30 | */ 31 | function cloneIndex(config) { 32 | 33 | var indexAddr = config.crateIndexAddr; 34 | var cacheDir = config.cacheDir; 35 | 36 | var localIndex = path.join(cacheDir, localIndexName); 37 | 38 | var p; 39 | if (fs.existsSync(localIndex)) { 40 | p = util.runCmd('git pull origin master', {cwd: localIndex}); 41 | } else { 42 | p = util.runCmd('mkdir -p ' + localIndex); 43 | p = util.runCmd('git clone ' + indexAddr + ' ' + localIndex); 44 | } 45 | 46 | p = p.then(function(x) { 47 | debug('Repository exists'); 48 | }); 49 | 50 | return p; 51 | } 52 | 53 | /** 54 | * Find all files in the repository which are not git files. 55 | */ 56 | function findFiles(directory) { 57 | return new Promise(function(resolve, reject) { 58 | var filesFound = []; 59 | var dir = path.resolve(directory); 60 | var dirLength = dir.length + 1; // Avoid the unnecessary . 61 | var emitter = walk(directory); 62 | 63 | // Store files 64 | emitter.on('file', function(_filename, stat) { 65 | var filename = path.resolve(_filename); 66 | filename = filename.slice(dirLength); 67 | // Ignore gitfiles and top level files (e.g. config.json) 68 | if (!/^[.]git\//.test(filename) && filename.indexOf(path.sep) !== -1) { 69 | filesFound.push(filename); 70 | } 71 | }); 72 | 73 | // Handle errors 74 | emitter.on('error', function(err) { 75 | reject(err); 76 | }); 77 | 78 | // Fails are found files which could not be stat'd 79 | emitter.on('fail', function(fail) { 80 | debug('Failed to read a file! %s', fail); 81 | }); 82 | 83 | // Resolve when all files are read 84 | emitter.on('end', function() { 85 | resolve(filesFound); 86 | }); 87 | }); 88 | }; 89 | 90 | /** 91 | * Read all versions of a given descriptor file and parse them into 92 | * JSON 93 | */ 94 | function readFile(dir, filename) { 95 | return fs.readFile(path.join(dir, filename), 'utf-8').then(function(filedata) { 96 | var files = filedata.split('\n'); 97 | return files.filter(function(f) { return !!f }).map(function(f) { 98 | return JSON.parse(f); 99 | }); 100 | }); 101 | } 102 | 103 | 104 | /** 105 | * Load the crate index from the remote address. 106 | */ 107 | function loadCrates(config) { 108 | return util.serial(function() { 109 | var indexAddr = config.crateIndexAddr; 110 | var cacheDir = config.cacheDir; 111 | 112 | var localIndex = path.join(cacheDir, localIndexName); 113 | 114 | var p = Promise.resolve(); 115 | 116 | p = p.then(function() { 117 | debug('repos asserted'); 118 | return findFiles(localIndex) 119 | }); 120 | 121 | p = p.then(function(filenames) { 122 | debug('files found'); 123 | return Promise.all(filenames.map(function(filename) { 124 | return readFile(localIndex, filename); 125 | })); 126 | }); 127 | 128 | p = p.then(function(res) { 129 | debug('files read'); 130 | var flat = []; 131 | res.forEach(function(r) { 132 | Array.prototype.push.apply(flat, r); 133 | }); 134 | return flat; 135 | }); 136 | 137 | return p; 138 | }); 139 | } 140 | 141 | /** 142 | * Given the resolved output from `loadCrates`, return a map from crate 143 | * names to arrays of crate data. 144 | */ 145 | function getMostRecentRevs(crates) { 146 | var map = {}; 147 | 148 | // maps in js have a default key called "constructor" and there's a crate 149 | // called "constructor"... 150 | map["constructor"] = null; 151 | 152 | crates.forEach(function(c) { 153 | if (map[c.name] == null) { 154 | map[c.name] = c; 155 | } else { 156 | if (semver.lt(map[c.name].vers, c.vers)) { 157 | map[c.name] = c; 158 | } 159 | } 160 | }); 161 | 162 | return map; 163 | } 164 | 165 | /** 166 | * Given the resolved output from `loadCrates`, return a map from crate 167 | * names to arrays of dependencies, using data from the most recent crate revisions 168 | * (so it is not perfectly accurate). 169 | */ 170 | function getDag(crates) { 171 | var mostRecent = getMostRecentRevs(crates); 172 | var map = { }; 173 | for (var k in mostRecent) { 174 | var crate = mostRecent[k]; 175 | var deps = []; 176 | crate.deps.forEach(function(dep) { 177 | deps.push(dep.name); 178 | }); 179 | map[crate.name] = deps; 180 | } 181 | return map; 182 | } 183 | 184 | /** 185 | * Return a map from crate names to number of transitive downstream users. 186 | */ 187 | function getPopularityMap(crates) { 188 | 189 | var users = { }; 190 | 191 | // Set users of every crate to 0 192 | crates.forEach(function(crate) { 193 | users[crate.name] = 0; 194 | }); 195 | 196 | var dag = getDag(crates); 197 | for (var crateName in dag) { 198 | var depStack = dag[crateName]; 199 | while (depStack && depStack.length != 0) { 200 | var nextDep = depStack.pop(); 201 | if (users[nextDep] == null) { 202 | debug("dep " + nextDep + " is unknown. probably filtered out earlier"); 203 | users[nextDep] = 0; 204 | } 205 | assert(users[nextDep] != null); 206 | users[nextDep] += 1; 207 | 208 | if (dag[nextDep]) { 209 | depStack.concat(dag[nextDep]); 210 | } 211 | } 212 | } 213 | 214 | return users; 215 | } 216 | 217 | exports.updateCaches = updateCaches; 218 | exports.cloneIndex = cloneIndex; 219 | exports.loadCrates = loadCrates; 220 | exports.getMostRecentRevs = getMostRecentRevs; 221 | exports.getDag = getDag; 222 | exports.getPopularityMap = getPopularityMap; 223 | -------------------------------------------------------------------------------- /crater-db.js: -------------------------------------------------------------------------------- 1 | /* 2 | * Stores and retrieves results from test runs. 3 | */ 4 | 5 | 'use strict'; 6 | 7 | var debug = require('debug')(__filename.slice(__dirname.length + 1)); 8 | var Promise = require('promise'); 9 | var pg = require('pg'); 10 | var util = require('./crater-util'); 11 | 12 | /** 13 | * Connects to a PostgreSQL DB and returns a promise of an opaque type 14 | * accepted as context to other functions here. 15 | */ 16 | function connect(config) { 17 | var credentials = config.dbCredentials; 18 | 19 | var dbctx = new Promise(function(resolve, reject) { 20 | var client = new pg.Client({ 21 | database: config.dbName, 22 | user: credentials.username, 23 | password: credentials.password, 24 | host: credentials.host || null, 25 | port: credentials.port || null 26 | }); 27 | 28 | client.connect(function(err) { 29 | if (!err) { 30 | resolve({ client: client }); 31 | } else { 32 | reject(err); 33 | } 34 | }); 35 | }); 36 | return dbctx.then(function(dbctx) { 37 | var p = populate(dbctx); 38 | var p = p.then(function() { return dbctx; }); 39 | return p; 40 | }); 41 | } 42 | 43 | function disconnect(dbctx) { 44 | dbctx.client.end(); 45 | return Promise.resolve(); 46 | } 47 | 48 | /** 49 | * Creates the tables of a database return a promise of nothing. Taks 50 | * a promise of a database context created by `connect`. 51 | */ 52 | function populate(dbctx) { 53 | var q = "create table if not exists \ 54 | build_results ( \ 55 | toolchain text not null, \ 56 | crate_name text not null, crate_vers text not null, \ 57 | status text not null, \ 58 | task_id text not null, \ 59 | primary key ( \ 60 | toolchain, crate_name, crate_vers ) ) \ 61 | "; 62 | return new Promise(function (resolve, reject) { 63 | dbctx.client.query(q, function(e, r) { 64 | if (e) { reject(e); } 65 | else { resolve(r); } 66 | }); 67 | }).then(function() { 68 | return new Promise(function(resolve, reject) { 69 | var q = "create table if not exists \ 70 | custom_toolchains ( \ 71 | toolchain text not null, \ 72 | url text not null, \ 73 | task_id text not null, \ 74 | primary key (toolchain) )"; 75 | dbctx.client.query(q, function(e, r) { 76 | if (e) { reject(e); } 77 | else { resolve(r); } 78 | }); 79 | }); 80 | }).then(function() { 81 | return new Promise(function(resolve, reject) { 82 | var q = "create table if not exists \ 83 | crate_versions ( \ 84 | name text not null, \ 85 | version text not null, \ 86 | primary key (name, version) )"; 87 | dbctx.client.query(q, function(e, r) { 88 | if (e) { reject(e); } 89 | else { resolve(r); } 90 | }); 91 | }); 92 | }).then(function() { 93 | return new Promise(function(resolve, reject) { 94 | var q = "create table if not exists \ 95 | crate_rank ( \ 96 | name text not null, \ 97 | rank integer not null, \ 98 | primary key (name) )"; 99 | dbctx.client.query(q, function(e, r) { 100 | if (e) { reject(e); } 101 | else { resolve(r); } 102 | }); 103 | }); 104 | }).then(function() { 105 | return new Promise(function(resolve, reject) { 106 | var q = "create table if not exists \ 107 | dep_edges ( \ 108 | name text not null, \ 109 | dep text not null, \ 110 | primary key (name, dep) )"; 111 | dbctx.client.query(q, function(e, r) { 112 | if (e) { reject(e); } 113 | else { resolve(r); } 114 | }); 115 | }); 116 | }); 117 | } 118 | 119 | /** 120 | * Destroys the tables. 121 | */ 122 | function depopulate(dbctx) { 123 | var q = "drop table if exists build_results"; 124 | debug(q); 125 | return new Promise(function (resolve, reject) { 126 | dbctx.client.query(q, function(e, r) { 127 | if (e) { reject(e); } 128 | else { resolve(r); } 129 | }); 130 | }).then(function() { 131 | var q = "drop table if exists custom_toolchains"; 132 | debug(q); 133 | return new Promise(function(resolve, reject) { 134 | dbctx.client.query(q, function(e, r) { 135 | if (e) { reject(e); } 136 | else { resolve(r); } 137 | }); 138 | }); 139 | }).then(function() { 140 | var q = "drop table if exists crate_versions"; 141 | return new Promise(function(resolve, reject) { 142 | dbctx.client.query(q, function(e, r) { 143 | if (e) { reject(e); } 144 | else { resolve(r); } 145 | }); 146 | }); 147 | }).then(function() { 148 | var q = "drop table if exists crate_rank"; 149 | return new Promise(function(resolve, reject) { 150 | dbctx.client.query(q, function(e, r) { 151 | if (e) { reject(e); } 152 | else { resolve(r); } 153 | }); 154 | }); 155 | }).then(function() { 156 | var q = "drop table if exists dep_edges"; 157 | return new Promise(function(resolve, reject) { 158 | dbctx.client.query(q, function(e, r) { 159 | if (e) { reject(e); } 160 | else { resolve(r); } 161 | }); 162 | }); 163 | }); 164 | } 165 | 166 | /** 167 | * Adds a build result and returns a promise of nothing. buildResult should 168 | * look like `{ toolchain: ..., crateName: ..., crateVers: ..., status: ..., 169 | * taskId: ... }`. 170 | */ 171 | function addBuildResult(dbctx, buildResult) { 172 | return new Promise(function (resolve, reject) { 173 | var f = function(e, r) { 174 | dbctx.client.query('commit', function(err, res) { 175 | if (e) { reject(e); } 176 | else { resolve(); } 177 | }); 178 | }; 179 | 180 | dbctx.client.query('begin', function(err, res) { 181 | if (err) { 182 | reject(err); 183 | return; 184 | } 185 | var p = getBuildResult(dbctx, buildResult); 186 | p.then(function(r) { 187 | if (r == null) { 188 | var q = "insert into build_results values ($1, $2, $3, $4, $5)"; 189 | debug(q); 190 | dbctx.client.query(q, [util.toolchainToString(buildResult.toolchain), 191 | buildResult.crateName, 192 | buildResult.crateVers, 193 | buildResult.status, 194 | buildResult.taskId], 195 | f); 196 | } else { 197 | var q = "update build_results set status = $4, task_id = $5 where \ 198 | toolchain = $1 and crate_name = $2 and crate_vers = $3"; 199 | debug(q); 200 | dbctx.client.query(q, [util.toolchainToString(buildResult.toolchain), 201 | buildResult.crateName, 202 | buildResult.crateVers, 203 | buildResult.status, 204 | buildResult.taskId], 205 | f); 206 | } 207 | }).catch(function(e) { 208 | reject(e); 209 | }); 210 | }); 211 | 212 | }); 213 | } 214 | 215 | /** 216 | * Adds a build result and returns a promise of a build 217 | * result. buildResultKey should look like `{ toolchain: ..., 218 | * crateName: ..., crateVers: ... }`. 219 | * 220 | * Returns a promised null if there is no build result for the key. 221 | */ 222 | function getBuildResult(dbctx, buildResultKey) { 223 | var q = "select * from build_results where \ 224 | toolchain = $1 and crate_name = $2 and crate_vers = $3"; 225 | debug(q); 226 | return new Promise(function (resolve, reject) { 227 | var f = function(e, r) { 228 | if (e) { reject(e); } 229 | else { 230 | if (r.rows.length > 0) { 231 | var row = r.rows[0]; 232 | resolve({ 233 | toolchain: util.parseToolchain(row.toolchain), 234 | crateName: row.crate_name, 235 | crateVers: row.crate_vers, 236 | status: row.status, 237 | taskId: row.task_id 238 | }); 239 | } else { 240 | resolve(null); 241 | } 242 | } 243 | }; 244 | 245 | dbctx.client.query(q, [util.toolchainToString(buildResultKey.toolchain), 246 | buildResultKey.crateName, 247 | buildResultKey.crateVers], 248 | f); 249 | }); 250 | } 251 | 252 | /** 253 | * Returns a promise of an array of pairs of build results for a given 254 | * pair of toolchains. Each element of the array looks like 255 | * `{ crateName: ..., crateVers: ..., from: ..., to: ... }`, 256 | * and `from` and `to` look like `{ succes: bool }`. 257 | */ 258 | function getResultPairs(dbctx, fromToolchain, toToolchain) { 259 | var q = "select a.crate_name, a.crate_vers, a.status as from_status, b.status as to_status, \ 260 | a.task_id as from_task_id, b.task_id as to_task_id \ 261 | from build_results a, build_results b \ 262 | where a.toolchain = $1 and b.toolchain = $2 \ 263 | and a.crate_name = b.crate_name and a.crate_vers = b.crate_vers \ 264 | order by a.crate_name, a.crate_vers"; 265 | debug(q); 266 | return new Promise(function(resolve, reject) { 267 | var f = function(e, r) { 268 | if (e) { reject(e); } 269 | else { 270 | var results = [] 271 | r.rows.forEach(function(row) { 272 | debug("result row: " + JSON.stringify(row)); 273 | results.push({ 274 | crateName: row.crate_name, 275 | crateVers: row.crate_vers, 276 | from: { status: row.from_status, taskId: row.from_task_id }, 277 | to: { status: row.to_status, taskId: row.to_task_id } 278 | }); 279 | }); 280 | resolve(results); 281 | } 282 | }; 283 | 284 | dbctx.client.query(q, [util.toolchainToString(fromToolchain), 285 | util.toolchainToString(toToolchain)], 286 | f); 287 | }); 288 | } 289 | 290 | function getResults(dbctx, toolchain) { 291 | var q = "select * from build_results \ 292 | where toolchain = $1 order by crate_name"; 293 | debug(q); 294 | return new Promise(function(resolve, reject) { 295 | var f = function(e, r) { 296 | if (e) { reject(e); } 297 | else { 298 | var results = []; 299 | r.rows.forEach(function(row) { 300 | results.push({ 301 | crateName: row.crate_name, 302 | crateVers: row.crate_vers, 303 | status: row.status, 304 | taskId: row.task_id 305 | }); 306 | }); 307 | resolve(results); 308 | } 309 | }; 310 | 311 | dbctx.client.query(q, [util.toolchainToString(toolchain)], f); 312 | }); 313 | } 314 | 315 | function addCustomToolchain(dbctx, custom) { 316 | return new Promise(function(resolve, reject) { 317 | var f = function(e, r) { 318 | dbctx.client.query('commit', function(err, res) { 319 | if (e) { reject(e); } 320 | else { resolve(); } 321 | }); 322 | }; 323 | 324 | dbctx.client.query('begin', function(err, res) { 325 | if (err) { reject(err); return; } 326 | 327 | var p = getCustomToolchain(dbctx, custom.toolchain); 328 | p.then(function(r) { 329 | if (r == null) { 330 | var q = "insert into custom_toolchains values ($1, $2, $3)"; 331 | debug(q); 332 | dbctx.client.query(q, [util.toolchainToString(custom.toolchain), 333 | custom.url, 334 | custom.taskId], f); 335 | } else { 336 | var q = "update custom_toolchains set url = $2, task_id = $3 where toolchain = $1"; 337 | debug(q); 338 | dbctx.client.query(q, [util.toolchainToString(custom.toolchain), 339 | custom.url, 340 | custom.taskId], f); 341 | } 342 | }).catch(function(e) { 343 | reject(e); 344 | }); 345 | }); 346 | }); 347 | } 348 | 349 | function getCustomToolchain(dbctx, toolchain) { 350 | var q = "select * from custom_toolchains where toolchain = $1"; 351 | debug(q); 352 | return new Promise(function(resolve, reject) { 353 | var f = function(e, r) { 354 | if (e) { reject(e); } 355 | else { 356 | if (r.rows.length > 0) { 357 | var row = r.rows[0]; 358 | resolve({ 359 | toolchain: toolchain, 360 | url: row.url, 361 | taskId: row.task_id 362 | }); 363 | } else { 364 | resolve(null); 365 | } 366 | } 367 | }; 368 | 369 | dbctx.client.query(q, [util.toolchainToString(toolchain)], f); 370 | }); 371 | } 372 | 373 | exports.connect = connect; 374 | exports.disconnect = disconnect; 375 | exports.populate = populate; 376 | exports.depopulate = depopulate; 377 | exports.addBuildResult = addBuildResult; 378 | exports.getBuildResult = getBuildResult; 379 | exports.getResultPairs = getResultPairs; 380 | exports.getResults = getResults; 381 | exports.addCustomToolchain = addCustomToolchain; 382 | exports.getCustomToolchain = getCustomToolchain; 383 | -------------------------------------------------------------------------------- /crater-util.js: -------------------------------------------------------------------------------- 1 | 'use strict'; 2 | 3 | var Promise = require('promise'); 4 | var _fs = require('graceful-fs'); // Need to parse many files at once 5 | var fs = { 6 | readFile: Promise.denodeify(_fs.readFile), 7 | readFileSync: _fs.readFileSync 8 | }; 9 | var exec = require('child_process').exec; 10 | var http = require('http'); 11 | var https = require('https'); 12 | var assert = require('assert'); 13 | var async = require('async'); 14 | var url = require('url'); 15 | 16 | var defaultRustDistAddr = "http://static-rust-lang-org.s3-us-west-1.amazonaws.com/dist"; 17 | var defaultCrateIndexAddr = "https://github.com/rust-lang/crates.io-index"; 18 | var defaultCacheDir = "./cache"; 19 | var defaultDlRootAddr = "https://crates.io/api/v1/crates"; 20 | var defaultDbCredentialsFile = "./pg-credentials.json"; 21 | var defaultPulseCredentialsFile = "./pulse-credentials.json"; 22 | var defaultTcCredentialsFile = "./tc-credentials.json"; 23 | var defaultDbName = "crater"; 24 | 25 | /** 26 | * Parses a string toolchain identifier into an object { channel: string, date: string } 27 | */ 28 | function parseToolchain(toolchainName) { 29 | if (toolchainName == null) { return null; } 30 | 31 | var ret_channel; 32 | var ret_date; 33 | ["nightly", "beta", "stable"].forEach(function(channel) { 34 | var prefix = channel + "-"; 35 | var ix = toolchainName.indexOf(prefix); 36 | if (ix != -1) { 37 | ret_channel = channel; 38 | ret_date = toolchainName.slice(prefix.length); 39 | } 40 | }); 41 | 42 | if (ret_channel) { 43 | return { 44 | channel: ret_channel, 45 | archiveDate: ret_date, 46 | }; 47 | } else { 48 | // It must be a 40-character sha 49 | if (toolchainName.length == 40) { 50 | return { 51 | customSha: toolchainName 52 | } 53 | } 54 | return null; 55 | } 56 | } 57 | 58 | function toolchainToString(toolchain) { 59 | assert((toolchain.channel && toolchain.archiveDate) || toolchain.customSha); 60 | if (!toolchain.customSha) { 61 | return toolchain.channel + "-" + toolchain.archiveDate; 62 | } else { 63 | return toolchain.customSha; 64 | } 65 | } 66 | 67 | function downloadToMem(addr) { 68 | 69 | var addr2 = url.parse(addr); 70 | var opts = { 71 | host: addr2.host, 72 | hostname: addr2.host, 73 | port: addr2.port, 74 | path: addr2.pathname, 75 | pathname: addr2.pathname, 76 | method: 'GET', 77 | headers: { 78 | "User-Agent": "crater - the Rust crate tester" 79 | } 80 | }; 81 | 82 | if (addr.lastIndexOf("https", 0) === 0) { 83 | return new Promise(function(resolve, reject) { 84 | https.get(opts, function(res) { 85 | var data = ''; 86 | 87 | res.on('error', function(e) { reject(e); }); 88 | res.on('data', function(d) { data += d; }); 89 | res.on('end', function() { 90 | resolve(data); 91 | }); 92 | }); 93 | }); 94 | } else if (addr.lastIndexOf("http", 0) === 0) { 95 | return new Promise(function(resolve, reject) { 96 | http.get(opts, function(res) { 97 | var data = ''; 98 | 99 | res.on('error', function(e) { reject(e); }); 100 | res.on('data', function(d) { data += d; }); 101 | res.on('end', function() { 102 | resolve(data); 103 | }); 104 | }); 105 | }); 106 | } else { 107 | return Promise.denodeify(fs.readFile)(addr, 'utf-8'); 108 | } 109 | } 110 | 111 | function runCmd(command, options) { 112 | return new Promise(function(resolve, reject) { 113 | exec(command, options, function(err, sout, serr) { 114 | if (err) { 115 | reject(err); 116 | } 117 | resolve({stdout: sout, stderr: serr}); 118 | }); 119 | }); 120 | } 121 | 122 | function rustDate(date) { 123 | var year = date.getUTCFullYear().toString(); 124 | var month = (date.getMonth() + 1).toString(); 125 | if (month.length == 1) { 126 | month = "0" + month; 127 | } 128 | var day = date.getDate().toString(); 129 | if (day.length == 1) { 130 | day = "0" + day; 131 | } 132 | 133 | return year + "-" + month + "-" + day; 134 | } 135 | 136 | function loadDefaultConfig() { 137 | return { 138 | dbName: defaultDbName, 139 | rustDistAddr: defaultRustDistAddr, 140 | crateIndexAddr: defaultCrateIndexAddr, 141 | dlRootAddr: defaultDlRootAddr, 142 | cacheDir: defaultCacheDir, 143 | dbCredentials: loadCredentials(defaultDbCredentialsFile), 144 | pulseCredentials: loadCredentials(defaultPulseCredentialsFile), 145 | tcCredentials: loadCredentials(defaultTcCredentialsFile) 146 | }; 147 | } 148 | 149 | function loadCredentials(credentialsFile) { 150 | return JSON.parse(fs.readFileSync(credentialsFile, "utf8")); 151 | } 152 | 153 | function workDispatcher(task, cb) { 154 | task(cb); 155 | } 156 | 157 | // A queue used to serialize access to the on-disk git repo and caches, 158 | // to avoid corruption. 159 | var actionQueue = async.queue(workDispatcher, 1); 160 | 161 | /** 162 | * Takes a function that returns a promise and ensures that no other serial promises execute 163 | * until is resolved. Returns a promise of that resolved value. 164 | */ 165 | function serial(f) { 166 | return new Promise(function(resolve, reject) { 167 | actionQueue.push(function(dispatcherCb) { 168 | f().then(function(r) { 169 | dispatcherCb(); 170 | resolve(r); 171 | }).catch(function(e) { 172 | dispatcherCb(); 173 | reject(e); 174 | }); 175 | }); 176 | }); 177 | } 178 | 179 | exports.parseToolchain = parseToolchain; 180 | exports.toolchainToString = toolchainToString; 181 | exports.downloadToMem = downloadToMem; 182 | exports.runCmd = runCmd; 183 | exports.rustDate = rustDate; 184 | exports.loadDefaultConfig = loadDefaultConfig; 185 | exports.defaultDbCredentialsFile = defaultDbCredentialsFile; 186 | exports.defaultPulseCredentialsFile = defaultPulseCredentialsFile; 187 | exports.defaultTcCredentialsFile = defaultTcCredentialsFile; 188 | exports.serial = serial; 189 | -------------------------------------------------------------------------------- /docker/Dockerfile: -------------------------------------------------------------------------------- 1 | FROM ubuntu:16.04 2 | MAINTAINER Brian Anderson 3 | 4 | RUN apt-get update 5 | 6 | # Baseline tools 7 | RUN apt-get install -y build-essential \ 8 | git file python2.7 \ 9 | perl curl git libc6-dev-i386 gcc-multilib g++-multilib llvm llvm-dev 10 | RUN apt-get build-dep -y clang llvm 11 | 12 | # Package compatibility 13 | 14 | # Servo 15 | RUN apt-get install -y libz-dev \ 16 | freeglut3-dev \ 17 | libfreetype6-dev libgl1-mesa-dri libglib2.0-dev xorg-dev \ 18 | gperf g++ cmake python-virtualenv \ 19 | libssl-dev libbz2-dev libosmesa6-dev libxmu6 libxmu-dev 20 | 21 | # sdl2 22 | RUN apt-get install -y libsdl2-dev 23 | 24 | # rustqlite 25 | RUN apt-get install -y libsqlite3-dev 26 | 27 | # netlib-provider 28 | RUN apt-get install -y gfortran 29 | 30 | # gdk-sys 31 | RUN apt-get install -y libgtk-3-dev 32 | -------------------------------------------------------------------------------- /mocha.opts: -------------------------------------------------------------------------------- 1 | --ui tdd 2 | --timeout 120s 3 | --reporter spec 4 | test.js 5 | -------------------------------------------------------------------------------- /monitor.js: -------------------------------------------------------------------------------- 1 | /* 2 | * Monitors TaskCluster messages and stores Crater results for later analysis. 3 | */ 4 | 5 | 'use strict'; 6 | 7 | var debug = require('debug')(__filename.slice(__dirname.length + 1)); 8 | var Promise = require('promise'); 9 | var tc = require('taskcluster-client'); 10 | var fs = require('fs'); 11 | var db = require('./crater-db'); 12 | var assert = require('assert'); 13 | var util = require('./crater-util'); 14 | 15 | var defaultPulseCredentialsFile = "./pulse-credentials.json"; 16 | var defaultTcCredentialsFile = "./tc-credentials.json"; 17 | 18 | function main() { 19 | var config = util.loadDefaultConfig(); 20 | var pulseCredentials = config.pulseCredentials; 21 | var tcCredentials = config.tcCredentials; 22 | 23 | db.connect(config).then(function(dbctx) { 24 | 25 | var tcQueue = new tc.Queue({ credentials: tcCredentials }); 26 | 27 | var pulseListener = new tc.PulseListener({ 28 | prefetch: 50, // fetch 50 messages at a time 29 | credentials: pulseCredentials, 30 | queueName: "crater-monitor" // create a durable queue 31 | }); 32 | 33 | var queueEvents = new tc.QueueEvents(); 34 | 35 | pulseListener.bind(queueEvents.taskDefined("route.crater.#")); 36 | pulseListener.bind(queueEvents.taskPending("route.crater.#")); 37 | pulseListener.bind(queueEvents.taskRunning("route.crater.#")); 38 | pulseListener.bind(queueEvents.artifactCreated("route.crater.#")); 39 | pulseListener.bind(queueEvents.taskCompleted("route.crater.#")); 40 | pulseListener.bind(queueEvents.taskFailed("route.crater.#")); 41 | pulseListener.bind(queueEvents.taskException("route.crater.#")); 42 | 43 | pulseListener.on('message', function(m) { 44 | debug("msg: " + JSON.stringify(m)); 45 | 46 | var taskId = m.payload.status.taskId; 47 | var state = m.payload.status.state; 48 | 49 | assert(taskId); 50 | assert(state); 51 | 52 | // Using a single db connection, don't clobber it with concurrency 53 | util.serial(function() { 54 | return new Promise(function(resolve, reject) { 55 | recordResultForTask(dbctx, tcQueue, taskId, state, m); 56 | resolve(null); 57 | }).catch(function(e) { reject(e); }) 58 | }) 59 | 60 | }); 61 | 62 | pulseListener.resume().then(function() { 63 | debug("listening"); 64 | }); 65 | }).catch(function(e) { console.log(e); }); 66 | } 67 | 68 | function recordResultForTask(dbctx, tcQueue, taskId, state, m) { 69 | // Get the task from TC 70 | debug("requesting task for " + taskId); 71 | var task = tcQueue.task(taskId); 72 | task.then(function(task) { 73 | debug("task: " + JSON.stringify(task)); 74 | var extra = task.extra.crater; 75 | 76 | if (extra.taskType == "crate-build") { 77 | 78 | var toolchain = extra.toolchain; 79 | var crateName = extra.crateName; 80 | var crateVers = extra.crateVers; 81 | 82 | assert(toolchain); 83 | assert(crateName); 84 | assert(crateVers); 85 | 86 | var status = "unknown"; 87 | if (state == "completed") { 88 | status = "success"; 89 | } else if (state == "failed") { 90 | status = "failure"; 91 | } else if (state == "exception") { 92 | status = "exception"; 93 | } else /*if (state == "pending" || state == "running")*/ { 94 | status = "unknown"; 95 | } 96 | 97 | var buildResult = { 98 | toolchain: toolchain, 99 | crateName: crateName, 100 | crateVers: crateVers, 101 | status: status, 102 | taskId: taskId 103 | }; 104 | console.log("adding build result: " + JSON.stringify(buildResult)); 105 | return db.addBuildResult(dbctx, buildResult); 106 | } else if (extra.taskType == "custom-build") { 107 | if (state == "completed") { 108 | debug("custom build success") 109 | var run = m.payload.status.runs.length - 1; 110 | var toolchain = util.parseToolchain(extra.toolchainGitSha); 111 | var url = "https://queue.taskcluster.net/v1/task/" + taskId + 112 | "/runs/" + run + "/artifacts/public/rustc-dev-x86_64-unknown-linux-gnu.tar.gz"; 113 | var custom = { 114 | toolchain: toolchain, 115 | url: url, 116 | taskId: taskId 117 | }; 118 | console.log("adding custom toolchain: " + JSON.stringify(custom)); 119 | return db.addCustomToolchain(dbctx, custom); 120 | } else if (state == "failure" || state == "exception") { 121 | console.log("custom toolchain build failed: " + taskId); 122 | } 123 | } else { 124 | console.log("unknown task type " + extra.taskType); 125 | return Promise.resolve(); 126 | } 127 | }).catch(function(e) { console.log("error: " + e) }); 128 | } 129 | 130 | main(); 131 | -------------------------------------------------------------------------------- /notes.md: -------------------------------------------------------------------------------- 1 | # Notes 2 | 3 | Additional crate data, including add date, can be dl'd from e.g. https://crates.io/api/v1/crates/toml/0.1.0 4 | 5 | An example of the paylod we need to create 6 | 7 | ``` 8 | { 9 | "image": "ubuntu:13.10", 10 | "command": [ 11 | "/bin/bash", 12 | "-c", 13 | "apt-get update && apt-get install curl -y && (curl -sf https://raw.githubusercontent.com/brson/taskcluster-crater/master/run-crater-task.sh | sh)" 14 | ], 15 | "env": { 16 | "CRATER_RUST_INSTALLER": "https://static.rust-lang.org/dist/rust-nightly-x86_64-unknown-linux-gnu.tar.gz", 17 | "CRATER_CRATE_FILE": "https://crates.io/api/v1/crates/toml/0.1.18/download" 18 | }, 19 | "maxRunTime": 600 20 | } 21 | ``` 22 | 23 | # taskcluster 24 | 25 | 13:16 <@jonasfj> brson, we don't have docs for setting up taskcluster... it's not something that you easily duplication (many moving parts involved, AMIs, S3 buckets, azure tables, azure queues, 2-3 heroku projects at 26 | minimum, DNS) 27 | 13:16 <@jonasfj> I think there is going to be some focus on setting up staging envs next quarter as the currently deployment is going into production... 28 | 13:18 <@jonasfj> what are you looking to do? You're most well come tome play around our TC deployment... that partly what it's for... we'll lock it more down in the future (using scopes) but for now it's pretty open if you 29 | have an @mozilla,com email. 30 | 13:20 <@jonasfj> s/duplication/do/ (wow, how did I write that...) 31 | 13:24 < brson> jonasfj: I am working on regressino testing for rust, and I'm ready to start figuring out how to schedule builds 32 | 13:25 <@jonasfj> hmm... lightsofapollo ^do we have story for scheduled tasks yet? I know we talked about it -- and I've seen chatter on IRC about this for gecko, but I'm not sure were the bootstrap lives... 33 | 13:26 <@jonasfj> I suspect it's just matter of adding something in the place that handles the github hooks... but jlal knows this integration stuff... 34 | 13:27 < brson> i don't know that i need github hooks, but i have some custom logic that wants to produce lots of builds 35 | 13:27 < brson> perhaps as a batch job every 24 hours 36 | 13:27 < brson> i don't know how to get started 37 | 13:27 <@jonasfj> oh... 38 | 13:28 <@jonasfj> brson, oh, so it's not running on TC yet? 39 | 13:29 <@jonasfj> basically we have end-points that can accept tasks or entire task-graphs... so if you create a heroku node or something that submits your jobs periodically you've solved the integration issue... (I don't 40 | think we have generic story for integration yet) 41 | 13:29 <@jonasfj> for running on TC... just make it run in docker.... such that input for "docker run" is image name, env vars and a simple command 42 | 13:30 <@jonasfj> then you can try it using our task create or (taskcluster-cli tools) 43 | 13:30 <@jonasfj> if you haven't, try out: https://tools.taskcluster.net/task-creator/ 44 | 13:31 <@jonasfj> (remember click login, and use an @mozilla.com email, and click authorse button) 45 | 13:31 < brson> jonasfj: ok, I will try to create tasks with task-creator. once i've done that and understand how tasks work, what do i use to start scripting those tasks? 46 | 13:32 <@jonasfj> you can create tasks using the API: http://docs.taskcluster.net/queue/api-docs/#createTask 47 | 13:32 <@jonasfj> or task graphs (if you have dependencies) using http://docs.taskcluster.net/scheduler/api-docs/#createTaskGraph 48 | 13:32 <@jonasfj> all of this is supported by taskcluster-client: https://github.com/taskcluster/taskcluster-client 49 | 13:33 <@jonasfj> note, docs for the task.payload property is present here: http://docs.taskcluster.net/docker-worker/ 50 | 13:33 < brson> jonasfj: great, thank you. is there any risk of me breaking things by submitting tasks? 51 | 13:34 <@jonasfj> brson, not really.... I'll buy you a beer if you break things by accident... :) 52 | 53 | 15:38 <@jonasfj> brson, you can declare artifafacts to be exported... 54 | 15:39 <@jonasfj> the log should already be exported as the public/logs/terminal.log artfiact 55 | 15:39 <@jonasfj> (note: artifacts starting with public/ are public and GETing them requires no credentials) 56 | 15:40 <@jonasfj> the easiest way to export artifacts... is to just declare the you want the contents of a folder uploaded: 57 | 15:40 < brson> jonasfj: ok. seems like when i use taskcluster-client to schedule a task i'll probably get a taskid back i can store 58 | 15:40 <@jonasfj> task.payload. "artifacts": { 59 | 15:40 <@jonasfj> "public/test-logs": { 60 | 15:40 <@jonasfj> "type": "directory", 61 | 15:40 <@jonasfj> "path": "/home/tester/git_checkout/test-logs", 62 | 15:40 <@jonasfj> "expires": "2016-02-28T03:37:46.430Z" 63 | 15:40 <@jonasfj> } 64 | 15:40 <@jonasfj> } 65 | 15:40 <@jonasfj> brson, yes... 66 | 15:40 < brson> ooh, thanks 67 | 15:40 <@jonasfj> brson, you can also do smart things :) 68 | 15:41 <@jonasfj> like listen on pulse with a custom route... 69 | 15:41 <@lightsofapollo> brson: from what I can see you probably have everything you need by default unless you need to rely on exit codes other then 0/1 ? 70 | 15:41 <@jonasfj> listening for task-completed for example: 71 | 15:41 <@jonasfj> http://docs.taskcluster.net/queue/exchanges/#taskCompleted 72 | 73 | 74 | Get credentials at https://auth.taskcluster.net/ 75 | 76 | 10:47 <@lightsofapollo> brson: nice... If you want stuff to move faster try using b2gtest or gaia worker type 77 | 10:48 < brson> lightsofapollo: because they already have workers running? 78 | 10:48 <@lightsofapollo> brson: yeah 79 | 10:48 <@lightsofapollo> we only really use cli for internal testing so usually nothing is running 80 | 81 | 12:17 <@jonasfj> brson, did a quick a example illustrating use of custom routes, as opposed to storing the taskIds: https://gist.github.com/jonasfj/c7648158a561eb95c426 82 | 12:17 <@jonasfj> in case you're interested. Basically we allow you add custom routing keys to tasks... 83 | 12:17 <@jonasfj> which will be used when we publish messages about the task. 84 | 12:18 <@jonasfj> so you just listen for your messages on task-completed exchange with a routingkey that matches your custom one. 85 | 12:19 <@jonasfj> (right now nothing is locked down, everybody gets '*' scope, but custom routing keys are scopes so in future you can make an exclusive claim to one) 86 | 87 | 88 | 13:13 <@jonasfj> brson, reposting from when it off yester day.. 89 | 13:13 <@jonasfj> brson, for the record (as you're nolonger around) no... task.extra is not available in pulse messages (pulse messages documented here: http://docs.taskcluster.net/queue/exchanges/) 90 | 13:13 <@jonasfj> brson, but you can load the task definition from the queue, using queue.getTask(taskId) and then read the extra part back from there... 91 | 13:13 <@jonasfj> (we include as little as possible in the pulse messages to keep pulse fast) 92 | 13:13 <@jonasfj> as a hack one can insert an extra key in the custom routing key... but this is very limited.. and has character restrictings on "."; it's mostly useful if you're injecting a revision number. 93 | 13:13 <@jonasfj> if you're looking at injecting repository + revision task.extra is a better place to store it... or task.tags (but that strictly strings) 94 | 95 | dl https://crates.io/api/v1/crates/toml/0.1.0 to get metadata 96 | 97 | See details about the cratertest worker type at http://aws-provisioner.taskcluster.net/ 98 | 99 | # Weekly Report 100 | 101 | Date: 2015/03/09 102 | 103 | The most recent stable release is XXX. 104 | The most recent beta release is XXX. 105 | The most recent nightly release is XXX. 106 | 107 | There are currently X root regressions from stable to beta. 108 | There are currently X root regressions from beta to nightly. 109 | 110 | There are currently X regressions from stable to beta. 111 | There are currently X regressions from beta to nightly. 112 | 113 | From stable to beta: 114 | * X crates tested: X working / X not working / X regressed / X fixed 115 | * X crates not tested: X old / X exception / X no-data 116 | 117 | From beta to nightly: 118 | * X crates tested: X working / X not working / X regressed / X fixed 119 | * X crates not tested: X old / X exception / X no-data 120 | 121 | ## Beta root regressions, by popularity: 122 | 123 | * [toml-0.1.18](link) 124 | 125 | ## Nightly root regressions, by popularity: 126 | 127 | * [toml-0.1.18](link) 128 | 129 | ## Beta non-root regressions, by popularity: 130 | 131 | * [toml-0.1.18](link) 132 | 133 | ## Nightly non-root regressions, by popularity: 134 | 135 | * [toml-0.1.18](link) 136 | 137 | # Comparison Report 138 | 139 | Like weekly but for two arbitrary toolchains. 140 | 141 | # Score Board 142 | 143 | Show the top X packages in order and their build status, for all channels 144 | 145 | # Single-toolchain report 146 | 147 | * Root breakage, by popularity 148 | 149 | # Next Steps 150 | 151 | * store custom builds on s3 152 | * build crates against custom rust 153 | * sort reports by crate popularity 154 | * add scoreboard report 155 | 156 | # info about worker types 157 | 158 | 12:19 < jonasfj> brson, cratertest is currently an r3.xlarge configurated with capacity = 5, which means it'll run up to 5 tasks in parallel... Reducing capacity would give you more ram...You can also move to r3.2xlarge :) 159 | 12:21 < jonasfj> as long as you moving within the r3 series I think it'll work.. c3 series is probably good too... but c4 might require some worker-level changes to how we mount disks... 160 | 12:21 < jonasfj> r3.largeI2I15.25I1 x 32 161 | 12:21 < jonasfj> r3.xlargeI4I30.5I1 x 80 162 | 12:21 < jonasfj> r3.2xlargeI8I61I1 x 160 163 | 12:21 < jonasfj> r3.4xlargeI16I122I1 x 320 164 | 12:21 < jonasfj> r3.8xlarge 165 | 12:21 < jonasfj> 32 166 | 12:21 < jonasfj> 244 167 | 12:21 < jonasfj> 2 x 320 168 | 12:22 < jonasfj> ahh, that did work well... 169 | 12:22 < jonasfj> my point was moving to r3.2xlarge and reducing capacity to 2 should give you 30G ram per container 170 | 12:23 < jonasfj> (we currently don't enforce fair sharing between containers) 171 | 12:26 < jonasfj> all of this can be done on aws-provisioner.taskcluster.net 172 | 12:28 < jonasfj> brson, I created rustbuild workertype (better name later); it's an r3.2xlarge (60G ram 8 cores) capacity one so will only run one container at the time... 173 | 12:28 < jonasfj> (also limited to no more than one instance... but you can change that if you want more... small limits are good when testing. 174 | 12:29 < jonasfj> anyways, I recreated your tasks for testing without the route (so it won't interfere with your listener): 175 | 12:29 < jonasfj> brson, http://docs.taskcluster.net/tools/task-inspector/#oezansp3QYmx3HQfRCjXPA 176 | 12:29 < jonasfj> ahh, submitted it with wrong workerType, my bad :) 177 | 12:29 -!- kang [kang@moz-t24d28.ca.comcast.net] has quit [Quit: WeeChat 1.1.1] 178 | 12:32 < jonasfj> brson, meant: http://docs.taskcluster.net/tools/task-inspector/#tISmmRsyRBewU_tEIsewww 179 | 12:32 < jonasfj> it'll probably start running soon, then we'll see if it 60g is enough ram 180 | 12:35 < jonasfj> note, we often use different workertypes for tests and builds. Builds easily saturates CPU and ram, so running multiple builds in parallel is usually pointless.. 181 | 12:37 < jonasfj> tests on the other hand are often slow... not running in parallel... They are more linear time "intensive", than cpu intensive... so running them in parallel pays off.. 182 | 183 | # info about docker 184 | 185 | 13:25 < jonasfj> it's pretty trivial... and fast I always test locally... 186 | 13:25 < jonasfj> not in our config... but we could have missed something... 187 | 13:26 < jonasfj> docker run -ti --rm -e ENV_VAR_FROM_TASK=VALUE ubuntu:13.10 188 | 13:26 < jonasfj> -ti is for interactive terminal output 189 | 13:26 < garndt> wonder how bad my virtualbox vm will die if I try it 190 | 13:27 < jonasfj> --rm deletes the container when you disconnect... 191 | 192 | # REST 193 | 194 | 14:02 < brson> besides not being able to build rust, i've got the major functionality i need scaped together, but it's just a bunch of modules and cli tools. now i want to put it behind a HTTP REST interface. what's the easy 195 | way to do that in node? 196 | 14:04 < jonasfj> what do you mean? do you want to setup to github hook... or offer a REST API for people to hit? Who is the consumer... 197 | 14:08 < brson> jonasfj: I want the functionality of my tool to be exposed as a REST API so I can add a web frontend to it 198 | 14:11 -!- bhearsum [bhearsum@moz-n6cgoa.dsl.bell.ca] has quit [Quit: mooooonday] 199 | 14:11 <@lightsofapollo> brson: hapi / express are decent node frameworks 200 | 14:11 <@lightsofapollo> http frameworks 201 | 14:12 * lightsofapollo would have loved to see this as a rust only thing ... someday 202 | 203 | # a working rust build task 204 | 205 | https://gist.github.com/jonasfj/91526a796c9f1cdc5c87 206 | http://docs.taskcluster.net/tools/task-inspector/#IDs5uUXuS9SGuqvYfaCQ9w/0 207 | 208 | # getting live logs 209 | 210 | http://docs.taskcluster.net/tools/task-inspector/#xW41f4pVTtm854sp_uX1nw/0 211 | 212 | 18:47 < brson> while i've got your attention, what url will the uploaded artifacts be available at when they do upload? 213 | 18:49 < garndt> https://queue.taskcluster.net/v1/task//runs//artifacts/ 214 | 18:50 < garndt> usually the path in the task definition is something like "public/some/path/some_file" or similar 215 | 216 | # switching aws provisioner 217 | 218 | 11:25 < lightsofapollo> brson: ah have we had you switch to new provisioner yet ? 219 | 11:26 < lightsofapollo> brson: pmoore|sick wrote the go bindings (he should be alive again hopefully by monday :)) 220 | 11:26 < lightsofapollo> bhearsum|mtg: ah nice :) I can't wait until we get all of this in the tree ... it's going to be so good 221 | 11:26 -!- kgrandon [Adium@moz-tqj7vj.2rkg.9kg1.0101.2620.IP] has quit [Quit: Leaving.] 222 | 11:32 < brson> lightsofapollo: no, i haven't switched provisioners 223 | 11:32 < lightsofapollo> brson: okay- we are going to keep old one around but the new one now supports us-east-1/us-west-1/us-west-2 224 | 11:33 < lightsofapollo> (i.e. your not going to run out of nodes anytime soon...) 225 | 11:40 < brson> lightsofapollo: how do i switch? 226 | 11:41 < lightsofapollo> brson: change "aws-provisioner" to "aws-provisioner-v1" 227 | 11:41 < lightsofapollo> brson: then the new home for the UI is https://tools.taskcluster.net/aws-provisioner/ 228 | 11:42 < lightsofapollo> brson: bunch of new features... so you can configure multiple instances use ondemand (in addition to spot) and we are expanding to new regions (frankfurt is next...) 229 | -------------------------------------------------------------------------------- /package.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "taskcluster-crater", 3 | "version": "0.0.1", 4 | "description": "A collection of tools to test Rust Crates", 5 | "main": "index.js", 6 | "scripts": { 7 | "test": "mocha --opts mocha.opts" 8 | }, 9 | "repository": { 10 | "type": "git", 11 | "url": "https://github.com/jhford/taskcluster-crater.git" 12 | }, 13 | "keywords": [ 14 | "rust", 15 | "ci", 16 | "crate", 17 | "taskcluster" 18 | ], 19 | "author": "John Ford ", 20 | "license": "ISC", 21 | "bugs": { 22 | "url": "https://github.com/jhford/taskcluster-crater/issues" 23 | }, 24 | "homepage": "https://github.com/jhford/taskcluster-crater", 25 | "dependencies": { 26 | "debug": "^2.1.1", 27 | "promise": "^6.1.0", 28 | "walkdir": "^0.0.7", 29 | "graceful-fs": "^3.0.5", 30 | "taskcluster-client": "^0.21.4", 31 | "slugid": "^1.0.3", 32 | "pg": "^4.3.0", 33 | "async": "^0.9.0", 34 | "express": "^4.12.3", 35 | "sleep": "^3.0.0" 36 | } 37 | } 38 | -------------------------------------------------------------------------------- /print-report.js: -------------------------------------------------------------------------------- 1 | 'use strict'; 2 | 3 | var debug = require('debug')(__filename.slice(__dirname.length + 1)); 4 | var reports = require('./reports'); 5 | var util = require('./crater-util'); 6 | var dist = require('./rust-dist'); 7 | var crateIndex = require('./crate-index'); 8 | var db = require('./crater-db'); 9 | var fs = require('fs'); 10 | 11 | function main() { 12 | var reportSpec = getReportSpecFromArgs(); 13 | if (!reportSpec) { 14 | console.log("can't parse report spec"); 15 | process.exit(1); 16 | } 17 | 18 | var config = util.loadDefaultConfig(); 19 | crateIndex.updateCaches(config).then(function() { 20 | printReport(config, reportSpec); 21 | }); 22 | } 23 | 24 | function printReport(config, reportSpec) { 25 | if (reportSpec.type == "current") { 26 | var date = reportSpec.date; 27 | reports.createCurrentReport(date, config).then(function(report) { 28 | console.log("# Current Report"); 29 | console.log(); 30 | console.log("* current stable is " + report.stable); 31 | console.log("* current beta is " + report.beta); 32 | console.log("* current nightly is " + report.nightly); 33 | }).done(); 34 | } else if (reportSpec.type == "weekly") { 35 | var date = reportSpec.date; 36 | db.connect(config).then(function(dbctx) { 37 | var p = reports.createWeeklyReport(date, dbctx, config); 38 | return p.then(function(report) { 39 | console.log("# Weekly Report"); 40 | console.log(); 41 | console.log("Date: " + report.date); 42 | console.log(); 43 | console.log("## Current releases"); 44 | console.log(); 45 | console.log("* The most recent stable release is " + report.currentReport.stable + "."); 46 | console.log("* The most recent beta release is " + report.currentReport.beta + "."); 47 | console.log("* The most recent nightly release is " + report.currentReport.nightly + "."); 48 | console.log(); 49 | console.log("## Coverage"); 50 | console.log(); 51 | console.log("From stable to beta:"); 52 | console.log("* " + report.beta.statuses.length + " crates tested: " + 53 | report.beta.statusSummary.working + " working / " + 54 | report.beta.statusSummary.notWorking + " not working / " + 55 | report.beta.statusSummary.regressed + " regressed / " + 56 | report.beta.statusSummary.fixed + " fixed / " + 57 | report.beta.statusSummary.unknown + " unknown."); 58 | console.log(); 59 | console.log("From beta to nightly:"); 60 | console.log("* " + report.nightly.statuses.length + " crates tested: " + 61 | report.nightly.statusSummary.working + " working / " + 62 | report.nightly.statusSummary.broken + " broken / " + 63 | report.nightly.statusSummary.fixed + " fixed / " + 64 | report.nightly.statusSummary.unknown + " unknown."); 65 | console.log(); 66 | console.log("## Regressions"); 67 | console.log(); 68 | console.log("* There are currently " + report.beta.rootRegressions.length + 69 | " root regressions from stable to beta."); 70 | console.log("* There are currently " + report.nightly.rootRegressions.length + 71 | " root regressions from beta to nightly."); 72 | console.log("* There are currently " + report.beta.regressions.length + 73 | " regressions from stable to beta."); 74 | console.log("* There are currently " + report.nightly.regressions.length + 75 | " regressions from beta to nightly."); 76 | console.log(); 77 | console.log("## Beta root regressions, sorted by rank:"); 78 | printCrateList(report.beta.rootRegressions); 79 | console.log("## Nightly root regressions, sorted by rank:"); 80 | printCrateList(report.nightly.rootRegressions); 81 | console.log("## Beta non-root regressions, sorted by rank:"); 82 | printCrateList(report.beta.regressions); 83 | console.log("## Nightly non-root regressions, sorted by rank:"); 84 | printCrateList(report.nightly.regressions); 85 | console.log("## Beta broken, sorted by rank:"); 86 | printCrateList(report.beta.broken); 87 | console.log("## Nightly broken, sorted by rank:"); 88 | printCrateList(report.nightly.broken); 89 | console.log("## Beta fixed, sorted by rank:"); 90 | printCrateList(report.beta.fixed); 91 | console.log("## Nightly fixed, sorted by rank:"); 92 | printCrateList(report.nightly.fixed); 93 | }).then(function() { 94 | return db.disconnect(dbctx); 95 | }); 96 | }).done(); 97 | } else if (reportSpec.type == "comparison") { 98 | var date = reportSpec.date; 99 | db.connect(config).then(function(dbctx) { 100 | var p = reports.createComparisonReport(reportSpec.fromToolchain, reportSpec.toToolchain, 101 | dbctx, config); 102 | return p.then(function(report) { 103 | var from = util.toolchainToString(report.fromToolchain); 104 | var to = util.toolchainToString(report.toToolchain); 105 | console.log("# Regression report " + from + " vs. " + to);; 106 | console.log(); 107 | console.log("* From: " + from); 108 | console.log("* To: " + to); 109 | console.log(); 110 | console.log("## Coverage"); 111 | console.log(); 112 | console.log("* " + report.statuses.length + " crates tested: " + 113 | report.statusSummary.working + " working / " + 114 | report.statusSummary.broken + " broken / " + 115 | report.statusSummary.regressed + " regressed / " + 116 | report.statusSummary.fixed + " fixed / " + 117 | report.statusSummary.unknown + " unknown."); 118 | console.log(); 119 | console.log("## Regressions"); 120 | console.log(); 121 | console.log("* There are " + report.rootRegressions.length + " root regressions"); 122 | console.log("* There are " + report.regressions.length + " regressions"); 123 | console.log(); 124 | console.log("## Root regressions, sorted by rank:"); 125 | printCrateList(report.rootRegressions); 126 | console.log("## Non-root regressions, sorted by rank:"); 127 | printCrateList(report.nonRootRegressions); 128 | console.log("## Broken, sorted by rank:"); 129 | printCrateList(report.broken); 130 | console.log("## Fixed, sorted by rank:"); 131 | printCrateList(report.fixed); 132 | console.log("## Working, sorted by rank:"); 133 | printCrateList(report.working); 134 | }).then(function() { 135 | return db.disconnect(dbctx); 136 | }); 137 | }).done(); 138 | } else if (reportSpec.type == "popularity") { 139 | reports.createPopularityReport(config).then(function(report) { 140 | console.log("# Popularity report"); 141 | console.log(""); 142 | report.forEach(function(r) { 143 | console.log("* " + r.pop + " [" + r.crateName + "](" + r.registryUrl + ")"); 144 | }); 145 | }); 146 | } else if (reportSpec.type == "toolchain") { 147 | db.connect(config).then(function(dbctx) { 148 | return reports.createToolchainReport(reportSpec.toolchain, dbctx, config).then(function(report) { 149 | console.log("# Toolchain report for " + util.toolchainToString(report.toolchain)); 150 | console.log(""); 151 | console.log("* " + report.successes.length + " successes / " + report.failures.length + " failures"); 152 | console.log("* " + report.rootFailures.length + " root failures / " + report.nonRootFailures.length + " non-root failures"); 153 | console.log(""); 154 | console.log("## Root failures, sorted by rank"); 155 | console.log(""); 156 | report.rootFailures.forEach(function(r) { 157 | var s = "* [" + r.crateName + "-" + r.crateVers + "](" + r.inspectorLink + ")"; 158 | console.log(s); 159 | }); 160 | console.log(""); 161 | console.log("## Non-root failures, sorted by rank"); 162 | console.log(""); 163 | report.nonRootFailures.forEach(function(r) { 164 | var s = "* [" + r.crateName + "-" + r.crateVers + "](" + r.inspectorLink + ")"; 165 | console.log(s); 166 | }); 167 | console.log(""); 168 | console.log("## Successes, sorted by rank"); 169 | console.log(""); 170 | report.successes.forEach(function(r) { 171 | var s = "* [" + r.crateName + "-" + r.crateVers + "](" + r.inspectorLink + ")"; 172 | console.log(s); 173 | }); 174 | console.log(""); 175 | 176 | return db.disconnect(dbctx); 177 | }); 178 | }).done(); 179 | } else { 180 | console.log("unknown report type"); 181 | } 182 | } 183 | 184 | function printCrateList(statuses) { 185 | console.log(); 186 | statuses.forEach(function(reg) { 187 | var fromLink = reg.from.inspectorLink; 188 | var toLink = reg.to.inspectorLink; 189 | var s = "* [" + reg.crateName + "-" + reg.crateVers + "](" + reg.registryUrl + ") " + 190 | "([before](" + fromLink + ")) " + 191 | "([after](" + toLink + "))"; 192 | console.log(s); 193 | }); 194 | console.log(); 195 | } 196 | 197 | function getReportSpecFromArgs() { 198 | if (process.argv[2] == "current") { 199 | return { 200 | type: "current", 201 | date: process.argv[3] || util.rustDate(new Date(Date.now())) 202 | }; 203 | } else if (process.argv[2] == "weekly") { 204 | return { 205 | type: "weekly", 206 | date: process.argv[3] || util.rustDate(new Date(Date.now())) 207 | }; 208 | } else if (process.argv[2] == "comparison") { 209 | if (!process.argv[3] || !process.argv[4]) { 210 | return null; 211 | } 212 | return { 213 | type: "comparison", 214 | fromToolchain: util.parseToolchain(process.argv[3]), 215 | toToolchain: util.parseToolchain(process.argv[4]) 216 | }; 217 | } else if (process.argv[2] == "popularity") { 218 | return { 219 | type: "popularity" 220 | }; 221 | } else if (process.argv[2] == "toolchain") { 222 | return { 223 | type: "toolchain", 224 | toolchain: util.parseToolchain(process.argv[3]) 225 | }; 226 | } else { 227 | return null; 228 | } 229 | } 230 | 231 | main(); 232 | -------------------------------------------------------------------------------- /reports.js: -------------------------------------------------------------------------------- 1 | 'use strict'; 2 | 3 | var debug = require('debug')(__filename.slice(__dirname.length + 1)); 4 | var util = require('./crater-util'); 5 | var crateIndex = require('./crate-index'); 6 | var Promise = require('promise'); 7 | var db = require('./crater-db'); 8 | var assert = require('assert'); 9 | var dist = require('./rust-dist'); 10 | 11 | function createToolchainReport(toolchain, dbctx, config) { 12 | return db.getResults(dbctx, toolchain).then(function(results) { 13 | return results.map(function(result) { 14 | result.registryUrl = makeRegistryUrl(result.crateName); 15 | result.inspectorLink = makeInspectorLink(result.taskId); 16 | return result; 17 | }); 18 | }).then(function(results) { 19 | return sortByPopularity(results, config); 20 | }).then(function(results) { 21 | // Partition into successful results and failful results 22 | var successes = []; 23 | var failures = []; 24 | results.forEach(function(result) { 25 | if (result.status == "success") { 26 | successes.push(result); 27 | } else if (result.status == "failure") { 28 | failures.push(result); 29 | } 30 | }); 31 | 32 | return getRootFailures(failures, config).then(function(rootFailures) { 33 | var nonRootFailures = getNonRootFailures(failures, rootFailures); 34 | return { 35 | toolchain: toolchain, 36 | successes: successes, 37 | failures: failures, 38 | rootFailures: rootFailures, 39 | nonRootFailures: nonRootFailures 40 | }; 41 | }); 42 | }); 43 | } 44 | 45 | function createComparisonReport(fromToolchain, toToolchain, dbctx, config) { 46 | var statuses = calculateStatuses(dbctx, fromToolchain, toToolchain); 47 | return statuses.then(function(statuses) { 48 | return sortByPopularity(statuses, config); 49 | }).then(function(statuses) { 50 | var statusSummary = calculateStatusSummary(statuses); 51 | var regressions = extractWithStatus(statuses, "regressed"); 52 | var rootRegressions = getRootFailures(regressions, config); 53 | return rootRegressions.then(function(rootRegressions) { 54 | var nonRootRegressions = getNonRootFailures(regressions, rootRegressions); 55 | 56 | var working = extractWithStatus(statuses, "working"); 57 | var broken = extractWithStatus(statuses, "broken"); 58 | var fixed = extractWithStatus(statuses, "fixed"); 59 | 60 | return { 61 | fromToolchain: fromToolchain, 62 | toToolchain: toToolchain, 63 | statuses: statuses, 64 | statusSummary: statusSummary, 65 | regressions: regressions, 66 | rootRegressions: rootRegressions, 67 | nonRootRegressions: nonRootRegressions, 68 | working: working, 69 | broken: broken, 70 | fixed: fixed 71 | }; 72 | }); 73 | }); 74 | } 75 | 76 | /** 77 | * Returns a promise of the data for a 'weekly report'. 78 | */ 79 | function createWeeklyReport(date, dbctx, config) { 80 | return createCurrentReport(date, config).then(function(currentReport) { 81 | var stableToolchain = null; 82 | if (currentReport.stable) { 83 | stableToolchain = { channel: "stable", archiveDate: currentReport.stable }; 84 | } 85 | var betaToolchain = null; 86 | if (currentReport.beta) { 87 | betaToolchain = { channel: "beta", archiveDate: currentReport.beta }; 88 | } 89 | var nightlyToolchain = null; 90 | if (currentReport.nightly) { 91 | nightlyToolchain = { channel: "nightly", archiveDate: currentReport.nightly }; 92 | } 93 | 94 | var betaReport = createComparisonReport(stableToolchain, betaToolchain, dbctx, config); 95 | return betaReport.then(function(betaReport) { 96 | var nightlyReport = createComparisonReport(betaToolchain, nightlyToolchain, dbctx, config); 97 | return nightlyReport.then(function(nightlyReport) { 98 | return { 99 | date: date, 100 | currentReport: currentReport, 101 | beta: betaReport, 102 | nightly: nightlyReport 103 | }; 104 | }); 105 | }); 106 | }); 107 | } 108 | 109 | /** 110 | * Returns promise of array of `{ crateName, crateVers, status }`, 111 | * where `status` is either 'working', 'broken', 'regressed', 112 | * 'fixed'. 113 | */ 114 | function calculateStatuses(dbctx, fromToolchain, toToolchain) { 115 | 116 | if (fromToolchain == null || toToolchain == null) { 117 | return new Promise(function(resolve, reject) { resolve([]); }); 118 | } 119 | 120 | return db.getResultPairs(dbctx, fromToolchain, toToolchain).then(function(buildResults) { 121 | return buildResults.map(function(buildResult) { 122 | var status = null; 123 | if (buildResult.from.status == "success" && buildResult.to.status == "success") { 124 | status = "working"; 125 | } else if (buildResult.from.status == "failure" && buildResult.to.status == "failure") { 126 | status = "broken"; 127 | } else if (buildResult.from.status == "success" && buildResult.to.status == "failure") { 128 | status = "regressed"; 129 | } else if (buildResult.from.status == "failure" && buildResult.to.status == "success") { 130 | status = "fixed"; 131 | } else { 132 | status = "unknown"; 133 | } 134 | 135 | // Just modify the intermediate result 136 | buildResult.status = status; 137 | buildResult.from.inspectorLink = makeInspectorLink(buildResult.from.taskId); 138 | buildResult.to.inspectorLink = makeInspectorLink(buildResult.to.taskId); 139 | buildResult.registryUrl = makeRegistryUrl(buildResult.crateName); 140 | 141 | return buildResult; 142 | }); 143 | }); 144 | } 145 | 146 | function sortByPopularity(statuses, config) { 147 | return crateIndex.loadCrates(config).then(function(crates) { 148 | var popMap = crateIndex.getPopularityMap(crates); 149 | 150 | var sorted = statuses.slice(); 151 | sorted.sort(function(a, b) { 152 | var aPop = popMap[a.crateName]; 153 | var bPop = popMap[b.crateName]; 154 | if (aPop == bPop) { return 0; } 155 | if (aPop < bPop) { return 1; } 156 | if (aPop > bPop) { return -1; } 157 | }); 158 | 159 | return sorted; 160 | }); 161 | } 162 | 163 | function calculateStatusSummary(statuses) { 164 | var working = 0; 165 | var broken = 0; 166 | var regressed = 0; 167 | var fixed = 0; 168 | var unknown = 0; 169 | statuses.forEach(function(status) { 170 | if (status.status == "working") { 171 | working += 1; 172 | } else if (status.status == "broken") { 173 | broken += 1; 174 | } else if (status.status == "regressed") { 175 | regressed += 1; 176 | } else if (status.status == "fixed") { 177 | fixed += 1; 178 | } else { 179 | assert(status.status == "unknown"); 180 | unknown += 1; 181 | } 182 | }); 183 | 184 | return { 185 | working: working, 186 | broken: broken, 187 | regressed: regressed, 188 | fixed: fixed, 189 | unknown: unknown 190 | }; 191 | } 192 | 193 | function extractWithStatus(statuses, needed) { 194 | var result = []; 195 | statuses.forEach(function(status) { 196 | if (status.status == needed) { 197 | result.push(status); 198 | } 199 | }); 200 | return result; 201 | } 202 | 203 | function getRootFailures(regressions, config) { 204 | var regressionMap = {}; 205 | regressions.forEach(function(r) { 206 | regressionMap[r.crateName] = r; 207 | }); 208 | 209 | return crateIndex.loadCrates(config).then(function(crates) { 210 | var dag = crateIndex.getDag(crates); 211 | var independent = []; 212 | regressions.forEach(function(reg) { 213 | var isIndependent = true; 214 | var depStack = dag[reg.crateName]; 215 | if (depStack == null) { 216 | // No info about this crate? Happens in the test suite. 217 | debug("no deps for " + reg.crateName); 218 | } 219 | while (depStack && depStack.length != 0 && isIndependent) { 220 | 221 | var nextDep = depStack.pop(); 222 | if (regressionMap[nextDep]) { 223 | debug(reg.crateName + " depends on regressed " + nextDep); 224 | isIndependent = false; 225 | } 226 | 227 | if (dag[nextDep]) { 228 | depStack.concat(dag[nextDep]); 229 | } 230 | } 231 | if (isIndependent) { 232 | debug(reg.crateName + " is an independent regression"); 233 | independent.push(reg); 234 | } 235 | }); 236 | return independent; 237 | }); 238 | } 239 | 240 | function getNonRootFailures(regs, rootRegs) { 241 | var rootRegMap = {}; 242 | rootRegs.forEach(function(r) { 243 | rootRegMap[r.crateName] = r; 244 | }); 245 | 246 | var dependent = [] 247 | regs.forEach(function(reg) { 248 | if (!rootRegMap[reg.crateName]) { 249 | dependent.push(reg); 250 | } 251 | }); 252 | 253 | return dependent; 254 | } 255 | 256 | /** 257 | * Returns a promise of a report on the current nightly/beta/stable revisions. 258 | */ 259 | function createCurrentReport(date, config) { 260 | return dist.getAvailableToolchains(config).then(function(toolchains) { 261 | var currentNightlyDate = null; 262 | toolchains.nightly.forEach(function(toolchainDate) { 263 | if (toolchainDate <= date) { 264 | currentNightlyDate = toolchainDate; 265 | } 266 | }); 267 | var currentBetaDate = null; 268 | toolchains.beta.forEach(function(toolchainDate) { 269 | if (toolchainDate <= date) { 270 | currentBetaDate = toolchainDate; 271 | } 272 | }); 273 | var currentStableDate = null; 274 | toolchains.stable.forEach(function(toolchainDate) { 275 | if (toolchainDate <= date) { 276 | currentStableDate = toolchainDate; 277 | } 278 | }); 279 | 280 | return { 281 | nightly: currentNightlyDate, 282 | beta: currentBetaDate, 283 | stable: currentStableDate 284 | }; 285 | 286 | }); 287 | } 288 | 289 | function createPopularityReport(config) { 290 | return crateIndex.loadCrates(config).then(function(crates) { 291 | var popMap = crateIndex.getPopularityMap(crates); 292 | 293 | return crates.map(function(crate) { 294 | return { 295 | crateName: crate.name, 296 | registryUrl: makeRegistryUrl(crate.name), 297 | pop: popMap[crate.name] 298 | }; 299 | }); 300 | }).then(function(crates) { 301 | var map = {} 302 | return crates.filter(function(crate) { 303 | if (map[crate.crateName]) { 304 | return false; 305 | } else { 306 | map[crate.crateName] = crate; 307 | return true; 308 | } 309 | }); 310 | }).then(function(crates) { 311 | return sortByPopularity(crates, config); 312 | }); 313 | } 314 | 315 | function makeRegistryUrl(name) { 316 | return "https://crates.io/crates/" + name; 317 | } 318 | 319 | function makeInspectorLink(taskId) { 320 | var inspectorRoot = "https://tools.taskcluster.net/task-inspector/#"; 321 | return inspectorRoot + taskId; 322 | } 323 | 324 | function createScoreboardReport(toolchain, config) { 325 | return 326 | } 327 | 328 | exports.createWeeklyReport = createWeeklyReport; 329 | exports.createCurrentReport = createCurrentReport; 330 | exports.createComparisonReport = createComparisonReport; 331 | exports.createPopularityReport = createPopularityReport; 332 | exports.createScoreboardReport = createScoreboardReport; 333 | exports.createToolchainReport = createToolchainReport; 334 | -------------------------------------------------------------------------------- /rs/Cargo.lock: -------------------------------------------------------------------------------- 1 | [root] 2 | name = "crater" 3 | version = "0.1.0" 4 | dependencies = [ 5 | "crater-api 0.1.0", 6 | "crater-bus 0.1.0", 7 | "crater-db 0.1.0", 8 | "crater-engine 0.1.0", 9 | "env_logger 0.3.2 (registry+https://github.com/rust-lang/crates.io-index)", 10 | "hyper 0.7.1 (registry+https://github.com/rust-lang/crates.io-index)", 11 | "iron 0.2.6 (registry+https://github.com/rust-lang/crates.io-index)", 12 | "log 0.3.4 (registry+https://github.com/rust-lang/crates.io-index)", 13 | "mount 0.0.9 (registry+https://github.com/rust-lang/crates.io-index)", 14 | "router 0.1.0 (registry+https://github.com/rust-lang/crates.io-index)", 15 | "rustc-serialize 0.3.16 (registry+https://github.com/rust-lang/crates.io-index)", 16 | "taskcluster 0.1.0", 17 | ] 18 | 19 | [[package]] 20 | name = "advapi32-sys" 21 | version = "0.1.2" 22 | source = "registry+https://github.com/rust-lang/crates.io-index" 23 | dependencies = [ 24 | "winapi 0.2.5 (registry+https://github.com/rust-lang/crates.io-index)", 25 | "winapi-build 0.1.1 (registry+https://github.com/rust-lang/crates.io-index)", 26 | ] 27 | 28 | [[package]] 29 | name = "aho-corasick" 30 | version = "0.4.0" 31 | source = "registry+https://github.com/rust-lang/crates.io-index" 32 | dependencies = [ 33 | "memchr 0.1.7 (registry+https://github.com/rust-lang/crates.io-index)", 34 | ] 35 | 36 | [[package]] 37 | name = "bitflags" 38 | version = "0.3.3" 39 | source = "registry+https://github.com/rust-lang/crates.io-index" 40 | 41 | [[package]] 42 | name = "bufstream" 43 | version = "0.1.1" 44 | source = "registry+https://github.com/rust-lang/crates.io-index" 45 | 46 | [[package]] 47 | name = "byteorder" 48 | version = "0.4.2" 49 | source = "registry+https://github.com/rust-lang/crates.io-index" 50 | 51 | [[package]] 52 | name = "cfg-if" 53 | version = "0.1.0" 54 | source = "registry+https://github.com/rust-lang/crates.io-index" 55 | 56 | [[package]] 57 | name = "conduit-mime-types" 58 | version = "0.7.3" 59 | source = "registry+https://github.com/rust-lang/crates.io-index" 60 | dependencies = [ 61 | "rustc-serialize 0.3.16 (registry+https://github.com/rust-lang/crates.io-index)", 62 | ] 63 | 64 | [[package]] 65 | name = "cookie" 66 | version = "0.2.2" 67 | source = "registry+https://github.com/rust-lang/crates.io-index" 68 | dependencies = [ 69 | "openssl 0.7.4 (registry+https://github.com/rust-lang/crates.io-index)", 70 | "rustc-serialize 0.3.16 (registry+https://github.com/rust-lang/crates.io-index)", 71 | "time 0.1.34 (registry+https://github.com/rust-lang/crates.io-index)", 72 | "url 0.5.2 (registry+https://github.com/rust-lang/crates.io-index)", 73 | ] 74 | 75 | [[package]] 76 | name = "crater-api" 77 | version = "0.1.0" 78 | dependencies = [ 79 | "rustc-serialize 0.3.16 (registry+https://github.com/rust-lang/crates.io-index)", 80 | ] 81 | 82 | [[package]] 83 | name = "crater-bus" 84 | version = "0.1.0" 85 | dependencies = [ 86 | "rustc-serialize 0.3.16 (registry+https://github.com/rust-lang/crates.io-index)", 87 | ] 88 | 89 | [[package]] 90 | name = "crater-db" 91 | version = "0.1.0" 92 | dependencies = [ 93 | "postgres 0.10.2 (registry+https://github.com/rust-lang/crates.io-index)", 94 | "rustc-serialize 0.3.16 (registry+https://github.com/rust-lang/crates.io-index)", 95 | ] 96 | 97 | [[package]] 98 | name = "crater-engine" 99 | version = "0.1.0" 100 | dependencies = [ 101 | "crater-bus 0.1.0", 102 | "log 0.3.4 (registry+https://github.com/rust-lang/crates.io-index)", 103 | "rustc-serialize 0.3.16 (registry+https://github.com/rust-lang/crates.io-index)", 104 | ] 105 | 106 | [[package]] 107 | name = "env_logger" 108 | version = "0.3.2" 109 | source = "registry+https://github.com/rust-lang/crates.io-index" 110 | dependencies = [ 111 | "log 0.3.4 (registry+https://github.com/rust-lang/crates.io-index)", 112 | "regex 0.1.44 (registry+https://github.com/rust-lang/crates.io-index)", 113 | ] 114 | 115 | [[package]] 116 | name = "error" 117 | version = "0.1.9" 118 | source = "registry+https://github.com/rust-lang/crates.io-index" 119 | dependencies = [ 120 | "traitobject 0.0.3 (registry+https://github.com/rust-lang/crates.io-index)", 121 | "typeable 0.1.2 (registry+https://github.com/rust-lang/crates.io-index)", 122 | ] 123 | 124 | [[package]] 125 | name = "gcc" 126 | version = "0.3.21" 127 | source = "registry+https://github.com/rust-lang/crates.io-index" 128 | dependencies = [ 129 | "advapi32-sys 0.1.2 (registry+https://github.com/rust-lang/crates.io-index)", 130 | "winapi 0.2.5 (registry+https://github.com/rust-lang/crates.io-index)", 131 | ] 132 | 133 | [[package]] 134 | name = "gdi32-sys" 135 | version = "0.1.1" 136 | source = "registry+https://github.com/rust-lang/crates.io-index" 137 | dependencies = [ 138 | "winapi 0.2.5 (registry+https://github.com/rust-lang/crates.io-index)", 139 | ] 140 | 141 | [[package]] 142 | name = "hpack" 143 | version = "0.2.0" 144 | source = "registry+https://github.com/rust-lang/crates.io-index" 145 | dependencies = [ 146 | "log 0.3.4 (registry+https://github.com/rust-lang/crates.io-index)", 147 | ] 148 | 149 | [[package]] 150 | name = "httparse" 151 | version = "1.1.0" 152 | source = "registry+https://github.com/rust-lang/crates.io-index" 153 | 154 | [[package]] 155 | name = "hyper" 156 | version = "0.7.1" 157 | source = "registry+https://github.com/rust-lang/crates.io-index" 158 | dependencies = [ 159 | "cookie 0.2.2 (registry+https://github.com/rust-lang/crates.io-index)", 160 | "httparse 1.1.0 (registry+https://github.com/rust-lang/crates.io-index)", 161 | "language-tags 0.2.0 (registry+https://github.com/rust-lang/crates.io-index)", 162 | "log 0.3.4 (registry+https://github.com/rust-lang/crates.io-index)", 163 | "mime 0.1.1 (registry+https://github.com/rust-lang/crates.io-index)", 164 | "num_cpus 0.2.10 (registry+https://github.com/rust-lang/crates.io-index)", 165 | "openssl 0.7.4 (registry+https://github.com/rust-lang/crates.io-index)", 166 | "rustc-serialize 0.3.16 (registry+https://github.com/rust-lang/crates.io-index)", 167 | "solicit 0.4.4 (registry+https://github.com/rust-lang/crates.io-index)", 168 | "time 0.1.34 (registry+https://github.com/rust-lang/crates.io-index)", 169 | "traitobject 0.0.1 (registry+https://github.com/rust-lang/crates.io-index)", 170 | "typeable 0.1.2 (registry+https://github.com/rust-lang/crates.io-index)", 171 | "unicase 1.1.0 (registry+https://github.com/rust-lang/crates.io-index)", 172 | "url 0.5.2 (registry+https://github.com/rust-lang/crates.io-index)", 173 | ] 174 | 175 | [[package]] 176 | name = "iron" 177 | version = "0.2.6" 178 | source = "registry+https://github.com/rust-lang/crates.io-index" 179 | dependencies = [ 180 | "conduit-mime-types 0.7.3 (registry+https://github.com/rust-lang/crates.io-index)", 181 | "error 0.1.9 (registry+https://github.com/rust-lang/crates.io-index)", 182 | "hyper 0.7.1 (registry+https://github.com/rust-lang/crates.io-index)", 183 | "lazy_static 0.1.15 (registry+https://github.com/rust-lang/crates.io-index)", 184 | "log 0.3.4 (registry+https://github.com/rust-lang/crates.io-index)", 185 | "modifier 0.1.0 (registry+https://github.com/rust-lang/crates.io-index)", 186 | "num_cpus 0.2.10 (registry+https://github.com/rust-lang/crates.io-index)", 187 | "plugin 0.2.6 (registry+https://github.com/rust-lang/crates.io-index)", 188 | "typemap 0.3.3 (registry+https://github.com/rust-lang/crates.io-index)", 189 | "url 0.5.2 (registry+https://github.com/rust-lang/crates.io-index)", 190 | ] 191 | 192 | [[package]] 193 | name = "kernel32-sys" 194 | version = "0.2.1" 195 | source = "registry+https://github.com/rust-lang/crates.io-index" 196 | dependencies = [ 197 | "winapi 0.2.5 (registry+https://github.com/rust-lang/crates.io-index)", 198 | "winapi-build 0.1.1 (registry+https://github.com/rust-lang/crates.io-index)", 199 | ] 200 | 201 | [[package]] 202 | name = "language-tags" 203 | version = "0.2.0" 204 | source = "registry+https://github.com/rust-lang/crates.io-index" 205 | 206 | [[package]] 207 | name = "lazy_static" 208 | version = "0.1.15" 209 | source = "registry+https://github.com/rust-lang/crates.io-index" 210 | 211 | [[package]] 212 | name = "libc" 213 | version = "0.2.4" 214 | source = "registry+https://github.com/rust-lang/crates.io-index" 215 | 216 | [[package]] 217 | name = "libressl-pnacl-sys" 218 | version = "2.1.6" 219 | source = "registry+https://github.com/rust-lang/crates.io-index" 220 | dependencies = [ 221 | "pnacl-build-helper 1.4.10 (registry+https://github.com/rust-lang/crates.io-index)", 222 | ] 223 | 224 | [[package]] 225 | name = "log" 226 | version = "0.3.4" 227 | source = "registry+https://github.com/rust-lang/crates.io-index" 228 | dependencies = [ 229 | "libc 0.2.4 (registry+https://github.com/rust-lang/crates.io-index)", 230 | ] 231 | 232 | [[package]] 233 | name = "matches" 234 | version = "0.1.2" 235 | source = "registry+https://github.com/rust-lang/crates.io-index" 236 | 237 | [[package]] 238 | name = "memchr" 239 | version = "0.1.7" 240 | source = "registry+https://github.com/rust-lang/crates.io-index" 241 | dependencies = [ 242 | "libc 0.2.4 (registry+https://github.com/rust-lang/crates.io-index)", 243 | ] 244 | 245 | [[package]] 246 | name = "mime" 247 | version = "0.1.1" 248 | source = "registry+https://github.com/rust-lang/crates.io-index" 249 | dependencies = [ 250 | "log 0.3.4 (registry+https://github.com/rust-lang/crates.io-index)", 251 | "serde 0.6.6 (registry+https://github.com/rust-lang/crates.io-index)", 252 | ] 253 | 254 | [[package]] 255 | name = "modifier" 256 | version = "0.1.0" 257 | source = "registry+https://github.com/rust-lang/crates.io-index" 258 | 259 | [[package]] 260 | name = "mount" 261 | version = "0.0.9" 262 | source = "registry+https://github.com/rust-lang/crates.io-index" 263 | dependencies = [ 264 | "iron 0.2.6 (registry+https://github.com/rust-lang/crates.io-index)", 265 | "sequence_trie 0.0.13 (registry+https://github.com/rust-lang/crates.io-index)", 266 | "url 0.2.38 (registry+https://github.com/rust-lang/crates.io-index)", 267 | ] 268 | 269 | [[package]] 270 | name = "net2" 271 | version = "0.2.20" 272 | source = "registry+https://github.com/rust-lang/crates.io-index" 273 | dependencies = [ 274 | "cfg-if 0.1.0 (registry+https://github.com/rust-lang/crates.io-index)", 275 | "kernel32-sys 0.2.1 (registry+https://github.com/rust-lang/crates.io-index)", 276 | "libc 0.2.4 (registry+https://github.com/rust-lang/crates.io-index)", 277 | "winapi 0.2.5 (registry+https://github.com/rust-lang/crates.io-index)", 278 | "ws2_32-sys 0.2.1 (registry+https://github.com/rust-lang/crates.io-index)", 279 | ] 280 | 281 | [[package]] 282 | name = "num" 283 | version = "0.1.29" 284 | source = "registry+https://github.com/rust-lang/crates.io-index" 285 | dependencies = [ 286 | "rand 0.3.12 (registry+https://github.com/rust-lang/crates.io-index)", 287 | "rustc-serialize 0.3.16 (registry+https://github.com/rust-lang/crates.io-index)", 288 | ] 289 | 290 | [[package]] 291 | name = "num_cpus" 292 | version = "0.2.10" 293 | source = "registry+https://github.com/rust-lang/crates.io-index" 294 | dependencies = [ 295 | "kernel32-sys 0.2.1 (registry+https://github.com/rust-lang/crates.io-index)", 296 | "libc 0.2.4 (registry+https://github.com/rust-lang/crates.io-index)", 297 | "winapi 0.2.5 (registry+https://github.com/rust-lang/crates.io-index)", 298 | ] 299 | 300 | [[package]] 301 | name = "openssl" 302 | version = "0.7.4" 303 | source = "registry+https://github.com/rust-lang/crates.io-index" 304 | dependencies = [ 305 | "bitflags 0.3.3 (registry+https://github.com/rust-lang/crates.io-index)", 306 | "gcc 0.3.21 (registry+https://github.com/rust-lang/crates.io-index)", 307 | "lazy_static 0.1.15 (registry+https://github.com/rust-lang/crates.io-index)", 308 | "libc 0.2.4 (registry+https://github.com/rust-lang/crates.io-index)", 309 | "openssl-sys 0.7.4 (registry+https://github.com/rust-lang/crates.io-index)", 310 | "openssl-sys-extras 0.7.4 (registry+https://github.com/rust-lang/crates.io-index)", 311 | ] 312 | 313 | [[package]] 314 | name = "openssl-sys" 315 | version = "0.7.4" 316 | source = "registry+https://github.com/rust-lang/crates.io-index" 317 | dependencies = [ 318 | "gdi32-sys 0.1.1 (registry+https://github.com/rust-lang/crates.io-index)", 319 | "libc 0.2.4 (registry+https://github.com/rust-lang/crates.io-index)", 320 | "libressl-pnacl-sys 2.1.6 (registry+https://github.com/rust-lang/crates.io-index)", 321 | "pkg-config 0.3.6 (registry+https://github.com/rust-lang/crates.io-index)", 322 | "user32-sys 0.1.2 (registry+https://github.com/rust-lang/crates.io-index)", 323 | ] 324 | 325 | [[package]] 326 | name = "openssl-sys-extras" 327 | version = "0.7.4" 328 | source = "registry+https://github.com/rust-lang/crates.io-index" 329 | dependencies = [ 330 | "gcc 0.3.21 (registry+https://github.com/rust-lang/crates.io-index)", 331 | "libc 0.2.4 (registry+https://github.com/rust-lang/crates.io-index)", 332 | "openssl-sys 0.7.4 (registry+https://github.com/rust-lang/crates.io-index)", 333 | ] 334 | 335 | [[package]] 336 | name = "phf" 337 | version = "0.7.9" 338 | source = "registry+https://github.com/rust-lang/crates.io-index" 339 | dependencies = [ 340 | "phf_shared 0.7.9 (registry+https://github.com/rust-lang/crates.io-index)", 341 | ] 342 | 343 | [[package]] 344 | name = "phf_codegen" 345 | version = "0.7.9" 346 | source = "registry+https://github.com/rust-lang/crates.io-index" 347 | dependencies = [ 348 | "phf_generator 0.7.9 (registry+https://github.com/rust-lang/crates.io-index)", 349 | "phf_shared 0.7.9 (registry+https://github.com/rust-lang/crates.io-index)", 350 | ] 351 | 352 | [[package]] 353 | name = "phf_generator" 354 | version = "0.7.9" 355 | source = "registry+https://github.com/rust-lang/crates.io-index" 356 | dependencies = [ 357 | "phf_shared 0.7.9 (registry+https://github.com/rust-lang/crates.io-index)", 358 | "rand 0.3.12 (registry+https://github.com/rust-lang/crates.io-index)", 359 | ] 360 | 361 | [[package]] 362 | name = "phf_shared" 363 | version = "0.7.9" 364 | source = "registry+https://github.com/rust-lang/crates.io-index" 365 | 366 | [[package]] 367 | name = "pkg-config" 368 | version = "0.3.6" 369 | source = "registry+https://github.com/rust-lang/crates.io-index" 370 | 371 | [[package]] 372 | name = "plugin" 373 | version = "0.2.6" 374 | source = "registry+https://github.com/rust-lang/crates.io-index" 375 | dependencies = [ 376 | "typemap 0.3.3 (registry+https://github.com/rust-lang/crates.io-index)", 377 | ] 378 | 379 | [[package]] 380 | name = "pnacl-build-helper" 381 | version = "1.4.10" 382 | source = "registry+https://github.com/rust-lang/crates.io-index" 383 | dependencies = [ 384 | "tempdir 0.3.4 (registry+https://github.com/rust-lang/crates.io-index)", 385 | ] 386 | 387 | [[package]] 388 | name = "postgres" 389 | version = "0.10.2" 390 | source = "registry+https://github.com/rust-lang/crates.io-index" 391 | dependencies = [ 392 | "bufstream 0.1.1 (registry+https://github.com/rust-lang/crates.io-index)", 393 | "byteorder 0.4.2 (registry+https://github.com/rust-lang/crates.io-index)", 394 | "log 0.3.4 (registry+https://github.com/rust-lang/crates.io-index)", 395 | "net2 0.2.20 (registry+https://github.com/rust-lang/crates.io-index)", 396 | "phf 0.7.9 (registry+https://github.com/rust-lang/crates.io-index)", 397 | "phf_codegen 0.7.9 (registry+https://github.com/rust-lang/crates.io-index)", 398 | "rustc-serialize 0.3.16 (registry+https://github.com/rust-lang/crates.io-index)", 399 | ] 400 | 401 | [[package]] 402 | name = "rand" 403 | version = "0.3.12" 404 | source = "registry+https://github.com/rust-lang/crates.io-index" 405 | dependencies = [ 406 | "advapi32-sys 0.1.2 (registry+https://github.com/rust-lang/crates.io-index)", 407 | "libc 0.2.4 (registry+https://github.com/rust-lang/crates.io-index)", 408 | "winapi 0.2.5 (registry+https://github.com/rust-lang/crates.io-index)", 409 | ] 410 | 411 | [[package]] 412 | name = "regex" 413 | version = "0.1.44" 414 | source = "registry+https://github.com/rust-lang/crates.io-index" 415 | dependencies = [ 416 | "aho-corasick 0.4.0 (registry+https://github.com/rust-lang/crates.io-index)", 417 | "memchr 0.1.7 (registry+https://github.com/rust-lang/crates.io-index)", 418 | "regex-syntax 0.2.2 (registry+https://github.com/rust-lang/crates.io-index)", 419 | ] 420 | 421 | [[package]] 422 | name = "regex-syntax" 423 | version = "0.2.2" 424 | source = "registry+https://github.com/rust-lang/crates.io-index" 425 | 426 | [[package]] 427 | name = "route-recognizer" 428 | version = "0.1.11" 429 | source = "registry+https://github.com/rust-lang/crates.io-index" 430 | 431 | [[package]] 432 | name = "router" 433 | version = "0.1.0" 434 | source = "registry+https://github.com/rust-lang/crates.io-index" 435 | dependencies = [ 436 | "iron 0.2.6 (registry+https://github.com/rust-lang/crates.io-index)", 437 | "route-recognizer 0.1.11 (registry+https://github.com/rust-lang/crates.io-index)", 438 | ] 439 | 440 | [[package]] 441 | name = "rustc-serialize" 442 | version = "0.3.16" 443 | source = "registry+https://github.com/rust-lang/crates.io-index" 444 | 445 | [[package]] 446 | name = "sequence_trie" 447 | version = "0.0.13" 448 | source = "registry+https://github.com/rust-lang/crates.io-index" 449 | 450 | [[package]] 451 | name = "serde" 452 | version = "0.6.6" 453 | source = "registry+https://github.com/rust-lang/crates.io-index" 454 | dependencies = [ 455 | "num 0.1.29 (registry+https://github.com/rust-lang/crates.io-index)", 456 | ] 457 | 458 | [[package]] 459 | name = "solicit" 460 | version = "0.4.4" 461 | source = "registry+https://github.com/rust-lang/crates.io-index" 462 | dependencies = [ 463 | "hpack 0.2.0 (registry+https://github.com/rust-lang/crates.io-index)", 464 | "log 0.3.4 (registry+https://github.com/rust-lang/crates.io-index)", 465 | ] 466 | 467 | [[package]] 468 | name = "taskcluster" 469 | version = "0.1.0" 470 | 471 | [[package]] 472 | name = "tempdir" 473 | version = "0.3.4" 474 | source = "registry+https://github.com/rust-lang/crates.io-index" 475 | dependencies = [ 476 | "rand 0.3.12 (registry+https://github.com/rust-lang/crates.io-index)", 477 | ] 478 | 479 | [[package]] 480 | name = "time" 481 | version = "0.1.34" 482 | source = "registry+https://github.com/rust-lang/crates.io-index" 483 | dependencies = [ 484 | "kernel32-sys 0.2.1 (registry+https://github.com/rust-lang/crates.io-index)", 485 | "libc 0.2.4 (registry+https://github.com/rust-lang/crates.io-index)", 486 | "winapi 0.2.5 (registry+https://github.com/rust-lang/crates.io-index)", 487 | ] 488 | 489 | [[package]] 490 | name = "traitobject" 491 | version = "0.0.1" 492 | source = "registry+https://github.com/rust-lang/crates.io-index" 493 | 494 | [[package]] 495 | name = "traitobject" 496 | version = "0.0.3" 497 | source = "registry+https://github.com/rust-lang/crates.io-index" 498 | 499 | [[package]] 500 | name = "typeable" 501 | version = "0.1.2" 502 | source = "registry+https://github.com/rust-lang/crates.io-index" 503 | 504 | [[package]] 505 | name = "typemap" 506 | version = "0.3.3" 507 | source = "registry+https://github.com/rust-lang/crates.io-index" 508 | dependencies = [ 509 | "unsafe-any 0.4.1 (registry+https://github.com/rust-lang/crates.io-index)", 510 | ] 511 | 512 | [[package]] 513 | name = "unicase" 514 | version = "1.1.0" 515 | source = "registry+https://github.com/rust-lang/crates.io-index" 516 | 517 | [[package]] 518 | name = "unsafe-any" 519 | version = "0.4.1" 520 | source = "registry+https://github.com/rust-lang/crates.io-index" 521 | dependencies = [ 522 | "traitobject 0.0.3 (registry+https://github.com/rust-lang/crates.io-index)", 523 | ] 524 | 525 | [[package]] 526 | name = "url" 527 | version = "0.2.38" 528 | source = "registry+https://github.com/rust-lang/crates.io-index" 529 | dependencies = [ 530 | "matches 0.1.2 (registry+https://github.com/rust-lang/crates.io-index)", 531 | "rustc-serialize 0.3.16 (registry+https://github.com/rust-lang/crates.io-index)", 532 | "uuid 0.1.18 (registry+https://github.com/rust-lang/crates.io-index)", 533 | ] 534 | 535 | [[package]] 536 | name = "url" 537 | version = "0.5.2" 538 | source = "registry+https://github.com/rust-lang/crates.io-index" 539 | dependencies = [ 540 | "matches 0.1.2 (registry+https://github.com/rust-lang/crates.io-index)", 541 | "rustc-serialize 0.3.16 (registry+https://github.com/rust-lang/crates.io-index)", 542 | "uuid 0.1.18 (registry+https://github.com/rust-lang/crates.io-index)", 543 | ] 544 | 545 | [[package]] 546 | name = "user32-sys" 547 | version = "0.1.2" 548 | source = "registry+https://github.com/rust-lang/crates.io-index" 549 | dependencies = [ 550 | "winapi 0.2.5 (registry+https://github.com/rust-lang/crates.io-index)", 551 | "winapi-build 0.1.1 (registry+https://github.com/rust-lang/crates.io-index)", 552 | ] 553 | 554 | [[package]] 555 | name = "uuid" 556 | version = "0.1.18" 557 | source = "registry+https://github.com/rust-lang/crates.io-index" 558 | dependencies = [ 559 | "rand 0.3.12 (registry+https://github.com/rust-lang/crates.io-index)", 560 | "rustc-serialize 0.3.16 (registry+https://github.com/rust-lang/crates.io-index)", 561 | ] 562 | 563 | [[package]] 564 | name = "winapi" 565 | version = "0.2.5" 566 | source = "registry+https://github.com/rust-lang/crates.io-index" 567 | 568 | [[package]] 569 | name = "winapi-build" 570 | version = "0.1.1" 571 | source = "registry+https://github.com/rust-lang/crates.io-index" 572 | 573 | [[package]] 574 | name = "ws2_32-sys" 575 | version = "0.2.1" 576 | source = "registry+https://github.com/rust-lang/crates.io-index" 577 | dependencies = [ 578 | "winapi 0.2.5 (registry+https://github.com/rust-lang/crates.io-index)", 579 | "winapi-build 0.1.1 (registry+https://github.com/rust-lang/crates.io-index)", 580 | ] 581 | 582 | [metadata] 583 | "checksum advapi32-sys 0.1.2 (registry+https://github.com/rust-lang/crates.io-index)" = "307c92332867e586720c0222ee9d890bbe8431711efed8a1b06bc5b40fc66bd7" 584 | "checksum aho-corasick 0.4.0 (registry+https://github.com/rust-lang/crates.io-index)" = "8f36f237c490deb976b38aca6369182dceb5a7af249aabf41c0ba5a964bac5ed" 585 | "checksum bitflags 0.3.3 (registry+https://github.com/rust-lang/crates.io-index)" = "32866f4d103c4e438b1db1158aa1b1a80ee078e5d77a59a2f906fd62a577389c" 586 | "checksum bufstream 0.1.1 (registry+https://github.com/rust-lang/crates.io-index)" = "7e493de6a8aed51697088e36dc5245ea3edd34e6872e32c732e5f996ed5b23b2" 587 | "checksum byteorder 0.4.2 (registry+https://github.com/rust-lang/crates.io-index)" = "96c8b41881888cc08af32d47ac4edd52bc7fa27fef774be47a92443756451304" 588 | "checksum cfg-if 0.1.0 (registry+https://github.com/rust-lang/crates.io-index)" = "de1e760d7b6535af4241fca8bd8adf68e2e7edacc6b29f5d399050c5e48cf88c" 589 | "checksum conduit-mime-types 0.7.3 (registry+https://github.com/rust-lang/crates.io-index)" = "95ca30253581af809925ef68c2641cc140d6183f43e12e0af4992d53768bd7b8" 590 | "checksum cookie 0.2.2 (registry+https://github.com/rust-lang/crates.io-index)" = "2fa3d3deaa24f00707d1806cd880e851bb1733571599797ba892d39638d504f9" 591 | "checksum env_logger 0.3.2 (registry+https://github.com/rust-lang/crates.io-index)" = "b6bbe7c0b619c81b9a1fd122ab3c7ef19a7b27cdba3c8486314b6f275ca211a4" 592 | "checksum error 0.1.9 (registry+https://github.com/rust-lang/crates.io-index)" = "a6e606f14042bb87cc02ef6a14db6c90ab92ed6f62d87e69377bc759fd7987cc" 593 | "checksum gcc 0.3.21 (registry+https://github.com/rust-lang/crates.io-index)" = "ca10e3e1f1c8278047da19b94dc17c4397861150d5fbcea052eedb1d9847d356" 594 | "checksum gdi32-sys 0.1.1 (registry+https://github.com/rust-lang/crates.io-index)" = "65256ec4dc2592e6f05bfc1ca3b956a4e0698aa90b1dff1f5687d55a5a3fd59a" 595 | "checksum hpack 0.2.0 (registry+https://github.com/rust-lang/crates.io-index)" = "3d2da7d3a34cf6406d9d700111b8eafafe9a251de41ae71d8052748259343b58" 596 | "checksum httparse 1.1.0 (registry+https://github.com/rust-lang/crates.io-index)" = "27eb5cd45d161db11659afaddd88190fbf189a426b9477b68b45a1ab7e6228a1" 597 | "checksum hyper 0.7.1 (registry+https://github.com/rust-lang/crates.io-index)" = "384207e64cf1e509e63d95e16ce0059927dbd509afb5369b59b6dbf5ed84e7bf" 598 | "checksum iron 0.2.6 (registry+https://github.com/rust-lang/crates.io-index)" = "6d02706746b2d95c50f463fc619de64efe10682adfa17fa18dc95e0a92b10237" 599 | "checksum kernel32-sys 0.2.1 (registry+https://github.com/rust-lang/crates.io-index)" = "b5b5e7edf375e6d26243bde172f1d5ed1446f4a766fc9b7006e1fd27258243f1" 600 | "checksum language-tags 0.2.0 (registry+https://github.com/rust-lang/crates.io-index)" = "ed3ef0cf3121570b78e685deb11670e53c575c3eb40cdffc39c4328db8e31244" 601 | "checksum lazy_static 0.1.15 (registry+https://github.com/rust-lang/crates.io-index)" = "547fd11b0400fbab39f7a6df53c648cfb01b977233015d9e0784d595a5a6c449" 602 | "checksum libc 0.2.4 (registry+https://github.com/rust-lang/crates.io-index)" = "10569e57695cc2c91ca4214357907649c9e242dc822c9ae623d0e0b0d68aa4d9" 603 | "checksum libressl-pnacl-sys 2.1.6 (registry+https://github.com/rust-lang/crates.io-index)" = "cbc058951ab6a3ef35ca16462d7642c4867e6403520811f28537a4e2f2db3e71" 604 | "checksum log 0.3.4 (registry+https://github.com/rust-lang/crates.io-index)" = "b72ac28ae1b8682bad8b149f4c009c123c40923c1cc1d63acda6508276a84611" 605 | "checksum matches 0.1.2 (registry+https://github.com/rust-lang/crates.io-index)" = "15305656809ce5a4805b1ff2946892810992197ce1270ff79baded852187942e" 606 | "checksum memchr 0.1.7 (registry+https://github.com/rust-lang/crates.io-index)" = "dc66b0957bf6ae6590681ceac49b0df16823d43037d49aaf2ee658d483af30ab" 607 | "checksum mime 0.1.1 (registry+https://github.com/rust-lang/crates.io-index)" = "6d6d35d8a5f47095d5bfa759d7587034ddb72f4f8bab5022483475bd436601e0" 608 | "checksum modifier 0.1.0 (registry+https://github.com/rust-lang/crates.io-index)" = "41f5c9112cb662acd3b204077e0de5bc66305fa8df65c8019d5adb10e9ab6e58" 609 | "checksum mount 0.0.9 (registry+https://github.com/rust-lang/crates.io-index)" = "c55c3e1b8c0804d450c6c74df19e6469caadfbb4930dfa3b237e4b03aea66aaf" 610 | "checksum net2 0.2.20 (registry+https://github.com/rust-lang/crates.io-index)" = "c892ac76ae192130c23b0f38c2482452b8e6656e55d5356e00f2fda845840707" 611 | "checksum num 0.1.29 (registry+https://github.com/rust-lang/crates.io-index)" = "57d22dea157e908496e5f1d5d662fcc75fecbc923aecf847f3c0ab14952c44e2" 612 | "checksum num_cpus 0.2.10 (registry+https://github.com/rust-lang/crates.io-index)" = "d8662bc744a09992bce099d65444efec4138089edf4f1f3661df0006652a6557" 613 | "checksum openssl 0.7.4 (registry+https://github.com/rust-lang/crates.io-index)" = "e9fc8c20bff188efae54bee18cd948037062c68dd1309c14420fdbcc09ee2645" 614 | "checksum openssl-sys 0.7.4 (registry+https://github.com/rust-lang/crates.io-index)" = "effd4a580fd79180235ef63e7f88157874c56eed2161c708a51596fc908afe31" 615 | "checksum openssl-sys-extras 0.7.4 (registry+https://github.com/rust-lang/crates.io-index)" = "f66e7841823b0c076888ae4133c794516ed5e4f49b5d916467d2c092b91a7e18" 616 | "checksum phf 0.7.9 (registry+https://github.com/rust-lang/crates.io-index)" = "dad7ea10056c8f4d72ea5ef72ca55cad5438194ad229a6dfe0f25bbf9df32b9b" 617 | "checksum phf_codegen 0.7.9 (registry+https://github.com/rust-lang/crates.io-index)" = "32988ed250cc48d9800703edfdfabf0e54d014e16af18f213bd1a9effc46e8c6" 618 | "checksum phf_generator 0.7.9 (registry+https://github.com/rust-lang/crates.io-index)" = "123476dfa95f1042a54d81939162de053b32a0fdbcdf4f5ca8db9a0af4b99541" 619 | "checksum phf_shared 0.7.9 (registry+https://github.com/rust-lang/crates.io-index)" = "cd65402249906c6d063657f38ddaed57ac98189c1a67fb1138cfd05196dc054b" 620 | "checksum pkg-config 0.3.6 (registry+https://github.com/rust-lang/crates.io-index)" = "e22680bde24ca417a3a7fc1036ead9acf0a0af007217c955f61c0eaa903edb2f" 621 | "checksum plugin 0.2.6 (registry+https://github.com/rust-lang/crates.io-index)" = "1a6a0dc3910bc8db877ffed8e457763b317cf880df4ae19109b9f77d277cf6e0" 622 | "checksum pnacl-build-helper 1.4.10 (registry+https://github.com/rust-lang/crates.io-index)" = "61c9231d31aea845007443d62fcbb58bb6949ab9c18081ee1e09920e0cf1118b" 623 | "checksum postgres 0.10.2 (registry+https://github.com/rust-lang/crates.io-index)" = "d8d8266b5d9aed31eb2c52d93d7411355a96bb47a92d323429c76460382bd969" 624 | "checksum rand 0.3.12 (registry+https://github.com/rust-lang/crates.io-index)" = "5eee40bdf3d293e1648490ab47e5471d9ab3e455e6b0bd48e558c454be4a015e" 625 | "checksum regex 0.1.44 (registry+https://github.com/rust-lang/crates.io-index)" = "e0940ad6bf8abf79e3210809a6a49e199fc8e00b7deafc0d9394157f56f5401e" 626 | "checksum regex-syntax 0.2.2 (registry+https://github.com/rust-lang/crates.io-index)" = "fcf1e805b0a23c845be2a303136d840a1511284727bc1f1fc32d079552ef901f" 627 | "checksum route-recognizer 0.1.11 (registry+https://github.com/rust-lang/crates.io-index)" = "4f0a750d020adb1978f5964ea7bca830585899b09da7cbb3f04961fc2400122d" 628 | "checksum router 0.1.0 (registry+https://github.com/rust-lang/crates.io-index)" = "77e4ebec45b604464852f33efd3b8dd64ad95b131b7718041152592bc8a09c0d" 629 | "checksum rustc-serialize 0.3.16 (registry+https://github.com/rust-lang/crates.io-index)" = "1a48546a64cae47d06885e9bccadb99d0547d877a94c5167fa451ea33a484456" 630 | "checksum sequence_trie 0.0.13 (registry+https://github.com/rust-lang/crates.io-index)" = "d5b4eb0f7d1ff9b9666d8b8ff543f3705dd464025269a5b0e1988ffa60ca1be8" 631 | "checksum serde 0.6.6 (registry+https://github.com/rust-lang/crates.io-index)" = "dc6b43c62f340c279a3608c1e6ed7b144b1e08ee74fac2a372500dc5d576af25" 632 | "checksum solicit 0.4.4 (registry+https://github.com/rust-lang/crates.io-index)" = "172382bac9424588d7840732b250faeeef88942e37b6e35317dce98cafdd75b2" 633 | "checksum tempdir 0.3.4 (registry+https://github.com/rust-lang/crates.io-index)" = "0b62933a3f96cd559700662c34f8bab881d9e3540289fb4f368419c7f13a5aa9" 634 | "checksum time 0.1.34 (registry+https://github.com/rust-lang/crates.io-index)" = "8c4aeaa1c95974f5763c3a5ac0db95a19793589bcea5d22e161b5587e3aad029" 635 | "checksum traitobject 0.0.1 (registry+https://github.com/rust-lang/crates.io-index)" = "07eaeb7689bb7fca7ce15628319635758eda769fed481ecfe6686ddef2600616" 636 | "checksum traitobject 0.0.3 (registry+https://github.com/rust-lang/crates.io-index)" = "9dc23794ff47c95882da6f9d15de9a6be14987760a28cc0aafb40b7675ef09d8" 637 | "checksum typeable 0.1.2 (registry+https://github.com/rust-lang/crates.io-index)" = "1410f6f91f21d1612654e7cc69193b0334f909dcf2c790c4826254fbb86f8887" 638 | "checksum typemap 0.3.3 (registry+https://github.com/rust-lang/crates.io-index)" = "653be63c80a3296da5551e1bfd2cca35227e13cdd08c6668903ae2f4f77aa1f6" 639 | "checksum unicase 1.1.0 (registry+https://github.com/rust-lang/crates.io-index)" = "305831f6dc1c00f2609eeb98854c2199f5fb11e3118881fcbbf49e5f48bf5ab9" 640 | "checksum unsafe-any 0.4.1 (registry+https://github.com/rust-lang/crates.io-index)" = "b351086021ebc264aea3ab4f94d61d889d98e5e9ec2d985d993f50133537fd3a" 641 | "checksum url 0.2.38 (registry+https://github.com/rust-lang/crates.io-index)" = "cbaa8377a162d88e7d15db0cf110c8523453edcbc5bc66d2b6fffccffa34a068" 642 | "checksum url 0.5.2 (registry+https://github.com/rust-lang/crates.io-index)" = "9953ea60b40337ad33bd9395167a833f296fec66df83ed3f5cb23da28566c309" 643 | "checksum user32-sys 0.1.2 (registry+https://github.com/rust-lang/crates.io-index)" = "6717129de5ac253f5642fc78a51d0c7de6f9f53d617fc94e9bae7f6e71cf5504" 644 | "checksum uuid 0.1.18 (registry+https://github.com/rust-lang/crates.io-index)" = "78c590b5bd79ed10aad8fb75f078a59d8db445af6c743e55c4a53227fc01c13f" 645 | "checksum winapi 0.2.5 (registry+https://github.com/rust-lang/crates.io-index)" = "dc3583688b861fcd83c2823d37cf2cd2446c233dd7ba3f97884d1a7302817537" 646 | "checksum winapi-build 0.1.1 (registry+https://github.com/rust-lang/crates.io-index)" = "2d315eee3b34aca4797b2da6b13ed88266e6d612562a0c46390af8299fc699bc" 647 | "checksum ws2_32-sys 0.2.1 (registry+https://github.com/rust-lang/crates.io-index)" = "d59cefebd0c892fa2dd6de581e937301d8552cb44489cdff035c6187cb63fa5e" 648 | -------------------------------------------------------------------------------- /rs/Cargo.toml: -------------------------------------------------------------------------------- 1 | [package] 2 | name = "crater" 3 | version = "0.1.0" 4 | authors = ["Brian Anderson "] 5 | 6 | # The web server, serves API calls via JSON, possibly HTML, and 7 | # possibly coordinates the job engine. 8 | [[bin]] 9 | name = "crater-web" 10 | path = "crater-web/main.rs" 11 | 12 | # The user interface 13 | [[bin]] 14 | name = "crater-cli" 15 | path = "crater-cli/main.rs" 16 | 17 | # Dependencies for the above two. 18 | # FIXME: I'd rather not mix the deps for them so that crater-cli can 19 | # be built without building iron. 20 | [dependencies] 21 | iron = "0.2.6" 22 | hyper = "0.7.1" 23 | router = "0.1.0" 24 | mount = "0.0.9" 25 | log = "0.3.2" 26 | env_logger = "0.3.1" 27 | rustc-serialize = "0.3.16" 28 | 29 | # Coordinates builds into crater jobs 30 | [dependencies.crater-engine] 31 | path = "crater-engine" 32 | 33 | # Abstracts the AMQP implementation 34 | [dependencies.crater-bus] 35 | path = "crater-bus" 36 | 37 | # Abstracts the PostgreSQL database 38 | [dependencies.crater-db] 39 | path = "crater-db" 40 | 41 | # Shared defs for the HTTP API 42 | [dependencies.crater-api] 43 | path = "crater-api" 44 | 45 | # Bindings to TaskCluster 46 | [dependencies.taskcluster] 47 | path = "taskcluster" 48 | -------------------------------------------------------------------------------- /rs/crater-api/Cargo.toml: -------------------------------------------------------------------------------- 1 | [package] 2 | name = "crater-api" 3 | version = "0.1.0" 4 | authors = ["Brian Anderson "] 5 | 6 | [lib] 7 | name = "crater_api" 8 | path = "lib.rs" 9 | 10 | [dependencies] 11 | rustc-serialize = "*" 12 | -------------------------------------------------------------------------------- /rs/crater-api/lib.rs: -------------------------------------------------------------------------------- 1 | extern crate rustc_serialize; 2 | 3 | pub mod v1 { 4 | use std::error::Error as StdError; 5 | use std::fmt::{self, Display, Formatter}; 6 | 7 | #[derive(RustcEncodable, RustcDecodable)] 8 | #[derive(Debug)] 9 | pub struct Auth { 10 | pub name: String, 11 | pub token: String 12 | } 13 | 14 | #[derive(RustcEncodable, RustcDecodable)] 15 | #[derive(Debug)] 16 | pub struct SelfTestRequest { 17 | pub auth: Auth 18 | } 19 | 20 | /// Build a compiler from a git repo and a commit sha 21 | #[derive(RustcEncodable, RustcDecodable)] 22 | #[derive(Debug)] 23 | pub struct CustomBuildRequest { 24 | pub auth: Auth, 25 | pub repo_url: String, 26 | pub commit_sha: String 27 | } 28 | 29 | #[derive(RustcEncodable, RustcDecodable)] 30 | #[derive(Debug)] 31 | pub struct CrateBuildRequest { 32 | pub auth: Auth, 33 | pub toolchain: String 34 | } 35 | 36 | #[derive(RustcEncodable, RustcDecodable)] 37 | #[derive(Debug)] 38 | pub struct ReportRequest { 39 | pub auth: Auth, 40 | pub kind: ReportKind 41 | } 42 | 43 | #[derive(RustcEncodable, RustcDecodable)] 44 | #[derive(Debug)] 45 | pub enum ReportKind { 46 | Comparison { 47 | toolchain_from: String, 48 | toolchain_to: String 49 | }, 50 | Toolchain(String) 51 | } 52 | 53 | /// Responses from running one of the v1 nodejs scripts 54 | #[derive(RustcEncodable, RustcDecodable)] 55 | #[derive(Debug)] 56 | pub struct StdIoResponse { 57 | pub stdout: String, 58 | pub stderr: String, 59 | pub success: bool 60 | } 61 | 62 | impl StdError for StdIoResponse { 63 | fn description(&self) -> &str { &self.stderr } 64 | } 65 | 66 | impl Display for StdIoResponse { 67 | fn fmt(&self, f: &mut Formatter) -> Result<(), fmt::Error> { 68 | f.write_str(self.description()) 69 | } 70 | } 71 | 72 | impl From for Result { 73 | fn from(e: StdIoResponse) -> Result { 74 | if e.success { 75 | Ok(e.stdout) 76 | } else { 77 | Err(e) 78 | } 79 | } 80 | } 81 | } 82 | 83 | -------------------------------------------------------------------------------- /rs/crater-bus/Cargo.toml: -------------------------------------------------------------------------------- 1 | [package] 2 | name = "crater-bus" 3 | version = "0.1.0" 4 | authors = ["Brian Anderson "] 5 | 6 | [lib] 7 | name = "crater_bus" 8 | path = "lib.rs" 9 | 10 | [dependencies] 11 | rustc-serialize = "*" 12 | 13 | #[dependencies.amqp] 14 | #git = "https://github.com/Antti/rust-amqp" 15 | -------------------------------------------------------------------------------- /rs/crater-bus/lib.rs: -------------------------------------------------------------------------------- 1 | extern crate rustc_serialize; 2 | 3 | #[derive(RustcEncodable, RustcDecodable)] 4 | #[derive(Clone)] 5 | pub struct Config; 6 | 7 | pub fn connect(config: Config) -> Result { 8 | Ok(Bus) 9 | } 10 | 11 | pub struct Bus; 12 | 13 | impl Bus { 14 | pub fn listen(&self) -> Result { 15 | Ok(Listener) 16 | } 17 | } 18 | 19 | pub struct Listener; 20 | 21 | impl Listener { 22 | pub fn recv(&self) -> Result, Error> { 23 | Ok(None) 24 | } 25 | } 26 | 27 | pub enum Msg { 28 | } 29 | 30 | #[derive(Debug)] 31 | pub enum Error { } 32 | -------------------------------------------------------------------------------- /rs/crater-cli/main.rs: -------------------------------------------------------------------------------- 1 | extern crate hyper; 2 | extern crate rustc_serialize; 3 | #[macro_use] 4 | extern crate log; 5 | extern crate env_logger; 6 | extern crate crater_api as api; 7 | 8 | use rustc_serialize::json; 9 | use std::convert::From; 10 | use std::env; 11 | use std::error::Error as StdError; 12 | use std::fmt::{self, Display, Formatter}; 13 | use std::fs::File; 14 | use std::io::{self, Read}; 15 | use api::v1; 16 | use std::io::Write; 17 | 18 | enum Opts { 19 | CustomBuild { repo_url: String, commit_sha: String }, 20 | CrateBuild { toolchain: String }, 21 | Report { kind: v1::ReportKind }, 22 | SelfTest 23 | } 24 | 25 | #[derive(RustcEncodable, RustcDecodable)] 26 | pub struct Config { 27 | server_url: String, 28 | username: String, 29 | auth_token: String 30 | } 31 | 32 | fn main() { 33 | run().unwrap(); 34 | } 35 | 36 | fn run() -> Result<(), Error> { 37 | try!(env_logger::init()); 38 | 39 | let config = try!(load_config()); 40 | 41 | let ref args: Vec = env::args().collect(); 42 | let opts = try!(parse_opts(args)); 43 | 44 | try!(run_run(config, opts)); 45 | 46 | Ok(()) 47 | } 48 | 49 | fn load_config() -> Result { 50 | let mut path = try!(::std::env::current_dir()); 51 | path.push("crater-cli-config.json"); 52 | 53 | let mut file = try!(File::open(path)); 54 | 55 | let mut s = String::new(); 56 | try!(file.read_to_string(&mut s)); 57 | 58 | return Ok(try!(json::decode(&s))); 59 | } 60 | 61 | fn parse_opts(args: &[String]) -> Result { 62 | if args.len() < 2 { return Err(Error::OptParse) } 63 | 64 | if args[1] == "custom-build" { 65 | let repo_url = try!(args.get(2).ok_or(Error::OptParse)); 66 | let commit_sha = try!(args.get(3).ok_or(Error::OptParse)); 67 | Ok(Opts::CustomBuild { repo_url: repo_url.clone(), 68 | commit_sha: commit_sha.clone() }) 69 | } else if args[1] == "crate-build" { 70 | let toolchain = try!(args.get(2).ok_or(Error::OptParse)); 71 | Ok(Opts::CrateBuild { toolchain: toolchain.clone() }) 72 | } else if args[1] == "report" { 73 | let ref kind = try!(args.get(2).ok_or(Error::OptParse)); 74 | let kind = try!(parse_report_kind(kind, &args[3..])); 75 | Ok(Opts::Report { kind: kind }) 76 | } else if args[1] == "self-test" { 77 | Ok(Opts::SelfTest) 78 | } else { 79 | Err(Error::OptParse) 80 | } 81 | } 82 | 83 | fn parse_report_kind(kind: &str, args: &[String]) -> Result { 84 | if kind == "comparison" { 85 | let from = try!(args.get(0).ok_or(Error::OptParse)); 86 | let to = try!(args.get(1).ok_or(Error::OptParse)); 87 | Ok(v1::ReportKind::Comparison { toolchain_from: from.clone(), 88 | toolchain_to: to.clone() }) 89 | } else if kind == "toolchain" { 90 | let toolchain = try!(args.get(0).ok_or(Error::OptParse)); 91 | Ok(v1::ReportKind::Toolchain(toolchain.clone())) 92 | } else { 93 | Err(Error::OptParse) 94 | } 95 | } 96 | 97 | fn run_run(config: Config, opts: Opts) -> Result<(), Error> { 98 | let client_v1 = client_v1::Ctxt::new(config); 99 | let res = match opts { 100 | Opts::CustomBuild { repo_url, commit_sha } => { 101 | client_v1.custom_build(repo_url, commit_sha) 102 | } 103 | Opts::CrateBuild { toolchain } => { 104 | client_v1.crate_build(toolchain) 105 | } 106 | Opts::Report { kind } => { 107 | client_v1.report(kind) 108 | } 109 | Opts::SelfTest => { 110 | client_v1.self_test() 111 | } 112 | }; 113 | 114 | match res { 115 | Ok(s) => { 116 | println!("{}", s); 117 | Ok(()) 118 | } 119 | Err(Error::StdIoError(ref e)) => { 120 | try!(writeln!(std::io::stderr(), 121 | "server reported an error executing node.js process")); 122 | try!(writeln!(std::io::stderr(), "")); 123 | try!(writeln!(std::io::stderr(), "{}", e.stderr)); 124 | Ok(()) 125 | } 126 | Err(e) => Err(e) 127 | } 128 | } 129 | 130 | #[derive(Debug)] 131 | pub enum Error { 132 | OptParse, 133 | StdError(Box), 134 | StdIoError(v1::StdIoResponse) 135 | } 136 | 137 | impl StdError for Error { 138 | fn description(&self) -> &str { 139 | match *self { 140 | Error::OptParse => "bad arguments", 141 | Error::StdError(ref e) => e.description(), 142 | Error::StdIoError(ref e) => &*e.stderr 143 | } 144 | } 145 | 146 | fn cause(&self) -> Option<&StdError> { 147 | match *self { 148 | Error::StdError(ref e) => Some(&**e), 149 | _ => None 150 | } 151 | } 152 | } 153 | 154 | impl Display for Error { 155 | fn fmt(&self, f: &mut Formatter) -> Result<(), fmt::Error> { 156 | f.write_str(self.description()) 157 | } 158 | } 159 | 160 | 161 | impl From for Error { 162 | fn from(e: io::Error) -> Error { 163 | Error::StdError(Box::new(e)) 164 | } 165 | } 166 | 167 | impl From for Error { 168 | fn from(e: json::DecoderError) -> Error { 169 | Error::StdError(Box::new(e)) 170 | } 171 | } 172 | 173 | impl From for Error { 174 | fn from(e: json::EncoderError) -> Error { 175 | Error::StdError(Box::new(e)) 176 | } 177 | } 178 | 179 | impl From for Error { 180 | fn from(e: hyper::Error) -> Error { 181 | Error::StdError(Box::new(e)) 182 | } 183 | } 184 | 185 | impl From for Error { 186 | fn from(e: log::SetLoggerError) -> Error { 187 | Error::StdError(Box::new(e)) 188 | } 189 | } 190 | 191 | impl From for Error { 192 | fn from(e: v1::StdIoResponse) -> Error { 193 | Error::StdIoError(e) 194 | } 195 | } 196 | 197 | mod client_v1 { 198 | use super::{Config, Error}; 199 | use hyper::Client; 200 | use api::v1; 201 | use rustc_serialize::json; 202 | use std::io::Read; 203 | use rustc_serialize::Encodable; 204 | 205 | pub struct Ctxt { 206 | config: Config 207 | } 208 | 209 | impl Ctxt { 210 | pub fn new(config: Config) -> Ctxt { 211 | Ctxt { config: config } 212 | } 213 | 214 | /// Returns the stdout from `node schedule-tasks.js custom-build` 215 | pub fn custom_build(&self, repo_url: String, commit_sha: String) -> Result { 216 | let req = v1::CustomBuildRequest { 217 | auth: self.auth(), 218 | repo_url: repo_url, commit_sha: commit_sha 219 | }; 220 | stdio_req(&self.config, "custom_build", req) 221 | } 222 | 223 | pub fn crate_build(&self, toolchain: String) -> Result { 224 | let req = v1::CrateBuildRequest { 225 | auth: self.auth(), 226 | toolchain: toolchain 227 | }; 228 | stdio_req(&self.config, "crate_build", req) 229 | } 230 | 231 | pub fn report(&self, kind: v1::ReportKind) -> Result { 232 | let req = v1::ReportRequest { 233 | auth: self.auth(), 234 | kind: kind 235 | }; 236 | stdio_req(&self.config, "report", req) 237 | } 238 | 239 | pub fn self_test(&self) -> Result { 240 | let req = v1::SelfTestRequest { 241 | auth: self.auth() 242 | }; 243 | stdio_req(&self.config, "self-test", req) 244 | } 245 | 246 | fn auth(&self) -> v1::Auth { 247 | v1::Auth { 248 | name: self.config.username.clone(), 249 | token: self.config.auth_token.clone() 250 | } 251 | } 252 | } 253 | 254 | fn stdio_req(config: &Config, name: &str, ref req: T) -> Result 255 | where T: Encodable { 256 | let ref api_url = format!("{}/api/v1/{}", config.server_url, name); 257 | info!("api endpoint: {}", api_url); 258 | let ref req_str = try!(json::encode(req)); 259 | 260 | let mut client = Client::new(); 261 | let mut http_res = try!(client.post(api_url).body(req_str).send()); 262 | let ref mut res_str = String::new(); 263 | try!(http_res.read_to_string(res_str)); 264 | 265 | let res: v1::StdIoResponse = try!(json::decode(res_str)); 266 | let stdout = try!(Result::from(res)); 267 | 268 | Ok(stdout) 269 | } 270 | } 271 | -------------------------------------------------------------------------------- /rs/crater-db/Cargo.toml: -------------------------------------------------------------------------------- 1 | [package] 2 | name = "crater-db" 3 | version = "0.1.0" 4 | authors = ["Brian Anderson "] 5 | 6 | [lib] 7 | name = "crater_db" 8 | path = "lib.rs" 9 | 10 | [dependencies] 11 | postgres = "^0.10.2" 12 | rustc-serialize = "*" 13 | -------------------------------------------------------------------------------- /rs/crater-db/lib.rs: -------------------------------------------------------------------------------- 1 | #![cfg_attr(test, feature(std_misc))] 2 | 3 | extern crate postgres; 4 | extern crate rustc_serialize; 5 | 6 | use std::error::Error as StdError; 7 | 8 | use postgres::{Connection, SslMode}; 9 | 10 | #[derive(PartialEq, Debug)] 11 | pub struct BuildResult { 12 | pub toolchain: String, 13 | pub crate_name: String, 14 | pub crate_vers: String, 15 | pub status: String, 16 | pub task_id: String 17 | } 18 | 19 | pub struct BuildResultKey { 20 | pub toolchain: String, 21 | pub crate_name: String, 22 | pub crate_vers: String 23 | } 24 | 25 | #[derive(RustcEncodable, RustcDecodable)] 26 | pub struct Config { 27 | pub dbname: String, 28 | pub username: String, 29 | pub password: String, 30 | pub host: String, 31 | pub port: u16 32 | } 33 | 34 | pub struct Database { 35 | conn: Connection 36 | } 37 | 38 | impl Database { 39 | /// Connects and updates the db to the correct scheme 40 | pub fn connect(credentials: &Config 41 | ) -> Result> { 42 | let url = make_url(&credentials.dbname, 43 | &credentials.username, 44 | &credentials.password, 45 | &credentials.host, 46 | credentials.port); 47 | let conn = try!(Connection::connect(&url[..], &SslMode::None)); 48 | 49 | let db = Database { conn: conn }; 50 | 51 | try!(db.create_or_upgrade_tables()); 52 | 53 | Ok(db) 54 | } 55 | 56 | fn create_or_upgrade_tables(&self) -> Result<(), Box> { 57 | let q = "create table if not exists \ 58 | build_results ( \ 59 | toolchain text not null, \ 60 | crate_name text not null, crate_vers text not null, \ 61 | status text not null, \ 62 | task_id text not null, \ 63 | primary key ( \ 64 | toolchain, crate_name, crate_vers ) )"; 65 | try!(self.conn.execute(q, &[])); 66 | 67 | let q = "create table if not exists \ 68 | custom_toolchains ( \ 69 | toolchain text not null, \ 70 | status text not null, \ 71 | task_id text not null, \ 72 | primary key (toolchain) )"; 73 | try!(self.conn.execute(q, &[])); 74 | 75 | let q = "create table if not exists \ 76 | crate_versions ( \ 77 | name text not null, \ 78 | version text not null, \ 79 | primary key (name, version) )"; 80 | try!(self.conn.execute(q, &[])); 81 | 82 | let q = "create table if not exists \ 83 | crate_rank ( \ 84 | name text not null, \ 85 | rank integer not null, \ 86 | primary key (name) )"; 87 | try!(self.conn.execute(q, &[])); 88 | 89 | let q = "create table if not exists \ 90 | dep_edges ( \ 91 | name text not null, \ 92 | dep text not null, \ 93 | primary key (name, dep) )"; 94 | try!(self.conn.execute(q, &[])); 95 | 96 | Ok(()) 97 | } 98 | 99 | pub fn delete_tables_and_close(self) -> Result<(), Box> { 100 | let q = "drop table if exists build_results"; 101 | try!(self.conn.execute(q, &[])); 102 | 103 | let q = "drop table if exists custom_toolchains"; 104 | try!(self.conn.execute(q, &[])); 105 | 106 | let q = "drop table if exists crate_versions"; 107 | try!(self.conn.execute(q, &[])); 108 | 109 | let q = "drop table if exists crate_rank"; 110 | try!(self.conn.execute(q, &[])); 111 | 112 | let q = "drop table if exists dep_edges"; 113 | try!(self.conn.execute(q, &[])); 114 | 115 | Ok(()) 116 | } 117 | 118 | pub fn add_build_result(&self, build_result: &BuildResult) -> Result<(), Box> { 119 | let upsert_retry_limit = 10; 120 | for _ in &[0 .. upsert_retry_limit] { 121 | let update_q = "update build_results set status = $4, task_id = $5 where \ 122 | toolchain = $1 and crate_name = $2 and crate_vers = $3"; 123 | 124 | let r = self.conn.execute(update_q, &[ 125 | &build_result.toolchain, 126 | &build_result.crate_name, 127 | &build_result.crate_vers, 128 | &build_result.status, 129 | &build_result.task_id]); 130 | match r { 131 | Ok(rows) if rows > 0 => return Ok(()), 132 | Ok(_) => (/* pass */), 133 | Err(err) => return Err(Box::new(err)) 134 | } 135 | 136 | let insert_q = "insert into build_results values ($1, $2, $3, $4, $5)"; 137 | 138 | let r = self.conn.execute(insert_q, &[ 139 | &build_result.toolchain, 140 | &build_result.crate_name, 141 | &build_result.crate_vers, 142 | &build_result.status, 143 | &build_result.task_id]); 144 | if r.is_ok() { return Ok(()) } 145 | } 146 | 147 | Err(Box::from(Error::UpsertFailure)) 148 | } 149 | 150 | pub fn get_build_result(&self, key: &BuildResultKey) -> Result> { 151 | let q = "select * from build_results where \ 152 | toolchain = $1 and crate_name = $2 and crate_vers = $3"; 153 | let stmt = try!(self.conn.prepare(q)); 154 | for row in try!(stmt.query(&[&key.toolchain, &key.crate_name, &key.crate_vers])) { 155 | return Ok(BuildResult { 156 | toolchain: row.get(0), 157 | crate_name: row.get(1), 158 | crate_vers: row.get(2), 159 | status: row.get(3), 160 | task_id: row.get(4) 161 | }) 162 | } 163 | 164 | Err(Box::from(Error::DbEmptyResultFailure)) 165 | } 166 | } 167 | 168 | #[derive(Debug)] 169 | enum Error { 170 | UpsertFailure, 171 | DbEmptyResultFailure 172 | } 173 | 174 | impl StdError for Error { 175 | fn description(&self) -> &str { 176 | match *self { 177 | Error::UpsertFailure => "upsert failure", 178 | Error::DbEmptyResultFailure => "no results", 179 | } 180 | } 181 | } 182 | 183 | impl ::std::fmt::Display for Error { 184 | fn fmt(&self, f: &mut ::std::fmt::Formatter) -> Result<(), ::std::fmt::Error> { 185 | try!(f.write_str(self.description())); 186 | Ok(()) 187 | } 188 | } 189 | 190 | fn make_url(dbname: &str, username: &str, password: &str, host: &str, port: u16) -> String { 191 | format!("postgres://{}:{}@{}:{}/{}", username, password, host, port, dbname) 192 | } 193 | 194 | // Tests expect username/password/db's 'crater-test' to exist 195 | #[cfg(test)] 196 | mod test { 197 | use super::*; 198 | use std::sync::{StaticMutex, MUTEX_INIT}; 199 | 200 | static LOCK: StaticMutex = MUTEX_INIT; 201 | 202 | fn connect() -> Database { 203 | let credentials = Config { 204 | dbname: "crater-test".to_string(), 205 | username: "crater-test".to_string(), 206 | password: "crater-test".to_string(), 207 | host: "localhost".to_string(), 208 | port: 5432 209 | }; 210 | Database::connect(&credentials).unwrap() 211 | } 212 | 213 | fn dbtest(f: &Fn()) { 214 | let _g = LOCK.lock().unwrap_or_else(|p| p.into_inner()); 215 | { connect().delete_tables_and_close().unwrap(); } 216 | f(); 217 | } 218 | 219 | #[test] 220 | fn connect_and_disconnect() { 221 | dbtest(&|| { 222 | let _ = connect(); 223 | }) 224 | } 225 | 226 | #[test] 227 | fn add_result_once() { 228 | dbtest(&|| { 229 | let expected = BuildResult { 230 | toolchain: String::from("nightly-2015-01-01"), 231 | crate_name: String::from("num"), 232 | crate_vers: String::from("1.0.0"), 233 | status: String::from("success"), 234 | task_id: String::from("my-task-id") 235 | }; 236 | let db = connect(); 237 | assert!(db.add_build_result(&expected).is_ok()); 238 | 239 | let actual = db.get_build_result(&BuildResultKey { 240 | toolchain: expected.toolchain.clone(), 241 | crate_name: expected.crate_name.clone(), 242 | crate_vers: expected.crate_vers.clone() 243 | }).unwrap(); 244 | 245 | assert_eq!(expected, actual); 246 | }) 247 | } 248 | 249 | #[test] 250 | fn add_result_twice() { 251 | dbtest(&|| { 252 | let expected = BuildResult { 253 | toolchain: String::from("nightly-2015-01-01"), 254 | crate_name: String::from("num"), 255 | crate_vers: String::from("1.0.0"), 256 | status: String::from("success"), 257 | task_id: String::from("my-task-id") 258 | }; 259 | let db = connect(); 260 | assert!(db.add_build_result(&expected).is_ok()); 261 | 262 | let expected = BuildResult { 263 | toolchain: String::from("nightly-2015-01-01"), 264 | crate_name: String::from("num"), 265 | crate_vers: String::from("1.0.0"), 266 | status: String::from("failure"), 267 | task_id: String::from("my-task-id-2") 268 | }; 269 | 270 | let db = connect(); 271 | assert!(db.add_build_result(&expected).is_ok()); 272 | 273 | let actual = db.get_build_result(&BuildResultKey { 274 | toolchain: expected.toolchain.clone(), 275 | crate_name: expected.crate_name.clone(), 276 | crate_vers: expected.crate_vers.clone() 277 | }).unwrap(); 278 | 279 | assert_eq!(expected, actual); 280 | }) 281 | } 282 | } 283 | -------------------------------------------------------------------------------- /rs/crater-engine/Cargo.toml: -------------------------------------------------------------------------------- 1 | [package] 2 | name = "crater-engine" 3 | version = "0.1.0" 4 | authors = ["Brian Anderson "] 5 | 6 | [lib] 7 | name = "crater_engine" 8 | path = "lib.rs" 9 | 10 | [dependencies] 11 | rustc-serialize = "*" 12 | log = "*" 13 | 14 | [dependencies.crater-bus] 15 | path = "../crater-bus" -------------------------------------------------------------------------------- /rs/crater-engine/lib.rs: -------------------------------------------------------------------------------- 1 | extern crate rustc_serialize; 2 | extern crate crater_bus as bus; 3 | #[macro_use] 4 | extern crate log; 5 | 6 | use std::error::Error as StdError; 7 | use std::fmt::{self, Display, Formatter}; 8 | 9 | #[derive(RustcEncodable, RustcDecodable)] 10 | #[derive(Clone)] 11 | pub struct Config { 12 | bus_config: bus::Config 13 | } 14 | 15 | pub fn initialize(config: Config) -> Result { 16 | let bus = try!(bus::connect(config.bus_config)); 17 | 18 | Ok(Engine { 19 | bus: bus 20 | }) 21 | } 22 | 23 | pub struct Engine { 24 | bus: bus::Bus 25 | } 26 | 27 | impl Engine { 28 | pub fn run(self) -> Result<(), Error> { 29 | info!("starting crater engine"); 30 | let listener = try!(self.bus.listen()); 31 | 32 | loop { 33 | match try!(listener.recv()) { 34 | Some(_) => { 35 | } 36 | None => { 37 | return Ok(()); 38 | } 39 | } 40 | } 41 | } 42 | } 43 | 44 | #[derive(Debug)] 45 | pub enum Error { 46 | BusError(bus::Error) 47 | } 48 | 49 | impl StdError for Error { 50 | fn description(&self) -> &str { 51 | "message bus error" 52 | } 53 | } 54 | 55 | impl Display for Error { 56 | fn fmt(&self, f: &mut Formatter) -> Result<(), fmt::Error> { 57 | f.write_str(self.description()) 58 | } 59 | } 60 | 61 | impl From for Error { 62 | fn from(e: bus::Error) -> Error { 63 | Error::BusError(e) 64 | } 65 | } 66 | -------------------------------------------------------------------------------- /rs/crater-web/main.rs: -------------------------------------------------------------------------------- 1 | extern crate iron; 2 | extern crate hyper; 3 | extern crate router; 4 | extern crate mount; 5 | #[macro_use] 6 | extern crate log; 7 | extern crate env_logger; 8 | extern crate crater_db as db; 9 | extern crate rustc_serialize; 10 | extern crate crater_engine as engine; 11 | extern crate crater_api as api; 12 | 13 | use iron::mime::Mime; 14 | use iron::prelude::*; 15 | use iron::status; 16 | use mount::Mount; 17 | use router::Router; 18 | use rustc_serialize::json; 19 | use std::convert::From; 20 | use std::error::Error as StdError; 21 | use std::fmt::{self, Display, Formatter}; 22 | use std::fs::File; 23 | use std::io::{self, Read}; 24 | use std::sync::Arc; 25 | use std::thread; 26 | 27 | #[derive(RustcEncodable, RustcDecodable)] 28 | struct Config { 29 | host: String, 30 | port: u16, 31 | db: db::Config, 32 | engine: engine::Config, 33 | users: Vec<(String, String)> 34 | } 35 | 36 | fn main() { 37 | run().unwrap(); 38 | } 39 | 40 | fn run() -> Result<(), Error> { 41 | try!(env_logger::init()); 42 | 43 | let config = try!(load_config()); 44 | 45 | // Start the job engine that listens to the pulse server, creates 46 | // taskcluster tasks, and updates the database with results. 47 | try!(start_engine(config.engine.clone())); 48 | 49 | // Blocks until the process is killed 50 | run_web_server(&config) 51 | } 52 | 53 | fn load_config() -> Result { 54 | let mut path = try!(::std::env::current_dir()); 55 | path.push("crater-web-config.json"); 56 | 57 | let mut file = try!(File::open(path)); 58 | 59 | let mut s = String::new(); 60 | try!(file.read_to_string(&mut s)); 61 | 62 | return Ok(try!(json::decode(&s))); 63 | } 64 | 65 | fn start_engine(engine_config: engine::Config) -> Result<(), Error> { 66 | 67 | let engine = try!(engine::initialize(engine_config)); 68 | 69 | thread::spawn(|| { 70 | // FIXME: error handling 71 | engine.run().unwrap(); 72 | }); 73 | 74 | Ok(()) 75 | } 76 | 77 | fn run_web_server(config: &Config) -> Result<(), Error> { 78 | let static_router = static_router(); 79 | let api_router_v1 = api_router_v1(config.users.clone()); 80 | 81 | let mut mount = Mount::new(); 82 | mount.mount("/api/v1/", api_router_v1); 83 | mount.mount("/", static_router); 84 | 85 | let addr = format!("{}:{}", config.host, config.port); 86 | let _ = try!(Iron::new(mount).http(&*addr)); 87 | 88 | return Ok(()); 89 | } 90 | 91 | fn api_router_v1(users: Vec<(String, String)>) -> Router { 92 | let api_ctxt_master = Arc::new(api_v1::Ctxt::new(users)); 93 | let mut router = Router::new(); 94 | 95 | let api_ctxt = api_ctxt_master.clone(); 96 | router.post("/custom_build", move |r: &mut Request| { 97 | let mut body = String::new(); 98 | try!(r.body.read_to_string(&mut body).map_err(|e| Error::from(e))); 99 | let payload = try!(api_ctxt.custom_build(&body)); 100 | Ok(Response::with((status::Ok, payload)).set(known_mime_type("application/json"))) 101 | }); 102 | let api_ctxt = api_ctxt_master.clone(); 103 | router.post("/crate_build", move |r: &mut Request| { 104 | let mut body = String::new(); 105 | try!(r.body.read_to_string(&mut body).map_err(|e| Error::from(e))); 106 | let payload = try!(api_ctxt.crate_build(&body)); 107 | Ok(Response::with((status::Ok, payload)).set(known_mime_type("application/json"))) 108 | }); 109 | let api_ctxt = api_ctxt_master.clone(); 110 | router.post("/report", move |r: &mut Request| { 111 | let mut body = String::new(); 112 | try!(r.body.read_to_string(&mut body).map_err(|e| Error::from(e))); 113 | let payload = try!(api_ctxt.report(&body)); 114 | Ok(Response::with((status::Ok, payload)).set(known_mime_type("application/json"))) 115 | }); 116 | let api_ctxt = api_ctxt_master.clone(); 117 | router.post("/self-test", move |r: &mut Request| { 118 | let mut body = String::new(); 119 | try!(r.body.read_to_string(&mut body).map_err(|e| Error::from(e))); 120 | let payload = try!(api_ctxt.self_test(&body)); 121 | Ok(Response::with((status::Ok, payload)).set(known_mime_type("application/json"))) 122 | }); 123 | 124 | return router; 125 | } 126 | 127 | fn static_router() -> Router { 128 | let mut router = Router::new(); 129 | router.get("/", move |_: &mut Request| { 130 | let (payload, mime_type) = try!(get_static_file_and_mime_type("index.html")); 131 | Ok(Response::with((status::Ok, payload)).set(mime_type)) 132 | }); 133 | router.get("*", move |r: &mut Request| { 134 | let last = r.url.path.last().expect("path is supposed to be non-empty"); 135 | let filename = if last == "" { 136 | String::from("index.html") 137 | } else { 138 | last.clone() 139 | }; 140 | 141 | let (payload, mime_type) = try!(get_static_file_and_mime_type(&filename)); 142 | 143 | Ok(Response::with((status::Ok, payload)).set(mime_type)) 144 | }); 145 | 146 | return router; 147 | } 148 | 149 | fn get_static_file_and_mime_type(name: &str) -> Result<(String, Mime), Error> { 150 | let payload = try!(get_static_file(&name)); 151 | let mime_type = known_mime_type(try!(get_mime_type(&name))); 152 | 153 | return Ok((payload, mime_type)); 154 | } 155 | 156 | fn known_mime_type(mime_type: &str) -> Mime { 157 | mime_type.parse().ok().expect("shouldn't create mime types that don't parse") 158 | } 159 | 160 | /// Loads a file from the './static' directory 161 | fn get_static_file(name: &str) -> Result { 162 | let mut path = try!(::std::env::current_dir()); 163 | 164 | let asset_dir = "static"; 165 | 166 | path.push(asset_dir); 167 | path.push(name); 168 | 169 | let mut file = try!(File::open(path)); 170 | 171 | let mut s = String::new(); 172 | try!(file.read_to_string(&mut s)); 173 | 174 | return Ok(s); 175 | } 176 | 177 | fn get_mime_type(name: &str) -> Result<&'static str, Error> { 178 | if name.ends_with(".html") { 179 | Ok("text/html") 180 | } else if name.ends_with(".js") { 181 | Ok("application/x-javascript") 182 | } else if name.ends_with(".css") { 183 | Ok("text/css") 184 | } else { 185 | Err(Error::BadMimeType) 186 | } 187 | } 188 | 189 | #[derive(Debug)] 190 | pub enum Error { 191 | BadMimeType, 192 | StdError(Box), 193 | AuthError 194 | } 195 | 196 | impl StdError for Error { 197 | fn description(&self) -> &str { 198 | match *self { 199 | Error::BadMimeType => "bad mime type", 200 | Error::StdError(ref e) => e.description(), 201 | Error::AuthError => "authentication failure" 202 | } 203 | } 204 | 205 | fn cause(&self) -> Option<&StdError> { 206 | match *self { 207 | Error::StdError(ref e) => Some(&**e), 208 | _ => None 209 | } 210 | } 211 | } 212 | 213 | impl Display for Error { 214 | fn fmt(&self, f: &mut Formatter) -> Result<(), fmt::Error> { 215 | f.write_str(self.description()) 216 | } 217 | } 218 | 219 | impl From for IronError { 220 | fn from(e: Error) -> IronError { 221 | // FIXME 222 | IronError::new(e, status::InternalServerError) 223 | } 224 | } 225 | 226 | impl From for Error { 227 | fn from(e: io::Error) -> Error { 228 | Error::StdError(Box::new(e)) 229 | } 230 | } 231 | 232 | impl From for Error { 233 | fn from(e: json::EncoderError) -> Error { 234 | Error::StdError(Box::new(e)) 235 | } 236 | } 237 | 238 | impl From for Error { 239 | fn from(e: json::DecoderError) -> Error { 240 | Error::StdError(Box::new(e)) 241 | } 242 | } 243 | 244 | impl From for Error { 245 | fn from(e: hyper::Error) -> Error { 246 | Error::StdError(Box::new(e)) 247 | } 248 | } 249 | 250 | impl From for Error { 251 | fn from(e: log::SetLoggerError) -> Error { 252 | Error::StdError(Box::new(e)) 253 | } 254 | } 255 | 256 | impl From for Error { 257 | fn from(e: engine::Error) -> Error { 258 | Error::StdError(Box::new(e)) 259 | } 260 | } 261 | 262 | impl From for Error { 263 | fn from(e: std::string::FromUtf8Error) -> Error { 264 | Error::StdError(Box::new(e)) 265 | } 266 | } 267 | 268 | mod api_v1 { 269 | use super::Error; 270 | use rustc_serialize::json; 271 | use api::v1; 272 | 273 | pub struct Ctxt { 274 | users: Vec<(String, String)> 275 | } 276 | 277 | impl Ctxt { 278 | pub fn new(users: Vec<(String, String)>) -> Ctxt { 279 | Ctxt { users: users } 280 | } 281 | 282 | pub fn custom_build(&self, req: &str) -> Result { 283 | let ref req: v1::CustomBuildRequest = try!(json::decode(req)); 284 | 285 | info!("custom_build: {:?}", req); 286 | 287 | try!(self.authorize(&req.auth)); 288 | 289 | let script = "schedule-tasks.js"; 290 | let ref args = ["custom-build", &*req.repo_url, &*req.commit_sha]; 291 | let res = try!(node_exec(script, args)); 292 | Ok(res) 293 | } 294 | 295 | pub fn crate_build(&self, req: &str) -> Result { 296 | let ref req: v1::CrateBuildRequest = try!(json::decode(req)); 297 | 298 | info!("crate_build: {:?}", req); 299 | 300 | try!(self.authorize(&req.auth)); 301 | 302 | let script = "schedule-tasks.js"; 303 | let ref args = ["crate-build", &*req.toolchain, "--most-recent-only"]; 304 | let res = try!(node_exec(script, args)); 305 | Ok(res) 306 | } 307 | 308 | pub fn report(&self, req: &str) -> Result { 309 | let ref req: v1::ReportRequest = try!(json::decode(req)); 310 | 311 | info!("report: {:?}", req); 312 | 313 | try!(self.authorize(&req.auth)); 314 | 315 | let script = "print-report.js"; 316 | let res = match req.kind { 317 | v1::ReportKind::Comparison { 318 | ref toolchain_from, ref toolchain_to 319 | } => { 320 | let ref args = ["comparison", &**toolchain_from, &**toolchain_to]; 321 | try!(node_exec(script, args)) 322 | } 323 | v1::ReportKind::Toolchain(ref t) => { 324 | let ref args = ["toolchain", &**t]; 325 | try!(node_exec(script, args)) 326 | } 327 | }; 328 | Ok(res) 329 | } 330 | 331 | pub fn self_test(&self, req: &str) -> Result { 332 | let ref req: v1::SelfTestRequest = try!(json::decode(req)); 333 | 334 | info!("self-test: {:?}", req); 335 | 336 | try!(self.authorize(&req.auth)); 337 | 338 | let ref res = v1::StdIoResponse { 339 | stdout: String::from("self-test succeeded"), 340 | stderr: String::from(""), 341 | success: true 342 | }; 343 | 344 | Ok(try!(json::encode(res))) 345 | } 346 | 347 | fn authorize(&self, auth: &v1::Auth) -> Result<(), Error> { 348 | for &(ref name, ref token) in &self.users { 349 | if name == &auth.name && token == &auth.token { 350 | return Ok(()); 351 | } 352 | } 353 | 354 | Err(Error::AuthError) 355 | } 356 | 357 | } 358 | 359 | fn node_exec(script: &str, args: &[&str]) -> Result { 360 | use std::process::Command; 361 | 362 | info!("running node: {} {:?}", script, args); 363 | 364 | let script_slice: &[&str] = &[script]; 365 | let ref mut real_args = Vec::from(script_slice); 366 | real_args.extend(args.into_iter()); 367 | 368 | // Back up from the 'rs' directory to get to the js stuff 369 | let dir = ".."; 370 | 371 | let output = try!(Command::new("node") 372 | .args(real_args) 373 | .current_dir(dir) 374 | .output()); 375 | 376 | let ref res = v1::StdIoResponse { 377 | stdout: try!(String::from_utf8(output.stdout)), 378 | stderr: try!(String::from_utf8(output.stderr)), 379 | success: output.status.success() 380 | }; 381 | 382 | Ok(try!(json::encode(res))) 383 | } 384 | } 385 | 386 | -------------------------------------------------------------------------------- /rs/static/index.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | Hi! 8 | 9 | -------------------------------------------------------------------------------- /rs/taskcluster/Cargo.toml: -------------------------------------------------------------------------------- 1 | [package] 2 | name = "taskcluster" 3 | version = "0.1.0" 4 | authors = ["Brian Anderson "] 5 | 6 | [lib] 7 | name = "taskcluster" 8 | path = "lib.rs" 9 | 10 | [dependencies] 11 | -------------------------------------------------------------------------------- /rs/taskcluster/lib.rs: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/rust-lang-nursery/taskcluster-crater/76922c57919cc342014c2b4df432172e55b350aa/rs/taskcluster/lib.rs -------------------------------------------------------------------------------- /run-crater-task.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | 3 | set -u -e 4 | 5 | say() { 6 | echo 7 | echo "# $1" 8 | echo 9 | } 10 | 11 | main() { 12 | local task_type="${CRATER_TASK_TYPE-}" 13 | if [ -z "$task_type" ]; then 14 | say "CRATER_TASK_TYPE not defined" 15 | exit 1 16 | fi 17 | 18 | if [ "$task_type" = "crate-build" ]; then 19 | local rust_installer="${CRATER_RUST_INSTALLER-}" 20 | local std_installer="${CRATER_STD_INSTALLER-}" 21 | local cargo_installer="${CRATER_CARGO_INSTALLER-}" 22 | local crate_file="${CRATER_CRATE_FILE-}" 23 | 24 | if [ -z "$rust_installer" ]; then 25 | say "CRATER_RUST_INSTALLER not defined" 26 | exit 1 27 | fi 28 | 29 | if [ -z "$crate_file" ]; then 30 | say "CRATER_CRATE_FILE not defined" 31 | exit 1 32 | fi 33 | 34 | say "Installing Rust from $rust_installer" 35 | curl --retry 5 -Lf "$rust_installer" -o installer.tar.gz 36 | mkdir ./rust-install 37 | tar xzf installer.tar.gz -C ./rust-install --strip-components=1 38 | ./rust-install/install.sh 39 | 40 | if [ -n "$std_installer" ]; then 41 | say "Installing std from $std_installer" 42 | curl --retry 5 -Lf "$std_installer" -o std-installer.tar.gz 43 | mkdir ./std-install 44 | tar xzf std-installer.tar.gz -C ./std-install --strip-components=1 45 | ./std-install/install.sh 46 | fi 47 | 48 | if [ -n "$cargo_installer" ]; then 49 | say "Installing Cargo from $cargo_installer" 50 | curl --retry 5 -Lf "$cargo_installer" -o cargo-installer.tar.gz 51 | mkdir ./cargo-install 52 | tar xzf cargo-installer.tar.gz -C ./cargo-install --strip-components=1 53 | ./cargo-install/install.sh 54 | fi 55 | 56 | say "Printing toolchain versions" 57 | 58 | rustc --version 59 | cargo --version 60 | 61 | say "Downloading crate from $crate_file" 62 | curl --retry 5 -fL "$crate_file" -o crate.tar.gz 63 | mkdir ./crate 64 | tar xzf crate.tar.gz -C ./crate --strip-components=1 65 | 66 | say "Replacing path dependencies in Cargo.toml" 67 | if [ -e ./crate/Cargo.toml ]; then 68 | # Replaces any line beginning with 'path' with an empty line, if that line 69 | # occurs inside a [dependencies.*] section 70 | sed -i '/\[dependencies.*\]/,/\[[^db].*\]/ s/^\w*path.*//' ./crate/Cargo.toml 71 | sed -i '/\[dev-dependencies.*\]/,/\[[^db].*\]/ s/^\w*path.*//' ./crate/Cargo.toml 72 | sed -i '/\[build-dependencies.*\]/,/\[[^db].*\]/ s/^\w*path.*//' ./crate/Cargo.toml 73 | # Remove any 'path = "...",' text from inside { }, with trailing comma 74 | sed -i '/\[dependencies.*\]/,/\[[^db].*\]/ s/path *= *\"[^ ]*" *,//' ./crate/Cargo.toml 75 | sed -i '/\[dev-dependencies.*\]/,/\[[^db].*\]/ s/path *= *\"[^ ]*" *,//' ./crate/Cargo.toml 76 | sed -i '/\[build-dependencies.*\]/,/\[[^db].*\]/ s/path *= *\"[^ ]*" *,//' ./crate/Cargo.toml 77 | # Same, but w/ leading trailing comma 78 | sed -i '/\[dependencies.*\]/,/\[[^db].*\]/ s/, *path *= *\"[^ ]*"//' ./crate/Cargo.toml 79 | sed -i '/\[dev-dependencies.*\]/,/\[[^db].*\]/ s/, *path *= *\"[^ ]*"//' ./crate/Cargo.toml 80 | sed -i '/\[build-dependencies.*\]/,/\[[^db].*\]/ s/, *path *= *\"[^ ]*"//' ./crate/Cargo.toml 81 | else 82 | say "Cargo.toml does not exist!" 83 | fi 84 | 85 | say "Fetching dependencies" 86 | local count=0 87 | local max=5 88 | local sleep_time=1 89 | for i in 1 2 4 8 16; do 90 | set +e 91 | (cd ./crate && cargo fetch) 92 | set -e 93 | if [ $? = 0 ]; then 94 | break 95 | fi 96 | say "Cargo fetch failed. Trying again in $i s." 97 | sleep "$i" 98 | done 99 | if [ $? != 0 ]; then 100 | say "Cargo fetch failed completely!" 101 | exit 1 102 | fi 103 | 104 | say "Building crate" 105 | (cd ./crate && cargo build) 106 | 107 | # FIXME would like to test 108 | #(cd ./crate && cargo test) 109 | elif [ "$task_type" = "custom-build" ]; then 110 | local git_repo="${CRATER_TOOLCHAIN_GIT_REPO-}" 111 | local commit_sha="${CRATER_TOOLCHAIN_GIT_SHA-}" 112 | 113 | if [ -z "$git_repo" ]; then 114 | say "CRATER_TOOLCHAIN_GIT_REPO not defined" 115 | exit 1 116 | fi 117 | 118 | if [ -z "$commit_sha" ]; then 119 | say "CRATER_TOOLCHAIN_GIT_SHA not defined" 120 | exit 1 121 | fi 122 | 123 | say "Cloning git repo" 124 | git clone "$git_repo" rust && (cd rust && git reset "$commit_sha" --hard) 125 | 126 | say "Configuring" 127 | (cd rust && ./configure --enable-extended --build=x86_64-unknown-linux-gnu --host=x86_64-unknown-linux-gnu --target=x86_64-unknown-linux-gnu) 128 | 129 | say "Building" 130 | (cd rust && make -j2 && make dist) 131 | 132 | say "Renaming installer" 133 | mv rust/build/dist/rustc-*-x86_64-unknown-linux-gnu.tar.gz \ 134 | rust/build/dist/rustc-dev-x86_64-unknown-linux-gnu.tar.gz 135 | mv rust/build/dist/rust-std-*-x86_64-unknown-linux-gnu.tar.gz \ 136 | rust/build/dist/rust-std-dev-x86_64-unknown-linux-gnu.tar.gz 137 | mv rust/build/dist/cargo-*-x86_64-unknown-linux-gnu.tar.gz \ 138 | rust/build/dist/cargo-dev-x86_64-unknown-linux-gnu.tar.gz 139 | 140 | else 141 | say "Unknown task type" 142 | exit 1 143 | fi 144 | } 145 | 146 | main "$@" 147 | 148 | -------------------------------------------------------------------------------- /rust-dist.js: -------------------------------------------------------------------------------- 1 | 'use strict'; 2 | 3 | var debug = require('debug')(__filename.slice(__dirname.length + 1)); 4 | var Promise = require('promise'); 5 | var http = require('http'); 6 | var fs = require('fs'); 7 | var assert = require('assert'); 8 | var util = require('./crater-util'); 9 | 10 | /** 11 | * Download the JSON index and return a promised JSON object. If 12 | * `distAddr` is null the default remote address is used. 13 | */ 14 | function downloadIndex(config) { 15 | var distAddr = config.rustDistAddr; 16 | 17 | var index = distAddr + "/index.json"; 18 | 19 | return util.downloadToMem(index).then(function(data) { 20 | return JSON.parse(data); 21 | }); 22 | } 23 | 24 | /** 25 | * Converts the object returned by `downloadIndex` to a more concise form: 26 | * 27 | * { nightly: [dates], beta: [dates], stable: [dates] } 28 | * 29 | * Each is sorted from most recent to oldest. 30 | */ 31 | function getAvailableToolchainsFromIndex(index) { 32 | // The index is kinda hacky and has an extra level of directory indirection. 33 | // Peel it off here. 34 | assert(index.ds.length == 1); 35 | var index = index.ds[0].children; 36 | var dirs = index.ds; 37 | 38 | var nightly = []; 39 | var beta = []; 40 | var stable = []; 41 | 42 | for (var i = 0; i < dirs.length; i++) { 43 | var dir = dirs[i]; 44 | var name = dir.name; 45 | var files = dir.children.fs; 46 | for (var j = 0; j < files.length; j++) { 47 | var file = files[j]; 48 | if (file.name == "channel-rust-nightly") { 49 | nightly.push(name); 50 | } else if (file.name == "channel-rust-beta") { 51 | beta.push(name); 52 | } else if (file.name == "channel-rust-stable") { 53 | stable.push(name); 54 | } 55 | } 56 | } 57 | 58 | nightly.sort(); 59 | beta.sort(); 60 | stable.sort(); 61 | 62 | var toolchains = { 63 | nightly: nightly, 64 | beta: beta, 65 | stable: stable 66 | }; 67 | 68 | return toolchains; 69 | } 70 | 71 | /** 72 | * Downloads the Rust channel index and pulls out the available toolchains 73 | * into an object with the shape: 74 | * 75 | * { nightly: [dates], beta: [dates], stable: [dates] } 76 | * 77 | * Each is sorted from most recent to oldest. 78 | * Returns a promise. 79 | */ 80 | function getAvailableToolchains(config) { 81 | var p = downloadIndex(config); 82 | p = p.then(function(index) { 83 | return getAvailableToolchainsFromIndex(index); 84 | }); 85 | return p; 86 | } 87 | 88 | function installerUrlForToolchain(toolchain, triple, config) { 89 | assert(toolchain.channel); 90 | 91 | var rustDistAddr = config.rustDistAddr; 92 | 93 | var manifest = rustDistAddr + "/" + toolchain.archiveDate + "/channel-rust-" + toolchain.channel; 94 | 95 | debug("manifest addr: " + manifest); 96 | 97 | return util.downloadToMem(manifest).then(function(data) { 98 | var lines = data.match(/^.*([\n\r]+|$)/gm); 99 | var res = null; 100 | lines.forEach(function(line) { 101 | var line = line.trim(); 102 | if (line.indexOf(triple) != -1 && line.indexOf(".tar.gz", line.length - ".tar.gz".length) !== -1) { 103 | res = line.trim(); 104 | } 105 | }); 106 | 107 | if (res) { 108 | var url = rustDistAddr + "/" + toolchain.archiveDate + "/" + res; 109 | return url; 110 | } else { 111 | return Promise.reject("no installer found for triple " + triple); 112 | } 113 | }); 114 | } 115 | 116 | exports.downloadIndex = downloadIndex; 117 | exports.getAvailableToolchainsFromIndex = getAvailableToolchainsFromIndex; 118 | exports.getAvailableToolchains = getAvailableToolchains; 119 | exports.installerUrlForToolchain = installerUrlForToolchain; 120 | -------------------------------------------------------------------------------- /schedule-tasks.js: -------------------------------------------------------------------------------- 1 | 'use strict'; 2 | 3 | var debug = require('debug')(__filename.slice(__dirname.length + 1)); 4 | var fs = require('fs'); 5 | var util = require('./crater-util'); 6 | var tc = require('taskcluster-client'); 7 | var Promise = require('promise'); 8 | var slugid = require('slugid'); 9 | var scheduler = require('./scheduler'); 10 | var crateIndex = require('./crate-index'); 11 | var db = require('./crater-db'); 12 | 13 | function main() { 14 | var options = parseOptionsFromArgs(); 15 | if (!options) { 16 | console.log("can't parse options"); 17 | process.exit(1); 18 | } 19 | 20 | debug("scheduling for toolchain %s", JSON.stringify(options)); 21 | 22 | var config = util.loadDefaultConfig(); 23 | 24 | crateIndex.updateCaches(config).then(function() { 25 | if (options.type == "crate-build") { 26 | db.connect(config).then(function(dbctx) { 27 | Promise.resolve().then(function() { 28 | return scheduler.createSchedule(options, config, dbctx); 29 | }).then(function(schedule) { 30 | return scheduler.scheduleBuilds(dbctx, schedule, config); 31 | }).then(function(tasks) { 32 | console.log("created " + tasks.length + " tasks"); 33 | }).then(function() { 34 | db.disconnect(dbctx); 35 | }).catch(function(e) { 36 | console.log("error: " + e); 37 | db.disconnect(dbctx); 38 | }).done(); 39 | }); 40 | } else { 41 | Promise.resolve().then(function() { 42 | return scheduler.scheduleCustomBuild(options, config); 43 | }).catch(function(e) { 44 | console.log("error: " + e); 45 | }).done(); 46 | } 47 | }).catch(function(e) { 48 | console.log("error: " + e); 49 | }).done(); 50 | } 51 | 52 | function parseOptionsFromArgs() { 53 | var type = process.argv[2]; 54 | if (type == "crate-build") { 55 | var toolchain = util.parseToolchain(process.argv[3]) 56 | var top = null; 57 | var mostRecentOnly = false; 58 | var crateName = null; 59 | var skipExisting = false; 60 | for (var i = 4; i < process.argv.length; i++) { 61 | if (process.argv[i] == "--top") { 62 | top = parseInt(process.argv[i + 1]); 63 | } 64 | if (process.argv[i] == "--most-recent-only") { 65 | mostRecentOnly = true; 66 | } 67 | if (process.argv[i] == "--name") { 68 | crateName = process.argv[i + 1]; 69 | } 70 | if (process.argv[i] == "--skip-existing") { 71 | skipExisting = true; 72 | } 73 | } 74 | 75 | return { 76 | type: "crate-build", 77 | toolchain: toolchain, 78 | top: top, 79 | mostRecentOnly: mostRecentOnly, 80 | crateName: crateName, 81 | skipExisting: skipExisting 82 | }; 83 | } else if (type == "custom-build") { 84 | var gitRepo = process.argv[3]; 85 | var commitSha = process.argv[4]; 86 | if (!gitRepo || !commitSha) { 87 | return null; 88 | } 89 | 90 | return { 91 | type: "custom-build", 92 | gitRepo: gitRepo, 93 | commitSha: commitSha 94 | }; 95 | } 96 | } 97 | 98 | main(); 99 | -------------------------------------------------------------------------------- /scheduler.js: -------------------------------------------------------------------------------- 1 | 'use strict'; 2 | 3 | var debug = require('debug')(__filename.slice(__dirname.length + 1)); 4 | var util = require('./crater-util'); 5 | var crateIndex = require('./crate-index'); 6 | var Promise = require('promise'); 7 | var async = require('async'); 8 | var slugid = require('slugid'); 9 | var tc = require('taskcluster-client'); 10 | var assert = require('assert'); 11 | var dist = require('./rust-dist'); 12 | var db = require('./crater-db'); 13 | var sleep = require('sleep'); 14 | var https = require('https'); 15 | 16 | var customBuildMaxRunTimeInSeconds = 240 * 60; 17 | var crateBuildMaxRunTimeInSeconds = 10 * 60; 18 | 19 | /** 20 | * Create a schedule of tasks for execution by `scheduleBuilds`. 21 | */ 22 | function createSchedule(schedOpts, config, dbctx) { 23 | return crateIndex.loadCrates(config).then(function(crates) { 24 | if (schedOpts.crateName) { 25 | return retainMatchingNames(crates, schedOpts.crateName); 26 | } else { 27 | return crates; 28 | } 29 | }).then(function(crates) { 30 | if (schedOpts.top) { 31 | return retainTop(crates, schedOpts.top); 32 | } else { 33 | return crates; 34 | } 35 | }).then(function(crates) { 36 | if (schedOpts.mostRecentOnly) { 37 | return retainMostRecent(crates); 38 | } else { 39 | return crates; 40 | } 41 | }).then(function(crates) { 42 | if (schedOpts.skipExisting) { 43 | return removeCratesWithCompleteResults(crates, dbctx, schedOpts.toolchain); 44 | } else { 45 | return crates; 46 | } 47 | }).then(function(crates) { 48 | return createScheduleForCratesForToolchain(crates, schedOpts.toolchain); 49 | }); 50 | } 51 | 52 | function removeCratesWithCompleteResults(crates, dbctx, toolchain) { 53 | // Look up every crate's results and throw out the build request 54 | // if it exists, by first setting it to null then filtering it out. 55 | // (async doesn't have 'filterLimit'). 56 | return Promise.denodeify(async.mapLimit)(crates, 1, function(crate, cb) { 57 | var buildResultKey = { 58 | toolchain: toolchain, 59 | crateName: crate.name, 60 | crateVers: crate.vers 61 | }; 62 | db.getBuildResult(dbctx, buildResultKey).then(function(buildResult) { 63 | if (buildResult) { 64 | if (buildResult.status == "success" || buildResult.status == "failure") { 65 | // Already have a result, map this crate to null 66 | debug("existing result for " + crate.name + "-" + crate.vers); 67 | cb(null, null); 68 | } else { 69 | // Have a result, but not a usable one. Possibly an exception 70 | debug("bad existing result for " + crate.name + "-" + crate.vers); 71 | cb(null, crate); 72 | } 73 | } else { 74 | debug("no existing result for " + crate.name + "-" + crate.vers); 75 | cb(null, crate); 76 | } 77 | }).catch(function(e) { 78 | cb(e, null); 79 | }); 80 | }).then(function(crates) { 81 | return crates.filter(function(crate) { 82 | if (crate) { return true; } else { return false; } 83 | }); 84 | }); 85 | } 86 | 87 | function retainTop(crates, count) { 88 | var popMap = crateIndex.getPopularityMap(crates); 89 | var sorted = crates.slice(); 90 | sorted.sort(function(a, b) { 91 | var aPop = popMap[a.name]; 92 | var bPop = popMap[b.name]; 93 | if (aPop == bPop) { return 0; } 94 | if (aPop < bPop) { return 1; } 95 | if (aPop > bPop) { return -1; } 96 | }); 97 | 98 | // We want the to *count* unique crate names, but to keep 99 | // all revisions. 100 | var finalSorted = []; 101 | var seenCrateNames = {}; 102 | for (var i = 0; i < sorted.length; i++) { 103 | var crate = sorted[i]; 104 | seenCrateNames[crate.name] = 0; 105 | if (Object.keys(seenCrateNames).length > count) { 106 | break; 107 | } 108 | 109 | finalSorted.push(crate); 110 | } 111 | 112 | return finalSorted; 113 | } 114 | 115 | function retainMostRecent(crates, count) { 116 | var mostRecent = crateIndex.getMostRecentRevs(crates); 117 | 118 | var result = []; 119 | 120 | crates.forEach(function(crate) { 121 | var recent = mostRecent[crate.name]; 122 | if (crate.vers == recent.vers) { 123 | result.push(crate); 124 | } 125 | }); 126 | 127 | return result; 128 | } 129 | 130 | function retainMatchingNames(crates, name) { 131 | var result = []; 132 | 133 | crates.forEach(function(crate) { 134 | if (crate.name == name) { 135 | result.push(crate); 136 | } 137 | }); 138 | 139 | return result; 140 | } 141 | 142 | function createScheduleForCratesForToolchain(crates, toolchain) { 143 | // Convert to scheduler commands 144 | var tasks = []; 145 | crates.forEach(function(crate) { 146 | var task = { 147 | toolchain: toolchain, 148 | crateName: crate.name, 149 | crateVers: crate.vers 150 | } 151 | tasks.push(task); 152 | }); 153 | return tasks; 154 | } 155 | 156 | function scheduleBuilds(dbctx, schedule, config) { 157 | var tcCredentials = config.tcCredentials; 158 | 159 | assert(tcCredentials != null); 160 | 161 | // FIXME: For testing, just schedule five builds instead of thousands 162 | if (schedule.length > 5) { 163 | //schedule = schedule.slice(0, 5) 164 | } 165 | 166 | var queue = new tc.Queue({ 167 | credentials: tcCredentials 168 | }); 169 | 170 | var total = schedule.length; 171 | var i = 1; 172 | 173 | return Promise.denodeify(async.mapLimit)(schedule, 50, function(schedule, cb) { 174 | createTaskDescriptorForCrateBuild(dbctx, schedule, config).then(function(taskDesc) { 175 | debug("createTask payload: " + JSON.stringify(taskDesc)); 176 | 177 | var taskId = slugid.v4(); 178 | 179 | debug("creating task " + i + " of " + total + " for " + schedule.crateName + "-" + schedule.crateVers); 180 | i = i + 1; 181 | 182 | queue.createTask(taskId, taskDesc) 183 | .catch(function(e) { 184 | // TODO: How to handle a single failure here? 185 | console.log("error creating task for " + JSON.stringify(schedule)); 186 | console.log("error is " + e); 187 | cb(e, null); 188 | }).then(function(result) { 189 | console.log("created task for " + JSON.stringify(schedule)); 190 | console.log("inspector link: https://tools.taskcluster.net/task-inspector/#" + taskId); 191 | 192 | cb(null, result); 193 | }); 194 | }).catch(function(e) { 195 | cb(e, null); 196 | }).done(); 197 | }) 198 | 199 | return p; 200 | } 201 | 202 | function createTaskDescriptorForCrateBuild(dbctx, schedule, config) { 203 | var dlRootAddr = config.dlRootAddr; 204 | 205 | debug("creating task descriptor for " + JSON.stringify(schedule)); 206 | 207 | var crateName = schedule.crateName; 208 | var crateVers = schedule.crateVers; 209 | 210 | assert(crateName != null); 211 | assert(crateVers != null); 212 | 213 | var p = installerUrlsForToolchain(dbctx, schedule.toolchain, config) 214 | return p.then(function(installerUrls) { 215 | var crateUrl = dlRootAddr + "/" + crateName + "/" + crateVers + "/download"; 216 | var taskName = util.toolchainToString(schedule.toolchain) + "-vs-" + crateName + "-" + crateVers; 217 | 218 | var env = { 219 | "CRATER_RUST_INSTALLER": installerUrls.rustInstallerUrl, 220 | "CRATER_CRATE_FILE": crateUrl 221 | }; 222 | 223 | if (installerUrls.stdInstallerUrl) { 224 | env["CRATER_STD_INSTALLER"] = installerUrls.stdInstallerUrl; 225 | } 226 | if (installerUrls.cargoInstallerUrl) { 227 | env["CRATER_CARGO_INSTALLER"] = installerUrls.cargoInstallerUrl; 228 | } 229 | 230 | var extra = { 231 | "toolchain": schedule.toolchain, 232 | "crateName": crateName, 233 | "crateVers": crateVers 234 | }; 235 | 236 | return createTaskDescriptor(taskName, env, extra, 237 | "crate-build", crateBuildMaxRunTimeInSeconds, "cratertest", 238 | { }, 960 /* deadline in minutes */); 239 | }); 240 | } 241 | 242 | // FIXME Too many arguments 243 | function createTaskDescriptor(taskName, env, extra, taskType, maxRunTime, workerType, artifacts, deadlineInMinutes) { 244 | 245 | var createTime = new Date(Date.now()); 246 | var deadlineTime = new Date(createTime.getTime() + deadlineInMinutes * 60000); 247 | 248 | var cmd = "cd /home && curl -sfL https://raw.githubusercontent.com/brson/taskcluster-crater/master/run-crater-task.sh -o ./run.sh && sh ./run.sh"; 249 | 250 | env.CRATER_TASK_TYPE = taskType; 251 | extra.taskType = taskType; 252 | 253 | var task = { 254 | "provisionerId": "aws-provisioner-v1", 255 | "workerType": workerType, 256 | "created": createTime.toISOString(), 257 | "deadline": deadlineTime.toISOString(), 258 | "retries": 5, 259 | "routes": [ 260 | "crater.#" 261 | ], 262 | "payload": { 263 | "image": "brson/crater:3", 264 | "command": [ "/bin/bash", "-c", cmd ], 265 | "env": env, 266 | "maxRunTime": maxRunTime, 267 | "artifacts": artifacts 268 | }, 269 | "metadata": { 270 | "name": "Crater task " + taskName, 271 | "description": "Testing Rust crates for Rust language regressions", 272 | "owner": "banderson@mozilla.com", 273 | "source": "http://github.com/brson/taskcluster-crater" 274 | }, 275 | "extra": { 276 | "crater": extra 277 | } 278 | }; 279 | return task; 280 | } 281 | 282 | function installerUrlsForToolchain(dbctx, toolchain, config) { 283 | if (toolchain.channel) { 284 | return dist.installerUrlForToolchain(toolchain, "x86_64-unknown-linux-gnu", config) 285 | .then(function(url) { 286 | return { 287 | rustInstallerUrl: url, 288 | stdInstallerUrl: null, 289 | cargoInstallerUrl: null 290 | }; 291 | }); 292 | } else { 293 | debug(toolchain); 294 | assert(toolchain.customSha); 295 | var cargoBaseUrl = "https://s3.amazonaws.com/rust-lang-ci/cargo-builds/"; 296 | return db.getCustomToolchain(dbctx, toolchain).then(function(custom) { 297 | var stdUrl = custom.url.replace("rustc-", "rust-std-"); 298 | var cargoUrl = custom.url.replace("rustc-", "cargo-"); 299 | return { 300 | rustInstallerUrl: custom.url, 301 | stdInstallerUrl: stdUrl, 302 | cargoInstallerUrl: cargoUrl, 303 | }; 304 | }); 305 | } 306 | } 307 | 308 | var cargoNightlySha = null; 309 | 310 | function getCargoNightlySha(dbctx) { 311 | if (cargoNightlySha) { 312 | return cargoNightlySha; 313 | } 314 | // curl -H "Accept: application/vnd.github.3.sha" -sSf https://api.github.com/repos/rust-lang/cargo/commits/master 315 | var url = "https://api.github.com/repos/rust-lang/cargo/commits/master"; 316 | cargoNightlySha = util.downloadToMem(url).then(function (data) { 317 | return JSON.parse(data).sha; 318 | }); 319 | return cargoNightlySha; 320 | } 321 | 322 | /** 323 | * Schedules a build and upload of a custom build. Fails if `uniqueName` 324 | * has already been taken. 325 | */ 326 | function scheduleCustomBuild(options, config) { 327 | var gitRepo = options.gitRepo; 328 | var commitSha = options.commitSha; 329 | 330 | if (commitSha.length != 40) { 331 | return Promise.reject("bogus sha"); 332 | } 333 | 334 | var tcCredentials = config.tcCredentials; 335 | 336 | var queue = new tc.Queue({ credentials: tcCredentials }); 337 | var taskId = slugid.v4(); 338 | var taskDesc = createTaskDescriptorForCustomBuild(gitRepo, commitSha); 339 | return queue.createTask(taskId, taskDesc).then(function(result) { 340 | console.log("created task for " + gitRepo); 341 | console.log("inspector link: https://tools.taskcluster.net/task-inspector/#" + taskId); 342 | return result; 343 | }); 344 | } 345 | 346 | function createTaskDescriptorForCustomBuild(gitRepo, commitSha) { 347 | 348 | var taskName = "build-" + commitSha; 349 | 350 | var env = { 351 | "CRATER_TOOLCHAIN_GIT_REPO": gitRepo, 352 | "CRATER_TOOLCHAIN_GIT_SHA": commitSha, 353 | }; 354 | 355 | var extra = { 356 | toolchainGitRepo: gitRepo, 357 | toolchainGitSha: commitSha 358 | }; 359 | 360 | var twoMonths = 60 /*s*/ * (24 * 60) /*m*/ * (30 * 2) /*d*/; 361 | 362 | var expiry = new Date(Date.now()); 363 | expiry.setDate(expiry.getDate() + 60); 364 | 365 | // Upload the installer 366 | var artifacts = { 367 | "public/rustc-dev-x86_64-unknown-linux-gnu.tar.gz": { 368 | type: "file", 369 | path: "/home/rust/build/dist/rustc-dev-x86_64-unknown-linux-gnu.tar.gz", 370 | expires: expiry 371 | }, 372 | "public/rust-std-dev-x86_64-unknown-linux-gnu.tar.gz": { 373 | type: "file", 374 | path: "/home/rust/build/dist/rust-std-dev-x86_64-unknown-linux-gnu.tar.gz", 375 | expires: expiry 376 | }, 377 | "public/cargo-dev-x86_64-unknown-linux-gnu.tar.gz": { 378 | type: "file", 379 | path: "/home/rust/build/dist/cargo-dev-x86_64-unknown-linux-gnu.tar.gz", 380 | expires: expiry 381 | } 382 | }; 383 | 384 | var deadlineInMinutes = 60 * 24; // Rust can take a long time to build successfully 385 | 386 | return createTaskDescriptor(taskName, env, extra, 387 | "custom-build", customBuildMaxRunTimeInSeconds, "rustbuild", 388 | artifacts, deadlineInMinutes); 389 | } 390 | 391 | exports.createSchedule = createSchedule; 392 | exports.scheduleBuilds = scheduleBuilds; 393 | exports.scheduleCustomBuild = scheduleCustomBuild; 394 | -------------------------------------------------------------------------------- /test.js: -------------------------------------------------------------------------------- 1 | var debug = require('debug')(__filename.slice(__dirname.length + 1)); 2 | var assert = require('assert'); 3 | var fs = require('fs'); 4 | var Promise = require('promise'); 5 | 6 | var crates = require('./crate-index'); 7 | var dist = require('./rust-dist'); 8 | var util = require('./crater-util'); 9 | var db = require('./crater-db'); 10 | var scheduler = require('./scheduler'); 11 | var reports = require('./reports'); 12 | 13 | var testDataDir = "./test"; 14 | var testDistDir = testDataDir + "/dist"; 15 | var testCrateIndexAddr = testDataDir + "/crates.io-index"; 16 | 17 | var tmpDir = "./testtmp"; 18 | 19 | var testConfig = { 20 | rustDistAddr: testDataDir + "/dist", 21 | crateIndexAddr: testDataDir + "/crates.io-index", 22 | dlRootAddr: testDataDir + "/versions", 23 | cacheDir: tmpDir + "/cache", 24 | dbName: "crater-test", 25 | dbCredentials: { 26 | username: "crater-test", 27 | password: "crater-test", 28 | host: "localhost", 29 | port: 5432 30 | } 31 | }; 32 | 33 | var liveConfig = util.loadDefaultConfig(); 34 | 35 | function rmTempDir() { 36 | return util.runCmd("rm -Rf '" + tmpDir + "'"); 37 | } 38 | 39 | function cleanTempDir() { 40 | return rmTempDir().then(function() { 41 | return util.runCmd("mkdir -p '" + tmpDir + "'"); 42 | }); 43 | } 44 | 45 | function cleanTempDb() { 46 | return db.connect(testConfig).then(function(dbctx) { 47 | debug("a"); 48 | var p = db.depopulate(dbctx); 49 | var p = p.then(function() { db.disconnect(dbctx); }); 50 | return p; 51 | }); 52 | } 53 | 54 | function runBeforeEach(done) { 55 | cleanTempDir() 56 | .then(function() { 57 | return crates.updateCaches(testConfig); 58 | }) 59 | .then(function() { done(); }) 60 | .catch(function(e) { done(e); }); 61 | } 62 | 63 | function runAfterEach(done) { 64 | rmTempDir() 65 | .then(function() { done(); }) 66 | .catch(function(e) { done(e); }); 67 | } 68 | 69 | function runBeforeEachDb(done) { 70 | cleanTempDir() 71 | .then(function() { return cleanTempDb(); }) 72 | .then(function() { done(); }) 73 | .catch(function(e) { done(e); }); 74 | } 75 | 76 | suite("local crate-index tests", function() { 77 | 78 | beforeEach(runBeforeEach); 79 | afterEach(runAfterEach); 80 | 81 | test("load crates", function(done) { 82 | var p = crates.loadCrates(testConfig); 83 | p = p.then(function(crates) { 84 | assert(crates.length > 0); 85 | done(); 86 | }); 87 | p = p.catch(function(e) { done(e) }); 88 | }); 89 | 90 | test("get most recent revs", function(done) { 91 | var p = crates.loadCrates(testConfig); 92 | p.then(function(crateData) { 93 | return crates.getMostRecentRevs(crateData); 94 | }).then(function(recent) { 95 | assert(recent["toml"].vers == "0.1.20"); 96 | assert(recent["obj-rs"].vers == "0.4.7"); 97 | done(); 98 | }).catch(function(e) { done(e); }); 99 | }); 100 | 101 | test("get dag", function(done) { 102 | var p = crates.loadCrates(testConfig); 103 | p.then(function(crateData) { 104 | return crates.getDag(crateData); 105 | }).then(function(crateData) { 106 | assert(crateData["piston"][0] == "pistoncore-input"); 107 | done(); 108 | }).catch(function(e) { done(e); }); 109 | }); 110 | 111 | test("get popularity map", function(done) { 112 | var p = crates.loadCrates(testConfig); 113 | p.then(function(crateData) { 114 | var pop = crates.getPopularityMap(crateData); 115 | assert(pop.time == 92); 116 | assert(pop.num == 66); 117 | assert(pop.piston == 1); 118 | done(); 119 | }).catch(function(e) { done(e); }); 120 | }); 121 | 122 | // Test that concurrent access to the crate index doesn't break things because 123 | // of interleaved I/O. 124 | test("concurrent no clobber", function(done) { 125 | var responses = 0; 126 | for (var i = 0; i < 4; i++) { 127 | crates.loadCrates(testConfig).then(function(crateData) { 128 | assert(crateData.length == 8827); 129 | responses += 1; 130 | if (responses == 4) { 131 | done(); 132 | } 133 | }).catch(function(e) { done(e); }); 134 | } 135 | }); 136 | 137 | }); 138 | 139 | suite("local rust-dist tests", function() { 140 | 141 | beforeEach(runBeforeEach); 142 | afterEach(runAfterEach); 143 | 144 | test("download dist index", function(done) { 145 | var p = dist.downloadIndex(testConfig); 146 | p = p.then(function(index) { 147 | assert(index.ds[0].children.fs[1].name == "channel-rust-beta"); 148 | done(); 149 | }); 150 | p = p.catch(function(e) { done(e) }); 151 | }); 152 | 153 | test("get available toolchains", function(done) { 154 | var p = dist.downloadIndex(testConfig); 155 | p = p.then(function(index) { 156 | var toolchains = dist.getAvailableToolchainsFromIndex(index); 157 | assert(toolchains.nightly.indexOf("2015-02-20") != -1); 158 | assert(toolchains.beta.indexOf("2015-02-20") != -1); 159 | assert(toolchains.stable.indexOf("2015-02-20") == -1); 160 | done(); 161 | }); 162 | p = p.catch(function(e) { done(e) }); 163 | }); 164 | 165 | test("get available toolchains sorted", function(done) { 166 | var p = dist.downloadIndex(testConfig); 167 | p = p.then(function(index) { 168 | var toolchains = dist.getAvailableToolchainsFromIndex(index); 169 | assert(toolchains.nightly.indexOf("2015-02-20") != -1); 170 | assert(toolchains.beta.indexOf("2015-02-20") != -1); 171 | assert(toolchains.stable.indexOf("2015-02-20") == -1); 172 | done(); 173 | }); 174 | p = p.catch(function(e) { done(e) }); 175 | }); 176 | 177 | test("get available toolchains without separate download", function(done) { 178 | var p = dist.getAvailableToolchains(testConfig); 179 | p = p.then(function(toolchains) { 180 | assert(toolchains.nightly.indexOf("2015-02-20") != -1); 181 | assert(toolchains.beta.indexOf("2015-02-20") != -1); 182 | assert(toolchains.stable.indexOf("2015-02-20") == -1); 183 | done(); 184 | }); 185 | p = p.catch(function(e) { done(e) }); 186 | }); 187 | 188 | test("get installer url", function(done) { 189 | var toolchain = { channel: "beta", archiveDate: "2015-03-03" }; 190 | dist.installerUrlForToolchain(toolchain, "x86_64-unknown-linux-gnu", testConfig) 191 | .then(function(url) { 192 | assert(url == testDistDir + "/2015-03-03/rust-1.0.0-alpha.2-x86_64-unknown-linux-gnu.tar.gz"); 193 | done(); 194 | }).catch(function(e) { done(e) }); 195 | }); 196 | }); 197 | 198 | suite("local utility tests", function() { 199 | 200 | beforeEach(runBeforeEach); 201 | afterEach(runAfterEach); 202 | 203 | test("parse toolchain", function() { 204 | assert(util.parseToolchain(null) == null); 205 | assert(util.parseToolchain("nightly") == null); 206 | var actual = JSON.stringify(util.parseToolchain("nightly-2015-03-01")); 207 | var ex = JSON.stringify({ channel: "nightly", archiveDate: "2015-03-01" }); 208 | assert(actual == ex); 209 | var actual = JSON.stringify(util.parseToolchain("beta-2015-03-01")); 210 | var ex = JSON.stringify({ channel: "beta", archiveDate: "2015-03-01" }); 211 | assert(actual == ex); 212 | var actual = JSON.stringify(util.parseToolchain("stable-2015-03-01")); 213 | var ex = JSON.stringify({ channel: "stable", archiveDate: "2015-03-01" }); 214 | assert(actual == ex); 215 | }); 216 | 217 | }); 218 | 219 | suite("database tests", function() { 220 | beforeEach(runBeforeEachDb); 221 | afterEach(runAfterEach); 222 | 223 | test("populate and depopulate", function(done) { 224 | db.connect(testConfig).then(function(dbctx) { 225 | var p = db.populate(dbctx); 226 | var p = p.then(function() { return db.depopulate(dbctx); }); 227 | var p = p.then(function() { return db.disconnect(dbctx); }); 228 | var p = p.then(function() { done(); }); 229 | return p; 230 | }).catch(function(e) { done(e); }); 231 | }); 232 | 233 | test("add build result", function(done) { 234 | db.connect(testConfig).then(function(dbctx) { 235 | var actual = { 236 | toolchain: util.parseToolchain("nightly-2015-03-01"), 237 | crateName: "toml", 238 | crateVers: "1.0", 239 | status: "success", 240 | taskId: "foo" 241 | }; 242 | var p = Promise.resolve(); 243 | var p = p.then(function() { return db.addBuildResult(dbctx, actual); }); 244 | var p = p.then(function() { return db.getBuildResult(dbctx, actual); }); 245 | var p = p.then(function(br) { assert(JSON.stringify(br) == JSON.stringify(actual)); }); 246 | var p = p.then(function() { return db.disconnect(dbctx); }); 247 | var p = p.then(function() { done(); }); 248 | return p; 249 | }).catch(function(e) { done(e); }); 250 | }); 251 | 252 | test("upsert build result", function(done) { 253 | db.connect(testConfig).then(function(dbctx) { 254 | var actual = { 255 | toolchain: util.parseToolchain("nightly-2015-03-01"), 256 | crateName: "toml", 257 | crateVers: "1.0", 258 | status: "success", 259 | taskId: "foo" 260 | }; 261 | var p = Promise.resolve(); 262 | // Call addBuildResult twice, an insert then an update 263 | var p = p.then(function() { return db.addBuildResult(dbctx, actual); }); 264 | var p = p.then(function() { return db.addBuildResult(dbctx, actual); }); 265 | var p = p.then(function() { return db.getBuildResult(dbctx, actual); }); 266 | var p = p.then(function(br) { assert(JSON.stringify(br) == JSON.stringify(actual)); }); 267 | var p = p.then(function() { return db.disconnect(dbctx); }); 268 | var p = p.then(function() { done(); }); 269 | return p; 270 | }).catch(function(e) { done(e); }); 271 | }); 272 | 273 | test("get null build result", function(done) { 274 | db.connect(testConfig).then(function(dbctx) { 275 | var req = { 276 | toolchain: util.parseToolchain("nightly-2015-03-01"), 277 | crateName: "toml", 278 | crateVers: "1.0" 279 | }; 280 | var p = Promise.resolve(); 281 | var p = p.then(function() { return db.getBuildResult(dbctx, req); }); 282 | var p = p.then(function(br) { assert(br == null); }); 283 | var p = p.then(function() { return db.disconnect(dbctx); }); 284 | var p = p.then(function() { done(); }); 285 | return p; 286 | }).catch(function(e) { done(e); }); 287 | }); 288 | 289 | test("get result pairs", function(done) { 290 | db.connect(testConfig).then(function(dbctx) { 291 | var oldResult1 = { 292 | toolchain: util.parseToolchain("beta-2015-03-01"), 293 | crateName: "num", 294 | crateVers: "1.0", 295 | status: "success", 296 | taskId: "t1" 297 | }; 298 | var oldResult2 = { 299 | toolchain: util.parseToolchain("beta-2015-03-01"), 300 | crateName: "toml", 301 | crateVers: "1.1", 302 | status: "success", 303 | taskId: "t2" 304 | }; 305 | var newResult1 = { 306 | toolchain: util.parseToolchain("nightly-2015-03-02"), 307 | crateName: "num", 308 | crateVers: "1.0", 309 | status: 'failure', 310 | taskId: "t3" 311 | }; 312 | var newResult2 = { 313 | toolchain: util.parseToolchain("nightly-2015-03-02"), 314 | crateName: "toml", 315 | crateVers: "1.1", 316 | status: "success", 317 | taskId: "t2" 318 | }; 319 | var fromToolchain = util.parseToolchain("beta-2015-03-01"); 320 | var toToolchain = util.parseToolchain("nightly-2015-03-02"); 321 | var p = Promise.resolve(); 322 | var p = p.then(function() { return db.addBuildResult(dbctx, oldResult1); }); 323 | var p = p.then(function() { return db.addBuildResult(dbctx, oldResult2); }); 324 | var p = p.then(function() { return db.addBuildResult(dbctx, newResult1); }); 325 | var p = p.then(function() { return db.addBuildResult(dbctx, newResult2); }); 326 | var p = p.then(function() { return db.getResultPairs(dbctx, fromToolchain, toToolchain); }); 327 | var p = p.then(function(results) { 328 | assert(results[0].crateName == "num"); 329 | assert(results[0].crateVers == "1.0"); 330 | assert(results[0].from.status == "success"); 331 | assert(results[0].to.status == "failure"); 332 | assert(results[1].crateName == "toml"); 333 | assert(results[1].crateVers == "1.1"); 334 | assert(results[1].from.status == "success"); 335 | assert(results[1].to.status == "success"); 336 | }); 337 | var p = p.then(function() { return db.disconnect(dbctx); }); 338 | var p = p.then(function() { done(); }); 339 | return p; 340 | }).catch(function(e) { done(e); }); 341 | }); 342 | 343 | test("add custom toolchain", function(done) { 344 | db.connect(testConfig).then(function(dbctx) { 345 | var toolchain = util.parseToolchain("aaaaabbbbbaaaaabbbbbaaaaabbbbbaaaaabbbbb"); 346 | var custom = { 347 | toolchain: toolchain, 348 | url: "http://foo", 349 | taskId: "myTask" 350 | }; 351 | var p = Promise.resolve(); 352 | var p = p.then(function() { return db.addCustomToolchain(dbctx, custom); }); 353 | var p = p.then(function() { return db.getCustomToolchain(dbctx, toolchain); }); 354 | var p = p.then(function(c) { assert(JSON.stringify(c) == JSON.stringify(custom)); }); 355 | var p = p.then(function() { return db.disconnect(dbctx); }); 356 | var p = p.then(function() { done(); }); 357 | return p; 358 | }).catch(function(e) { done(e); }); 359 | }); 360 | 361 | }); 362 | 363 | suite("scheduler tests", function() { 364 | 365 | beforeEach(runBeforeEach); 366 | afterEach(runAfterEach); 367 | 368 | test("schedule top crates", function(done) { 369 | var dbctx; 370 | db.connect(testConfig).then(function(d) { 371 | dbctx = d; 372 | }).then(function() { 373 | var options = { toolchain: { channel: "nightly", archiveDate: "2015-03-03" }, top: 2 }; 374 | return scheduler.createSchedule(options, testConfig, dbctx); 375 | }).then(function(schedule) { 376 | return db.disconnect(dbctx).then(function() { return schedule; } ); 377 | }).then(function(schedule) { 378 | assert(schedule[0].crateName == "winapi"); 379 | assert(schedule[schedule.length - 1].crateName == "libc"); 380 | done(); 381 | }).catch(function(e) { done(e); }).done(); 382 | }); 383 | 384 | test("schedule most recent", function(done) { 385 | var dbctx; 386 | db.connect(testConfig).then(function(d) { 387 | dbctx = d; 388 | }).then(function() { 389 | var options = { toolchain: { channel: "nightly", archiveDate: "2015-03-03" }, top: 2, mostRecentOnly: true }; 390 | return scheduler.createSchedule(options, testConfig, dbctx); 391 | }).then(function(schedule) { 392 | return db.disconnect(dbctx).then(function() { return schedule; } ); 393 | }).then(function(schedule) { 394 | assert(schedule.length == 2); 395 | assert(schedule[0].crateName == "winapi"); 396 | assert(schedule[1].crateName == "libc"); 397 | done(); 398 | }).catch(function(e) { done(e); }).done(); 399 | }); 400 | 401 | test("schedule skip existing results", function(done) { 402 | var toolchain = { channel: "nightly", archiveDate: "2015-03-03" }; 403 | var dbctx; 404 | db.connect(testConfig).then(function(d) { 405 | dbctx = d; 406 | 407 | // Add a successful result, that we don't want rescheduled again 408 | var buildResult = { 409 | toolchain: toolchain, 410 | crateName: "libc", 411 | crateVers: "0.1.6", 412 | status: "success", 413 | taskId: "whatever" 414 | }; 415 | return db.addBuildResult(dbctx, buildResult); 416 | }).then(function() { 417 | var options = { 418 | toolchain: toolchain, 419 | top: 2, 420 | mostRecentOnly: true, 421 | skipExisting: true 422 | }; 423 | return scheduler.createSchedule(options, testConfig, dbctx); 424 | }).then(function(schedule) { 425 | return db.disconnect(dbctx).then(function() { return schedule; } ); 426 | }).then(function(schedule) { 427 | assert(schedule.length == 1); 428 | assert(schedule[0].crateName == "winapi"); 429 | done(); 430 | }).catch(function(e) { done(e); }).done(); 431 | }); 432 | 433 | }); 434 | 435 | suite("report tests", function() { 436 | 437 | beforeEach(runBeforeEach); 438 | afterEach(runAfterEach); 439 | 440 | test("current report", function(done) { 441 | reports.createCurrentReport("2015-03-03", testConfig).then(function(report) { 442 | assert(report.nightly == "2015-02-26"); 443 | assert(report.beta == "2015-02-20"); 444 | assert(report.stable == null); 445 | done(); 446 | }).catch(function(e) { done(e); }); 447 | }); 448 | 449 | test("weekly report", function(done) { 450 | var oldResultWorking = { 451 | toolchain: util.parseToolchain("beta-2015-02-20"), 452 | crateName: "num", 453 | crateVers: "1.0", 454 | status: "success", 455 | taskId: "t" 456 | }; 457 | var newResultWorking = { 458 | toolchain: util.parseToolchain("nightly-2015-02-26"), 459 | crateName: "num", 460 | crateVers: "1.0", 461 | status: "success", 462 | taskId: "t" 463 | }; 464 | var oldResultNotWorking = { 465 | toolchain: util.parseToolchain("beta-2015-02-20"), 466 | crateName: "op", 467 | crateVers: "1.0", 468 | status: 'failure', 469 | taskId: "t" 470 | }; 471 | var newResultNotWorking = { 472 | toolchain: util.parseToolchain("nightly-2015-02-26"), 473 | crateName: "op", 474 | crateVers: "1.0", 475 | status: 'failure', 476 | taskId: "t" 477 | }; 478 | var oldResultRegressed = { 479 | toolchain: util.parseToolchain("beta-2015-02-20"), 480 | crateName: "plot", 481 | crateVers: "1.0", 482 | status: "success", 483 | taskId: "t" 484 | }; 485 | var newResultRegressed = { 486 | toolchain: util.parseToolchain("nightly-2015-02-26"), 487 | crateName: "plot", 488 | crateVers: "1.0", 489 | status: 'failure', 490 | taskId: "t" 491 | }; 492 | var oldResultFixed = { 493 | toolchain: util.parseToolchain("beta-2015-02-20"), 494 | crateName: "quux", 495 | crateVers: "1.0", 496 | status: 'failure', 497 | taskId: "t" 498 | }; 499 | var newResultFixed = { 500 | toolchain: util.parseToolchain("nightly-2015-02-26"), 501 | crateName: "quux", 502 | crateVers: "1.0", 503 | status: "success", 504 | taskId: "t" 505 | }; 506 | var dbctx; 507 | db.connect(testConfig).then(function(d) { 508 | dbctx = d; 509 | }).then(function() { return db.addBuildResult(dbctx, oldResultWorking); 510 | }).then(function() { return db.addBuildResult(dbctx, newResultWorking); 511 | }).then(function() { return db.addBuildResult(dbctx, oldResultNotWorking); 512 | }).then(function() { return db.addBuildResult(dbctx, newResultNotWorking); 513 | }).then(function() { return db.addBuildResult(dbctx, oldResultRegressed); 514 | }).then(function() { return db.addBuildResult(dbctx, newResultRegressed); 515 | }).then(function() { return db.addBuildResult(dbctx, oldResultFixed); 516 | }).then(function() { return db.addBuildResult(dbctx, newResultFixed); 517 | }).then(function() { 518 | return reports.createWeeklyReport("2015-03-03", dbctx, testConfig); 519 | }).then(function(report) { 520 | return db.disconnect(dbctx).then(function() { return report; } ); 521 | }).then(function(report) { 522 | assert(report.date == "2015-03-03"); 523 | 524 | assert(report.currentReport.nightly == "2015-02-26"); 525 | assert(report.currentReport.beta == "2015-02-20"); 526 | assert(report.currentReport.stable == null); 527 | 528 | assert(report.beta.statuses.length == 0); 529 | assert(report.nightly.statuses[0].status == "working"); 530 | assert(report.nightly.statuses[1].status == "broken"); 531 | assert(report.nightly.statuses[2].status == "regressed"); 532 | assert(report.nightly.statuses[3].status == "fixed"); 533 | 534 | assert(report.nightly.statusSummary.working == 1); 535 | assert(report.nightly.statusSummary.broken == 1); 536 | assert(report.nightly.statusSummary.regressed == 1); 537 | assert(report.nightly.statusSummary.fixed == 1); 538 | 539 | assert(report.nightly.regressions[0].crateName = "plot"); 540 | 541 | done(); 542 | }).catch(function(e) { done(e) }); 543 | }); 544 | 545 | test("weekly report prune regressed leaves", function(done) { 546 | var oldResultRegressed = { 547 | toolchain: util.parseToolchain("beta-2015-02-20"), 548 | crateName: "piston", 549 | crateVers: "0.0.7", 550 | status: "success", 551 | taskId: "t" 552 | }; 553 | var newResultRegressed = { 554 | toolchain: util.parseToolchain("nightly-2015-02-26"), 555 | crateName: "piston", 556 | crateVers: "0.0.7", 557 | status: 'failure', 558 | taskId: "t" 559 | }; 560 | var oldResultRegressedDep = { 561 | toolchain: util.parseToolchain("beta-2015-02-20"), 562 | crateName: "pistoncore-input", 563 | crateVers: "0.0.5", 564 | status: "success", 565 | taskId: "t" 566 | }; 567 | var newResultRegressedDep = { 568 | toolchain: util.parseToolchain("nightly-2015-02-26"), 569 | crateName: "pistoncore-input", 570 | crateVers: "0.0.5", 571 | status: 'failure', 572 | taskId: "t" 573 | }; 574 | var dbctx; 575 | db.connect(testConfig).then(function(d) { 576 | dbctx = d; 577 | }).then(function() { return db.addBuildResult(dbctx, oldResultRegressed); 578 | }).then(function() { return db.addBuildResult(dbctx, newResultRegressed); 579 | }).then(function() { return db.addBuildResult(dbctx, oldResultRegressedDep); 580 | }).then(function() { return db.addBuildResult(dbctx, newResultRegressedDep); 581 | }).then(function() { 582 | return reports.createWeeklyReport("2015-03-03", dbctx, testConfig); 583 | }).then(function(report) { 584 | return db.disconnect(dbctx).then(function() { return report; } ); 585 | }).then(function(report) { 586 | 587 | // 'piston' is not a root regression 588 | assert(report.nightly.rootRegressions.length == 1); 589 | assert(report.nightly.rootRegressions[0].crateName == "pistoncore-input"); 590 | 591 | done(); 592 | }).catch(function(e) { done(e) }); 593 | }); 594 | 595 | test("weekly report sort by popularity", function(done) { 596 | var oldResultRegressed = { 597 | toolchain: util.parseToolchain("beta-2015-02-20"), 598 | crateName: "piston", 599 | crateVers: "0.0.7", 600 | status: "success", 601 | taskId: "t" 602 | }; 603 | var newResultRegressed = { 604 | toolchain: util.parseToolchain("nightly-2015-02-26"), 605 | crateName: "piston", 606 | crateVers: "0.0.7", 607 | status: 'failure', 608 | taskId: "t" 609 | }; 610 | var oldResultRegressedDep = { 611 | toolchain: util.parseToolchain("beta-2015-02-20"), 612 | crateName: "url", 613 | crateVers: "0.0.5", 614 | status: "success", 615 | taskId: "t" 616 | }; 617 | var newResultRegressedDep = { 618 | toolchain: util.parseToolchain("nightly-2015-02-26"), 619 | crateName: "url", 620 | crateVers: "0.0.5", 621 | status: 'failure', 622 | taskId: "t" 623 | }; 624 | var dbctx; 625 | db.connect(testConfig).then(function(d) { 626 | dbctx = d; 627 | }).then(function() { return db.addBuildResult(dbctx, oldResultRegressed); 628 | }).then(function() { return db.addBuildResult(dbctx, newResultRegressed); 629 | }).then(function() { return db.addBuildResult(dbctx, oldResultRegressedDep); 630 | }).then(function() { return db.addBuildResult(dbctx, newResultRegressedDep); 631 | }).then(function() { 632 | return reports.createWeeklyReport("2015-03-03", dbctx, testConfig); 633 | }).then(function(report) { 634 | return db.disconnect(dbctx).then(function() { return report; } ); 635 | }).then(function(report) { 636 | 637 | assert(report.nightly.regressions[0].crateName == "url"); 638 | assert(report.nightly.regressions[1].crateName == "piston"); 639 | 640 | done(); 641 | }).catch(function(e) { done(e) }); 642 | }); 643 | 644 | }); 645 | 646 | suite("live network tests", function() { 647 | 648 | beforeEach(runBeforeEach); 649 | afterEach(runAfterEach); 650 | 651 | test("download dist index", function(done) { 652 | var p = dist.downloadIndex(liveConfig); 653 | p = p.then(function(index) { 654 | done() 655 | }); 656 | p = p.catch(function(e) { done(e) }); 657 | }); 658 | 659 | test("load crates", function(done) { 660 | var p = crates.cloneIndex(liveConfig); 661 | p = p.then(function() { return crates.loadCrates(liveConfig); }); 662 | p = p.then(function(crates) { 663 | assert(crates.length > 0); 664 | done(); 665 | }); 666 | p = p.catch(function(e) { done(e) }); 667 | }); 668 | 669 | }); 670 | 671 | -------------------------------------------------------------------------------- /test/dist/2015-03-03/channel-rust-beta: -------------------------------------------------------------------------------- 1 | rust-1.0.0-alpha.2-i686-apple-darwin.pkg 2 | rust-1.0.0-alpha.2-i686-apple-darwin.tar.gz 3 | rust-1.0.0-alpha.2-i686-pc-windows-gnu.exe 4 | rust-1.0.0-alpha.2-i686-pc-windows-gnu.msi 5 | rust-1.0.0-alpha.2-i686-pc-windows-gnu.tar.gz 6 | rust-1.0.0-alpha.2-i686-unknown-linux-gnu.tar.gz 7 | rust-1.0.0-alpha.2-x86_64-apple-darwin.pkg 8 | rust-1.0.0-alpha.2-x86_64-apple-darwin.tar.gz 9 | rust-1.0.0-alpha.2-x86_64-pc-windows-gnu.exe 10 | rust-1.0.0-alpha.2-x86_64-pc-windows-gnu.msi 11 | rust-1.0.0-alpha.2-x86_64-pc-windows-gnu.tar.gz 12 | rust-1.0.0-alpha.2-x86_64-unknown-linux-gnu.tar.gz 13 | -------------------------------------------------------------------------------- /test/versions/toml/0.1.18: -------------------------------------------------------------------------------- 1 | {"version":{"crate":"toml","created_at":"2015-02-25T22:53:39Z","dl_path":"/api/v1/crates/toml/0.1.18/download","downloads":511,"features":{},"id":5627,"links":{"authors":"/api/v1/crates/toml/0.1.18/authors","dependencies":"/api/v1/crates/toml/0.1.18/dependencies","version_downloads":"/api/v1/crates/toml/0.1.18/downloads"},"num":"0.1.18","updated_at":"2015-02-25T22:53:39Z","yanked":false}} -------------------------------------------------------------------------------- /test/versions/toml/0.1.7: -------------------------------------------------------------------------------- 1 | {"version":{"crate":"toml","created_at":"2015-01-25T22:53:39Z","dl_path":"/api/v1/crates/toml/0.1.18/download","downloads":511,"features":{},"id":5627,"links":{"authors":"/api/v1/crates/toml/0.1.18/authors","dependencies":"/api/v1/crates/toml/0.1.18/dependencies","version_downloads":"/api/v1/crates/toml/0.1.18/downloads"},"num":"0.1.18","updated_at":"2015-02-25T22:53:39Z","yanked":false}} --------------------------------------------------------------------------------