├── AD-AE-AppendixForm.pdf ├── LICENSE ├── README.md ├── SC-reproducibility-initiative.md └── collect_environment.sh /AD-AE-AppendixForm.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SC-Tech-Program/Author-Kit/7f8d0b2ea3329b4b566246077d2ca20177b6c993/AD-AE-AppendixForm.pdf -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | BSD 3-Clause License 2 | 3 | Copyright (c) 2018, SC-Tech-Program 4 | All rights reserved. 5 | 6 | Redistribution and use in source and binary forms, with or without 7 | modification, are permitted provided that the following conditions are met: 8 | 9 | * Redistributions of source code must retain the above copyright notice, this 10 | list of conditions and the following disclaimer. 11 | 12 | * Redistributions in binary form must reproduce the above copyright notice, 13 | this list of conditions and the following disclaimer in the documentation 14 | and/or other materials provided with the distribution. 15 | 16 | * Neither the name of the copyright holder nor the names of its 17 | contributors may be used to endorse or promote products derived from 18 | this software without specific prior written permission. 19 | 20 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" 21 | AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 22 | IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 23 | DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE 24 | FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL 25 | DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR 26 | SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER 27 | CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, 28 | OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 29 | OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 30 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Author-Kit 2 | 3 | This repository contains materials to help submitting authors of the SC Conference Technical Program. 4 | 5 | > **Find answers to Frequently Asked Questions in the [`FAQs`](https://github.com/SC-Tech-Program/FAQs) repository.** 6 | 7 | ## AD/AE Appendix 8 | 9 | Artifact Description (AD) appendices are _required_ of all paper submissions to the SC19 Technical Program. 10 | Artifact Evaluation (AE) appendices are optional but encouraged. 11 | 12 | The AD/AE Appendix will be automatically generated during submission, after the authors respond an online form. 13 | All authors are encouraged to familiarize themselves with the questions in this form, with plenty of time before sumbission. 14 | See the PDF printout of the online form: [AD-AE-AppendixForm.pdf](https://github.com/SC-Tech-Program/Author-Kit/blob/master/AD-AE-AppendixForm.pdf) 15 | 16 | ## Reproducibility Resources 17 | 18 | **Example script for machine-generated environment data** 19 | 20 | In an effort to facilitate preparing the SC Artifact Description appendix, we provide a sample Bash shell script, [`collect_environment.sh`](collect_environment.sh), that queries the operating system for a variety of information. As written, the script is somewhat Linux-specific, but it should serve to provide ideas for similar scripts that run under other operating systems. The script should be run on one of the nodes that was used to gather the paper's experimental data (i.e., a compute node, not a head node). 21 | 22 | At the time of this writing, `collect_environment.sh` invokes the following commands: 23 | 24 | * [`lsb_release`](https://linux.die.net/man/1/lsb_release): report the Linux distribution 25 | * [`uname`](https://linux.die.net/man/1/uname): report the kernel version and base architecture 26 | * [`lscpu`](https://linux.die.net/man/1/lscpu) (or `cat /proc/cpuinfo`): report CPU characteristics 27 | * `cat /proc/meminfo`: report virtual and physical memory capacity 28 | * [`lspci`](https://linux.die.net/man/8/lspci): list all PCI devices 29 | * [`lshw`](https://linux.die.net/man/1/lshw): list all hardware devices 30 | * [`lsblk`](https://linux.die.net/man/8/lsblk): list all block devices 31 | * [`lsscsi`](https://linux.die.net/man/8/lsscsi): list all SCSI devices 32 | * [`env`](https://linux.die.net/man/1/env): list all environment variables currently in use 33 | * [`module list`](https://linux.die.net/man/1/module): list all loaded [Environment Modules](http://modules.sourceforge.net/) 34 | * [`nvidia-smi`](https://developer.nvidia.com/nvidia-system-management-interface): report characteristics and configuration of NVIDIA GPUs 35 | 36 | There is some redundancy in information provided by the preceding tools, but note that not every tools is available on every system. The goal is to acquire as much information as possible on as many types of systems as possible. 37 | -------------------------------------------------------------------------------- /SC-reproducibility-initiative.md: -------------------------------------------------------------------------------- 1 | # SC Reproducibility Initiative 2 | 3 | **TL;DR** — After 3 years of increasing momentum, the SC Reproducibility Initiative makes Artifact Description (AD) 4 | Appendices mandatory for all papers submitted to the SC19 Technical Program (but optional for workshop papers and 5 | posters). Artifact Evaluation (AE) is still optional, but highly encouraged. AD/AE Appendices will now be auto-generated from author 6 | responses to a standard form, embedded in the SC online submission system. In 2019, the SC Reproducibility Initiative 7 | introduces three new tracks/committees, under the SC Reproducibility Chair: (1) AD/AE Appendices, (2) Reproducibility 8 | Challenge, (3) Journal Special Issue. 9 | 10 | ## New SC Reproducibility Tracks/Committees 11 | 12 | **AD/AE Appendices.** 13 | 14 | This committee has the following charge: 15 | * Implement a program of actions to mentor authors in the preparation of the appendices. This may include webinars, 16 | blog posts, expanding the reproducibility FAQs, etc. 17 | * Inspect the appendices of submitted papers, follow artifact URLs to check their contents, and support authors with 18 | appendix improvement, if needed. 19 | 20 | **Reproducibility Challenge.** 21 | 22 | This committee has the following charge: 23 | * Select a paper accepted to the previous conference, to be used as source of the Reproducibility Challenge in the 24 | Students Cluster Competition (SCC) of the upcoming conference. 25 | * Work with the authors of the selected paper to build the challenge benchmark for the SCC teams. 26 | * Continue working until the time of the conference in crafting the rules and requirements for the challenge, and 27 | ensuring compatibility with the various hardware used by the SCC teams. 28 | 29 | **Journal Special Issue.** 30 | 31 | This committee has the following charge: 32 | * Review full papers submitted by the SCC teams for publication in the journal special issue, reporting on the 33 | results of the efforts to reproduce the challenge benchmark. 34 | * Work with student teams to assist them in revision of their papers, for a successful final publication of the 35 | reports in the journal special issue. 36 | 37 | ## Artifact Description / Article Evaluation (AD/AE) Appendix Form 38 | 39 | All SC19 submitting authors to the Technical Program must complete this form to describe the computational 40 | artifacts their results rely on: software, data, and hardware. The form is standard, and embedded in the 41 | conference submission system. 42 | 43 | **Note:** If your paper used no computational artifacts, you will respond "No" to the first question and be done. 44 | 45 | Submissions will be double-blind reviewed, and reviewers will have all the information in the AD/AE form _except_ 46 | for the URLs pointing to the artifacts. The paper and the appendix, including URLs to artifacts, will be available 47 | to the AD/AE Appendices Committee. The committee may provide feedback to authors in a double-open arrangement 48 | (authors and committee members are known to each other). The AD/AE Appendices Committee will not share any information 49 | with the PC other than to confirm whether artifacts indicated as available are in fact available. "Available" means 50 | that the provided URLs properly resolve, and that the author-created artifacts are persistently archived and have a 51 | global, unique identifier. Accepted papers with available artifacts will display an ACM badge. 52 | 53 | **Additional notes:** 54 | * The form targets both Artifact Description (required) and Evaluation (optional). 55 | * Availability of software, data, and hardware artifacts must be explicitly indicated. 56 | * Papers should still be self contained: they should describe the experimental methodology enough for the PC to 57 | evaluate the contribution. 58 | * Sample scripts for machine-generated environment data will be provided at: https://github.com/SC-Tech-Program 59 | * Authors are encouraged to familiarize themselves early with the AD/AE Appendix requirements, and to direct questions to: 60 | ad-ae-appendices@info.supercomputing.org 61 | 62 | 63 | ## History of the SC Reproducibility Initiative 64 | 65 | **2015** — The SC steering committee approved the reproducibility initiative. Authors of SC15 papers were invited to submit 66 | an AD Appendix after the conference: one paper did so, became the source for the SC16 Student Cluster Competition reproducibility 67 | challenge and the first SC paper to display an ACM badge. 68 | 69 | **2016** — Authors submitting to the SC16 conference could optionally submit an AD Appendix: 9 authors submitted one, 3 were 70 | finalists and one was selected to become the source for the SC17 Student Cluster Competition reproducibility challenge. 71 | 72 | **2017** — SC made the AD Appendix a requirement to be considered for the Best Paper or Best Student Paper awards. SC17 also 73 | introduced the Computational Results Analysis (CRA) appendix. 40% of submitted and 50% of accepted papers included an AD appendix; 74 | 9 submitted papers (6 accepted) included a CRA appendix. 75 | 76 | **2018** — SC extended the option of submitting AD appendices to workshops and posters. The CRA appendix was renamed Artifact 77 | Evaluation (AE) appendix, and limited to 4 pp. AD appendices were limited to 2 pp. and remained optional but required for 78 | consideration as Best Paper/Best Student Paper, and also Best Poster/Best Student Poster. 79 | 80 | **2019** — AD appendices will be mandatory for all submissions, AE appendices are still optional, and both will be submitted 81 | via a standard form in the conference submission system. Three new TP tracks, with their respective committees and chairs, 82 | are introduced in support of the SC Reproducibility Initiative. 83 | 84 | 85 | ## Rationale and results of Reproducibility at SC 86 | 87 | The SC Conference Series is committed to introducing activities that highlight, enhance, and reward participant efforts to 88 | improve reproducibility. Each year, authors have engaged more with the Reproducibility Initiative, and the benefits have become 89 | more clear to our community. 90 | 91 | The AD/AE Appendices improve the _quality_ of SC papers, and increase _efficiency_ of our collective endeavors: 92 | * The AD appendix provides information (details of software, hardware and data) that should be part of any publication with computational results. 93 | * The new AD/AE Appendices form (starting 2019) standardizes the items that authors should include. 94 | * If an SC paper used no computational artifacts (software, hardware or data), the author will click “No” in answer to the first question, and be done. 95 | * In the AE appendix, authors describe how they verified and validated their computational results, supporting their trustworthiness and providing best-practice descriptions to the community that can be adopted by others who want to improve their own efforts. 96 | * AD/AE Appendices are reviewed in a double-open format by a dedicated committee, independent of the Technical Program committee (which maintains a double-blind process). 97 | 98 | ## Engaging the Student Cluster Competition 99 | 100 | The SC Reproducibility Challenge committee, now a part of the Technical Program, works with the SC Reproducibility Chair 101 | to select a paper from the previous conference, which will be used as source of the reproducibility challenge the next year. 102 | This committee will work with the authors of the selected paper to develop the challenge benchmark, and ensure the SCC teams 103 | have the best chance at succeeding. The SCC teams write extended reports of their results and their efforts with the 104 | reproducibility challenge, which are submitted for a journal special issue (where they are peer reviewed before acceptance). 105 | The SC paper selected for the reproducibility challenge receives a special recognition at the SC Awards Ceremony. 106 | 107 | 108 | ## SC Reproducibility Resources for Authors 109 | 110 | Please visit the GitHub repository to find resources and answers to FAQs. 111 | https://github.com/SC-Tech-Program 112 | 113 | --- 114 | *— By the SC19 Reproducibility Chair, Lorena A. Barba, PhD, derived in part from the [SC18 Reproducibility Initiative](https://sc18.supercomputing.org/submit/sc-reproducibility-initiative/) web page.* 115 | -------------------------------------------------------------------------------- /collect_environment.sh: -------------------------------------------------------------------------------- 1 | #! /bin/sh 2 | 3 | # Linux data-gathering commands; adjust as necessary for your platform. 4 | # 5 | # Be sure to remove any information from the output that would violate 6 | # SC's double-blind review policies. 7 | 8 | env | sed "s/$USER/USER/g" 9 | set -x 10 | lsb_release -a 11 | uname -a 12 | lscpu || cat /proc/cpuinfo 13 | cat /proc/meminfo 14 | inxi -F -c0 15 | lsblk -a 16 | lsscsi -s 17 | module list 18 | nvidia-smi 19 | (lshw -short -quiet -sanitize || lspci) | cat 20 | --------------------------------------------------------------------------------