├── README.md └── templates ├── responsibleNLPresearch.docx ├── responsibleNLPresearch.pdf └── responsibleNLPresearch.tex /README.md: -------------------------------------------------------------------------------- 1 | # responsibleNLPresearch 2 | 3 | This repository contains templates and other documents regarding responsible NLP research practices in the ACL community. 4 | 5 | You can fill in: 6 | * the Word template (in Word) 7 | * the PDF generated by compiling the LaTex 8 | * the LaTex template (see the usage notes in the source) 9 | 10 | Suggestions for improving either template are most welcome; pull requests welcome. 11 | 12 | For the current version of the responsible NLP research guidelines, please see https://aclrollingreview.org/responsibleNLPresearch/. 13 | -------------------------------------------------------------------------------- /templates/responsibleNLPresearch.docx: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/acl-org/responsibleNLPresearch/ee6289ee7c61a54b281afbca15ebe0925cc97b96/templates/responsibleNLPresearch.docx -------------------------------------------------------------------------------- /templates/responsibleNLPresearch.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/acl-org/responsibleNLPresearch/ee6289ee7c61a54b281afbca15ebe0925cc97b96/templates/responsibleNLPresearch.pdf -------------------------------------------------------------------------------- /templates/responsibleNLPresearch.tex: -------------------------------------------------------------------------------- 1 | %----------------------------------------------------------------------------------- 2 | % Responsible NLP research checklist version 0.1 3 | % 4 | % This checklist was created by Marine Carpuat, Marie-Catherine de Marneffe and Ivan Vladimir Meza Ruiz, 5 | % the NAACL 2022 program chairs, working with Jesse Dodge and the ARR editors-in-chief Pascale Fung, 6 | % Goran Glavas, Sebastian Riedel and Amanda Stent. Source updated by Vilém Zouhar. 7 | % Additional feedback was provided by Anna Rogers, Margot Mieskes and the NAACL 2022 reproducibility chairs. 8 | % The checklist was reviewed by the ACL Ethics Committee. 9 | % 10 | % Subject to change. 11 | %------------------------------------------------------------------------------------ 12 | 13 | \documentclass{article} 14 | \usepackage[utf8]{inputenc} 15 | \usepackage{xcolor} 16 | \usepackage[letterpaper, total={6.5in, 9in}]{geometry} 17 | \usepackage{hyperref} 18 | \usepackage{parskip} 19 | \usepackage{longtable} 20 | \newcounter{myenumi} 21 | \newcounter{myenumii}[myenumi] 22 | \usepackage{titling} 23 | \usepackage{titlesec} 24 | \usepackage{etoolbox} 25 | 26 | % Enviroment for numbered tabular 27 | \newenvironment{enumtabular}[2] 28 | {% 29 | \setcounter{myenumi}{0}% 30 | \setcounter{myenumii}{0}% 31 | \renewcommand{\item}{#1\refstepcounter{myenumi}\themyenumi.\ \ }% 32 | \begin{longtable}{#2} 33 | } 34 | {\end{longtable}} 35 | 36 | 37 | % Command for choice menu 38 | \newcommand{\cm}[3]{\mbox{\ChoiceMenu[height=0.3cm,width=0.3cm,bordercolor=,name=#1,radio,radiosymbol=\ding{108},default=#3]{}{#2}}} 39 | 40 | % Command for text field 41 | \newcommand{\tf}[3][0.78]{\mbox{\TextField[bordercolor=,name=#2,value={#3},multiline=true,height=4em, width=#1\textwidth]{\noindent \parbox{0.11\textwidth}{Section or\\Justification}}}} 42 | 43 | % Command for checkbox 44 | \newcommand{\cb}[1]{\mbox{\CheckBox[bordercolor=,name=#1,height=0.4cm,width=0.4cm]{}{\ }}} 45 | 46 | % Fix choice menu 47 | \makeatletter 48 | \patchcmd{\HyField@FlagsRadioButton}{\HyField@SetFlag{Ff}{Radio}}{}{}{} 49 | \makeatother 50 | 51 | \titleformat{\section}{\Large\bfseries}{\Alph{section}}{1em}{} 52 | \titleformat{\subsection}{\normalfont\bfseries}{\Alph{section}\arabic{subsection}}{1em}{} 53 | 54 | \title{Responsible NLP Research Checklist\vspace{-3em}} 55 | \date{} 56 | 57 | \begin{document} 58 | \maketitle 59 | 60 | Members of the ACL are responsible for adhering to the \href{https://www.aclweb.org/portal/content/acl-code-ethics}{ACL code of ethics}. The ARR Responsible NLP Research checklist is designed to encourage best practices for responsible research, addressing issues of research ethics, societal impact and reproducibility. 61 | 62 | Please read the \href{https://aclrollingreview.org/responsibleNLPresearch/}{Responsible NLP Research checklist guidelines} for information on how to answer these questions. Note that not answering positively to a question is not grounds for rejection. 63 | 64 | All supporting evidence can appear either in the main paper or the supplemental material. For each question, if you answer {\bf Yes}, provide the section number; if you answer {\bf No}, provide a justification. 65 | 66 | Please do \textbf{not} modify, reorder, delete or add questions, question options or other wording of this document. 67 | 68 | % USAGE: 69 | % Use the last argument in the \cm and \tf commands to mark your answer, e.g.: 70 | % \cm{mainClaims}{Yes,No,N/A}{N/A} 71 | % \tf[0.85]{mainClaimsJustification}{This does not apply...} 72 | 73 | \section{For every submission} 74 | 75 | \subsection{Did you discuss the \textit{limitations} of your work?} 76 | If you answer {\bf Yes}, provide the section number; if you answer {\bf No}, provide a justification. \\[0.3cm] 77 | \begin{Form} 78 | \begin{tabular}{l} 79 | \cm{mainClaims}{Yes,No,N/A}{}\\[0.2cm] 80 | \tf[0.85]{mainClaimsJustification}{} 81 | \end{tabular} 82 | \end{Form} \\[0.3cm] 83 | 84 | \subsection{Did you discuss any potential \textit{risks} of your work?} 85 | If you answer {\bf Yes}, provide the section number; if you answer {\bf No}, provide a justification. \\[0.3cm] 86 | \begin{Form} 87 | \begin{tabular}{l} 88 | \cm{risks}{Yes,No,N/A}{}\\[0.2cm] 89 | \tf[0.85]{risksJustification}{} 90 | \end{tabular} 91 | \end{Form} 92 | 93 | \subsection{Do the abstract and introduction summarize the paper’s main claims?} 94 | If you answer {\bf Yes}, provide the section number; if you answer {\bf No}, provide a justification. \\[0.3cm] 95 | \begin{Form} 96 | \begin{tabular}{l} 97 | \cm{abstractIntro}{Yes,No,N/A}{}\\[0.2cm] 98 | \tf[0.85]{abstractIntroJustification}{} 99 | \end{tabular} 100 | \end{Form} 101 | 102 | \section{Did you use or create \textit{scientific artifacts}?} 103 | If you answer {\bf Yes}, provide the section number; if you answer {\bf No}, you can skip the rest of this section. \\[0.3cm] 104 | \begin{Form} 105 | \begin{tabular}{l} 106 | \cm{createArtifacts}{Yes,No}{}\\[0.2cm] 107 | \end{tabular} 108 | \end{Form} 109 | 110 | If yes: 111 | \subsection{Did you cite the creators of artifacts you used?} 112 | If you answer {\bf Yes}, provide the section number; if you answer {\bf No}, provide a justification. \\[0.3cm] 113 | \begin{Form} 114 | \begin{tabular}{l} 115 | \cm{citeCreators}{Yes,No,N/A}{}\\[0.2cm] 116 | \tf{citeCreatorsJustification}{} 117 | \end{tabular} 118 | \end{Form} \\[0.3cm] 119 | 120 | \subsection{Did you discuss the \textit{license or terms} for use and/or distribution of any artifacts?} 121 | If you answer {\bf Yes}, provide the section number; if you answer {\bf No}, provide a justification. \\[0.3cm] 122 | \begin{Form} 123 | \begin{tabular}{l} 124 | \cm{legalGrounds}{Yes,No,N/A}{}\\[0.2cm] 125 | \tf{legalGroundsJustification}{} 126 | \end{tabular} 127 | \end{Form} \\[0.3cm] 128 | 129 | \subsection{Did you discuss if your use of existing artifact(s) was consistent with their \textit{intended use}, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)?} 130 | If you answer {\bf Yes}, provide the section number; if you answer {\bf No}, provide a justification. \\[0.3cm] 131 | \begin{Form} 132 | \begin{tabular}{l} 133 | \cm{intendedUse}{Yes,No,N/A}{}\\[0.2cm] 134 | \tf{intendedUseJustification}{} 135 | \end{tabular} 136 | \end{Form} \\[0.3cm] 137 | 138 | \subsection{Did you discuss the steps taken to check whether the data that was collected/used contains any \textit{information that names or uniquely identifies individual people} or \textit{offensive content}, and the steps taken to protect / anonymize it?} 139 | If you answer {\bf Yes}, provide the section number; if you answer {\bf No}, provide a justification. \\[0.3cm] 140 | \begin{Form} 141 | \begin{tabular}{l} 142 | \cm{personallyIdentifiableInformationOrOffensiveContent}{Yes,No,N/A}{}\\[0.2cm] 143 | \tf{personallyIdentifiableInformationOrOffensiveContentJustification}{} 144 | \end{tabular} 145 | \end{Form} \\[0.3cm] 146 | 147 | \subsection{Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.?} 148 | If you answer {\bf Yes}, provide the section number; if you answer {\bf No}, provide a justification. \\[0.3cm] 149 | \begin{Form} 150 | \begin{tabular}{l} 151 | \cm{documentation}{Yes,No,N/A}{}\\[0.2cm] 152 | \tf{documentationJustification}{} 153 | \end{tabular} 154 | \end{Form} \\[0.3cm] 155 | 156 | \subsection{Did you report relevant statistics like the number of examples, details of train/test/dev splits, etc. for the data that you used/created?} 157 | If you answer {\bf Yes}, provide the section number; if you answer {\bf No}, provide a justification. \\[0.3cm] 158 | \begin{Form} 159 | \begin{tabular}{l} 160 | \cm{relevantStatistics}{Yes,No,N/A}{}\\[0.2cm] 161 | \tf{relevantStatisticsJustification}{} 162 | \end{tabular} 163 | \end{Form} \\[0.3cm] 164 | 165 | \section{Did you run \textit{computational experiments}?} 166 | If you answer {\bf Yes}, provide the section number; if you answer {\bf No}, you can skip the rest of this section. \\[0.3cm] 167 | \begin{Form} 168 | \begin{tabular}{l} 169 | \cm{computationalExperiments}{Yes,No}{} 170 | \end{tabular} 171 | \end{Form} 172 | 173 | If yes: 174 | \subsection{Did you report the \textit{number of parameters} in the models used, the \textit{total computational budget} (e.g., GPU hours), and \textit{computing infrastructure} used?} 175 | If you answer {\bf Yes}, provide the section number; if you answer {\bf No}, provide a justification. \\[0.3cm] 176 | \begin{Form} 177 | \begin{tabular}{l} 178 | \cm{reportReproducibility}{Yes,No,N/A}{}\\[0.2cm] 179 | \tf{reportReproducibilityJustification}{} 180 | \end{tabular} 181 | \end{Form} \\[0.3cm] 182 | 183 | \subsection{Did you discuss the experimental setup, including \textit{hyperparameter search} and \textit{best-found hyperparameter} values?} 184 | If you answer {\bf Yes}, provide the section number; if you answer {\bf No}, provide a justification. \\[0.3cm] 185 | \begin{Form} 186 | \begin{tabular}{l} 187 | \cm{bestFoundHyperparameter}{Yes,No,N/A}{}\\[0.2cm] 188 | \tf{bestFoundHyperparameterJustification}{} 189 | \end{tabular} 190 | \end{Form} \\[0.3cm] 191 | 192 | \subsection{Did you report \textit{descriptive statistics} about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run?} 193 | If you answer {\bf Yes}, provide the section number; if you answer {\bf No}, provide a justification. \\[0.3cm] 194 | \begin{Form} 195 | \begin{tabular}{l} 196 | \cm{descriptiveStatistics}{Yes,No,N/A}{}\\[0.2cm] 197 | \tf{descriptiveStatisticsJustification}{} 198 | \end{tabular} 199 | \end{Form} \\[0.3cm] 200 | 201 | \subsection{If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)?} 202 | If you answer {\bf Yes}, provide the section number; if you answer {\bf No}, provide a justification. \\[0.3cm] 203 | \begin{Form} 204 | \begin{tabular}{l} 205 | \cm{existingPackages}{Yes,No,N/A}{}\\[0.2cm] 206 | \tf{existingPackagesJustification}{} 207 | \end{tabular} 208 | \end{Form} \\[0.3cm] 209 | 210 | \section{Did you use \textit{human annotators} (e.g., crowdworkers) or \textit{research with human subjects}?} If you answer {\bf Yes}, provide the section number; if you answer {\bf No}, you can skip the rest of this section. \\[0.3cm] 211 | \begin{Form} 212 | \begin{tabular}{l} 213 | \cm{hummanAnnotators}{Yes,No}{}\\ 214 | \end{tabular} 215 | \end{Form} 216 | 217 | If yes: 218 | \subsection{Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.?} 219 | If you answer {\bf Yes}, provide the section number; if you answer {\bf No}, provide a justification. \\[0.3cm] 220 | \begin{Form} 221 | \begin{tabular}{l} 222 | \cm{fullTextInstructions}{Yes,No,N/A}{}\\[0.2cm] 223 | \tf{fullTextInstructionsJustification}{} 224 | \end{tabular} 225 | \end{Form} \\[0.3cm] 226 | 227 | \subsection{Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such \textit{payment is adequate} given the participants’ demographic (e.g., country of residence)?} 228 | If you answer {\bf Yes}, provide the section number; if you answer {\bf No}, provide a justification. \\[0.3cm] 229 | \begin{Form} 230 | \begin{tabular}{l} 231 | \cm{payment}{Yes,No,N/A}{}\\[0.2cm] 232 | \tf{paymentJustification}{} 233 | \end{tabular} 234 | \end{Form} \\[0.3cm] 235 | 236 | \subsection{Did you discuss whether and how \textit{consent} was obtained from people whose data you're using/curating (e.g., did your instructions explain how the data would be used)?} 237 | If you answer {\bf Yes}, provide the section number; if you answer {\bf No}, provide a justification. \\[0.3cm] 238 | \begin{Form} 239 | \begin{tabular}{l} 240 | \cm{consent}{Yes,No,N/A}{}\\[0.2cm] 241 | \tf{consentJustification}{} 242 | \end{tabular} 243 | \end{Form} \\[0.3cm] 244 | 245 | \subsection{Was the data collection protocol \textit{approved (or determined exempt)} by an ethics review board?} 246 | If you answer {\bf Yes}, provide the section number; if you answer {\bf No}, provide a justification. \\[0.3cm] 247 | \begin{Form} 248 | \begin{tabular}{l} 249 | \cm{ethicsAmountSpent}{Yes,No,N/A}{}\\[0.2cm] 250 | \tf{ethicsAmountSpentJustification}{} 251 | \end{tabular} 252 | \end{Form} \\[0.3cm] 253 | 254 | \subsection{Did you report the basic demographic and geographic characteristics of the \textit{annotator} population that is the source of the data?} 255 | If you answer {\bf Yes}, provide the section number; if you answer {\bf No}, provide a justification. \\[0.3cm] 256 | \begin{Form} 257 | \begin{tabular}{l} 258 | \cm{annotator}{Yes,No,N/A}{}\\[0.2cm] 259 | \tf{annotatorJustification}{} 260 | \end{tabular} 261 | \end{Form} \\[0.3cm] 262 | 263 | \end{document} 264 | --------------------------------------------------------------------------------