├── .gitignore ├── Cover_Art ├── ACF_nolabels.eps ├── LaTeX │ ├── text1-crop.eps │ ├── text1-crop.pdf │ ├── text1.pdf │ ├── text1.tex │ ├── text2-crop.eps │ ├── text2-crop.pdf │ ├── text2.pdf │ └── text2.tex ├── MATLAB │ ├── ACF.fig │ ├── ACF_nolabels.fig │ ├── for_Dan_S2.fig │ ├── pressure_trace.fig │ └── pressure_trace_nolabels.fig ├── concept1.fig ├── cover_art.eps ├── cover_art.fig ├── cover_art.pdf ├── cover_art.png ├── pressure_trace_nolabels.eps ├── text-engineeringdecisions.eps └── text-scientificconclusions.eps ├── Metrology_Vocabulary_JCGM_200_2012.pdf ├── README.md ├── Revision_1 ├── Diff_Submission-Revision.pdf └── Response_Letter.pdf ├── Revision_2 ├── Diff_Revions1-Revision2.pdf └── Response_Letter.pdf ├── Uncertainty_Guide_JCGM_100_2008_E.1-dmz.pdf ├── checklist.tex ├── conclusion.tex ├── definitions.tex ├── enhanced-sampling.tex ├── equilibration-burn-in.tex ├── figures ├── 1d-landscape-tslow.png ├── combinedcluster.pdf ├── combinedcluster │ ├── combinedcluster.agr │ └── combinedcluster.pdf ├── decorr-graphic.png ├── hyperbola.png ├── rmsd.pdf ├── rmsd │ ├── plot.gpt │ ├── rmsd.dat │ ├── rmsd.pdf │ ├── rmsds.dat │ └── rmsds.pdf ├── rmsds.pdf └── tequil-time-trace.png ├── from-google-doc.md ├── global-sampling.tex ├── livecoms.cls ├── main.pdf ├── main.tex ├── pre-sim-sanity.tex ├── quick-and-dirty.tex ├── refs.bib ├── releases ├── LiveCoMS_Article_ASAP_V1.pdf ├── LiveCoMS_Article_V1.pdf └── header_V1.png ├── scope.tex ├── specific-observables.tex └── vancouver-livecoms.bst /.gitignore: -------------------------------------------------------------------------------- 1 | # ignore the crap that latex generates 2 | 3 | *.aux 4 | *.bbl 5 | *.blg 6 | *.log 7 | *.out 8 | *.bib-stamp 9 | 10 | # ignore the crap that vim generates 11 | *.sw* 12 | 13 | # ignore backups from XFig 14 | *.fig.bak 15 | 16 | # ignore the pdf that latex produces 17 | # can't say *.pdf because there will be pdfs in the figures directory 18 | # that we want to keep 19 | *JCGM*.pdf 20 | 21 | 22 | -------------------------------------------------------------------------------- /Cover_Art/LaTeX/text1-crop.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/dmzuckerman/Sampling-Uncertainty/8cd46e29067297e7ff6288f56d0e131c214d6fa6/Cover_Art/LaTeX/text1-crop.pdf -------------------------------------------------------------------------------- /Cover_Art/LaTeX/text1.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/dmzuckerman/Sampling-Uncertainty/8cd46e29067297e7ff6288f56d0e131c214d6fa6/Cover_Art/LaTeX/text1.pdf -------------------------------------------------------------------------------- /Cover_Art/LaTeX/text1.tex: -------------------------------------------------------------------------------- 1 | % \documentclass[pre,preprint,amsmath,amssymb]{revtex4-1} 2 | \documentclass{article} 3 | \usepackage[screen,nopanel]{pdfscreen} 4 | 5 | \margins{0.05in}{0.05in}{0.05in}{0.05in} 6 | \screensize{1in}{3.5in} 7 | 8 | \usepackage{color} 9 | 10 | \begin{document} 11 | 12 | \backgroundcolor{white} 13 | \pagestyle{empty} 14 | 15 | \Huge 16 | Scientific Conclusions 17 | 18 | 19 | 20 | \end{document} 21 | 22 | 23 | %%% Local Variables: 24 | %%% mode: latex 25 | %%% TeX-master: t 26 | %%% End: 27 | 28 | 29 | -------------------------------------------------------------------------------- /Cover_Art/LaTeX/text2-crop.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/dmzuckerman/Sampling-Uncertainty/8cd46e29067297e7ff6288f56d0e131c214d6fa6/Cover_Art/LaTeX/text2-crop.pdf -------------------------------------------------------------------------------- /Cover_Art/LaTeX/text2.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/dmzuckerman/Sampling-Uncertainty/8cd46e29067297e7ff6288f56d0e131c214d6fa6/Cover_Art/LaTeX/text2.pdf -------------------------------------------------------------------------------- /Cover_Art/LaTeX/text2.tex: -------------------------------------------------------------------------------- 1 | % \documentclass[pre,preprint,amsmath,amssymb]{revtex4-1} 2 | \documentclass{article} 3 | \usepackage[screen,nopanel]{pdfscreen} 4 | 5 | \margins{0.05in}{0.05in}{0.05in}{0.05in} 6 | \screensize{1in}{3.5in} 7 | 8 | \usepackage{color} 9 | 10 | \begin{document} 11 | 12 | \backgroundcolor{white} 13 | \pagestyle{empty} 14 | 15 | \Huge 16 | Engineering Decisions 17 | 18 | 19 | 20 | \end{document} 21 | 22 | 23 | %%% Local Variables: 24 | %%% mode: latex 25 | %%% TeX-master: t 26 | %%% End: 27 | 28 | 29 | -------------------------------------------------------------------------------- /Cover_Art/MATLAB/ACF.fig: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/dmzuckerman/Sampling-Uncertainty/8cd46e29067297e7ff6288f56d0e131c214d6fa6/Cover_Art/MATLAB/ACF.fig -------------------------------------------------------------------------------- /Cover_Art/MATLAB/ACF_nolabels.fig: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/dmzuckerman/Sampling-Uncertainty/8cd46e29067297e7ff6288f56d0e131c214d6fa6/Cover_Art/MATLAB/ACF_nolabels.fig -------------------------------------------------------------------------------- /Cover_Art/MATLAB/for_Dan_S2.fig: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/dmzuckerman/Sampling-Uncertainty/8cd46e29067297e7ff6288f56d0e131c214d6fa6/Cover_Art/MATLAB/for_Dan_S2.fig -------------------------------------------------------------------------------- /Cover_Art/MATLAB/pressure_trace.fig: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/dmzuckerman/Sampling-Uncertainty/8cd46e29067297e7ff6288f56d0e131c214d6fa6/Cover_Art/MATLAB/pressure_trace.fig -------------------------------------------------------------------------------- /Cover_Art/MATLAB/pressure_trace_nolabels.fig: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/dmzuckerman/Sampling-Uncertainty/8cd46e29067297e7ff6288f56d0e131c214d6fa6/Cover_Art/MATLAB/pressure_trace_nolabels.fig -------------------------------------------------------------------------------- /Cover_Art/concept1.fig: -------------------------------------------------------------------------------- 1 | #FIG 3.2 Produced by xfig version 3.2.5c 2 | Landscape 3 | Center 4 | Metric 5 | A4 6 | 100.00 7 | Single 8 | -2 9 | 1200 2 10 | 2 5 0 1 0 -1 50 -1 -1 0.000 0 0 -1 0 0 5 11 | 0 pressure_trace.eps 12 | 450 450 9000 450 9000 4447 450 4447 450 450 13 | 2 5 0 1 0 -1 50 -1 -1 0.000 0 0 -1 0 0 5 14 | 0 text2-crop.eps 15 | 1350 6840 4308 6840 4308 7155 1350 7155 1350 6840 16 | 2 5 0 1 0 -1 50 -1 -1 0.000 0 0 -1 0 0 5 17 | 0 text1-crop.eps 18 | 1350 5265 4376 5265 4376 5535 1350 5535 1350 5265 19 | 2 5 0 1 0 -1 50 -1 -1 0.000 0 0 -1 0 0 5 20 | 0 avg_CI-crop.eps 21 | 5445 5985 9765 5985 9765 6435 5445 6435 5445 5985 22 | 2 5 0 1 0 -1 50 -1 -1 0.000 0 0 -1 0 0 5 23 | 0 stats_graphic.png 24 | 11160 2205 15617 2205 15617 5355 11160 5355 11160 2205 25 | 2 5 0 1 0 -1 50 -1 -1 0.000 0 0 -1 0 0 5 26 | 0 text3-crop.eps 27 | 11295 1688 15717 1688 15717 2025 11295 2025 11295 1688 28 | 2 1 0 2 0 7 50 -1 -1 0.000 0 0 -1 1 0 2 29 | 2 1 2.00 90.00 180.00 30 | 5400 6255 4365 6975 31 | 2 1 0 2 0 7 50 -1 -1 0.000 0 0 -1 1 0 2 32 | 2 1 2.00 90.00 180.00 33 | 5400 6255 4455 5445 34 | 3 0 0 2 0 7 50 -1 -1 0.000 0 1 0 3 35 | 2 1 2.00 120.00 180.00 36 | 8685 765 12600 225 13725 1530 37 | 0.000 1.000 0.000 38 | 3 0 0 2 0 7 50 -1 -1 0.000 0 1 0 3 39 | 2 1 2.00 120.00 180.00 40 | 13770 5130 12645 5940 9900 6255 41 | 0.000 1.000 0.000 42 | -------------------------------------------------------------------------------- /Cover_Art/cover_art.fig: -------------------------------------------------------------------------------- 1 | #FIG 3.2 Produced by xfig version 3.2.5c 2 | Landscape 3 | Center 4 | Metric 5 | A4 6 | 100.00 7 | Single 8 | -2 9 | 1200 2 10 | 2 5 0 1 0 -1 50 -1 -1 0.000 0 0 -1 0 0 5 11 | 0 pressure_trace_nolabels.eps 12 | 900 450 5400 450 5400 3085 900 3085 900 450 13 | 2 5 0 1 0 -1 50 -1 -1 0.000 0 0 -1 0 0 5 14 | 0 text-scientificconclusions.eps 15 | 9810 1080 12836 1080 12836 1350 9810 1350 9810 1080 16 | 2 5 0 1 0 -1 50 -1 -1 0.000 0 0 -1 0 0 5 17 | 0 text-engineeringdecisions.eps 18 | 9810 1890 12768 1890 12768 2205 9810 2205 9810 1890 19 | 2 5 0 1 0 -1 50 -1 -1 0.000 0 0 -1 0 0 5 20 | 0 ACF_nolabels.eps 21 | 5175 2700 9675 2700 9675 5335 5175 5335 5175 2700 22 | 3 0 0 3 0 7 50 -1 -1 0.000 0 1 0 3 23 | 2 1 2.00 150.00 300.00 24 | 3150 3015 3825 4050 5175 4050 25 | 0.000 1.000 0.000 26 | 3 0 0 3 0 7 50 -1 -1 0.000 0 1 0 3 27 | 2 1 2.00 150.00 300.00 28 | 7425 2655 8415 1620 9765 1620 29 | 0.000 1.000 0.000 30 | -------------------------------------------------------------------------------- /Cover_Art/cover_art.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/dmzuckerman/Sampling-Uncertainty/8cd46e29067297e7ff6288f56d0e131c214d6fa6/Cover_Art/cover_art.pdf -------------------------------------------------------------------------------- /Cover_Art/cover_art.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/dmzuckerman/Sampling-Uncertainty/8cd46e29067297e7ff6288f56d0e131c214d6fa6/Cover_Art/cover_art.png -------------------------------------------------------------------------------- /Cover_Art/pressure_trace_nolabels.eps: -------------------------------------------------------------------------------- 1 | %!PS-Adobe-3.0 EPSF-3.0 2 | %%Creator: (MATLAB, The Mathworks, Inc. Version 9.3.0.713579 \(R2017b\). Operating System: Linux) 3 | %%Title: /home/local/NIST/dsideriu/Documents/LiveCoMS/Sampling-Uncertainty/Cover_Art/pressure_trace_nolabels.eps 4 | %%CreationDate: 2018-06-19T11:06:25 5 | %%Pages: (atend) 6 | %%BoundingBox: 0 0 514 301 7 | %%LanguageLevel: 3 8 | %%EndComments 9 | %%BeginProlog 10 | %%BeginResource: procset (Apache XML Graphics Std ProcSet) 1.2 0 11 | %%Version: 1.2 0 12 | %%Copyright: (Copyright 2001-2003,2010 The Apache Software Foundation. License terms: http://www.apache.org/licenses/LICENSE-2.0) 13 | /bd{bind def}bind def 14 | /ld{load def}bd 15 | /GR/grestore ld 16 | /GS/gsave ld 17 | /RM/rmoveto ld 18 | /C/curveto ld 19 | /t/show ld 20 | /L/lineto ld 21 | /ML/setmiterlimit ld 22 | /CT/concat ld 23 | /f/fill ld 24 | /N/newpath ld 25 | /S/stroke ld 26 | /CC/setcmykcolor ld 27 | /A/ashow ld 28 | /cp/closepath ld 29 | /RC/setrgbcolor ld 30 | /LJ/setlinejoin ld 31 | /GC/setgray ld 32 | /LW/setlinewidth ld 33 | /M/moveto ld 34 | /re {4 2 roll M 35 | 1 index 0 rlineto 36 | 0 exch rlineto 37 | neg 0 rlineto 38 | cp } bd 39 | /_ctm matrix def 40 | /_tm matrix def 41 | /BT { _ctm currentmatrix pop matrix _tm copy pop 0 0 moveto } bd 42 | /ET { _ctm setmatrix } bd 43 | /iTm { _ctm setmatrix _tm concat } bd 44 | /Tm { _tm astore pop iTm 0 0 moveto } bd 45 | /ux 0.0 def 46 | /uy 0.0 def 47 | /F { 48 | /Tp exch def 49 | /Tf exch def 50 | Tf findfont Tp scalefont setfont 51 | /cf Tf def /cs Tp def 52 | } bd 53 | /ULS {currentpoint /uy exch def /ux exch def} bd 54 | /ULE { 55 | /Tcx currentpoint pop def 56 | gsave 57 | newpath 58 | cf findfont cs scalefont dup 59 | /FontMatrix get 0 get /Ts exch def /FontInfo get dup 60 | /UnderlinePosition get Ts mul /To exch def 61 | /UnderlineThickness get Ts mul /Tt exch def 62 | ux uy To add moveto Tcx uy To add lineto 63 | Tt setlinewidth stroke 64 | grestore 65 | } bd 66 | /OLE { 67 | /Tcx currentpoint pop def 68 | gsave 69 | newpath 70 | cf findfont cs scalefont dup 71 | /FontMatrix get 0 get /Ts exch def /FontInfo get dup 72 | /UnderlinePosition get Ts mul /To exch def 73 | /UnderlineThickness get Ts mul /Tt exch def 74 | ux uy To add cs add moveto Tcx uy To add cs add lineto 75 | Tt setlinewidth stroke 76 | grestore 77 | } bd 78 | /SOE { 79 | /Tcx currentpoint pop def 80 | gsave 81 | newpath 82 | cf findfont cs scalefont dup 83 | /FontMatrix get 0 get /Ts exch def /FontInfo get dup 84 | /UnderlinePosition get Ts mul /To exch def 85 | /UnderlineThickness get Ts mul /Tt exch def 86 | ux uy To add cs 10 mul 26 idiv add moveto Tcx uy To add cs 10 mul 26 idiv add lineto 87 | Tt setlinewidth stroke 88 | grestore 89 | } bd 90 | /QT { 91 | /Y22 exch store 92 | /X22 exch store 93 | /Y21 exch store 94 | /X21 exch store 95 | currentpoint 96 | /Y21 load 2 mul add 3 div exch 97 | /X21 load 2 mul add 3 div exch 98 | /X21 load 2 mul /X22 load add 3 div 99 | /Y21 load 2 mul /Y22 load add 3 div 100 | /X22 load /Y22 load curveto 101 | } bd 102 | /SSPD { 103 | dup length /d exch dict def 104 | { 105 | /v exch def 106 | /k exch def 107 | currentpagedevice k known { 108 | /cpdv currentpagedevice k get def 109 | v cpdv ne { 110 | /upd false def 111 | /nullv v type /nulltype eq def 112 | /nullcpdv cpdv type /nulltype eq def 113 | nullv nullcpdv or 114 | { 115 | /upd true def 116 | } { 117 | /sametype v type cpdv type eq def 118 | sametype { 119 | v type /arraytype eq { 120 | /vlen v length def 121 | /cpdvlen cpdv length def 122 | vlen cpdvlen eq { 123 | 0 1 vlen 1 sub { 124 | /i exch def 125 | /obj v i get def 126 | /cpdobj cpdv i get def 127 | obj cpdobj ne { 128 | /upd true def 129 | exit 130 | } if 131 | } for 132 | } { 133 | /upd true def 134 | } ifelse 135 | } { 136 | v type /dicttype eq { 137 | v { 138 | /dv exch def 139 | /dk exch def 140 | /cpddv cpdv dk get def 141 | dv cpddv ne { 142 | /upd true def 143 | exit 144 | } if 145 | } forall 146 | } { 147 | /upd true def 148 | } ifelse 149 | } ifelse 150 | } if 151 | } ifelse 152 | upd true eq { 153 | d k v put 154 | } if 155 | } if 156 | } if 157 | } forall 158 | d length 0 gt { 159 | d setpagedevice 160 | } if 161 | } bd 162 | /RE { % /NewFontName [NewEncodingArray] /FontName RE - 163 | findfont dup length dict begin 164 | { 165 | 1 index /FID ne 166 | {def} {pop pop} ifelse 167 | } forall 168 | /Encoding exch def 169 | /FontName 1 index def 170 | currentdict definefont pop 171 | end 172 | } bind def 173 | %%EndResource 174 | %%BeginResource: procset (Apache XML Graphics EPS ProcSet) 1.0 0 175 | %%Version: 1.0 0 176 | %%Copyright: (Copyright 2002-2003 The Apache Software Foundation. License terms: http://www.apache.org/licenses/LICENSE-2.0) 177 | /BeginEPSF { %def 178 | /b4_Inc_state save def % Save state for cleanup 179 | /dict_count countdictstack def % Count objects on dict stack 180 | /op_count count 1 sub def % Count objects on operand stack 181 | userdict begin % Push userdict on dict stack 182 | /showpage { } def % Redefine showpage, { } = null proc 183 | 0 setgray 0 setlinecap % Prepare graphics state 184 | 1 setlinewidth 0 setlinejoin 185 | 10 setmiterlimit [ ] 0 setdash newpath 186 | /languagelevel where % If level not equal to 1 then 187 | {pop languagelevel % set strokeadjust and 188 | 1 ne % overprint to their defaults. 189 | {false setstrokeadjust false setoverprint 190 | } if 191 | } if 192 | } bd 193 | /EndEPSF { %def 194 | count op_count sub {pop} repeat % Clean up stacks 195 | countdictstack dict_count sub {end} repeat 196 | b4_Inc_state restore 197 | } bd 198 | %%EndResource 199 | %FOPBeginFontDict 200 | %%IncludeResource: font Courier-Oblique 201 | %%IncludeResource: font Courier-BoldOblique 202 | %%IncludeResource: font Courier-Bold 203 | %%IncludeResource: font ZapfDingbats 204 | %%IncludeResource: font Symbol 205 | %%IncludeResource: font Helvetica 206 | %%IncludeResource: font Helvetica-Oblique 207 | %%IncludeResource: font Helvetica-Bold 208 | %%IncludeResource: font Helvetica-BoldOblique 209 | %%IncludeResource: font Times-Roman 210 | %%IncludeResource: font Times-Italic 211 | %%IncludeResource: font Times-Bold 212 | %%IncludeResource: font Times-BoldItalic 213 | %%IncludeResource: font Courier 214 | %FOPEndFontDict 215 | %%BeginResource: encoding WinAnsiEncoding 216 | /WinAnsiEncoding [ 217 | /.notdef /.notdef /.notdef /.notdef /.notdef 218 | /.notdef /.notdef /.notdef /.notdef /.notdef 219 | /.notdef /.notdef /.notdef /.notdef /.notdef 220 | /.notdef /.notdef /.notdef /.notdef /.notdef 221 | /.notdef /.notdef /.notdef /.notdef /.notdef 222 | /.notdef /.notdef /.notdef /.notdef /.notdef 223 | /.notdef /.notdef /space /exclam /quotedbl 224 | /numbersign /dollar /percent /ampersand /quotesingle 225 | /parenleft /parenright /asterisk /plus /comma 226 | /hyphen /period /slash /zero /one 227 | /two /three /four /five /six 228 | /seven /eight /nine /colon /semicolon 229 | /less /equal /greater /question /at 230 | /A /B /C /D /E 231 | /F /G /H /I /J 232 | /K /L /M /N /O 233 | /P /Q /R /S /T 234 | /U /V /W /X /Y 235 | /Z /bracketleft /backslash /bracketright /asciicircum 236 | /underscore /quoteleft /a /b /c 237 | /d /e /f /g /h 238 | /i /j /k /l /m 239 | /n /o /p /q /r 240 | /s /t /u /v /w 241 | /x /y /z /braceleft /bar 242 | /braceright /asciitilde /bullet /Euro /bullet 243 | /quotesinglbase /florin /quotedblbase /ellipsis /dagger 244 | /daggerdbl /circumflex /perthousand /Scaron /guilsinglleft 245 | /OE /bullet /Zcaron /bullet /bullet 246 | /quoteleft /quoteright /quotedblleft /quotedblright /bullet 247 | /endash /emdash /asciitilde /trademark /scaron 248 | /guilsinglright /oe /bullet /zcaron /Ydieresis 249 | /space /exclamdown /cent /sterling /currency 250 | /yen /brokenbar /section /dieresis /copyright 251 | /ordfeminine /guillemotleft /logicalnot /sfthyphen /registered 252 | /macron /degree /plusminus /twosuperior /threesuperior 253 | /acute /mu /paragraph /middot /cedilla 254 | /onesuperior /ordmasculine /guillemotright /onequarter /onehalf 255 | /threequarters /questiondown /Agrave /Aacute /Acircumflex 256 | /Atilde /Adieresis /Aring /AE /Ccedilla 257 | /Egrave /Eacute /Ecircumflex /Edieresis /Igrave 258 | /Iacute /Icircumflex /Idieresis /Eth /Ntilde 259 | /Ograve /Oacute /Ocircumflex /Otilde /Odieresis 260 | /multiply /Oslash /Ugrave /Uacute /Ucircumflex 261 | /Udieresis /Yacute /Thorn /germandbls /agrave 262 | /aacute /acircumflex /atilde /adieresis /aring 263 | /ae /ccedilla /egrave /eacute /ecircumflex 264 | /edieresis /igrave /iacute /icircumflex /idieresis 265 | /eth /ntilde /ograve /oacute /ocircumflex 266 | /otilde /odieresis /divide /oslash /ugrave 267 | /uacute /ucircumflex /udieresis /yacute /thorn 268 | /ydieresis 269 | ] def 270 | %%EndResource 271 | %FOPBeginFontReencode 272 | /Courier-Oblique findfont 273 | dup length dict begin 274 | {1 index /FID ne {def} {pop pop} ifelse} forall 275 | /Encoding WinAnsiEncoding def 276 | currentdict 277 | end 278 | /Courier-Oblique exch definefont pop 279 | /Courier-BoldOblique findfont 280 | dup length dict begin 281 | {1 index /FID ne {def} {pop pop} ifelse} forall 282 | /Encoding WinAnsiEncoding def 283 | currentdict 284 | end 285 | /Courier-BoldOblique exch definefont pop 286 | /Courier-Bold findfont 287 | dup length dict begin 288 | {1 index /FID ne {def} {pop pop} ifelse} forall 289 | /Encoding WinAnsiEncoding def 290 | currentdict 291 | end 292 | /Courier-Bold exch definefont pop 293 | /Helvetica findfont 294 | dup length dict begin 295 | {1 index /FID ne {def} {pop pop} ifelse} forall 296 | /Encoding WinAnsiEncoding def 297 | currentdict 298 | end 299 | /Helvetica exch definefont pop 300 | /Helvetica-Oblique findfont 301 | dup length dict begin 302 | {1 index /FID ne {def} {pop pop} ifelse} forall 303 | /Encoding WinAnsiEncoding def 304 | currentdict 305 | end 306 | /Helvetica-Oblique exch definefont pop 307 | /Helvetica-Bold findfont 308 | dup length dict begin 309 | {1 index /FID ne {def} {pop pop} ifelse} forall 310 | /Encoding WinAnsiEncoding def 311 | currentdict 312 | end 313 | /Helvetica-Bold exch definefont pop 314 | /Helvetica-BoldOblique findfont 315 | dup length dict begin 316 | {1 index /FID ne {def} {pop pop} ifelse} forall 317 | /Encoding WinAnsiEncoding def 318 | currentdict 319 | end 320 | /Helvetica-BoldOblique exch definefont pop 321 | /Times-Roman findfont 322 | dup length dict begin 323 | {1 index /FID ne {def} {pop pop} ifelse} forall 324 | /Encoding WinAnsiEncoding def 325 | currentdict 326 | end 327 | /Times-Roman exch definefont pop 328 | /Times-Italic findfont 329 | dup length dict begin 330 | {1 index /FID ne {def} {pop pop} ifelse} forall 331 | /Encoding WinAnsiEncoding def 332 | currentdict 333 | end 334 | /Times-Italic exch definefont pop 335 | /Times-Bold findfont 336 | dup length dict begin 337 | {1 index /FID ne {def} {pop pop} ifelse} forall 338 | /Encoding WinAnsiEncoding def 339 | currentdict 340 | end 341 | /Times-Bold exch definefont pop 342 | /Times-BoldItalic findfont 343 | dup length dict begin 344 | {1 index /FID ne {def} {pop pop} ifelse} forall 345 | /Encoding WinAnsiEncoding def 346 | currentdict 347 | end 348 | /Times-BoldItalic exch definefont pop 349 | /Courier findfont 350 | dup length dict begin 351 | {1 index /FID ne {def} {pop pop} ifelse} forall 352 | /Encoding WinAnsiEncoding def 353 | currentdict 354 | end 355 | /Courier exch definefont pop 356 | %FOPEndFontReencode 357 | %%EndProlog 358 | %%Page: 1 1 359 | %%PageBoundingBox: 0 0 514 301 360 | %%BeginPageSetup 361 | [1 0 0 -1 0 301] CT 362 | %%EndPageSetup 363 | GS 364 | [0.75 0 0 0.75 0 0.25] CT 365 | 1 GC 366 | N 367 | 0 0 685 401 re 368 | f 369 | GR 370 | GS 371 | [0.75 0 0 0.75 0 0.25] CT 372 | 1 GC 373 | N 374 | 0 0 685 401 re 375 | f 376 | GR 377 | GS 378 | [0.75 0 0 0.75 0 0.25] CT 379 | 1 GC 380 | N 381 | 89 357 M 382 | 620 357 L 383 | 620 30 L 384 | 89 30 L 385 | cp 386 | f 387 | GR 388 | GS 389 | [0.75 0 0 0.75 0 0.25] CT 390 | 0.149 GC 391 | 2 setlinecap 392 | 1 LJ 393 | 0.667 LW 394 | N 395 | 89 357 M 396 | 620 357 L 397 | S 398 | GR 399 | GS 400 | [0.75 0 0 0.75 0 0.25] CT 401 | 0.149 GC 402 | 2 setlinecap 403 | 1 LJ 404 | 0.667 LW 405 | N 406 | 89 30 M 407 | 620 30 L 408 | S 409 | GR 410 | GS 411 | [0.75 0 0 0.75 0 0.25] CT 412 | 0.149 GC 413 | 2 setlinecap 414 | 1 LJ 415 | 0.667 LW 416 | N 417 | 89 357 M 418 | 89 30 L 419 | S 420 | GR 421 | GS 422 | [0.75 0 0 0.75 0 0.25] CT 423 | 0.149 GC 424 | 2 setlinecap 425 | 1 LJ 426 | 0.667 LW 427 | N 428 | 620 357 M 429 | 620 30 L 430 | S 431 | GR 432 | GS 433 | [0.75 0 0 0.75 266.62104 48.07973] CT 434 | /Times-Bold 32 F 435 | GS 436 | [1 0 0 1 0 0] CT 437 | -68 -7 moveto 438 | 1 -1 scale 439 | (Raw Data) t 440 | GR 441 | GR 442 | GS 443 | [0.75 0 0 0.75 0 0.25] CT 444 | 0 0.447 0.741 RC 445 | 1 LJ 446 | 2.667 LW 447 | N 448 | 90.058 200.918 M 449 | 91.116 203.453 L 450 | 92.173 204.034 L 451 | 93.231 202.886 L 452 | 94.289 200.437 L 453 | 95.347 197.316 L 454 | 96.404 194.248 L 455 | 97.462 191.91 L 456 | 98.52 190.777 L 457 | 99.578 191.059 L 458 | 100.635 192.747 L 459 | 101.693 195.69 L 460 | 102.751 199.697 L 461 | 103.809 204.607 L 462 | 104.867 210.333 L 463 | 105.924 216.87 L 464 | 106.982 224.224 L 465 | 108.04 232.323 L 466 | 109.098 240.969 L 467 | 110.155 249.86 L 468 | 111.213 258.664 L 469 | 112.271 267.12 L 470 | 113.329 275.095 L 471 | 114.386 282.589 L 472 | 115.444 289.695 L 473 | 116.502 296.525 L 474 | 117.56 303.134 L 475 | 118.618 309.438 L 476 | 119.675 315.177 L 477 | 120.733 319.919 L 478 | 121.791 323.128 L 479 | 122.849 324.287 L 480 | 123.906 322.991 L 481 | 124.964 318.998 L 482 | 126.022 312.232 L 483 | 127.08 302.758 L 484 | 128.137 290.759 L 485 | 129.195 276.529 L 486 | 130.253 260.464 L 487 | 131.311 243.077 L 488 | 132.369 225.061 L 489 | 133.426 207.261 L 490 | 134.484 190.533 L 491 | 135.542 175.568 L 492 | 136.6 162.768 L 493 | 137.657 152.196 L 494 | 138.715 143.647 L 495 | 139.773 136.761 L 496 | 140.831 131.139 L 497 | 141.888 126.394 L 498 | 142.946 122.181 L 499 | 144.004 118.185 L 500 | 145.062 114.153 L 501 | 146.12 109.93 L 502 | 147.177 105.442 L 503 | 148.235 100.732 L 504 | 149.293 96.016 L 505 | 150.351 91.766 L 506 | 151.408 88.667 L 507 | 152.466 87.515 L 508 | 153.524 89.066 L 509 | 154.582 93.839 L 510 | 155.639 101.922 L 511 | 156.697 112.879 L 512 | 157.755 125.799 L 513 | 158.813 139.51 L 514 | 159.871 152.857 L 515 | 160.928 164.957 L 516 | 161.986 175.274 L 517 | 163.044 183.599 L 518 | 164.102 189.968 L 519 | 165.159 194.555 L 520 | 166.217 197.569 L 521 | 167.275 199.201 L 522 | 168.333 199.615 L 523 | 169.39 199.007 L 524 | 170.448 197.655 L 525 | 171.506 195.959 L 526 | 172.564 194.378 L 527 | 173.622 193.343 L 528 | 174.679 193.126 L 529 | 175.737 193.742 L 530 | 176.795 194.922 L 531 | 177.853 196.134 L 532 | 178.91 196.679 L 533 | 179.968 195.88 L 534 | 181.026 193.249 L 535 | 182.084 188.597 L 536 | 183.141 182.039 L 537 | 184.199 173.941 L 538 | 185.257 164.83 L 539 | 186.315 155.364 L 540 | 187.373 146.299 L 541 | 188.43 138.434 L 542 | 189.488 132.501 L 543 | 190.546 129.048 L 544 | 191.604 128.314 L 545 | 192.661 130.16 L 546 | 193.719 134.094 L 547 | 194.777 139.407 L 548 | 195.835 145.371 L 549 | 196.892 151.404 L 550 | 197.95 157.098 L 551 | 199.008 162.231 L 552 | 200.066 166.704 L 553 | 201.124 170.419 L 554 | 202.181 173.188 L 555 | 203.239 174.712 L 556 | 204.297 174.686 L 557 | 205.355 172.931 L 558 | 206.412 169.534 L 559 | 207.47 164.886 L 560 | 208.528 159.65 L 561 | 209.586 154.644 L 562 | 210.643 150.675 L 563 | 211.701 148.34 L 564 | 212.759 147.924 L 565 | 213.817 149.343 L 566 | 214.874 152.14 L 567 | 215.932 155.562 L 568 | 216.99 158.741 L 569 | 218.048 160.925 L 570 | 219.106 161.652 L 571 | 220.163 160.807 L 572 | 221.221 158.573 L 573 | 222.279 155.336 L 574 | 223.337 151.54 L 575 | 224.394 147.585 L 576 | 225.452 143.817 L 577 | 226.51 140.595 L 578 | 227.568 138.32 L 579 | 228.626 137.447 L 580 | 229.683 138.47 L 581 | 230.741 141.882 L 582 | 231.799 148.038 L 583 | 232.857 157.055 L 584 | 233.914 168.701 L 585 | 234.972 182.367 L 586 | 236.03 197.106 L 587 | 237.088 211.77 L 588 | 238.145 225.195 L 589 | 239.203 236.424 L 590 | 240.261 244.927 L 591 | 241.319 250.662 L 592 | 242.376 254.036 L 593 | 243.434 255.762 L 594 | 244.492 256.67 L 595 | 245.55 257.489 L 596 | 246.608 258.679 L 597 | 247.665 260.383 L 598 | 248.723 262.462 L 599 | 249.781 264.56 L 600 | 250.839 266.261 L 601 | 251.896 267.237 L 602 | 252.954 267.296 L 603 | 254.012 266.35 L 604 | 255.07 264.295 L 605 | 256.128 260.937 L 606 | 257.185 255.988 L 607 | 258.243 249.148 L 608 | 259.301 240.295 L 609 | 260.359 229.653 L 610 | 261.416 217.829 L 611 | 262.474 205.749 L 612 | 263.532 194.479 L 613 | 264.59 185.02 L 614 | 265.647 178.134 L 615 | 266.705 174.228 L 616 | 267.763 173.295 L 617 | 268.821 174.912 L 618 | 269.878 178.317 L 619 | 270.936 182.563 L 620 | 271.994 186.7 L 621 | 273.052 189.998 L 622 | 274.11 192.086 L 623 | 275.167 192.973 L 624 | 276.225 192.903 L 625 | 277.283 192.205 L 626 | 278.341 191.151 L 627 | 279.398 189.907 L 628 | 280.456 188.542 L 629 | 281.514 187.064 L 630 | 282.572 185.442 L 631 | 283.629 183.659 L 632 | 284.687 181.732 L 633 | 285.745 179.706 L 634 | 286.803 177.615 L 635 | 287.861 175.429 L 636 | 288.918 173.015 L 637 | 289.976 170.198 L 638 | 291.034 166.87 L 639 | 292.092 163.079 L 640 | 293.149 159.104 L 641 | 294.207 155.429 L 642 | 295.265 152.679 L 643 | 296.323 151.522 L 644 | 297.38 152.508 L 645 | 298.438 155.92 L 646 | 299.496 161.688 L 647 | 300.554 169.428 L 648 | 301.612 178.506 L 649 | 302.669 188.178 L 650 | 303.727 197.703 L 651 | 304.785 206.379 L 652 | 305.843 213.585 L 653 | 306.9 218.8 L 654 | 307.958 221.672 L 655 | 309.016 222.033 L 656 | 310.074 219.896 L 657 | 311.131 215.467 L 658 | 312.189 209.182 L 659 | 313.247 201.7 L 660 | 314.305 193.831 L 661 | 315.363 186.414 L 662 | 316.42 180.14 L 663 | 317.478 175.419 L 664 | 318.536 172.337 L 665 | 319.594 170.711 L 666 | 320.651 170.175 L 667 | 321.709 170.371 L 668 | 322.767 171.068 L 669 | 323.825 172.227 L 670 | 324.882 173.949 L 671 | 325.94 176.333 L 672 | 326.998 179.284 L 673 | 328.056 182.456 L 674 | 329.114 185.346 L 675 | 330.171 187.496 L 676 | 331.229 188.664 L 677 | 332.287 188.888 L 678 | 333.345 188.397 L 679 | 334.402 187.501 L 680 | 335.46 186.472 L 681 | 336.518 185.439 L 682 | 337.576 184.325 L 683 | 338.633 182.856 L 684 | 339.691 180.647 L 685 | 340.749 177.321 L 686 | 341.807 172.585 L 687 | 342.865 166.323 L 688 | 343.922 158.629 L 689 | 344.98 149.767 L 690 | 346.038 140.069 L 691 | 347.096 129.84 L 692 | 348.153 119.35 L 693 | 349.211 108.856 L 694 | 350.269 98.663 L 695 | 351.327 89.15 L 696 | 352.384 80.802 L 697 | 353.442 74.181 L 698 | 354.5 69.863 L 699 | 355.558 68.301 L 700 | 356.616 69.733 L 701 | 357.673 74.142 L 702 | 358.731 81.252 L 703 | 359.789 90.564 L 704 | 360.847 101.426 L 705 | 361.904 113.191 L 706 | 362.962 125.346 L 707 | 364.02 137.58 L 708 | 365.078 149.765 L 709 | 366.135 161.891 L 710 | 367.193 173.995 L 711 | 368.251 186.11 L 712 | 369.309 198.199 L 713 | 370.367 210.143 L 714 | 371.424 221.752 L 715 | 372.482 232.793 L 716 | 373.54 243.058 L 717 | 374.598 252.358 L 718 | 375.655 260.528 L 719 | 376.713 267.447 L 720 | 377.771 273.033 L 721 | 378.829 277.228 L 722 | 379.886 279.979 L 723 | 380.944 281.185 L 724 | 382.002 280.677 L 725 | 383.06 278.284 L 726 | 384.118 273.887 L 727 | 385.175 267.472 L 728 | 386.233 259.195 L 729 | 387.291 249.411 L 730 | 388.349 238.647 L 731 | 389.406 227.542 L 732 | 390.464 216.718 L 733 | 391.522 206.692 L 734 | 392.58 197.858 L 735 | 393.637 190.472 L 736 | 394.695 184.649 L 737 | 395.753 180.326 L 738 | 396.811 177.259 L 739 | 397.869 175.029 L 740 | 398.926 173.089 L 741 | 399.984 170.859 L 742 | 401.042 167.869 L 743 | 402.1 163.905 L 744 | 403.157 159.083 L 745 | 404.215 153.787 L 746 | 405.273 148.562 L 747 | 406.331 144.003 L 748 | 407.388 140.638 L 749 | 408.446 138.826 L 750 | 409.504 138.714 L 751 | 410.562 140.234 L 752 | 411.62 143.151 L 753 | 412.677 147.113 L 754 | 413.735 151.712 L 755 | 414.793 156.515 L 756 | 415.851 161.07 L 757 | 416.908 164.922 L 758 | 417.966 167.666 L 759 | 419.024 169.006 L 760 | 420.082 168.835 L 761 | 421.139 167.264 L 762 | 422.197 164.645 L 763 | 423.255 161.522 L 764 | 424.313 158.489 L 765 | 425.371 155.999 L 766 | 426.428 154.247 L 767 | 427.486 153.133 L 768 | 428.544 152.333 L 769 | 429.602 151.447 L 770 | 430.659 150.161 L 771 | 431.717 148.359 L 772 | 432.775 146.161 L 773 | 433.833 143.885 L 774 | 434.89 141.983 L 775 | 435.948 140.953 L 776 | 437.006 141.238 L 777 | 438.064 143.153 L 778 | 439.122 146.809 L 779 | 440.179 152.079 L 780 | 441.237 158.629 L 781 | 442.295 165.947 L 782 | 443.353 173.405 L 783 | 444.41 180.357 L 784 | 445.468 186.271 L 785 | 446.526 190.757 L 786 | 447.584 193.561 L 787 | 448.641 194.487 L 788 | 449.699 193.357 L 789 | 450.757 189.991 L 790 | 451.815 184.242 L 791 | 452.872 176.102 L 792 | 453.93 165.797 L 793 | 454.988 153.828 L 794 | 456.046 140.933 L 795 | 457.104 128.031 L 796 | 458.161 116.118 L 797 | 459.219 106.135 L 798 | 460.277 98.835 L 799 | 461.335 94.695 L 800 | 462.392 93.91 L 801 | 463.45 96.45 L 802 | 464.508 102.144 L 803 | 465.566 110.781 L 804 | 466.624 122.125 L 805 | 467.681 135.903 L 806 | 468.739 151.738 L 807 | 469.797 169.048 L 808 | 470.855 186.962 L 809 | 471.912 204.357 L 810 | 472.97 219.935 L 811 | 474.028 232.368 L 812 | 475.086 240.503 L 813 | 476.143 243.574 L 814 | 477.201 241.314 L 815 | 478.259 234.001 L 816 | 479.317 222.353 L 817 | 480.375 207.389 L 818 | 481.432 190.268 L 819 | 482.49 172.115 L 820 | 483.548 153.927 L 821 | 484.606 136.516 L 822 | 485.663 120.523 L 823 | 486.721 106.448 L 824 | 487.779 94.672 L 825 | 488.837 85.448 L 826 | 489.894 78.918 L 827 | 490.952 75.109 L 828 | 492.01 73.932 L 829 | 493.068 75.201 L 830 | 494.125 78.626 L 831 | 495.183 83.807 L 832 | 496.241 90.277 L 833 | 497.299 97.594 L 834 | 498.357 105.423 L 835 | 499.414 113.609 L 836 | 500.472 122.201 L 837 | 501.53 131.373 L 838 | 502.588 141.297 L 839 | 503.645 152.03 L 840 | 504.703 163.475 L 841 | 505.761 175.362 L 842 | 506.819 187.285 L 843 | 507.876 198.764 L 844 | 508.934 209.359 L 845 | 509.992 218.693 L 846 | 511.05 226.492 L 847 | 512.108 232.605 L 848 | 513.165 237.043 L 849 | 514.223 239.956 L 850 | 515.281 241.604 L 851 | 516.339 242.304 L 852 | 517.396 242.393 L 853 | 518.454 242.227 L 854 | 519.512 242.126 L 855 | 520.57 242.28 L 856 | 521.628 242.702 L 857 | 522.685 243.209 L 858 | 523.743 243.447 L 859 | 524.801 242.957 L 860 | 525.859 241.294 L 861 | 526.916 238.161 L 862 | 527.974 233.543 L 863 | 529.032 227.771 L 864 | 530.09 221.461 L 865 | 531.147 215.34 L 866 | 532.205 210.03 L 867 | 533.263 205.931 L 868 | 534.321 203.2 L 869 | 535.378 201.777 L 870 | 536.436 201.487 L 871 | 537.494 202.151 L 872 | 538.552 203.669 L 873 | 539.61 206.049 L 874 | 540.667 209.403 L 875 | 541.725 213.898 L 876 | 542.783 219.69 L 877 | 543.841 226.851 L 878 | 544.898 235.316 L 879 | 545.956 244.861 L 880 | 547.014 255.132 L 881 | 548.072 265.73 L 882 | 549.129 276.285 L 883 | 550.187 286.504 L 884 | 551.245 296.197 L 885 | 552.303 305.296 L 886 | 553.361 313.809 L 887 | 554.418 321.735 L 888 | 555.476 329.014 L 889 | 556.534 335.494 L 890 | 557.592 340.937 L 891 | 558.649 345.019 L 892 | 559.707 347.399 L 893 | 560.765 347.8 L 894 | 561.823 346.087 L 895 | 562.88 342.295 L 896 | 563.938 336.605 L 897 | 564.996 329.335 L 898 | 566.054 320.88 L 899 | 567.112 311.635 L 900 | 568.169 301.965 L 901 | 569.227 292.15 L 902 | 570.285 282.332 L 903 | 571.343 272.552 L 904 | 572.4 262.811 L 905 | 573.458 253.118 L 906 | 574.516 243.521 L 907 | 575.574 234.148 L 908 | 576.631 225.201 L 909 | 577.689 216.937 L 910 | 578.747 209.619 L 911 | 579.805 203.455 L 912 | 580.863 198.515 L 913 | 581.92 194.631 L 914 | 582.978 191.38 L 915 | 584.036 188.15 L 916 | 585.094 184.323 L 917 | 586.151 179.464 L 918 | 587.209 173.443 L 919 | 588.267 166.491 L 920 | 589.325 159.185 L 921 | 590.382 152.374 L 922 | 591.44 147.051 L 923 | 592.498 144.203 L 924 | 593.556 144.658 L 925 | 594.614 148.943 L 926 | 595.671 157.212 L 927 | 596.729 169.209 L 928 | 597.787 184.293 L 929 | 598.845 201.522 L 930 | 599.902 219.784 L 931 | 600.96 237.958 L 932 | 602.018 255.01 L 933 | 603.076 270.121 L 934 | 604.133 282.771 L 935 | 605.191 292.785 L 936 | 606.249 300.279 L 937 | 607.307 305.638 L 938 | 608.365 309.404 L 939 | 609.422 312.113 L 940 | 610.48 314.106 L 941 | 611.538 315.421 L 942 | 612.596 315.876 L 943 | 613.653 315.223 L 944 | 614.711 313.33 L 945 | 615.769 310.272 L 946 | 616.827 306.353 L 947 | 617.884 302.053 L 948 | 618.942 297.941 L 949 | S 950 | GR 951 | %%Trailer 952 | %%Pages: 1 953 | %%EOF 954 | -------------------------------------------------------------------------------- /Metrology_Vocabulary_JCGM_200_2012.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/dmzuckerman/Sampling-Uncertainty/8cd46e29067297e7ff6288f56d0e131c214d6fa6/Metrology_Vocabulary_JCGM_200_2012.pdf -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Sampling-Uncertainty 2 | Best Practices article for LiveCoMS (Living Journal of Computational 3 | Molecular Science) [http://www.livecomsjournal.org/] 4 | ACCEPTED ON 17 OCT 2018 5 | 6 | Link to published versions: https://doi.org/10.33011/livecoms.1.1.5067 7 | 8 | This is an article motivated by the Best Practices workshop organized by Michael Shirts on behalf of MolSSI, 9 | held at NIST on Aug 24-25, 2017. 10 | 11 | The article is intended as a guide for beginning molecular simulators (and beyond) 12 | to understand and assess sampling quality, 'convergence', and statistical uncertainty. 13 | 14 | Comments and suggestions from the community are welcomed. 15 | It is hoped that this article will be regularly updated to maximize its utility for the community. 16 | Significant contributions will be considered as a potential basis for authorhsip. 17 | 18 | The initial draft scope of the article: 19 | This article is geared toward studies attempting to quantify observables and derive reliable error bars 20 | based on “standard” canonical sampling methods (e.g., molecular dynamics and Monte Carlo) and 21 | associated “enhanced” sampling methods. Some problems and systems may be better studied with 22 | cruder techniques and analyses, which will not be covered here. 23 | -------------------------------------------------------------------------------- /Revision_1/Diff_Submission-Revision.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/dmzuckerman/Sampling-Uncertainty/8cd46e29067297e7ff6288f56d0e131c214d6fa6/Revision_1/Diff_Submission-Revision.pdf -------------------------------------------------------------------------------- /Revision_1/Response_Letter.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/dmzuckerman/Sampling-Uncertainty/8cd46e29067297e7ff6288f56d0e131c214d6fa6/Revision_1/Response_Letter.pdf -------------------------------------------------------------------------------- /Revision_2/Diff_Revions1-Revision2.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/dmzuckerman/Sampling-Uncertainty/8cd46e29067297e7ff6288f56d0e131c214d6fa6/Revision_2/Diff_Revions1-Revision2.pdf -------------------------------------------------------------------------------- /Revision_2/Response_Letter.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/dmzuckerman/Sampling-Uncertainty/8cd46e29067297e7ff6288f56d0e131c214d6fa6/Revision_2/Response_Letter.pdf -------------------------------------------------------------------------------- /Uncertainty_Guide_JCGM_100_2008_E.1-dmz.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/dmzuckerman/Sampling-Uncertainty/8cd46e29067297e7ff6288f56d0e131c214d6fa6/Uncertainty_Guide_JCGM_100_2008_E.1-dmz.pdf -------------------------------------------------------------------------------- /checklist.tex: -------------------------------------------------------------------------------- 1 | \section{Best Practices Checklist} 2 | 3 | Our overall recommendations are summarized in the checklist presented on the following page, which should facilitate avoiding common errors and adhering to good practices. 4 | 5 | \begin{Checklists*}[p!] 6 | \begin{checklist}{Quantifying Uncertainty and Sampling Quality in Molecular Simulation} 7 | \begin{itemize} 8 | \item 9 | \textbf{Plan your study carefully by starting with pre-simulation sanity checks.} 10 | There is no guarantee that any method, enhanced or otherwise, can sample the system of interest. 11 | See Sec.\ \ref{sec:sanity} 12 | \begin{itemize} 13 | \item Consult best-practices papers on simulation background and planning/setup. 14 | See: \url{https://github.com/MobleyLab/basic_simulation_training} 15 | \item Estimate whether system timescales are known experimentally and feasible computationally based on published literature. 16 | If timescales are too long for straight-ahead MD, investigate enhanced-sampling methods for systems of similar complexity. 17 | The same concept applies to MC, based on the number of MC trial moves instead of actual time. 18 | \item Read up on sampling assessment and uncertainty estimation, from this article or another source (e.g., Ref. \cite{Grossfield2009}). 19 | Understanding uncertainty will help in the \emph{planning} of a simulation (e.g., ensure collection of sufficient data). 20 | \item Consider multiple runs instead of a single simulation. 21 | Diverse starting structures enable a check on sampling for equilibrium ensembles, which should not depend on the starting structure. 22 | Multiple runs may be especially useful in assessing uncertainty for enhanced sampling methods. 23 | \item Check and validate your code/method via a simple benchmark system. 24 | See: \url{https://github.com/shirtsgroup/software-physical-validation} 25 | \end{itemize} 26 | \vspace{-0.325\baselineskip} %Line spacing after a sub-list is too large 27 | 28 | \item 29 | \textbf{Do not ``cherry-pick'' data that provides hoped-for outcomes.} 30 | This practice is ethically questionable and, at a minimum, can significantly bias your conclusions. 31 | Use all of the available data unless there is an objective and compelling reason not to, e.g., the simulation setup was incorrect or a sampling metric indicated that the simulation was not equilibrated. 32 | When used, sampling metrics should be applied uniformly to \emph{all} simulations to further avoid bias. 33 | 34 | \item 35 | \textbf{Perform simple, semiquantitative checks which can rule out (but not ensure) sufficient sampling.} It is easier to diagnose insufficient sampling than to demonstrate good sampling. See Sec.\ \ref{sec:quick}. 36 | \begin{itemize} 37 | \item Critically examine the time series of a number of observables, both those of interest \emph{and others}. 38 | Is each time series fluctuating about an average value or drifting overall? 39 | What states are expected and what are seen? 40 | Are there a significant number of transitions between states? 41 | \item If multiple runs have been performed, compare results (e.g., time series, distributions, etc.) from different simulations. 42 | \item An individual trajectory can be divided into two parts and analyzed as if two simulations had been run. 43 | \end{itemize} 44 | \vspace{-0.325\baselineskip} %Line spacing after a sub-list is too large 45 | 46 | \item 47 | \textbf{Remove an ``equilibration'' (a.k.a. ``burn-in'', or transient) portion of a single MD or MC trajectory} and perform analyses only on the remaining ``production'' portion of trajectory. An initial configuration is unlikely to be representative of the desired ensemble and the system must be allowed to relax so that low probability states are not overrepresented in collected data. See Sec.\ \ref{sec:equil}. 48 | 49 | \item 50 | \textbf{Consider computing a quantitative measure of global sampling}, i.e., attempt to estimate the number of statistically independent samples in a trajectory. Sequential configurations are highly correlated because one configuration is generated from the preceding one, and estimating the degree of correlation is essential to understanding overall simulation quality. 51 | See Secs.\ \ref{sec:global} and \ref{sec:autocorrelation}. 52 | 53 | \item 54 | \textbf{Quantify uncertainty in specific observables of interest using confidence intervals.} 55 | The statistical uncertainty in, e.g., the \hyperref[def:arith_mean]{\emph{arithmetic mean}} of an observable decreases as more independent samples are obtained and can be much smaller than the \hyperref[def:exp_st_dev]{experimental standard deviation} of that observable. 56 | See Sec.\ \ref{sec:specific}. 57 | 58 | \item 59 | \textbf{Use special care when designing uncertainty analyses for simulations with enhanced sampling methods.} 60 | The use of multiple, potentially correlated trajectories within a single enhanced-sampling simulation can invalidate the assumptions underpinning traditional analyses of uncertainty. 61 | See Sec.\ \ref{sec:enhanced}. 62 | 63 | \item 64 | \textbf{Report a complete description of your uncertainty quantification procedure, detailed enough to permit reproduction of reported findings.} 65 | Describe the meaning and basis of uncertainties given in figures or tables in the captions for those items, e.g., "Error bars represent 95\% confidence intervals based on bootstrapping results from the independent simulations." 66 | Provide expanded discussion of or references for the uncertainty analysis if the method is non-trivial. 67 | We strongly urge publication of unprocessed simulation data (measurements/observations) and post-processing scripts, perhaps using public data or software repositories, so that readers can exactly reproduce the processed results and uncertainty estimates. 68 | The non-uniformity of uncertainty quantification procedures in the modern literature underscores the value of clarity and transparency going forward. 69 | 70 | \end{itemize} 71 | \end{checklist} 72 | \end{Checklists*} 73 | -------------------------------------------------------------------------------- /conclusion.tex: -------------------------------------------------------------------------------- 1 | \section{Concluding Discussion} 2 | 3 | As computational scientists, we often spend vast resources modeling complex systems. With the thought and care involved in setting up these simulations, it is therefore surprising that significantly less time may be spent thinking about how to analyze and understand the validity of the generated data. Do our simulations not deserve better? 4 | 5 | We have spent some twenty odd pages telling you, reader, how and why you should run more simulations and do more analyses to vet the reliability of any given result. Given that any one of these tasks could take as long as running a production simulation, we face the invariable reality that uncertainty quantification (UQ) can substantially increase the time required to complete a project. 6 | Thus, we wish to adjust our readers' expectations: the time needed to perform a simulation study is not the time spent simulating. Rather, it is the time needed to: (i) generate data; {\it (ii) thoughtfully analyze it (whether by means of a posteriori UQ methods or additional simulations); and (iii) clearly communicate the means to and interpretations of the resulting uncertainties.} 7 | 8 | Ultimately we take the perspective that this extra effort gives us more confidence in our computational results. While this benefit may seem {\it soft}, note that industrial stakeholders actively use simulations and simulated results to make costly economic decisions. Thus, it could be argued that by not doing UQ, we invariably diminish the usefulness of simulations, and thus contribute to debate concerning their reliability and financial value. 9 | 10 | UQ goes hand-in-hand with the quest for \emph{reproducibility.} We should always keep in mind the fundamental principle that scientific results are reproducible. If we cannot state the certainty with which we believe a result, we cannot assess its reproducibility. If our results cannot be reproduced, what value do they have? 11 | 12 | UQ is an evolving field, but the underlying principles are not expected to change. 13 | We intend that this article will be updated to include best practices across an ever-broadening array of techniques, but even so, individual studies may require some adaptation and creativity. 14 | It is fair to say that UQ is a practitioners' field. You know your data best and should be able to assess its quality based on fundamental statistical principles and (variations of) the approaches described here. 15 | 16 | As a final note, we encourage readers to comment or add to this guide using the issue tracker of its GitHub repository [\githubrepository]. 17 | -------------------------------------------------------------------------------- /definitions.tex: -------------------------------------------------------------------------------- 1 | \subsection{Key Definitions} 2 | 3 | In order to make the discussion that follows more precise, we first define key terms used in subsequent sections. 4 | We caution that while many of these concepts are familiar, our terminology follows the {\it International Vocabulary of Metrology} (VIM) \citep{JCGM:VIM2012}, a standard that sometimes differs from the conventional or common language of engineering statistics. For additional information about or clarification of the statistical meaning of terms in the VIM, we suggest that readers consult the {\it Guide to the expression of uncertainty in measurement} (GUM) \citep{JCGM:GUM2008}. 5 | 6 | For clarity, we highlight a few differences between conventional terms and the VIM usage employed throughout this article. 7 | Readers should study the term ``standard uncertainty'' which is sometimes estimated by (in common parlance) the ``standard error of the mean''; however, the VIM term for the latter is the ``experimental standard deviation of the mean.'' 8 | In cases of lexical ambiguity, the reader should assume that we hold to the definition of terms as given in the VIM. 9 | 10 | Note also that the glossary is presented in a logical, rather than alphabetical order. We strongly encourage reading it through in its entirety because of the structure and potentially unfamiliar terminology. Importantly, we also recommend reading the discussion that immediately follows, since this (i) explains the rationale for adopting the chosen language, (ii) discusses the limited relationship between statistics and uncertainty quantification, and (iii) thereby clarifies our perspective on best-practices. 11 | 12 | 13 | \subsubsection{Glossary of Statistical Terms} 14 | \begin{itemize} 15 | 16 | \item {\bf Random quantity}: A quantity whose numerical value is inherently unknowable or unpredictable. 17 | Observations or measurements taken from a molecular simulation are treated as random quantities\footnote{Most molecular simulations (even those using pseudo-random number generators) are deterministic in that the sequence of visited states is generated by a fixed and known algorithm. As such, the simulation output is never truly random. 18 | In practice, however, the chaotic nature of the simulation allows for application of the principles of statistics to the analysis of simulation observations. 19 | Thus, observations/measurements taken at points along the simulation may be treated as random quantities. 20 | See Ref.~\cite{Leimkuhler} for more discussion of this rather deep point.}. 21 | 22 | \item {\bf True value:} The value of a quantity that is consistent with its definition and is the objective of an idealized measurement or simulation. The adjective ``true'' is often dropped when reference to the definition is clear by context \citep{JCGM:GUM2008,JCGM:VIM2012}. 23 | \label{def:true_value} 24 | 25 | \item {\bf Expectation value}: If $P(x)$ is the probability density of a continuous random quantity $x$, then the expectation value is given by the formula 26 | \begin{align} 27 | \expval{x} = \int {\rm d}x\, P(x) x. 28 | \label{linavg_cont} 29 | \end{align} 30 | In the case that $x$ adopts discrete values $x_1,x_2,...$ with corresponding fractional (absolute) probabilities $P(x_j)$, we instead write 31 | \begin{align} 32 | \expval{x} = \sum_j x_j P(x_j). 33 | \label{linavg} 34 | \end{align} 35 | Note that $P(x)$ is dimensionless when $x$ is discrete as shown above. 36 | When $x$ is continuous, as in Eq.~\ref{linavg_cont}, $P(x)$ must have units reciprocal to $x$, e.g., if $x$ has units of kg, then $P(x)$ has units of 1/kg. 37 | Furthermore, whether $x$ is discrete or continuous, $P(x)$ should always be normalized to ensure a total probability of unity. 38 | 39 | \item {\bf Variance}:\footnote{The true probability density $P(x)$ is inherently unknowable, given that we can only collect a finite amount of data about $x$. As such, we can only estimate its properties (e.g., mean and variance) and approximate its analytical form (e.g.\ see the end of Ref.\ \cite{SMC}).} Taking $P(x)$ as defined previously, the variance of a random quantity is a measure of how much it can fluctuate, given by the formula 40 | \begin{align} 41 | \sigma_x^2 = \int {\rm d}x\, P(x) \left( x - \expval{x} \right)^2. 42 | \end{align} 43 | If $x$ assume discrete values, the corresponding definition becomes 44 | \begin{align} 45 | \sigma_x^2 = \sum_j P(x_j) \left( x_j - \expval{x} \right)^2. 46 | \end{align} 47 | 48 | \item {\bf Standard Deviation}: The positive square root of the variance, denoted $\sigma_x$.\label{def:st_dev} This is a measure of the width of the distribution of $x$, and is, in itself, \emph{not} a measure of the statistical uncertainty; see below. 49 | 50 | \item {\bf Arithmetic mean}: An \emph{estimate} of the (true) expectation value of a random quantity, given by the formula 51 | % 52 | \begin{equation} 53 | \mean{x} = \dfrac{1}{n} \sum_{j=1}^{n} x_j \label{def:arith_mean} 54 | \end{equation} 55 | % 56 | where $x_j$ is an experimental or simulated realization of the random variable and $n$ is the number of samples. 57 | \smallskip 58 | 59 | \textbf{\textit{Remark:}} This quantity is often called the ``sample mean.'' 60 | Note that a proper realization of a random variable (with no systematic bias) will yield values distributed according to $P(x)$, so $\mean{x} \rightarrow \expval{x}$ as $n \rightarrow \infty$. 61 | 62 | 63 | \item {\bf Standard Uncertainty}: Uncertainty in a result (e.g., estimation of a true value) as expressed in terms of a \hyperref[def:st_dev]{standard deviation}.\footnote{The definition of standard uncertainty does not specify how to calculate the standard deviation. This choice ultimately rests with the modeler and should be dictated by the details of the uncertainty relevant to the problem at hand. Intuitively, this quantity should reflect the degree to which an estimate would vary if recomputed using new and independent data.} 64 | \label{def:std_unc} 65 | 66 | \item {\bf Experimental standard deviation:\footnote{ 67 | The term ``experimental'' can refer to simulated data, since these are the results of numerical experiments. 68 | } 69 | } An \emph{estimate} of the (true) standard deviation of a random variable, given by the formula\footnote{The factor of $n-1$ (as opposed to $n$) appearing in the denominator of Eq.~\ref{def:exp_st_dev} is needed to ensure that the variance estimate is {\it unbiased}, meaning that on average $\var{x}$ is equal to the true variance. Physically, we can interpret the $-1$ as accounting for the fact that one degree-of-freedom (e.g., piece of data) is lost via the appearance of $\mean{x}$ in the definition of $\stdev{x}$. Equivalently, it accounts for the fact that the arithmetic mean is linearly correlated with each $x_j$ (cf.\ \hyperref[def:unc_obs]{Linearly Uncorrelated Observables}).} 70 | \begin{equation} 71 | \stdev{x} = \sqrt{\dfrac{\sum_{j=1}^n\left(x_j - \mean{x}\right)^2}{n-1}} \label{def:exp_st_dev} 72 | \end{equation} 73 | \smallskip 74 | The square of the experimental standard deviation, denoted $\var{x}$, is the experimental variance.\\ 75 | \textbf{\textit{Remark:}} This quantity is often called the ``sample standard deviation.'' 76 | Additionally, $\stdev{x}$ is a statistical property of the specific set of observations $\left\{x_1,x_2,...,x_n\right\}$, not of the random quantity $x$ in general. 77 | Thus, $\stdev{x}$ is sometimes written as $\stdev{x_j}$ for emphasis of this property. 78 | 79 | \item {\bf Linearly uncorrelated observables}: If quantities $x$ and $y$ have mean values $\expval{x}$ and $\expval{y}$, then $x$ and $y$ are linearly uncorrelated if 80 | \begin{equation} 81 | \expval{ \left(x - \expval{x} \right) \left(y - \expval{y} \right) } = 0 \label{def:unc_obs} 82 | \end{equation} 83 | 84 | \textbf{\textit{Remark:}} The concepts of linear uncorrelation and independence of random variables are often conflated. Two variables can be correlated even if Eq.~\ref{def:unc_obs} is 0, e.g.~when a scatter plot of the two variables forms a circle. Truly independent variables have zero linear and higher-order correlations, such that the joint density of two random variables $x$ and $y$ can be decomposed as $P(x,y)=P(x)P(y)$, which is a stronger condition than linear uncorrelation. Empirically testing for independence, however, is not practical, nor is it necessary for any of the estimates discussed in this work. 85 | 86 | \item {\bf Experimental standard deviation of the mean}: An estimate of the standard deviation of the distribution of the \hyperref[def:arith_mean]{arithmetic mean}, given by the formula 87 | % 88 | \begin{equation} 89 | \stdevmean{\mean{x}} = \dfrac{\stdev{x}}{\sqrt{n}}, \label{def:exp_st_dev_mean} 90 | \end{equation} 91 | where the realizations of $x_j$ are assumed to be linearly uncorrelated.\footnote{The true variance of the mean goes as $\sigma_x^2/n$, which assumes exact knowledge of $\sigma_x$. Thus, the factor of $n$ (as opposed to $n-1$) appearing in Eq.~\eqref{def:exp_st_dev_mean} is motivated by the observation that $\langle \stdev{x}^2 \rangle = \sigma_x^2$, i.e., the experimental standard deviation provides an unbiased estimate of the experimental standard deviation of the mean. It is important and somewhat counterintuitive, however, that Eq.~\eqref{def:exp_st_dev_mean} actually {\it underestimates} the true standard deviation of the mean, which is a trivial consequence of Jensen's inequality. } 92 | 93 | \smallskip 94 | \textbf{\textit{Remark:}} This quantity is often called the ``standard error.'' 95 | 96 | %%% Terms that might be added in a future iteration: 97 | % \item {\bf Precision}: The amount of variability in an estimate (e.g., based on repeating a given simulation protocol multiple times). 98 | %\item {\bf Accuracy}: The degree to which a result agrees with a reference value. 99 | 100 | \item {\bf Raw data}: The numbers that the computer program directly generates as it proceeds through a sequence of states. 101 | For example, a MC simulation generates a sequence of configurations, for which there are associated properties such as the instantaneous pressure, temperature, volume, etc. 102 | \label{def:raw_data} 103 | \item {\bf Derived observables}: Quantities derived from ``non-trivial'' analyses of raw data, e.g., properties that may not be computed for a single configuration such as free energies. 104 | \label{def:deriv_obs} 105 | 106 | 107 | \item {\bf Correlation time}: In time-series data of a random quantity $x(t)$ (e.g., a physical property from a MC or MD trajectory; the sequence of trials moves is treated as a ``time series'' in MC), the correlation time (denoted here as $\tau$) is the {\it longest} separation time $\Delta t$ over which $x(t)$ and $x(t+\Delta t)$ remain (linearly) correlated.\footnote{Generally speaking, MC and MD trajectories generate new configurations from preceding ones.} (See Eq.~\ref{def:obs_ACF} for mathematical definition and Sec.~\ref{sec:autocorrelation} for discussion.) 108 | Thus, the correlation time can be interpreted as the time over which the system retains memory of its previous states. 109 | Such correlations are often {\bf stationary}, meaning that $\tau$ is independent of $t$. 110 | Roughly speaking, the total simulation time divided by the longest correlation time yields an order-of-magnitude estimate of the number of (linearly) {\it uncorrelated} samples generated by a simulation. See Sec.~\ref{sec:autocorrelation}. Note that the correlation time can be infinite. 111 | \label{def:corr_time} 112 | 113 | \item {\bf Two-sided confidence interval}: An interval, typically stated as $\expval{x} = \mean{x} \pm U$, which is expected to contain the possible values attributed to $\expval{x}$ given the experimental measurements of $x_j$ and a certain {\it level of confidence}, denoted $p$. 114 | The size of the confidence interval, known as the {\it expanded uncertainty}, is defined by $U = k \stdevmean{\mean{x}}$ where $k$ is the \hyperref[def:coveragefactor]{\it coverage factor} \cite{JCGM:VIM2012}.\footnote{This conceptual description of a confidence interval is only applicable when certain conditions are met, including the important stipulation that all uncertainty contained in $\stdevmean{\mean{x}}$ is determined only by statistical evaluation of the random experimental measurements of $x_j$ \cite{JCGM:GUM2008}.} 115 | The level of confidence $p$ is typically given as a percentage, e.g., 95~\%. Hence, the confidence interval is typically described as ``the $p$~\% confidence interval'' for a given value of $p$. 116 | \label{def:conf_int} 117 | 118 | \item {\bf Coverage Factor}: 119 | The factor $k$ which is multiplied by the experimental standard deviation of the mean \hyperref[def:exp_st_dev_mean]{$\stdevmean{\mean{x}}$} to obtain the expanded uncertainty, typically in the range of 2 to 3. In general, $k$ is selected based on the chosen level of confidence $p$ and probability distribution that characterizes the measurement result $x_j$. For Gaussian-distributed data, $k$ is determined from the $t$-distribution, based on the level of confidence $p$ and the number of measurements in the experimental sample.\footnote{For discussion regarding the selection of $k$ for non-Gaussian-distributed data, consult Annex G of Ref.~\cite{JCGM:GUM2008}.} See Sec.~\ref{sec:conf_int} for further discussion on the selection of $k$ and the resultant computation of confidence intervals. 120 | \label{def:coveragefactor} 121 | 122 | \end{itemize} 123 | 124 | \subsubsection{Terminology and its relation to our broader perspective on uncertainty} 125 | 126 | As surveyed by Refs.~\citep{JCGM:GUM2008,JCGM:VIM2012}, the discussion that originally motivated many of theses definitions appears rather philosophical. However, there are practical issues at stake related to both the content of the definitions as well as the need to adopt their usage. We review such issues now. 127 | 128 | At the heart of the matter is the observation that any uncertainty analysis, no matter how thorough, is inherently subjective. This can can be understood, for example, by noting that the arithmetic mean is itself actually a random quantity that only approximates the true expectation value.\footnote{Notably, the same observation applies to the experimental standard deviation and the corresponding experimental standard deviation of the mean.} Because its variation relative to the true value depends on the number of samples (notwithstanding a little bad luck), one could therefore argue that a better mean is always obtained by collecting more data. We cannot collect data indefinitely, however, so the {\it quality} of an estimate necessarily depends on a {\it choice} of when to stop. Ultimately, this discussion forces us to acknowledge that {\it the role of any uncertainty estimate is to facilitate decision making,} and, as such, the thoroughness of any analysis should be tailored to the decision at hand. 129 | 130 | Practically speaking, the definitions as put forth by the VIM attempt to reflect this perspective while also capturing ideas that the statistics community have long found useful. For example, the concept of an ``experimental standard deviation of the mean'' is nothing more than the ``standard error of the mean.'' However, the adjective ``experimental'' explicitly acknowledges that the estimate is in fact obtained from observation (and not analytical results), while the use of ``deviation'' in place of ``error'' emphasizes that the latter is unknowable. Similar considerations apply to the term ``experimental standard deviation,'' which is more commonly referred to as the ``sample standard deviation.'' 131 | 132 | It is important to note that subjectivity as identified in this discussion does not arise just from questions of sampling. In particular, methods such as parametric bootstrap and correlation analyses (discussed below) invoke modeling assumptions that can never be objectively tested. Moreover, experts may not even agree on how to compute a derived quantity, which leads to ambiguity in what we mean by a ``true value'' \cite{patrone1}. That we should consider these issues carefully and assess their impacts on any prediction is reflected in the definition of the ``standard uncertainty,'' which does not actually tell us how to compute uncertainties. {\it Rather it is the task of the modeler to consider the impacts of their assumptions and choices when formulating a final uncertainty estimate. To this end, the language we use plays a large role in how well these considerations are communicated.} 133 | 134 | As a final thought, we reiterate that the goal of an uncertainty analysis is not necessarily to perform the most thorough computations possible, but rather to communicate clearly and openly what has been assumed and done. We cannot predict every use-case for data that we generate, nor can we anticipate the decisions that will be made on the basis of our predictions. The importance of clearly communicating therefore rests on the fact that in doing so, we allow others to decide for themselves whether our analysis is sufficient or requires revisiting. To this end, consistent and precise use of language plays an important, if understated role. 135 | -------------------------------------------------------------------------------- /enhanced-sampling.tex: -------------------------------------------------------------------------------- 1 | \section{Assessing Uncertainty in Enhanced Sampling Simulations} 2 | \label{sec:enhanced} 3 | 4 | While recent advances in computational hardware have allowed MD simulations of systems with biological relevance to routinely reach timescales ranging from hundreds of \si{\nano\second} to \si{\micro\second}, in many cases this is still not long enough to obtain equilibrated (i.e., Boltzmann-weighted) structural populations. Intrinsic timescales of the systems may be much longer. 5 | Enhanced sampling methods can be used to obtain well-converged ensembles faster than conventional MD. In general, enhanced sampling methods work through a combination of modifying the underlying energy landscape and/or thermodynamic parameters to increase the rate at which energy barriers are crossed along with some form of reweighting to recover the unbiased ensemble \cite{Zuckerman2011}. However, such methods do not guarantee a converged ensemble, and care must be taken when using and evaluating enhanced sampling methods. 6 | 7 | Generally speaking, uncertainty analysis is more challenging for data generated by an enhanced sampling method. 8 | The family of enhanced equilibrium sampling methods, examples of which include replica exchange and variants \cite{Swendsen-1986,Sugita1999,Okamoto-2000}, local elevation \cite{Huber1994}, conformational flooding \cite{Grubmueller1995}, metadynamics \cite{Bussi2006a,Laio2008}, and adaptive biasing force \cite{Darve2001,Darve2008,Comer2015} to name a few, are complex and the resulting data may have a highly non-trivial correlation structure. 9 | In replica exchange, for example, the ensemble at a temperature of interest will be based on multiple return visits of different sequentially correlated trajectories. 10 | 11 | Before performing an enhanced-sampling simulation, consider carefully whether the technique is needed, and consult the literature for best practices in setting up a simulation. 12 | Even a straightforward MD simulation requires considerable planning, and the complexity is much greater for enhanced techniques. 13 | 14 | Given the subtleties of these sampling approaches, when possible, consider taking a ``bottom line'' approach, and assessing sampling based on multiple independent runs. 15 | The variance among these runs, if the approach is not biased, will help to quantify the overall sampling. 16 | Note that methods applicable to global assessment of multiple trajectories (Sec.\ \ref{sec:globalMultiTraj}) should be valid for analyzing multiple runs of an arbitrary method. 17 | However, a caveat for the approach of Zhang et al.\ \cite{Zhang2010} is that some dynamics trajectory segments would be required to perform state construction by kinetic clustering. 18 | 19 | \subsection{Replica Exchange Molecular Dynamics} 20 | One of the most popular enhanced sampling methods is replica exchange MD (REMD) \citep{Sugita1999}; see also \cite{Swendsen-1986}. Broadly speaking, REMD consists of running parallel MD simulations on a number of non-interacting replicas of a system, each with a different Hamiltonian and/or thermodynamic parameters (e.g., temperature), and periodically exchanging system coordinates between replicas according to a Metropolis criterion which maintains Boltzmann-factor sampling for all replicas. 21 | 22 | 23 | 24 | In order to assess the results of a REMD simulation, it is important to consider not just the overall convergence of the simulation to the correct Boltzmann-weighted ensemble of structures (via e.g., combined clustering, see Sec.~\ref{sec:quick}), but how efficiently the REMD simulation is doing so. These concepts are termed "thermodynamic efficiency" and "mixing efficiency" by Abraham and Gready \citep{Abraham2008}, and it is quite possible to achieve one without the other; both must be assessed. In order for sampling to be efficient, coordinates must be able to move freely in replica space. 25 | 26 | In practical settings, several metrics are often used to assess these two efficiencies, a few of which we list below. In these definitions, note that we refer to both "coordinate trajectories" and "replica trajectories". A "coordinate trajectory" follows an individual system's continuous trajectory as it traverses replica space (e.g., a system experiencing multiple temperatures as it is exchanged during a temperature REMD simulation). A "replica trajectory" is the sequence of configurations corresponding to a single replica under fixed Hamiltonian and thermodynamic conditions, (e.g., all structures at a temperature of 300 K in a temperature REMD simulation). Thus, a replica trajectory consists of concatenated coordinate-trajectory segments and \textit{vice versa}. 27 | 28 | Below are several checks that should be applied to REMD simulation data. 29 | 30 | \begin{itemize} 31 | \item Exchange acceptance. The exchange acceptance rate (i.e., the number of exchanges divided by the number of exchange attempts) between neighboring replicas should be roughly equivalent to each other and to the target acceptance rate. A low exchange acceptance between neighboring replicas relative to the average exchange acceptance rate creates a bottleneck in replica space which in turn can lead to poor sampling of the overall configuration space. In such cases, the replica spacing may need to be decreased or additional replicas used. Conversely, a high exchange acceptance rate between neighboring replicas relative to the average exchange acceptance rate may indicate that more resources than necessary are being used to simulate replicates, and that good sampling can be achieved with fewer replicas or larger replica spacing. 32 | \item Replica round trips. The time taken for a coordinate trajectory to travel from the lowest replica to the highest and back is called the replica "round trip" time. Over the course of a REMD simulation, any given coordinate trajectory should make multiple round trips. The rationale behind this is that every replica should contribute to enhancing the sampling of every set of starting coordinates. 33 | One can look at the average, minimum, and maximum round trip times among the coordinate trajectories: these should be similar for any given set of coordinates. See e.g., Fig.\ 6 in \citep{Roe2014}. %\textcolor{red}{PNP comment: not clear to me why these should be equivalent. Isn't there a distribution of times?} 34 | If they are not, it is likely due to one or more bottlenecks in replica space which can be identified by a relatively low exchange acceptance rate (see the previous bullet point). 35 | \item Replica residence time. The time a coordinate trajectory spends at a replica is called the "replica residence time". For replica sampling to be efficient, the replica residence time for each set of starting coordinates at each replica should be roughly equivalent. If it is not (i.e., if a set of starting coordinates is spending a much larger amount of time at certain replicas compared to the overall average) this can also indicate one or more bottlenecks in replica space. An example of this is shown in Fig.\ 7 in Ref.\ \citep{Roe2014}. 36 | \item Distributions of quantities calculated from coordinate trajectories. If all coordinates are moving freely in replica space, they should eventually converge to the same ensemble of structures. Comparing distributions of various quantities from coordinate trajectories can provide a measure of how converged the simulation is. For example, one can compare the distribution of RMSD values of coordinate trajectories to a common reference structure; see e.g., Fig.\ 8 in Ref.\ \citep{Henriksen2013}. Poor overlap can be an indication that replica efficiency is poor or the simulation is not yet converged. 37 | \end{itemize} 38 | 39 | All of the above quantities (replica residence time, round trip time, lifetimes etc) can be calculated with CPPTRAJ \citep{Roe2013}, which is freely available from \url{https://github.com/Amber-MD/cpptraj} or as part of AmberTools (\url{http://ambermd.org}). 40 | 41 | It may also be useful to perform multiple REMD runs. Using the standard uncertainty among runs can quantify uncertainty and provide the basis for a confidence interval with an appropriate coverage factor - see definitions in Sec.\ \ref{sec:scope}. If the ensembles produced depend significantly on the set of starting configurations, that is a sign of incomplete sampling. 42 | 43 | \subsection{Weighted Ensemble simulations} 44 | 45 | The weighted ensemble (WE) method orchestrates an ensemble of trajectories that are intermittently pruned or replicated in order to enhance sampling of difficult-to-access regions of configuration space \cite{Huber-1996}. 46 | The final set of trajectories can be visualized as a tree structure based on the occasional replication and pruning events. 47 | WE is an unbiased method that can be used to sample rare transient behavior \cite{Zhang2010a} as well as steady states \cite{Bhatt2010a} including equilibrium \cite{Suarez2014}. 48 | 49 | Like other enhanced sampling methods, WE's tree of trajectories has a complex correlation structure requiring care for uncertainty analysis. 50 | It is important to understand the basic theory and limitations of the WE method, as is discussed in a 51 | \href{https://westpa.github.io/westpa/overview.html}{WE overview document}. 52 | 53 | From a practical standpoint, the safest way to assess uncertainty in WE simulations is to run multiple instances (which can be seeded from identical or different starting structures depending on the desired calculation) from which a variance and standard uncertainty in any observable can be calculated. 54 | Note particularly that WE tracks the time evolution of observables as the system relaxes (perhaps quite slowly) to equilibrium or another steady state \cite{Zhang2010a}; hence, the variance computed in an observable from multiple runs should be based on values at the same time point. 55 | 56 | When it is necessary to estimate uncertainty based on a single WE run, the user should treat the (ensemble-weighted) value of an observable measured over time much like an observable in a standard single MD simulation; this is because the correlations in ensemble averages are sequential in time. 57 | First, as discussed in Sec.\ \ref{sec:quick}, the time trace of the observable should be inspected for relaxation to a nearly constant value about which fluctuations occur. 58 | A transient/equilibration period should be removed in analogy to MD - see Sec.\ \ref{sec:equil} - and then best practices for single observable uncertainties should be followed as described in Sec.\ \ref{sec:specific}. 59 | Despite this rather neat analogy to conventional MD, experience has shown that run-to-run variance in WE simulations of challenging systems can be large, so multiple runs are advised. In the future, variance-reduction techniques may alleviate the need for multiple runs. 60 | -------------------------------------------------------------------------------- /equilibration-burn-in.tex: -------------------------------------------------------------------------------- 1 | % !TEX root = ./main.tex 2 | 3 | \section{Determining and removing an equilibration or `burn-in' portion of a trajectory} 4 | \label{sec:equil} 5 | 6 | \begin{figure} 7 | \centering 8 | \includegraphics[width=0.9\linewidth]{figures/tequil-time-trace} 9 | \caption{ 10 | \label{fig:tequil} 11 | The equilibration and production segments of a trajectory. 12 | ``Equilibration'' over the time $t_{\mathrm{equil}}$ represents transient behavior while the initial configuration relaxes toward configurations more representative of the equilibrium ensemble. 13 | Readers are encouraged to select $t_{\mathrm{equil}}$ in a systematic way based on published literature. 14 | If you find strong sensitivity of ``production'' data to the choice of $t_{\mathrm{equil}}$, this suggests additional sampling is required. 15 | } 16 | \end{figure} 17 | 18 | The ``equilibration'' or ``burn-in'' time $t_{\mathrm{equil}}$ represents the initial part of a single continuous trajectory (whether from MD or MC) that is \emph{discarded} for purposes of data analysis of \emph{equilibrium or steady-state properties}; 19 | the remaining trajectory data are often called ``production'' data. 20 | See Fig.\ \ref{fig:tequil}. 21 | Discarding data may seem counterproductive, but there is no reason to expect that the initial configurations of a trajectory will be important in the ensemble ultimately obtained. 22 | Including early-time data, therefore, can systematically \emph{bias} results. 23 | 24 | To illustrate these points, consider the process of relaxing an initial, crystalline configuration of a protein to its amorphous counterpart in an aqueous environment. 25 | While the initial structure might seem to be intrinsically valuable, remember that configurations representative of the crystal structure may never appear in an aqueous system. 26 | As a result, the initial structure may be subject to unphysical forces and/or transitions that provide useless, if not misleading information about the system behavior.\footnote{In MD modeling of structural polymers (e.g., thermoset polymers), the problem of unphysical forces can be so severe that simulations become numerically unstable and crash. This frequently manifests as systems that explode and/or tear themselves apart. As a result, relaxation is often performed using Monte Carlo moves that minimize energy without reference to velocities and forces.} 27 | Note that relaxation/equilibration should be viewed as a means to an end: for equilibrium sampling, we only care that the relaxed state is representative of {\it any} local energy minimum that the system might sample, not how we arrived at that state, which is ultimately why data generated during equilibration can be discarded. 28 | 29 | The RMSD trace in Fig. \ref{f:rmsd} illustrates typical behavior of a system undergoing relaxation. Note the very rapid RMSD increase in the first $\approx$ 200 ns. Part of this increase is simply entropic: the volume of phase space within 1 {\AA} of a protein structure is extremely small, so that the process of thermalizing rapidly increases the RMSD from the starting structure, \emph{regardless of how favorable or representative that structure is}. Thus, examining that initial rapid increase is not helpful in determining an equilibration time. However, in this case, the RMSD continues to increase past 3 {\AA}, which is larger than the amplitude of simple thermal fluctuations (shown by Fig.\ \ref{f:rmsd}B), indicating an initial drift to a new structure, followed by sampling. 30 | 31 | Accepting that some data should be discarded, it is not hard to see that we want to avoid discarding too much data, given that many systems of interest are extremely expensive to simulate. In statistical terms, we want to remove bias but also minimize uncertainty (variance) through adequate sampling. Before addressing this problem, however, we emphasize that the very notion of separating a trajectory into equilibration and production segments only makes sense if the system has indeed reached configurations important in the equilibrium ensemble. While it is generally impossible to guarantee this has occurred, some easy checks for determining that this has not occurred are described in Sec.\ \ref{sec:quick}. \emph{It is essential to perform those basic checks before analyzing data with a more sophisticated approach} that may assume a trajectory has a substantial amount of true equilibrium sampling. 32 | 33 | 34 | A robust approach to determining the equilibration time is discussed in \cite{Chodera-2016}, which generalizes the notion of reverse cumulative averaging~\cite{Yang2004} to observables that do not necessarily have Gaussian distributions. 35 | The key idea is to analyze time-series data considering the effect of discarding various trial values of the initial equilibration interval, $t_{\mathrm{equil}}$ (Fig.\ \ref{fig:tequil}), and selecting the value that maximizes the effective number of uncorrelated samples of the remaining production region. 36 | This effective sample size is estimated from the number of samples in the production region divided by the number of temporally correlated samples required to produce one effectively uncorrelated sample, based on an auto-correlation analysis. 37 | At sufficiently large $t_{\mathrm{equil}}$, the majority of the initial relaxation transient is excluded, and the method selects the largest production region for which correlation times remain short to maximize the number of uncorrelated samples. 38 | Care must be taken in the case that the simulation is insufficiently long to sample many transitions among kinetically metastable states, however, or else this approach can simply result in restricting the production region to the last sampled metastable basin. 39 | A simpler qualitative analysis based on comparing forward and reverse estimates of observables \cite{Klimovich2015} may also be helpful. 40 | Readers may want to compare auto-correlation times for individual observables to the global ``decorrelation time'' \cite{Lyman2007a} described in Sec.\ \ref{sec:global}. 41 | As another general check, if values of observables estimated from the production phase depend sensitively on the choice of $t_{\mathrm{equil}}$, it is likely that further sampling is required. 42 | 43 | 44 | -------------------------------------------------------------------------------- /figures/1d-landscape-tslow.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/dmzuckerman/Sampling-Uncertainty/8cd46e29067297e7ff6288f56d0e131c214d6fa6/figures/1d-landscape-tslow.png -------------------------------------------------------------------------------- /figures/combinedcluster.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/dmzuckerman/Sampling-Uncertainty/8cd46e29067297e7ff6288f56d0e131c214d6fa6/figures/combinedcluster.pdf -------------------------------------------------------------------------------- /figures/combinedcluster/combinedcluster.agr: -------------------------------------------------------------------------------- 1 | # Grace project file 2 | # 3 | @version 50123 4 | @page size 792, 612 5 | @page scroll 5% 6 | @page inout 5% 7 | @link page off 8 | @map font 8 to "Courier", "Courier" 9 | @map font 10 to "Courier-Bold", "Courier-Bold" 10 | @map font 11 to "Courier-BoldOblique", "Courier-BoldOblique" 11 | @map font 9 to "Courier-Oblique", "Courier-Oblique" 12 | @map font 14 to "Courier-Regular", "Courier-Regular" 13 | @map font 15 to "Dingbats-Regular", "Dingbats-Regular" 14 | @map font 4 to "Helvetica", "Helvetica" 15 | @map font 6 to "Helvetica-Bold", "Helvetica-Bold" 16 | @map font 7 to "Helvetica-BoldOblique", "Helvetica-BoldOblique" 17 | @map font 5 to "Helvetica-Oblique", "Helvetica-Oblique" 18 | @map font 20 to "NimbusMonoL-Bold", "NimbusMonoL-Bold" 19 | @map font 21 to "NimbusMonoL-BoldOblique", "NimbusMonoL-BoldOblique" 20 | @map font 22 to "NimbusMonoL-Regular", "NimbusMonoL-Regular" 21 | @map font 23 to "NimbusMonoL-RegularOblique", "NimbusMonoL-RegularOblique" 22 | @map font 24 to "NimbusRomanNo9L-Medium", "NimbusRomanNo9L-Medium" 23 | @map font 25 to "NimbusRomanNo9L-MediumItalic", "NimbusRomanNo9L-MediumItalic" 24 | @map font 26 to "NimbusRomanNo9L-Regular", "NimbusRomanNo9L-Regular" 25 | @map font 27 to "NimbusRomanNo9L-RegularItalic", "NimbusRomanNo9L-RegularItalic" 26 | @map font 28 to "NimbusSansL-Bold", "NimbusSansL-Bold" 27 | @map font 29 to "NimbusSansL-BoldCondensed", "NimbusSansL-BoldCondensed" 28 | @map font 30 to "NimbusSansL-BoldCondensedItalic", "NimbusSansL-BoldCondensedItalic" 29 | @map font 31 to "NimbusSansL-BoldItalic", "NimbusSansL-BoldItalic" 30 | @map font 32 to "NimbusSansL-Regular", "NimbusSansL-Regular" 31 | @map font 33 to "NimbusSansL-RegularCondensed", "NimbusSansL-RegularCondensed" 32 | @map font 34 to "NimbusSansL-RegularCondensedItalic", "NimbusSansL-RegularCondensedItalic" 33 | @map font 35 to "NimbusSansL-RegularItalic", "NimbusSansL-RegularItalic" 34 | @map font 36 to "StandardSymbolsL-Regular", "StandardSymbolsL-Regular" 35 | @map font 12 to "Symbol", "Symbol" 36 | @map font 38 to "Symbol-Regular", "Symbol-Regular" 37 | @map font 2 to "Times-Bold", "Times-Bold" 38 | @map font 3 to "Times-BoldItalic", "Times-BoldItalic" 39 | @map font 1 to "Times-Italic", "Times-Italic" 40 | @map font 0 to "Times-Roman", "Times-Roman" 41 | @map font 43 to "URWBookmanL-DemiBold", "URWBookmanL-DemiBold" 42 | @map font 44 to "URWBookmanL-DemiBoldItalic", "URWBookmanL-DemiBoldItalic" 43 | @map font 45 to "URWBookmanL-Light", "URWBookmanL-Light" 44 | @map font 46 to "URWBookmanL-LightItalic", "URWBookmanL-LightItalic" 45 | @map font 47 to "URWChanceryL-MediumItalic", "URWChanceryL-MediumItalic" 46 | @map font 48 to "URWGothicL-Book", "URWGothicL-Book" 47 | @map font 49 to "URWGothicL-BookOblique", "URWGothicL-BookOblique" 48 | @map font 50 to "URWGothicL-Demi", "URWGothicL-Demi" 49 | @map font 51 to "URWGothicL-DemiOblique", "URWGothicL-DemiOblique" 50 | @map font 52 to "URWPalladioL-Bold", "URWPalladioL-Bold" 51 | @map font 53 to "URWPalladioL-BoldItalic", "URWPalladioL-BoldItalic" 52 | @map font 54 to "URWPalladioL-Italic", "URWPalladioL-Italic" 53 | @map font 55 to "URWPalladioL-Roman", "URWPalladioL-Roman" 54 | @map font 56 to "Utopia-Bold", "Utopia-Bold" 55 | @map font 57 to "Utopia-BoldItalic", "Utopia-BoldItalic" 56 | @map font 58 to "Utopia-Italic", "Utopia-Italic" 57 | @map font 59 to "Utopia-Regular", "Utopia-Regular" 58 | @map font 13 to "ZapfDingbats", "ZapfDingbats" 59 | @map color 0 to (255, 255, 255), "white" 60 | @map color 1 to (0, 0, 0), "black" 61 | @map color 2 to (255, 0, 0), "red" 62 | @map color 3 to (0, 255, 0), "green" 63 | @map color 4 to (0, 0, 255), "blue" 64 | @map color 5 to (255, 255, 0), "yellow" 65 | @map color 6 to (188, 143, 143), "brown" 66 | @map color 7 to (220, 220, 220), "grey" 67 | @map color 8 to (148, 0, 211), "violet" 68 | @map color 9 to (0, 255, 255), "cyan" 69 | @map color 10 to (255, 0, 255), "magenta" 70 | @map color 11 to (255, 165, 0), "orange" 71 | @map color 12 to (114, 33, 188), "indigo" 72 | @map color 13 to (103, 7, 72), "maroon" 73 | @map color 14 to (64, 224, 208), "turquoise" 74 | @map color 15 to (0, 139, 0), "green4" 75 | @reference date 0 76 | @date wrap off 77 | @date wrap year 1950 78 | @default linewidth 1.0 79 | @default linestyle 1 80 | @default color 1 81 | @default pattern 1 82 | @default font 0 83 | @default char size 1.000000 84 | @default symbol size 1.000000 85 | @default sformat "%.8g" 86 | @background color 0 87 | @page background fill on 88 | @timestamp off 89 | @timestamp 0.03, 0.03 90 | @timestamp color 1 91 | @timestamp rot 0 92 | @timestamp font 0 93 | @timestamp char size 1.000000 94 | @timestamp def "Fri Oct 20 14:35:42 2017" 95 | @r0 off 96 | @link r0 to g0 97 | @r0 type above 98 | @r0 linestyle 1 99 | @r0 linewidth 1.0 100 | @r0 color 1 101 | @r0 line 0, 0, 0, 0 102 | @r1 off 103 | @link r1 to g0 104 | @r1 type above 105 | @r1 linestyle 1 106 | @r1 linewidth 1.0 107 | @r1 color 1 108 | @r1 line 0, 0, 0, 0 109 | @r2 off 110 | @link r2 to g0 111 | @r2 type above 112 | @r2 linestyle 1 113 | @r2 linewidth 1.0 114 | @r2 color 1 115 | @r2 line 0, 0, 0, 0 116 | @r3 off 117 | @link r3 to g0 118 | @r3 type above 119 | @r3 linestyle 1 120 | @r3 linewidth 1.0 121 | @r3 color 1 122 | @r3 line 0, 0, 0, 0 123 | @r4 off 124 | @link r4 to g0 125 | @r4 type above 126 | @r4 linestyle 1 127 | @r4 linewidth 1.0 128 | @r4 color 1 129 | @r4 line 0, 0, 0, 0 130 | @g0 on 131 | @g0 hidden false 132 | @g0 type XY 133 | @g0 stacked false 134 | @g0 bar hgap 0.000000 135 | @g0 fixedpoint off 136 | @g0 fixedpoint type 0 137 | @g0 fixedpoint xy 0.000000, 0.000000 138 | @g0 fixedpoint format general general 139 | @g0 fixedpoint prec 6, 6 140 | @with g0 141 | @ world 0, 0, 0.5, 0.5 142 | @ stack world 0, 0, 0, 0 143 | @ znorm 1 144 | @ view 0.150000, 0.150000, 0.850000, 0.850000 145 | @ title "" 146 | @ title font 0 147 | @ title size 1.500000 148 | @ title color 1 149 | @ subtitle "" 150 | @ subtitle font 0 151 | @ subtitle size 1.000000 152 | @ subtitle color 1 153 | @ xaxes scale Normal 154 | @ yaxes scale Normal 155 | @ xaxes invert off 156 | @ yaxes invert off 157 | @ xaxis on 158 | @ xaxis type zero false 159 | @ xaxis offset 0.000000 , 0.000000 160 | @ xaxis bar on 161 | @ xaxis bar color 1 162 | @ xaxis bar linestyle 1 163 | @ xaxis bar linewidth 1.0 164 | @ xaxis label "Frac. 1" 165 | @ xaxis label layout para 166 | @ xaxis label place auto 167 | @ xaxis label char size 1.000000 168 | @ xaxis label font 0 169 | @ xaxis label color 1 170 | @ xaxis label place normal 171 | @ xaxis tick on 172 | @ xaxis tick major 0.1 173 | @ xaxis tick minor ticks 1 174 | @ xaxis tick default 6 175 | @ xaxis tick place rounded true 176 | @ xaxis tick in 177 | @ xaxis tick major size 1.000000 178 | @ xaxis tick major color 1 179 | @ xaxis tick major linewidth 1.0 180 | @ xaxis tick major linestyle 1 181 | @ xaxis tick major grid off 182 | @ xaxis tick minor color 1 183 | @ xaxis tick minor linewidth 1.0 184 | @ xaxis tick minor linestyle 1 185 | @ xaxis tick minor grid off 186 | @ xaxis tick minor size 0.500000 187 | @ xaxis ticklabel on 188 | @ xaxis ticklabel format general 189 | @ xaxis ticklabel prec 5 190 | @ xaxis ticklabel formula "" 191 | @ xaxis ticklabel append "" 192 | @ xaxis ticklabel prepend "" 193 | @ xaxis ticklabel angle 0 194 | @ xaxis ticklabel skip 0 195 | @ xaxis ticklabel stagger 0 196 | @ xaxis ticklabel place normal 197 | @ xaxis ticklabel offset auto 198 | @ xaxis ticklabel offset 0.000000 , 0.010000 199 | @ xaxis ticklabel start type auto 200 | @ xaxis ticklabel start 0.000000 201 | @ xaxis ticklabel stop type auto 202 | @ xaxis ticklabel stop 0.000000 203 | @ xaxis ticklabel char size 1.000000 204 | @ xaxis ticklabel font 0 205 | @ xaxis ticklabel color 1 206 | @ xaxis tick place both 207 | @ xaxis tick spec type none 208 | @ yaxis on 209 | @ yaxis type zero false 210 | @ yaxis offset 0.000000 , 0.000000 211 | @ yaxis bar on 212 | @ yaxis bar color 1 213 | @ yaxis bar linestyle 1 214 | @ yaxis bar linewidth 1.0 215 | @ yaxis label "Frac. 2" 216 | @ yaxis label layout para 217 | @ yaxis label place auto 218 | @ yaxis label char size 1.000000 219 | @ yaxis label font 0 220 | @ yaxis label color 1 221 | @ yaxis label place normal 222 | @ yaxis tick on 223 | @ yaxis tick major 0.1 224 | @ yaxis tick minor ticks 1 225 | @ yaxis tick default 6 226 | @ yaxis tick place rounded true 227 | @ yaxis tick in 228 | @ yaxis tick major size 1.000000 229 | @ yaxis tick major color 1 230 | @ yaxis tick major linewidth 1.0 231 | @ yaxis tick major linestyle 1 232 | @ yaxis tick major grid off 233 | @ yaxis tick minor color 1 234 | @ yaxis tick minor linewidth 1.0 235 | @ yaxis tick minor linestyle 1 236 | @ yaxis tick minor grid off 237 | @ yaxis tick minor size 0.500000 238 | @ yaxis ticklabel on 239 | @ yaxis ticklabel format general 240 | @ yaxis ticklabel prec 5 241 | @ yaxis ticklabel formula "" 242 | @ yaxis ticklabel append "" 243 | @ yaxis ticklabel prepend "" 244 | @ yaxis ticklabel angle 0 245 | @ yaxis ticklabel skip 0 246 | @ yaxis ticklabel stagger 0 247 | @ yaxis ticklabel place normal 248 | @ yaxis ticklabel offset auto 249 | @ yaxis ticklabel offset 0.000000 , 0.010000 250 | @ yaxis ticklabel start type auto 251 | @ yaxis ticklabel start 0.000000 252 | @ yaxis ticklabel stop type auto 253 | @ yaxis ticklabel stop 0.000000 254 | @ yaxis ticklabel char size 1.000000 255 | @ yaxis ticklabel font 0 256 | @ yaxis ticklabel color 1 257 | @ yaxis tick place both 258 | @ yaxis tick spec type none 259 | @ altxaxis off 260 | @ altyaxis off 261 | @ legend on 262 | @ legend loctype view 263 | @ legend 0.2, 0.8 264 | @ legend box color 1 265 | @ legend box pattern 1 266 | @ legend box linewidth 1.0 267 | @ legend box linestyle 1 268 | @ legend box fill color 0 269 | @ legend box fill pattern 1 270 | @ legend font 0 271 | @ legend char size 1.000000 272 | @ legend color 1 273 | @ legend length 4 274 | @ legend vgap 1 275 | @ legend hgap 1 276 | @ legend invert false 277 | @ frame type 0 278 | @ frame linestyle 1 279 | @ frame linewidth 1.0 280 | @ frame color 1 281 | @ frame pattern 1 282 | @ frame background color 0 283 | @ frame background pattern 0 284 | @ s0 hidden false 285 | @ s0 type xy 286 | @ s0 symbol 0 287 | @ s0 symbol size 1.000000 288 | @ s0 symbol color 1 289 | @ s0 symbol pattern 1 290 | @ s0 symbol fill color 1 291 | @ s0 symbol fill pattern 0 292 | @ s0 symbol linewidth 1.0 293 | @ s0 symbol linestyle 1 294 | @ s0 symbol char 65 295 | @ s0 symbol char font 0 296 | @ s0 symbol skip 0 297 | @ s0 line type 1 298 | @ s0 line linestyle 1 299 | @ s0 line linewidth 1.0 300 | @ s0 line color 1 301 | @ s0 line pattern 1 302 | @ s0 baseline type 0 303 | @ s0 baseline off 304 | @ s0 dropline off 305 | @ s0 fill type 0 306 | @ s0 fill rule 0 307 | @ s0 fill color 1 308 | @ s0 fill pattern 1 309 | @ s0 avalue off 310 | @ s0 avalue type 2 311 | @ s0 avalue char size 1.000000 312 | @ s0 avalue font 0 313 | @ s0 avalue color 1 314 | @ s0 avalue rot 0 315 | @ s0 avalue format general 316 | @ s0 avalue prec 3 317 | @ s0 avalue prepend "" 318 | @ s0 avalue append "" 319 | @ s0 avalue offset 0.000000 , 0.000000 320 | @ s0 errorbar on 321 | @ s0 errorbar place both 322 | @ s0 errorbar color 1 323 | @ s0 errorbar pattern 1 324 | @ s0 errorbar size 1.000000 325 | @ s0 errorbar linewidth 1.0 326 | @ s0 errorbar linestyle 1 327 | @ s0 errorbar riser linewidth 1.0 328 | @ s0 errorbar riser linestyle 1 329 | @ s0 errorbar riser clip off 330 | @ s0 errorbar riser clip length 0.100000 331 | @ s0 comment "" 332 | @ s0 legend "" 333 | @ s1 hidden false 334 | @ s1 type xy 335 | @ s1 symbol 1 336 | @ s1 symbol size 1.000000 337 | @ s1 symbol color 4 338 | @ s1 symbol pattern 1 339 | @ s1 symbol fill color 4 340 | @ s1 symbol fill pattern 0 341 | @ s1 symbol linewidth 2.0 342 | @ s1 symbol linestyle 1 343 | @ s1 symbol char 65 344 | @ s1 symbol char font 0 345 | @ s1 symbol skip 0 346 | @ s1 line type 0 347 | @ s1 line linestyle 1 348 | @ s1 line linewidth 1.0 349 | @ s1 line color 4 350 | @ s1 line pattern 1 351 | @ s1 baseline type 0 352 | @ s1 baseline off 353 | @ s1 dropline off 354 | @ s1 fill type 0 355 | @ s1 fill rule 0 356 | @ s1 fill color 1 357 | @ s1 fill pattern 1 358 | @ s1 avalue off 359 | @ s1 avalue type 2 360 | @ s1 avalue char size 1.000000 361 | @ s1 avalue font 0 362 | @ s1 avalue color 1 363 | @ s1 avalue rot 0 364 | @ s1 avalue format general 365 | @ s1 avalue prec 3 366 | @ s1 avalue prepend "" 367 | @ s1 avalue append "" 368 | @ s1 avalue offset 0.000000 , 0.000000 369 | @ s1 errorbar on 370 | @ s1 errorbar place both 371 | @ s1 errorbar color 4 372 | @ s1 errorbar pattern 1 373 | @ s1 errorbar size 1.000000 374 | @ s1 errorbar linewidth 1.0 375 | @ s1 errorbar linestyle 1 376 | @ s1 errorbar riser linewidth 1.0 377 | @ s1 errorbar riser linestyle 1 378 | @ s1 errorbar riser clip off 379 | @ s1 errorbar riser clip length 0.100000 380 | @ s1 comment "Cols 8:9" 381 | @ s1 legend "20 ns" 382 | @ s2 hidden false 383 | @ s2 type xy 384 | @ s2 symbol 8 385 | @ s2 symbol size 1.000000 386 | @ s2 symbol color 10 387 | @ s2 symbol pattern 1 388 | @ s2 symbol fill color 10 389 | @ s2 symbol fill pattern 1 390 | @ s2 symbol linewidth 2.0 391 | @ s2 symbol linestyle 1 392 | @ s2 symbol char 65 393 | @ s2 symbol char font 0 394 | @ s2 symbol skip 0 395 | @ s2 line type 0 396 | @ s2 line linestyle 1 397 | @ s2 line linewidth 1.0 398 | @ s2 line color 10 399 | @ s2 line pattern 1 400 | @ s2 baseline type 0 401 | @ s2 baseline off 402 | @ s2 dropline off 403 | @ s2 fill type 0 404 | @ s2 fill rule 0 405 | @ s2 fill color 1 406 | @ s2 fill pattern 1 407 | @ s2 avalue off 408 | @ s2 avalue type 2 409 | @ s2 avalue char size 1.000000 410 | @ s2 avalue font 0 411 | @ s2 avalue color 1 412 | @ s2 avalue rot 0 413 | @ s2 avalue format general 414 | @ s2 avalue prec 3 415 | @ s2 avalue prepend "" 416 | @ s2 avalue append "" 417 | @ s2 avalue offset 0.000000 , 0.000000 418 | @ s2 errorbar on 419 | @ s2 errorbar place both 420 | @ s2 errorbar color 10 421 | @ s2 errorbar pattern 1 422 | @ s2 errorbar size 1.000000 423 | @ s2 errorbar linewidth 1.0 424 | @ s2 errorbar linestyle 1 425 | @ s2 errorbar riser linewidth 1.0 426 | @ s2 errorbar riser linestyle 1 427 | @ s2 errorbar riser clip off 428 | @ s2 errorbar riser clip length 0.100000 429 | @ s2 comment "Cols 8:9" 430 | @ s2 legend "100 ns" 431 | @ s3 hidden false 432 | @ s3 type xy 433 | @ s3 symbol 9 434 | @ s3 symbol size 1.000000 435 | @ s3 symbol color 11 436 | @ s3 symbol pattern 1 437 | @ s3 symbol fill color 11 438 | @ s3 symbol fill pattern 1 439 | @ s3 symbol linewidth 2.0 440 | @ s3 symbol linestyle 1 441 | @ s3 symbol char 65 442 | @ s3 symbol char font 0 443 | @ s3 symbol skip 0 444 | @ s3 line type 0 445 | @ s3 line linestyle 1 446 | @ s3 line linewidth 1.0 447 | @ s3 line color 11 448 | @ s3 line pattern 1 449 | @ s3 baseline type 0 450 | @ s3 baseline off 451 | @ s3 dropline off 452 | @ s3 fill type 0 453 | @ s3 fill rule 0 454 | @ s3 fill color 1 455 | @ s3 fill pattern 1 456 | @ s3 avalue off 457 | @ s3 avalue type 2 458 | @ s3 avalue char size 1.000000 459 | @ s3 avalue font 0 460 | @ s3 avalue color 1 461 | @ s3 avalue rot 0 462 | @ s3 avalue format general 463 | @ s3 avalue prec 3 464 | @ s3 avalue prepend "" 465 | @ s3 avalue append "" 466 | @ s3 avalue offset 0.000000 , 0.000000 467 | @ s3 errorbar on 468 | @ s3 errorbar place both 469 | @ s3 errorbar color 11 470 | @ s3 errorbar pattern 1 471 | @ s3 errorbar size 1.000000 472 | @ s3 errorbar linewidth 1.0 473 | @ s3 errorbar linestyle 1 474 | @ s3 errorbar riser linewidth 1.0 475 | @ s3 errorbar riser linestyle 1 476 | @ s3 errorbar riser clip off 477 | @ s3 errorbar riser clip length 0.100000 478 | @ s3 comment "Cols 8:9" 479 | @ s3 legend "800 ns" 480 | @target G0.S0 481 | @type xy 482 | 0 0 483 | 0.5 0.5 484 | & 485 | @target G0.S1 486 | @type xy 487 | 0.449 0.2525 488 | 0.1785 0.1145 489 | 0.0055 0.2315 490 | 0.01 0.0215 491 | 0.0005 0.016 492 | & 493 | @target G0.S2 494 | @type xy 495 | 0.2066 0.1517 496 | 0.2029 0.1521 497 | 0.0875 0.2047 498 | 0.0505 0.0332 499 | 0.0298 0.0388 500 | 0.0161 0.0144 501 | 0.0114 0.0137 502 | 0.0082 0.0153 503 | 0.011 0.0058 504 | 0.0098 0.0056 505 | 0.0072 0.0082 506 | 0.0038 0.0073 507 | 0.0038 0.0069 508 | 0.004 0.0033 509 | 0.0031 0.0034 510 | 0.0041 0.0023 511 | 0.0027 0.0026 512 | 0.0045 0.0005 513 | 0.0015 0.0034 514 | 0.0015 0.0026 515 | 0.002 0.002 516 | 0.001 0.0027 517 | 0.0006 0.0031 518 | 0.0025 0.0008 519 | 0.0017 0.0013 520 | & 521 | @target G0.S3 522 | @type xy 523 | 0.1529 0.1479 524 | 0.1216 0.1438 525 | 0.1142 0.1007 526 | 0.0918 0.0387 527 | 0.0669 0.0766 528 | 0.0392 0.046 529 | 0.0232 0.0199 530 | 0.0126 0.0155 531 | 0.0107 0.0147 532 | 0.0098 0.0135 533 | 0.0096 0.0085 534 | 0.0087 0.0104 535 | 0.0076 0.0076 536 | 0.0064 0.0062 537 | 0.0049 0.0053 538 | 0.0032 0.0063 539 | 0.0028 0.0047 540 | 0.003 0.0019 541 | & 542 | -------------------------------------------------------------------------------- /figures/combinedcluster/combinedcluster.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/dmzuckerman/Sampling-Uncertainty/8cd46e29067297e7ff6288f56d0e131c214d6fa6/figures/combinedcluster/combinedcluster.pdf -------------------------------------------------------------------------------- /figures/decorr-graphic.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/dmzuckerman/Sampling-Uncertainty/8cd46e29067297e7ff6288f56d0e131c214d6fa6/figures/decorr-graphic.png -------------------------------------------------------------------------------- /figures/hyperbola.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/dmzuckerman/Sampling-Uncertainty/8cd46e29067297e7ff6288f56d0e131c214d6fa6/figures/hyperbola.png -------------------------------------------------------------------------------- /figures/rmsd.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/dmzuckerman/Sampling-Uncertainty/8cd46e29067297e7ff6288f56d0e131c214d6fa6/figures/rmsd.pdf -------------------------------------------------------------------------------- /figures/rmsd/plot.gpt: -------------------------------------------------------------------------------- 1 | set terminal pdfcairo enhanced font "Arial, 9" size 5cm,5cm 2 | set encoding iso_8859_1 3 | 4 | 5 | set output "rmsd.pdf" 6 | set xlabel "Time (ns)" 7 | set xrange [0:3000] 8 | set ylabel "RMSD (\305)" 9 | plot "rmsd.dat" u 1:2 notitle w l 10 | 11 | set output "rmsds.pdf" 12 | set xrange [0:150] 13 | set yrange [0:150] 14 | set xtics ("0" 0, "500" 25, "1000" 50, "1500" 75, "2000" 100, "2500" 125, "3000" 150) 15 | set ytics ("0" 0, "500" 25, "1000" 50, "1500" 75, "2000" 100, "2500" 125, "3000" 150) 16 | set xlabel "Time (ns)" 17 | set ylabel "Time (ns)" 18 | set view map 19 | set size square 20 | splot "rmsds.dat" matrix notitle w image 21 | 22 | -------------------------------------------------------------------------------- /figures/rmsd/rmsd.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/dmzuckerman/Sampling-Uncertainty/8cd46e29067297e7ff6288f56d0e131c214d6fa6/figures/rmsd/rmsd.pdf -------------------------------------------------------------------------------- /figures/rmsd/rmsds.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/dmzuckerman/Sampling-Uncertainty/8cd46e29067297e7ff6288f56d0e131c214d6fa6/figures/rmsd/rmsds.pdf -------------------------------------------------------------------------------- /figures/rmsds.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/dmzuckerman/Sampling-Uncertainty/8cd46e29067297e7ff6288f56d0e131c214d6fa6/figures/rmsds.pdf -------------------------------------------------------------------------------- /figures/tequil-time-trace.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/dmzuckerman/Sampling-Uncertainty/8cd46e29067297e7ff6288f56d0e131c214d6fa6/figures/tequil-time-trace.png -------------------------------------------------------------------------------- /from-google-doc.md: -------------------------------------------------------------------------------- 1 | ## Best Practices Draft Doc for sampling/convergence assessment & ensemble averages/uncertainties 2 | Alan Grossfield (agrossfield on GitHub), Pascal Merz, Paul Patrone, Daniel Roe, Andrew Schultz, Daniel Siderius, Daniel M. Zuckerman [PLEASE ADD YOUR NAME IF YOU HAVE ADDED TO THE DOC!!!] 3 | Grossfield, Alan [Alan_Grossfield@URMC.Rochester.edu]; zuckermd@ohsu.edu; Pascal.Merz@Colorado.EDU; Siderius, Daniel (Fed) [daniel.siderius@nist.gov]‎‎; ajs42@buffalo.edu‎; alan_grossfield@urmc.rochester.edu‎; daniel.r.roe@gmail.com; Patrone, Paul (Fed) [paul.patrone@nist.gov] 4 | 5 | # Scope 6 | This article is geared toward studies attempting to quantify observables and derive reliable error bars based on “standard” canonical sampling methods (e.g., molecular dynamics and Monte Carlo) and associated “enhanced” sampling methods. Some problems and systems may be better studied with cruder techniques and analyses, which will not be covered here. 7 | 8 | # Key Definitions [to be refined] 9 | - Precision 10 | - The amount of variability in an estimate 11 | - Accuracy 12 | - Vs nature/experiment 13 | - Vs. outcome of perfect/infinite simulation 14 | - Raw data - the set of numbers that the computer program spits out as it runs 15 | - Derived observables - quantities derived based on ‘non-trivial’ analyses based on the raw data 16 | 17 | # CHECKLIST 18 | - Pre-simulation sanity checks and planning tips: There is no guarantee that any method (enhanced or otherwise) can sample any given system 19 | - See best-practices papers on simulation background and planning/setup [Link out to simulation background and preparation documents - github] 20 | - Are system timescales known experimentally and feasible computationally based on published literature? 21 | - If timescales are too long for straight-ahead MD, is an enhanced method being used for which there are precedents for systems of similar complexity? 22 | - Read a good article or book on sampling assessment (this one or a reference herein). Understanding error is a technical endeavor. 23 | - Key concept: Connection between the equilibrium ensemble and individual trajectory (may or may not reach equilibrium); equilibrium vs. non-equilibrium. 24 | - Consider multiple runs vs. single run. Multiple runs may be especially useful in assessing uncertainty for enhanced sampling methods. 25 | - Make initial configuration as diverse as possible … but note that if results depend on initial configs, that implies insufficient sampling (to be pedantic, it always does if sampling is finite, but it’s about figuring out variability and confidence intervals 26 | - Look for automated construction methods for reproducibility 27 | - Check your code/method via a simple benchmark system. [Link out to software validation doc] 28 | - Perform a quick-and-dirty data checks which can rule out (but not ensure) sufficient sampling: Necessary vs. sufficient 29 | - Look at time series -- think in advance about what states should exist. How many transitions do you see? If you have 1 transition, you can’t talk about populations 30 | - Plot as many properties as you can think of, even if they’re not interesting 31 | - Visualize the trajectory -- look for slow motions. BE SKEPTICAL! 32 | - Compare observable different fractions of a run (DMZ thirds idea) 33 | - Andrew: short vs. very short 34 | - Daniel R: Compare runs from different initial conditions - be sure initial conditions are ‘different enough’ 35 | - Remove an ‘equilibration’/’burn in’/transient portion of a single MD or MC trajectory and perform analyses only on remaining ‘production’ portion of trajectory. 36 | - Consider computing a quantitative measure of global sampling [biomolecular vs. materials problems] - i.e., how many statistically independent samples do you have? 37 | - Quantifying error in specific observables of interest [general approaches] 38 | - Key concept: Difference between between scale of variation (measured by variance) and statistical uncertainty 39 | - If you’re using an enhanced sampling method, … 40 | - Key concept: Complexity of correlations in advanced methods 41 | 42 | # Item 6 - Computing error in specific observables 43 | - Basics: how to report, what goal to shoot for, significant figures 44 | - When should you not trust uncertainties 45 | - Unknown unknowns 46 | - If calculating a derived quantity, consider error in conversion from raw data 47 | - Propagation of error (this doesn’t mean publishing your results!) 48 | - Taylor series expansion can handle cases where derived quantity is a direct function of measured data 49 | - Wikipedia: Propagation of uncertainty 50 | - Generate synthetic data with noise model [Paul] 51 | - Data filtering [Paul] 52 | - Bootstrapping [Andrew/Dan S] 53 | used for cases where the derived quantity is not simple function of the measured data. 54 | An Introduction to the Bootstrap 55 | - Need to know correlation to correctly estimate sample size 56 | - Otherwise just gives relative uncertainty 57 | - Correlation time analysis [Dan Z] 58 | - Block averaging [Dan Z/Alan?/Dan S] Flyvjberg and Petersen 59 | - ‘Dark uncertainty’ analysis [Paul] 60 | - MOST OF THESE ALGORITHMS FAIL IF THE TRAJECTORY IS WAY TOO SHORT 61 | - If you miss the timescale by enough, you can’t tell 62 | - YOU HAVE TO THINK ABOUT THIS IN ADVANCE 63 | - Link out to transport doc 64 | 65 | # Item 7 - Enhanced sampling [Write as supplements, which can later be broken out as separate documents] 66 | - Compare to standard sampling (e.g., straight MD) for a simple system 67 | - Complex correlation structures in complex methods indicate comparison of multiple independent runs will be useful 68 | - Explain what to compare [Dan Z] 69 | - Replica exchange [Daniel Roe] 70 | - Round-trips (necessary but not sufficient) 71 | - Compare ‘coordinate trajectories’ - distributions from temperature/Hamiltonian-wandering trajectories should match 72 | - Examine replica residence times 73 | - Weighted ensemble (WE) 74 | - See this WE overview doc, particularly limitations section 75 | - Key concept: ‘Tree’ of trajectories generated by WE leads to strong correlations, requiring care 76 | - Key concept: WE simulation generically relaxes from the initial distribution toward the ultimate distribution which could be equilibrium (if no feedback/recycling or external driving) or a non-equilibrium steady state (if feedback from specified target to initial state) 77 | - Safest approach: Use multiple runs, which are fully independent. Perform as many runs as needed to reduce the statistical uncertainty (std err of mean) for quantity of interest 78 | - For a single observable, the time course of the value can be analyzed using the usual methods of analyzing time-correlated data (see above - e.g., block-averaging) 79 | 80 | 81 | # Original notes/discussion below here ... needs to be harvested for text 82 | 83 | Questions 84 | Should our document be self-contained? That is, should we write so that a graduate student could understand and implement all the methods noted, or do we rely on published works? 85 | I think making it self-contained would make for an enormous document and a huge amount of work. My preference would be to go with qualitative descriptions of techniques, prescriptions and rules of thumb for good practice, and extensive references for the details. 86 | 87 | (Paul Patrone: I second this. Another NIST colleague and I wrote a book chapter on UQ for MD meant as an introductory text for graduate students and novice modelers. It’s close to 100 pages long and took ~ 1 year to write. That would be too difficult for this document.) 88 | 89 | 90 | (PP) The more I read this document, I feel like we need to clarify for ourselves what our scope is. “Sampling / Uncertainty Assessment” is a very vague term to me. Is this meant to be an “overview” of uncertainty? What types of uncertainties do we focus on (systematic, statistical, etc.)? Do we focus on reducing uncertainties, quantifying them, both? Assessing confidence in a result? For what purpose? Do we only discuss issues of sampling in service of quantifying uncertainties, or do questions of sampling arise independently of uncertainty assessments? 91 | 92 | To put forth one perspective on this, I'll play devil’s advocate and beat a (dead ?) horse a bit. A few folks seem to be advocating the perspective that any discussion of sampling / convergence should not poach (too much) from other groups works, e.g. on physical validation or best practices for specific properties. I would perhaps argue a different perspective, namely that any discussion of sampling / convergence assessment can’t be divorced from the specifics of the simulation in question. (I'm just going to lump convergence / sampling / uncertainty estimates into the term “uncertainty quantification” or UQ). 93 | 94 | Uncertainty assessment and related issues of convergence (at least as I see them), are always tied to a question that we are trying to address or decision that someone is trying to make. UQ basically builds confidence that a decision is the “best” one. That being said, I don't think it's useful to do UQ in a vacuum or do analyses that don't help to inform a decision. As an example, I don't see too much of a point in checking that I've converged to the “correct” ensemble if all I care about is computing an ensemble average. So, not only can I forget about verifying the ensemble, I can use a thermostat (e.g. Andersen) that only gives me correct ensemble average. If, on the other hand, I care specifically about fluctuations, I may want use Nose-Hoover and then verify in whatever way I can that the ensemble is correct. As a different example, if I'm trying to estimate a material property such as yield strain, then I should be able to check that the material (approximately) recovers it's shape before the onset of yield. I would argue that's part of the uncertainty assessment because if the simulation doesn't do that, then I don't have a good estimate of when the material yielded and can’t make decisions on the basis of that simulation. 95 | 96 | So, all this being said, I see this document as a general overview of UQ issues that arise in simulations, with an eye always towards the reasons why and contexts in which we do these assessments. We need not go into detail on every material property (that's for other groups to address) but I don't think we should shy away from the greater context in which we do UQ. 97 | 98 | Anyways, I'm happy to get off of this horse if folks see this document differently. But I do think it would be useful to clarify what we're up to here and how this document fits into the overall picture of this workshop and the other documents that folks are writing. 99 | 100 | (AJS) It just doesn’t seem practical to enumerate how to compute and/or diagnose trouble with every kind of property and then to use this document as the basis for a “checklist” (that was the original objective of all this) that someone would use to convince themselves that they have sampled properly and have reasonable estimates of uncertainty. If we want people to use the resulting checklist, it has to be compact and not list stuff that’s not relevant to what they’re doing. Several of the specific properties mentioned below do have other groups working to produce a document (and checklist) for those properties. The group doing fluctuations (for instance) ought to have something about how to run the simulation, how to determine that the sampling is adequate and the issues (specific to fluctuations) in estimating uncertainties. It seem our best option is to point to those documents from here and to work with them to ensure that the relevant issues are covered in one of the documents. My 2 cents. 101 | 102 | DMZ: Here’s my view. I do think we want to adhere, for the most part, to the original goal of producing a doc that will be useful to beginning simulators (e.g., grad student in early years). There is consensus that the document will not explain all the theory but link out to extensive existing literature. As others have said, inevitably there will be overlap with the documents being produced by other groups (which doesn’t seem like a real problem) but I do think it’s ok for us to offer suggestions on assessing error in specific properties of general interest where some of us may have a lot of expertise. The more guidance we have for that beginning graduate student the better. One clean line that I think we can draw around our efforts is to exclude underlying/systematic errors in the models/force fields that we are simulating. I think we should stick to the issue of how the beginner knows whether to rely on their data as an accurate portrayal of the model/force field that has been assumed. BTW I think the goal is to produce and maintain a ‘living document’ that will be updated over time, which I think is a nice model. If one of us (or someone else) in the future wants to add discussion of analyses specific to another property, that will make the document still more helpful to the beginner. 103 | 104 | AG: I think Paul’s idea is interesting and true, but I’d say that the solution for his vision is the aggregate of _all_ of the documents coming from this workshop -- taken together, they’re trying to cover all of the sources of error and incorrectness. Like Dan, I think our charter was to focus on one particular area -- statistical error -- as separate from the many sources of systematic error. To cover all of the other sources of error would require a book, which is more effort than I think we want. 105 | 106 | Basic ideas and themes 107 | What can cause errors? 108 | Implementation errors (bad code, bad algorithms) 109 | Force field 110 | Finite size artifacts 111 | Timescale mismatch (experiments all do some implicit averaging) 112 | All of the above cause systematic errors 113 | No matter how long you run, these problems won’t go away 114 | Statistical errors are a separate issue 115 | Ergodic theorem only true in infinite time limit 116 | Finite sampling time implies statistical error 117 | Trivial to quantify in simple case -- stderr, etc 118 | Hard in the context of simulations with long time correlations 119 | Must be resolved in order to tackle the causes of systematic error 120 | Can’t know a new force field is better until you know the error bars on your “measurement” 121 | Realization and Verification of “convergence” 122 | Aim is to address the question: how does one ensure that the simulation is representative of thermodynamic ensemble OR whatever statistics govern the system of interest 123 | Generally involves at least two steps 124 | Simulating a transient/relaxation period during which we “burn in” or equilibrate system 125 | Verification that subsequent simulations represent the desired ensemble 126 | 127 | Start with sanity checking in advance (BEFORE DOING THE CALCULATION) 128 | What are the relevant time scales? 129 | Roughly how many relevant states are there? 130 | Is it even plausible to equilibrate? To converge? 131 | Have systems of similar complexity been well-sampled in the past, based on ‘best practices’ checks? 132 | Recall: most “internal consistency” measures of statistical uncertainty fail if your simulations are orders of magnitude too short. No rescue for stupid design. 133 | 134 | Uncertainty estimation (sometimes goes by the moniker “Uncertainty quantification”) 135 | Aim is generally to put error bars / confidence intervals on estimates, and possibly assess inconsistencies with other “target results” (i.e. validate the result) 136 | First task is generally to estimate uncertainties associated with the simulation itself 137 | Requires, e.g., understanding whether simulated data is correlated or statistically independent 138 | Requires understanding / modeling / accounting for the timescales over which random variables may fluctuate 139 | Requires formulating appropriate definition of uncertainty: e.g. sample variance or other statistical measures of uncertainty in a mean 140 | Other questions that arise / must be addressed 141 | How can we say when two averages are different? (P values) 142 | How do we verify that simulated data is consistent with theory (more examples below)? 143 | How do we address finite-size effects and their impact on uncertainty (this is relevant, e.g. for isothermal compressibility)? See J. J. Salacuse, A. R. Denton, and P. A. Egelstaff, Phys. Rev. E 53, 2382 (1996). 144 | How do we represent complicated observables, such as the radial distribution function or probability densities? See http://dx.doi.org/10.1063/1.4977516 145 | Second task may be to calibrate simulation inputs, (this task may not be necessary and/or may be a part of or precede the verification steps above) 146 | Validation against target data 147 | Often must be addressed in the context of target data that comes with its own uncertainties 148 | May be used to estimate quantities such as “model-form” error 149 | May use a variety of tools, such as polynomial chaos, Kriging analyses, etc. 150 | Point to relevant literature about uncertainty quantification; e.g. “Guide for Verification and Validation in Computational Solid Mechanics (ASME, 2006).” Also, Ralph Smith’s book. 151 | 152 | 153 | 154 | 155 | II. Realizing and verifying convergence of a simulation 156 | Methods for realizing convergence 157 | Within a single simple MD trajectories 158 | Chodera’s method http://pubs.acs.org/doi/abs/10.1021/acs.jctc.5b00784 159 | Rule of thumb?? Divide trajectory into thirds and compare distributions of observable for the last two thirds (i.e., 2nd third vs. 3rd third, with 1st third discarded as presumed equilibration period) 160 | Better yet -- all scalar quantities should be subject to block averaging if you’ve got one trajectory. It’s cheap, just make it part of standard operating procedure. STILL FAILS IF YOU’RE SUFFICIENTLY OFF ON TIME SCALE. 161 | What about ideas based on / motivated by fluctuation dissipation theorems? Small deviations from equilibrium often have a characteristic exponential decay time back to equilibrium. Shouldn’t observables converge according to these decay times? (I’m speculating a bit here) 162 | Here we describe a method that is similar to looking for convergence in mean square of observables: https://doi.org/10.1016/j.polymer.2016.01.074 163 | Convergence using multiple simple MD trajectories 164 | Often useful if you know there are long-lived states (even if it’s in principle an equilibrium system) 165 | Lets you use standard statistical approaches, treat each trajectory as 1 independent measurement 166 | May be necessary if, for example, the system takes a long time (microseconds or longer) to transition between metastable states 167 | May be necessary if individual simulations become irreversibly trapped in certain configurations (e.g. as simulations of crosslinked polymers do) 168 | This is a non-equilibrium circumstance, so totally different set of problems -- you’re characterizing a single relaxation per trajectory 169 | Methods for realizing convergence 170 | Ensemble methods - replica exchange, etc. (THIS IS A WHOLE OTHER BALLGAME) 171 | Just run multiple simulations (e.g. for crosslinked polymers) 172 | Methods for assessing convergence 173 | Shirts’ method (along with spectral Monte Carlo) to verify that an “equilibrated” distribution is consistent with Boltzmann statistics or grand-canonical ensemble. See M. R. Shirts, Journal of Chemical Theory and Computation 9, 909 (2013). 174 | Method motivated by law of large numbers 175 | Basically, compute standard error for N measurements and stop simulating when standard error falls below threshold 176 | NOTE: convergence methods based on total energy are tricky and ill-advised without extreme care 177 | Recommendation is based on the observation that discrete integrators (such as Verlet methods used in most MD packages) actually sample points from a shadow-Hamiltonian that is a perturbation of the analytical one that we write down. 178 | As a result, the energies output by MD (which are computed from the original, unperturbed Hamiltonian) fluctuate, especially in microcanonical ensemble. Thus, it appears that such simulations never converge to the correct ensemble. 179 | In order to assess energy conservation, it is therefore necessary to know the shadow Hamiltonian, or at least approximate it. This computation (whether done analytically or numerically), is very expensive. 180 | Alternatively, estimate order of magnitude of shadow Hamiltonian perturbation and make sure energy is conserved to within that precision 181 | See B. Leimkuhler and C. Matthews, Molecular Dynamics with Deterministic and Stochastic Numerical Methods (Springer International Publishing, Switzerland, 2015). 182 | Like most scalar observables, the energy is highly degenerate - many configs have same energy. The energy also includes all degrees of freedom in a system, unlike many specific observables of interest. Thus, despite its prominent place in statistical mechanics, the energy is unlikely to be a good indicator of overall sampling 183 | Convergence of structural ensembles from multiple independent simulations can be assessed via “combined clustering”: see e.g. http://pubs.acs.org/doi/abs/10.1021/ct400862k (Figs 3+4) 184 | Similarly, convergence of dynamics from multiple independent simulations can be assessed by overlap of principal component projection histograms: http://www.sciencedirect.com/science/article/pii/S0304416514003092 (Fig 3) 185 | 186 | 187 | 188 | 189 | III. Uncertainty estimation (UQ) 190 | Single MD trajectories 191 | At least two types of UQ associated with raw simulated data and data analysis thereof 192 | For raw simulated data, many techniques for estimating uncertainties 193 | Auto-correlation analysis 194 | For scalar properties such as pressure, temperature, etc., keep simulated data from every timestep 195 | For system in equilibrium, assume stationary autocorrelation function; estimate in terms of sample covariance 196 | Build uncertainty estimate in terms of associated correlation matrix 197 | Block averaging - note that correlation time is implicit 198 | Decorrelation time analysis - doesn’t require any user selection of observables [Lyman & Zuckerman, JPCB 2007 http://pubs.acs.org/doi/abs/10.1021/jp073061t] 199 | Analysis according to methods proposed by Salacuse: J. J. Salacuse, A. R. Denton, and P. A. Egelstaff, Phys. Rev. E 53, 2382 (1996) and J. J. Salacuse, A. R. Denton, P. A. Egelstaff, M. Tau, and L. Reatto, Phys. Rev. E 53, 2390 (1996). 200 | 201 | Data analysis of simulated data 202 | Generally is used to extract material properties from raw data, such as Tg (glass-transition temp) from density-temperature curves or modulus from stress-strain data 203 | Propagation of uncertainty can be used to estimate uncertainties of derived quantities (free energy, coexistence, etc). 204 | Must account for correlation between quantities measured in a single simulation. 205 | Bootstrapping can be used for cases where the derived quantity is not simple function of the measured data. 206 | Analyses often attempt to mimic what one would do with experimental data, which raises several issues 207 | Does the simulated data actually have characteristics of experimental data? 208 | In the context of Tg, one can ask: does my density-temperature curve have asymptotic regimes as identified, e.g., by a hyperbola? https://doi.org/10.1016/j.polymer.2016.01.074 209 | In the context of yield, one could ask: does the system recovers its original dimensions if allowed to relax after applying a strain that is less than the yield strain? https://doi.org/10.1016/j.polymer.2017.03.046 210 | Does simulated data have significantly more noise than experimental data, and if so, how does that affect analysis? 211 | Bootstrap and synthetic dataset methods can be used to estimate uncertainties associated with analysis applied to a single dataset 212 | Multiple independent analysis methods can be applied to a dataset to check for consistency between them 213 | Do the experiment and simulation operate on similar time scales? Experimental “shutter speed” can mean that the correct comparison is not with the ensemble generated in MD, but on some averaged/blurred version thereof. Especially true of fluctuation quantities. 214 | 215 | Multiple simple MD trajectories 216 | Estimates from multiple realizations of the same system may not have overlapping error bars 217 | “Between-simulation” or “dark-uncertainty” estimates can be used to generate a consensus mean estimate of the quantity of interest. 218 | See, e.g. https://doi.org/10.1016/j.polymer.2016.01.074 and A. L. Rukhin, Metrologia 46, 323 (2009). 219 | Ensemble methods - replica exchange, etc.. 220 | Connection to global sampling 221 | Generic strategies for evaluating enhanced sampling techniques which may have complex internal correlation structures 222 | Use of multiple independent (enhanced) simulations which are guaranteed to be independent. 223 | Validation / comparison with target data 224 | Can be used to generate model form error 225 | See, e.g. papers by Kennedy and O’Hagan 226 | M. C. Kennedy and A. O’Hagan, Journal of the Royal Statistical Society: Series B (Statistical Methodology) 63, 425 (2001). 227 | M. C. Kennedy, C. W. Anderson, S. Conti, and A. OHagan, Reliability Engineering & System Safety 91, 1301 (2006), the Fourth International Conference on Sensitivity Analysis of Model Output (SAMO 2004). 228 | 229 | 230 | 231 | Global measures of sampling - should avoid input of coordinates if possible 232 | Background: Connection to single-observable uncertainty … in principle, good global sampling should imply good ‘local’ sampling of particular observables … use idea of timescales to connect global/local 233 | Single simple MD trajectories 234 | Decorrelation time analysis - doesn’t require any user selection of observables [Lyman & Zuckerman, JPCB 2007 http://pubs.acs.org/doi/abs/10.1021/jp073061t] 235 | Nemec & Hoffman http://pubs.acs.org/doi/abs/10.1021/acs.jctc.6b00823 236 | Multiple simple MD trajectories 237 | Effective sample size 238 | Grossfield & co 239 | Zuckerman & co: JCTC 2010 http://pubs.acs.org/doi/abs/10.1021/ct1002384 240 | Also Nemec & Hoffman (same as above) 241 | 242 | 243 | 244 | 245 | -------------------------------------------------------------------------------- /global-sampling.tex: -------------------------------------------------------------------------------- 1 | \section{Quantification of Global Sampling} 2 | \label{sec:global} 3 | 4 | 5 | With ideal trajectory data, one would hope to be able to compute arbitrary observables with reasonably small error bars. 6 | During a simulation, it is not uncommon to monitor specific observables of interest, but after the data are obtained, it may prove necessary to compute observables not previously considered. 7 | These points motivate the task of estimating global sampling quality, which can be framed most simply in the context of single-trajectory data: 8 | ``Among the very large number of simulation frames (snapshots), how many are statistically independent?'' 9 | This number is called the \emph{effective sample size.} 10 | From a dynamical perspective evoking auto-correlation ideas, which also apply to Monte Carlo data, how long must one wait before the system completely loses memory of its prior configuration? 11 | The methods noted in this section build on ideas already presented in Sec.\ \ref{sec:quick} on qualitative sampling analysis, but attempt to go a step further to quantify sampling quality. 12 | 13 | We emphasize that \emph{no single method described here has emerged as a clear best practice.} 14 | However, because the global assessment methods provide a powerful window into overall sampling quality, which could easily be masked in the analysis of single observables (Sec.\ \ref{sec:specific}), we strongly encourage their use. 15 | The reader is encouraged to try one or more of the approaches in order to understand the limitations of their data. 16 | 17 | 18 | A key caveat is needed before proceeding. 19 | Analysis of trajectory data generally cannot make inferences about parts of configuration space not visited \cite{Grossfield2009}. 20 | It is generally impossible to know whether configurational states absent from a trajectory are appropriately absent because they are highly improbable (extremely high energy) or because the simulation simply failed to visit them because of a high barrier or random chance. 21 | 22 | \subsection{Global sampling assessment for a single trajectory} 23 | Two methods applicable for a single trajectory were previously introduced by some of the present authors, exploiting the fact that trajectories typically are correlated in time. 24 | That is, each configuration evolves from and is most similar to the immediately preceding configuration; 25 | this picture holds for standard MD and Markov-chain MC. 26 | Both analysis methods are implemented as part of the software package LOOS \cite{LOOS,LOOS-JCC}. 27 | 28 | \begin{figure} 29 | \centering 30 | \includegraphics[width=0.8\linewidth]{decorr-graphic.png} 31 | \caption{The basis for ``decorrelation analysis'' \cite{Lyman2007a}. 32 | From a continuous trajectory (blue curve), configurations can be extracted at equally spaced time points (filled circles) with each such configuration categorized as belonging to one of a set of arbitrary states (delineated by straight black lines). 33 | If the configurations are statistically independent -- if they are sufficiently decorrelated -- then their statistical behavior will match that predicted by a multinomial distribution consistent with the trajectory's fractional populations in each state. 34 | A range of time-spacings can be analyzed to determined if and when such independence occurs.} 35 | \label{fig:decorr} 36 | \end{figure} 37 | 38 | Lyman and Zuckerman proposed a global ``decorrelation'' analysis by mapping a trajectory to a discretization of configuration space (set of all $x, y, z$ coordinates) and analyzing the resulting statistics \cite{Lyman2007a}. 39 | See Fig.\ \ref{fig:decorr}. 40 | Configuration space is discretized into bins based on Voronoi cells \footnote{A Voronoi cell is defined to be the set of configurations closest to a given reference structure.} of structurally similar configurations, e.g., using RMSD defined in Eq.\ \ref{eq:rmsd} or another configurational similarity measure; reference configurations for the Voronoi binning are chosen at random or more systematically as described in \cite{Lyman2007a}. 41 | Once configuration space is discretized, the trajectory frames can be classified accordingly, leading to a discrete (i.e., 'multinomial') distribution (Fig.\ \ref{fig:decorr}). 42 | The analysis method is based on the observation that the variance for any bin of a multinomial distribution is known, given the bin populations (from trajectory counts) and a specified number of independent samples drawn from the distribution \cite{Lyman2007a}. 43 | The knowledge of the expected variance allows testing of increasing waiting times between configurations drawn from the trajectory to determine when and if the variance approaches that expected for independent samples. 44 | The minimum waiting time yielding agreement with ideal (i.e., uncorrelated) statistics yields an estimate for the decorrelation/memory time, which in turns implies an overall effective sample size. 45 | 46 | 47 | 48 | A second method, employing block covariance analysis (BCOM), was presented by Romo and Grossfield \cite{Romo2011} building on ideas by Hess \cite{Hess2002}. In essence, the method combines two standard error analysis techniques --- block averaging \cite{Flyvbjerg-1989} and bootstrapping \cite{Tibshirani1998} --- with covariance overlap, which quantitatively measures the similarity of modes determined from principal component analysis (PCA) \cite{Hess2002}. PCA in essence generates a new coordinate system for representing the fluctuation in the system while tracking the importance of each vector; the central idea of the method is to exploit the fact that as sampling improves, the modes generated by PCA should become more similar, and the covariance overlap will approach unity in the limit of infinite sampling. 49 | 50 | When applying BCOM, the principal components are computed from subsets of the trajectory, and the similarity of the modes evaluated as a function of subset size; as the subsets get larger, the resulting modes become more similar. This is done both for contiguous blocks of trajectory data (block averaging), and again for randomly chosen subsets of trajectory frames (bootstrapping); taking the ratio of the two values as a function of block size yields the degree of correlation in the data. Fitting that ratio to a sum of exponentials allows one to extract the relaxation times in the sampling. The key advantage of this method over others is that it implicitly takes into account the number of substates; the longest correlation time is the time required not to make a transition, but to sample a scattering of the relevant states. 51 | 52 | \subsection{Global sampling assessment for multiple independent trajectories} 53 | \label{sec:globalMultiTraj} 54 | When sampling is performed using multiple independent trajectories (whether MD or MC), additional care is required. 55 | Analyses based solely on the assumption of sequential correlations may break down because of the unknown relationship between separate trajectories. 56 | 57 | Zhang et al.\ extended the decorrelation/variance analysis noted above, while still retaining the basic strategy of inferring sample size based on variance \cite{Zhang2010}. 58 | To enable assessment of multiple trajectories, the new approach focused on conformational state populations, arguing that the states fundamentally underlie equilibrium observables. 59 | Here, a state is defined as a finite region of configuration space, which ideally consists of configurations among which transitions are faster than transitions among states; 60 | in practice, such states can be approximated based on kinetic-clustering Voronoi cells according to the inter-state transition times \cite{Zhang2010}. 61 | Once states are defined, the approach then uses the variances in state populations among trajectories to estimate the effective sample size, motivated by the decorrelation approach \cite{Lyman2007a} described above. 62 | 63 | Nemec and Hoffmann proposed related sampling measures geared specifically for analyzing and comparing multiple trajectories \cite{Nemec2017}. 64 | These measures again do not require user input of specific observables but only a measure of the difference between conformations, which was taken to the be the RMSD. 65 | Nemec and Hoffmann provide formulas for quantifying the conformational overlap among trajectories (addressing whether the same configurational states were sampled) and the density agreement (addressing whether conformational regions were sampled with equal probabilities). 66 | 67 | 68 | -------------------------------------------------------------------------------- /livecoms.cls: -------------------------------------------------------------------------------- 1 | % A template for LiveCoMS submissions. 2 | % 3 | % adapted from elife template, v1.4 4 | \NeedsTeXFormat{LaTeX2e} 5 | \ProvidesClass{livecoms}[2017/08/10, v0.5...] 6 | 7 | \RequirePackage[english]{babel} 8 | 9 | \RequirePackage{calc} 10 | \RequirePackage{etoolbox} 11 | \RequirePackage{regexpatch} 12 | \RequirePackage{ifxetex,ifluatex} 13 | 14 | \newif\ifxetexorluatex 15 | \ifxetex 16 | \xetexorluatextrue 17 | \else 18 | \ifluatex 19 | \xetexorluatextrue 20 | \else 21 | \xetexorluatexfalse 22 | \fi 23 | \fi 24 | 25 | \newif\if@reqslineno 26 | \DeclareOption{lineno}{\@reqslinenotrue} 27 | 28 | %% the type of document this is. The current types: 29 | % bestpractices, editorial, tutorial, review, comparison, lessons 30 | \newif\if@bestpractices 31 | \DeclareOption{bestpractices}{\@bestpracticestrue} 32 | 33 | \newif\if@editorial 34 | \DeclareOption{editorial}{\@editorialtrue} 35 | 36 | \newif\if@tutorial 37 | \DeclareOption{tutorial}{\@tutorialtrue} 38 | 39 | \newif\if@review 40 | \DeclareOption{review}{\@reviewtrue} 41 | 42 | \newif\if@comparison 43 | \DeclareOption{comparison}{\@comparisontrue} 44 | 45 | \newif\if@lessons 46 | \DeclareOption{lessons}{\@lessonstrue} 47 | 48 | %Publication Information 49 | \newif\if@pubversion 50 | \DeclareOption{pubversion}{\@pubversiontrue} 51 | 52 | \newif\if@ASAPversion 53 | \DeclareOption{ASAPversion}{\@ASAPversiontrue} 54 | 55 | %% Linespacing. 56 | \newif\if@onehalfspacing 57 | \newif\if@doublespacing 58 | \DeclareOption{onehalfspacing}{\@onehalfspacingtrue} 59 | \DeclareOption{doublespacing}{\@doublespacingtrue} 60 | 61 | \DeclareOption*{\PassOptionsToClass{\CurrentOption}{extarticle}} 62 | \ExecuteOptions{} 63 | \ProcessOptions\relax 64 | \LoadClass{extarticle} 65 | 66 | \RequirePackage{amsmath} 67 | \RequirePackage{amssymb} 68 | \RequirePackage{mdframed} 69 | 70 | \RequirePackage{lineno} 71 | \if@reqslineno\linenumbers\fi 72 | 73 | \ifxetexorluatex 74 | \RequirePackage[no-math]{fontspec} 75 | \setmainfont[Ligatures = TeX, 76 | Extension = .ttf, 77 | UprightFont = *-Regular, 78 | BoldFont = *-Bold, 79 | ItalicFont = *-Italic, 80 | BoldItalicFont = *-BoldItalic] 81 | {OpenSans} 82 | \else 83 | \RequirePackage[T1]{fontenc} 84 | \RequirePackage[utf8]{inputenc} 85 | \RequirePackage[default]{opensans} 86 | \renewcommand{\ttdefault}{lmtt} 87 | \fi 88 | 89 | \RequirePackage{microtype} 90 | 91 | % Trueno/Open Sans requires a bigger "single" linespread. 92 | \linespread{1.2} 93 | \if@onehalfspacing\linespread{1.5}\fi 94 | \if@doublespacing\linespread{2.0}\fi 95 | 96 | \emergencystretch 3em 97 | 98 | \RequirePackage{graphicx,xcolor} 99 | \definecolor{LiveCoMSDarkBlue}{HTML}{273B81} 100 | \definecolor{LiveCoMSLightBlue}{HTML}{0A9DD9} 101 | \definecolor{LiveCoMSMediumGrey}{HTML}{6D6E70} 102 | \definecolor{LiveCoMSLightGrey}{HTML}{929497} 103 | 104 | \RequirePackage{booktabs} 105 | \RequirePackage{authblk} 106 | 107 | % Modified page geometry for LiveComs 108 | \RequirePackage[%left=6cm,% 109 | %marginparwidth=4cm,% 110 | %marginparsep=0.5cm,% 111 | left=2cm, 112 | right=1.3cm,% 113 | top=2cm,% 114 | bottom=2.5cm,% 115 | headheight=21pt,% 116 | headsep=2\baselineskip,% 117 | columnsep=2em,% 118 | letterpaper]{geometry}% 119 | \RequirePackage{changepage} 120 | 121 | \RequirePackage{silence} 122 | \WarningFilter{caption}{The option `hypcap=true' will be ignored} 123 | \WarningFilter{microtype}{Unknown slot} 124 | 125 | \RequirePackage[labelfont={bf},% 126 | labelsep=period,% 127 | justification=justified,% 128 | singlelinecheck=false,% 129 | tableposition=top,font=small] 130 | {caption} 131 | 132 | % \captionsetup*[table]{skip=\medskipamount} 133 | 134 | 135 | \RequirePackage[square,numbers,sort&compress]{natbib} 136 | \RequirePackage{natmove} 137 | \renewcommand{\bibfont}{\small} 138 | % modifed from https://github.com/gbhutani/vancouver_authoryear_bibstyle/ 139 | \IfFileExists{vancouver-livecoms.bst} 140 | {\bibliographystyle{vancouver-livecoms}} 141 | {\PackageWarning{elife}{vancouver-livecoms.bst not found; falling back to apalike bibliography style.}\bibliographystyle{apalike}} 142 | % Make author in citation italic 143 | \renewcommand{\NAT@nmfmt}[1]{{\bfseries\itshape\color{LiveCoMSMediumGrey} #1}} 144 | 145 | % ...as well as the year 146 | \xpatchcmd{\NAT@citex} 147 | {\@citea\NAT@hyper@{\NAT@nmfmt{\NAT@nm}\NAT@date}} 148 | {\@citea\NAT@hyper@{\NAT@nmfmt{\NAT@nm}\NAT@nmfmt{\NAT@date}}} 149 | {}{\PackageWarning{LiveCoMS}{Failed to patch year format in citation}} 150 | 151 | \xpatchcmd{\NAT@citex} 152 | {\else\unskip\NAT@spacechar\NAT@hyper@{\NAT@date}} 153 | {\else\unskip\NAT@spacechar\NAT@hyper@{\NAT@nmfmt{\NAT@date}}} 154 | {}{\PackageWarning{LiveCoMS}{Failed to patch year format in citation}} 155 | 156 | \xpatchcmd{\NAT@citex} 157 | {\hyper@natlinkbreak{\NAT@aysep\NAT@spacechar}{\@citeb\@extra@b@citeb}\NAT@date} 158 | {\hyper@natlinkbreak{\NAT@nmfmt{\NAT@aysep\NAT@spacechar}}{\@citeb\@extra@b@citeb}\NAT@nmfmt{\NAT@date}} 159 | {}{\PackageWarning{LiveCoMS}{Failed to patch year format in citation}} 160 | 161 | \xpatchcmd{\NAT@citex} 162 | {\@citea\NAT@hyper@{\NAT@date}} 163 | {\@citea\NAT@hyper@{\NAT@nmfmt{\NAT@date}}} 164 | {}{\PackageWarning{LiveCoMS}{Failed to patch year format in citation}} 165 | 166 | \xpatchcmd{\NAT@citex} 167 | {{\@citeb\@extra@b@citeb}\NAT@date} 168 | {{\@citeb\@extra@b@citeb}\NAT@nmfmt{\NAT@date}} 169 | {}{\PackageWarning{LiveCoMS}{Failed to patch year format in citation}} 170 | %% There, we're finally done with patching the year in citations. 171 | 172 | % 173 | % headers and footers 174 | % 175 | 176 | \RequirePackage{fancyhdr} % custom headers/footers 177 | \RequirePackage{lastpage} % Number of pages in the document 178 | \pagestyle{fancy} % Enables the custom headers/footers 179 | %% Next two lines unnecessary for LiveComs 180 | % \addtolength{\headwidth}{\marginparsep} 181 | % \addtolength{\headwidth}{\marginparwidth} 182 | 183 | %% different document types listed here 184 | 185 | \newif\ifdocumenttype 186 | \documenttypefalse 187 | 188 | \if@bestpractices 189 | \documenttypetrue 190 | \newcommand{\documenttype}{Best Practices Guide} 191 | \else 192 | % nothing 193 | \fi 194 | 195 | \if@editorial 196 | \documenttypetrue 197 | \newcommand{\documenttype}{Editorial} 198 | \else 199 | % nothing 200 | \fi 201 | 202 | \if@tutorial 203 | \documenttypetrue 204 | \newcommand{\documenttype}{Tutorial} 205 | \else 206 | % nothing 207 | \fi 208 | 209 | \if@review 210 | \documenttypetrue 211 | \newcommand{\documenttype}{Perpetual Review} 212 | \else 213 | % nothing 214 | \fi 215 | 216 | \if@comparison 217 | \documenttypetrue 218 | \newcommand{\documenttype}{Molecular Simulation Comparison} 219 | \else 220 | % nothing 221 | \fi 222 | 223 | \if@lessons 224 | \documenttypetrue 225 | \newcommand{\documenttype}{``Lessons Learned'' Document} 226 | \else 227 | % nothing 228 | \fi 229 | 230 | \fancyhf{} 231 | \ifdocumenttype 232 | \chead{% 233 | \setlength{\fboxsep}{3pt} 234 | \colorbox{LiveCoMSMediumGrey}{\begin{minipage}{\headwidth}\centering\color{white} A LiveCoMS \documenttype\end{minipage}}% 235 | } 236 | \fi 237 | 238 | % Publication information in document footer 239 | % *ONLY INCLUDED IF "pubversion" CLASS OPTION IS INVOKED* 240 | \def\@publishedDOI{} 241 | \def\@publishedvolume{} 242 | \def\@publishedyear{} 243 | \def\@publishedarticlenum{} 244 | \def\@publisheddatereceived{} 245 | \def\@publisheddateaccepted{} 246 | \def \DOIprefix{10.XXXX} %May be able to use this later 247 | \newcommand{\pubDOI}[1]{% 248 | \appto{\@publishedDOI}{#1}{}{} 249 | } 250 | \newcommand{\pubvolume}[1]{% 251 | \appto{\@publishedvolume}{#1}{}{} 252 | } 253 | \newcommand{\pubissue}[1]{% 254 | \appto{\@publishedissue}{#1}{}{} 255 | } 256 | \newcommand{\pubyear}[1]{% 257 | \appto{\@publishedyear}{#1}{}{} 258 | } 259 | \newcommand{\articlenum}[1]{% 260 | \appto{\@publishedarticlenum}{#1}{}{} 261 | } 262 | \newcommand{\datereceived}[1]{% 263 | \appto{\@publisheddatereceived}{#1}{}{} 264 | } 265 | \newcommand{\dateaccepted}[1]{% 266 | \appto{\@publisheddateaccepted}{#1}{}{} 267 | } 268 | 269 | %-------------------------------------------------------- 270 | % Footers 271 | % 1. Error Check for conflicting class options 272 | \if@pubversion 273 | \if@ASAPversion 274 | \ClassError{livecoms} 275 | {Nope nope nope, you cannot invoke 'pubversion' and 'ASAPversion' simultaneously. Please correct the class options.} 276 | \fi 277 | \fi 278 | % 2. Publication Version: put submission/acceptance dates in left footer and citation information in right footer 279 | %%% DWS NOTE: would be nice if the left footer was in an if A-or-B type logical statement 280 | \if@pubversion 281 | \lfoot{\ifthenelse{\value{page}=1} 282 | {\small\color{LiveCoMSMediumGrey}Received: \@publisheddatereceived \\ Accepted: \@publisheddateaccepted} 283 | {~\\~} 284 | }% 285 | \rfoot{\small\color{LiveCoMSMediumGrey}\href{https://doi.org/\@publishedDOI}{https://doi.org/\@publishedDOI}\\ 286 | {\it Living J. Comp. Mol. Sci.} \@publishedyear, \@publishedvolume\nobreak\hspace{.05em}(\@publishedissue), \@publishedarticlenum 287 | }% 288 | \fi 289 | % 3. ASAP Version: put submission/acceptance dates in left footer and "ASAP Version" in right footer 290 | \if@ASAPversion 291 | \lfoot{\ifthenelse{\value{page}=1} 292 | {\small\color{LiveCoMSMediumGrey}Received: \@publisheddatereceived \\ Accepted: \@publisheddateaccepted} 293 | {~\\~} 294 | }% 295 | \rfoot{\small\color{LiveCoMSMediumGrey}\href{https://doi.org/\@publishedDOI}{https://doi.org/\@publishedDOI}\\ 296 | {\it Living J. Comp. Mol. Sci.} ASAP Version 297 | }% 298 | \fi 299 | % 4. Page Number in center of footer 300 | \cfoot{\small\color{white} \vspace{\baselineskip} \small\color{LiveCoMSMediumGrey} \thepage\space of\space\pageref{LastPage}}% 301 | \preto{\footrule}{\color{LiveCoMSMediumGrey}} 302 | \renewcommand{\headrulewidth}{0pt}% % No header rule 303 | \renewcommand{\footrulewidth}{0.4pt}% % No footer rule 304 | %---------------------------------------------------------- 305 | 306 | % 307 | % section/subsection/paragraph set-up 308 | % Updated for LiveComs 309 | % \setcounter{secnumdepth}{0} 310 | \RequirePackage[explicit]{titlesec} 311 | \titleformat{\section} 312 | {\LARGE\bfseries\raggedright} 313 | {\thesection}{1em}{#1}[] 314 | \titleformat{\subsection} 315 | {\Large\bfseries\raggedright\color{LiveCoMSMediumGrey}} 316 | {\thesubsection}{1em}{#1}[] 317 | \titleformat{\subsubsection} 318 | {\large\raggedright\color{LiveCoMSMediumGrey}} 319 | {\thesubsubsection}{1em}{#1}[] 320 | \titleformat{\paragraph} 321 | {\large\raggedright\color{LiveCoMSMediumGrey}} 322 | {\theparagraph}{1em}{#1}[] 323 | \titlespacing*{\section}{0pc}{3ex \@plus4pt \@minus3pt}{0pt} 324 | \titlespacing*{\subsection}{0pc}{2.5ex \@plus3pt \@minus2pt}{0pt} 325 | \titlespacing*{\subsubsection}{0pc}{2ex \@plus2.5pt \@minus1.5pt}{0pt} 326 | \titlespacing*{\paragraph}{0pc}{1.5ex \@plus2pt \@minus1pt}{0pt} 327 | 328 | \RequirePackage{enumitem} 329 | \setlist{noitemsep} 330 | 331 | \RequirePackage{alphalph} 332 | \newalphalph{\fnsymbolmult}[mult]{\@fnsymbol}{5} 333 | 334 | \newcounter{authorfn} 335 | \setcounter{authorfn}{1} 336 | \newcommand{\authfn}[1]{% 337 | \fnsymbolmult{\numexpr\value{authorfn}+#1}% 338 | } 339 | 340 | \def\@correspondence{} 341 | \def\@contribution{} 342 | \def\@presentaddress{} 343 | \def\@deceased{} 344 | % Added blurb for LiveComs 345 | \def\@blurb{} 346 | \def\@orcidblock{} 347 | 348 | 349 | \newcommand{\corr}[2]{% 350 | \ifx\empty\@correspondence\else\appto{\@correspondence}{; }{}{}\fi 351 | \appto{\@correspondence}{% 352 | \url{#1}% 353 | \ifx\empty#2\else\space(#2)\fi 354 | }{}{}% 355 | } 356 | 357 | \newcommand{\contrib}[2][]{ 358 | \appto{\@contribution}{% 359 | \ifx\empty#1\else\textsuperscript{#1}\fi 360 | #2\\ 361 | }{}{} 362 | } 363 | 364 | \newcommand{\presentadd}[2][]{ 365 | \ifx\empty\@presentaddress\else\appto{\@presentaddress}{; }{}{}\fi 366 | \appto{\@presentaddress}{% 367 | \ifx\empty#1\else\textsuperscript{#1}\fi 368 | #2% 369 | }{}{} 370 | } 371 | 372 | \newcommand{\deceased}[1]{\def\@deceased{\textsuperscript{#1}Deceased}} 373 | 374 | % Added for LiveComs 375 | \newcommand{\blurb}[1]{\def\@blurb{#1}} 376 | 377 | \newcommand{\orcid}[2]{% 378 | \ifx\empty\@orcidblock\else\appto{\@orcidblock}{\\}{}{}\fi 379 | \appto{\@orcidblock}{% 380 | #1:\space% 381 | \ifx\empty#2\else\href{https://orcid.org/#2}{#2} \fi 382 | }{}{}% 383 | } 384 | 385 | 386 | 387 | \reversemarginpar 388 | 389 | % 390 | % custom title page 391 | % 392 | \renewcommand{\Authfont}{\bfseries\large\raggedright} 393 | \renewcommand{\Affilfont}{\mdseries\large\raggedright} 394 | \renewcommand{\Authands}{, } 395 | \setlength{\affilsep}{16pt} 396 | \renewcommand{\AB@affilsepx}{; \protect\Affilfont} 397 | 398 | \newcommand{\themetadata}{% 399 | \textbf{*For correspondence:\\} \@correspondence\par 400 | \ifx\empty\@contribution\else 401 | \bigskip\@contribution\par\fi 402 | \ifx\empty\@presentaddress\else 403 | \textbf{Present address: }\@presentaddress\par\fi 404 | \ifx\empty\@deceased\else\@deceased\par\fi 405 | } 406 | 407 | \patchcmd{\@author}{\AB@authlist\\[\affilsep]\AB@affillist}{\AB@authlist\\[\affilsep] 408 | %% Removed for LiveComs; will be placed after abstract in frontmatter 409 | % \marginpar{\raggedright\footnotesize\themetadata\par} 410 | \AB@affillist}{}{} 411 | 412 | %% Added for LiveComs 413 | \RequirePackage{environ} 414 | \RequirePackage{textpos} 415 | 416 | %% Abstract outside frontmatter will throw an error! 417 | \RenewEnviron{abstract}{% 418 | \ClassError{livecoms} 419 | {Nope nope nope, please put the abstract inside the frontmatter environment.} 420 | {Please put the abstract inside the frontmatter environment.} 421 | } 422 | 423 | \NewEnviron{frontmatter}{% 424 | %% Define abstract's behavior when placed in frontmatter 425 | \renewenvironment{abstract}{% 426 | \setlength{\parindent}{0pt} %\raggedright 427 | \raisebox{-16pt-\baselineskip}[0pt][0pt]{\makebox[0pt][r]{\parbox[t]{3cm}{% 428 | \raggedright\itshape\footnotesize\@blurb\par\medskip% 429 | This version dated \@date% 430 | }\hspace*{1cm}}}% 431 | \textcolor{LiveCoMSMediumGrey}{\rule{\textwidth}{2pt}} 432 | \vskip16pt 433 | \textcolor{LiveCoMSLightBlue}{\large\bfseries\abstractname\space} 434 | }{% 435 | \vskip8pt 436 | \textcolor{LiveCoMSMediumGrey}{\rule{\textwidth}{2pt}} 437 | \vskip16pt 438 | } 439 | \twocolumn[% 440 | \protecting{%\begin{minipage}[b]{3cm} 441 | % \small\itshape 442 | % \raggedright\@blurb 443 | % \end{minipage} 444 | \hfill 445 | \begin{minipage}[b]{\textwidth-4cm} 446 | \BODY 447 | \themetadata% 448 | \end{minipage}}\vspace*{2\baselineskip} 449 | ]% 450 | } 451 | 452 | \renewcommand{\maketitle}{% 453 | \vskip36pt% 454 | {\color{LiveCoMSDarkBlue}\raggedright\bfseries\fontsize{22}{27}\selectfont \@title\par}% 455 | \vskip16pt 456 | {\@author\par} 457 | \vskip8pt 458 | } 459 | 460 | \newcommand{\makeorcid}{% 461 | % \textbf{*For correspondence:\\} \@correspondence\par 462 | % \textbf{ORCID:\\} \@correspondence\par 463 | \textbf{ORCID:\\} \@orcidblock\par 464 | } 465 | 466 | %% Insert a grey line to separate floats from main text 467 | \newcommand{\topfigrule}{\vskip8pt\noindent{\rule{\linewidth}{1pt}}} 468 | \newcommand{\botfigrule}{\noindent{\rule{\linewidth}{1pt}}\vskip8pt} 469 | 470 | \RequirePackage{newfloat} 471 | \RequirePackage{wrapfig} 472 | \AtEndEnvironment{wrapfigure}{\vskip8pt\noindent{\rule{\hsize}{1pt}}} 473 | % \RequirePackage[lflt]{floatflt} 474 | % \AtEndEnvironment{floatingfigure}{\vskip8pt\noindent\textcolor{LiveCoMSMediumGrey}{\rule{\hsize}{2pt}}} 475 | 476 | \DeclareFloatingEnvironment[placement=hbt,name=Box]{featurebox} 477 | \captionsetup[featurebox]{font={Large,bf,color=LiveCoMSDarkBlue}} 478 | 479 | \newcounter{featurefigure} 480 | \newcounter{featuretable} 481 | \AtBeginEnvironment{featurebox}{% 482 | \setcounter{featurefigure}{0}% 483 | \setcounter{featuretable}{0}% 484 | \newcommand{\featurefig}[1]{% 485 | \refstepcounter{featurefigure}% 486 | \vskip\smallskipamount% 487 | {\small\textbf{\color{LiveCoMSDarkBlue}Box \arabic{featurebox} Figure \arabic{featurefigure}.}\space #1\par}\medskip} 488 | \newcommand{\featuretable}[1]{% 489 | \refstepcounter{featuretable}% 490 | \vskip\smallskipamount% 491 | {\small\textbf{\color{LiveCoMSDarkBlue}Box \arabic{featurebox} Table \arabic{featuretable}.}\space #1\par}\medskip} 492 | 493 | } 494 | \apptocmd{\featurebox}{% 495 | \begin{mdframed}[linewidth=0pt,backgroundcolor=LiveCoMSLightBlue!10,fontcolor=LiveCoMSDarkBlue] 496 | \if@reqslineno\addtolength{\linenumbersep}{1em}\internallinenumbers\fi% 497 | }{}{} 498 | \pretocmd{\endfeaturebox}{\end{mdframed}}{}{} 499 | 500 | %% Starred version for LiveComs two-column 501 | \AtBeginEnvironment{featurebox*}{% 502 | \setcounter{featurefigure}{0}% 503 | \setcounter{featuretable}{0}% 504 | \newcommand{\featurefig}[1]{% 505 | \refstepcounter{featurefigure}% 506 | \vskip\smallskipamount% 507 | {\small\textbf{\color{LiveCoMSDarkBlue}Box \arabic{featurebox} Figure \arabic{featurefigure}.}\space #1\par}\medskip} 508 | \newcommand{\featuretable}[1]{% 509 | \refstepcounter{featuretable}% 510 | \vskip\smallskipamount% 511 | {\small\textbf{\color{LiveCoMSDarkBlue}Box \arabic{featurebox} Table \arabic{featuretable}.}\space #1\par}\medskip} 512 | } 513 | \expandafter\apptocmd\csname featurebox*\endcsname{% 514 | \begin{mdframed}[linewidth=0pt,backgroundcolor=LiveCoMSLightBlue!10,fontcolor=LiveCoMSDarkBlue] 515 | \if@reqslineno\addtolength{\linenumbersep}{1em}\internallinenumbers\fi% 516 | }{}{} 517 | \expandafter\pretocmd\csname endfeaturebox*\endcsname{\end{mdframed}}{}{} 518 | 519 | %% Unnecessary for LiveComs 520 | % \newenvironment{fullwidth}{% 521 | % \begin{adjustwidth}{-4.5cm}{} 522 | % }{\end{adjustwidth}} 523 | 524 | %% Provide support for pseudocode and algorithms 525 | \RequirePackage{algorithm,algpseudocode} 526 | \captionsetup[algorithm]{% 527 | labelfont={bf},font=small,labelsep=period, 528 | justification=raggedright,singlelinecheck=false} 529 | \newcommand\fs@notopruled{\def\@fs@cfont{\bfseries}\let\@fs@capt\floatc@ruled 530 | \def\@fs@pre{}% \hrule height.8pt depth0pt \kern2pt}% 531 | \def\@fs@post{} %\kern2pt\hrule\relax}% 532 | \def\@fs@mid{\medskip\kern2pt\hrule\kern2pt}% 533 | \let\@fs@iftopcapt\iftrue} 534 | \floatstyle{notopruled} 535 | \restylefloat{algorithm} 536 | \newcommand{\algorithmautorefname}{Algorithm} 537 | \newcommand{\ALG}[1]{\autoref{alg:#1}} 538 | 539 | %% Update some appendix sectional styles 540 | \appto{\appendix}{% 541 | \@addtoreset{figure}{section} 542 | \@addtoreset{table}{section} 543 | \@addtoreset{featurebox}{section} 544 | \@addtoreset{algorithm}{section} 545 | % \numberwithin{figure}{section} 546 | % \numberwithin{table}{section} 547 | % \numberwithin{featurebox}{section} 548 | \titleformat{\section} 549 | {\LARGE\bfseries\color{LiveCoMSDarkBlue}} 550 | {\appendixname\ \thesection}{1em}{#1}[] 551 | 552 | \captionsetup*[figure]{name={Appendix \thesection\ Figure },font={color=LiveCoMSDarkBlue,small},skip=\smallskipamount}% 553 | 554 | \captionsetup*[table]{name={Appendix \thesection\ Table },font={color=LiveCoMSDarkBlue,small}}% 555 | } 556 | 557 | \newcounter{figsupp} 558 | \setcounter{figsupp}{0} 559 | \newcounter{data} 560 | \setcounter{data}{0} 561 | \def\supplist{} 562 | 563 | \RequirePackage{newfile} 564 | \newoutputstream{suppinfo} 565 | \openoutputfile{\jobname.suppinfo}{suppinfo} 566 | 567 | 568 | \AtBeginEnvironment{figure}{% 569 | \setcounter{figsupp}{0} 570 | \setcounter{data}{0} 571 | %% Updated 2017/06/30 to allow optional argument 572 | \newcommand{\figsupp}[3][]{% 573 | \refstepcounter{figsupp}% 574 | {% 575 | \ifstrequal{#1}{none}{}{% 576 | \small\textbf{Figure~\thefigure--Figure supplement \arabic{figsupp}.} \ifstrempty{#1}{#2}{#1}}\par} 577 | \addtostream{suppinfo}{% 578 | \noindent\protect\begin{minipage}{\linewidth} 579 | \protect #3\noexpand\par 580 | \textbf{Figure \thefigure--Figure supplement \arabic{figsupp}.} #2\noexpand\par 581 | \vskip8pt 582 | \protect\end{minipage} 583 | \vskip16pt 584 | } 585 | } 586 | \newcommand{\figdata}[1]{% 587 | \refstepcounter{data} 588 | {\small\textbf{Figure~\thefigure--source data \arabic{data}.} #1}\par 589 | } 590 | } 591 | 592 | %% Added for LiveComs (two columns) 593 | \AtBeginEnvironment{figure*}{% 594 | \setcounter{figsupp}{0} 595 | \setcounter{data}{0} 596 | %% Updated 2017/06/30 to allow optional argument 597 | \newcommand{\figsupp}[3][]{% 598 | \refstepcounter{figsupp}% 599 | {% 600 | \ifstrequal{#1}{none}{}{% 601 | \small\textbf{Figure~\thefigure--Figure supplement \arabic{figsupp}.} \ifstrempty{#1}{#2}{#1}}\par} 602 | \addtostream{suppinfo}{% 603 | \noindent\protect\begin{minipage}{\linewidth} 604 | \protect #3\noexpand\par 605 | \textbf{Figure \thefigure--Figure supplement \arabic{figsupp}.} #2\noexpand\par 606 | \vskip8pt 607 | \protect\end{minipage} 608 | \vskip16pt 609 | } 610 | } 611 | \newcommand{\figdata}[1]{% 612 | \refstepcounter{data} 613 | {\small\textbf{Figure~\thefigure--source data \arabic{data}.} #1}\par 614 | } 615 | } 616 | 617 | \AtBeginEnvironment{table}{% 618 | \setcounter{data}{0} 619 | \newcommand{\tabledata}[1]{% 620 | \refstepcounter{data} 621 | {\small\textbf{Table~\thetable--source data \arabic{data}.} #1}\par 622 | } 623 | } 624 | 625 | %% Added for LiveComs (twocolumns) 626 | \AtBeginEnvironment{table*}{% 627 | \setcounter{data}{0} 628 | \newcommand{\tabledata}[1]{% 629 | \refstepcounter{data} 630 | {\small\textbf{Table~\thetable--source data \arabic{data}.} #1}\par 631 | } 632 | } 633 | 634 | %% Checklists as floats 635 | \RequirePackage{fontawesome} 636 | \DeclareFloatingEnvironment[placement=hbtp,name=Checklists]{Checklists} 637 | \newcounter{checklist} 638 | \AtBeginEnvironment{Checklists}{% 639 | \setcounter{checklist}{0} 640 | \mdfsetup{skipabove=0pt,skipbelow=0pt, 641 | frametitleaboveskip=12pt,innerbottommargin=12pt, 642 | hidealllines=true, 643 | frametitlefont=\Large\bfseries\color{LiveCoMSLightBlue}} 644 | }{}{} 645 | 646 | \AtBeginEnvironment{Checklists*}{% 647 | \setcounter{checklist}{0} 648 | \mdfsetup{skipabove=0pt,skipbelow=0pt, 649 | frametitleaboveskip=12pt,innerbottommargin=12pt, 650 | hidealllines=true, 651 | frametitlefont=\Large\bfseries\color{LiveCoMSLightBlue}} 652 | }{}{} 653 | 654 | \newenvironment{checklist}[1]{% 655 | \stepcounter{checklist} 656 | \ifnumodd{\thechecklist} 657 | {\def\cl@bgcolor{gray!12}} 658 | {\def\cl@bgcolor{gray!25}} 659 | \begin{mdframed}[ 660 | frametitle=\MakeUppercase{#1}, 661 | backgroundcolor=\cl@bgcolor] 662 | \setlist[itemize]{label=$\Box$,leftmargin=*} 663 | }{\end{mdframed}} 664 | 665 | \AtEndDocument{% 666 | \closeoutputstream{suppinfo} 667 | % \pagestyle{empty} 668 | \renewcommand{\footrule}{} 669 | \rfoot{} 670 | \input{\jobname.suppinfo} 671 | } 672 | 673 | %% Use more traditional Appendix section approach 674 | % \newcounter{appendix} 675 | % \setcounter{appendix}{0} 676 | % \newenvironment{appendixbox}{% 677 | % \setcounter{figure}{0} 678 | % \setcounter{table}{0} 679 | % \refstepcounter{appendix}% 680 | % \clearpage% 681 | % \patchcmd{\ttlf@section}{LiveCoMSMediumGrey}{LiveCoMSDarkBlue}{}{} 682 | % \noindent{\bfseries\Large\color{LiveCoMSMediumGrey}Appendix \arabic{appendix}\par} 683 | % \nolinenumbers% 684 | % %% Remove box colours for LiveComs 685 | % \begin{mdframed}[hidealllines=true, 686 | % % backgroundcolor=LiveCoMSLightBlue!10, 687 | % fontcolor=LiveCoMSDarkBlue, 688 | % % leftline=true,linecolor=LiveCoMSLightBlue,linewidth=1em 689 | % ] 690 | % \if@reqslineno\addtolength{\linenumbersep}{2em}\internallinenumbers\fi 691 | % }{% 692 | % \end{mdframed} 693 | % } 694 | 695 | \RequirePackage[colorlinks=true,allcolors=black,citecolor=LiveCoMSLightBlue,linkcolor=LiveCoMSMediumGrey,urlcolor=LiveCoMSLightBlue]{hyperref} 696 | \urlstyle{sf} 697 | 698 | % Other desired commands 699 | \renewcommand{\equationautorefname}{Eq.} 700 | \newcommand{\FIG}[1]{\autoref{fig:#1}} 701 | \newcommand{\TABLE}[1]{\autoref{tab:#1}} 702 | \newcommand{\EQ}[1]{\autoref{eq:#1}} 703 | \newcommand{\BOX}[1]{\autoref{box:#1}} 704 | \let\oldautoref\autoref 705 | \renewcommand{\autoref}[1]{\emph{\textbf{\oldautoref{#1}}}} 706 | 707 | \endinput 708 | -------------------------------------------------------------------------------- /main.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/dmzuckerman/Sampling-Uncertainty/8cd46e29067297e7ff6288f56d0e131c214d6fa6/main.pdf -------------------------------------------------------------------------------- /main.tex: -------------------------------------------------------------------------------- 1 | %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 2 | %%% LIVECOMS ARTICLE TEMPLATE FOR BEST PRACTICES GUIDE 3 | %%% ADAPTED FROM ELIFE ARTICLE TEMPLATE (8/10/2017) 4 | %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 5 | %%% PREAMBLE 6 | \documentclass[9pt,bestpractices,pubversion]{livecoms} 7 | % Use the 'onehalfspacing' option for 1.5 line spacing 8 | % Use the 'doublespacing' option for 2.0 line spacing 9 | % Use the 'lineno' option for adding line numbers. 10 | % The 'bestpractices' option for indicates that this is a best practices guide. 11 | % Omit the bestpractices option to remove the marking as a LiveCoMS paper. 12 | % Please note that these options may affect formatting. 13 | 14 | \usepackage{lipsum} % Required to insert dummy text 15 | \usepackage[version=4]{mhchem} 16 | \usepackage{siunitx} 17 | \usepackage{bm} 18 | \DeclareSIUnit\Molar{M} 19 | \usepackage[italic]{mathastext} 20 | \newcommand{\versionnumber}{1.0} % you should update the minor version number in preprints and major version number of submissions. 21 | \newcommand{\githubrepository}{\url{https://github.com/dmzuckerman/Sampling-Uncertainty}} %this should be the main github repository for this article 22 | \graphicspath{{figures/}} 23 | %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 24 | %%% ARTICLE SETUP 25 | %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 26 | \title{Best Practices for Quantification of Uncertainty and Sampling Quality in Molecular Simulations [Article v\versionnumber]} 27 | 28 | \author[1*\authfn{1}]{Alan Grossfield} 29 | \author[2*\authfn{1}]{Paul N. Patrone} 30 | \author[3*\authfn{1}]{Daniel R. Roe} 31 | \author[4*\authfn{1}]{Andrew J. Schultz} 32 | \author[5*\authfn{1}]{Daniel W. Siderius} 33 | \author[6*\authfn{1}]{Daniel M. Zuckerman} 34 | \affil[1]{University of Rochester Medical Center, Department of Biochemistry and Biophysics} 35 | \affil[2]{Applied Computational and Mathematics Division, National Institute of Standards and Technology} 36 | \affil[3]{Laboratory of Computational Biology, National Heart Lung and Blood Institute, National Institutes of Health} 37 | \affil[4]{Department of Chemical and Biological Engineering, University at Buffalo, The State University of New York} 38 | \affil[5]{Chemical Sciences Division, National Institute of Standards and Technology} 39 | \affil[6]{Department of Biomedical Engineering, Oregon Health \& Science University} 40 | 41 | \corr{alan_grossfield@urmc.rochester.edu}{AG} 42 | \corr{paul.patrone@nist.gov}{PNP} 43 | \corr{daniel.roe@nih.gov}{DRR} 44 | \corr{ajs42@buffalo.edu}{AJS} 45 | \corr{daniel.siderius@nist.gov}{DWS} 46 | \corr{zuckermd@ohsu.edu}{DMZ} 47 | 48 | \orcid{Alan Grossfield}{0000-0002-5877-2789} 49 | \orcid{Paul N. Patrone}{0000-0002-9446-8186} 50 | \orcid{Daniel R. Roe}{0000-0002-5834-2447} 51 | \orcid{Andrew J. Schultz}{0000-0002-1180-0156} 52 | \orcid{Daniel W. Siderius}{0000-0002-6260-7727} 53 | \orcid{Daniel M. Zuckerman}{0000-0001-7662-2031} 54 | 55 | \contrib[\authfn{1}]{These authors contributed equally to this work.} 56 | 57 | \blurb{This LiveCoMS document is maintained online on GitHub at \githubrepository; to provide feedback, suggestions, or help improve it, please visit the GitHub repository and participate via the issue tracker.\\ 58 | \bigskip 59 | Contribution of the National Institute of Standards and Technology, not subject to US copyright. 60 | } 61 | 62 | %% Common symbols 63 | \newcommand{\stdev}[1]{s\left( #1 \right)} 64 | \newcommand{\stdevmean}[1]{s\left( #1 \right)} 65 | \newcommand{\expval}[1]{\langle #1 \rangle} 66 | \newcommand{\mean}[1]{\bar{#1}} 67 | \newcommand{\expvar}[1]{\sigma_{#1}^2} 68 | \newcommand{\var}[1]{s^2\left( #1 \right)} 69 | 70 | %\presentadd[\authfn{3}]{Department, Institute, Country} 71 | %\presentadd[\authfn{4}]{Department, Institute, Country} 72 | 73 | %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 74 | %%% PUBLICATION INFORMATION 75 | %%% Fill out these parameters when available 76 | %%% These are used when the "pubversion" option is invoked 77 | %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 78 | \pubDOI{10.33011/livecoms.1.1.5067} 79 | \pubvolume{1} 80 | \pubissue{1} 81 | \pubyear{2019} 82 | \articlenum{5067} 83 | \datereceived{20 June 2018} 84 | \dateaccepted{17 October 2018} 85 | 86 | %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 87 | %%% ARTICLE START 88 | %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 89 | 90 | \begin{document} 91 | 92 | \begin{frontmatter} %NOTE: to make the document single column, just take out the {frontmatter} sentinels 93 | \maketitle 94 | 95 | \begin{abstract} 96 | The quantitative assessment of uncertainty and sampling quality is essential in molecular simulation. 97 | Many systems of interest are highly complex, often at the edge of current computational capabilities. Modelers must therefore analyze and communicate statistical uncertainties so that ``consumers'' of simulated data understand its significance and limitations. This article covers key analyses appropriate for trajectory data generated by conventional simulation methods such as molecular dynamics and (single Markov chain) Monte Carlo. It also provides guidance for analyzing some `enhanced' sampling approaches. We do not discuss \emph{systematic} errors arising, e.g., from inaccuracy in the chosen model or force field. 98 | \end{abstract} 99 | 100 | \end{frontmatter} 101 | 102 | \section{Introduction: Scope and definitions} 103 | \label{sec:scope} 104 | \input{scope} 105 | \input{definitions} 106 | \input{checklist.tex} 107 | \input{pre-sim-sanity.tex} 108 | \input{quick-and-dirty.tex} 109 | \input{equilibration-burn-in.tex} 110 | \input{global-sampling.tex} 111 | \input{specific-observables.tex} 112 | \input{enhanced-sampling.tex} 113 | \input{conclusion.tex} 114 | 115 | 116 | \section*{Acknowledgments} 117 | The authors appreciate helpful discussions with Pascal T. Merz (University of Colorado-Boulder), comments on the text from John Chodera (Memorial Sloan Kettering Cancer Center), Lillian T.~Chong (University of Pittsburgh) and William R.~Smith (University of Guelph), and valuable feedback from Harold W.~Hatch, Richard A.~Messerly, Raymond D.~Mountain, and Andrew M.~Dienstfrey in their roles as NIST reviewers. 118 | DMZ acknowledges support from NIH Grant GM115805. 119 | 120 | \section*{Disclaimer} 121 | Certain commercially available items may be identified in this paper. This identification does not imply recommendation by NIST, nor does it imply that it is the best available for the purposes described. 122 | 123 | \section*{Author Information} 124 | \makeorcid 125 | 126 | \bibliography{refs} 127 | 128 | \end{document} 129 | -------------------------------------------------------------------------------- /pre-sim-sanity.tex: -------------------------------------------------------------------------------- 1 | \section{Pre-simulation ``sanity checks'' and planning tips} 2 | \label{sec:sanity} 3 | 4 | Sampling a molecular system that is complex enough to be ``interesting'' in modern science is often extremely challenging, and similar difficulties apply to studies of ``simple'' systems \cite{Schappals2017}. 5 | Therefore, a small amount of effort spent planning a study can pay off many times over. In the worst case, a poorly planned study can lead to weeks or months of simulations and analyses that yield questionable results. 6 | 7 | With this in mind, one of the objectives of this document is to provide a set of benchmark practices against which reviewers and other scientists can judge the quality of a given work. If you read this guide in its entirety \emph{before} performing a simulation, you will have a much better sense of what constitutes (in our minds) a thoughtful simulation study. Thus, we strongly advise that readers review and understand the concepts presented here, as well as in related reviews \cite{Grossfield2009,JCGM:GUM2008,PatroneUQreview} 8 | 9 | In a generic sense, the overall goal of a computational study is to be able to draw statistically significant conclusions regarding a particular phenomenon. To this end, ``good statistics'' usually follow from repeated observations of a quantity-of-interest. While such information can be obtained in a number of ways, time-series data are a natural output of many simulations and is therefore a commonly used to achieve the desired sampling.\footnote{Note that the ``time-series'' descriptor here can also refer to a sequence of states in the Markov chain of a Monte Carlo simulation.} 10 | It is important to recognize that time-series data usually displays a certain amount of autocorrelation in the sense that the numerical values of nearby points in the series tend to cluster close to one another. Intuition dictates that correlated data does not reveal fully ``new'' information about the quantity-of-interest, and so we require uncorrelated samples to achieve meaningful sampling \cite{PatroneAIAA}.\footnote{This intuition is, strictly speaking, misguided in that anti-correlated samples actually {\it increases} our knowledge of a given random quantity relative to decorrelated samples. See, for example, the discussion in Ref.~\cite{PatroneAIAA}.} 11 | 12 | Thus, it is critical to ask: \emph{what are the pertinent timescales of the system?} 13 | Unfortunately, this question must be answered individually for each system. You will want to study the experimental and computational literature for your particular system, although we warn that a published prior simulation of a given length does not in itself validate a new simulation of a similar or slightly increased length. In the end, your data should be examined using statistical tools, such as the autocorrelation analysis described in Secs.~\ref{sec:zeroth} and \ref{sec:autocorrelation}. Be warned that a system may possess states (regions of configuration space) that, although important, are \emph{never} visited in a given simulation set because of insufficient computational time \cite{Grossfield2009} and, furthermore, this type of error will not be discovered through the analyses presented below. 14 | Finally, note that "system" here does not necessarily refer to a complete simulation (e.g., a biological system with protein, solvent, ions, etc); it can also refer to some subset of the simulation for which data are desired. For example, if one is only interested in the dynamics of a binding site in a protein, it probably is not necessary to observe the unfolding and refolding of that protein as well. 15 | 16 | One general strategy that will allow you to understand the relevant timescales in a system is to perform several repeats of the same simulation protocol. As described below, repeats can be used to assess variance in \emph{any} observable within the time you have run your simulation. 17 | When performing repeat simulations, it is generally advised to use different starting states which are as diverse as possible; then, differences among the runs can be an indicator of inadequate sampling of the equilibrium distribution. 18 | Alternatively, performing multiple runs from the same starting state will yield behavior particular to that starting state; information about (potential) equilibrium is obtained only if the runs are long enough. 19 | 20 | \begin{figure} 21 | \centering 22 | \includegraphics[width=0.9\linewidth]{figures/1d-landscape-tslow} 23 | \caption{ 24 | \label{fig:landscape} 25 | Schematic illustration of a free energy landscape dominated by a slow process. 26 | The timescales associated with a system will often reflect ``activated'' (energy-climbing) processes, although they could also indicate diffusion times for traversing a rough landscape with many small barriers. 27 | In the figure, the largest barrier is associated with the slowest timescale $t_{\mathrm{slow}}$, and the danger for conventional MD simulations is that the total length of the simulation may be inadequate to generate the barrier crossing. 28 | } 29 | \end{figure} 30 | 31 | A toy model illustrates some of these timescale issues and their effects on sampling. 32 | Consider the ``double well'' free energy landscape shown in Fig.\ \ref{fig:landscape}, and note that the slowest timescale is associated with crossing the largest barrier. Generally, you should expect that the value of \emph{any} observable (e.g., $x$ itself or another coordinate not shown or a function of those coordinates) will depend on which of the two dominant basins the system occupies. In turn, the equilibrium average of an observable will require sampling the two basins according to their equilibrium populations. In order to directly sample these basins, however, the length of a trajectory will have to be orders of magnitude greater than the slowest timescale, i.e., the largest barrier should be crossed multiple times. Only in this way can the relative populations of states be inferred from time spent in each state. Stated differently, the equilibrium populations follow from the transition rates \cite{Zuckerman2011,Chou11,Kolmogoroff1936} which can be estimated from multiple events. For completeness, we note that there is no guarantee that sampling of a given system will be limited by a dominant barrier. Instead, a system could exhibit a generally rough landscape with many pathways between states of interest. 33 | Nevertheless, the same cautions apply. 34 | 35 | What should be done if a determination is made that a system's timescales are too long for direct simulation? 36 | The two main options would be to consider a more simplified (``coarse-grained'') model \cite{Merchant2011,Kmiecik2016} or an enhanced sampling technique (see Sec.~\ref{sec:enhanced}). 37 | Modelers should keep in mind that enhanced sampling methods are not foolproof but have their own limitations which should be considered carefully. 38 | 39 | Lastly, whatever simulation protocol you pursue, be sure to use a well-validated piece of software [\url{https://github.com/shirtsgroup/software-physical-validation}]. 40 | If you are using your own code, check it against independent simulations on other software for a system that can be readily sampled, e.g., Ref.~\cite{NIST_SRSW}. 41 | 42 | -------------------------------------------------------------------------------- /quick-and-dirty.tex: -------------------------------------------------------------------------------- 1 | % !TEX root = ./main.tex 2 | 3 | \section{Qualitative and semiquantitative checks that can rule out good sampling} 4 | \label{sec:quick} 5 | 6 | It is difficult to establish with certainty that good sampling has been achieved, but it is not difficult to \emph{rule out} high-quality sampling. 7 | Here we elaborate on some relatively simple tests that can quickly bring out inadequacies in sampling. 8 | 9 | Generally speaking, analysis routines that extract information from raw simulated data are often formulated on the basis of physical intuition about how that data should behave. Before proceeding to quantitative data analysis and uncertainty quantification, it is therefore useful to assess the extent to which data conforms to these expectations and the requirements imposed by either the modeler or the analysis routines. Such tasks help reduce subjectivity of predictions and offer insight into when a simulation protocol should be revisited to better understand its meaningfulness \cite{patrone1}. Unfortunately, general recipes for assessing data quality are impossible to formulate, owing to the range of physical quantities of interest to modelers. Nonetheless, several example procedures will help clarify the matter. 10 | 11 | \subsection{Zeroth-order system-wide tests} 12 | \label{sec:zeroth} 13 | 14 | The simplest test for poor sampling is lack of equilibration: if the system is still noticeably relaxing from its starting conformation, statistical sampling has not even begun, and thus by definition is poor. As a result, the very first test should be to verify that the basic equilibration has occurred. To check for this, one should inspect the time series for a number of simple scalar values, such as potential energy, system size (and area, if you are simulating a membrane or other system where one dimension is distinct from the others), temperature (if you are simulating in the NVE ensemble), and/or density (if simulating in the isothermal-isobaric ensemble). 15 | 16 | Simple visual inspection is often sufficient to determine that the simulation is systematically changing, although more sophisticated methods have been proposed (see Sec.\ \ref{sec:equil}). If \emph{any} value appears to be systematically changing, %then the system is not equilibrated. 17 | then the system may not be equilibrated and further investigation is warranted. 18 | See, for example, the time trace in Fig.\ \ref{f:rmsd}. 19 | After the rapid rise, the value shows slower changes; 20 | however, it does not fluctuate repeatedly about an average value, implying it has not been well-sampled. 21 | 22 | 23 | \subsection{Tests based on configurational distance measures (e.g., RMSD)} 24 | \label{sec:bio_RMSD} 25 | 26 | \begin{figure} 27 | \centering 28 | \includegraphics[width=0.9\linewidth]{figures/rmsd} 29 | \includegraphics[width=0.9\linewidth]{figures/rmsds} 30 | \caption{RMSD as a measure of convergence. The upper panel shows the 31 | $\alpha$-carbon RMSD of the protein rhodopsin from its starting structure as a 32 | function of time. The lower panel shows the all-to-all RMSD map computed from the same 33 | trajectory. Color scale is the RMSD, in {\AA}. Data from Ref.\ \cite{Grossfield-2015}.} 34 | \label{f:rmsd} 35 | \end{figure} 36 | 37 | 38 | Because a system with $N$ particles has a $3N$ dimensional configuration-space (the full set of $x, y, z$ coordinates), it is generally difficult to assess the extent to which the simulation has adequately explored these degrees-of-freedom. Thus, modelers often project out all but a few degrees of freedom, e.g., monitoring a ``distance'' in configuration space as described below or keeping track of only certain dihedral angles. In this lower dimensional subspace, it can be easier to track transitions between states and monitor similarity between configurations. However, the interpretation of such analyses requires care. 39 | 40 | By employing a configuration-space ``distance'', several useful qualitative checks can be performed. 41 | Such a distance is commonly employed in biomolecular simulations (e.g., RMSD, defined below) but analogous measures could be employed for other types of systems. 42 | A configuration-space distance is a simple scalar function quantifying the similarity between two molecular configurations and can be used in a variety of ways to probe sampling. 43 | 44 | To understand the basic idea behind using a distance to assess sampling, consider first a one-dimensional system, as sketched in Fig.\ 1. If we perform a simulation and monitor the $x$ coordinate alone, without knowing anything about the landscape, we can get an idea of the sampling performed simply by monitoring $x$ as a function of time. If we see numerous transitions among apparent metastable regions (where the $x$ values fluctuates rapidly about a local mean), we can conclude that sampling likely was adequate \emph{for the configuration space seen in the simulation.} An important caveat is that we know nothing about states that were never visited. On the other hand, if the time trace of $x$ changes primarily in just one direction or exhibits few transitions among apparent metastable regions, we can conclude that sampling was poor -- again without knowledge of the energy landscape. 45 | 46 | The same basic procedures (and more) can be followed once we precisely define a configurational distance between two configurations. A typical example is the root-mean-square deviation, 47 | \begin{equation} 48 | \label{eq:rmsd} 49 | \mathrm{RMSD}(\mathbf{r}, \mathbf{s}) = \sqrt{\frac{1}{N}\sum_{i=1}^{N} \left| \mathbf{r}_i - \mathbf{s}_i \right|^2} \;, 50 | \end{equation} 51 | where $\mathbf{r}_i$ and $\mathbf{s}_i$ are the Cartesian coordinates of atom $i$ in two distinct configurations $\mathbf{r}$ and $\mathbf{s}$ which have been optimally aligned \cite{Kabsch1976}, so that the RMSD is the minimum ``distance'' between the configurations. 52 | It is not uncommon to use only a subset of the atoms (e.g., protein backbone, only secondary structure elements) when computing the RMSD, in order to filter out the higher-frequency fluctuations. 53 | Another configuration-space metric is the dihedral angle distance which sums over all distances for pairs of selected angles. 54 | Note that configurational distances \emph{generally} suffer from the degeneracy problem: the fact that many different configurations can be the same distance from any given reference. This is analogous to 55 | the increasing number of points in three-dimensional space 56 | with increasing radial distance from a reference point, except 57 | much worse because of the dimensionality. For an 58 | exploration of expected RMSD distributions for biomolecular 59 | systems see the work of Pitera \cite{Pitera2014}. 60 | 61 | 62 | Some qualitative tools for assessing global sampling based on RMSD were reviewed 63 | in prior work \cite{Grossfield2009}. The classic time-series plot of RMSD with 64 | respect to a crystal or other single reference structure (Fig.\ \ref{f:rmsd}) can immediately 65 | indicate whether the structure is still systematically changing. Although this 66 | kind of plot was historically used as a sampling test, it should really be 67 | considered as another equilibration test like those discussed above. Moreover, 68 | it is not even a particularly good test of equilibration, because the degeneracy 69 | of RMSD means you cannot tell if the simulation is exploring new states that are 70 | equidistant from the chosen reference. The upper panel of Fig.\ \ref{f:rmsd} 71 | shows a typical curve of this sort, taken from a simulation of the G 72 | protein-coupled receptor rhodopsin \cite{Grossfield-2015}; the curve increases 73 | rapidly over the few nanoseconds and then roughly plateaus. It is difficult to 74 | assign meaning to the other features on the curve. 75 | 76 | A better RMSD-based convergence measure is the all-to-all RMSD plot; taking the 77 | RMSD of each snapshot in the trajectory with respect to all others allows you to 78 | use RMSD for what it does best, identifying very similar structures. The lower 79 | panel of Fig.\ \ref{f:rmsd} shows an example of this kind of plot, applied to 80 | the same rhodopsin trajectory. By definition, all such plots have values of zero 81 | along the diagonal, and occupation of a given state shows up as a block of 82 | similar RMSD along the diagonal; in this case, there are 2 main states, with one 83 | transition occurring roughly 800 ns into the trajectory. Off diagonal ``peaks'' 84 | (regions of low RMSD between structures sampled far apart in time) indicate that 85 | the system is revisiting previously sampled states, a necessary condition for 86 | good statistics. In this case, the initial state is never sampled after the 87 | first transition as seen from the lack of low RMSD values following $\sim$800 ns with respect to configurations prior to that point; 88 | however, there are a number of small transitions within the second state based on low RMSD values occurring among configurations following $\sim$800 ns. 89 | 90 | \subsection{Analyzing the qualitative behavior of data} 91 | 92 | 93 | In many cases, analysis of simulated outputs relies on determining or extracting information from a regime in which data are expected to behave a certain way. For example, we might anticipate that a given dataset should have linear regimes or more generically look like a convex function. However, typical sources of fluctuations in simulations often introduce noise that can distort the character of data and thereby render such analyses difficult or even impossible to approach objectively. It is therefore often useful to systematically assess the extent to which raw data conforms to our expectations and requirements. 94 | 95 | In the context of materials science, simulations of yield-strain $\epsilon_y$ (loosely speaking, the deformation at which a material fails) provide one such example. In particular, intuition and experiments tells us that upon deforming a material by a fraction $1+\epsilon$, it should recover its original dimensions if $\epsilon \le \epsilon_y$ and have a residual strain $\epsilon_r = \epsilon - \epsilon_y$ if $\epsilon \ge \epsilon_y$ \cite{patrone2}. Thus, residual-strain data should exhibit bilinear behavior, with slopes indicating whether the material is in the pre- or post-yield regime. 96 | 97 | In experimental data, these regimes are generally distinct and connected by a sharp transition. In simulated data, however, the transition in $\epsilon_r$ around yield is generally smooth and not piece-wise linear, owing to the timescale limitations of MD. Thus, it is useful to perform analyses that can objectively identify the asymptotic regimes without need for input from a modeler. One way to achieve this is by fitting residual strain to a hyperbola. In doing so, the proximity of data to the asymptotes illustrates the extent to which simulated $\epsilon_r$ conforms to the expectation that $\epsilon_r=0$ when $\epsilon < \epsilon_y$. See Fig.~\ref{fig:yield} and Refs.~\cite{patrone1,patrone2} for more examples and discussion. 98 | 99 | \begin{figure} 100 | \centering 101 | \includegraphics[width=0.9\linewidth]{figures/hyperbola.png} 102 | \caption{Residual strain $\epsilon_r$ as a function of applied strain $\epsilon$. Blue $\times$ denote simulated data, whereas the smooth curve is a hyperbola fit to the data. The green lines are asymptotes; their intersection can be taken as an estimate of $\epsilon_y$. Bounds on yield are computed by the synthetic data method discussed in Sec.~\ref{sec:bootstrap}. {\it From, ``Estimation and uncertainty quantification of yield via strain recovery simulations,'' P.\ Patrone, CAMX 2016 Conference Proceedings. Reprinted courtesy of the National Institute of Standards and Technology, U.S. Department of Commerce. Not copyrightable in the United States.}} 103 | \label{fig:yield} 104 | \end{figure} 105 | 106 | While extending this approach to other types of simulations invariably depends on the problem at hand, we recognize a few generic principles. In particular, it is sometimes possible to test the quality of data by fitting it to {\it global} (not piece-wise or local!) functions that exhibit characteristics we desire of the former. By testing the goodness of this fit, we can assess the extent to which the data captures the entire structure of the fit-function and therefore conforms to expectations. We note that this task can even be done in the absence of a known fit function, given only more generic properties such as convexity. See, for example, the discussion in Ref.~\cite{PatroneAIAA}. 107 | 108 | 109 | \subsection{Tests based on independent simulations and related ideas} 110 | 111 | \begin{figure} 112 | \centering 113 | \includegraphics[width=0.9\linewidth]{figures/combinedcluster} 114 | \caption{Combined clustering between two independent trajectories as a measure of convergence. The X axis is the population of a cluster from trajectory 1, while the Y axis is the population of that cluster from trajectory 2. Cluster populations are show after 20, 100, and 800 ns of sampling. The simulations used to generate the data used in this plot are described in Ref.\ \citep{Roe2014}.} 115 | \label{f:combinedcluster} 116 | \end{figure} 117 | 118 | When estimating any statistical property, multiple measurements are required to characterize the underlying model with high confidence. Consider, for example, the probability that an unbiased coin will land heads-up as estimated in terms of the relative fraction coin-flips that give this result. This fraction approximated in terms of a single flip (measurement) will always yield a grossly incorrect probability, since only one outcome (heads or tails) can ever be represented by this procedure. However, as more flips (measurements) are made, the relative fraction of outcomes will converge to the correct probability, i.e., the former represents and increasingly good estimate of the latter. 119 | 120 | In an analogous way, we often use ``convergence'' in the context of simulations to describe the extent to which an estimator (e.g., an arithmetic mean) approaches some true value (i.e., the corresponding expectation of an observable) with increasing amounts of data. In many cases, however, the true value is not known {\it a priori}, so that we cannot be sure what value a given estimator should be approaching. In such cases, it is common to use the overlap of independent estimates and confidence intervals as a proxy for convergence because the associated clustering suggests a shared if unknown mean. Conversely, lack of such ``convergence'' is a strong indication that sampling is poor. 121 | 122 | 123 | There are two approaches to obtaining independent measurements. Arguably the best is to have multiple independent simulations, each with different initial conditions. Ideally these conditions should be chosen so as to span the space to be sampled, which provides confidence that simulations are not being trapped in a local minimum. Consider, for example, the task of sampling the $\phi$ and $\psi$ torsions of alanine dipeptide. To accomplish this, one could initialize these angles in the alpha-helical conformation and then run a second simulation initialized in the polyproline II conformation. It is important to note, however, that the starting conditions only need to be varied enough so that the desired space is sampled. For example, if the goal is to sample protein folding and unfolding, there should be some simulations started from the folded conformation and some from the unfolded, but if it is not important to consider protein folding, initial unfolded conformations may not be needed. 124 | 125 | However, the ``many short trajectories'' strategy has a number of limitations that must be also be considered. First, as a rule one does not know the underlying ensemble in advance (else, we might not need to do the simulation!), which complicates the generation of a diverse set of initial states. When simulating large biomolecules (e.g. proteins or nucleic acids), ``diverse'' initial structures are often constructed using the crystal or NMR structure coupled with randomized placement of surrounding water molecules, ions, etc. If the true ensemble contains protein states with significant structural variations, it is possible that no number of short simulations would actually capture transitions, particularly if the transitions themselves are slow. In that case, each individual trajectory must be of significant duration in order to have any meaning relevant to the underlying ensemble. The minimum duration needed to achieve significance is highly system dependent, and estimating it in advance requires an understanding of the relevant timescales in the system and what properties are to be calculated. Second, one must equilibrate each new trajectory, which can appreciably increase the computational cost of running many short trajectories, depending on the quality of the initial states. 126 | 127 | One can also try to estimate statistical uncertainties directly from a single simulation by dividing it into two or more subsets (``blocks''). However this can at times be problematic because it can be more difficult to tell if the system is biased by shared initial conditions (e.g., trapped in a local energy minimum). Those employing this approach should take extra care to assess their results (see Sec.~\ref{sec:blockavg}). 128 | 129 | 130 | Autocorrelation analyses applied to trajectory blocks can be used to better understand the extent to which a time series represents an equilibrated system. In particular, systems at steady state (which includes equilibrium) by definition have statistical properties that are time-invariant. Thus, correlations between a single observable at different times depend only on the relative spacing (or ``lag'') between the time steps. That is, the autocorrelation function of observable $x$, denoted $C$, has the stationarity property 131 | \begin{equation} 132 | C(x_k,x_{k+j}) \equiv \dfrac{ 133 | \overline{ 134 | \left(x_{k{\color{white}j}} - \overline{x} \right) 135 | \left(x_{k+j} - \overline{x} \right) 136 | } 137 | } 138 | {s\left(x\right)^2} 139 | \Rightarrow C_j 140 | \label{def:obs_ACF} 141 | \end{equation} 142 | where $C_j$ is independent of the time step $k$. With this in mind, one can partition a given time series into continuous blocks, compute the autocorrelation for a collection of lags $j$, and compare between blocks. Estimates of $C_j$ that are independent of the block suggest an equilibrated (or at least a steady-state) system, whereas significant changes in the autocorrelation may indicate an unequilibrated system. Importantly, this technique can help to distinguish long-timescale trends in apparently equilibrated data. 143 | 144 | 145 | \subsubsection*{Combined Clustering} 146 | 147 | Cluster analysis is a means by which data points are grouped together based on a similarity (or distance) metric. For example, cluster analysis can be used to identify the major conformational substates of a biomolecule from molecular dynamics trajectory data using coordinate RMSD as a distance metric. For an in-depth discussion of cluster analysis as applied to biomolecular simulations data, see Ref.\ \citep{Shao2007}. 148 | 149 | One useful technique for evaluating convergence of structure populations is so-called "combined clustering". Briefly, in this method two or more independent trajectories are combined into a single trajectory (or a single trajectory is divided into two or more parts), on which cluster analysis is performed. Clusters represents groupings of configurations for which intra-group similarity is higher than inter-group similarity \citep{Okur2006}. 150 | 151 | The resulting clusters are then split according to the trajectory (or part of the trajectory) they originally came from. If simulations are converged then each part will have similar populations for any given cluster. Indications of poor convergence are large deviations in cluster populations, or clusters that show up in one part but not others. Figure \ref{f:combinedcluster} shows results from combined clustering of two independent trajectories as a plot of cluster population fraction from the first trajectory compared to the second. If the two independent trajectories are perfectly converged then all points should fall on the X=Y line. As simulation time increases the cluster populations from the independent trajectories are in better agreement, which indicates the simulations are converging. For another example of performing combined cluster analysis see Ref.\ \citep{Bergonzo2014}. 152 | -------------------------------------------------------------------------------- /releases/LiveCoMS_Article_ASAP_V1.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/dmzuckerman/Sampling-Uncertainty/8cd46e29067297e7ff6288f56d0e131c214d6fa6/releases/LiveCoMS_Article_ASAP_V1.pdf -------------------------------------------------------------------------------- /releases/LiveCoMS_Article_V1.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/dmzuckerman/Sampling-Uncertainty/8cd46e29067297e7ff6288f56d0e131c214d6fa6/releases/LiveCoMS_Article_V1.pdf -------------------------------------------------------------------------------- /releases/header_V1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/dmzuckerman/Sampling-Uncertainty/8cd46e29067297e7ff6288f56d0e131c214d6fa6/releases/header_V1.png -------------------------------------------------------------------------------- /scope.tex: -------------------------------------------------------------------------------- 1 | \subsection{Scope} 2 | 3 | Simulating molecular systems that are interesting by today's standards, whether for biomolecular research, materials science, or a related field, is a challenging task. 4 | However, computational scientists are often dazzled by the system-specific issues that emerge from such problems and fail to recognize that even ``simple'' simulations (e.g., alkanes) require significant care \cite{Schappals2017}. In particular, questions often arise regarding the best way to adequately sample the desired phase-space or estimate uncertainties. And while such questions are not unique to molecular modeling, their importance cannot be overstated: the usefulness of a simulated result ultimately hinges on being able to confidently and accurately report uncertainties along with any given prediction \cite{Nicholls2014}. In the context of techniques such as molecular dynamics (MD) and Monte Carlo (MC), these considerations are especially important, given that even large-scale modern computing resources do not guarantee adequate sampling. 5 | 6 | This article therefore aims to provide best-practices for reporting simulated observables, assessing confidence in simulations, and deriving uncertainty estimates (more colloquially, ``error bars'') based on a variety of statistical techniques applicable to physics-based sampling methods and their associated ``enhanced'' counterparts. As a general rule, we advocate a tiered approach to computational modeling. In particular, workflows should begin with back-of-the-envelope calculations to determine the feasibility of a given computation, followed by the actual simulation(s). Semi-quantitative checks can then be used to check for adequate sampling and assess the quality of data. Only once these steps have been performed should one actually construct estimates of observables and uncertainties. In this way, modelers avoid unnecessary waste by continuously gauging the likelihood that subsequent steps will be successful. Moreover, this approach can help to identify seemingly reasonable data that may have little value for prediction and/or be the result of a poorly run simulation. 7 | 8 | It is worth emphasizing that in the last few years, many works have developed and advocated for uncertainty quantification (UQ) methods not traditionally used in the MD and MC communities. In some cases, these methods buck trends that have become longstanding conventions, e.g., the practice of only using uncorrelated data to construct statistical estimates. One goal of this manuscript is therefore to advocate newer UQ methods when these are demonstrably better. Along these lines, we wish to remind the reader that better results are not only obtained from faster computers, but also by using data more thoughtfully. 9 | It is also important to appreciate that debate continues even among professional statisticians on what analyses to perform and report \cite{Leek2017}. 10 | 11 | The reader should be aware that there is not a ``one-size-fits-all'' approach to UQ. Ultimately, we take the perspective that uncertainty quantification in its broadest sense aims to provide actionable information for making decisions, e.g., in an industrial research and development setting or in planning future academic studies. A simulation protocol and subsequent analysis of its results should therefore take into account the intended audience and/or decisions to be made on the basis of the computation. In some cases, quick-and-dirty workflows can indeed be useful if the goal is to only provide order-of-magnitude estimates of some quantity. We also note that uncertainties can often be estimated through a variety of techniques, and there may not be consensus as to which, if any, are best. {\it Thus, a critical component of any UQ analysis is communication, e.g., of the assumptions being made, the UQ tools used, and the way that results are interpreted.} Educated decisions can only be made through an understanding of both the process of estimating uncertainty and its numerical results. 12 | 13 | While UQ is a central topic of this manuscript, our scope is limited to issues associated with sampling and related uncertainty estimates. We do not address systematic errors arising from inaccuracy of force-fields, the underlying model, or parametric choices such as the choice of a thermostat time-constant. See, for example, Refs.~\cite{Leimkuhler,Rizzi2,Rizzi3,Rizzi4} for methods that address such problems. Similarly, we did not address bugs and other implementation errors, which will generally introduce systematic errors. Finally, we do not consider model-form error and related issues that arise when comparing simulated predictions with experiment. Rather, we take the raw trajectory data at face value, assuming that it is a valid description of the system of interest.\footnote{In more technical UQ language, we restrict our scope to {\it verification} of simulation results, as opposed to {\it validation}. Readers may consult \url{https://github.com/MobleyLab/basic_simulation_training} and \url{https://github.com/shirtsgroup/software-physical-validation} regarding foundations of molecular simulation and validation of simulation results, respectively.} 14 | -------------------------------------------------------------------------------- /specific-observables.tex: -------------------------------------------------------------------------------- 1 | \section{Computing uncertainty in specific observables} 2 | \label{sec:specific} 3 | 4 | \subsection{Basics} 5 | 6 | Here we address the simple but critical question, ``What error bar should I report?'' 7 | In general, there is no one-best practice for choosing error bars \cite{Nicholls2014}. However, in the context of simulations, we can nonetheless identify common goals when reporting such estimates: 1) to help authors and readers better understand uncertainty in data; and 2) to provide readers with realistic information about the reproducibility of a given result. 8 | 9 | With this in mind, we recommend the following: (a) in fields where there is a definitive standard for reporting uncertainty, the authors should follow existing conventions; (b) otherwise, such as for biomolecular simulations, \emph{authors should report (and graph) their best estimates of 95~\% \hyperref[def:conf_int]{confidence intervals}.} 10 | (c) when feasible and especially for a small number of independent measurements ($n < 10$), authors should consider plotting all of the points instead of an average with error bars. 11 | 12 | We emphasize that as opposed to standard uncertainties [$\stdevmean{\mean{x}}$], confidence intervals have several practical benefits that justify their usage. In particular, they directly quantify the range in which the average value of an observed quantity is expected to fall, which is more relatable to everyday experience than, say, the moments of a probability distribution. As such, confidence intervals can help authors and readers better understand the implications of an uncertainty analysis. Moreover, downstream consumers of a given paper may include less statistically oriented readers for whom confidence intervals are a more meaningful measure of variation. 13 | 14 | In a related vein, error bars expressed in integer multiples of $\stdevmean{\mean{x}}$ can be misinterpreted as unrealistically under or overestimating uncertainty if taken at face value. For example, reporting $3\stdevmean{\mean{x}}$ uncertainties for a normal random variable amounts to a 99.7~\% level of confidence, which is likely to be a significant overestimate for many applications. On the other hand, $1\stdevmean{\mean{x}}$ uncertainties only correspond to a 68~\% level of confidence, which may be too low. Given that many readers may not take the time to make such conversions in their heads, we feel that it is safest for modelers to explicitly state the confidence level of their error bar or reported confidence interval. 15 | 16 | In recommending 95~\% confidence intervals, we are admittedly attempting to address a social issue that nevertheless has important implications for science as a whole. In particular, the authors of a study and the reputation of their field do not benefit in the long run by under-representing uncertainty, since this may lead to incorrect conclusions. Just as importantly, many of the same problems can arise if uncertainties are reported in a technically correct but obscure and difficult-to-interpret manner. For example, $1\stdevmean{\mean{x}}$ error bars may not overlap and thereby mask the inability to statistically distinguish two quantities, since the corresponding confidence intervals are only 68~\%. With this in mind, we therefore wish to emphasize that visual impressions conveyed by figures in a paper are of primary importance. Regardless of what a research paper may explain carefully in text, error bars on graphs create a lasting impression and must be as informative and accurate as possible. If 95~\% confidence intervals are reported, the expert reader can easily estimate the smaller standard uncertainty (especially if it is noted in the text), but showing a graph with overly small error bars is bound to mislead most readers, even experts who do not search out the fine print. 17 | 18 | As a final note, we remind readers that only significant figures should be reported. 19 | Additional digits beyond the precision implicit in the uncertainty are unhelpful at best, and potentially misleading to readers who may not be aware of the limitations of simulations or statistical analyses generally. 20 | For example, if the mean of a quantity is calculated to be $1.23456$ with uncertainty $\pm 0.1$ based on a 95~\% confidence interval, then only two significant figures should be reported for the mean ($1.2$). 21 | 22 | 23 | \subsection{Overview of procedures for computing a confidence interval} 24 | We remind readers that they should perform the semiquantitative sampling checks (Sec.\ \ref{sec:quick}) before attempting to quantify uncertainty. If the observable of interest is not fluctuating about a mean value but largely increasing or decreasing during the course of a simulation, a reliable quantitative estimate for the observable or its associated uncertainty cannot be obtained. 25 | 26 | For observables passing the qualitative tests noted above in Sec.\ \ref{sec:quick}, we advocate obtaining confidence intervals in one of two ways: 27 | \begin{itemize} 28 | \item For observables that are Gaussian-distributed (or assumed to be, as an approximation or due to lack of information), an appropriately chosen \emph{coverage factor} $k$ (typically in the range of 2 to 3; see Sec.~\ref{sec:conf_int} for further details) is multiplied by the standard uncertainty $\stdevmean{\mean{x}}$ to yield the expanded uncertainty, which estimates the 95~\% confidence interval. 29 | \item For non-Gaussian observables, a \emph{bootstrapping} approach (Sec.~\ref{sec:bootstrap}) should be used. 30 | An example of a potentially non-Gaussian observable is a rate-constant, which must be positive but could exhibit significant variance. As such, a confidence interval estimated with a coverage factor may lead to an unphysical negative lower limit. 31 | In contrast, bootstrapping does not assume an underlying distribution but instead constructs a confidence interval based on the recorded data values, and the limits cannot fall outside the extreme data values; 32 | nevertheless, bootstrapped confidence intervals can have shortcomings \cite{Schenker1985,Chernick2009}. 33 | Bootstrapping is also sometimes useful for estimating uncertainties associated with \hyperref[def:deriv_obs]{derived observables}. 34 | \end{itemize} 35 | 36 | Below we describe approaches for estimating the standard uncertainty $\stdevmean{\mean{x}}$ from a single trajectory with a coverage factor $k$ as well as the bootstrapping approach for direct confidence-interval estimation. Whether using a coverage factor and standard uncertainty or bootstrapping, one requires an estimate for the independent number of observations in a given simulation. This requires care, but may be accomplished based on the effective sample size described in Sec.~\ref{sec:global}, via block averaging, or by analysis of a time-correlation function. However, these methods have their limitations and must be used with caution. In particular, both block averaging and autocorrelation analyses will produce effective sample sizes that depend on the quantity of interest. To produce reliable answers, one must therefore identify and track the slowest relevant degree of freedom in the system, which can be a non-trivial task. Even apparently fast-varying properties may have significant statistical error if they are coupled to slower varying ones, and this error in uncertainty estimation may not be readily identifiable by solely examining the fast-varying time series. 37 | 38 | In the absence of a reliable estimate for the number of independent observations, one can perform $n$ independent simulations and calculate the standard deviation $\stdev{x}$ for quantity $x$ (which could be the ensemble average of a raw data output or a derived observable) among the $n$ simulations, yielding a standard uncertainty of $\stdevmean{\mean{x}} = \stdev{x} / \sqrt{n}$. When computing the uncertainty with this approach, it is important to ensure that each starting configuration is also independent or else to recognize and report that the uncertainty refers to simulations started from a particular configuration. The means to obtain independent starting configurations is system-dependent, but might involve repeating the protocol used to construct a configuration (solvating a protein, inserting liquid molecules in a box, etc.), and/or using different seeds to generate random configurations, velocities, etc. However, readers are cautioned that \emph{for complex systems, it may be effectively impossible to generate truly independent starting configurations pertinent to the ensemble of interest.} For example, a simulation of a protein in water will nearly always start from the experimental structure, which introduces some correlation in the resulting simulations even when the remaining simulation components (water, salt, etc.) are regenerated {\it de novo}. 39 | 40 | \subsection{Dealing with correlated time-series data} 41 | 42 | When samples of a simulated observable are independent, the experimental standard deviation of the mean (i.e., Eq.~\hyperref[def:exp_st_dev]{\ref{def:exp_st_dev_mean}}) can be used as an estimate of the corresponding standard uncertainty. Due to correlations, however, the number of independent samples in a simulation is neither equal to the number of observations nor known {\it a priori}; thus Eq.~\ref{def:exp_st_dev_mean} is not directly useful. To overcome this problem, a variety of techniques have been developed to estimate the effective number of independent samples in a dataset. Two methods in particular have gained considerable traction in recent years: (i) autocorrelation analyses, which directly estimate the number of independent samples in a time series; and (ii) block averaging, which projects a time series onto a smaller dataset of (approximately) independent samples. We now discuss these methods in more detail. 43 | 44 | 45 | \subsubsection{Autocorrelation method for estimating the standard uncertainty}\label{sec:autocorrelation} 46 | 47 | Conceptually, autocorrelation analyses directly compute the effective number of independent samples $N_{\rm ind}$ in a time series, taking into account ``redundant'' (or even possibly new) information arising from correlations.\footnote{It is worth pointing out that correlations do not always provide redundant information. Consider, for example, the time series $1,-1,1,-1,1,-1,...$. In the limit that the number of elements goes to infinity, the arithmetic mean also converges to zero. However, a block of $2n$ entries also has a mean of zero, so that (anti)correlations effectively increase the amount of information. See also Ref.~\cite{PatroneAIAA}. } In particular, this approach invokes the fact that the statistical properties of steady-state simulations (e.g., those in equilibrium or non-equilibrium steady state) are, by definition, time-invariant. As such, correlations between an observable computed at two different times depends only on the lag (i.e., difference) between those times, not their absolute values. 48 | 49 | This observation motivates one to compute an autocorrelation function. Specifically, one computes the stationary autocorrelation function $C_j$ as given in Eq.~\ref{def:obs_ACF} for a set of lags $j$. 50 | Then, the number of independent samples is estimated by\footnote{The reader should note that both the autocorrelation function (Eq.~\ref{def:obs_ACF}) and the number of independent samples (Eq.~\ref{def:Nind}) may be written in different forms\cite{Grossfield2009,Chodera-2016}. Our convention here presents the observations as a list $\left\{x_j\right\}$ in which the time interval (Molecular Dynamics) or trial spacing (Monte Carlo) of adjacent $x_j$ is implicitly fixed. For time-series data, one could alternately write both the observations and autocorrelation function as continuous functions of time, e.g., $x\left(t\right)$ and $C\left(\tau\right)$ where $\tau$ is the lag time. In that case, $N_{\rm ind}$ is written as a division of the total simulation time by the time integral of $C\left(\tau\right)$\cite{Grossfield2009}.} 51 | % 52 | \begin{equation} 53 | N_{\rm ind} = \dfrac{N}{1+2 \sum_{j=1}^{N_{\rm max}} C_{j}} 54 | \label{def:Nind} 55 | \end{equation} 56 | % 57 | where $N_{\rm max}$ is an appropriately chosen maximum number of lags (see below). Note that $N_{\rm ind}$ need not be an integer. Finally, the standard uncertainty is estimated via 58 | % 59 | \begin{equation} 60 | \stdevmean{\mean{x}} = \dfrac{\stdev{x}}{\sqrt{N_{\rm ind}}} 61 | \label{def:ACF_std_unc} 62 | \end{equation} 63 | % 64 | We note that the \hyperref[def:exp_st_dev]{experimental standard deviation} of the observable $x$ is used in Eq.~\ref{def:ACF_std_unc} to estimate the uncertainty. Strictly speaking, the standard uncertainty should be estimated using the true standard deviation of $x$ (e.g., $\sigma_x$); given that the true standard deviation is unknown, the experimental standard deviation is used in its place as an estimate of $\sigma_x$ \cite{PatroneAIAA}. 65 | 66 | In evaluating Eq.~\eqref{def:Nind}, the value of $N_{max}$ must be chosen with some care. 67 | Roughly speaking, $N_{max}$ should be large enough so that the sum is converged and insensitive to the choice of upper bound. Although a very large value of $N_{max}$ might seem necessary for slowly decaying autocorrelation functions, appropriate truncations of the sum will introduce negligible error, even if the correlation time is infinite. We refer readers to discussions elsewhere on this topic, for example, Refs.~\cite{Janke_Statistical_2002,Shirts_Statistically_2008,Grossfield2009,Gowers_Automated_2018}. In typical situations, $N_{max}$ can be set to any value greater than $\tau$, since in principle $C_{j}=0$ for all $j > \tau$. However, care must be exercised to avoid integrating pure noise over too large of an interval, since this can generate Brownian motion; see, for example, Ref.~\cite{acfnoise} and references contained therein. 68 | 69 | %In evaluating Eq.~\eqref{def:Nind}, we observe that $\tau$ may in fact be infinite, e.g. if the autocorrelation time decays exponentially or according to some power law. In principle, this does not significantly alter the sum in Eq.~\eqref{def:Nind} (except that $N_{\max} \to \infty$), since the sum is usually convergent. However, in such cases there may be ambiguity in how to pick a value of $N_{\max}$ that is useful for practical computation, since appropriate truncations of the sum will introduce negligible error. Unfortunately a deeper analysis of this issue and recommendations on how to proceed are largely problem specific, and we direct the reader to discussions elsewhere. See, for example, Refs.~\cite{Janke_Statistical_2002,Shirts_Statistically_2008,Grossfield2009,Gowers_Automated_2018}. 70 | 71 | %In the situations in which $N_{\max}$ is finite (or a finite approximation has been chosen), we also observe that this quantity can be set to any value greater than $\tau$, since $C_j = 0$ (or $C_j \approx 0$ in the case of finite approximation) for all $j>\tau$. In practice, however, setting $N_{\rm max}$ too large can sometimes cause the sum to be dominated by noise appearing in the estimate of $C_j$. See, for example, Ref.~\cite{acfnoise} and references contained therein. Thus, care must be taken when choosing an appropriate value of $N_{\rm max}$, making sure that the estimate is sufficiently large so that the sum is converged but otherwise independent of this upper limit. 72 | 73 | \subsubsection{Block averaging method for estimating the standard uncertainty}\label{sec:blockavg} 74 | 75 | The main idea behind block averaging is to permit the direct usage of Eq.~\ref{def:exp_st_dev} by projecting the original dataset onto one comprised of only independent samples, so that there is no need to compute $N_{\rm ind}$. Acknowledging that typical MD time series have a finite-correlation time $\tau$, we recognize that a continuous block of $M$ data-points will only be correlated with its adjacent blocks through its first and last $\tau$ points, provided $\tau$ is small compared to the block size $M$. That is, correlations will be on the order of $\tau / M$, which goes to zero in the limit of large blocks. 76 | 77 | This observation motivates a technique known as block averaging \cite{Friedberg1970,Flyvbjerg-1989,FrenkelSmit2002,Grossfield2009}. 78 | Briefly, the set of $N$ observations $\left\{x_1, ..., x_N\right\}$ are converted to a set of $M$ ``block averages'' $\left\{x^b_1, ..., x^b_{M}\right\}$, where a block average $x^b_j$ is the \hyperref[def:arith_mean]{arithmetic mean} of $n$ (the block size) sequential measurements of $x$: 79 | % 80 | \begin{equation} 81 | x^b_j = \dfrac{\sum\limits_{k=1+(j-1)n}^{jn} x_k}{n} 82 | \end{equation} 83 | % 84 | From this set of block averages, one may then compute the arithmetic mean of the block averages, $\mean{x}^b$, which is an estimator for $\expval{x}$.\footnote{Note that in general $\mean{x} \ne \mean{x}^b$ if the block averages discard some observations of $x_l$, for example, when the total number of observations is not a multiple of the block size.} 85 | Following, one computes the \hyperref[def:exp_st_dev]{experimental standard deviation} of the block averages, $\stdev{\mean{x}^b}$, using Eq.~\ref{def:exp_st_dev}. 86 | Lastly, the standard uncertainty of $\mean{x}^b$ is just the \hyperref[def:exp_st_dev_mean]{experimental standard deviation of the mean} given the set of $M$ block averages: 87 | % 88 | \begin{equation} 89 | \stdevmean{\mean{x}^b} = \dfrac{\stdev{x^b}}{\sqrt{M}} 90 | \end{equation} 91 | % 92 | This standard uncertainty may then be used to calculate a confidence interval on $\mean{x}^b$. 93 | 94 | It is important to note that for statistical purposes, the blocks must all be of the same size in order to be identically distributed, and thereby satisfy the requirements of Eq.~\ref{def:exp_st_dev_mean}. It is also important to systematically assess the impact of block size on the corresponding estimates. In particular, as the blocks get longer, the block averages should decorrelate and $\stdevmean{\mean{x}^b}$ should plateau \cite{Flyvbjerg-1989,Grossfield2009}. 95 | Another approach is to measure the block correlation and to use it to improve the selection of the block size and, hence, uncertainty estimate \cite{Kolafa1986}. 96 | We stress that this final step of adjusting the block size and recomputing the block standard uncertainty is absolutely necessary. Otherwise, the blocks may be correlated, yielding an uncertainty that is not meaningful. 97 | 98 | 99 | \subsection{Propagation of uncertainty}\label{subsec:linear} 100 | 101 | Oftentimes we run simulations for the purposes of computing derived quantities, i.e., those that arise from some analysis applied to raw data. In such cases, it is necessary to propagate uncertainties in the raw data through the corresponding analysis to arrive at the uncertainties associated with the derived quantity. Frequently, this can be accomplished through a linear propagation analysis using Taylor series, which yields simple and useful formulas. 102 | 103 | The foundation for this approach lies in rigorous results for the propagation of error through linear functions of random variables. 104 | For a derived observable that is a linear function of $M$ uncorrelated raw data measurements, e.g., 105 | % 106 | \begin{equation} 107 | F\left( \left\{x_i\right\} \right) = c + \sum_{i=1}^M a_i x_i, 108 | \label{def:linear_derived_obs} 109 | \end{equation} 110 | % 111 | where $c$ is a constant, the experimental variance of $F$ may be rigorously expressed as \cite{NIST_Sematech_eHandbook} 112 | % 113 | \begin{equation} 114 | \var{F} = \sum_{i=1}^M a_i^2 \var{x_i} 115 | \label{def:var_linear_derived} 116 | \end{equation} 117 | % 118 | A key assumption in Eq.~\ref{def:var_linear_derived} is that the raw data, $\left\{x_i\right\}$, are \hyperref[def:unc_obs]{linearly uncorrelated} (see Eq.~\ref{def:unc_obs}). If any observed quantities are correlated, the uncertainty in $F$ must include ``covariance'' terms. The reader may consult Sec.\ 2.5.5, ``Propagation of error considerations'' in Ref.~\cite{NIST_Sematech_eHandbook} for further discussion. For reasons of tractability, we restrict the discussion here to linearly uncorrelated observables or the assumption thereof. 119 | 120 | The situation for a nonlinear derived quantity is much more complicated and, as a result, rigorous expressions for the uncertainty of such functions are rarely used in practice. As a simplification, however, one approximates the nonlinear derived quantity as a Taylor-series expansion about a reference point, i.e., 121 | % 122 | \begin{equation} 123 | F\left( \left\{x_i\right\} \right) 124 | \approx 125 | F\left( \left\{\mean{x_i}\right\} \right) 126 | + \sum_{j=1}^M \left. 127 | \left( \dfrac{\partial F}{\partial x_j}\right) 128 | \right|_{\left\{\mean{x_i}\right\}} 129 | \left(x_j - \mean{x_j}\right) + \cdots 130 | \label{def:nonlinear_derived_obs_taylor} 131 | \end{equation} 132 | % 133 | The deviation of a particular measurement from its mean, $\epsilon_i = x_i - \mean{x_i}$, is itself a random quantity, and the uncertainty in those measurements is propagated into uncertainty in $F$. 134 | Note that the ratio $\epsilon_i/\mean{x_i}$ is the so-called ``noise-to-signal'' ratio, which vanishes in the limit of a precise measurement. 135 | With this {\it linear} approximation of $F$, which is analogous to Eq.~\ref{def:linear_derived_obs} with $a_i = \left( \partial F/\partial x_i\right)$, and the assumption that the raw data are uncorrelated, the variance in $F$ may be approximated by 136 | % 137 | \begin{eqnarray} 138 | \var{F} & \approx & \sum_{j=1}^M 139 | \left. 140 | \left( \dfrac{\partial F}{\partial x_j} \right)^2 141 | \right|_{\left\{\mean{x_i}\right\}} 142 | \var{x_j} 143 | \label{def:var_nonlinear_derived} 144 | \end{eqnarray} 145 | % 146 | 147 | A simple example illustrates this procedure. 148 | Consider, in particular, the task of estimating the uncertainty in a measurement of density, $\rho=m/V$, from a time series of volumes output by a constant pressure simulation, where $m$ is the (constant) system mass and $V$ is the (fluctuating) system volume. 149 | Application of Eq.~\ref{def:var_nonlinear_derived} to the definition of $\rho$ yields 150 | % 151 | \begin{eqnarray} 152 | \var{\rho} & \approx & \left( \dfrac{N}{\mean{V}^2} \right)^2 \var{V} \\ 153 | \stdev{\rho} & \approx & \mean{\rho} \dfrac{\stdev{V}}{\mean{V}} 154 | \label{ex:density_linearized} 155 | \end{eqnarray} 156 | % 157 | with $\mean{\rho} = N/\mean{V}$. 158 | This approximation of the experimental standard deviation may be used to estimate a confidence interval on $\mean{\rho}$ or for other purposes. 159 | 160 | In general, approximations in the spirit of Eq.~\ref{ex:density_linearized} are useful and easy to generalize to higher-dimensional settings in which the derived observable is a nonlinear combination of many data-points or sets. 161 | However, the method does have limits. 162 | In particular, it rests on the assumption of a small noise-to-signal ratio, which may not be valid for all simulated data. 163 | If there is doubt as to the quality of an estimate, the uncertainty should therefore be estimated with alternative approaches such as bootstrapping in order to validate the linear approximation. 164 | See also the pooling analysis of Ref.~\cite{patrone1} for a method of assessing the validity of linear approximations. 165 | 166 | \subsection{From standard uncertainty to confidence interval for Gaussian variables}\label{sec:conf_int} 167 | 168 | Once a standard uncertainty value is obtained for a Gaussian-distributed random variable with mean $\left< x \right>$, and the number of independent samples $n$ has been estimated, the 95~\%-confidence interval $\left[ \mean{x} -k \, \stdevmean{\mean{x}} , \mean{x} +k \, \stdevmean{\mean{x}} \right]$ can be constructed on the basis of an established look-up table (or a statistics software model) for the coverage factor $k$ based on $n$. The theoretical basis for the table is the ``Student'' or ``$t$'' distribution, which is \emph{not} Gaussian, but governs the behavior of an \emph{average} derived from $n$ independent Gaussian variables \cite{JCGM:GUM2008}. Table \ref{tab:coveragefactors} lists $k$ for two-sided 95~\% confidence intervals for select values of $n$. 169 | 170 | \begin{table} 171 | \centering 172 | \begin{tabular}{S S} 173 | \toprule 174 | {$n$ (independent samples)} & {$k$ (coverage factor)} \\ 175 | \hline 176 | 6 & 2.57 \\ 177 | 11 & 2.23 \\ 178 | 16 & 2.13 \\ 179 | 21 & 2.09\\ 180 | 26 & 2.06 \\ 181 | 51 & 2.01 \\ 182 | 101 & 1.98 \\ 183 | \bottomrule 184 | \end{tabular} 185 | \caption{Coverage factors $k$ required for a two-sided 95~\% confidence interval for a Gaussian variable \cite{JCGM:GUM2008}. Note that $k$ increases with decreasing sample size. This in turn implies that smaller samples yield higher uncertainty for a given estimation of the experimental standard deviation.} 186 | \label{tab:coveragefactors} 187 | \end{table} 188 | 189 | As a reminder, multi-modally distributed variables with multiple peaks in their distributions cannot be considered Gaussian random variables. Variables with a strict upper or lower limit (such as a non-negative quantity) and long-tailed distributions are also not Gaussian. These cases should be treated with bootstrapping. 190 | 191 | \subsection{Bootstrapping}\label{sec:bootstrap} 192 | 193 | Bootstrapping is an approach to uncertainty estimation that does not assume a particular distribution for the observable of interest or a particular kind of relationship between the observable and variables directly obtained from simulation \cite{Tibshirani1998}. A full discussion of bootstrapping and resampling methods is outside the scope of this article; we will cover the broad strokes here, and suggest interested readers consult excellent discussions elsewhere for more details (e.g. Refs.~\cite{Schenker1985}, \cite{Chernick2009}, and, particularly, \cite{Tibshirani1998}). 194 | 195 | In nonparametric bootstrapping, new, ``synthetic'' data sets (corresponding to hypothetical simulation runs) are created by drawing $n$ samples (configurations) from the original collection that was generated during the actual run. The same sample may be selected twice, while others may not be selected at all in a process called ``sampling with replacement.'' In doing so, these synthetic sets will be different even though they all have the same number of samples and draw from the same pool of data. Having created a new set, the data are analyzed to determine the derived quantity of interest, and this process is repeated to produce multiple estimates of the quantity. The distribution of ``synthetic'' observables can be directly used to construct a 95~\% confidence interval from the 2.5 percentile to the 97.5 percentile value. 196 | Readers are cautioned that bootstrapping confidence intervals are not quantitatively reliable in certain cases such as with small-sample sizes or distributions that are skewed or heavy-tailed \cite{Schenker1985,Chernick2009}. 197 | 198 | The process described above assumes that the original simulation data are uncorrelated. If this is not the case, then the resampling method can be reformulated in one of two ways. The first option is to estimate the number of independent samples in the original set (e.g., using an autocorrelation method \cite{Chodera-2016,Lyman2007a}) and to pull only that many samples to create the new data sets. The second option is to group the samples into blocks that are uncorrelated based on analyzing varying block sizes (see Sec.~\ref{sec:blockavg}) and to then use the block averages as the samples for bootstrapping. 199 | 200 | Alternatively, one could use the difference between errors estimated via block averaging and bootstrapping as a measure of the correlation; if one tracks the bootstrapped and block-averaged estimates of a quantity's uncertainty as a function of block size, the only difference between the two modes of calculation is whether the data are correlated. The decay in the ratio of the two quantities as a function of time is a measure of the correlation time in the sample \cite{Romo2011}. 201 | 202 | \subsubsection{Bootstrapping variants} 203 | 204 | An alternate approach that can directly account for correlations is called parametric bootstrapping. The main idea behind this method is to model the original data as a deterministic function (which can be zero, constant, or have free parameters) plus additive noise. The parameters of this model, including the structure of the noise (i.e., its covariance), can be determined through a statistical inference procedure. Having calibrated the model, random number generators can be used to sample the noise, which is then added back to the trial function to generate a synthetic data set. As with the nonparametric bootstrap, the generated data can be used to compute the derived quantity of interest, and the uncertainty can be obtained from the statistics of the values compute with different generated sets. 205 | 206 | To further clarify the procedure of parametric bootstrapping, consider the simplest case in which the data are a collection of uncorrelated random variables fluctuating about a constant mean. In this situation, one could estimate (I) the deterministic part of a parametric model using the sample mean $\mean{x}$ of the data, and (II) the stochastic part as a Gaussian random variable whose variance equals the sample variance. If instead the data are correlated (e.g., as in a time series of simulated observables), one can postulate a covariance function to describe the structure of this randomness. Often these covariance functions are formulated with free parameters (often called ``hyperparameters'') that characterize properties such as the noise-scale and characteristic length of correlations \cite{Rasmussen}. In such cases, determining the hyperparameters may require more sophisticated techniques such as maximum likelihood analyses or Bayesian approaches; see, for example, Ref.~\cite{Rasmussen}. See also Refs.~\cite{patrone1,patrone2,patrone3} for examples and practical implementations applied to cases in which the deterministic component of the data is not constant. 207 | 208 | It is important to note that various bootstrapping approaches can and often are used as uncertainty propagation tools. Nonetheless, care should be exercised when using such methods with either nonlinear functions or naturally constrained functions. For example, consider application of the parametric bootstrap technique to a free energy calculation, 209 | % 210 | \begin{equation} 211 | \beta \Delta A = - \ln \left[ \overline{\exp\left(-\beta \Delta U\right)} \right] 212 | \end{equation} 213 | % 214 | If synthetic samples are drawn for $\exp(-\beta \Delta U)$ from, say, a Gaussian centered at 1 with a standard deviation of 0.5, new estimates will eventually output negative numbers. This is, of course, mathematically nonsensical for the exponential function. Thus, one should be aware of any distributional assumptions imposed either by the physics of the problem or the mathematical analyses of synthetic data. 215 | 216 | Lastly, an alternative to bootstrapping is the ``jackknife'' method \cite{Quenouille_Approximate_1949,Quenouille_Notes_1956,Tukey_Bias_1958} (also outlined in Ref.~\cite{Tibshirani1998}). It operates similar to the bootstrap as a resampling technique, but it uses synthetic data sets created by subtraction of some number of samples rather than replacement; as such it is often categorized as a variant of the bootstrap (even though it predates the bootstrap). Since it operates by sample deletion, it may be better suited to smaller data sets for which a few samples may be overrepresented in the synthetic data set created by the bootstrap replacement technique. Ultimately, though, the results are similar in that the jackknife technique creates a distribution of derived observables that can be used to compute an arithmetic mean, an estimate of the standard uncertainty, and confidence intervals. 217 | 218 | \subsection{Dark uncertainty analyses} 219 | 220 | In some cases, multiple simulations of the same physical observable $\tau$ may yield predictions whose error bars do not overlap. This situation can arise, for example, in simulations of the glass transition temperature when undersampling the crosslinked network structure of certain polymers. In such cases, it is reasonable to postulate an unaccounted for source of uncertainty, which we colorfully refer to as ``dark uncertainty'' \cite{patrone1}. In the context of a statistical model, we postulate that the probability of a simulation output depends on the unobserved or ``true'' mean value $\bar \tau$, an uncertainty $\sigma_i^2$ whose value is specific to the simulation (estimated, e.g., according to uncertainty propagation), and the unaccounted-for dark uncertainty $y^2$. (For simplicity, the $\sigma_i^2$ and $y^2$ should be treated as variances.) 221 | 222 | While details are beyond this scope of this document, such a model motivates an estimate of $\bar \tau$ of the form 223 | \begin{align} 224 | \bar \tau \approx \mathcal T \propto \sum_i \frac{T_i}{\sigma_i^2 + y^2}, \label{eq:darkmean} 225 | \end{align} 226 | where $\mathcal T$ is a ``consensus'' or weighted-mean estimate of the true mean $\bar \tau$, $T_i$ is the prediction from the $i$th simulation, $\sigma_i^2$ is its associated ``within-simulation'' uncertainty, and $y^2$ is the ``dark'' or ``between-simulation'' uncertainty; note that the latter does not depend on $i$. The variable $y^2$ can be estimated from a maximum-likelihood analysis of the data and amounts to numerically solving a relatively simple nonlinear equation (see Ref.~\cite{patrone1}). Equation~\ref{eq:darkmean} is useful insofar as it weights simulated results according to their certainty while reducing the impact of overconfident predictions (e.g., having small $\sigma_i^2$). Additional details on this method are provided in Ref.~\cite{patrone1} and the references contained therein. 227 | 228 | \subsection{Propagation across multiple steps} 229 | In some instances, it is useful and/or necessary to propagate uncertainty across different simulation steps. This occurs, for example, when the property of a constant pressure system can only be computed using constant volume simulations (e.g., for certain viscosity calculations). In such cases, uncertainty in the system volume must be accounted for in the final property prediction, which carries its own uncertainties associated with the simulation protocol, input parameters, etc. Related issues arise when attempting to account for uncertainty in force-field parameters \cite{Rizzi2,Rizzi3,Rizzi4}. 230 | 231 | Addressing these uncertainty propagation tasks may be as simple as performing a linear or bootstrap analysis as previously described, but at each step of the simulation protocol. In other cases, especially when propagation must be performed between different simulations, such approaches are computationally expensive and, thus, infeasible. A variety of methods (surrogate modeling, polynomial chaos, Gaussian-process regression, etc.) are being actively explored by the community, but in many instances few, if any approaches, have emerged as widely accepted strategies. We encourage the reader to stay informed of current literature, some of which is referenced herein \cite{smith2013uncertainty}. 232 | --------------------------------------------------------------------------------