├── Queues.md
├── README.md
├── Topic Outline.md
├── applications.md
├── cluster_overview.md
├── cluster_utilization.md
├── code_of_conduct.md
├── directory_structure.md
├── facilities_statement.md
├── intro_to_bash.md
├── nodes.md
├── questions_comments.md
├── researchIT_team.md
└── storage.md


/Queues.md:
--------------------------------------------------------------------------------
  1 | The high performance computing (HPC) resources of The Jackson Laboratory represent a shared resource available to all JAX researchers, and in order to keep these resources available to all researchers in a consistent and fair manner, a number of walltime-based queues have been implemented on these resources.  These queues allow the Information Technology department the ability to better plan maintenance and schedule upgrade windows on these systems, while providing a more consistent and stable operating environment for JAX HPC users.
  2 | 
  3 | ### Identifying the partitions
  4 | 
  5 | ### squeue
  6 | 
  7 | The squeue account provides details about many things including the partitions configured currently
  8 | 
  9 | ~~~
 10 | squeue
 11 | squeue -u <username>
 12 | squeue --help
 13 | ~~~
 14 | 
 15 | Examples:
 16 | 
 17 | squeue
 18 | 
 19 | ~~~
 20 | JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
 21 |             643231       gpu RnnTrain   zhaoyu PD       0:00      1 (Resources)
 22 |             643232       gpu RnnTrain   zhaoyu PD       0:00      1 (Priority)
 23 | 639633_[68-135%30]       gpu track_gr   sheppk PD       0:00      1 (JobArrayTaskLimit)
 24 |        552775_7337   compute checkBug   zhaoyu PD       0:00      1 (launch failed requeued held)
 25 |             627649   compute    build   tewher  R 1-03:00:18      1 sumner054
 26 |             627650   compute    build   tewher  R 1-03:00:18      1 sumner054
 27 |             627651   compute    build   tewher  R 1-03:00:18      1 sumner054
 28 |             627652   compute    build   tewher  R 1-03:00:18      1 sumner054
 29 |             627648   compute    build   tewher  R 1-03:00:28      1 sumner054
 30 | ~~~
 31 | 
 32 | squeue -u tewher
 33 | 
 34 | ~~~
 35 | JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
 36 |             627649   compute    build   tewher  R 1-03:02:14      1 sumner054
 37 |             627650   compute    build   tewher  R 1-03:02:14      1 sumner054
 38 |             627651   compute    build   tewher  R 1-03:02:14      1 sumner054
 39 |             627652   compute    build   tewher  R 1-03:02:14      1 sumner054
 40 |             627648   compute    build   tewher  R 1-03:02:24      1 sumner054
 41 |             638914   compute    build   tewher  R   22:07:11      1 sumner014
 42 |             638981   compute    build   tewher  R   22:07:11      1 sumner014
 43 |             638982   compute    build   tewher  R   22:07:11      1 sumner039
 44 |             638984   compute    build   tewher  R   22:07:11      1 sumner054
 45 |             638985   compute    build   tewher  R   22:07:11      1 sumner014
 46 | ~~~
 47 | 
 48 | squeue --help
 49 | 
 50 | ~~~
 51 | Usage: squeue [OPTIONS]
 52 |   -A, --account=account(s)        comma separated list of accounts
 53 | 				  to view, default is all accounts
 54 |   -a, --all                       display jobs in hidden partitions
 55 |       --array-unique              display one unique pending job array
 56 |                                   element per line
 57 |       --federation                Report federated information if a member
 58 |                                   of one
 59 |   -h, --noheader                  no headers on output
 60 |       --hide                      do not display jobs in hidden partitions
 61 |   -i, --iterate=seconds           specify an interation period
 62 |   -j, --job=job(s)                comma separated list of jobs IDs
 63 |                                   to view, default is all
 64 |       --local                     Report information only about jobs on the
 65 |                                   local cluster. Overrides --federation.
 66 |   -l, --long                      long report
 67 |   -L, --licenses=(license names)  comma separated list of license names to view
 68 |   -M, --clusters=cluster_name     cluster to issue commands to.  Default is
 69 |                                   current cluster.  cluster with no name will
 70 |                                   reset to default. Implies --local.
 71 |   -n, --name=job_name(s)          comma separated list of job names to view
 72 |       --noconvert                 don't convert units from their original type
 73 |                                   (e.g. 2048M won't be converted to 2G).
 74 |   -o, --format=format             format specification
 75 |   -O, --Format=format             format specification
 76 |   -p, --partition=partition(s)    comma separated list of partitions
 77 | 				  to view, default is all partitions
 78 |   -q, --qos=qos(s)                comma separated list of qos's
 79 | 				  to view, default is all qos's
 80 |   -R, --reservation=name          reservation to view, default is all
 81 |   -r, --array                     display one job array element per line
 82 |       --sibling                   Report information about all sibling jobs
 83 |                                   on a federated cluster. Implies --federation.
 84 |   -s, --step=step(s)              comma separated list of job steps
 85 | 				  to view, default is all
 86 |   -S, --sort=fields               comma separated list of fields to sort on
 87 |       --start                     print expected start times of pending jobs
 88 |   -t, --states=states             comma separated list of states to view,
 89 | 				  default is pending and running,
 90 | 				  '--states=all' reports all states
 91 |   -u, --user=user_name(s)         comma separated list of users to view
 92 |       --name=job_name(s)          comma separated list of job names to view
 93 |   -v, --verbose                   verbosity level
 94 |   -V, --version                   output version information and exit
 95 |   -w, --nodelist=hostlist         list of nodes to view, default is
 96 | 				  all nodes
 97 | 
 98 | Help options:
 99 |   --help                          show this help message
100 |   --usage                         display a brief summary of squeue options
101 | ~~~
102 | 
103 | ### Identifying the QOS (Queues)
104 | 
105 | ### `sacctmgr`
106 | 
107 | The `sacctmgr` (or Slurm Account Manager) command shows the current QOS (aka queues) on the HPC environment.
108 | 
109 | Examples:
110 | 
111 | sacctmgr
112 | 
113 | ~~~
114 | sacctmgr
115 | sacctmgr show qos
116 | sacctmgr --help
117 | ~~~
118 | 
119 | sacctmgr show qos
120 | 
121 | ~~~
122 | $ sacctmgr show qos
123 | ~~~
124 | 
125 | ~~~
126 | Name   Priority  GraceTime    Preempt PreemptMode                                    Flags UsageThres UsageFactor       GrpTRES   GrpTRESMins GrpTRESRunMin GrpJobs GrpSubmit     GrpWall       MaxTRES MaxTRESPerNode   MaxTRESMins     MaxWall     MaxTRESPU MaxJobsPU MaxSubmitPU     MaxTRESPA MaxJobsPA MaxSubmitPA       MinTRES
127 | ---------- ---------- ---------- ---------- ----------- ---------------------------------------- ---------- ----------- ------------- ------------- ------------- ------- --------- ----------- ------------- -------------- ------------- ----------- ------------- --------- ----------- ------------- --------- ----------- -------------
128 |       long          0   00:00:00                cluster                                                        1.000000                                                                                                                    14-00:00:00      cpu=3600
129 |      batch          0   00:00:00                cluster                                                        1.000000                                                                                                                     3-00:00:00      cpu=3600
130 | ~~~
131 | 
132 | sacctmgr  --help
133 | 
134 | ~~~
135 | $ sacctmgr --help
136 | ~~~
137 | 
138 | ~~~
139 | sacctmgr [<OPTION>] [<COMMAND>]
140 |     Valid <OPTION> values are:
141 |      -h or --help: equivalent to "help" command
142 |      -i or --immediate: commit changes immediately
143 |      -n or --noheader: no header will be added to the beginning of output
144 |      -o or --oneliner: equivalent to "oneliner" command
145 |      -p or --parsable: output will be '|' delimited with a '|' at the end
146 |      -P or --parsable2: output will be '|' delimited without a '|' at the end
147 |      -Q or --quiet: equivalent to "quiet" command
148 |      -r or --readonly: equivalent to "readonly" command
149 |      -s or --associations: equivalent to "associations" command
150 |      -v or --verbose: equivalent to "verbose" command
151 |      -V or --version: equivalent to "version" command
152 | 
153 |   <keyword> may be omitted from the execute line and sacctmgr will execute
154 |   in interactive mode. It will process commands as entered until explicitly
155 |   terminated.
156 | 
157 |   Valid <COMMAND> values are:
158 |      add <ENTITY> <SPECS>     add entity
159 |      archive <DUMP/LOAD> <SPECS>
160 |                               Archive past jobs and/or steps, or load them
161 |                               back into the databse.
162 |      associations             when using show/list will list the
163 |                               associations associated with the entity.
164 |      clear stats              clear server statistics
165 |      delete <ENTITY> <SPECS>  delete the specified entity(s)
166 |      dump <CLUSTER> [File=<FILENAME>]
167 |                               dump database information of the
168 |                               specified cluster to the flat file.
169 |                               Will default to clustername.cfg if no file
170 |                               is given.
171 |      exit                     terminate sacctmgr
172 |      help                     print this description of use.
173 |      list <ENTITY> [<SPECS>]  display info of identified entity, default
174 |                               is display all.
175 |      load <FILE> [<SPECS>]    read in the file to update the database
176 |                               with the file contents. <SPECS> here consist
177 |                               of 'cluster=', and 'clean'.  The 'cluster='
178 |                               will override the cluster name given in the
179 |                               file.  The 'clean' option will remove what is
180 |                               already in the system for this cluster and
181 |                               replace it with the file.  If the clean option
182 |                               is not given only new additions or
183 |                               modifications will be done, no deletions.
184 |      modify <ENTITY> <SPECS>  modify entity
185 |      oneliner                 report output one record per line.
186 |      parsable                 output will be | delimited with an ending '|'
187 |      parsable2                output will be | delimited without an ending '|'
188 |      quiet                    print no messages other than error messages.
189 |      quit                     terminate this command.
190 |      readonly                 makes it so no modification can happen.
191 |      reconfigure              reread the slurmdbd.conf on the DBD.
192 |      shutdown                 shutdown the server.
193 |      show                     same as list
194 |      verbose                  enable detailed logging.
195 |      version                  display tool version number.
196 |      !!                       Repeat the last command entered.
197 | 
198 |   <ENTITY> may be "account", "association", "cluster",
199 |                   "configuration", "coordinator", "event",
200 |                   "federation", "job", "problem", "qos",
201 |                   "resource", "reservation", "runawayjobs", "stats"
202 |                   "transaction", "tres", "user" or "wckey"
203 | 
204 |   <SPECS> are different for each command entity pair.
205 |        list account       - Clusters=, Descriptions=, Format=,
206 |                             Names=, Organizations=, Parents=, WithAssoc,
207 |                             WithDeleted, WithCoordinators, WithRawQOS,
208 |                             and WOPLimits
209 |        add account        - Clusters=, DefaultQOS=, Description=, Fairshare=,
210 |                             GrpTRESMins=, GrpTRES=, GrpJobs=, GrpMemory=,
211 |                             GrpNodes=, GrpSubmitJob=, GrpWall=, MaxTRESMins=,
212 |                             MaxTRES=, MaxJobs=, MaxNodes=, MaxSubmitJobs=,
213 |                             MaxWall=, Names=, Organization=, Parent=,
214 |                             and QosLevel=
215 |        modify account     - (set options) DefaultQOS=, Description=,
216 |                             Fairshare=, GrpTRESMins=, GrpTRESRunMins=,
217 |                             GrpTRES=, GrpJobs=, GrpMemory=, GrpNodes=,
218 |                             GrpSubmitJob=, GrpWall=, MaxTRESMins=, MaxTRES=,
219 |                             MaxJobs=, MaxNodes=, MaxSubmitJobs=, MaxWall=,
220 |                             Names=, Organization=, Parent=, and QosLevel=
221 |                             RawUsage= (with admin privileges only)
222 |                             (where options) Clusters=, DefaultQOS=,
223 |                             Descriptions=, Names=, Organizations=,
224 |                             Parent=, and QosLevel=
225 |        delete account     - Clusters=, DefaultQOS=, Descriptions=, Names=,
226 |                             Organizations=, and Parents=
227 | 
228 |        list associations  - Accounts=, Clusters=, Format=, ID=, OnlyDefaults,
229 |                             Partitions=, Parent=, Tree, Users=,
230 |                             WithSubAccounts, WithDeleted, WOLimits,
231 |                             WOPInfo, and WOPLimits
232 | 
233 |        list cluster       - Classification=, DefaultQOS=, Federation=,
234 |                             Flags=, Format=, Names=, RPC= WithFed and
235 |                             WOLimits
236 |        add cluster        - DefaultQOS=, Fairshare=, Federation=, FedState=,
237 |                             GrpTRES=, GrpJobs=, GrpMemory=, GrpNodes=,
238 |                             GrpSubmitJob=, MaxTRESMins=, MaxJobs=,
239 |                             MaxNodes=, MaxSubmitJobs=, MaxWall=, Name=,
240 |                             QosLevel= and Weight=
241 |        modify cluster     - (set options) DefaultQOS=, Fairshare=,
242 |                             Federation=, FedState=, GrpTRES=, GrpJobs=,
243 |                             GrpMemory=, GrpNodes=, GrpSubmitJob=,
244 |                             MaxTRESMins=, MaxJobs=, MaxNodes=,
245 |                             MaxSubmitJobs=, MaxWall=, QosLevel= and Weight=
246 |                             (where options) Classification=, Federation=,
247 |                             Flags=, and Names=
248 |        delete cluster     - Classification=, DefaultQOS=, Flags=, and Names=
249 | 
250 |        add coordinator    - Accounts=, and Names=
251 |        delete coordinator - Accounts=, and Names=
252 | 
253 |        list events        - All_Clusters, All_Time, Clusters=, End=, Events=,
254 |                             Format=, MaxCPUs=, MinCPUs=, Nodes=, Reason=,
255 |                             Start=, States=, and User=
256 | 
257 |        list federation    - Names=, Format= and Tree
258 |        add federation     - Flags=, Clusters= and Name=
259 |        modify federation  - (set options) Clusters= and Flags=
260 |                             (where options) Names=
261 |        delete federation  - Names=
262 | 
263 |        modify job         - (set options) DerivedExitCode=, Comment=
264 |                             (where options) JobID=, Cluster=
265 | 
266 |        list qos           - Descriptions=, Format=, Id=, Names=,
267 |                             PreemptMode=, and WithDeleted
268 |        add qos            - Description=, Flags=, GraceTime=, GrpJobs=,
269 |                             GrpSubmitJob=, GrpTRES=, GrpTRESMins=, GrpWall=,
270 |                             MaxJobs=, MaxSubmitJobsPerUser=, MaxTRESMins=,
271 |                             MaxTRESPerJob=, MaxTRESPerNode=, MaxTRESPerUser=,
272 |                             MaxWall=, Names=, Preempt=, PreemptMode=,
273 |                             Priority=, UsageFactor=, and UsageThreshold=
274 |        modify qos         - (set options) Description=, Flags=, GraceTime=,
275 |                             GrpJobs=, GrpSubmitJob=, GrpTRES=, GrpTRESMins=,
276 |                             GrpWall=,
277 |                             MaxJobs=, MaxSubmitJobsPerUser=, MaxTRESMins=,
278 |                             MaxTRESPerJob=, MaxTRESPerNode=, MaxTRESPerUser=,
279 |                             MaxWall=, Names=, Preempt=, PreemptMode=,
280 |                             Priority=, RawUsage= (admin only),
281 |                             UsageFactor=, and UsageThreshold=
282 |                             (where options) Descriptions=, ID=, Names=
283 |                             and PreemptMode=
284 |        delete qos         - Descriptions=, ID=, Names=, and PreemptMode=
285 | 
286 |        list resource      - Clusters=, Descriptions=, Flags=, Format=, Ids=,
287 |                             Names=, PercentAllowed=, ServerType=, Servers=,
288 |                             and WithClusters
289 |        add resource       - Clusters=, Count=, Descriptions=, Flags=,
290 |                             ServerType=, Names=, PercentAllowed=, Server=,
291 |                             and Type=
292 |        modify resource    - (set options) Count=, Flags=, Manager=,
293 |                             PercentAllowed=,
294 |                             (where options) Clusters=, Names=, Servers=,
295 |        delete resource    - Clusters=, Names=
296 | 
297 |        list reservation   - Clusters=, End=, ID=, Names=, Nodes=, Start=
298 | 
299 |        list runawayjobs   - Cluster=, Format=
300 | 
301 |        clear stats
302 |        list stats
303 | 
304 |        list transactions  - Accounts=, Action=, Actor=, Clusters=, End=,
305 |                             Format=, ID=, Start=, User=, and WithAssoc
306 | 
307 |        list tres          - ID=, Name=, Type=, WithDeleted
308 | 
309 |        list user          - AdminLevel=, DefaultAccount=,
310 |                             DefaultWCKey=, Format=, Names=,
311 |                             QosLevel=, WithAssoc, WithCoordinators,
312 |                             WithDeleted, WithRawQOS, and WOPLimits
313 |        add user           - Accounts=, AdminLevel=, Clusters=,
314 |                             DefaultAccount=, DefaultQOS=, DefaultWCKey=,
315 |                             Fairshare=, MaxTRESMins=, MaxTRES=,
316 |                             MaxJobs=, MaxNodes=, MaxSubmitJobs=, MaxWall=,
317 |                             Names=, Partitions=, and QosLevel=
318 |        modify user        - (set options) AdminLevel=, DefaultAccount=,
319 |                             DefaultQOS=, DefaultWCKey=, Fairshare=,
320 |                             MaxTRESMins=, MaxTRES=, MaxJobs=, MaxNodes=,
321 |                             MaxSubmitJobs=, MaxWall=, NewName=,
322 |                             and QosLevel=,
323 |                             RawUsage= (with admin privileges only)
324 |                             (where options) Accounts=, AdminLevel=,
325 |                             Clusters=, DefaultAccount=, Names=,
326 |                             Partitions=, and QosLevel=
327 |        delete user        - Accounts=, AdminLevel=, Clusters=,
328 |                             DefaultAccount=, DefaultWCKey=, and Names=
329 | 
330 |        list wckey         - Clusters=, End=, Format=, ID=, Names=,
331 |                             Start=, User=, and WithDeleted
332 | 
333 |        archive dump       - Directory=, Events, Jobs,
334 |                             PurgeEventAfter=, PurgeJobAfter=,
335 |                             PurgeStepAfter=, PurgeSuspendAfter=,
336 |                             Script=, Steps, and Suspend
337 | 
338 |        archive load       - File=, or Insert=
339 | 
340 |   Format options are different for listing each entity pair.
341 | 
342 |   One can get an number of characters by following the field option with
343 |   a %NUMBER option.  i.e. format=name%30 will print 30 chars of field name.
344 | 
345 |        Account            - Account, Coordinators, Description, Organization
346 | 
347 |        Association        - Account, Cluster, DefaultQOS, Fairshare,
348 |                             GrpTRESMins, GrpTRESRunMins, GrpTRES, GrpJobs,
349 |                             GrpMemory, GrpNodes, GrpSubmitJob, GrpWall,
350 |                             ID, LFT, MaxTRESMins, MaxTRES,
351 |                             MaxJobs, MaxNodes, MaxSubmitJobs, MaxWall, QOS,
352 |                             ParentID, ParentName, Partition, RGT,
353 |                             User, WithRawQOS
354 | 
355 |        Cluster            - Classification, Cluster, ClusterNodes,
356 |                             ControlHost, ControlPort, DefaultQOS,
357 |                             Fairshare, Flags, GrpTRESMins, GrpTRES GrpJobs,
358 |                             GrpMemory, GrpNodes, GrpSubmitJob, MaxTRESMins,
359 |                             MaxTRES, MaxJobs, MaxNodes, MaxSubmitJobs,
360 |                             MaxWall, NodeCount, PluginIDSelect, RPC, TRES
361 | 
362 |        Event              - Cluster, ClusterNodes, Duration, End,
363 |                             Event, EventRaw, NodeName, Reason, Start,
364 |                             State, StateRaw, TRES, User
365 | 
366 |        QOS                - Description, Flags, GraceTime, GrpJobs,
367 |                             GrpSubmitJob, GrpTRES, GrpTRESMins, GrpWall,
368 |                             MaxJobs, MaxSubmitJobsPerUser, MaxTRESMins,
369 |                             MaxTRESPerJob, MaxTRESPerNode, MaxTRESPerUser,
370 |                             MaxWall, Name, Preempt, PreemptMode,
371 |                             Priority, UsageFactor, UsageThreshold
372 | 
373 |        Resource           - Cluster, Count, CountAllowed, CountUsed,
374 |                             Description, Flags, Manager, Name,
375 |                             PercentAllowed, PercentUsed, Server, Type
376 | 
377 |        Reservation        - Assoc, Cluster, End, Flags, ID, Name,
378 |                             NodeNames, Start, TRES, UnusedWall
379 | 
380 |        RunAwayJobs        - Cluster, ID, Name, Partition, State,
381 |                             TimeStart, TimeEnd
382 | 
383 |        Transactions       - Action, Actor, Info, TimeStamp, Where
384 | 
385 |        TRES               - ID, Name, Type
386 | 
387 |        User               - AdminLevel, Coordinators, DefaultAccount,
388 |                             DefaultWCKey, User
389 | 
390 |        WCKey              - Cluster, ID, Name, User
391 | 
392 |        Account/User WithAssoc option will also honor
393 |        all of the options for Association.
394 | 
395 | 
396 |   All commands entitys, and options are case-insensitive.
397 | ~~~
398 | 
399 | 
400 | ### Interactive Jobs 
401 | 
402 | * Currently no restrictions on interactive job count
403 | * Jobs will run as long as the QOS allows
404 | 
405 | ### Batch Queue (`batch`)
406 | 
407 | * Up to 700 cores running per user, max walltime = 72 hours.
408 | * This queue allows users to submit a large number of medium-runtime, high-resource jobs to the Jax HPC clusters.  This queue does not allow interactivity.  Users should use interactivity available in the `interactive` queue. Jobs that do not specify a QOS will not be scheduled to run.
409 | 
410 | ### Long Queue (`long`)
411 | 
412 | * 504 hours max walltime.
413 | * This queue is where long-runtime jobs should be submitted.
414 | 
415 | ### High Memory (`high_mem`)
416 |   
417 | * Jobs requiring greater then 768gb of memory.
418 | * 72 hours max walltime.
419 | * Please request approval to access the 'high_mem' queue via helpdesk. A cluster job ID will be required to confirm the requested access. A limited set of resources dedicated to running jobs that have memory requirements greater than 7 GB.  
420 | 
421 | ### GPU (`gpu`)
422 | 
423 | * Interactivity allowed. Job arrays allowed.
424 | * 72 hours max walltime.
425 | * This queue is for jobs requiring NVIDIA GPU technology.  Jobs observed not utilizing GPU resources will be terminated without warning. A limited set of resources dedicated to running jobs that require GPU support.
426 | 
427 | ### Special / Reserved Queue (`special`)
428 | 
429 | * The `special` queue is reserved for special requests by researchers in conjunction with the IT department.  Users should consult with IT if their jobs require more than 3 weeks to complete or have other needs that the above queues do not address.  Output showing the walltime exceeded from your job in the `long` queue or other reasons the job can not function within the existing queue configurations will be required.
430 | 


--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
1 | # IntroToHPC
2 | HPC and queue usage class syllabus
3 | 


--------------------------------------------------------------------------------
/Topic Outline.md:
--------------------------------------------------------------------------------
 1 | 1. [Intro to bash](intro_to_bash.md)						.5 hrs
 2 | 2. Cluster Resource Overview				3 hrs
 3 |    - [Facilities Statement](facilities_statement.md)				.25 hrs
 4 |    - [Cluster Overview](cluster_overview.md)				.5 hrs
 5 |      - Login Nodes
 6 |      - Compute nodes
 7 |      - Specialized nodes
 8 |      - Queues
 9 |      - Dev nodes
10 |     - Storage
11 |       - Policies
12 |       - [Directory structure and utilization](directory_structure.md)
13 |          - Permissions by file/directory		.5 hrs
14 |          - /projects
15 |          - /data
16 |          - /fastscratch
17 |          - /gt_delivery
18 |          - /home
19 |     - [Applications](applications.md)					.5 hrs
20 |       - Userspace commands
21 |       - Application builds
22 |       - Central application tree
23 |     - [Cluster resource utilization](cluster_utilization.md)		.5 hrs
24 |       - Queue templates
25 | 3. [Questions/Comments](questions_comments.md)				.25 hrs
26 | 


--------------------------------------------------------------------------------
/applications.md:
--------------------------------------------------------------------------------
 1 | ### Software Usage on the Cluster
 2 | 
 3 | On Sumner, we have overhauled how researchers are able to install and easily access the software they need. Users are still able to download and install software into their userspace (`/projects` or `/home` directories). Additionally, low level development tools such as `gcc`, `openMPI`, `openJDK`, and basic libraires are still available via the `module` system. The new and exciting way we are able to provide software on Sumner, however, is through software containers. Singularity containers are a cutting edge way of creating your own custom software modules. Containers not only provide you with the software you need, but also a contained environment which ensure that your software runs the exact same way whether it's on your laptop, on the HPC resources, with a collaborator, or in the cloud. 
 4 | 
 5 | ---
 6 | ### Userspace Installation
 7 | 
 8 | The simplest method of installing software for use with Sumner is to install directly into your userspace. This includes any directories you have permissions to write to and modify. This includes your group's `/projects` directory or your personal `/home` directory. You also have write permissions on `/fastscratch` and `/tmp`, but these are not recommended due to their ephemeral nature. Most software installations will default to writing under a `/usr/` or `/lib/` path. Since these locations are shared by everyone, these "global" trees are not writable by the typical HPC user. Instead, many software installations will allow you to perform a "local" install in a directory of your choosing. Please consult the documentation for your software to find how to change the install path for your software. 
 9 | 
10 | ### Environment Modules
11 | 
12 | On Helix and Cadillac, common software was managed through the Environment Modules package. Now, only low-level software such as libraries, compilers, and programming language binaries are available through modules. To see what modules are available, run the command `module avail`. For more information about how to use the `module` command, see the [documentation](https://modules.readthedocs.io/en/latest/). 
13 | 
14 | ### Singularity Containers
15 | 
16 | The newest and most liberating change to software installations on HPC is the introduction of Singularity. Containerization allows you to install whatever software you want inside of a singular, stand-alone Singularity image file (`.sif`). This file contains your software, custom environment, and metadata all in one. 
17 | 
18 | To use Singularity, all you have to do is load the module by running `module load singularity` in a running job session on Sumner. Singularity can download new containers (`singularity pull`), run existing ones (`singularity run/exec/shell`), or upload new containers to registries (`singularity push`). For a full tutorial on Singularity, please see the educational materials posted on the [Cyberinfrastructure SharePoint](https://jacksonlaboratory.sharepoint.com/sites/ResearchIT/SitePages/Containerization-with-Singularity--101.aspx). 
19 | 
20 | In addition to the Singularity module, there is also additional infrastructure available to help you use Singularity to its full potential at JAX. We have made a container builder available to facilitate the building of your own custom containers without having to distribute `sudo` permissions for the `singularity build` command. The builder is itself a container which sends a specified recipe to a build server, then downloads the resulting `.sif` file. To use the builder, all you have to do is run `singularity run http://s3-far.jax.org/builder/builder`. Alternatively, you can download the container by running `singularity pull http://s3-far.jax.org/builder/builder`, or using other tools like `wget` or `curl`. The first method is suggested because it doesn't leave the possibility that you are using an oudated builder client. 
21 | 
22 | Additionally, we are a container registry soon which will allow users from every group at JAX to upload containers and share them with everyone in the lab. For more updates on this, please contact [matt.bradley@jax.org](mailto:matt.bradley@jax.org). 
23 | 


--------------------------------------------------------------------------------
/cluster_overview.md:
--------------------------------------------------------------------------------
 1 | # Abbreviations
 2 | 
 3 | 
 4 | * np = CPU core count
 5 | * gpu = Graphical Processing Unit
 6 | 
 7 | # Cluster Overview
 8 | 
 9 | ## Login Nodes
10 | 
11 | | Sumner | Winter |
12 | | ------------- | ---------------- |
13 | | login.sumner.jax.org | login.winter.jax.org |
14 | 
15 | ## Compute nodes
16 | 
17 | [Node List Breakdown](nodes.md)
18 | 
19 | ## Queues
20 | [Queues](Queues.md)
21 | 
22 | ## Storage policies
23 | 
24 | [Storage Detail](storage.md)
25 | 


--------------------------------------------------------------------------------
/cluster_utilization.md:
--------------------------------------------------------------------------------
  1 | # Cluster Utilization
  2 | 
  3 | ## HPC Live
  4 | 
  5 | [HPC Live](https://hpclive.jax.org/d/_yYwcFxZk/sumner-dashboard?orgId=2&refresh=5m)
  6 | 
  7 | ## Userspace Cluster Commands
  8 | 
  9 | ~~~
 10 | sacct
 11 | Section: Slurm Commands (1)
 12 | Updated: Slurm Commands
 13 | Index
 14 |  
 15 | 
 16 | NAME
 17 | sacct - displays accounting data for all jobs and job steps in the Slurm job accounting log or Slurm database
 18 |  
 19 | 
 20 | SYNOPSIS
 21 | sacct [OPTIONS...]
 22 |  
 23 | 
 24 | DESCRIPTION
 25 | Accounting information for jobs invoked with Slurm are either logged in the job accounting log file or saved to the Slurm database, as configured with the AccountingStorageType parameter.
 26 | 
 27 | The sacct command displays job accounting data stored in the job accounting log file or Slurm database in a variety of forms for your analysis. The sacct command displays information on jobs, job steps, status, and exitcodes by default. You can tailor the output with the use of the --format= option to specify the fields to be shown.
 28 | 
 29 | For the root user, the sacct command displays job accounting data for all users, although there are options to filter the output to report only the jobs from a specified user or group.
 30 | 
 31 | For the non-root user, the sacct command limits the display of job accounting data to jobs that were launched with their own user identifier (UID) by default. Data for other users can be displayed with the --allusers, --user, or --uid options.
 32 | 
 33 | Elapsed time fields are presented as [days-]hours:minutes:seconds[.microseconds]. Only 'CPU' fields will ever have microseconds.
 34 | 
 35 | The default input file is the file named in the AccountingStorageLoc parameter in slurm.conf.
 36 | 
 37 | Note:
 38 | If designated, the slurmdbd.conf option PrivateData may further restrict the accounting data visible to users which are not SlurmUser, root, or a user with AdminLevel=Admin. See the slurmdbd.conf man page for additional details on restricting access to accounting data.
 39 | Note:
 40 | If the AccountingStorageType is set to "accounting_storage/filetxt", space characters embedded within account names, job names, and step names will be replaced by underscores. If account names with embedded spaces are needed, it is recommended that a database type of accounting storage be configured.
 41 | Note:
 42 | The contents of Slurm's database are maintained in lower case. This may result in some sacct output differing from that of other Slurm commands.
 43 | Note:
 44 | Much of the data reported by sacct has been generated by the wait3() and getrusage() system calls. Some systems gather and report incomplete information for these calls; sacct reports values of 0 for this missing data. See your systems getrusage (3) man page for information about which data are actually available on your system.
 45 |  
 46 | OPTIONS
 47 | -a, --allusers
 48 | Displays all users jobs when run by user root or if PrivateData is not configured to jobs. Otherwise display the current user's jobs
 49 | 
 50 | -A account_list , --accounts=account_list
 51 | Displays jobs when a comma separated list of accounts are given as the argument.
 52 | 
 53 | -b, --brief
 54 | Displays a brief listing, which includes the following data:
 55 | jobid
 56 | status
 57 | exitcode
 58 | 
 59 | -c, --completion
 60 | Use job completion data instead of job accounting. The JobCompType parameter in the slurm.conf file must be defined to a non-none option. Does not support federated cluster information (local data only).
 61 | 
 62 | -C constraint_list, --constraints=constraint_list
 63 | Comma separated list to filter jobs based on what constraints/features the job requested. Multiple options will be treated as 'and' not 'or', so the job would need all constraints specified to be returned not one or the other.
 64 | 
 65 | --delimiter=characters
 66 | ASCII characters used to separate the fields when specifying the -p or -P options. The default delimiter is a '|'. This option is ignored if -p or -P options are not specified.
 67 | 
 68 | -D, --duplicates
 69 | If Slurm job ids are reset, some job numbers will probably appear more than once in the accounting log file but refer to different jobs. Such jobs can be distinguished by the "submit" time stamp in the data records.
 70 | When data for specific jobs are requested with the --jobs option, sacct returns the most recent job with that number. This behavior can be overridden by specifying --duplicates, in which case all records that match the selection criteria will be returned.
 71 | NOTE: Revoked federated sibling jobs are hidden unless the --duplicates option is specified.
 72 | 
 73 | -e, --helpformat
 74 | Print a list of fields that can be specified with the --format option.
 75 | Fields available:
 76 | 
 77 | Account             AdminComment        AllocCPUS           AllocGRES
 78 | AllocNodes          AllocTRES           AssocID             AveCPU
 79 | AveCPUFreq          AveDiskRead         AveDiskWrite        AvePages
 80 | AveRSS              AveVMSize           BlockID             Cluster
 81 | Comment             Constraints         ConsumedEnergy      ConsumedEnergyRaw
 82 | CPUTime             CPUTimeRAW          DBIndex             DerivedExitCode
 83 | Elapsed             ElapsedRaw          Eligible            End
 84 | ExitCode            Flags               GID                 Group
 85 | JobID               JobIDRaw            JobName             Layout
 86 | MaxDiskRead         MaxDiskReadNode     MaxDiskReadTask     MaxDiskWrite
 87 | MaxDiskWriteNode    MaxDiskWriteTask    MaxPages            MaxPagesNode
 88 | MaxPagesTask        MaxRSS              MaxRSSNode          MaxRSSTask
 89 | MaxVMSize           MaxVMSizeNode       MaxVMSizeTask       McsLabel
 90 | MinCPU              MinCPUNode          MinCPUTask          NCPUS
 91 | NNodes              NodeList            NTasks              Priority
 92 | Partition           QOS                 QOSRAW              Reason
 93 | ReqCPUFreq          ReqCPUFreqMin       ReqCPUFreqMax       ReqCPUFreqGov
 94 | ReqCPUS             ReqGRES             ReqMem              ReqNodes
 95 | ReqTRES             Reservation         ReservationId       Reserved
 96 | ResvCPU             ResvCPURAW          Start               State
 97 | Submit              Suspended           SystemCPU           SystemComment
 98 | Timelimit           TimelimitRaw        TotalCPU            TRESUsageInAve
 99 | TRESUsageInMax      TRESUsageInMaxNode  TRESUsageInMaxTask  TRESUsageInMin
100 | TRESUsageInMinNode  TRESUsageInMinTask  TRESUsageInTot      TRESUsageOutAve
101 | TRESUsageOutMax     TRESUsageOutMaxNode TRESUsageOutMaxTask TRESUsageOutMin
102 | TRESUsageOutMinNode TRESUsageOutMinTask TRESUsageOutTot     UID
103 | User                UserCPU             WCKey               WCKeyID
104 | WorkDir
105 | NOTE: When using with Ave[RSS|VM]Size or their values in TRESUsageIn[Ave|Tot]. They represent the average/total of the highest watermarks over all ranks in the step. When using sstat they represent the average/total at the moment the command was run.
106 | 
107 | NOTE: TRESUsage*Min* values represent the lowest highwater mark in the step.
108 | The section titled "Job Accounting Fields" describes these fields.
109 | 
110 | -E end_time, --endtime=end_time
111 | Select jobs in any state before the specified time. If states are given with the -s option return jobs in this state before this period. See the DEFAULT TIME WINDOW for more details.
112 | Valid time formats are...
113 | 
114 | HH:MM[:SS] [AM|PM]
115 | MMDD[YY] or MM/DD[/YY] or MM.DD[.YY]
116 | MM/DD[/YY]-HH:MM[:SS]
117 | YYYY-MM-DD[THH:MM[:SS]]
118 | 
119 | 
120 | --federation
121 | Show jobs from the federation if a member of one.
122 | 
123 | -f file, --file=file
124 | Causes the sacct command to read job accounting data from the named file instead of the current Slurm job accounting log file. Only applicable when running the filetxt plugin.
125 | 
126 | -F flag_list, --flags=flag_list
127 | Comma separated list to filter jobs based on what various ways the jobs were handled. Current flags are SchedSubmit, SchedMain, SchedBackfill. These particular options describe the scheduler that started the job.
128 | 
129 | -g gid_list, --gid=gid_list --group=group_list
130 | Displays the statistics only for the jobs started with the GID or the GROUP specified by the gid_list or the group_list operand, which is a comma-separated list. Space characters are not allowed. Default is no restrictions.
131 | 
132 | -h, --help
133 | Displays a general help message.
134 | 
135 | -i, --nnodes=N
136 | Return jobs which ran on this many nodes (N = min[-max])
137 | 
138 | -j job(.step) , --jobs=job(.step)
139 | Displays information about the specified job(.step) or list of job(.step)s.
140 | The job(.step) parameter is a comma-separated list of jobs. Space characters are not permitted in this list. NOTE: A step id of 'batch' will display the information about the batch step.
141 | By default sacct shows only jobs with Eligible time, but with this option the non-eligible will be also shown. NOTE: If --state is also specified, as non-eligible are not PD, then non-eligible jobs will not be displayed. See the DEFAULT TIME WINDOW for details about how this option changes the default -S and -E options.
142 | 
143 | -k, --timelimit-min
144 | Only send data about jobs with this timelimit. If used with timelimit_max this will be the minimum timelimit of the range. Default is no restriction.
145 | 
146 | -K, --timelimit-max
147 | Ignored by itself, but if timelimit_min is set this will be the maximum timelimit of the range. Default is no restriction.
148 | 
149 | --local
150 | Show only jobs local to this cluster. Ignore other clusters in this federation (if any). Overrides --federation.
151 | 
152 | -l, --long
153 | Equivalent to specifying:
154 | --format=jobid,jobname,partition,maxvmsize,maxvmsizenode,maxvmsizetask, avevmsize,maxrss,maxrssnode,maxrsstask,averss,maxpages,maxpagesnode, maxpagestask,avepages,mincpu,mincpunode,mincputask,avecpu,ntasks, alloccpus,elapsed,state,exitcode,maxdiskread,maxdiskreadnode,maxdiskreadtask, avediskread,maxdiskwrite,maxdiskwritenode,maxdiskwritetask,avediskwrite, allocgres,reqgres,avecpufreq,reqcpufreqmin,reqcpufreqmax,reqcpufreqgov, tresusageinmax,tresusageinmaxn,tresusageinmaxt,tresusageoutmax,tresusageoutmaxn, tresusageoutmaxt,tresusageoutave,tresusageinave
155 | 
156 | -L, --allclusters
157 | Display jobs ran on all clusters. By default, only jobs ran on the cluster from where sacct is called are displayed.
158 | 
159 | -M cluster_list, --clusters=cluster_list
160 | Displays the statistics only for the jobs started on the clusters specified by the cluster_list operand, which is a comma-separated list of clusters. Space characters are not allowed in the cluster_list. A value of 'all' will query to run on all clusters. The default is current cluster you are executing the sacct command on or all clusters in the federation when executed on a federated cluster. This option implicitly sets the --local option.
161 | 
162 | -n, --noheader
163 | No heading will be added to the output. The default action is to display a header.
164 | 
165 | --noconvert
166 | Don't convert units from their original type (e.g. 2048M won't be converted to 2G).
167 | 
168 | -N node_list, --nodelist=node_list
169 | Display jobs that ran on any of these node(s). node_list can be a ranged string.
170 | 
171 | --name=jobname_list
172 | Display jobs that have any of these name(s).
173 | 
174 | -o, --format
175 | Comma separated list of fields. (use "--helpformat" for a list of available fields).
176 | NOTE: When using the format option for listing various fields you can put a %NUMBER afterwards to specify how many characters should be printed.
177 | 
178 | e.g. format=name%30 will print 30 characters of field name right justified. A %-30 will print 30 characters left justified.
179 | 
180 | When set, the SACCT_FORMAT environment variable will override the default format. For example:
181 | 
182 | SACCT_FORMAT="jobid,user,account,cluster"
183 | 
184 | 
185 | -p, --parsable
186 | output will be '|' delimited with a '|' at the end
187 | 
188 | -P, --parsable2
189 | output will be '|' delimited without a '|' at the end
190 | 
191 | -q, --qos
192 | Only send data about jobs using these qos. Default is all.
193 | 
194 | -r, --partition
195 | Comma separated list of partitions to select jobs and job steps from. The default is all partitions.
196 | 
197 | 
198 | -R reason_list, --reason=reason_list
199 | Comma separated list to filter jobs based on what reason the job wasn't scheduled outside resources/priority.
200 | 
201 | -s state_list, --state=state_list
202 | Selects jobs based on their state during the time period given. Unless otherwise specified, the start and end time will be the current time when the --state option is specified and only currently running jobs can be displayed. A start and/or end time must be specified to view information about jobs not currently running. See the JOB STATE CODES section below for a list of state designators. Multiple state names may be specified using comma separators. Either the short or long form of the state name may be used (e.g. CA or CANCELLED) and the name is case insensitive (i.e. ca and CA both work).
203 | NOTE: Note for a job to be selected in the PENDING state it must have "EligibleTime" in the requested time interval or different from "Unknown". The "EligibleTime" is displayed by the "scontrol show job" command. For example jobs submitted with the "--hold" option will have "EligibleTime=Unknown" as they are pending indefinitely.
204 | 
205 | NOTE: When specifying states and no start time is given the default start time is 'now'. This is only when -j is not used. If -j is used the start time will default to 'Epoch'. In both cases if no end time is given it will default to 'now'. See the DEFAULT TIME WINDOW for more details.
206 | 
207 | 
208 | -S, --starttime
209 | Select jobs in any state after the specified time. Default is 00:00:00 of the current day, unless the '-s' or '-j' options are used. If the '-s' option is used, then the default is 'now'. If states are given with the '-s' option then only jobs in this state at this time will be returned. If the '-j' option is used, then the default time is Unix Epoch 0. See the DEFAULT TIME WINDOW for more details.
210 | Valid time formats are...
211 | 
212 | HH:MM[:SS] [AM|PM]
213 | MMDD[YY] or MM/DD[/YY] or MM.DD[.YY]
214 | MM/DD[/YY]-HH:MM[:SS]
215 | YYYY-MM-DD[THH:MM[:SS]]
216 | 
217 | 
218 | -T, --truncate
219 | Truncate time. So if a job started before --starttime the start time would be truncated to --starttime. The same for end time and --endtime.
220 | 
221 | -u uid_list, --uid=uid_list, --user=user_list
222 | Use this comma separated list of uids or user names to select jobs to display. By default, the running user's uid is used.
223 | 
224 | --units=[KMGTP]
225 | Display values in specified unit type. Takes precedence over --noconvert option.
226 | 
227 | --usage
228 | Display a command usage summary.
229 | 
230 | -v, --verbose
231 | Primarily for debugging purposes, report the state of various variables during processing.
232 | 
233 | -V, --version
234 | Print version.
235 | 
236 | --whole-hetjob=[yes|no]
237 | When querying heterogeneous jobs, Slurm by default retrieves the information about all the components of the job if the het_job_id (leader id) is selected. If a non-leader heterogeneous job component id is selected only that component is retrieved by default. This behavior can be changed by using this option. If set to 'yes' or no value is set, then information about all the components will be retrieved no matter which component is selected in the job filter. Otherwise, if set to 'no' then only the selected heterogeneous job components will be retrieved, even when selecting the leader.
238 | 
239 | -W wckey_list, --wckeys=wckey_list
240 | Displays the statistics only for the jobs started on the wckeys specified by the wckey_list operand, which is a comma-separated list of wckey names. Space characters are not allowed in the wckey_list. Default is all wckeys.
241 | 
242 | -x associd_list, --associations=assoc_list
243 | Displays the statistics only for the jobs running under the association ids specified by the assoc_list operand, which is a comma-separated list of association ids. Space characters are not allowed in the assoc_list. Default is all associations.
244 | 
245 | -X, --allocations
246 | Only show statistics relevant to the job allocation itself, not taking steps into consideration.
247 |  
248 | JOB ACCOUNTING FIELDS
249 | The following describes each job accounting field:
250 | ALL
251 | Print all fields listed below.
252 | 
253 | AllocCPUs
254 | Count of allocated CPUs. Equivalent to NCPUS.
255 | 
256 | AllocGRES
257 | Names and counts of generic resources allocated.
258 | 
259 | AllocNodes
260 | Number of nodes allocated to the job/step. 0 if the job is pending.
261 | 
262 | AllocTres
263 | Trackable resources. These are the resources allocated to the job/step after the job started running. For pending jobs this should be blank. For more details see AccountingStorageTRES in slurm.conf.
264 | 
265 | Account
266 | Account the job ran under.
267 | 
268 | AssocID
269 | Reference to the association of user, account and cluster.
270 | 
271 | AveCPU
272 | Average (system + user) CPU time of all tasks in job.
273 | 
274 | AveCPUFreq
275 | Average weighted CPU frequency of all tasks in job, in kHz.
276 | 
277 | AveDiskRead
278 | Average number of bytes read by all tasks in job.
279 | 
280 | AveDiskWrite
281 | Average number of bytes written by all tasks in job.
282 | 
283 | AvePages
284 | Average number of page faults of all tasks in job.
285 | 
286 | AveRSS
287 | Average resident set size of all tasks in job.
288 | 
289 | AveVMSize
290 | Average Virtual Memory size of all tasks in job.
291 | 
292 | Cluster
293 | Cluster name.
294 | 
295 | Comment
296 | The job's comment string when the AccountingStoreJobComment parameter in the slurm.conf file is set (or defaults) to YES. The Comment string can be modified by invoking sacctmgr modify job or the specialized sjobexitmod command.
297 | 
298 | ConsumedEnergy
299 | Total energy consumed by all tasks in job, in joules. Note: Only in case of exclusive job allocation this value reflects the jobs' real energy consumption.
300 | 
301 | CPUTime
302 | Time used (Elapsed time * CPU count) by a job or step in HH:MM:SS format.
303 | 
304 | CPUTimeRAW
305 | Time used (Elapsed time * CPU count) by a job or step in cpu-seconds.
306 | 
307 | DerivedExitCode
308 | The highest exit code returned by the job's job steps (srun invocations). Following the colon is the signal that caused the process to terminate if it was terminated by a signal. The DerivedExitCode can be modified by invoking sacctmgr modify job or the specialized sjobexitmod command.
309 | 
310 | Elapsed
311 | The jobs elapsed time.
312 | The format of this fields output is as follows:
313 | [DD-[HH:]]MM:SS
314 | as defined by the following:
315 | DD
316 | days
317 | hh
318 | hours
319 | mm
320 | minutes
321 | ss
322 | seconds
323 | 
324 | Eligible
325 | When the job became eligible to run in the same format as End.
326 | 
327 | End
328 | Termination time of the job. Format output is, YYYY-MM-DDTHH:MM:SS, unless changed through the SLURM_TIME_FORMAT environment variable.
329 | 
330 | ExitCode
331 | The exit code returned by the job script or salloc, typically as set by the exit() function. Following the colon is the signal that caused the process to terminate if it was terminated by a signal.
332 | 
333 | GID
334 | The group identifier of the user who ran the job.
335 | 
336 | Group
337 | The group name of the user who ran the job.
338 | 
339 | JobID
340 | The identification number of the job or job step.
341 | Regular jobs are in the form:
342 | JobID[.JobStep]
343 | Array jobs are in the form:
344 | ArrayJobID_ArrayTaskID
345 | Heterogeneous jobs are in the form:
346 | HetJobID+HetJobOffset.
347 | 
348 | JobIDRaw
349 | The identification number of the job or job step. Prints the JobID in the form JobID[.JobStep] for regular, heterogeneous and array jobs..
350 | 
351 | JobName
352 | The name of the job or job step. The slurm_accounting.log file is a space delimited file. Because of this if a space is used in the jobname an underscore is substituted for the space before the record is written to the accounting file. So when the jobname is displayed by sacct the jobname that had a space in it will now have an underscore in place of the space.
353 | 
354 | Layout
355 | What the layout of a step was when it was running. This can be used to give you an idea of which node ran which rank in your job.
356 | 
357 | MaxDiskRead
358 | Maximum number of bytes read by all tasks in job.
359 | 
360 | MaxDiskReadNode
361 | The node on which the maxdiskread occurred.
362 | 
363 | MaxDiskReadTask
364 | The task ID where the maxdiskread occurred.
365 | 
366 | MaxDiskWrite
367 | Maximum number of bytes written by all tasks in job.
368 | 
369 | MaxDiskWriteNode
370 | The node on which the maxdiskwrite occurred.
371 | 
372 | MaxDiskWriteTask
373 | The task ID where the maxdiskwrite occurred.
374 | 
375 | MaxPages
376 | Maximum number of page faults of all tasks in job.
377 | 
378 | MaxPagesNode
379 | The node on which the maxpages occurred.
380 | 
381 | MaxPagesTask
382 | The task ID where the maxpages occurred.
383 | 
384 | MaxRSS
385 | Maximum resident set size of all tasks in job.
386 | 
387 | MaxRSSNode
388 | The node on which the maxrss occurred.
389 | 
390 | MaxRSSTask
391 | The task ID where the maxrss occurred.
392 | 
393 | MaxVMSize
394 | Maximum Virtual Memory size of all tasks in job.
395 | 
396 | MaxVMSizeNode
397 | The node on which the maxvmsize occurred.
398 | 
399 | MaxVMSizeTask
400 | The task ID where the maxvmsize occurred.
401 | 
402 | MinCPU
403 | Minimum (system + user) CPU time of all tasks in job.
404 | 
405 | MinCPUNode
406 | The node on which the mincpu occurred.
407 | 
408 | MinCPUTask
409 | The task ID where the mincpu occurred.
410 | 
411 | NCPUS
412 | Total number of CPUs allocated to the job. Equivalent to AllocCPUS.
413 | 
414 | NodeList
415 | List of nodes in job/step.
416 | 
417 | NNodes
418 | Number of nodes in a job or step. If the job is running, or ran, this count will be the number allocated, else the number will be the number requested.
419 | 
420 | NTasks
421 | Total number of tasks in a job or step.
422 | 
423 | Priority
424 | Slurm priority.
425 | 
426 | Partition
427 | Identifies the partition on which the job ran.
428 | 
429 | QOS
430 | Name of Quality of Service.
431 | 
432 | QOSRAW
433 | Numeric id of Quality of Service.
434 | 
435 | ReqCPUFreq
436 | Requested CPU frequency for the step, in kHz. Note: This value applies only to a job step. No value is reported for the job.
437 | 
438 | ReqCPUS
439 | Number of requested CPUs.
440 | 
441 | ReqGRES
442 | Names and counts of generic resources requested.
443 | 
444 | ReqMem
445 | Minimum required memory for the job, in MB. A 'c' at the end of number represents Memory Per CPU, a 'n' represents Memory Per Node. Note: This value is only from the job allocation, not the step.
446 | 
447 | ReqNodes
448 | Requested minimum Node count for the job/step.
449 | 
450 | ReqTres
451 | Trackable resources. These are the minimum resource counts requested by the job/step at submission time. For more details see AccountingStorageTRES in slurm.conf.
452 | 
453 | Reservation
454 | Reservation Name.
455 | 
456 | ReservationId
457 | Reservation Id.
458 | 
459 | Reserved
460 | How much wall clock time was used as reserved time for this job. This is derived from how long a job was waiting from eligible time to when it actually started. Format is the same as Elapsed.
461 | 
462 | ResvCPU
463 | How many CPU seconds were used as reserved time for this job. Format is the same as Elapsed.
464 | 
465 | ResvCPURAW
466 | How many CPU seconds were used as reserved time for this job. Format is in processor seconds.
467 | 
468 | Start
469 | Initiation time of the job in the same format as End.
470 | 
471 | State
472 | Displays the job status, or state. See the JOB STATE CODES section below for a list of possible states.
473 | If more information is available on the job state than will fit into the current field width (for example, the uid that CANCELLED a job) the state will be followed by a "+". You can increase the size of the displayed state using the "%NUMBER" format modifier described earlier.
474 | 
475 | NOTE: The RUNNING state will return suspended jobs as well. In order to print suspended jobs you must request SUSPENDED at a different call from RUNNING.
476 | 
477 | NOTE: The RUNNING state will return any jobs completed (cancelled or otherwise) in the time period requested as the job was also RUNNING during that time. If you are only looking for jobs that finished, please choose the appropriate state(s) without the RUNNING state.
478 | 
479 | 
480 | Submit
481 | The time the job was submitted in the same format as End.
482 | NOTE: If a job is requeued, the submit time is reset. To obtain the original submit time it is necessary to use the -D or --duplicate option to display all duplicate entries for a job.
483 | 
484 | 
485 | Suspended
486 | The amount of time a job or job step was suspended. Format is the same as Elapsed.
487 | 
488 | SystemCPU
489 | The amount of system CPU time used by the job or job step. Format is the same as Elapsed.
490 | NOTE: SystemCPU provides a measure of the task's parent process and does not include CPU time of child processes.
491 | 
492 | 
493 | Timelimit
494 | What the timelimit was/is for the job. Format is the same as Elapsed.
495 | 
496 | TimelimitRaw
497 | What the timelimit was/is for the job. Format is in number of minutes.
498 | 
499 | TotalCPU
500 | The sum of the SystemCPU and UserCPU time used by the job or job step. The total CPU time of the job may exceed the job's elapsed time for jobs that include multiple job steps. Format is the same as Elapsed.
501 | NOTE: TotalCPU provides a measure of the task's parent process and does not include CPU time of child processes.
502 | 
503 | 
504 | TresUsageInAve
505 | Tres average usage in by all tasks in job. NOTE: If corresponding TresUsageInMaxTask is -1 the metric is node centric instead of task.
506 | 
507 | TresUsageInMax
508 | Tres maximum usage in by all tasks in job. NOTE: If corresponding TresUsageInMaxTask is -1 the metric is node centric instead of task.
509 | 
510 | TresUsageInMaxNode
511 | Node for which each maximum TRES usage out occurred.
512 | 
513 | TresUsageInMaxTask
514 | Task for which each maximum TRES usage out occurred.
515 | 
516 | TresUsageOutAve
517 | Tres average usage out by all tasks in job. NOTE: If corresponding TresUsageOutMaxTask is -1 the metric is node centric instead of task.
518 | 
519 | TresUsageOutMax
520 | Tres maximum usage out by all tasks in job. NOTE: If corresponding TresUsageOutMaxTask is -1 the metric is node centric instead of task.
521 | 
522 | TresUsageOutMaxNode
523 | Node for which each maximum TRES usage out occurred.
524 | 
525 | TresUsageOutMaxTask
526 | Task for which each maximum TRES usage out occurred.
527 | 
528 | UID
529 | The user identifier of the user who ran the job.
530 | 
531 | User
532 | The user name of the user who ran the job.
533 | 
534 | UserCPU
535 | The amount of user CPU time used by the job or job step. Format is the same as Elapsed.
536 | NOTE: UserCPU provides a measure of the task's parent process and does not include CPU time of child processes.
537 | 
538 | 
539 | WCKey
540 | Workload Characterization Key. Arbitrary string for grouping orthogonal accounts together.
541 | 
542 | WCKeyID
543 | Reference to the wckey.
544 |  
545 | JOB STATE CODES
546 | BF BOOT_FAIL
547 | Job terminated due to launch failure, typically due to a hardware failure (e.g. unable to boot the node or block and the job can not be requeued).
548 | CA CANCELLED
549 | Job was explicitly cancelled by the user or system administrator. The job may or may not have been initiated.
550 | CD COMPLETED
551 | Job has terminated all processes on all nodes with an exit code of zero.
552 | DL DEADLINE
553 | Job terminated on deadline.
554 | F FAILED
555 | Job terminated with non-zero exit code or other failure condition.
556 | NF NODE_FAIL
557 | Job terminated due to failure of one or more allocated nodes.
558 | OOM OUT_OF_MEMORY
559 | Job experienced out of memory error.
560 | PD PENDING
561 | Job is awaiting resource allocation.
562 | PR PREEMPTED
563 | Job terminated due to preemption.
564 | R RUNNING
565 | Job currently has an allocation.
566 | RQ REQUEUED
567 | Job was requeued.
568 | RS RESIZING
569 | Job is about to change size.
570 | RV REVOKED
571 | Sibling was removed from cluster due to other cluster starting the job.
572 | S SUSPENDED
573 | Job has an allocation, but execution has been suspended and CPUs have been released for other jobs.
574 | TO TIMEOUT
575 | Job terminated upon reaching its time limit.
576 |  
577 | DEFAULT TIME WINDOW
578 | The options --starttime and --endtime define the time window between which sacct is going to search. For historical and practical reasons their default values (i.e. the default time window) depends on other options: --jobs and --state.
579 | 
580 | Depending on if --jobs and/or --state are specified, the default values of --starttime and --endtime options are:
581 | 
582 | WITHOUT EITHER --jobs NOR --state specified:
583 | ---starttime
584 | Defaults to Midnight.
585 | ---endtime
586 | Defaults to Now.
587 | 
588 | WITH --jobs AND WITHOUT --state specified:
589 | ---starttime
590 | Defaults to Epoch 0.
591 | ---endtime
592 | Defaults to Now.
593 | 
594 | WITHOUT --jobs AND WITH --state specified:
595 | ---starttime
596 | Defaults to Now.
597 | ---endtime
598 | Defaults to --starttime and to Now if --starttime is not specified.
599 | 
600 | WITH BOTH --jobs AND --state specified:
601 | ---starttime
602 | Defaults to Epoch 0.
603 | ---endtime
604 | Defaults to --starttime or to Now if --starttime is not specified.
605 | NOTE: With -v/--verbose a message about the actual time window in use is shown.
606 | 
607 |  
608 | 
609 | PERFORMANCE
610 | Executing sacct sends a remote procedure call to slurmdbd. If enough calls from sacct or other Slurm client commands that send remote procedure calls to the slurmdbd daemon come in at once, it can result in a degradation of performance of the slurmdbd daemon, possibly resulting in a denial of service.
611 | 
612 | Do not run sacct or other Slurm client commands that send remote procedure calls to slurmdbd from loops in shell scripts or other programs. Ensure that programs limit calls to sacct to the minimum necessary for the information you are trying to gather.
613 | 
614 |  
615 | 
616 | ENVIRONMENT VARIABLES
617 | Some sacct options may be set via environment variables. These environment variables, along with their corresponding options, are listed below. (Note: Commandline options will always override these settings.)
618 | 
619 | SACCT_FEDERATION
620 | Same as --federation
621 | SACCT_LOCAL
622 | Same as --local
623 | SLURM_CONF
624 | The location of the Slurm configuration file.
625 | SLURM_TIME_FORMAT
626 | Specify the format used to report time stamps. A value of standard, the default value, generates output in the form "year-month-dateThour:minute:second". A value of relative returns only "hour:minute:second" if the current day. For other dates in the current year it prints the "hour:minute" preceded by "Tomorr" (tomorrow), "Ystday" (yesterday), the name of the day for the coming week (e.g. "Mon", "Tue", etc.), otherwise the date (e.g. "25 Apr"). For other years it returns a date month and year without a time (e.g. "6 Jun 2012"). All of the time stamps use a 24 hour format.
627 | A valid strftime() format can also be specified. For example, a value of "%a %T" will report the day of the week and a time stamp (e.g. "Mon 12:34:56").
628 | 
629 |  
630 | EXAMPLES
631 | This example illustrates the default invocation of the sacct command:
632 | # sacct
633 | Jobid      Jobname    Partition    Account AllocCPUS State     ExitCode
634 | ---------- ---------- ---------- ---------- ---------- ---------- --------
635 | 2          script01   srun       acct1               1 RUNNING           0
636 | 3          script02   srun       acct1               1 RUNNING           0
637 | 4          endscript  srun       acct1               1 RUNNING           0
638 | 4.0                   srun       acct1               1 COMPLETED         0
639 | 
640 | This example shows the same job accounting information with the brief option.
641 | 
642 | # sacct --brief
643 |      Jobid     State  ExitCode
644 | ---------- ---------- --------
645 | 2          RUNNING           0
646 | 3          RUNNING           0
647 | 4          RUNNING           0
648 | 4.0        COMPLETED         0
649 | # sacct --allocations
650 | Jobid      Jobname    Partition Account    AllocCPUS  State     ExitCode
651 | ---------- ---------- ---------- ---------- ------- ---------- --------
652 | 3          sja_init   andy       acct1            1 COMPLETED         0
653 | 4          sjaload    andy       acct1            2 COMPLETED         0
654 | 5          sja_scr1   andy       acct1            1 COMPLETED         0
655 | 6          sja_scr2   andy       acct1           18 COMPLETED         2
656 | 7          sja_scr3   andy       acct1           18 COMPLETED         0
657 | 8          sja_scr5   andy       acct1            2 COMPLETED         0
658 | 9          sja_scr7   andy       acct1           90 COMPLETED         1
659 | 10         endscript  andy       acct1          186 COMPLETED         0
660 | 
661 | This example demonstrates the ability to customize the output of the sacct command. The fields are displayed in the order designated on the command line.
662 | 
663 | # sacct --format=jobid,elapsed,ncpus,ntasks,state
664 |      Jobid    Elapsed      Ncpus   Ntasks     State
665 | ---------- ---------- ---------- -------- ----------
666 | 3            00:01:30          2        1 COMPLETED
667 | 3.0          00:01:30          2        1 COMPLETED
668 | 4            00:00:00          2        2 COMPLETED
669 | 4.0          00:00:01          2        2 COMPLETED
670 | 5            00:01:23          2        1 COMPLETED
671 | 5.0          00:01:31          2        1 COMPLETED
672 | 
673 | This example demonstrates the use of the -T (--truncate) option when used with -S (--starttime) and -E (--endtime). When the -T option is used, the start time of the job will be the specified -S value if the job was started before the specified time, otherwise the time will be the job's start time. The end time will be the specified -E option if the job ends after the specified time, otherwise it will be the jobs end time.
674 | 
675 | NOTE: If no -s (--state) option is given sacct will display jobs that ran during the specified time, otherwise it returns jobs that were in the state requested during that period of time.
676 | 
677 | Without -T (normal operation) sacct output would be like this.
678 | 
679 | # sacct -S2014-07-03-11:40 -E2014-07-03-12:00 -X -ojobid,start,end,state
680 |     JobID                 Start                  End        State
681 | --------- --------------------- -------------------- ------------
682 | 2         2014-07-03T11:33:16   2014-07-03T11:59:01   COMPLETED
683 | 3         2014-07-03T11:35:21   Unknown               RUNNING
684 | 4         2014-07-03T11:35:21   2014-07-03T11:45:21   COMPLETED
685 | 5         2014-07-03T11:41:01   Unknown               RUNNING
686 | By adding the -T option the job's start and end times are truncated to reflect only the time requested. If a job started after the start time requested or finished before the end time requested those times are not altered. The -T option is useful when determining exact run times during any given period.
687 | 
688 | # sacct -T -S2014-07-03-11:40 -E2014-07-03-12:00 -X -ojobid,jobname,user,start,end,state
689 |     JobID                 Start                  End        State
690 | --------- --------------------- -------------------- ------------
691 | 2         2014-07-03T11:40:00   2014-07-03T11:59:01   COMPLETED
692 | 3         2014-07-03T11:40:00   2014-07-03T12:00:00   RUNNING
693 | 4         2014-07-03T11:40:00   2014-07-03T11:45:21   COMPLETED
694 | 5         2014-07-03T11:41:01   2014-07-03T12:00:00   RUNNING
695 | 
696 | 
697 |  
698 | COPYING
699 | Copyright (C) 2005-2007 Copyright Hewlett-Packard Development Company L.P.
700 | Copyright (C) 2008-2010 Lawrence Livermore National Security. Produced at Lawrence Livermore National Laboratory (cf, DISCLAIMER).
701 | Copyright (C) 2010-2014 SchedMD LLC.
702 | This file is part of Slurm, a resource management program. For details, see <https://slurm.schedmd.com/>.
703 | 
704 | Slurm is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.
705 | 
706 | Slurm is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.  
707 | 
708 | FILES
709 | /etc/slurm.conf
710 | Entries to this file enable job accounting and designate the job accounting log file that collects system job accounting.
711 | /var/log/slurm_accounting.log
712 | The default job accounting log file. By default, this file is set to read and write permission for root only.
713 |  
714 | SEE ALSO
715 | sstat(1), ps (1), srun(1), squeue(1), getrusage (2), time (2)
716 | ~~~
717 | 
718 | ~~~
719 | sacct --allocations
720 |        JobID    JobName  Partition    Account  AllocCPUS      State ExitCode 
721 | ------------ ---------- ---------- ---------- ---------- ---------- -------- 
722 | 231722       trim_nf_s+    compute                     1    PENDING      0:0 
723 | 362037             bash    compute                     1    RUNNING      0:0 
724 | 421324             bash    compute                     1    RUNNING      0:0 
725 | 422019       jupyterla+    compute                     1    RUNNING      0:0 
726 | 424106       sumn7_ecd+    compute                     3 CANCELLED+      0:0 
727 | 518679             bash    compute                     1    RUNNING      0:0 
728 | 519577       helloworl+    compute                     4    PENDING      0:0 
729 | 519705       helloworl+    compute                     4    PENDING      0:0 
730 | 519712       helloworld    compute                     1    PENDING      0:0 
731 | 519714       helloworld    compute                     1    PENDING      0:0 
732 | 522212          overlap    compute                    10    TIMEOUT      0:0 
733 | 522277         RnnTrain        gpu                     2  COMPLETED      0:0 
734 | 524482       jupyter.n+    compute                     8    TIMEOUT      0:0 
735 | 524653       sleep_seg+        gpu                    12  COMPLETED      0:0 
736 | 524724       sleep_seg+        gpu                    12  COMPLETED      0:0 
737 | 524727       sleep_seg+        gpu                    12  COMPLETED      0:0 
738 | 524733       sleep_seg+        gpu                    12  COMPLETED      0:0 
739 | 524754       sleep_seg+        gpu                    12  COMPLETED      0:0 
740 | 524755       sleep_seg+        gpu                    12  COMPLETED      0:0 
741 | 524849             wrap    compute                     8     FAILED      1:0 
742 | 524859          overlap    compute                    10 CANCELLED+      0:0 
743 | 524884             bash        gpu                     4    RUNNING      0:0 
744 | 528407_219       scRATE    compute                     4  COMPLETED      0:0 
745 | 528407_1         scRATE    compute                     4  COMPLETED      0:0 
746 | 528407_2         scRATE    compute                     4    RUNNING      0:0 
747 | 528407_3         scRATE    compute                     4  COMPLETED      0:0 
748 | 528407_4         scRATE    compute                     4    RUNNING      0:0 
749 | 528407_5         scRATE    compute                     4  COMPLETED      0:0 
750 | 528407_6         scRATE    compute                     4  COMPLETED      0:0 
751 | 528407_7         scRATE    compute                     4  COMPLETED      0:0 
752 | 528407_8         scRATE    compute                     4  COMPLETED      0:0 
753 | 528407_9         scRATE    compute                     4  COMPLETED      0:0 
754 | 528407_10        scRATE    compute                     4  COMPLETED      0:0 
755 | 528407_11        scRATE    compute                     4  COMPLETED      0:0 
756 | 528407_12        scRATE    compute                     4    RUNNING      0:0 
757 | 528407_13        scRATE    compute                     4    RUNNING      0:0 
758 | 528407_14        scRATE    compute                     4    RUNNING      0:0 
759 | 528407_15        scRATE    compute                     4  COMPLETED      0:0 
760 | 528407_16        scRATE    compute                     4    RUNNING      0:0 
761 | 528407_17        scRATE    compute                     4    RUNNING      0:0 
762 | 528407_18        scRATE    compute                     4  COMPLETED      0:0 
763 | 528407_19        scRATE    compute                     4  COMPLETED      0:0 
764 | 528407_20        scRATE    compute                     4  COMPLETED      0:0 
765 | 528407_21        scRATE    compute                     4  COMPLETED      0:0 
766 | 528407_22        scRATE    compute                     4  COMPLETED      0:0 
767 | 528407_23        scRATE    compute                     4    RUNNING      0:0 
768 | 528407_24        scRATE    compute                     4  COMPLETED      0:0 
769 | 528407_25        scRATE    compute                     4    RUNNING      0:0 
770 | 528407_26        scRATE    compute                     4  COMPLETED      0:0 
771 | 528407_27        scRATE    compute                     4  COMPLETED      0:0 
772 | 528407_28        scRATE    compute                     4  COMPLETED      0:0 
773 | 528407_29        scRATE    compute                     4    RUNNING      0:0 
774 | 528407_30        scRATE    compute                     4  COMPLETED      0:0 
775 | 528407_31        scRATE    compute                     4  COMPLETED      0:0 
776 | 528407_32        scRATE    compute                     4    RUNNING      0:0 
777 | 528407_33        scRATE    compute                     4  COMPLETED      0:0 
778 | 528407_34        scRATE    compute                     4    RUNNING      0:0 
779 | 528407_35        scRATE    compute                     4    RUNNING      0:0
780 | 
781 | ---
782 | sacct -u <userid> -S 2020-02-25 format=User,JobID,Jobname,partition,state,time,start,end,elapsed,MaxRss,MaxVMSize,nnodes,ncpus,nodelist
783 | ~~~
784 | 


--------------------------------------------------------------------------------
/code_of_conduct.md:
--------------------------------------------------------------------------------
 1 | Code of Conduct
 2 | 
 3 | Software Carpentry and Data Carpentry are dedicated to providing a welcoming and supportive environment for all people, regardless of background or identity. However, we recognise that some groups in our community are subject to historical and ongoing discrimination, and may be vulnerable or disadvantaged. Membership in such a specific group can be on the basis of characteristics such as gender, sexual orientation, disability, physical appearance, body size, race, nationality, sex, colour, ethnic or social origin, pregnancy, citizenship, familial status, veteran status, genetic information, religion or belief, political or any other opinion, membership of a national minority, property, birth, age, or choice of text editor. We do not tolerate harassment of participants on the basis of these categories, or for any other reason.
 4 | 
 5 | Harassment is any form of behaviour intended to exclude, intimidate, or cause discomfort. Because we are a diverse community, we may have different ways of communicating and of understanding the intent behind actions. Therefore we have chosen to prohibit certain forms of behaviour in our community, regardless of intent. Prohibited harassing behaviour includes but is not limited to:
 6 | 
 7 | * written or verbal comments which have the effect of excluding people on the basis of membership of a specific group listed above
 8 | * causing someone to fear for their safety, such as through stalking, following, or intimidation
 9 | * the display of sexual or violent images
10 | * unwelcome sexual attention
11 | * nonconsensual or unwelcome physical contact
12 | * sustained disruption of talks, events or communications
13 | * incitement to violence, suicide, or self-harm
14 | * continuing to initiate interaction (including photography or recording) with someone after being asked to stop
15 | * publication of private communication without consent
16 | 
17 | Behaviour not explicitly mentioned above may still constitute harassment. The list above should not be taken as exhaustive but rather as a guide to make it easier to enrich all of us and the communities in which we participate. All Carpentry interactions should be professional regardless of location: harassment is prohibited whether it occurs on- or offline, and the same standards apply to both.
18 | 
19 | Enforcement of the Code of Conduct will be respectful and not include any harassing behaviors.
20 | 
21 | Thank you for helping make this a welcoming, friendly community for all.
22 | 
23 | This code of conduct is a modified version of that used by PyCon, which in turn is forked from a template written by the Ada Initiative and hosted on the Geek Feminism Wiki. Contributors to this document: Adam Obeng, Aleksandra Pawlik, Bill Mills, Carol Willing, Erin Becker, Hilmar Lapp, Kara Woo, Karin Lagesen, Pauline Barmby, Sheila Miguez, Simon Waldman, Tracy Teal.
24 | 


--------------------------------------------------------------------------------
/directory_structure.md:
--------------------------------------------------------------------------------
 1 | ### Directory Structure Best Practice
 2 | Time spent at the beginning of the project to define folder hierarchy and file naming conventions will make it much easier to keep things organized and findable, both throughout the project and after project completion. Adhering to well-thought out naming conventions:
 3 | 
 4 | * helps prevent accidental overwrites or deletion
 5 | * makes it easier to locate specific data files
 6 | * makes collaborating on the same files less confusing
 7 | 
 8 | ### File naming best practices
 9 | 
10 | Include a few pieces of descriptive information in the filename, in a standard order, to make it clear what the file contains. For example, filenames could include:
11 | 
12 | * experiment name or acronym
13 | * researcher initials
14 | * date data collected
15 | * type of data
16 | * conditions
17 | * file version
18 | * file extension for application-specific files
19 | 
20 | #### Consider sort order:
21 | 
22 | If it is useful for files to stay in chronological order, a good convention is to start file names with `YYYYMMDD` or `YYMMDD`.
23 | 
24 | If you are using a sequential numbering system, use leading zeros to maintain sort order, e.g. `007` will sort before `700`.
25 | 
26 | Do not use special (i.e. non-alphanumeric) characters in names such as:  
27 | ```" / \ : * ? ‘ < > [ ] [ ] { }  ( ) & $ ~ ! @ #  % ^   , '```
28 | 
29 | These could be interpreted by programs or operating systems in unexpected ways.
30 | 
31 | Do not use spaces in file or folder names, as some operating systems will not recognize them and you will need to enclose them in quotation marks to reference them in scripts and programs. Alternatives to spaces in filenames:
32 | 
33 | * Underscores, e.g. file_name.xxx
34 | * Dashes, e.g. file-name.xxx
35 | * No separation, e.g. filename.xxx
36 | * Camel case, where the first letter of each section of text is capitalized, e.g. FileName.xxx
37 | * Keep names short, no more than 25 characters.
38 | 
39 | ### File Versioning Best Practices
40 | 
41 | File versioning ensures that you always understand what version of a file you are working with, and what are the working and final versions of files. Recommended file versioning practices:
42 | 
43 | * Include a version number at the end of the file name such as v01. Change this version number each time the file is saved.
44 | * For the final version, substitute the word FINAL for the version number.
45 | * Take advantage of the versioning capabilities available in collaborative workspaces such as github OSF, Google Drive, and Box.
46 | * Track versions of computer code with versioning software such as Git, Subversion, or CVS.
47 | 
48 | ### Directory Structure Best Practices
49 | Directories can be organized in many different ways. Consider what makes sense for your project and research team, and how people new to the project might look for files.
50 | 
51 | Once you determine how you want your directories to be organized, it is a good idea to stub out an empty directory structure to hold future data, and to document the contents of each directory in a readme file.
52 | 
53 | Directory Best Practices
54 | 
55 | * Organize directories hierarchically, with broader topics at the top level of the hierarchy and more specific topics lower in the structure.
56 | * Group files of similar information together in a single directory.
57 | * Name directories after aspects of the project rather than after the names of individual researchers.
58 | * Once you have decided on a directory structure, follow it consistently and audit it periodically.
59 | * Separate ongoing and completed work.
60 | 
61 | ### Sources
62 | http://guides.lib.umich.edu/datamanagement/files
63 | 


--------------------------------------------------------------------------------
/facilities_statement.md:
--------------------------------------------------------------------------------
 1 | # Research IT Resources
 2 | 
 3 | *Last Updated: 26-Feb-2020*
 4 | 
 5 | The Information Technology (IT) platforms at JAX enable computationally intensive, high-throughput, data-rich research to be conducted with the goal of developing personalized genomic medicine methodologies and practices.   The IT team supports our infrastructure and technology platforms, and includes several dedicated Research IT personnel.  Key IT platforms are summarized below:
 6 | 
 7 | 
 8 | 
 9 | Working storage (Tier 0) on Cadillac is provided by a Data Direct Networks Gridscalar GS7k GPFS storage appliance with 232 TB usable storage capacity.  The primary storage (Tier 1) on Cadillac is provided 25 Dell EMC Isilon scale-out NAS nodes combined for a total of 2.4 PB raw storage capacity.  A new storage array (Tier 2) has been brought online as of Q1 2018 adding approximately 2 PB of new non-computable capacity.
10 | 
11 | ## HPC Resources for all JAX Research:
12 | 
13 | ## Sumner Cluster:
14 | 
15 | Sumner is the HPC cluster physically located at the Farmington, CT campus.
16 | 
17 | Sumner includes 100 Supermicro X11DPT-B Series servers with 70 Intel Xeon Gold 6150 computable cores at 2.7GHz and 768 GB RAM creating a 7,000-core high performance compute cluster.
18 | 
19 | Specialized resources associated with Sumner include 2 nodes with 142 Intel Xeon Gold 6150 cores at 2.7GHz and 3 TB RAM available for workloads with extremely large memory requirements.
20 | 
21 | ## Winter Cluster:
22 | 
23 | Winter includes 8 Supermicro X11DGQ Series servers with 46 Intel Xeon Gold 6136 at 3.00GHz and 192 GB RAM.  Each server includes 4 Nvidia Tesla V100 32 GB GPU nvme cards.  This translates into 249.6 TFLOPS of double precision floating-point, 502.4 TFLOPS of single precision and 4,000 Tensor TFLOPS of combined, peak performance.
24 | 
25 | ## Cluster Accessible Storage:
26 | 
27 | Working storage (Tier 0) on Sumner is provided by a Data Direct Networks Gridscalar GS7k GPFS storage appliance with 522 TB usable storage capacity.  The primary storage (Tier 1) on Sumner is provided 27 Dell EMC Isilon scale-out NAS nodes combined for a total of 2.7 PB raw storage capacity.  A new storage array (Tier 2) has been brought online as of Q1 2018 adding approximately 2 PB of new non-computable capacity.
28 | 
29 | ## Archival Storage:
30 | 
31 | Archival storage (Tier 3) is provided at both Bar Harbor, ME and Farmington, CT by 2 Quantum Artico StorNext appliances with 72 TB front-end disk capacity backed by a 4 PB tape library at each site.  Data storage at this tier is replicated across both geographic sites.
32 | 
33 | ## Applications Platform:
34 | 
35 | Our applications platform at both sites supports standard software necessary for investigators to process and analyze their data, as well as providing the entire research community with the basic building blocks for them to build their own custom workflows.  Database development and deployment is supported by MySQL and other database management systems.
36 | 
37 | ## Network Infrastructure:
38 | 
39 | Our network platform supports scientific and enterprise business systems, using 40Gb core switches in our server farm that delivers at least 1Gb to user devices.  The environment includes wired and wireless network service and a redundant voice over IP (VOIP) system, and is protected by application firewalls. The network infrastructure for the HPC environment is comprised of a dedicated 100Gb backbone with 50Gb to each server in the cluster, and 40Gb to each storage node.  Internet service is delivered by a commercial service provider that can scale beyond 10Gb as demand for data transfer increases.
40 | 
41 | ## Scientific Services Support:
42 | 
43 | Gilbert is the HPC cluster serving the Genome Technologies Scientific Service.
44 | 
45 | Gilbert includes 10 HP Proliant XL Series servers with 28 cores at 2.3GHz and 512 GB RAM combined into a 280-core high performance compute cluster.  The primary storage (Tier 1) on Gilbert is provided by 10 Dell EMC Isilon scale-out NAS nodes combined for a total of 641.3 TB raw storage capacity.
46 | 
47 | Exome is the HPC cluster serving the Clinical Genomics Scientific Service.
48 | 
49 | Exome includes 8 HP Proliant SL Series servers with 16 cores at 2.6GHz and 256 GB RAM combined into a 128-core high performance compute cluster.  The primary storage (Tier 1) on Exome is provided by 4 Dell EMC Isilon scale-out NAS nodes combined for a total of 165.5 TB raw storage capacity.
50 | 


--------------------------------------------------------------------------------
/intro_to_bash.md:
--------------------------------------------------------------------------------
 1 | ## Common Linux Commands
 2 | | Command | Description |
 3 | | ------- | ----------- |
 4 | | cat [filename] | Display file’s contents to the standard output device(usually your monitor). |
 5 | | cd /directorypath	| Change to directory. |
 6 | | chmod [options] mode filename	| Change a file’s permissions. |
 7 | | chown [options] filename	| Change who owns a file. |
 8 | | clear	| Clear a command line screen/window for a fresh start. |
 9 | | cp [options] source destination	| Copy files and directories.|
10 | | date [options]	| Display or set the system date and time. |
11 | | df [options]	| Display used and available disk space. |
12 | | du [options]	| Show how much space each file takes up. |
13 | | file [options] filename	| Determine what type of data is within a file. |
14 | | find [pathname] [expression]	| Search for files matching a provided pattern. |
15 | | grep [options] pattern [filesname]	| Search files or output for a particular pattern. |
16 | | kill [options] pid	| Stop a process. If the process refuses to stop, use kill -9 pid. |
17 | | less [options] [filename]	| View the contents of a file one page at a time. |
18 | | ln [options] source [destination]	| Create a shortcut. |
19 | | locate filename	| Search a copy of your filesystem for the specified filename. |
20 | | ls [options]	| List directory contents. |
21 | | man [command]	| Display the help information for the specified command. |
22 | | mkdir [options] directory	| Create a new directory. |
23 | | mv [options] source destination	| Rename or move file(s) or directories. |
24 | | ps [options]	| Display a snapshot of the currently running processes. |
25 | | pwd	| Display the pathname for the current directory. |
26 | | rm [options] directory	| Remove (delete) file(s) and/or directories. |
27 | | rmdir [options] directory	| Delete empty directories. |
28 | | ssh [options] user@machine	| Remotely log in to another Linux machine, over the network. Leave an ssh session by typing exit. |
29 | | tail [options] [filename]	| Display the last n lines of a file (the default is 10). |
30 | | tar [options] filename | Store and extract files from a tarfile (.tar) or tarball (.tar.gz or .tgz). |
31 | | top	| Displays the resources being used on your system. Press q to exit. |
32 | | touch filename	| Create an empty file with the specified name. |
33 | | who [options]	| Display who is logged on. |
34 | | whoami	| Display current user. |
35 | 
36 | 
37 | ## Other Resources 
38 | * [Beginner's Guide to the Bash Terminal](https://youtu.be/oxuRxtrO2Ag)
39 | * [Software Carpentry - The Unix Shell](https://swcarpentry.github.io/shell-novice/)
40 | * [ExplainShell.com - Fun website for CLI explanation](https://explainshell.com)
41 | 


--------------------------------------------------------------------------------
/nodes.md:
--------------------------------------------------------------------------------
  1 | # Abbreviations
  2 | 
  3 | * np = CPU core count
  4 | * gpu = Graphical Processing Unit
  5 | 
  6 | # Cluster Node Breakdown
  7 | 
  8 | ## Login Nodes
  9 | 
 10 | | Sumner | Winter |
 11 | | ------------- | ---------------- |
 12 | | login.sumner.jax.org | login.winter.jax.org |
 13 | 
 14 | ## Compute nodes
 15 | 
 16 | | Sumner Compute | Winter Compute |
 17 | | ------- | ----------- |
 18 | | sumner014 np=70 768GB | winter200 np=46 192GB 4 V100 gpu |
 19 | | sumner015 np=70 768GB | winter201 np=46 192GB 4 V100 gpu |
 20 | | sumner016 np=70 768GB | winter202 np=46 192GB 4 V100 gpu |
 21 | | sumner017 np=70 768GB | winter203 np=46 192GB 4 V100 gpu |
 22 | | sumner018 np=70 768GB | winter204 np=46 192GB 4 V100 gpu |
 23 | | sumner019 np=70 768GB | winter205 np=46 192GB 4 V100 gpu |
 24 | | sumner020 np=70 768GB | winter206 np=46 192GB 4 V100 gpu |
 25 | | sumner021 np=70 768GB | winter207 np=46 192GB 4 V100 gpu |
 26 | | sumner022 np=70 768GB |
 27 | | sumner023 np=70 768GB |
 28 | | sumner024 np=70 768GB |
 29 | | sumner025 np=70 768GB |
 30 | | sumner026 np=70 768GB |
 31 | | sumner027 np=70 768GB |
 32 | | sumner028 np=70 768GB |
 33 | | sumner029 np=70 768GB |
 34 | | sumner030 np=70 768GB |
 35 | | sumner031 np=70 768GB |
 36 | | sumner032 np=70 768GB |
 37 | | sumner033 np=70 768GB |
 38 | | sumner034 np=70 768GB |
 39 | | sumner035 np=70 768GB |
 40 | | sumner036 np=70 768GB |
 41 | | sumner037 np=70 768GB |
 42 | | sumner038 np=70 768GB |
 43 | | sumner039 np=70 768GB |
 44 | | sumner040 np=70 768GB |
 45 | | sumner041 np=70 768GB |
 46 | | sumner042 np=70 768GB |
 47 | | sumner043 np=70 768GB |
 48 | | sumner044 np=70 768GB |
 49 | | sumner045 np=70 768GB |
 50 | | sumner046 np=70 768GB |
 51 | | sumner047 np=70 768GB |
 52 | | sumner048 np=70 768GB |
 53 | | sumner049 np=70 768GB |
 54 | | sumner050 np=70 768GB |
 55 | | sumner051 np=70 768GB |
 56 | | sumner052 np=70 768GB |
 57 | | sumner053 np=70 768GB |
 58 | | sumner054 np=70 768GB |
 59 | | sumner055 np=70 768GB |
 60 | | sumner056 np=70 768GB |
 61 | | sumner057 np=70 768GB |
 62 | | sumner058 np=70 768GB |
 63 | | sumner059 np=70 768GB |
 64 | | sumner060 np=70 768GB |
 65 | | sumner061 np=70 768GB |
 66 | | sumner062 np=70 768GB |
 67 | | sumner063 np=70 768GB |
 68 | | sumner064 np=70 768GB |
 69 | | sumner065 np=70 768GB |
 70 | | sumner066 np=70 768GB |
 71 | | sumner067 np=70 768GB |
 72 | | sumner068 np=70 768GB |
 73 | | sumner069 np=70 768GB |
 74 | | sumner070 np=70 768GB |
 75 | | sumner071 np=70 768GB |
 76 | | sumner072 np=70 768GB |
 77 | | sumner073 np=70 768GB |
 78 | | sumner074 np=70 768GB |
 79 | | sumner075 np=70 768GB |
 80 | | sumner076 np=70 768GB |
 81 | | sumner077 np=70 768GB |
 82 | | sumner078 np=70 768GB |
 83 | | sumner079 np=70 768GB |
 84 | | sumner080 np=70 768GB |
 85 | | sumner081 np=70 768GB |
 86 | | sumner082 np=70 768GB |
 87 | | sumner083 np=70 768GB |
 88 | | sumner084 np=70 768GB |
 89 | | sumner085 np=70 768GB |
 90 | | sumner086 np=70 768GB |
 91 | | sumner087 np=70 768GB |
 92 | | sumner088 np=70 768GB |
 93 | | sumner089 np=70 768GB |
 94 | | sumner090 np=70 768GB |
 95 | | sumner091 np=70 768GB |
 96 | | sumner092 np=70 768GB |
 97 | | sumner093 np=70 768GB |
 98 | | sumner094 np=70 768GB |
 99 | | sumner095 np=70 768GB |
100 | | sumner096 np=70 768GB |
101 | | sumner097 np=70 768GB |
102 | | sumner098 np=70 768GB |
103 | | sumner099 np=70 768GB |
104 | | sumner100 np=70 768GB |
105 | | sumner101 np=70 768GB |
106 | | sumner102 np=70 768GB |
107 | | sumner103 np=70 768GB |
108 | | sumner104 np=70 768GB |
109 | | sumner105 np=70 768GB |
110 | | sumner106 np=70 768GB |
111 | | sumner107 np=70 768GB |
112 | | sumner108 np=70 768GB |
113 | | sumner109 np=70 768GB |
114 | | sumner110 np=70 768GB |
115 | | sumner112 np=70 768GB |
116 | | sumner113 np=70 768GB |
117 | 
118 | ## Specialized nodes
119 | 
120 | | Sumner high_mem |
121 | | ---------- |
122 | | sumner114 np=142 3096GB |
123 | | sumner115 np=142 3096GB |
124 | 


--------------------------------------------------------------------------------
/questions_comments.md:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/TheJacksonLaboratory/IntroToHPC/4fe78be04937ec3ab1ac7a58d56143cc4f4cd8bf/questions_comments.md


--------------------------------------------------------------------------------
/researchIT_team.md:
--------------------------------------------------------------------------------
 1 | # Advanced Cyberinfrastructure Team
 2 | 
 3 | * Brendan Arbuckle, MS - Direrector, Information and Research Technologies	
 4 | 	* Shane Sanders, PhD - Senior Manager - Advanced Cyberinfrastructure
 5 | 	* David McKenzie, BS - Cyberinfrastructure Architect - Storage, Networking
 6 | 	* Jason Macklin, BA - Cyberinfrastructure Architect - High Performance Computing
 7 | 	* Aaron McDivitt, BA - System Administrator - High Performance Computing
 8 | 	* Richard Yanicky, MS - Systems Analyst - High Performance Computing
 9 | 	* Kurt Showmaker, PhD - Systems Analyst - High Performance Computing
10 | 


--------------------------------------------------------------------------------
/storage.md:
--------------------------------------------------------------------------------
 1 | Ok, now that we are connecting to Sumner through SSH, we should take a minute to examine and discuss the common directory structure of the system.
 2 | 
 3 | `/home` is where the home directories for every user are kept, and are accessible by all cluster nodes.  When you log in, you shold be in your `/home/userid` directory.  Let's check that by using `ssh` to connect back to Sumner, and then using the `pwd` command we learned earlier.
 4 | 
 5 | ~~~
 6 | ssh ssander@login.sumner.jax.org
 7 | pwd
 8 | ~~~
 9 | 
10 | Home directories have a quota of 50GB each, meaning that the maximum storage available to a single user's `/home/userid` folder is 50GB.  
11 | 
12 | `/projects` is the primary active storage directory for users and lab groups.  It is accessible by all cluster nodes.  Each labgroup has a `/projects/PIname-lab` folder with a quota of 75TB.
13 | 
14 | `/fastscratch` is a temporary directory pinned to the fastest in-house storage we have available.  It is accessible by all cluster nodes.  It has a total capacity of 150TB, and is meant to serve as 'scratch paper' for computational analysis.  Users can specify that their jobs output to `/fastscratch`, and then copy their important files back into their `/home` or `/project` directories.  Bioinformatics software can generate a large number of temporary and intermediate files, that are often no longer needed after the analysis completes, and can accidently consume large quantities of storage.  Files in the `/fastscratch` directory that are 10 days old will be erased automatically by the current storage system policy, and IT reserves the right to erase any and all data on `/fastscratch` without notice.
15 | 
16 | `/gt_delivery` is the delivery directory for sequence data from JAX's Genome Technologies group.  It is only available via the Globus UI or command-line utility.  More information is available at the Research IT Sharepoint site (https://jacksonlaboratory.sharepoint.com/sites/ResearchIT/SitePages/Globus-Data-Transfers.aspx)
17 | 


--------------------------------------------------------------------------------