├── .Rbuildignore ├── .gitignore ├── ChangeLog ├── DESCRIPTION ├── LICENSE ├── LICENSE.md ├── NAMESPACE ├── R ├── launch.R ├── launchr-package.r └── preload.r ├── README.md ├── inst └── demo │ ├── client_examples.r │ ├── tensor.r │ └── tensor_function_def.r ├── launchr.Rproj └── man ├── here_doc.Rd ├── launch.Rd ├── launchr-package.Rd ├── lnode_script.Rd ├── pbs_default.Rd └── ssh_args.Rd /.Rbuildignore: -------------------------------------------------------------------------------- 1 | ^.*\.Rproj$ 2 | ^\.Rproj\.user$ 3 | LICENSE.md 4 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | *~ 2 | *.swp 3 | .Rproj.user 4 | .Rhistory 5 | .DS_Store 6 | -------------------------------------------------------------------------------- /ChangeLog: -------------------------------------------------------------------------------- 1 | Release 0.2-0: 2 | * Reconfigured custom configuration as editing of a default script 3 | * Added more server start feedback 4 | * Improved documentation 5 | 6 | Release 0.1-1: 7 | * Added or-condo-login configuration 8 | * Fixed some minor issues 9 | 10 | Release 0.1-0: 11 | * Initial roll out with one configuration for Rhea 12 | -------------------------------------------------------------------------------- /DESCRIPTION: -------------------------------------------------------------------------------- 1 | Package: launchr 2 | Type: Package 3 | Title: Launch a distributed R server on a cluster from a remote R session 4 | Version: 0.1-1 5 | Description: Provides ease of use and configuration components for 6 | controlling a distributed set of R instances on a cluster from a 7 | remote R session (e.g. running on a laptop with RStudio). Other 8 | parallel approaches (multicore, multithreading, and GPU) are 9 | compatible with a launchr() distributed server session, which provides 10 | a means of specifying the number of R sessions per node, leaving cores 11 | available for these shared-memory applications. The intent is to 12 | provide an easy-to-use interactive SPMD-style develpment platform for 13 | distributed parallel programming in R. Leave BIG data on BIG platforms 14 | and control its analysis using R codes on your laptop. 15 | License: BSD 2-clause License + file LICENSE 16 | Depends: 17 | R (>= 3.2.0), 18 | remoter (>= 0.4-0) 19 | Imports: 20 | pbdRPC (>= 0.2-2) 21 | Enhances: pbdCS, pbdMPI, pbdDMAT, kazaam 22 | ByteCompile: yes 23 | Authors@R: c( 24 | person("George", "Ostrouchov", role=c("aut", "cre"), email="ostrouchovg@ornl.gov"), person("Drew", "Schmidt", role="aut"), person("Wei-Chen", "Chen", role="aut"), person("ORNL", role = "cph") 25 | ) 26 | URL: https://github.com/RBigData/launchr 27 | BugReports: https://github.com/RBigData/launchr/issues 28 | Maintainer: George Ostrouchov 29 | RoxygenNote: 6.1.0 30 | NeedsCompilation: no 31 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | YEAR: 2018 2 | COPYRIGHT HOLDER: Oak Ridge National Laboratory 3 | -------------------------------------------------------------------------------- /LICENSE.md: -------------------------------------------------------------------------------- 1 | Copyright 2018 Oak Ridge National Laboratory 2 | 3 | Redistribution and use in source and binary forms, with or without 4 | modification, are permitted provided that the following conditions are met: 5 | 6 | 1. Redistributions of source code must retain the above copyright notice, 7 | this list of conditions and the following disclaimer. 8 | 9 | 2. Redistributions in binary form must reproduce the above copyright notice, 10 | this list of conditions and the following disclaimer in the documentation 11 | and/or other materials provided with the distribution. 12 | 13 | THIS SOFTWARE IS PROVIDED BY COPYRIGHT HOLDER ``AS IS'' AND ANY EXPRESS OR 14 | IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF 15 | MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO 16 | EVENT SHALL COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, 17 | INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, 18 | BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, 19 | DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF 20 | LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE 21 | OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF 22 | ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 23 | -------------------------------------------------------------------------------- /NAMESPACE: -------------------------------------------------------------------------------- 1 | # Generated by roxygen2: do not edit by hand 2 | 3 | export(here_doc) 4 | export(kill_server) 5 | export(launch) 6 | export(pbs_default) 7 | import(pbdRPC) 8 | import(remoter) 9 | -------------------------------------------------------------------------------- /R/launch.R: -------------------------------------------------------------------------------- 1 | #' Launches a persistent pbdR server on a cluster 2 | #' @param server A string giving the hostname to run the pbdR server, 3 | #' resolvable on your local platform. 4 | #' @param user Username for server login. 5 | #' @param FUN A function that modifies the default ssh script. It is 6 | #' called with \code{FUN(script, ...)} so it has access to 7 | #' unmatched arguments in \code{launch()}. The default is the 8 | #' identity function. 9 | #' @param port The port to be used for communication with the 10 | #' server. Defaults to 55555. 11 | #' @param verbose If TRUE, print generated scripts before 12 | #' submission. A value > 1 will also prevent script submission. 13 | #' @param fn A list of file names that will be created on the 14 | #' server. Normally the defaults are fine. 15 | #' @param ... Arguments for script details. Typical parameters include 16 | #' \describe{ 17 | #' \item{nodes}{The number of nodes to allocate for the server.} 18 | #' \item{npernode}{The number of MPI ranks (R sessions) to start per node.} 19 | #' \item{walltime}{A string "hh:mm:ss" giving the maximum length of time in 20 | #' hours:minutes:seconds before server timeout shutdown.} 21 | #' \item{account}{Account used for your cluster allocation.} 22 | #' \item{modules}{A vector of strings giving the modules to load before 23 | #' starting the server.} 24 | #' \item{rwd}{A string giving a working directory on server.} 25 | #' } 26 | #' @return A shell script as a vector of strings. 27 | #' @export 28 | launch = function(server, 29 | user = Sys.getenv("USER"), 30 | FUN = function(script, ...) script, 31 | port = 55555, 32 | verbose = FALSE, 33 | fn = list( 34 | lnode_file = ".pbdR_server.sh", 35 | pbs_file = ".pbdR_server.pbs", 36 | head_node_file = ".pbdR_server_hnode", 37 | server_log = ".pbdR_server.o" 38 | ), 39 | ...) { 40 | 41 | ## files created on server in rwd directory 42 | 43 | ## generate default script 44 | preload = lnode_script(fn, port, ...) 45 | args = ssh_args(port) 46 | ## TODO try to infer script form server name 47 | ## server = strsplit(SERVER, "[.]")[1] 48 | 49 | ## now process any modifications to the default by the FUN filter 50 | ## does the function exist in user's environment? 51 | FUN = match.fun(FUN) 52 | preload = FUN(script = preload, ...) 53 | 54 | ## TODO find a place to store user-contributed functions. inst/examples?? 55 | ## file = system.file(paste0(deparse(substitute(FUN)), ".R"), 56 | ## "examples", package = "launchr") 57 | 58 | ## echo scripts if requested 59 | if(verbose) { 60 | cat("\nScript for login node:\n") 61 | print(preload) 62 | cat("\nLocal ssh parameters:\n") 63 | print(args) 64 | 65 | if(verbose > 1) { 66 | cat("\nPrinted scripts only. No launch\n") 67 | return(invisible(FALSE)) 68 | } 69 | } 70 | 71 | ## set the machine parameters 72 | rserver = pbdRPC::machine(hostname = server, user = user, 73 | exec.type = "ssh", args = args) 74 | 75 | ## start server 76 | pbdRPC::start_cs(machine = rserver, cmd = "", 77 | preload = paste0(preload, collapse = "\n")) 78 | 79 | return(invisible(rserver)) 80 | } 81 | 82 | ## submission cancel script (a hack so far) 83 | ## may leave portion of a tunnel??! 84 | #' @export 85 | kill_server = function(machine, rwd) { 86 | command = paste0( 87 | "'echo `hostname`'", 88 | "; PID=$(showq -u $USER | grep $USER | grep -o -E '^[0-9]+')", 89 | "; qdel $PID; cd ", rwd, 90 | "; echo 'killed' >> .pbdR_server_hnode" 91 | ) 92 | pbdRPC::kill_rr(machine, command) 93 | } 94 | -------------------------------------------------------------------------------- /R/launchr-package.r: -------------------------------------------------------------------------------- 1 | #' launchr 2 | #' 3 | #' TODO 4 | #' 5 | #' @references Project URL: \url{https://github.com/RBigData/launchr} 6 | #' @author George Ostrouchov 7 | #' 8 | #' @name launchr-package 9 | #' 10 | #' @import pbdRPC 11 | #' @import remoter 12 | #' 13 | #' @docType package 14 | #' @keywords package 15 | NULL 16 | -------------------------------------------------------------------------------- /R/preload.r: -------------------------------------------------------------------------------- 1 | 2 | #' Wraps a script with shell commands to write it to a file. 3 | #' Sometimes referred to as a "here document" 4 | #' @param script A vector of strings, usually each a shell command. 5 | #' @param file_name A string giving the file name to use. Careful not to 6 | #' conflict with an existing file name as it will be silently deleted and 7 | #' overwritten. 8 | #' @param eof A string to use for end-of-file. Must not occurr within the 9 | #' script and must be unique among potential nested uses of this function 10 | #' for a script. 11 | #' @return A vector of strings, each string a line of the script. 12 | #' @export 13 | here_doc = function(script, file_name, eof) { 14 | c(paste0("cat >> ", file_name, " << ", "'", eof, "'"), script, eof) 15 | } 16 | 17 | #' Creates a default pbs script as a vector of strings. Modification 18 | #' for platform-specific parameters should be done with a function via 19 | #' the FUN parameter of the \code{launch()} function rather than 20 | #' rewriting \code{pbs_default()}. 21 | #' @param nodes Number of nodes to use for the server 22 | #' @param npernode Number of R instances to run per node 23 | #' @param rwd Working directory on server. This is where script files 24 | #' are created. 25 | #' @param modules A vector of strings giving the modules to load 26 | #' before starting the server. 27 | #' @param walltime A string "hh:mm:ss" giving the maximum length of 28 | #' time in hours:minutes:seconds before server timeout 29 | #' shutdown. Note that while the server is running you may be 30 | #' incurring time charges to your account. 31 | #' @param account Account used for your cluster allocation 32 | #' @param warn_on_fork Set to \code{FALSE} to suppress a warning when 33 | #' forking a process (for example, when invoking 34 | #' \code{parallel::mclapply()}). 35 | #' @param fn A list providing names used for script files that are 36 | #' created in the server rwd directory. 37 | #' @return A shell script as a vector of strings. 38 | #' @export 39 | pbs_default = function(nodes = 1, npernode = 16, modules = "r", 40 | walltime = "01:00:00", account, 41 | warn_on_fork = TRUE, fn = NULL, rwd = "~/") { 42 | pbs = c( 43 | "#!/bin/bash", 44 | "#PBS -N pbdR_server", 45 | "#PBS -j oe", 46 | paste0("#PBS -o ", fn$server_log), 47 | paste0("#PBS -A ", account), 48 | paste0("#PBS -l walltime=", walltime), 49 | paste0("#PBS -l nodes=", nodes) 50 | ) 51 | 52 | mod_vec = vector("character", length(modules)) 53 | for(m in seq(along=modules)) 54 | mod_vec[m] = paste0("module load ", modules[m]) 55 | 56 | wof = ifelse(warn_on_fork, "", " --mca mpi_warn_on_fork 0 ") 57 | 58 | commands = c( 59 | "module list", 60 | paste0("cd ", rwd), 61 | paste0("hostname > ", fn$head_node_file), 62 | paste0("echo $PBS_JOBID >> ", fn$head_node_file), 63 | paste0("mpirun ", wof, " --map-by ppr:", npernode, 64 | ":node Rscript -e 'pbdCS::pbdserver()'") 65 | ) 66 | 67 | c(pbs, mod_vec, commands) 68 | } 69 | 70 | #' Constructs a shell script as a vector of strings. Its purpose is to run 71 | #' in a login node shell and submit a pbdR server batch job that 72 | #' runs for a specified time or until closed. It also opens an ssh tunnel 73 | #' from the login node to the running server head node. 74 | #' 75 | #' Modification for platform-specific parameters should be done with a 76 | #' function via the FUN parameter of the \code{launch()} function 77 | #' rather than rewriting \code{lnode_script()}. 78 | #' 79 | #' @param fn A list of file names that will be created on the 80 | #' server. Normally the defaults specified in \code{launch()} 81 | #' function are fine. 82 | #' @param port The port used by the server. 83 | #' @param rwd Remote working directory as a string. 84 | #' @param ... Unmatched arguments are passed down to \code{pbs_default()} 85 | #' @return A shell script as a vector of strings. 86 | lnode_script = function(fn, port, rwd = "~/", ...) { 87 | ## Returns a script as a vector of strings to run on a login node so that it 88 | ## starts a pbdR server and opens an ssh tunnel to its head node on 89 | ## the specified port 90 | 91 | ## set working directory and file names to be created there 92 | cd = paste0("cd ", rwd) # TODO where will this be matched? 93 | 94 | ## file cleanup command 95 | clean = paste("rm -f", fn$pbs_file, fn$head_node_file, fn$lnode_file, 96 | fn$server_log) 97 | 98 | ## commands to write pbs script to its file 99 | pbs_script = pbs_default(rwd = rwd, fn = fn, ...) 100 | make_pbs_file = here_doc(pbs_script, fn$pbs_file, "..PBS-EOF..") 101 | 102 | ## command to queue the script 103 | qsub = c( 104 | "echo 'Submitting pbdR server'", 105 | paste0("qsub ", fn$pbs_file) 106 | ) 107 | 108 | ## commands to wait for server start with progress dots 109 | wait_run = paste0("while [ ! -f ", fn$head_node_file, 110 | " ]; do sleep 1; echo -n '.'; done; echo '.' ") 111 | 112 | ## command to tunnel from login to head node 113 | tunnel = paste0("ssh -f -L ", port, ":localhost:", port, 114 | " -N \\$(cat ", fn$head_node_file, ")") 115 | 116 | ## commands to report login node and head node names 117 | tell = c( 118 | "echo 'server login node: ' \\$(hostname)", 119 | paste0("echo 'server head node: ' \\$(cat ", fn$head_node_file, ")"), 120 | "echo 'Server running ... ready for client connections'" 121 | ) 122 | 123 | lnode_script = c(cd, make_pbs_file, qsub, wait_run, tunnel, tell) 124 | 125 | ## commands to write login node script to its file 126 | make_lnode_file = here_doc(lnode_script, fn$lnode_file, "..LNODE-EOF..") 127 | 128 | ## command to run the login node script 129 | run_file = paste0("source ", fn$lnode_file) 130 | 131 | ## put it all together 132 | c(cd, clean, make_lnode_file, run_file) 133 | } 134 | 135 | 136 | #' Constructs a string of arguments for local ssh to forward a port 137 | #' @param port The port for server connection 138 | #' @return A string. 139 | ssh_args = function(port) { 140 | args = paste0(" -f -L ", port, ":localhost:", port) 141 | } 142 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # launchr 2 | 3 | * **Version:** 0.2-0 4 | * **License:** [BSD 2-Clause](http://opensource.org/licenses/BSD-2-Clause) 5 | * **Author:** George Ostrouchov 6 | * **Project home**: https://github.com/RBigData/launchr 7 | * **Bug reports**: https://github.com/RBigData/launchr/issues 8 | 9 | Provides ease of use and configuration components for controlling a 10 | distributed set of R instances on a cluster from a remote R session 11 | (e.g. running on a laptop with RStudio). The intent is to provide an 12 | easy-to-use interactive SPMD-style development platform for distributed 13 | parallel programming in R. Other parallel approaches (multicore, 14 | multithreading, and GPU) are compatible with a launchr() distributed 15 | server session, which provides a means of specifying the number of R 16 | sessions per node, potentially leaving cores available for further 17 | parallel shared-memory applications. 18 | 19 | Leave BIG data on BIG platforms and control its analysis using R codes 20 | on your laptop. 21 | 22 | A default script for launching on a PBS-managed cluster is constructed. A 23 | means of editing the script by an R function allows for custom 24 | modifications necessary for local configurations. An example 25 | modification is provided in the inst/demo directory. Please let us 26 | know about your configurations to help us provide an easier experience 27 | for everyone. 28 | 29 | For understanding how port forwarding works, see 30 | [stackexchange](https://unix.stackexchange.com/questions/115897/whats-ssh-port-forwarding-and-whats-the-difference-between-ssh-local-and-remot). 31 | 32 | When producing mass graphics in parallel on the remote server file 33 | system, use \code{grDevices::dev.off()} to bypass a modified 34 | \code{dev.off()} that intends to hijack graphics back to the client. 35 | 36 | ## Installation 37 | 38 | 43 | 44 | A development version is maintained on GitHub: 45 | 46 | ```r 47 | remotes::install_github("RBigData/launchr") 48 | ``` 49 | -------------------------------------------------------------------------------- /inst/demo/client_examples.r: -------------------------------------------------------------------------------- 1 | ## This script is intended for learning how to interact with a distributed 2 | ## server running multiple R sessions. 3 | ## 4 | ## Install with the following: 5 | ## devtools::install_github("RBigData/launchr") 6 | 7 | library(launchr) 8 | 9 | ## Custom function to replaces one line in default launch script by several lines 10 | cades = function(script, nodes, ...) { 11 | new_lines = c( 12 | ## ppn=32 due to policy to concentrate cores on nodes when < 32 13 | paste0("#PBS -l nodes=", nodes, ":ppn=32"), 14 | "#PBS -l qos=std", 15 | "#PBS -q batch", 16 | "#PBS -W group_list=cades-ccsd", 17 | "#PBS -m abe" 18 | ) 19 | ## new_lines replace one line in script 20 | line = which(startsWith(script, "#PBS -l nodes=")) 21 | c(script[1:(line - 1)], new_lines, script[(line + 1):length(script)]) 22 | } 23 | 24 | ## example of a launch using default scripts (set verbose <2 to launch) 25 | srv = launch(server = "rhea.ccs.ornl.gov", verbose = 2, 26 | nodes = 2, npernode = 16, modules = c("hdf5", "r"), account = "your_account", 27 | walltime = "01:00:00", rwd = "~/eof/hosvd_code", warn_on_fork = FALSE) 28 | 29 | ## example of a launch using custom modification of a default script 30 | ## (set verbose <2 to launch) 31 | srv = launch(server = "or-condo-login.ornl.gov", FUN = cades, verbose = 2, 32 | nodes = 2, npernode = 16, modules = c("R"), account = "ccsd", 33 | walltime = "01:00:00", rwd = "~/test", warn_on_fork = FALSE) 34 | 35 | ## server submitted in queue. You MUST wait for node report 36 | remoter::client() # submit ONLY when server head node reports 37 | comm.size() 38 | comm.rank() 39 | 40 | ## to see server session information and packages 41 | if(comm.rank() == 0) sessionInfo() 42 | if(comm.rank() == 0) loadedNamespaces() 43 | if(comm.rank() == 0) .libPaths() 44 | 45 | ## reduction across ranks retains dimensions 46 | x = c(1, 2) 47 | x 48 | allreduce(x) 49 | allreduce(x, op="max") 50 | x = matrix(1, nrow = 3, ncol = 4) 51 | x 52 | allreduce(x) 53 | x = array(1, dim = c(4, 3, 2)) 54 | x 55 | allreduce(x) 56 | 57 | ## remember that data on the server is different from the data on 58 | ## the client (your local R session) 59 | exit() 60 | x 61 | remoter::client() 62 | x 63 | 64 | x = matrix(comm.rank()*10, nrow = 3, ncol = 4) 65 | comm.print(x) # New concept: Who prints? 66 | comm.print(x, rank.print=19) # Can inspect data on other ranks 67 | 68 | exit(client.only=FALSE) # shuts down server and exits client 69 | 70 | -------------------------------------------------------------------------------- /inst/demo/tensor.r: -------------------------------------------------------------------------------- 1 | library(launchr) 2 | 3 | launch(nodes = 4, npernode = 16, server = "rhea.ccs.ornl.gov", 4 | modules = c("r", "hdf5"), account = "gen011", 5 | walltime = "01:00:00", rwd = "~/eof/hosvd_code") 6 | ## server submitted in queue. You MUST wait for it to start. 7 | remoter::client() # submit ONLY when server head node reports 8 | comm.size() 9 | 10 | ## 11 | ## Note: After an interactive development phase, the code below 12 | ## (without the client-server section above) would be run in batch. 13 | ## Plotting functions below will run in batch but are temporarily 14 | ## disabled for the client-server demo. 15 | ## 16 | suppressMessages(library(rhdf5)) # can read hdf5 in parallel chunks 17 | suppressMessages(library(pbdIO)) # some I/O helper functions 18 | suppressMessages(library(kazaam)) # work with skinny matrices 19 | suppressMessages(library(ggplot2)) # grammar of graphics plots 20 | cwd = "/lustre/atlas/world-shared/gen011/pbdR_demo/" 21 | window = 11 # number of time steps in window 22 | w_center = 400 # ref time step size (first time in window) 23 | maxplots = 24 # limit # plots 24 | out_dir = paste0("/lustre/atlas/world-shared/gen011/pbdR_demo/", 25 | "xgc-w", window, "-c", w_center, "/") 26 | 27 | source("tensor_function_def.r") 28 | 29 | myrank = comm.rank() 30 | commsize = comm.size() 31 | 32 | ## note file names and variable names specfic to demo data 33 | data_dir = "/lustre/atlas/world-shared/gen011/pbdR_demo/data/" 34 | file_mesh= paste0(data_dir, "xgc.mesh.h5") 35 | file_var = paste0(data_dir, "xgc.3d.#####.h5") 36 | var = "dpot" 37 | 38 | ## Crate output directories 39 | screedir = paste0(out_dir, "scree/") 40 | ref = sprintf("%0.5d", w_center) 41 | plotdir = paste0(out_dir, "plots", ref, "/") 42 | if(myrank == 0) { # only one rank should be creating a directory 43 | dir.create(out_dir, showWarnings = FALSE) 44 | dir.create(screedir, showWarnings = FALSE) 45 | dir.create(plotdir, showWarnings = FALSE) 46 | } 47 | barrier() # all ranks must wait for directory creation 48 | 49 | ## parallel read the xgc mesh file and allgather to ranks 50 | mesh = read_xgc_mesh(file_mesh)$rz 51 | 52 | ## Parallel read a window of steps and create tensor array as 53 | ## list with attributes. Mesh dimension is split across ranks. 54 | tens = read_xgc_window(file_var, var, w_center, window)$Data 55 | tdim = dim(tens) # tensor dimensions (1, 2, 3d) = (toro, time, mesh) 56 | 57 | ## u1 slices: want dimensions (1, 3d*2) - need tshaq 58 | u1tens = as.vector(tens) # (1, 2, 3d) 59 | dim(u1tens) = c(tdim[1], tdim[2]*tdim[3]) # (1, 2*3d) 60 | u1tens.s = tshaq(u1tens) # (1, 2*3d) tshaq 61 | u1svd = svd(u1tens.s) 62 | ## scree_plots(u1svd$d, paste0(ref, "u1"), myrank, screedir) 63 | 64 | ## u3 mesh: want dimensions (3d, 1*2) - need shaq of transpose 65 | u3tens = as.vector(tens) # (1, 2, 3d) 66 | dim(u3tens) = c(tdim[1]*tdim[2], tdim[3]) # dim (1*2, 3d) 67 | u3tens.s = shaq(t(u3tens)) # transposed so dim (3d, 1*2) shaq 68 | u3svd = svd(u3tens.s) 69 | ## scree_plots(u3svd$d, paste0(ref, "u3"), myrank) 70 | ## space_plots(mesh, u3svd$d, DATA(u3svd$u), maxplots, myrank, tag="u3") 71 | 72 | ## u2 time: want dimensions (2, 1*3d) - need tshaq 73 | ## Note that tens has dims (toro, time, mesh) and we want an unfolding 74 | ## with (time, toro*mesh). Respecting contiguous sections, we can 75 | ## use u3tens with rows indexed by (toro, time) -> (time, toro) 76 | rindex = rep(tdim[2]*(1:tdim[1] - 1), times=tdim[2]) + 77 | rep(1:tdim[2], each=tdim[1]) # reorders (1*2, 3d) to (2*1, 3d) 78 | u2tens = u3tens[rindex, ] # reordered to (2, 1, 3d) dim (2*1, 3d) 79 | dim(u2tens) = c(tdim[2], tdim[1]*tdim[3]) # dim (2, 1*3d) 80 | u2tens.s = tshaq(u2tens) # (2, 1*3d) 81 | u2svd = svd(u2tens.s) 82 | ## scree_plots(u2svd$d, paste0(ref, "u2"), myrank) 83 | 84 | ## Core tensor computation 85 | u1core1 = crossprod(u1svd$u, u1tens) # dim (1, 2*3d), all local op 86 | 87 | u3core1 = as.vector(u1core1) # (1, 2, 3d) 88 | dim(u3core1) = c(tdim[1]*tdim[2], tdim[3]) # dim (1*2, 3d) 89 | u2core1 = u3core1[rindex, ] # reordered to (2, 1, 3d) dim (2*1, 3d) 90 | dim(u2core1) = c(tdim[2], tdim[1]*tdim[3]) # dim (2, 1*3d) 91 | u2core21 = crossprod(u2svd$u, u2core1) # dim(2, 1*3d), all local 92 | 93 | u3core21 = as.vector(u2core21) # (2, 1, 3d) 94 | dim(u3core21) = c(tdim[2]*tdim[1], tdim[3]) # dim (2*1, 3d) 95 | u3core321 = crossprod(u3svd$u, shaq(t(u3core21))) # dim (3td, 2*1) 96 | ## Note that 3td = 2*1 as 3 > 2*1 leads to 3 - 2*1 zero eigenvalues 97 | ## so that leading dimension of u3core321 is 2*1 instead of 3. 98 | ## u3core ends up a local matrix that is replicated. The shaq 99 | ## crossprod collapses the long dimension. 100 | 101 | ## Now apply u2svd to whiten over time (it's a local operation!) 102 | ## removes time covariance structure iid implications and systematic bias. 103 | w2u2tens = crossprod(diag(1/u2svd$d) %*% u2svd$u, u2tens) # still (2*1, 3d) 104 | 105 | ## next reorder whitened to make u3 and redo u3 svd 106 | w2u3tens = as.vector(w2u2tens) # (2, 1, 3d) 107 | dim(w2u3tens) = c(tdim[2]*tdim[1], tdim[3]) # dim (2*1, 3d) 108 | w2u3tens.s = shaq(t(w2u3tens)) # transposed so dim (3d, 2*1) 109 | w2u3svd = svd(w2u3tens.s) 110 | ## scree_plots(w2u3svd$d, paste0(ref, "w2u3"), myrank) 111 | ## space_plots(mesh, w2u3svd$d, DATA(w2u3svd$u), maxplots, myrank, tag="w2u3") 112 | 113 | ## take u2-whitened and apply u1 whitening. Having systematic toro 114 | ## distances may have iid implications and systematic bias. 115 | w2u1tens = w2u3tens[order(rindex), ] # reverses (2*1, 3d) back to (1*2, 3d) 116 | dim(w2u1tens) = c(tdim[1], tdim[2]*tdim[3]) # dim (1, 2*3d) 117 | w1w2u1tens = crossprod(diag(1/u1svd$d) %*% u1svd$u, w2u1tens) 118 | 119 | ## next reoder w1w2u1 into u3 and redo u3 svd 120 | w1w2u3tens = as.vector(w1w2u1tens) # (1, 2, 3d) 121 | dim(w1w2u3tens) = c(tdim[1]*tdim[2], tdim[3]) # dim (1*2, 3d) 122 | w1w2u3tens.s = shaq(t(w1w2u3tens)) # transposed so dim (3d, 1*2) 123 | w1w2u3svd = svd(w1w2u3tens.s) 124 | ## scree_plots(w1w2u3svd$d, paste0(ref, "w1w2u3"), myrank) 125 | ## space_plots(mesh, w1w2u3svd$d, DATA(w1w2u3svd$u), maxplots, myrank, tag="w1w2u3") 126 | 127 | ## For batch, replace below by finalize() 128 | exit(client.only = FALSE) 129 | -------------------------------------------------------------------------------- /inst/demo/tensor_function_def.r: -------------------------------------------------------------------------------- 1 | read_xgc_mesh = function(file) { 2 | ## read mesh in parallel and allgather a full copy to everyone 3 | ## this allows multiple plots in parallel 4 | if(endsWith(file, ".bp")) { 5 | mesh = bp.read(file, c("n_n", "n_t")) 6 | iopair_t = comm.chunk(mesh$n_t, form = "iopair") 7 | iopair_n = comm.chunk(mesh$n_n, form = "iopair") 8 | buffer = 9 | bp.read(file, c("nd_connect_list", "rz"), 10 | start=list(c(iopair_t[1], 0), c(iopair_n[1], 0)), 11 | count=list(c(iopair_t[2], 3), c(iopair_n[2], 2))) 12 | 13 | nd_connect_list = do.call(c, allgather(buffer[[1]])) 14 | dim(nd_connect_list) = c(3, mesh$n_n) 15 | rz = do.call(c, allgather(unlist(buffer[[2]]))) 16 | dim(rz) = c(2, mesh$n_n) 17 | } else if(endsWith(file, ".h5")) { 18 | mesh = list(n_n=NULL, n_t=NULL) 19 | buffer = vector("list", 2) 20 | mesh$n_n = h5read(file, "n_n") 21 | 22 | iopair_n = comm.chunk(mesh$n_n, form = "iopair") 23 | ## rhdf5 needs to add 1 to C/Python written data! 24 | buffer = h5read(file, "coordinates/values", 25 | start=c(0 + 1, iopair_n[1] + 1), 26 | count=c(2, iopair_n[2])) 27 | rz = do.call(c, allgather(unlist(buffer))) 28 | dim(rz) = c(2, mesh$n_n) 29 | nd_connect_list = NULL 30 | } else { 31 | comm.print("Don't know file type") 32 | stop(11) 33 | } 34 | list(rz = t(rz), nd_connect_list = nd_connect_list) 35 | } 36 | 37 | read_xgc = function(file, var) { 38 | ## reads a 2-d array: a set of ploidal planes 39 | ## TODO make generic with a parameter on which dimension to split 40 | if(endsWith(file, ".bp")) { 41 | ## TODO not generic! needs to get dimensions from varname! 42 | par = bp.read(file, c("nnode", "nphi")) 43 | iopair = comm.chunk(par$nnode, form = "iopair") 44 | 45 | buffer = bp.read(file, var, start = c(iopair[1], 0), 46 | count = c(iopair[2], par$nphi))[[1]] 47 | } else if(endsWith(file, ".h5")) { 48 | ## hdf5 file 49 | gd = h5ls(file) # TODO There has to be a better way to get dims!! 50 | par = as.numeric(strsplit(gd[gd$name == var, ]$dim, "x")[[1]]) 51 | iopair = comm.chunk(par[2], form = "iopair") 52 | buffer = h5read(file, var, 53 | start=c(0 + 1, iopair[1] + 1), 54 | count=c(par[1], iopair[2])) 55 | } else { 56 | comm.print("Don't know file type") 57 | stop(11) 58 | } 59 | 60 | dim(buffer) = c(par[1], iopair[2]) 61 | buffer 62 | } 63 | 64 | read_xgc_window = function(file, var, time, window) { 65 | ## read time window 66 | ## TODO needs to be generic with indicators of what to do with which 67 | ## dimensions. 68 | 69 | rex = "#+" # get length of rex in file name to replace with time 70 | rl = attr(regexpr(rex, file), "match.length") 71 | 72 | start = time - window %/% 2 # center the window on time 73 | 74 | ## construct file name for each window element and read the file 75 | wdat = vector("list", window) 76 | for(w in 1:window) { 77 | file_t = sub(rex, sprintf(paste0("%0.", rl, "d"), start + w - 1), file) 78 | wdat[[w]] = read_xgc(file_t, var) 79 | } 80 | ## wdat is a list with each element a time step in the window 81 | ## column dimension is the mesh 82 | ## row dimension is the poloidal slice 83 | 84 | ## TODO convert this to S4 (with a tensor package?) 85 | tdim = c(dim(wdat[[1]])[1], window, dim(wdat[[1]])[2]) # dims of tensor 86 | tdist = c(FALSE, FALSE, TRUE) # which dimensions are distributed 87 | tdimname = c("toro", "time", "mesh") 88 | umat = do.call(rbind, wdat) # final unfolded distributed matrix 89 | dim(umat) = tdim # this should make it a 3d array or a tensor 90 | class(umat) = "array" 91 | list(Data=umat, tdist=tdist, tdimname=tdimname) 92 | } 93 | 94 | byrow_to_bycol_gather = function(dat) { 95 | rcols = comm.chunk(ncol(dat), form="vector", all.rank=TRUE) 96 | for(r in 1:comm.size()) { 97 | if(comm.rank() == r - 1L) { 98 | Vc = do.call(rbind, gather(dat[, rcols[[r]]], rank.dest = r - 1L)) 99 | } else { 100 | Vtemp = gather(dat[, rcols[[r]]], rank.dest = r - 1L) 101 | } 102 | } 103 | Vc 104 | } 105 | 106 | screePlot = function(d, file, which=1:length(d), cumvar=FALSE, 107 | log10=FALSE, percent=FALSE, ylims=NULL, 108 | width=5, height=4, sizeL=0.3, sizeP=0.4) { 109 | ## scree plot with options 110 | pdf(paste0(file, ".pdf"), width, height) 111 | df = data.frame(component = which, var = d[which], 112 | cumvar = cumsum(d[which])) 113 | if(percent) { 114 | df$var = 100*df$var/sum(d) 115 | df$cumvar = 100*df$cumvar/sum(d) 116 | } 117 | myplot = ggplot(df, aes(component, var)) + geom_line(size = sizeL) + 118 | geom_point(size = sizeP, color = "blue") 119 | if(cumvar) myplot = myplot + 120 | geom_line(aes(component, cumvar), size = sizeL) + 121 | geom_point(aes(component, cumvar), size = sizeP, 122 | color = "red") 123 | if(log10) myplot = myplot + scale_y_log10() 124 | if(!is.null(ylims[1])) myplot = myplot + expand_limits(y = ylims) 125 | ggplot2:::print.ggplot(myplot + theme_bw()) 126 | dev.off() 127 | } 128 | 129 | scree_plots = function(d, ttag, myrank, screedir) { 130 | ## need a ease of use concept for if-elses below? list with lapply? 131 | ## current version only works with 5+ ranks!! 132 | if(myrank == 0) { 133 | screePlot(d^2, paste0(screedir, "scree", ttag)) 134 | } else if(myrank == 1) { 135 | screePlot(d^2, paste0(screedir, "scree100Log", ttag), 1:100, log10=TRUE) 136 | } else if(myrank == 2) { 137 | screePlot(d^2, paste0(screedir, "screeLog", ttag), log10=TRUE) 138 | } else if(myrank == 3) { 139 | screePlot(d^2, paste0(screedir, "screeCumm", ttag), cumvar=TRUE) 140 | } else if(myrank == 4) { 141 | screePlot(d^2, paste0(screedir, "screeCummP", ttag), cumvar=TRUE, 142 | percent=TRUE) 143 | } 144 | } 145 | 146 | spacePlot = function(mesh, vals, file = "spacePlot", tagImage=FALSE) { 147 | ## plots vals on mesh to pdf file 148 | pdata = data.frame(x = mesh[, 1], y = mesh[, 2], val= vals) 149 | pdf(paste0(file, ".pdf"), 10, 8) 150 | myplot = ggplot(pdata, aes(x, y, color = val)) + geom_point(size=0.5) + 151 | scale_color_gradient2(low="blue", mid="green", high="red") + 152 | theme_void() 153 | if(tagImage) myplot = myplot + ggtitle(file) 154 | ggplot2:::print.ggplot(myplot) 155 | dev.off() 156 | } 157 | 158 | space_plots = function(mesh, d, Vr, maxplots, myrank, tag="") { 159 | ## Plot the poloidal slice coherent structures 160 | Vc = byrow_to_bycol_gather(Vr) 161 | nplots = min(maxplots, length(d)) 162 | myPCs = comm.chunk(nplots, form="vector", type="balance") 163 | my.d = d[myPCs] 164 | PCdigits = floor(log10(length(d)) + 1) 165 | ftag = paste0(plotdir, tag, "pc%0.", PCdigits, "d") 166 | for(i in seq_along(myPCs)) { 167 | spacePlot(mesh, Vc[, i]*my.d[i], sprintf(ftag, myPCs[i]), tagImage=TRUE) 168 | } 169 | } 170 | ## End function definition 171 | -------------------------------------------------------------------------------- /launchr.Rproj: -------------------------------------------------------------------------------- 1 | Version: 1.0 2 | 3 | RestoreWorkspace: Default 4 | SaveWorkspace: Default 5 | AlwaysSaveHistory: Default 6 | 7 | EnableCodeIndexing: Yes 8 | UseSpacesForTab: Yes 9 | NumSpacesForTab: 2 10 | Encoding: UTF-8 11 | 12 | RnwWeave: knitr 13 | LaTeX: pdfLaTeX 14 | 15 | BuildType: Package 16 | PackageUseDevtools: Yes 17 | PackageInstallArgs: --no-multiarch --with-keep.source 18 | PackageRoxygenize: rd,collate,namespace 19 | -------------------------------------------------------------------------------- /man/here_doc.Rd: -------------------------------------------------------------------------------- 1 | % Generated by roxygen2: do not edit by hand 2 | % Please edit documentation in R/preload.r 3 | \name{here_doc} 4 | \alias{here_doc} 5 | \title{Wraps a script with shell commands to write it to a file. 6 | Sometimes referred to as a "here document"} 7 | \usage{ 8 | here_doc(script, file_name, eof) 9 | } 10 | \arguments{ 11 | \item{script}{A vector of strings, usually each a shell command.} 12 | 13 | \item{file_name}{A string giving the file name to use. Careful not to 14 | conflict with an existing file name as it will be silently deleted and 15 | overwritten.} 16 | 17 | \item{eof}{A string to use for end-of-file. Must not occurr within the 18 | script and must be unique among potential nested uses of this function 19 | for a script.} 20 | } 21 | \value{ 22 | A vector of strings, each string a line of the script. 23 | } 24 | \description{ 25 | Wraps a script with shell commands to write it to a file. 26 | Sometimes referred to as a "here document" 27 | } 28 | -------------------------------------------------------------------------------- /man/launch.Rd: -------------------------------------------------------------------------------- 1 | % Generated by roxygen2: do not edit by hand 2 | % Please edit documentation in R/launch.R 3 | \name{launch} 4 | \alias{launch} 5 | \title{Launches a persistent pbdR server on a cluster} 6 | \usage{ 7 | launch(server, user = Sys.getenv("USER"), FUN = function(script, ...) 8 | script, port = 55555, verbose = FALSE, fn = list(lnode_file = 9 | ".pbdR_server.sh", pbs_file = ".pbdR_server.pbs", head_node_file = 10 | ".pbdR_server_hnode", server_log = ".pbdR_server.o"), ...) 11 | } 12 | \arguments{ 13 | \item{server}{A string giving the hostname to run the pbdR server, 14 | resolvable on your local platform.} 15 | 16 | \item{user}{Username for server login.} 17 | 18 | \item{FUN}{A function that modifies the default ssh script. It is 19 | called with \code{FUN(script, ...)} so it has access to 20 | unmatched arguments in \code{launch()}. The default is the 21 | identity function.} 22 | 23 | \item{port}{The port to be used for communication with the 24 | server. Defaults to 55555.} 25 | 26 | \item{verbose}{If TRUE, print generated scripts before 27 | submission. A value > 1 will also prevent script submission.} 28 | 29 | \item{fn}{A list of file names that will be created on the 30 | server. Normally the defaults are fine.} 31 | 32 | \item{...}{Arguments for script details. Typical parameters include 33 | \describe{ 34 | \item{nodes}{The number of nodes to allocate for the server.} 35 | \item{npernode}{The number of MPI ranks (R sessions) to start per node.} 36 | \item{walltime}{A string "hh:mm:ss" giving the maximum length of time in 37 | hours:minutes:seconds before server timeout shutdown.} 38 | \item{account}{Account used for your cluster allocation.} 39 | \item{modules}{A vector of strings giving the modules to load before 40 | starting the server.} 41 | \item{rwd}{A string giving a working directory on server.} 42 | }} 43 | } 44 | \value{ 45 | A shell script as a vector of strings. 46 | } 47 | \description{ 48 | Launches a persistent pbdR server on a cluster 49 | } 50 | -------------------------------------------------------------------------------- /man/launchr-package.Rd: -------------------------------------------------------------------------------- 1 | % Generated by roxygen2: do not edit by hand 2 | % Please edit documentation in R/launchr-package.r 3 | \docType{package} 4 | \name{launchr-package} 5 | \alias{launchr-package} 6 | \title{launchr} 7 | \description{ 8 | TODO 9 | } 10 | \references{ 11 | Project URL: \url{https://github.com/RBigData/launchr} 12 | } 13 | \author{ 14 | George Ostrouchov 15 | } 16 | \keyword{package} 17 | -------------------------------------------------------------------------------- /man/lnode_script.Rd: -------------------------------------------------------------------------------- 1 | % Generated by roxygen2: do not edit by hand 2 | % Please edit documentation in R/preload.r 3 | \name{lnode_script} 4 | \alias{lnode_script} 5 | \title{Constructs a shell script as a vector of strings. Its purpose is to run 6 | in a login node shell and submit a pbdR server batch job that 7 | runs for a specified time or until closed. It also opens an ssh tunnel 8 | from the login node to the running server head node.} 9 | \usage{ 10 | lnode_script(fn, port, rwd = "~/", ...) 11 | } 12 | \arguments{ 13 | \item{fn}{A list of file names that will be created on the 14 | server. Normally the defaults specified in \code{launch()} 15 | function are fine.} 16 | 17 | \item{port}{The port used by the server.} 18 | 19 | \item{rwd}{Remote working directory as a string.} 20 | 21 | \item{...}{Unmatched arguments are passed down to \code{pbs_default()}} 22 | } 23 | \value{ 24 | A shell script as a vector of strings. 25 | } 26 | \description{ 27 | Modification for platform-specific parameters should be done with a 28 | function via the FUN parameter of the \code{launch()} function 29 | rather than rewriting \code{lnode_script()}. 30 | } 31 | -------------------------------------------------------------------------------- /man/pbs_default.Rd: -------------------------------------------------------------------------------- 1 | % Generated by roxygen2: do not edit by hand 2 | % Please edit documentation in R/preload.r 3 | \name{pbs_default} 4 | \alias{pbs_default} 5 | \title{Creates a default pbs script as a vector of strings. Modification 6 | for platform-specific parameters should be done with a function via 7 | the FUN parameter of the \code{launch()} function rather than 8 | rewriting \code{pbs_default()}.} 9 | \usage{ 10 | pbs_default(nodes = 1, npernode = 16, modules = "r", 11 | walltime = "01:00:00", account, warn_on_fork = TRUE, fn = NULL, 12 | rwd = "~/") 13 | } 14 | \arguments{ 15 | \item{nodes}{Number of nodes to use for the server} 16 | 17 | \item{npernode}{Number of R instances to run per node} 18 | 19 | \item{modules}{A vector of strings giving the modules to load 20 | before starting the server.} 21 | 22 | \item{walltime}{A string "hh:mm:ss" giving the maximum length of 23 | time in hours:minutes:seconds before server timeout 24 | shutdown. Note that while the server is running you may be 25 | incurring time charges to your account.} 26 | 27 | \item{account}{Account used for your cluster allocation} 28 | 29 | \item{warn_on_fork}{Set to \code{FALSE} to suppress a warning when 30 | forking a process (for example, when invoking 31 | \code{parallel::mclapply()}).} 32 | 33 | \item{fn}{A list providing names used for script files that are 34 | created in the server rwd directory.} 35 | 36 | \item{rwd}{Working directory on server. This is where script files 37 | are created.} 38 | } 39 | \value{ 40 | A shell script as a vector of strings. 41 | } 42 | \description{ 43 | Creates a default pbs script as a vector of strings. Modification 44 | for platform-specific parameters should be done with a function via 45 | the FUN parameter of the \code{launch()} function rather than 46 | rewriting \code{pbs_default()}. 47 | } 48 | -------------------------------------------------------------------------------- /man/ssh_args.Rd: -------------------------------------------------------------------------------- 1 | % Generated by roxygen2: do not edit by hand 2 | % Please edit documentation in R/preload.r 3 | \name{ssh_args} 4 | \alias{ssh_args} 5 | \title{Constructs a string of arguments for local ssh to forward a port} 6 | \usage{ 7 | ssh_args(port) 8 | } 9 | \arguments{ 10 | \item{port}{The port for server connection} 11 | } 12 | \value{ 13 | A string. 14 | } 15 | \description{ 16 | Constructs a string of arguments for local ssh to forward a port 17 | } 18 | --------------------------------------------------------------------------------