├── Logging.php ├── README.md ├── addldapuser ├── htaccess ├── ipynb-launch ├── ipython-hydra.php └── sudoers_ipython /Logging.php: -------------------------------------------------------------------------------- 1 | log_file = $path; 24 | } 25 | public function logfile() { 26 | return($this->log_file); 27 | } 28 | // write message to the log file 29 | public function lwrite($message) { 30 | // if file pointer doesn't exist, then open log file 31 | if (!is_resource($this->fp)) { 32 | $this->lopen(); 33 | } 34 | // define script name 35 | $script_name = pathinfo($_SERVER['PHP_SELF'], PATHINFO_FILENAME); 36 | // define current time and suppress E_WARNING if using the system TZ settings 37 | // (don't forget to set the INI setting date.timezone) 38 | $time = @date('[d/M/Y:H:i:s]'); 39 | // write current time, script name and message to the log file 40 | fwrite($this->fp, "$time ($script_name) $message" . PHP_EOL); 41 | } 42 | // close log file (it's always a good idea to close a file when you're done with it) 43 | public function lclose() { 44 | fclose($this->fp); 45 | } 46 | // open log file (private method) 47 | private function lopen() { 48 | // in case of Windows set default log file 49 | if (strtoupper(substr(PHP_OS, 0, 3)) === 'WIN') { 50 | $log_file_default = 'c:/php/logfile.txt'; 51 | } 52 | // set default log file for Linux and other systems 53 | else { 54 | $log_file_default = '/tmp/logfile.txt'; 55 | } 56 | // define log file from lfile method or use previously set default 57 | $lfile = $this->log_file ? $this->log_file : $log_file_default; 58 | // open log file for writing only and place file pointer at the end of the file 59 | // (if the file does not exist, try to create it) 60 | $this->fp = fopen($lfile, 'a') or exit("Can't open $lfile!"); 61 | } 62 | } 63 | 64 | ?> 65 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | ipython-hydra 2 | ============= 3 | 4 | Set of scripts to automatically launch ipython notebooks for each user who hits the page. 5 | 6 | To set up ipython.stanford.edu, we did a standard linux-apache web server install and installed the latest ipython (with easy install). We also installed some other python packages that we wanted, like the nipy suite. 7 | 8 | Additional configuration: 9 | 10 | sudo su 11 | addgroup --system ipython 12 | echo “www-data ALL=(%ipython) NOPASSWD: /usr/local/bin/ipynb-launch” >> /etc/sudoers.d/ipython 13 | echo “www-data ALL=(root) NOPASSWD: /usr/local/bin/addldapuser” >> /etc/sudoers.d/ipython 14 | chmod 0440 /etc/sudoers.d/ipython 15 | mkdir /var/log/ipython 16 | chown www-data.ipython /var/log/ipython 17 | chmod g+w /var/log/ipython 18 | 19 | The php script calls the two bash scripts, where all the hard work happens: 20 | * addldapuser will add a new user to the system, if they don't already exist. It must be run as root. The password is disabled because we use Stanford's kerberos authentication, so if users log in via ssh they can authenticate with their SUNet ID and password. The new user is also added to the ipython group. If the user exists in Stanford LDAP, then their UID will be assigned by the LDAP result. Otherwise, a local UID is used. 21 | 22 | * ipynb-launch will configure a new user's home directory (if it hasn't already been done) by setting up the ipython config file and checking out the tutorial notebooks from github. If the notebooks already exist, git pull is run to get any new items. However, we ensure that any existing notebooks are kept as the user last left them to ensure that we don't corrupt any of their edits. (NOTE: this part of the code sucks; I'm sure there is a more elgant way to do this, perhaps with some git-magic.) Finally, a new ipython notebook is launched, if needed. The port that it listens on and the auto-generated password are stored in files within the user's home directory (in ~/.ipython/lock and ~/.ipython/pass). If a server is already running for that user, and it seems to be listening on the correct port, then no action is taken. 23 | 24 | After these scripts are called, the port and password are read from the user's home directory and a little intermediate page is returned to the client. This page just contains a little javascript to do a POST to the ipython notebook server login page with the auto-generated password. If all works as planned, the user will never see this hidden page nor the ipython login page. They should be taken directly to their active notebook server. Unfortunately, sometimes we've noticed that the auto-login fails, and the user is confronted with the ipython log in page requesting a password that they don't know. Reloading the ipython.stanford.edu page to let it redo the auto-login usually fixes it. (The user will also see the ipython login page if they click the logout button in the notebook. Is there any way to hack around this?) 25 | 26 | To run the latest (development) branch of ipython, get it from github and then run "sudo pip install -e ." from the ipython directory. This installs the necessary packages and symlinks IPython into your system (in /usr/local/...) so that you can run the latest code. 27 | 28 | -------------------------------------------------------------------------------- /addldapuser: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # Add a user to the system, if they don't already exist. This script must be run as root. 4 | # If the user exists in Stanford LDAP, we'll user their Stanford-wide UID. Otherwise, 5 | # they get assigned a local UID that's outside the Stanford UID range. 6 | 7 | # Copyright 2012 by Gunnaer Schaefer (gsfr@stanford.edu) and Bob Dougherty (bobd@stanford.edu) 8 | 9 | if [ ! $1 ]; then 10 | echo "Usage: $0 SUNetID" 11 | exit 999 12 | fi 13 | 14 | if id $1 &> /dev/null ; then 15 | if [ -z "`groups $1 | grep ipython`" ]; then 16 | /usr/sbin/adduser $1 ipython 17 | else 18 | echo "User $1 already exists and is a member of the ipython group." 19 | fi 20 | exit 1 21 | else 22 | ldapinfo=$(ldapsearch -x -h ldap.stanford.edu uid=$1) 23 | uid_num=$(echo "$ldapinfo" | grep uidNumber); uid_num=${uid_num##*: } 24 | firstname=$(echo "$ldapinfo" | grep suDisplayNameFirst); firstname=${firstname##*: } 25 | lastname=$(echo "$ldapinfo" | grep suDisplayNameLast); lastname=${lastname##*: } 26 | if [ -z $uid_num ]; then 27 | echo "User $1 does not exist in LDAP. Assigning a local UID." 28 | uid_num=$((`(echo 69999; cut -d':' -f3 /etc/passwd) | sort -n | tail -1` + 1)) 29 | fi 30 | echo "Creating user $1 ($firstname $lastname, uid = $uid_num)..." 31 | /usr/sbin/adduser --disabled-password --uid $uid_num --gecos "$firstname $lastname" $1 32 | /usr/sbin/adduser $1 ipython 33 | fi 34 | -------------------------------------------------------------------------------- /htaccess: -------------------------------------------------------------------------------- 1 | AuthType WebAuth 2 | Require valid-user 3 | -------------------------------------------------------------------------------- /ipynb-launch: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # Configre a user's home directory (if necessary) and launch an ipythonl notebook server 4 | # for them (also only if necessary). This script *MUST* be run as the target user! 5 | # It's safe to run this script as often as you like. 6 | 7 | # Copyright 2012 by Bob Dougherty (bobd@stanford.edu) 8 | 9 | # When we're run with sudo -u [someuser], the $USER and $HOME env vars reflect the caller, 10 | # not [someuser]. But whoami always returns the correct thing. 11 | user=`whoami` 12 | global_logfile=$1 13 | if [ -z "$global_logfile" ]; then 14 | global_logfile="/tmp/${user}_ipython_server.log" 15 | fi 16 | echo "ipynb_launch for user $user..." >$global_logfile 17 | 18 | # Hard-link ipython to apache_ipython so that we can easily detect process that 19 | # that launch, and safely "killall" when needed. 20 | IPEXEC="apache_ipython" 21 | IPEXEC_PATH="/usr/local/bin/${IPEXEC}" 22 | USERDIR="/home/${user}" 23 | IP="${USERDIR}/.ipython" 24 | LOG="${IP}/log" 25 | CONF="${IP}/profile_nbserver/ipython_notebook_config.py" 26 | NBDIR="${USERDIR}/bayfmri/notebooks" 27 | DATDIR="/data/" 28 | # TODO: hard-code the paths in the conf file, since we have them here anyway. 29 | 30 | if [ ! -f ${IP}/security/ssl_${user}.pem ]; then 31 | mkdir -p -m 700 ${IP}/security 32 | openssl req -new -newkey rsa:2048 -days 3652 -nodes -x509 -subj "/C=US/ST=CA/L=Stanford/O=Stanford University/CN=ipython.stanford.edu" -keyout ${IP}/security/ssl_${user}.pem -out ${IP}/security/ssl_${user}.pem 33 | fi 34 | if [ ! -e "$CONF" ]; then 35 | echo " Configuring home directory for user $user..." >>$global_logfile 36 | mkdir -p ${IP}/profile_nbserver 37 | echo "c = get_config()" > ${CONF} 38 | echo "c.NotebookApp.ip = 'ipython.stanford.edu'" >> ${CONF} 39 | # Ensure the port is within the range of ports that we have opened up in the firewall (9000-9999). 40 | # From the ipython code docs, the port is selected with the following algorithm: 41 | # The first 5 ports will be sequential, and the remaining n-5 will be randomly 42 | # selected in the range [port-2*n, port+2*n]. (n=port_retires) 43 | # WTF were they smoking?!?! Anyway, it's fine. We can deal with it. 44 | echo "c.NotebookApp.port = 9500" >> ${CONF} 45 | echo "c.NotebookApp.port_retries = 249" >> ${CONF} 46 | echo "c.NotebookApp.enable_mathjax = True" >> ${CONF} 47 | echo "c.NotebookApp.open_browser = False" >> ${CONF} 48 | echo "c.NotebookApp.certfile = u'${IP}/security/ssl_${user}.pem'" >> ${CONF} 49 | echo "c.NotebookApp.ipython_dir = u'${IP}'" >> ${CONF} 50 | echo "from IPython.lib import passwd" >> ${CONF} 51 | echo "with open('${IP}/pass','r') as fp:" >> ${CONF} 52 | echo " p = fp.read().strip()" >> ${CONF} 53 | echo "c.NotebookApp.password = unicode(passwd(p))" >> ${CONF} 54 | echo "c.IPKernelApp.pylab = 'inline'" >> ${CONF} 55 | echo "c.NotebookManager.notebook_dir = u'${NBDIR}'" >> ${CONF} 56 | fi 57 | if [ ! -e "$NBDIR" ]; then 58 | echo " Cloning notebooks for user $user..." >>$global_logfile 59 | git clone https://github.com/jbpoline/bayfmri.git ${USERDIR}/bayfmri 2>>$global_logfile 60 | else 61 | echo " Fetching notebooks for user $user..." >>$global_logfile 62 | cd ${NBDIR}/.. 2>>$global_logfile 63 | mod_files=`git diff --name-only` 64 | # TODO: rename the modded files so the user can see both their mods and the new stuff. 65 | if [ ! -z "$mod_files" ]; then 66 | mkdir ${NBDIR}_CACHE 2>>$global_logfile 67 | mv -f $mod_files ${NBDIR}_CACHE/ 2>>$global_logfile 68 | fi 69 | git pull 2>>$global_logfile 70 | git checkout 2>>$global_logfile 71 | cd 2>>$global_logfile 72 | if [ ! -z "$mod_files" ]; then 73 | mv -f ${NBDIR}_CACHE/* ${NBDIR}/ 2>>$global_logfile 74 | rmdir ${NBDIR}_CACHE 2>>$global_logfile 75 | fi 76 | fi 77 | if [ ! -e "${NBDIR}/ds107/" ]; then 78 | echo " Creating data directory for user $user..." >>$global_logfile 79 | cp -as /data/ds107 ${NBDIR}/ 2>>$global_logfile 80 | 81 | fi 82 | #if [ ! -e "$NBDIR/principles_statistics.ipynb" ]; then 83 | # echo " Copying in extra notebooks for user $user..." >>$global_logfile 84 | # git clone https://github.com/fperez/nipy-notebooks.git /tmp/${user}_nipy-notebooks 85 | # cp /tmp/${user}_nipy-notebooks/*.ipynb ${NBDIR}/ 86 | #fi 87 | 88 | # Check for an existing lock file 89 | PORT= 90 | if [ -f ${IP}/lock ]; then 91 | echo " Found an existing lock file for user $user..." >>$global_logfile 92 | # Check to be sure the server really is running 93 | PORT=( $(<${IP}/lock) ) 94 | PROC=`ps -u ${user} | grep ${IPEXEC}` 95 | if [ -n "$PORT" -a -n "$PROC" ]; then 96 | # TODO: is there a more specific test for an active ipython kernel? 97 | # Maybe something like pinging the port? 98 | echo " Port (${PORT}) and process (${PROC}) seem valid, checking socket..." >>$global_logfile 99 | SOCK=`netstat -nan | grep ${PORT} | grep LISTEN` 100 | echo " netstat status of ${PORT}: ${SOCK})." >>$global_logfile 101 | fi 102 | if [ -z "$PORT" -o -z "$PROC" -o -z "$SOCK" ]; then 103 | # seems something isn't right. Probably a stale lock file. 104 | # We'll just clean up and let a new kernel get launched below. 105 | echo " Cleaning up stale lock file." >>$global_logfile 106 | rm ${IP}/lock 107 | PORT= 108 | fi 109 | fi 110 | 111 | if [ -z $PORT ]; then 112 | # No usable kernel running. Check for rogue kernels, and kill them. 113 | echo " Killing any existing python processes for user $user..." >>$global_logfile 114 | killall -u $user $IPEXEC &>>$global_logfile 115 | echo " Launching a new kernel for user $user..." >>$global_logfile 116 | # Now fire up a fresh kernel. First we have to set the desired password. 117 | RANDOM=`date +%N` 118 | passwd=$( echo "$RANDOM" | md5sum ) 119 | passwd=${passwd:2:14} 120 | echo $passwd > ${IP}/pass 121 | # Note: if you try to pass text back from a backgrounded process, php seems to hang. 122 | # There's probably a way around this, but I kind of like the lock-file approach. 123 | export IPYTHONDIR=${IP} 124 | #nohup $IPEXEC notebook --profile=nbserver &> $LOG & 125 | $IPEXEC_PATH notebook --profile=nbserver &> $LOG & 126 | # We need to sleep for a bit here to allow ipynb to launch and write it's port to the log. 127 | sleep 1 128 | PORT=`sed -En 's!.*https://.*:([0-9]+)/*!\1!p' $LOG` 129 | echo $PORT > ${IP}/lock 130 | echo " New kernel for user $user running on port $PORT (status=$?, pid=$!)." >>$global_logfile 131 | exit 0 132 | else 133 | echo " Kernel already running on port ${PORT}..." >>$global_logfile 134 | # Exit status 1 means we are reconnecting to an existing kernel 135 | exit 1 136 | fi 137 | -------------------------------------------------------------------------------- /ipython-hydra.php: -------------------------------------------------------------------------------- 1 | lfile('/var/log/ipython/ipython_server.log'); 7 | $user = getenv('REMOTE_USER'); 8 | 9 | # Create the user, if they don't already exist. 10 | # THis script must be run as root. Make sure apache has 11 | # sudo privledge to run this. E.g.,: 12 | # 13 | exec("sudo /usr/local/bin/addldapuser ".$user, $out, $stat); 14 | if($stat==0){ 15 | $log->lwrite('Created new user '.$user.'.'); 16 | }else{ 17 | $log->lwrite('User not created ('.implode(', ',$out).').'); 18 | } 19 | 20 | unset($out); 21 | # The log file is owned by apache, so we'll let ipynb-launch write to a temp log 22 | # and then copy that into our log. This will also help our log be a little more coherent 23 | # when multiple processes are writing to it. (Might also consider locking the log with flock) 24 | $tmplog = '/tmp/ipython_'.$user.'_'.getmypid().'.log'; 25 | exec("sudo -n -u $user /usr/local/bin/ipynb-launch ".$tmplog, $out, $stat); 26 | $log->lwrite(file_get_contents($tmplog)); 27 | # TODO: Check return status. If 0, then a new kernel was launched. If 1, then exiting kernel was used. 28 | $port = trim(file_get_contents("/home/$user/.ipython/lock")); 29 | # The password might not be what we requested (e.g., if an existing kernel was used). 30 | $passwd = trim(file_get_contents("/home/$user/.ipython/pass")); 31 | $url = 'https://ipython.stanford.edu:'.$port; 32 | 33 | $log->lclose(); 34 | echo "
\n"; 35 | echo ""; 36 | echo "\n"; 37 | echo "
\n"; 38 | echo "\n"; 39 | ?> 40 | -------------------------------------------------------------------------------- /sudoers_ipython: -------------------------------------------------------------------------------- 1 | www-data ALL=(%ipython) NOPASSWD: /usr/local/bin/ipynb-launch 2 | www-data ALL=(root) NOPASSWD: /usr/local/bin/addldapuser 3 | --------------------------------------------------------------------------------