├── .gitignore ├── tljh_extras.sh ├── zip_home_folders.sh ├── copy_to_users.sh ├── LICENSE ├── settings_tljh_julia_TEMPLATE.sh ├── julia_install.sh ├── tljh_install.sh └── README.org /.gitignore: -------------------------------------------------------------------------------- 1 | settings_linode_tljh_julia.sh 2 | settings_linode_tljh_julia* 3 | -------------------------------------------------------------------------------- /tljh_extras.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # extra apt installs 4 | apt install pandoc # for pdf exports 5 | 6 | # make shared folder 7 | mkdir -p /srv/scratch 8 | chown root:jupyterhub-users /srv/scratch 9 | chmod 777 /srv/scratch 10 | chmod g+s /srv/scratch 11 | ln -s /srv/scratch /etc/skel/scratch 12 | -------------------------------------------------------------------------------- /zip_home_folders.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # 3 | # This zips the home of each user and puts a jupyter-user.zip in their home-folder 4 | # 5 | # TODO: think about scratch 6 | 7 | cd /home 8 | export users=jupyter-* 9 | 10 | for user in $users 11 | do 12 | sudo -u $user zip -r /tmp/$user.zip $user/ 13 | sudo -u $user mv /tmp/$user.zip /home/$user/ 14 | done 15 | -------------------------------------------------------------------------------- /copy_to_users.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # 3 | # This copies (recursively) files/dirs to all users and to /etc/skel 4 | 5 | mkdir /tmp/transfer 6 | cp -r $1 /tmp/transfer 7 | chmod -R a+rx /tmp/transfer/$1 8 | 9 | cp -r $1 /etc/skel 10 | 11 | cd /home 12 | export users=jupyter-* 13 | 14 | for user in $users 15 | do 16 | sudo -u $user cp -r /tmp/transfer/$1 /home/$user/ 17 | done 18 | 19 | rm -r /tmp/transfer 20 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Copyright (c) 2020 Mauro Werder, VAW, ETH Zurich 2 | 3 | Permission is hereby granted, free of charge, to any person obtaining a copy 4 | of this software and associated documentation files (the "Software"), to deal 5 | in the Software without restriction, including without limitation the rights 6 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 7 | copies of the Software, and to permit persons to whom the Software is 8 | furnished to do so, subject to the following conditions: 9 | 10 | The above copyright notice and this permission notice shall be included in all 11 | copies or substantial portions of the Software. 12 | 13 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 14 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 15 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 16 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 17 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 18 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 19 | SOFTWARE. -------------------------------------------------------------------------------- /settings_tljh_julia_TEMPLATE.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # 3 | # These are the settings, which (may) need adjusting 4 | 5 | ## Seetings for Linux-server install 6 | 7 | # leave this empty if you don't have a domain 8 | export domain= 9 | 10 | # this should be set to something irrespective 11 | export hostname= 12 | 13 | export fqdn=$hostname.$domain 14 | export timezone="Europe/Zurich" 15 | 16 | ## HTTPS setup, one of: none, self-cert, existing-cert, letsencrypt 17 | ############# 18 | export https_setup=none 19 | 20 | # needs a value for option "letsencrypt", otherwise leave empty 21 | export email4letsencrypt= 22 | 23 | # Need a values for option "existing-cert,", otherwise leave empty 24 | # Point to existing key & certificate files. 25 | # Note, they will be copied to /opt/tljh/state. 26 | export ssl_key= 27 | export ssl_cert= 28 | 29 | ## For TLJH 30 | ############ 31 | export jupyteradmin=admin 32 | export jupyteradmin_user=jupyter-$jupyteradmin 33 | # how long a user's server stays alive without user interaction 34 | export tljh_timeout=3600 35 | # memory limit of each user 36 | export tljh_limits_memory=2G 37 | # CPU-core limit of each user 38 | export tljh_limits_cpu=1 39 | # set path to get python and julia of TLJH 40 | # https://tljh.jupyter.org/en/latest/howto/env/user-environment.html#accessing-user-environment-outside-jupyterhub 41 | export PATH=/opt/tljh/user/bin:${PATH} 42 | 43 | ## For Julia 44 | ############ 45 | 46 | # version of Julia to install. If set to the empty string, Julia will not be installed. 47 | export julia_version="1.10.2" 48 | 49 | # Packages to be installed system-wide, separate entries with ":" 50 | # (IJulia gets installed irrespective) 51 | export julia_packages= 52 | -------------------------------------------------------------------------------- /julia_install.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # 3 | # Install the specified Julia version and packages, handling the environment 4 | # as needed for TheLittlestJupyterHub 5 | 6 | if [ "$1" == "-h" ]; then 7 | echo "Usage: `basename $0` " 8 | echo "Example: `basename $0` 1.6.1 Plots:StatsBase" 9 | exit 0 10 | fi 11 | if [ "$#" == "0" ]; then 12 | echo "No Julia install as input arguments to julia_install.sh are empty" 13 | exit 0 14 | fi 15 | if [ "$#" -gt 2 ]; then 16 | echo "Error: julia_install.sh needs two arguments." 17 | echo "Usage: `basename $0` " 18 | echo "Example: `basename $0` 1.6.1 Plots:StatsBase" 19 | exit 1 20 | fi 21 | 22 | # parse arguments 23 | export julia_version=$1 24 | export julia_packages=$2 25 | export julia_version_short=$(grep -o '^[0-9]*\.[0-9]*' <<< $julia_version) 26 | 27 | ## Download and unpack Julia 28 | export julia_archive_name="julia-$julia_version-linux-x86_64.tar.gz" 29 | export julia_parent_dir="/opt/julia" 30 | export julia_dir="$julia_parent_dir/$julia_version" 31 | wget -O $julia_archive_name https://julialang-s3.julialang.org/bin/linux/x64/$julia_version_short/$julia_archive_name 32 | mkdir -p julia-$julia_version 33 | tar zxf $julia_archive_name julia-$julia_version 34 | mkdir -p $julia_parent_dir 35 | mv -f julia-$julia_version $julia_dir 36 | # create symlinks 37 | ln -sf $julia_dir/bin/julia /usr/local/bin/julia$julia_version 38 | ln -sf $julia_dir/bin/julia /usr/local/bin/julia$julia_version_short 39 | ln -sf $julia_dir/bin/julia /usr/local/bin/julia 40 | 41 | ## Install Julia packages 42 | # This is the tricky bit and requires a bit of juggling with the DEPOT_PATH 43 | # and different environments. 44 | 45 | # the packages are installed into this depot: 46 | export julia_global_depot=$(julia -e 'print(DEPOT_PATH[2])') 47 | # (if not using this default, DEPOT_PATH will need to reflect this) 48 | mkdir -p $julia_global_depot 49 | 50 | # The corresponding environment is (another one could be chosen): 51 | export julia_global_env_dir=$(julia -e 'using Pkg; print(Pkg.envdir(DEPOT_PATH[2]))') 52 | export julia_global_env=$julia_global_env_dir/v$julia_version_short 53 | mkdir -p $julia_global_env 54 | touch $julia_global_env/Project.toml 55 | # Note, this env needs to be made available to the user in startup.jl or by other means. 56 | # --> see below 57 | 58 | # Install IJulia 59 | julia --project=$julia_global_env -e 'deleteat!(DEPOT_PATH, [1,3]); using Pkg; Pkg.update(); Pkg.add("IJulia"); Pkg.precompile()' 60 | # and make the kernel available to TLJH 61 | cp -r ~/.local/share/jupyter/kernels/julia-$julia_version_short /opt/tljh/user/share/jupyter/kernels/ 62 | 63 | # Install more packages 64 | if [ ! -z "$julia_packages" ] 65 | then 66 | julia --project=$julia_global_env -e 'deleteat!(DEPOT_PATH, [1,3]); using Pkg; Pkg.add.(split(ENV["julia_packages"], '\'':'\'')); Pkg.precompile()' 67 | fi 68 | 69 | # ensure all users can read General registry 70 | chmod -R a+rX $julia_global_depot 71 | 72 | # The installed packages are availabe to all users now. 73 | # But to avoid user-installs trying to write to the global Project.toml, 74 | # give them their own Project.toml by adding it to /etc/skel. 75 | # NOTE: already existing users will not get a fully working Julia install. Manually copy the files in /etc/skel for them. 76 | export julia_local_env_dir=$(julia -e 'using Pkg; print(Pkg.envdir("/etc/skel/.julia/"))') 77 | export julia_local_env=$julia_local_env_dir/v$julia_version_short 78 | mkdir -p $julia_local_env 79 | touch $julia_local_env/Project.toml 80 | mkdir -p /etc/skel/.julia/config 81 | echo "# Add load-path to globally installed packages" > /etc/skel/.julia/config/startup.jl 82 | echo "push!(LOAD_PATH, "\"$julia_global_env\"")" >> /etc/skel/.julia/config/startup.jl 83 | -------------------------------------------------------------------------------- /tljh_install.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # 3 | # This installs The Littlest JupyterHub with a Julia Kernel on a Ubuntu server. 4 | # (Tested on Linode.com with Ubuntu 20.04 LTS) 5 | # 6 | # Make sure to create a settings_tljh_julia.sh from 7 | # settings_tljh_julia_TEMPLATE.sh?" 8 | 9 | ## Settings 10 | # make sure settings file is there 11 | if [ ! -f "settings_tljh_julia.sh" ]; then 12 | echo "settings_tljh_julia.sh does not exist!" 13 | echo "Did you edit and rename settings_tljh_julia_TEMPLATE.sh?" 14 | echo "Exiting..." 15 | exit 1 16 | fi 17 | # get settings 18 | . ./settings_tljh_julia.sh 19 | 20 | # # IPv4 address of the server 21 | # ip4=$(ip -4 addr show eth0 | grep -oP '(?<=inet\s)\d+(\.\d+){3}') 22 | # # ip4=$(hostname -i | cut - -d" " -f2) 23 | 24 | ############################### 25 | ## Server (Linode) base install 26 | ############################### 27 | 28 | ## Update packages 29 | apt update 30 | apt upgrade -y 31 | 32 | timedatectl set-timezone $timezone 33 | 34 | # ## Networking 35 | # echo $hostname > /etc/hostname 36 | # hostname -F /etc/hostname 37 | # # This sets the Fully Qualified Domain Name 38 | # if [ ! -v $domain ]; then 39 | # echo $ip4 $fqdn >> /etc/hosts 40 | # fi 41 | 42 | ## SSH 43 | # disable password login if there is a SSH key for login 44 | if [ -f "/root/.ssh/authorized_keys" ]; then 45 | echo "PasswordAuthentication no" >> /etc/ssh/sshd_config 46 | systemctl restart sshd 47 | fi 48 | 49 | ## Firewall 50 | apt install ufw 51 | ufw default allow outgoing 52 | ufw default deny incoming 53 | ufw limit ssh 54 | ufw allow https 55 | ufw allow http 56 | # ufw enable 57 | yes | ufw enable 58 | 59 | ################################## 60 | ## The Littlest JupyterHub install 61 | ################################## 62 | # follows https://tljh.jupyter.org/en/latest/install/custom-server.html 63 | 64 | ## Install dependencies 65 | apt install python3 python3-dev git curl -y 66 | 67 | ## Install TLJH 68 | curl -L https://tljh.jupyter.org/bootstrap.py | python3 - --admin $jupyteradmin 69 | 70 | ## TLJH config 71 | ############## 72 | 73 | ## HTTPS setup 74 | case $https_setup in 75 | 76 | none) 77 | echo "HTTPS: Not setting up HTTPS" 78 | ;; 79 | 80 | self-cert) 81 | echo "HTPPS: Installing self-signed certificates" 82 | # this just uses default inputs for the key & value. If you want to set them use 83 | # a config file 84 | # https://www.openssl.org/docs/man1.0.2/man1/openssl-req.html#CONFIGURATION-FILE-FORMAT 85 | openssl req -newkey rsa:4096 -x509 -sha256 -days 365 -nodes -out tljh.crt -keyout tljh.key -batch 86 | mv tljh.key tljh.crt /opt/tljh/state 87 | tljh-config set https.enabled true 88 | tljh-config set https.tls.key /opt/tljh/state/tljh.key 89 | tljh-config set https.tls.cert /opt/tljh/state/tljh.crt 90 | tljh-config reload proxy 91 | ;; 92 | 93 | existing-cert) 94 | echo "HTPPS: Installing existing certificates" 95 | cp $ssl_key $ssl_cert /opt/tljh/state 96 | tljh-config set https.enabled true 97 | tljh-config set https.tls.key /opt/tljh/state/$(basename $ssl_key) 98 | tljh-config set https.tls.cert /opt/tljh/state/$(basename $ssl_cert) 99 | tljh-config reload proxy 100 | ;; 101 | 102 | letsencrypt) 103 | if [ ! -v $email4letsencrypt ]; then 104 | echo "HTPPS: Installing Let's Encrypt certificate" 105 | # https://tljh.jupyter.org/en/latest/howto/admin/https.html#howto-admin-https 106 | tljh-config set https.enabled true 107 | tljh-config set https.letsencrypt.email $email4letsencrypt 108 | tljh-config add-item https.letsencrypt.domains $fqdn 109 | tljh-config reload proxy 110 | else 111 | echo 'HTTPS: Option "https_setup=letsencrypt" but no value given for option "email4letsencrypt". Lets Encrypt needs an email. Exiting...' 112 | exit 1 113 | fi 114 | ;; 115 | 116 | *) 117 | echo 'HTTPS: Option for "https_setup" not valid. Exiting' 118 | exit 1 119 | ;; 120 | esac 121 | # run `tljh-config show` to check settings 122 | 123 | 124 | ## more TLJH config 125 | 126 | # Set timeout after which server shuts down 127 | # https://tljh.jupyter.org/en/latest/topic/idle-culler.html?highlight=timeout 128 | tljh-config set services.cull.timeout $tljh_timeout 129 | # Limit CPU & RAM 130 | # https://tljh.jupyter.org/en/latest/topic/tljh-config.html?highlight=environment#user-server-limits 131 | #+begin_src 132 | tljh-config set limits.memory $tljh_limits_memory 133 | tljh-config set limits.cpu $tljh_limits_cpu 134 | 135 | tljh-config reload 136 | 137 | ################################################# 138 | # Python and Julia package installs (system-wide) 139 | ################################################# 140 | # Including installing the Julia Jupyter-kernel 141 | 142 | ## Install python packages (that's easy) 143 | pip install numpy 144 | pip install matplotlib 145 | pip install scipy 146 | 147 | ./julia_install.sh $julia_version $julia_packages 148 | 149 | ########## 150 | # All done 151 | ########## 152 | echo "Install finished!" 153 | echo "Have a look at the tljh_extras.sh scripts for extras." 154 | echo " " 155 | echo "Probably it is also the time for a reboot, as likely a new Linux-kernel was installed." 156 | echo " " 157 | echo "Then login with your admin account $jupyteradmin on $fqdn (or $ip4 if you have no url)." 158 | echo "Create your users using the web-control panel." 159 | -------------------------------------------------------------------------------- /README.org: -------------------------------------------------------------------------------- 1 | #+Title: How to install The Littlest JupyterHub (TLJH) with a Julia kernel 2 | 3 | It sure took me some time to figure this one out. [[https://tljh.jupyter.org/en/latest/index.html][The Littlest JupyterHub]] 4 | install itself was a breeze but the Julia side of things was not as it 5 | was tricky to find the documentation. Thanks to @fredrikekre for the help! 6 | 7 | These are notes to myself plus a script to do the install on a 8 | Linode.com server running Ubuntu. But the script should work on any 9 | Ubuntu server and many Linux servers. It maybe useful to others too. 10 | If you have questions or corrections, open an issue/pull-request. 11 | 12 | Also, I figured out a setup to run TLJH within a virtual machine on a 13 | (real) server we have and expose it to the internet, see [[https://maurow.bitbucket.io/notes/multipass-vm-port-forwarding.html][here]]. 14 | 15 | *NOTE*: only run this on a /dedicated server/ (or virtual-machine) as it 16 | will mess with settings you probably don't want messed with! For 17 | security reasons, this is also the recommendation from the TLJH-folks ([[https://tljh.jupyter.org/en/latest/install/custom-server.html][link]]). 18 | 19 | *TL;DR* 20 | - clone this repo to your server: ~git clone https://github.com/mauro3/JupyterHubWithJulia.git~ 21 | - copy ~settings_tljh_julia_TEMPLATE.sh~ to ~settings_tljh_julia.sh~ 22 | - edit ~settings_tljh_julia.sh~ to your liking 23 | - optional but needed for HTTPS: give the server a domain name by setting 24 | the DNS records 25 | - run ~./tljh_install.sh~ (as root). This will take about 10-20min. 26 | - login on your JupyterHub website as root and make user accounts 27 | - done 28 | 29 | TODO: 30 | - make and use a system-image for Julia 31 | 32 | * The scripts tljh_install.sh and settings_tljh_julia.sh 33 | 34 | There are three scripts in this repo (and they are the authoritative 35 | source, although I will try to keep this README up to date): 36 | - settings-template is [[./settings_tljh_julia_TEMPLATE.sh][settings_tljh_julia_TEMPLATE.sh]] which needs to 37 | be filled in and renamed to ~settings_tljh_julia.sh~. 38 | - install script [[./tljh_install.sh][tljh_install.sh]] which does the full install 39 | - this calls into [[./julia_install.sh][julia_install.sh]] to install Julia 40 | 41 | In the following I will describe what these scipts do. If you execute 42 | the commands by hand: 43 | - do it as root user 44 | - first source ~. ./settings_tljh_julia.sh~ 45 | 46 | * Global settings 47 | All global settings are contained within 48 | ~settings_tljh_julia.sh~ which is derived from editing 49 | [[./settings_tljh_julia_TEMPLATE.sh]]. These settings are used 50 | throughout the install, so it is assumed that this file is sourced 51 | before executing the commands listed below (it is sourced 52 | automatically in ~tljh_install.sh~). 53 | 54 | * Linode (or any Ubutnu) server setup 55 | Linode.com offers Linux servers in the cloud. Currently there are 56 | quite a few $100-credit offers going around, that's why I started here. 57 | But I think the script should work on any Ubuntu 20.04 system. 58 | 59 | *NOTE*: again, only run these scripts on a dedicated server! Both for 60 | security reasons (TLJH-people recommendation) and because these 61 | scripts will likely mess with your setup. 62 | 63 | Install: 64 | - Ubuntu 20.04 LTS 65 | - choose at least 2GB of RAM (but much more for multi-user) 66 | 67 | Boot the server. Then, if you want HTTPS, set the domain record next. 68 | 69 | ** Setting a Domain Record via http://linode.com 70 | Having an actual url is needed for HTTPS access. 71 | 72 | If you have a Domain at Linode, otherwise see below: 73 | - Go to https://cloud.linode.com/domains/ 74 | - click "Add an A/AAAA Record" 75 | - Settings 76 | Hostname: whatever-your-hostname-is 77 | IP address: copy from your server 78 | TTL: 5min (this sets on how quickly records propagate through DNS on changes) 79 | - check in a few seconds/minutes whether the record propagated far 80 | enough for Let's Encrypt to pick it up with https://letsdebug.net/ 81 | (note that the error about no web-server running is ok. It just 82 | needs to be able to resolve the domain name.) 83 | 84 | For the domain transfer to Linode I followed 85 | https://merelycurious.me/post/connecting-namecheap-domain-to-linode; 86 | but you don't need to have your domain at Linode. 87 | 88 | *** Aside: HTTPS with Let's Encrypt 89 | *The Let's Encrypt setup I do via The Littelest JuptyterHub install, which is super easy.* 90 | But here some notes on how to do it without using the TLJH machinery: 91 | 92 | Let's encrypt: [[https://www.linode.com/docs/guides/install-lets-encrypt-to-create-ssl-certificates/][docs]] 93 | - root@localhost:/opt/letsencrypt# ./letsencrypt-auto certonly --standalone -d example.com -d jhub.example.com -d docker.example.com 94 | -> does not work 95 | - loosly follow instead 96 | https://www.rosehosting.com/blog/how-to-install-lets-encrypt-on-ubuntu-20-04-with-apache/ 97 | - apt install software-properties-common 98 | - apt update 99 | - apt install certbot 100 | - certbot certonly --standalone -d jhub.example.com 101 | - keys are in /etc/letencrypt 102 | 103 | 104 | ** The install 105 | 106 | The install is done within the section ~Server (Linode) base install~ 107 | of the install script [[./tljh_install.sh][tljh_install.sh]]. It does: 108 | - update system via ~apt~ 109 | - set time-zone 110 | - set networking stuff by hand (this is disabled in the script as it does not work in general) 111 | - disable SSH password login, if there is a /root/.ssh/authorized_keys 112 | file (i.e. assuming there is a key for passwordless login). *Note* 113 | this potentially locks you out of the system, if so disable it. 114 | - setup ~ufw~ firewall 115 | 116 | * The littelest JuliaHub (TLJH) install 117 | 118 | Up-front note: there are two terminals once TLJH is running: 119 | - the normal shell (via ssh) 120 | - the terminal in the web-interface 121 | The latter has some special environment variables set, namely the 122 | ~$PATH~. The setup described here uses the normal shell exclusively 123 | (by setting the ~$PATH~), this is a bit different to the docs on 124 | https://tljh.jupyter.org/en/latest/index.html. 125 | 126 | This part of the install is done in the section ~The Littlest 127 | JupyterHub install~ of [[./tljh_install.sh][tljh_install.sh]]. 128 | 129 | ** Install 130 | The install follows 131 | https://tljh.jupyter.org/en/latest/install/custom-server.html 132 | #+begin_src 133 | curl -L https://tljh.jupyter.org/bootstrap.py | python3 - --admin $jupyteradmin 134 | #+end_src 135 | 136 | *Don't create users yet.* 137 | 138 | ** HTTPS 139 | The install-script only does this if the ~$email4letsencrypt~ variable 140 | is set. 141 | 142 | - Using Let's Encrypt 143 | - this only works once the domain is set in DNS manager (see above). Once set, it should be 144 | very quick (seconds) for Let's Encrypt to be able to resolve the 145 | DNS. You can check with https://letsdebug.net/ whether it can 146 | resolve it. 147 | - It's done through TLJH 148 | https://tljh.jupyter.org/en/latest/howto/admin/https.html#howto-admin-https 149 | - If there are problems, the website https://letsdebug.net/ might help. 150 | 151 | 152 | #+begin_src 153 | tljh-config set https.enabled true 154 | tljh-config set https.letsencrypt.email $email4letsencrypt 155 | tljh-config add-item https.letsencrypt.domains $fqdn 156 | #+end_src 157 | Check and reload: 158 | #+begin_src 159 | tljh-config show 160 | tljh-config reload proxy 161 | #+end_src 162 | 163 | ** TLJH config 164 | *Set kernel shutdown time* 165 | 166 | The Jupyter-kernel of each user will shut down after some idle time, 167 | 10min by default. 168 | Probably increase this as Julia takes time to startup, so a shutdown 169 | is annoying. 170 | Ref: https://tljh.jupyter.org/en/latest/topic/idle-culler.html?highlight=timeout 171 | 172 | At the root-shell: 173 | #+begin_src 174 | tljh-config set services.cull.timeout 3600 175 | tljh-config reload 176 | #+end_src 177 | 178 | *Limit CPU & RAM* 179 | https://tljh.jupyter.org/en/latest/topic/tljh-config.html?highlight=environment#user-server-limits 180 | 181 | Note that Julia is quite memory hungry with one Julia notebook taking 182 | up at least 400MB and more once packages are used (Python notebooks 183 | start at 130MB). Thus, set at least 1GB of memory per user, better 184 | 2GB. 185 | 186 | #+begin_src 187 | tljh-config set limits.memory 2G 188 | tljh-config set limits.cpu 1 189 | tljh-config reload 190 | #+end_src 191 | 192 | ** A note on JupyterHub Passwords 193 | The default install is that each user sets their password on the first 194 | login: https://tljh.jupyter.org/en/latest/howto/auth/firstuse.html 195 | 196 | - change your own password on 197 | https://jhub.example.com/hub/auth/change-password 198 | - admin can reset user password with 199 | https://tljh.jupyter.org/en/latest/install/custom-server.html 200 | - admin can reset admin password with 201 | https://tljh.jupyter.org/en/latest/howto/admin/admin-users.html 202 | 203 | * Package & Julia install 204 | This was the tricky bit to figure out, or more precisely, the Julia 205 | side of it is a bit tricky. 206 | 207 | *If you follow along by hand, maybe backup your server now.* On Linode 208 | there is the "Manual Snapshot" option in the "Backups" tab, probably 209 | best to first shutdown the server though. That way you can get back 210 | to the good install. For me, the size of the Linode backup was 32GB. 211 | 212 | This part of the install is done in the section ~Python and Julia 213 | package installs (system-wide)~ of [[./tljh_install.sh][tljh_install.sh]] which calls out to 214 | [[./julia_install.sh][julia_install.sh]] for the heavy lifting. 215 | 216 | ** Machine-wide Python packages install 217 | See 218 | https://tljh.jupyter.org/en/latest/howto/env/user-environment.html, 219 | but note that we don't execute the commands at the web-terminal, thus 220 | dropping the ~sudo -E~. 221 | 222 | #+begin_src 223 | pip install numpy 224 | pip install matplotlib 225 | pip install scipy 226 | #+end_src 227 | This will spew some warnings about "WARNING: The directory 228 | '/home/jupyter-admin/.cache/pip/http' or its parent directory is not 229 | owned by the current user"; as far as I can tell, those are ok. 230 | 231 | ** Machine-wide Julia install 232 | This is in [[./julia_install.sh][julia_install.sh]]. 233 | *** Install Julia binaries 234 | 235 | Essentially downloads Julia binaries, puts them into the right place 236 | and adds some sym-links. See setion ~## Download and unpack Julia~ of [[./julia_install.sh][julia_install.sh]]. 237 | 238 | *** Julia package install 239 | This is where it gets a bit dicey. I do the following: 240 | - the root user installs the system-wide packages using the depot path 241 | ~DEPOT_PATH[2]~ (this is a variable defined within running Julia) 242 | and using an environment ~DEPOT_PATH[2]/environments//~. 243 | - install the packages: 244 | - IJulia to actually make the Jupyter notebooks work 245 | - any other packages 246 | - To make these packages available to users the ~LOAD_PATH~ of the 247 | users needs to be set accordingly. 248 | - Ideally, a sysimage would then be created with the installed 249 | packages for speedy startup: [[*Julia Sysimage (WIP)][Julia Sysimage (WIP)]]. But this I 250 | haven't tried yet. 251 | 252 | **** Global depot and environment 253 | Make special environment and global depot-folder: 254 | #+begin_src 255 | # the packages are installed into this depot: 256 | export julia_global_depot=$(julia -e 'print(DEPOT_PATH[2])') 257 | mkdir -p $julia_global_depot 258 | 259 | # The corresponding environment is (another one could be chosen): 260 | export julia_global_env_dir=$(julia -e 'using Pkg; print(Pkg.envdir(DEPOT_PATH[2]))') 261 | export julia_global_env=$julia_global_env_dir/v$julia_version_short 262 | mkdir -p $julia_global_env 263 | touch $julia_global_env/Project.toml 264 | #+end_src 265 | 266 | **** IJulia install 267 | The Julia kernel needs to be copied to the location where TLJH can use it. 268 | 269 | #+begin_src 270 | julia --project=$julia_global_env -e 'deleteat!(DEPOT_PATH, [1,3]); using Pkg; Pkg.update(); Pkg.add("IJulia"); Pkg.precompile()' 271 | cp -r ~/.local/share/jupyter/kernels/julia-$julia_version_short /opt/tljh/user/share/jupyter/kernels/ 272 | #+end_src 273 | 274 | Adapted from 275 | https://github.com/dclong/docker-jupyterhub-julia/blob/master/Dockerfile; 276 | note that the two `chmod` in that docker file are not needed here (in 277 | fact are bad, because global package updates then fail). 278 | 279 | **** Install more Julia packages 280 | Install more Julia packages as specified in the settings variable ~$julia_packages~: 281 | 282 | #+begin_src 283 | julia --project=$julia_global_env -e 'deleteat!(DEPOT_PATH, [1,3]); using Pkg; Pkg.update(); Pkg.add.(split(ENV["julia_packages"], '\'':'\'')); Pkg.precompile()' 284 | #+end_src 285 | 286 | Note, the precompilation is usable for all users. 287 | 288 | **** Set the user LOAD_PATH to pick up the global packages 289 | 290 | The installed packages are availabe to all users now but they don't 291 | have their own environment at the moment, give it to them: 292 | #+begin_src 293 | mkdir -p /etc/skel/.julia/environments/v1.4 294 | touch /etc/skel/.julia/environments/v1.4/Project.toml 295 | #+end_src 296 | This uses the ~/etc/skel~ directory which is used as template (by Linux) when a 297 | user is created. 298 | 299 | But now their own environment shadows the global one, to rectify this 300 | the global one needs to be specified explicitly: 301 | #+begin_src 302 | export julia_local_env_dir=$(julia -e 'using Pkg; print(Pkg.envdir("/etc/skel/.julia/"))') 303 | export julia_local_env=$julia_local_env_dir/v$julia_version_short 304 | mkdir -p $julia_local_env 305 | touch $julia_local_env/Project.toml 306 | mkdir -p /etc/skel/.julia/config 307 | echo "# Add load-path to globally installed packages" > /etc/skel/.julia/config/startup.jl 308 | echo "push!(LOAD_PATH, "\"$julia_global_env\"")" >> /etc/skel/.julia/config/startup.jl 309 | #+end_src 310 | 311 | **** Julia Sysimage (WIP) 312 | This is work in progress. If I get to run it, I'll update here. 313 | Create a sysimage with the globally installed packages. 314 | 315 | https://julialang.github.io/PackageCompiler.jl/dev/sysimages/ 316 | 317 | Precompile script ~tmp.jl~: 318 | #+begin_src 319 | using ... # installed packages 320 | 321 | # execute what is normally executed to make sysimage-compilation pick it up 322 | 323 | notebook() 324 | #+end_src 325 | 326 | All in all: 327 | 328 | ~create_sysimage([packages...], sysimage_path="/tmp/sysimg2.so", precompile_execution_file="tmp.jl")~ 329 | 330 | * All done, mostly 331 | The script [[./tljh_install.sh][tljh_install.sh]] finished with the last section. Here some 332 | additional and/or extra steps. 333 | 334 | ** Deployment for users 335 | 336 | *** Create users 337 | 338 | Login as admin user on the web-page and go to the "Admin" panel in the 339 | web interface. 340 | 341 | Note that the corresponding unix users will only be created upon their 342 | first login. 343 | 344 | *** Extra stuff to do 345 | 346 | See [[./tljh_extras.sh][tljh_extras.sh]]. 347 | 348 | 349 | - how to deploy notebooks? 350 | - See [[https://tljh.jupyter.org/en/latest/howto/content/nbgitpuller.html][nbgitpuller]] 351 | - add them to ~/etc/skel~ 352 | - copy directories/files to all users with [[./copy_to_users.sh][copy_to_users.sh]] script 353 | - [[https://tljh.jupyter.org/en/latest/howto/content/share-data.html#option-3-create-a-directory-for-users-to-share-notebooks-and-other-files][how to let the users collaborate]]: 354 | #+begin_src 355 | mkdir -p /srv/scratch 356 | chown root:jupyterhub-users /srv/scratch 357 | chmod 777 /srv/scratch 358 | chmod g+s /srv/scratch 359 | ln -s /srv/scratch /etc/skel/scratch 360 | #+end_src 361 | --------------------------------------------------------------------------------