├── Assets ├── gcp-3.png ├── gcp-4.png ├── gcp-s1.png ├── gcp-s2.png ├── gcp-s5.png ├── gcp-s6.png └── SingularityLogo.png ├── GPU.md ├── Examples ├── DefFiles │ └── lolcow ├── mpi │ ├── submit.sh │ ├── mpitest.c │ └── README.md └── gcp │ └── README.md ├── LICENSE.md ├── MISCELLANEOUS.md ├── INSTALL.md ├── INSTANCES.md ├── README.md ├── ADVANCED_USAGE.md ├── CLOUD_SERVICES.md └── BUILD_RUN.md /Assets/gcp-3.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ArangoGutierrez/Singularity-tutorial/HEAD/Assets/gcp-3.png -------------------------------------------------------------------------------- /Assets/gcp-4.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ArangoGutierrez/Singularity-tutorial/HEAD/Assets/gcp-4.png -------------------------------------------------------------------------------- /Assets/gcp-s1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ArangoGutierrez/Singularity-tutorial/HEAD/Assets/gcp-s1.png -------------------------------------------------------------------------------- /Assets/gcp-s2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ArangoGutierrez/Singularity-tutorial/HEAD/Assets/gcp-s2.png -------------------------------------------------------------------------------- /Assets/gcp-s5.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ArangoGutierrez/Singularity-tutorial/HEAD/Assets/gcp-s5.png -------------------------------------------------------------------------------- /Assets/gcp-s6.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ArangoGutierrez/Singularity-tutorial/HEAD/Assets/gcp-s6.png -------------------------------------------------------------------------------- /Assets/SingularityLogo.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ArangoGutierrez/Singularity-tutorial/HEAD/Assets/SingularityLogo.png -------------------------------------------------------------------------------- /GPU.md: -------------------------------------------------------------------------------- 1 | ## GPU computing 2 | 3 | In Singularity v3.0+ the `--nv` option will look for NVIDIA libraries on the host system and automatically bind mount them to the container so that GPUs work seamlessly. 4 | -------------------------------------------------------------------------------- /Examples/DefFiles/lolcow: -------------------------------------------------------------------------------- 1 | BootStrap: docker 2 | From: ubuntu:16.04 3 | 4 | %post 5 | apt-get -y update 6 | apt-get -y install fortune cowsay lolcat 7 | 8 | %environment 9 | export LC_ALL=C 10 | export PATH=/usr/games:$PATH 11 | 12 | %runscript 13 | fortune | cowsay | lolcat 14 | -------------------------------------------------------------------------------- /Examples/mpi/submit.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | #SBATCH -n 4 3 | #SBATCH -p GPU 4 | #SBATCH --ntasks-per-node=2 5 | 6 | SINGULARITYENV_LD_LIBRARY_PATH=/opt/openmpi/lib 7 | SINGULARITYENV_PREPEND_PATH=/opt/openmpi/bin 8 | 9 | mpiexec -n 4 singularity exec -B /opt/openmpi ~/centos7.sif ~/mpitest 10 | 11 | echo $? 12 | -------------------------------------------------------------------------------- /Examples/mpi/mpitest.c: -------------------------------------------------------------------------------- 1 | /* 2 | Copyright (c) 2018, Sylabs, Inc. All rights reserved. 3 | This software is licensed under a 3-clause BSD license. Please 4 | consult LICENSE.md file distributed with the sources of this project regarding 5 | your rights to use or distribute this software. 6 | */ 7 | 8 | #include 9 | #include 10 | #include 11 | 12 | int main(int argc, char** argv) { 13 | 14 | MPI_Init(NULL, NULL); 15 | 16 | int size; 17 | MPI_Comm_size(MPI_COMM_WORLD, &size); 18 | 19 | int rank; 20 | MPI_Comm_rank(MPI_COMM_WORLD, &rank); 21 | 22 | char proc_name[MPI_MAX_PROCESSOR_NAME]; 23 | int len; 24 | MPI_Get_processor_name(proc_name, &len); 25 | 26 | printf("Hello from proc %s, rank %d of %d\n", proc_name, rank, size); 27 | 28 | MPI_Finalize(); 29 | 30 | exit(0); 31 | } 32 | -------------------------------------------------------------------------------- /Examples/gcp/README.md: -------------------------------------------------------------------------------- 1 | # Getting started guidelines 2 | ## Running Singularity on GCP 3 | 4 | 5 | ### Via web GCP web platform 6 | Select SingularityPRO-25 from the Google Cloud Launcher console and click Launch on Compute Engine 7 | 8 | ### Deploy SingularityPRO-25 9 | 1. Choose a name for your SingularityPRO-25 instance 10 | 2. Choose a machine type with at least 6GB of RAM; SingularityPRO-25 is charge free, you will only pay for the machine resource 11 | 3. Click Deploy 12 | 13 | 14 | ### Wait for it... 15 | 16 | 4. See it run! Save the Admin URL, user and password for the next steps. 17 | 18 | 5. Click on the SSH icon 19 | 20 | We now have a running GCP instance with SingularityPRO-25 packed! 21 | 22 | 23 | ### Check and start deploying your containers 24 | 25 | -------------------------------------------------------------------------------- /LICENSE.md: -------------------------------------------------------------------------------- 1 | Redistribution and use in source and binary forms, with or without 2 | modification, are permitted provided that the following conditions are met: 3 | 4 | 1. Redistributions of source code must retain the above copyright notice, 5 | this list of conditions and the following disclaimer. 6 | 7 | 2. Redistributions in binary form must reproduce the above copyright notice, 8 | this list of conditions and the following disclaimer in the documentation 9 | and/or other materials provided with the distribution. 10 | 11 | 3. Neither the name of the copyright holder nor the names of its 12 | contributors may be used to endorse or promote products derived from this 13 | software without specific prior written permission. 14 | 15 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" 16 | AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 17 | IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE 18 | ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE 19 | LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR 20 | CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF 21 | SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS 22 | INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN 23 | CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) 24 | ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE 25 | POSSIBILITY OF SUCH DAMAGE. 26 | -------------------------------------------------------------------------------- /MISCELLANEOUS.md: -------------------------------------------------------------------------------- 1 | ## Miscellaneous Topics 2 | 3 | ### pipes and redirection 4 | 5 | As we demonstrated earlier, pipes and redirects work as expected between a container and host system. If you need to pipe the output of one command in your container to another command in your container things may be more complicated. 6 | 7 | ``` 8 | $ singularity exec lolcow.sif fortune | singularity exec lolcow.sif cowsay 9 | ``` 10 | 11 | ### X11 and OpenGL 12 | 13 | You can use Singularity containers to display graphics through common protocols. To do this, you need to install the proper graphics stack within the Singularity container. For instance if you want to display X11 graphics you must install `xorg` within your container. In an Ubuntu container the command would look like this. 14 | 15 | ``` 16 | $ apt-get install xorg 17 | ``` 18 | 19 | ### Using the network on the host system 20 | 21 | Network ports on the host system are accessible from within the container and work seamlessly. For example, you could install ipython within a container, start a jupyter notebook instance, and then connect to that instance using a browser running outside of the container on the host system or from another host. 22 | 23 | ### a note on SUID programs and daemons 24 | 25 | Some programs need root privileges to run. These often include services or daemons that start via the `init.d` or `system.d` systems and run in the background. For instance, `sshd` the ssh daemon that listens on port 22 and allows another user to connect to your computer requires root privileges. You will not be able to run it in a container unless you start the container as root. 26 | 27 | Other programs may set the SUID bit or capabilities to run as root or with elevated privileges without your knowledge. For instance, the well-known `ping` program actually runs with elevated privileges (and needs to since it sets up a raw network socket). This program will not run in a container unless you are root in the container. 28 | -------------------------------------------------------------------------------- /Examples/mpi/README.md: -------------------------------------------------------------------------------- 1 | # MPI 2 | One of the architecturally defined features in Singularity is that it can execute containers like they are native programs or scripts on a host computer. As a result, integration with schedulers is simple and runs exactly as you would expect. All standard input, output, error, pipes, IPC, and other communication pathways that locally running programs employ are synchronized with the applications running locally within the container. 3 | 4 | Additionally, because Singularity is not emulating a full hardware level virtualization paradigm, there is no need to separate out any sandboxed networks or file systems because there is no concept of user-escalation within a container. Users can run Singularity containers just as they run any other program on the HPC resource. 5 | 6 | Include the appropriate development tools into the container (notice we are calling 7 | singularity as root and the container is writable) 8 | 9 | ``` 10 | $ sudo singularity build --sandbox centos-7 library://centos:7 11 | $ sudo singularity exec -w centos-7.sif yum groupinstall -y "Development Tools" 12 | ``` 13 | ### Obtain the stable version of Open MPI 14 | ``` 15 | $ wget https://www.open-mpi.org/software/ompi/v2.1/downloads/openmpi-2.1.0.tar.bz2 16 | $ tar -xf openmpi-2.1.0.tar.bz2 17 | $ cd openmpi-2.1.0 18 | $ singularity exec ../centos-7 ./configure --prefix=/usr/local 19 | $ singularity exec ../centos-7 make -j$(nrpoc) 20 | ``` 21 | ### Install OpenMPI into the container 22 | 23 | > notice now running as root and container is writable 24 | 25 | ``` 26 | $ sudo singularity exec -w -B /home ../centos-7 make install 27 | ``` 28 | ### Build the OpenMPI ring example and place the binary in this directory 29 | ``` 30 | $ singularity exec centos-7 mpicc openmpi-2.1.0/examples/ring_c.c -o ring 31 | 32 | $ sudo singularity exec -w -B /home centos-7 cp ring /usr/bin/ring 33 | ``` 34 | 35 | Now that our sandbox has all the requirements to run mpi, let's save it as a SIF file 36 | 37 | ``` 38 | $ cd .. 39 | 40 | $ sudo singularity build centos7.sif centos-7 41 | ``` 42 | 43 | ### Run the MPI program within the container by calling the MPIRUN on the host 44 | 45 | > *WARNING* MPI must be pre installed on the host 46 | 47 | ``` 48 | $ mpirun -np 4 singularity exec centos-7.sif /usr/bin/ring 49 | ``` 50 | ### Submit mpi example over a SLURM infraestruture 51 | 52 | > *WARNING* You need to be in a SLURM head-node in order to run this last step 53 | 54 | ```bash 55 | mpicc -o mpitest mpitest.c 56 | sbatch -N 2 submit.sh 57 | ``` 58 | -------------------------------------------------------------------------------- /INSTALL.md: -------------------------------------------------------------------------------- 1 | ## Install 2 | 3 | Here we will install the latest tagged release from [GitHub](https://github.com/singularityware/singularity). If you prefer to install a different version or to install Singularity in a different location, see these [INSTALL.md](https://github.com/sylabs/singularity/blob/master/INSTALL.md) 4 | 5 | ### Install golang 6 | This is one of several ways to [install and configure golang](https://golang.org/doc/install). 7 | 8 | First, visit the [golang download page](https://golang.org/dl/) and pick a 9 | package archive to download. Copy the link address and download with `wget`. 10 | 11 | ``` 12 | $ export VERSION=1.13 OS=linux ARCH=amd64 13 | $ cd /tmp 14 | $ wget https://dl.google.com/go/go$VERSION.$OS-$ARCH.tar.gz 15 | ``` 16 | 17 | Then extract the archive to `/usr/local` (or use other instructions on go 18 | installation page). 19 | 20 | ``` 21 | $ sudo tar -C /usr/local -xzf go$VERSION.$OS-$ARCH.tar.gz 22 | ``` 23 | 24 | Finally, set up your environment for go 25 | 26 | ``` 27 | $ echo 'export GOPATH=${HOME}/go' >> ~/.bashrc 28 | $ echo 'export PATH=/usr/local/go/bin:${PATH}:${GOPATH}/bin' >> ~/.bashrc 29 | $ source ~/.bashrc 30 | ``` 31 | ### Install singularity from source 32 | 33 | We're going to compile Singularity from source code. First we'll need to make sure we have some development tools installed so that we can do that. On Ubuntu, run these commands to make sure you have all the necessary packages installed. 34 | 35 | ON a DEB based OS, these commmands should get you up to speed: 36 | 37 | ``` 38 | $ sudo apt-get update && sudo apt-get install -y build-essential libssl-dev uuid-dev libgpgme11-dev squashfs-tools libseccomp-dev 39 | ``` 40 | 41 | On RPM based OS's, these commmands should get you up to speed. 42 | 43 | ``` 44 | $ sudo yum update 45 | 46 | $ sudo yum groupinstall 'Development Tools' 47 | 48 | $ sudo yum install libtool libarchive-devel openssl-devel libuuid-devel libseccomp-devel 49 | ``` 50 | 51 | now clone the repo and build the source code 52 | 53 | ``` 54 | git clone https://github.com/sylabs/singularity.git 55 | ``` 56 | 57 | Finally it's time to build and install! 58 | 59 | ``` 60 | $ cd singularity 61 | 62 | $ git checkout v3.4.0 63 | 64 | $ ./mconfig -p /usr/local 65 | 66 | $ make -C builddir/ -j$(nproc) 67 | 68 | $ sudo make -C builddir/ install 69 | ``` 70 | 71 | If you want support for tab completion of Singularity commands, you need to source the appropriate file and add it to the bash completion directory in `/etc` so that it will be sourced automatically when you start another shell. 72 | 73 | ``` 74 | $ source /usr/local/etc/bash_completion.d/singularity 75 | ``` 76 | 77 | If everything went according to plan, you now have a working installation of Singularity. You can test your installation like so: 78 | 79 | ``` 80 | $ singularity run library://sylabsed/examples/lolcow:latest 81 | ``` 82 | 83 | You should see something like the following. 84 | 85 | ``` 86 | _________________________________________ 87 | / Of course you have a purpose -- to find \ 88 | \ a purpose. / 89 | ----------------------------------------- 90 | \ ^__^ 91 | \ (oo)\_______ 92 | (__)\ )\/\ 93 | ||----w | 94 | || || 95 | ``` 96 | 97 | Your cow will likely say something different (and be more colorful), but as long as you see a cow your installation is working properly. 98 | 99 | This command downloads and runs a container from [Sylabs Library](https://cloud.sylabs.io/library/_container/5b9e91c694feb900016ea40b). During this tutorial we will learn how to build a similar container from scratch. 100 | 101 | ### Install image building dependencies 102 | You may want to build images from their official repositories like debootstrap, yum or zypper, you "may" need to install these depending on the OS you are using 103 | 104 | e.g 105 | if you get the following error 106 | ``` 107 | vagrant@ubuntu-bionic:~$ sudo singularity build --sandbox lolcow DefFiles/lolcow 108 | WARNING: Authentication token file not found : Only pulls of public images will succeed 109 | INFO: Starting build... 110 | FATAL: While performing build: conveyor failed to get: debootstrap is not in PATH... Perhaps 'apt-get install' it: exec: "debootstrap": executable file not found in $PATH 111 | ``` 112 | 113 | that means that your host doesn't have debootstrap installed, but you are trying to run a build from it, so you need to install it in order to proceed with your build. 114 | 115 | ``` 116 | sudo apt-get install debootstrap 117 | ``` 118 | -------------------------------------------------------------------------------- /INSTANCES.md: -------------------------------------------------------------------------------- 1 | ## Singularity Instances 2 | 3 | Up to now all of our examples have run Singularity containers in the foreground. But what if you want to run a service like a web server or a database in a Singularity container in the background? 4 | 5 | #### lolcow (useless) example 6 | In Singularity v3.0, you can use the [`instance` command group](https://www.sylabs.io/guides/3.0/user-guide/quick_start.html#interact-with-images) to start and control container instances that run in the background. To demonstrate, let's start an instance of our `lolcow.sif` container running in the background. 7 | 8 | ``` 9 | $ singularity instance start lolcow.simg cow1 10 | ``` 11 | 12 | We can use the `instance list` command to show the instances that are currently running. 13 | 14 | ``` 15 | $ singularity instance.list 16 | DAEMON NAME PID CONTAINER IMAGE 17 | cow1 10794 /home/eduardo/lolcow.sif 18 | ``` 19 | 20 | We can connect to running instances using the `instance://` URI like so: 21 | 22 | ``` 23 | $ singularity shell instance://cow1 24 | Singularity: Invoking an interactive shell within container... 25 | 26 | Singularity lolcow.sif:~> ps -ef 27 | UID PID PPID C STIME TTY TIME CMD 28 | eduardo 1 0 0 19:05 ? 00:00:00 singularity-instance: eduardo [cow1] 29 | eduardo 3 0 0 19:06 pts/0 00:00:00 /bin/bash --norc 30 | eduardo 4 3 0 19:06 pts/0 00:00:00 ps -ef 31 | 32 | Singularity lolcow.sif:~> exit 33 | ``` 34 | 35 | Note that we've entered a new PID namespace, so that the `singularity-instance` process has PID number 1. 36 | 37 | You can start multiple instances running in the background, as long as you give them unique names. 38 | 39 | ``` 40 | $ singularity instance start lolcow.sif cow2 41 | 42 | $ singularity instance start lolcow.sif cow3 43 | 44 | $ singularity instance.list 45 | DAEMON NAME PID CONTAINER IMAGE 46 | cow1 10794 /home/eduardo/lolcow.sif 47 | cow2 10855 /home/eduardo/lolcow.sif 48 | cow3 10885 /home/eduardo/lolcow.sif 49 | ``` 50 | 51 | You can stop individual instances using their unique names or stop all instances with the `--all` option. 52 | 53 | ``` 54 | $ singularity instance stop cow1 55 | Stopping cow1 instance of /home/eduardo/lolcow.sif (PID=10794) 56 | 57 | $ singularity instance stop --all 58 | Stopping cow2 instance of /home/eduardo/lolcow.sif (PID=10855) 59 | Stopping cow3 instance of /home/eduardo/lolcow.sif (PID=10885) 60 | ``` 61 | 62 | #### nginx (useful) example 63 | 64 | These examples are not very useful because `lolcow.sif` doesn't run any services. Let's extend the example to something useful by running a local nginx web server in the background. This command will download the official nginx image from Docker Hub and start it in a background instance called "web". (The commands need to be executed as root so that nginx can run with the privileges it needs.) 65 | 66 | ``` 67 | $ sudo singularity instance.start docker://nginx web 68 | Docker image path: index.docker.io/library/nginx:latest 69 | Cache folder set to /root/.singularity/docker 70 | [3/3] |===================================| 100.0% 71 | Creating container runtime... 72 | 73 | $ sudo singularity instance.list 74 | DAEMON NAME PID CONTAINER IMAGE 75 | web 15379 /tmp/.singularity-runtime.MBzI4Hus/nginx 76 | ``` 77 | 78 | Now to start nginx running in the instance called web. 79 | 80 | ``` 81 | $ sudo singularity exec instance://web nginx 82 | ``` 83 | 84 | Now we have an nginx web server running on our localhost. We can verify that it is running with `curl`. 85 | 86 | ``` 87 | $ curl localhost 88 | 127.0.0.1 - - [02/Nov/2017:19:20:39 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.52.1" "-" 89 | 90 | 91 | 92 | Welcome to nginx! 93 | 100 | 101 | 102 |

Welcome to nginx!

103 |

If you see this page, the nginx web server is successfully installed and 104 | working. Further configuration is required.

105 | 106 |

For online documentation and support please refer to 107 | nginx.org.
108 | Commercial support is available at 109 | nginx.com.

110 | 111 |

Thank you for using nginx.

112 | 113 | 114 | ``` 115 | 116 | When finished, don't forget to stop all running instances like so: 117 | 118 | ``` 119 | $ sudo singularity instance stop --all 120 | ``` 121 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Creating and running software containers with Singularity 2 | ### How to use [Singularity](https://www.sylabs.io/)! 3 | 4 | 5 | 6 | - [Singularity Home](https://www.sylabs.io/) 7 | - [Singularity on GitHub](https://github.com/sylabs/singularity) 8 | - [Singularity on Google Groups](https://groups.google.com/a/lbl.gov/forum/#!forum/singularity) 9 | - [Sylabs Library](https://cloud.sylabs.io/library) 10 | 11 | ## Introduction 12 | 13 | ### What IS a software container anyway? (And what's it good for?) 14 | 15 | A container allows you to stick an application and all of its dependencies into a single package. This makes your application portable, shareable, and reproducible. 16 | 17 | Containers foster portability and reproducibility because they package **ALL** of an applications dependencies... including its own tiny operating system! 18 | 19 | This means your application won't break when you port it to a new environment. Your app brings its environment with it. 20 | 21 | Here are some examples of things you can do with containers: 22 | 23 | - Package an analysis pipeline so that it runs on your laptop, in the cloud, and in a high performance computing (HPC) environment to produce the same result. 24 | - Publish a paper and include a link to a container with all of the data and software that you used so that others can easily reproduce your results. 25 | - Install and run an application that requires a complicated stack of dependencies with a few keystrokes. 26 | - Create a pipeline or complex workflow where each individual program is meant to run on a different operating system. 27 | 28 | ### How do containers differ from virtual machines (VMs) 29 | 30 | Containers and VMs are both types of virtualization. But it's important to understand the differences between the two and know when to use each. 31 | 32 | **Virtual Machines** install every last bit of an operating system (OS) right down to the core software that allows the OS to control the hardware (called the _kernel_). This means that VMs: 33 | - Are complete in the sense that you can use a VM to interact with your computer via a different OS. 34 | - Are extremely flexible. For instance you an install a Windows VM on a Mac using software like [VirtualBox](https://www.virtualbox.org/wiki/VirtualBox). 35 | - Are slow and resource hungry. Every time you start a VM it has to bring up an entirely new OS. 36 | 37 | **Containers** share a kernel with the host OS. This means that Containers: 38 | - Are less flexible than VMs. For example, a Linux container must be run on a Linux host OS. (Although you can mix and match distributions.) In practice, containers are only extensively developed on Linux. 39 | - Are much faster and lighter weight than VMs. A container may be just a few MB. 40 | - Start and stop quickly and are suitable for running single apps. 41 | 42 | Because of their differences, VMs and containers serve different purposes and should be favored under different circumstances. 43 | - VMs are good for long running interactive sessions where you may want to use several different applications. (Checking email on Outlook and using Microsoft Word and Excel). 44 | - Containers are better suited to running one or two applications non-interactively in their own custom environments. 45 | 46 | ### Singularity 47 | 48 | [Singularity](https://www.sylabs.io/) is a 3 years container runtime software originally developed by Greg Kurtzer while at Lawrence Berkley National labs. It was developed with security, scientific software, and HPC systems in mind. 49 | 50 | **philosophy** 51 | 52 | Singularity assumes ([more or less](http://containers-ftw.org/SCI-F/)) that each application will have its own container. It does not seek to fully isolate containers from one another or the host system. 53 | Singularity assumes that you will have a build system where you are the root user, but that you will also have a production system where you may or may not be the root user. 54 | 55 | **strengths** 56 | - Easy to learn and use (relatively speaking) 57 | - Approved for HPC (installed on some of the biggest HPC systems in the world) 58 | - Can convert Docker containers to Singularity and run containers directly from Docker Hub 59 | - Sylabs container Library [Sylabs Library](https://cloud.sylabs.io/library) 60 | - [Singularity Hub](https://singularity-hub.org/)! 61 | - A place to build and host your containers similar to Docker Hub 62 | 63 | Singularity shines for scientific software running in an HPC environment. We will use it for the remainder of the class. 64 | 65 | ## Tutorial steps 66 | 67 | - [Step 1 - Install](INSTALL.md) 68 | - [Step 2 - Build and Run](BUILD_RUN.md) 69 | - [Step 3 - Cloud services](CLOUD_SERVICES.md) 70 | - [Step 4 - Instances](INSTANCES.md) 71 | - [Step 5 - GPU](GPU.md) 72 | 73 | Want to learn more: 74 | - [Miscellaneous](MISCELLANEOUS.md) 75 | - [Join the community](https://www.sylabs.io/singularity/join-the-community/) 76 | -------------------------------------------------------------------------------- /ADVANCED_USAGE.md: -------------------------------------------------------------------------------- 1 | ## Advanced Singularity usage 2 | 3 | ### Making containerized apps behave more like normal apps 4 | 5 | In the third hour we are going to consider an extended example describing a containerized application that takes a file as input, analyzes the data in the file, and produces another file as output. This is obviously a very common situation. 6 | 7 | Let's imagine that we want to use the cowsay program in our `lolcow.sif` to "analyze data". We should give our container an input file, it should reformat it (in the form of a cow speaking), and it should dump the output into another file. 8 | 9 | Here's an example. First I'll make some "data" 10 | 11 | ``` 12 | $ echo "The grass is always greener over the septic tank" > input 13 | ``` 14 | 15 | Now I'll "analyze" the "data" 16 | 17 | ``` 18 | $ cat input | singularity exec lolcow.sif cowsay > output 19 | ``` 20 | 21 | The "analyzed data" is saved in a file called `output`. 22 | 23 | ``` 24 | $ cat output 25 | ______________________________________ 26 | / The grass is always greener over the \ 27 | \ septic tank / 28 | -------------------------------------- 29 | \ ^__^ 30 | \ (oo)\_______ 31 | (__)\ )\/\ 32 | ||----w | 33 | || || 34 | ``` 35 | 36 | This _works..._ but the syntax is ugly and difficult to remember. 37 | 38 | Singularity supports a neat trick for making a container function as though it were an executable. We need to create a **runscript** inside the container. It turns out that our Singularity recipe file already contains a runscript. It causes our container to print a helpful message. 39 | 40 | ``` 41 | $ ./lolcow.sif 42 | This is what happens when you run the container... 43 | ``` 44 | 45 | Let's rewrite this runscript in the definition file and rebuild our container 46 | so that it does something more useful. 47 | 48 | ``` 49 | BootStrap: library 50 | From: ubuntu:latest 51 | 52 | %runscript 53 | #!/bin/bash 54 | if [ $# -ne 2 ]; then 55 | echo "Please provide an input and an output file." 56 | exit 1 57 | fi 58 | cat $1 | cowsay > $2 59 | 60 | %post 61 | echo "Hello from inside the container" 62 | sed -i 's/$/ universe/' /etc/apt/sources.list 63 | apt-get update 64 | apt-get -y install vim fortune cowsay lolcat 65 | 66 | %environment 67 | export PATH=/usr/games:$PATH 68 | export LC_ALL=C 69 | ``` 70 | 71 | Now we must rebuild out container to install the new runscript. 72 | 73 | ``` 74 | $ sudo singularity build --force lolcow.sif Singularity 75 | ``` 76 | 77 | Note the `--force` option which ensures our previous container is completely overwritten. 78 | 79 | After rebuilding our container, we can call the lolcow.sif as though it were an executable, and simply give it two arguments. One for input and one for output. 80 | 81 | ``` 82 | $ ./lolcow.sif 83 | Please provide an input and an output file. 84 | 85 | $ ./lolcow.sif input output2 86 | 87 | $ cat output2 88 | ______________________________________ 89 | / The grass is always greener over the \ 90 | \ septic tank / 91 | -------------------------------------- 92 | \ ^__^ 93 | \ (oo)\_______ 94 | (__)\ )\/\ 95 | ||----w | 96 | || || 97 | ``` 98 | 99 | ### Bind mounting host system directories into a container 100 | 101 | It's possible to create and modify files on the host system from within the container. In fact, that's exactly what we did in the previous example when we created output files in our home directory. 102 | 103 | Let's be more explicit. Consider this example. 104 | 105 | ``` 106 | $ singularity shell lolcow.sif 107 | 108 | Singularity lolcow.sif:~> echo wutini > ~/jawa.sez 109 | 110 | Singularity lolcow.sif:~> cat ~/jawa.sez 111 | wutini 112 | 113 | Singularity lolcow.sif:~> exit 114 | 115 | $ cat ~/jawa.sez 116 | wutini 117 | ``` 118 | 119 | Here we shelled into a container and created a file with some text in our home directory. Even after we exited the container, the file still existed. How did this work? 120 | 121 | There are several special directories that Singularity _bind mounts_ into 122 | your container by default. These include: 123 | 124 | - `/home/$USER` 125 | - `/tmp` 126 | - `/proc` 127 | - `/sys` 128 | - `/dev` 129 | 130 | You can specify other directories to bind using the `--bind` option or the environmental variable `$SINGULARITY_BINDPATH` 131 | 132 | Let's say we want to use our `cowsay.sif` container to "analyze data" and save results in a different directory. For this example, we first need to create a new directory with some data on our host system. 133 | 134 | ``` 135 | $ sudo mkdir /data 136 | 137 | $ sudo chown $USER:$USER /data 138 | 139 | $ echo 'I am your father' > /data/vader.sez 140 | ``` 141 | 142 | We also need a directory _within_ our container where we can bind mount the host system `/data` directory. We could create another directory in the `%post` section of our recipe file and rebuild the container, but our container already has a directory called `/mnt` that we can use for this example. 143 | 144 | Now let's see how bind mounts work. First, let's list the contents of `/mnt` within the container without bind mounting `/data` to it. 145 | 146 | ``` 147 | $ singularity exec lolcow.sif ls -l /mnt 148 | total 0 149 | ``` 150 | 151 | The `/mnt` directory within the container is empty. Now let's repeat the same command but using the `--bind` option to bind mount `/data` into the container. 152 | 153 | ``` 154 | $ singularity exec --bind /data:/mnt lolcow.sif ls -l /mnt 155 | total 4 156 | -rw-rw-r-- 1 ubuntu ubuntu 17 Jun 7 20:57 vader.sez 157 | ``` 158 | 159 | Now the `/mnt` directory in the container is bind mounted to the `/data` directory on the host system and we can see its contents. 160 | 161 | Now what about our earlier example in which we used a runscript to run a our container as though it were an executable? The `singularity run` command accepts the `--bind` option and can execute our runscript like so. 162 | 163 | ``` 164 | $ singularity run --bind /data:/mnt lolcow.sif /mnt/vader.sez /mnt/output3 165 | 166 | $ cat /data/output3 167 | __________________ 168 | < I am your father > 169 | ------------------ 170 | \ ^__^ 171 | \ (oo)\_______ 172 | (__)\ )\/\ 173 | ||----w | 174 | || || 175 | ``` 176 | 177 | But that's a cumbersome command. Instead, we could set the variable `$SINGULARITY_BINDPATH` and then use our container as before. 178 | 179 | ``` 180 | $ export SINGULARITY_BINDPATH=/data:/mnt 181 | 182 | $ ./lolcow.sif /mnt/output3 /mnt/metacow2 183 | 184 | $ ls -l /data/ 185 | total 12 186 | -rw-rw-r-- 1 ubuntu ubuntu 809 Jun 7 21:07 metacow2 187 | -rw-rw-r-- 1 ubuntu ubuntu 184 Jun 7 21:06 output3 188 | -rw-rw-r-- 1 ubuntu ubuntu 17 Jun 7 20:57 vader.sez 189 | 190 | $ cat /data/metacow2 191 | ________________________________________ 192 | / __________________ < I am your father \ 193 | | > | 194 | | | 195 | | ------------------ | 196 | | | 197 | | \ ^__^ | 198 | | | 199 | | \ (oo)\_______ | 200 | | | 201 | | (__)\ )\/\ | 202 | | | 203 | | ||----w | | 204 | | | 205 | \ || || / 206 | ---------------------------------------- 207 | \ ^__^ 208 | \ (oo)\_______ 209 | (__)\ )\/\ 210 | ||----w | 211 | || || 212 | ``` 213 | 214 | For a lot more info on how to bind mount host directories to your container, check out the [singularity documentation](https://www.sylabs.io/guides/3.0/user-guide/). 215 | -------------------------------------------------------------------------------- /CLOUD_SERVICES.md: -------------------------------------------------------------------------------- 1 | ## Sylabs cloud library 2 | 3 | In this sections we will build learn how to push our `lolcow.sif` image into the sylabs library, then to pull it from a different host, and how can we add security verification to those steps by signing our image with a PGP protocol. 4 | 5 | Also we will learn how to use the sylabs cloud builder to build our image without root privileges. 6 | 7 | ### The container library 8 | 9 | On 2018 Sylabs launched the Container Library, a comfortable home for SIF basded containers. Available as a cloud service, or for on-prem deployment, the Library will be available to manage, store and share containers. The cloud service portion will offer common Linux distributions, programming languages and AI frameworks, which will be updated regularly. A clear web interface and simple command-line syntax will let you search across containers and `singularity pull` them down to your system. 10 | 11 | but first we need to go to [Sylabs Cloud](https://cloud.sylabs.io/library) and create a login user, then create a token file and store it under `$HOME/.singularity/sylabs-token`, as default. (you can always store it in a prefered path and use the `-t/--tokenfile` flag to redirect to a new path, or set the token as an env var `SYLABS-TOKEN=A_VERY_LONG_TOKEN`) 12 | 13 | > push 14 | 15 | Now that we have an account in the container library, and the token file set, let's push our cow into the clouds! 16 | 17 | The Singularity push command allows you to upload your sif image to a library of your choosing. 18 | 19 | ``` 20 | eduardo@linux> singularity push lolcow.sif library://sylabsed/examples/lolcow:latest 21 | INFO: Now uploading lolcow.sif to the library 22 | 139.66 MiB / 139.66 MiB [=================================================================================================================================================================] 100.00% 2.75 MiB/s 50s 23 | INFO: Setting tag latest 24 | WARNING: latest replaces an existing tag 25 | ``` 26 | 27 | The warning is due the user `syalbsed` (aka me) already had a lolcow tagged as latest, so it must be replaced to the "new" image. 28 | 29 | > pull 30 | 31 | The 'pull' command allows you to download or build a container from a given URI. Supported URIs include: 32 | 33 | - library: Pull an image from the currently configured library 34 | 35 | library://[user[collection/[container[:tag]]]] 36 | 37 | - docker: Pull an image from Docker Hub 38 | 39 | docker://user/image:tag 40 | 41 | - shub: Pull an image from Singularity Hub to CWD 42 | 43 | shub://user/image:tag 44 | 45 | This tutorial focuses on the Sylabs Cloud library, so we will use the library URI to retrieve our image. 46 | 47 | My `lolcow` image is at https://cloud.sylabs.io/library/_container/5b9e91c694feb900016ea40b , you can go there to download the image from a UI environment or via CLI 48 | ``` 49 | # Pull with Singularity 50 | $ singularity pull library://sylabsed/examples/lolcow:latest 51 | # Pull by unique ID (reproducible even if tags change) 52 | $ singularity pull library://sylabsed/examples/lolcow:sha256.699eccab2e5c31043f540a9d5fbd3c8dc105e7355bbb7b855697aa223f5b71d0 53 | ``` 54 | **note** if you don't set a name `` the command `singularity pull` will by default set the image name based on the image name and tag name, in this case will be `lolcow_latest.sif` 55 | 56 | Now you know how to `push/pull` your SIF images! now let's make sure we are pulling the same image, our just add a security step to our workflow 57 | 58 | ### The remote builder 59 | 60 | So, what if I don't have root privileges on my host? 61 | 62 | For this concern, Sylabs has developed a Remote Build Service, first make sure you have a Sylabs cloud token - get one here. Save it to ~/.singularity/sylabs-token, and then build using the --remote flag: 63 | 64 | ``` 65 | eduardo@linux> singularity build --remote lolcow.sif lolcow.def 66 | searching for available build agent.........INFO: Starting build... 67 | Getting image source signatures 68 | Copying blob sha256:dca7be20e546564ad2c985dae3c8b0a259454f5637e98b59a3ca6509432ccd01 69 | 40.80 MiB / 40.80 MiB 1s 70 | Copying blob sha256:40bca54f5968c2bdb0d8516e6c2ca4d8f181326a06ff6efee8b4f5e1a36826b8 71 | 816 B / 816 B 0s 72 | Copying blob sha256:61464f23390e7d30cddfd10a22f27ae6f8f69cc4c1662af2c775f9d657266016 73 | 515 B / 515 B 0s 74 | Copying blob sha256:d99f0bcd5dc8b557254a1a18c6b78866b9bf460ab1bf2c73cc6aca210408dc67 75 | 854 B / 854 B 0s 76 | Copying blob sha256:120db6f90955814bab93a8ca1f19cbcad473fc22833f52f4d29d066135fd10b6 77 | 163 B / 163 B 0s 78 | Copying config sha256:473d4d9cf99523631d35a7645b7a5f276db55ec97da2f10ebf915e14b3c80552 79 | 2.62 KiB / 2.62 KiB 0s 80 | Writing manifest to image destination 81 | Storing signatures 82 | INFO: Creating SIF file... 83 | INFO: Build complete: /tmp/image-564730361 84 | INFO: Now uploading /tmp/image-564730361 to the library 85 | 36.42 MiB / 36.42 MiB 100.00% 37.88 MiB/s 0s 86 | INFO: Setting tag latest 87 | 36.42 MiB / 36.42 MiB [===================================================================================================================================================================] 100.00% 2.36 MiB/s 15s 88 | ``` 89 | 90 | Now you have you image, and is also stored in the cloud library, in case you need to re-used it for the future. 91 | 92 | ### Keystore 93 | 94 | The Sylabs [Keystore](https://cloud.sylabs.io/keystore) offers a way to easily search, visualize and share SIF signing keys, so that you can verify images downloaded from the Container Library, and allow others to verify images you create. 95 | 96 | Firsy we need to create a set of keys! 97 | 98 | The 'keys' command allows you to manage local OpenPGP key stores by creating a new store and new keys pairs. You can also list available keys from the default store. Finally, the keys command offers subcommands to communicate with an HKP key server to fetch and upload public keys. 99 | 100 | ``` 101 | eduardo@linux> singularity keys list 102 | Public key listing (/home/eduardo/.singularity/sypgp/pgp-public): 103 | 104 | ``` 105 | 106 | Here we can see that I currently don't have any keys on my host. 107 | 108 | ``` 109 | eduardo@linux> singularity keys newpair 110 | Enter your name (e.g., John Doe) : eduardo arango 111 | Enter your email address (e.g., john.doe@example.com) : eduardo@sylabs.io 112 | Enter optional comment (e.g., development keys) : oss all the things 113 | Generating Entity and OpenPGP Key Pair... Done 114 | Enter encryption passphrase : 115 | eduardo@linux> singularity keys list 116 | Public key listing (/home/eduardo/.singularity/sypgp/pgp-public): 117 | 118 | 0) U: eduardo arango (oss all the things) 119 | C: 2019-01-21 12:14:12 -0500 -05 120 | F: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 121 | L: 4096 122 | -------- 123 | ``` 124 | 125 | Now we have a keypair, now let's use this keys to sign our cow, so we can tell our cow from the herd, with the `singularity sign` command. 126 | 127 | The sign command allows a user to create a cryptographic signature on either a single data object or a list of data objects within the same SIF group. By default without parameters, the command searches for the primary partition and creates a verification block that is then added to the SIF container file. 128 | 129 | ``` 130 | eduardo@linux> singularity sign lolcow.sif 131 | Signing image: lolcow.sif 132 | Enter key passphrase: 133 | Signature created and applied to lolcow.sif 134 | ``` 135 | 136 | Now we are going to use the `verify` comand. The verify command allows a user to verify cryptographic signatures on SIF container files. There may be multiple signatures for data objects and multiple data objects signed. By default the command searches for the primary partition signature. If found, a list of all verification blocks applied on the primary partition is gathered so that data integrity (hashing) and signature verification is done for all those blocks. 137 | 138 | ``` 139 | eduardo@linux> singularity verify lolcow.sif 140 | Verifying image: lolcow.sif 141 | Data integrity checked, authentic and signed by: 142 | eduardo arango (oss all the things) , KeyID XXXXXXXXXXXXXXXX 143 | ``` 144 | 145 | That's my cow! 146 | 147 | But now... how can I verify my cow if I take it out from my herd, and take it to a local fair, I need to also bring my key pairs!, with `singularity keys push` , Upload an OpenPGP public key to a key server. 148 | 149 | ``` 150 | eduardo@linux> singularity keys push XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 151 | public key `XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' pushed to server successfully 152 | ``` 153 | 154 | Now I can go to the [Keystore](https://cloud.sylabs.io/keystore) and check that my key pair is there, and so I can pull it from a different host (with my Sylabs login credentials) and verify my image. 155 | 156 | 157 | ### Resources 158 | - [Library](https://cloud.sylabs.io/library) 159 | - [Remote builder](https://cloud.sylabs.io/builder) 160 | - [Keystore](https://cloud.sylabs.io/keystore) 161 | 162 | > Labnote 163 | - [Sylabs container library manage secure containers](https://www.sylabs.io/2018/05/sylabs-container-library-manage-secure-containers/) 164 | -------------------------------------------------------------------------------- /BUILD_RUN.md: -------------------------------------------------------------------------------- 1 | ## Building and Running Containers 2 | 3 | In the second hour we will build the preceding container from scratch. 4 | 5 | Simply typing `singularity` will give you an summary of all the commands you can use. Typing `singularity help ` will give you more detailed information about running an individual command. 6 | 7 | ### Building a basic container 8 | 9 | To build a singularity container, you must use the `build` command. The `build` command installs an OS, sets up your container's environment and installs the apps you need. To use the `build` command, we need a **recipe file** (also called a definition file). A Singularity recipe file is a set of instructions telling Singularity what software to install in the container. 10 | 11 | The Singularity source code contains several example definition files in the `/examples` subdirectory. Let's copy the ubuntu example to our home directory and inspect it. 12 | 13 | **Note:** You need to build containers on a file system where the sudo command can write files as root. This may not work in an HPC cluster setting if your home directory resides on a shared file server. If that's the case you may have to to `cd` to a local hard disk such as `/tmp`. 14 | 15 | ``` 16 | $ mkdir ../lolcow 17 | 18 | $ cp examples/ubuntu/Singularity ../lolcow/ 19 | 20 | $ cd ../lolcow 21 | 22 | $ vim Singularity 23 | ``` 24 | 25 | It should look something like this: 26 | 27 | ``` 28 | BootStrap: library 29 | From: ubuntu:16.04 30 | 31 | %runscript 32 | echo "This is what happens when you run the container..." 33 | 34 | %post 35 | echo "Hello from inside the container" 36 | apt-get -y install vim 37 | 38 | ``` 39 | 40 | See the [Singularity docs](https://www.sylabs.io/guides/3.0/user-guide/definition_files.html) for an explanation of each of these sections. 41 | 42 | Now let's use this recipe file as a starting point to build our `lolcow.img` container. Note that the build command requires `sudo` privileges, when used in combination with a recipe file. 43 | 44 | ``` 45 | $ sudo singularity build --sandbox lolcow Singularity 46 | ``` 47 | 48 | The `--sandbox` option in the command above tells Singularity that we want to build a special type of image (File system) for development purposes. 49 | 50 | Singularity can build containers in several different file formats. The default is to build a [SIF](https://github.com/sylabs/sif) image. SIF is an open source implementation of the Singularity Container Image Format that makes it easy to create complete and encapsulated container enviroments stored in a single file. 51 | 52 | But if you want to shell into a container and tinker with it (like we will do here), you should build a sandbox (which is really just a file system under a foler directory). This is great when you are still developing your container and don't yet know what should be included in the recipe file. 53 | 54 | When your build finishes, you will have a basic Ubuntu container saved in a local directory called `lolcow`. 55 | 56 | ### Using `shell` to explore and modify containers 57 | 58 | Now let's enter our new container and look around. 59 | 60 | ``` 61 | $ singularity shell lolcow 62 | ``` 63 | 64 | Depending on the environment on your host system you may see your prompt change. Let's look at what OS is running inside the container. 65 | 66 | ``` 67 | Singularity lolcow:~> cat /etc/os-release 68 | NAME="Ubuntu" 69 | VERSION="16.04.5 LTS (Xenial Xerus)" 70 | ID=ubuntu 71 | ID_LIKE=debian 72 | PRETTY_NAME="Ubuntu 16.04.5 LTS" 73 | VERSION_ID="16.04" 74 | HOME_URL="http://www.ubuntu.com/" 75 | SUPPORT_URL="http://help.ubuntu.com/" 76 | BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/" 77 | VERSION_CODENAME=xenial 78 | UBUNTU_CODENAME=xenial 79 | ``` 80 | 81 | No matter what OS is running on your host, your container is running Ubuntu 16.04! 82 | 83 | Let's try a few more commands: 84 | 85 | ``` 86 | Singularity lolcow:~> whoami 87 | eduardo 88 | 89 | Singularity lolcow:~> hostname 90 | eduardo-laptop 91 | ``` 92 | 93 | This is one of the core features of Singularity that makes it so attractive from a security standpoint. The user remains the same inside and outside of the container. 94 | 95 | Let's try installing some software. For building lolcow we first need the programs `fortune`, `cowsay`, and `lolcat` to produce the container that we saw in the first demo. 96 | 97 | ``` 98 | Singularity lolcow:~> sudo apt-get update && sudo apt-get -y install fortune cowsay lolcat 99 | bash: sudo: command not found 100 | ``` 101 | 102 | Whoops! 103 | 104 | Singularity complains that it can't find the `sudo` command. But even if you try to install `sudo` or change to root using `su`, you will find it impossible to elevate your privileges within the container. 105 | 106 | Once again, this is an important concept in Singularity. If you enter a container without root privileges, you are unable to obtain root privileges within the container. This insurance against privilege escalation is the reason that you will find Singularity installed in so many HPC environments. 107 | 108 | Let's exit the container and re-enter as root. 109 | 110 | ``` 111 | Singularity lolcow:~> exit 112 | 113 | $ sudo singularity shell --writable lolcow 114 | ``` 115 | 116 | Now we are the root user inside the container. Note also the addition of the `--writable` option. This option allows us to modify the container. The changes will actually be saved into the container and will persist across uses. 117 | 118 | Let's try installing some software again. 119 | 120 | ``` 121 | Singularity lolcow:~> apt-get update && apt-get -y install fortune cowsay lolcat 122 | ``` 123 | 124 | Now you should see the programs successfully installed. Let's try running the demo in this new container. 125 | 126 | ``` 127 | Singularity lolcow:~> fortune | cowsay | lolcat 128 | bash: lolcat: command not found 129 | bash: cowsay: command not found 130 | bash: fortune: command not found 131 | ``` 132 | 133 | Drat! It looks like the programs were not added to our `$PATH`. Let's add them and try again. 134 | 135 | ``` 136 | Singularity lolcow:~> export PATH=/usr/games:$PATH 137 | 138 | Singularity lolcow:~> fortune | cowsay | lolcat 139 | perl: warning: Setting locale failed. 140 | perl: warning: Please check that your locale settings: 141 | LANGUAGE = (unset), 142 | LC_ALL = (unset), 143 | LANG = "en_US.UTF-8" 144 | are supported and installed on your system. 145 | perl: warning: Falling back to the standard locale ("C"). 146 | ________________________________________ 147 | / Keep emotionally active. Cater to your \ 148 | \ favorite neurosis. / 149 | ---------------------------------------- 150 | \ ^__^ 151 | \ (oo)\_______ 152 | (__)\ )\/\ 153 | ||----w | 154 | || || 155 | ``` 156 | 157 | We're making progress, but we are now receiving a warning from perl. However, before we tackle that, let's think some more about the `$PATH` variable. 158 | 159 | We changed our path in this session, but those changes will disappear as soon as we exit the container just like they will when you exit any other shell. To make the changes permanent we should add them to the definition file and re-bootstrap the container. We'll do that in a minute. 160 | 161 | Now back to our perl warning. Perl is complaining that the locale is not set properly. Basically, perl wants to know where you are and what sort of language encoding it should use. Should you encounter this warning you can probably fix it with the `locale-gen` command or by setting `LC_ALL=C`. Here we'll just set the environment variable. 162 | 163 | ``` 164 | Singularity lolcow:~> export LC_ALL=C 165 | 166 | Singularity lolcow:~> fortune | cowsay | lolcat 167 | _________________________________________ 168 | / FORTUNE PROVIDES QUESTIONS FOR THE \ 169 | | GREAT ANSWERS: #19 A: To be or not to | 170 | \ be. Q: What is the square root of 4b^2? / 171 | ----------------------------------------- 172 | \ ^__^ 173 | \ (oo)\_______ 174 | (__)\ )\/\ 175 | ||----w | 176 | || || 177 | ``` 178 | 179 | Great! Things are working properly now. 180 | 181 | Although it is fine to shell into your Singularity container and make changes while you are debugging, you ultimately want all of these changes to be reflected in your recipe file. Otherwise if you need to reproduce it from scratch you will forget all of the changes you made. 182 | 183 | Let's update our definition file with the changes we made to this container. 184 | 185 | ``` 186 | Singularity lolcow:~> exit 187 | 188 | $ nano Singularity 189 | ``` 190 | 191 | Here is what our updated definition file should look like. 192 | 193 | ``` 194 | BootStrap: library 195 | From: ubuntu:16.04 196 | 197 | %post 198 | apt-get -y update 199 | apt-get -y install fortune cowsay lolcat 200 | 201 | %environment 202 | export LC_ALL=C 203 | export PATH=/usr/games:$PATH 204 | 205 | %runscript 206 | fortune | cowsay | lolcat 207 | ``` 208 | 209 | Let's rebuild the container with the new definition file. 210 | 211 | ``` 212 | $ sudo singularity build lolcow.sif Singularity 213 | ``` 214 | 215 | Note that we changed the name of the container. By omitting the `--sandbox` option, we are building our container in the standard Singularity squashfs file format. We are denoting the file format with the (optional) `.sif` extension (Singularity image format). A squashfs file is compressed and immutable making it a good choice for a production environment. 216 | 217 | Singularity stores a lot of [useful metadata](https://www.sylabs.io/guides/3.0/user-guide/environment_and_metadata.html). For instance, if you want to see the recipe file that was used to create the container you can use the `inspect` command like so: 218 | 219 | ``` 220 | $ singularity inspect --deffile lolcow.sif 221 | BootStrap: library 222 | From: ubuntu:16.04 223 | 224 | %post 225 | apt-get -y update 226 | apt-get -y install fortune cowsay lolcat 227 | 228 | %environment 229 | export LC_ALL=C 230 | export PATH=/usr/games:$PATH 231 | 232 | %runscript 233 | fortune | cowsay | lolcat 234 | ``` 235 | 236 | ### Blurring the line between the container and the host system. 237 | 238 | Singularity does not try to isolate your container completely from the host system. This allows you to do some interesting things. 239 | 240 | Using the exec command, we can run commands within the container from the host system. 241 | 242 | ``` 243 | $ singularity exec lolcow.sif cowsay 'How did you get out of the container?' 244 | _______________________________________ 245 | < How did you get out of the container? > 246 | --------------------------------------- 247 | \ ^__^ 248 | \ (oo)\_______ 249 | (__)\ )\/\ 250 | ||----w | 251 | || || 252 | ``` 253 | 254 | In this example, singularity entered the container, ran the `cowsay` command, displayed the standard output on our host system terminal, and then exited. 255 | 256 | You can also use pipes and redirection to blur the lines between the container and the host system. 257 | 258 | ``` 259 | $ singularity exec lolcow.sif cowsay moo > cowsaid 260 | 261 | $ cat cowsaid 262 | _____ 263 | < moo > 264 | ----- 265 | \ ^__^ 266 | \ (oo)\_______ 267 | (__)\ )\/\ 268 | ||----w | 269 | || || 270 | ``` 271 | 272 | We created a file called `cowsaid` in the current working directory with the output of a command that was executed within the container. 273 | 274 | We can also pipe things _into_ the container. 275 | 276 | ``` 277 | $ cat cowsaid | singularity exec lolcow.sif cowsay -n 278 | ______________________________ 279 | / _____ \ 280 | | < moo > | 281 | | ----- | 282 | | \ ^__^ | 283 | | \ (oo)\_______ | 284 | | (__)\ )\/\ | 285 | | ||----w | | 286 | \ || || / 287 | ------------------------------ 288 | \ ^__^ 289 | \ (oo)\_______ 290 | (__)\ )\/\ 291 | ||----w | 292 | || || 293 | ``` 294 | 295 | We've created a meta-cow (a cow that talks about cows). :stuck_out_tongue_winking_eye: 296 | --------------------------------------------------------------------------------