├── HDS-CDT_DL.md ├── README.md ├── buildMyGpuEnvironment.md ├── buildyourDependencies.md ├── clusterArchitecture.md ├── clusterRescomp.md ├── conda_essentials.md ├── docker_essentials.md ├── environment.yml ├── firstClusterUsage.md ├── githubTroubles.md ├── installations.md ├── putmyDataResComp.md ├── qsub2RescompScript.sh ├── screen.md ├── scriptForRunningParallelJobs.sh ├── sshToRemoteServer.md └── tensorboard.md /HDS-CDT_DL.md: -------------------------------------------------------------------------------- 1 | #### HDS-M05: Deep Learning 2 | 3 | - Login to your VM with GPU (*interactive for jupyter-notebook*) 4 | 5 | ```shell 6 | $ ssh -L 8888:127.0.0.1:8888 $user@bdicdtvm01.bmrc.ox.ac.uk 7 | ``` 8 | 9 | - Activate your virtual environment 10 | 11 | ```shell 12 | $ ml Anaconda3 13 | $ source activate base 14 | ``` 15 | 16 | - Copy ipython-Notebook downloaded to your home folder 17 | 18 | ```shell 19 | $ scp -r ~/Downloads/file.ipynb $user@bdicdtvm01.bmrc.ox.ac.uk:/cdthome/$user 20 | ``` 21 | 22 | - Start Jupyter-notebook where you have your ``.ipynb`` files 23 | 24 | ```shell 25 | $ jupyter-notebook 26 | ``` 27 | 28 | - Using **tensorboard** (make sure you are logged in interactive session) 29 | 30 | ```shell 31 | $ ssh -L 8889:127.0.0.1:8889 $user@bdicdtvm01.bmrc.ox.ac.uk 32 | $ ml Anaconda3 33 | $ source activate base 34 | $ tensorboard --logdir runs/ --port 8889 35 | $ http://127.0.0.1:8889/ 36 | ``` 37 | 38 | - Check GPU usage (e.g. memory, processing etc) 39 | 40 | ```shell 41 | $ ssh $user@bdicdtvm01.bmrc.ox.ac.uk 42 | $ watch nvidia-smi *comment: will continously monitor your session* 43 | ``` 44 | 45 | - Data for *segmentation task* (also provided on **Canvas**) 46 | 47 | ```shell 48 | /cdtshared/Cohort-2021/HDS-M05/segmentation/MontgomerySet 49 | ``` 50 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | **Lead contributors:** 2 | 3 | **Sharib Ali** , and 4 | 5 | **Avelino Javer** 6 | 7 | 8 | #### Basics on usages: What you will find? 9 | 10 | - [Conda basics and building virtual envs](conda_essentials.md) 11 | 12 | - [Docker deployment](docker_essentials.md) 13 | 14 | - [Cluster rescompX usage basic](clusterRescomp.md) 15 | 16 | - [Cluster rescompX gpu usage](buildMyGpuEnvironment.md) 17 | 18 | - [Buidling your own softwares and dependencies using cmake/gmake](buildyourDependencies.md) 19 | 20 | - [Manage your data and software builds for share](putmyDataResComp.md) 21 | 22 | - [Run my CPU script on cluster](https://github.com/sharibox/tutorial/blob/master/scriptForRunningParallelJobs.sh) 23 | 24 | - [Leaving your work place, but want to keep your cluster job via ssh keep running?, see here](https://github.com/sharibox/tutorial/blob/master/screen.md) 25 | 26 | - [About github, some general toubles!](https://github.com/sharibox/tutorial/blob/master/githubTroubles.md) 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | -------------------------------------------------------------------------------- /buildMyGpuEnvironment.md: -------------------------------------------------------------------------------- 1 | ## Getting ready: 2 | 3 | **The major requirements are:** 4 | 5 | * python 3 6 | * [pytorch](http://pytorch.org/) 7 | * tensorflow-gpu 8 | * CUDA (If you want to do any serious tests! But you can also run simple codes without CUDA/GPU) 9 | 10 | ### ssh to recomp1/2 11 | 12 | ```shell 13 | $ ssh username@rescomp1.well.ox.ac.uk 14 | ``` 15 | 16 | Make sure you are in rescomp home folder and nano .bashrc (put: `module use -a /mgmt/modules/eb/modules/all` and `module load Anaconda3/5.1.0`) so that it is linked to modules 17 | 18 | ### Creating virtual environment 19 | 20 | ```shell 21 | $ conda create -n aligpu 22 | $ source activate aligpu 23 | (aligpu) $ 24 | ``` 25 | 26 | ### Pytorch 27 | As explained [here](http://pytorch.org/). 28 | ```shell 29 | $ source activate aligpu 30 | (aligpu) $ conda install pytorch torchvision cuda80 -c soumith 31 | or 32 | (aligpu) $ conda install pytorch torchvision -c pytorch (default cuda 8.0, python3.5) 33 | or 34 | (aligpu) $ conda install pytorch torchvision cuda90 -c pytorch 35 | 36 | I did the last one! 37 | ``` 38 | 39 | ### Tensorflow and keras 40 | 41 | ```shell 42 | (aligpu) $conda install -c anaconda tensorflow-gpu 43 | 44 | ``` 45 | 46 | ```shell 47 | (aligpu) $ conda install -c anaconda keras-gpu 48 | ``` 49 | 50 | ### Visualization with tensorboard-pytorch 51 | 52 | [tensorboard](https://www.tensorflow.org/get_started/summaries_and_tensorboard) visualization, use [tensorboard-pytorch](https://github.com/lanpa/tensorboard-pytorch). 53 | 54 | ```shell 55 | $ source activate aligpu 56 | (aligpu) $ pip install tensorboardX 57 | (aligpu) $ pip install tensorflow 58 | ``` 59 | 60 | ### Extras (packages that can be a bit tricky to install and may not function) 61 | 62 | ```shell 63 | (aligpu) $ conda install -c anaconda qt 64 | (aligpu) $ conda install matplotlib (*you might have problem with this if qt latest version is not installed*) 65 | ``` 66 | 67 | ### Others 68 | 69 | For other packages just use 70 | 71 | ``` shell 72 | (aligpu) $ conda install $packagename 73 | (aligpu) $ pip install $packagename 74 | 75 | ``` 76 | ### Test 77 | 78 | - You have to go to the node where gpu is available 79 | 80 | ```shell 81 | $ qlogin -P rittscher.prjb -q gpu9.q -pe shmem 1 -l gpu=1 82 | $ source activate aligpu(your virtual env.) 83 | 84 | ``` 85 | 86 | - Check pytorch-gpu 87 | 88 | ```shell 89 | python -c "import torch; print('N GPU: {}'.format(torch.cuda.device_count()))" 90 | ``` 91 | For compG002, you should get: 92 | 93 | N GPU: 1 94 | 95 | - Check tensorflow-gpu 96 | 97 | ```shell 98 | $ python 99 | 100 | >> import tensorflow as tf 101 | >> print('version:',tf.__version__) 102 | >> from tensorflow.python.client import device_lib 103 | >> print(device_lib.list_local_devices()) 104 | 105 | ``` 106 | Same test as above but in one line (I recommend above as it will give you clean info.): 107 | 108 | ```shell 109 | python -c "import tensorflow as tf;print('version:',tf.__version__);from tensorflow.python.client import device_lib;print(device_lib.list_local_devices())" 110 | ``` 111 | 112 | For compG002, you should get: 113 | 114 | version: 1.7.0 115 | name: "/device:CPU:0"... (meaning 1 CPU) 116 | name: "/device:GPU:0... (meaning 1 GPU) 117 | 118 | 119 | -------------------------------------------------------------------------------- /buildyourDependencies.md: -------------------------------------------------------------------------------- 1 | ### Installed packages 2 | 3 | 4 | #### conda (installed with python 3.6, if conda missing) 5 | [link](https://conda.io/miniconda.html) 6 | 7 | bash Miniconda3-latest-MacOSX-x86_64.sh 8 | 9 | #### Essentials 10 | ``` 11 | cmake3.9.4 (conda install -c anaconda cmake) 12 | conda install -c anaconda *git* (also includes perl package 5.26) 13 | conda install -c menpo eigen (eigen 3.2.7) 14 | pip install numpy scipy matplotlib scikit-image scikit-learn ipython pandas 15 | ``` 16 | 17 | #### openCV 18 | [link](https://gist.github.com/jruizvar/0535fb8612afb105e0eef64051dc0d00) 19 | ``` 20 | git clone https://github.com/opencv/opencv.git 21 | git clone https://github.com/opencv/opencv_contrib.git 22 | ``` 23 | 24 | 25 | mkdir build 26 | cd build 27 | cmake -DPYTHON_DEFAULT_EXECUTABLE=python3 -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/usr/local -DBUILD_opencv_python2=False -DBUILD_opencv_python3=True -DPYTHON3_EXECUTABLE=python3 ../opencv-3.2.0 28 | make -j2 29 | sudo make install 30 | echo /usr/local/lib/python3.6/site-packages >> /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/opencv3.pth 31 | 32 | 33 | ###### Verify installation 34 | 35 | python3 -c "import cv2; print(cv2.__version__)" 36 | 37 | ###### For python2 support 38 | 39 | 40 | cmake .. -DBUILD_opencv_python2=True -DPYTHON2_LIBRARY=$CONDA_PREFIX/lib/libpython2.7.dylib -DPYTHON2_INCLUDE_DIR=$CONDA_PREFIX/include/python2.7 -DBUILD_opencv_python3=False \ 41 | \-DCMAKE_CONFIGURATION_TYPES=Release -DCMAKE_INSTALL_PREFIX=$CONDA_PREFIX -DMKL_INCLUDE_DIRS=/opt/intel/mkl/include\ 42 | -DMKL_ROOT_DIR=/opt/intel/mkl -DWITH_CUDA=OFF -DWITH_CUFFT=OFF -DWITH_MATLAB=OFF 43 | 44 | 45 | [FeatureMatchingTestSample](http://opencv-python-tutroals.readthedocs.io/en/latest/py_tutorials/py_feature2d/py_matcher/py_matcher.html) 46 | 47 | 48 | ###### Fixing problem with matplotlib 49 | 50 | - brew install python3 51 | - brew install freetype 52 | - brew install pkg-config 53 | - sudo pip3 install matplotlib 54 | 55 | 56 | #### ITK 57 | ``` 58 | git clone https://github.com/InsightSoftwareConsortium/ITK.git 59 | ``` 60 | 61 | Building with previous subversions:[link](https://itk.org/Wiki/ITK/Git/Download) 62 | 63 | Python wrapping [link](https://itk.org/Wiki/ITK/Python_Wrapping) 64 | 65 | `error: python library mismatch` 66 | 67 | `fix: -DPYTHON_EXECUTABLE=/usr/bin/python2.7` 68 | 69 | ITKPython Package ready [link](http://itkpythonpackage.readthedocs.io/en/latest/) 70 | 71 | 72 | 73 | #### brew install parallel 74 | 75 | 76 | #### Tensor flow with speed-up architechtures (SSE4.1 SSE4.2 AVX) 77 | 78 | - Follow this [link](https://github.com/lakshayg/tensorflow-build) and this [post](https://github.com/tensorflow/tensorflow/issues/8037) -------------------------------------------------------------------------------- /clusterArchitecture.md: -------------------------------------------------------------------------------- 1 | ## Cluster details at glance 2 | 3 | **CPU** 4 | 5 | Nodes | A | B | C | D | H | 6 | --- | --- | ---| ---| ---| --- 7 | *cores* | 288 | 640 | 1720 | 768 | 144 8 | *mem/node* | 32G | 96G | 256G | 384G | 2TB 9 | *mem/core* | 2.67G | 8G | 16G | 16G | 42.66G 10 | 11 | 12 | **GPU** 13 | 14 | Recommended to see: [here](https://rescomp.well.ox.ac.uk/wiki/Using_the_Rescomp_GPU_resources) <--- requires your credentials 15 | 16 | 17 | 18 | *Note: Check pdfs sent by Jon for more details* 19 | 20 | 21 | 22 | -------------------------------------------------------------------------------- /clusterRescomp.md: -------------------------------------------------------------------------------- 1 | ## ssh to clusterRescompX 2 | 3 | ```shell 4 | $ ssh username@rescomp1.well.ox.ac.uk 5 | ``` 6 | 7 | In OSX you need to install and open [XQuartz](https://www.xquartz.org/). 8 | 9 | Then loging to the server enabling X11 forwarding. 10 | 11 | ```shell 12 | $ ssh -X username@rescomp1.well.ox.ac.uk 13 | ``` 14 | 15 | Go ahead and open any GUI you need (e.g., $ gedit textfile). 16 | 17 | ## Load Anaconda 18 | 19 | Before executing of submiting a task you have to load the modules you will need: 20 | 21 | ```shell 22 | module use -a /mgmt/modules/eb/modules/all #Add the path where the Anacoda module is located. 23 | module load Anaconda3/5.1.0 24 | ``` 25 | `Recommended:` Put above two lines in your `.bashrc (nano ~/.bashrc)` 26 | 27 | ## Create your virtual environment 28 | 29 | Before installing any python package it is recommended to create an enviroment as: 30 | 31 | ```shell 32 | $ conda create --name myenv 33 | ``` 34 | 35 | You can create a new enviroment with all your required packages using a a .yml file like: 36 | 37 | ```shell 38 | $ conda create -f my_packages.yml 39 | ``` 40 | 41 | To activate the enviroment just type: 42 | 43 | ```shell 44 | $ source activate myenv 45 | ``` 46 | 47 | If you forgot any packages in your enviorment you can add new ones using: 48 | 49 | ```shell 50 | $ conda install package_name 51 | ``` 52 | 53 | More information about anaconda enviroments can be found [here](https://conda.io/docs/user-guide/tasks/manage-environments.html). 54 | 55 | 56 | ## Example of a script file to send to the cluster queue: 57 | 58 | ```shell 59 | #!/bin/bash 60 | 61 | #$ -P rittscher.prjb -q gpu9.q 62 | #$ -l gpu=1 63 | 64 | module use -a /mgmt/modules/eb/modules/all 65 | module load Anaconda3/5.1.0 66 | source activate pytorch90-env 67 | 68 | python -c "import torch; print('N GPU: {}'.format(torch.cuda.device_count()))" 69 | 70 | echo "Finished at :"`date` 71 | exit 0 72 | ``` 73 | 74 | ## Mount the group data directory using ssh (on mac): 75 | [Here](https://susanqq.github.io/jekyll/pixyll/2017/09/05/remotefiles/) are instructions. 76 | 77 | In brief install osxfuse and sshfs using homebrew. You can mount the files typing something like: 78 | 79 | ```shell 80 | $ sudo sshfs -o allow_other,defer_permissions username@rescomp1.well.ox.ac.uk:/well/rittscher/users/ /Volumes/rescomp1 81 | ``` 82 | 83 | ``note: for LINUX/Ubuntu defer_permissions has to be replaced with default_permissions`` 84 | 85 | ## Copy data to group data directory using ssh directly (linux/mac): 86 | 87 | ```shell 88 | $ scp -r fullPathofYourLocalDirectory username@rescomp1.well.ox.ac.uk:/well/rittscher/users/yourAccountName 89 | ``` 90 | 91 | or 92 | 93 | ```shell 94 | $ rsync -aP fullPathofYourLocalDirectory username@rescomp1.well.ox.ac.uk:/well/rittscher/users/yourAccountName 95 | ``` 96 | 97 | ## Start an interative session on a node: 98 | 99 | ```shell 100 | $ qlogin -P rittscher.prjb -q short.qb -pe shmem 1 (prj* -q short.q*, '*'--> a/b/c) 101 | ``` 102 | 103 | For GPU clusters you have to specify the number of gpu's or you will not be allowed to start the session. 104 | ```shell 105 | $ qlogin -P rittscher.prjb -q gpu8.q -pe shmem 1 -l gpu=1 106 | ``` 107 | 108 | ## Start a jupyter notebook in an interactive server: 109 | 110 | Start a remote session using tunneling: 111 | 112 | ```shell 113 | $ ssh -L 8080:localhost:8080 username@rescomp1.well.ox.ac.uk 114 | 115 | $ jupyter notebook --no-browser --port=8080 116 | ``` 117 | 118 | Ipython should print a link with an access token. You can then we can just copy 119 | and paste the link in your local browser and execute jupyter scripts. 120 | 121 | Sometimes the port might be in use. Then change the port and start again. 122 | 123 | You can also use jupyter on the GPU nodes by doing another tunneling between rescomp and your node. For example, to run a jupyter notebook on port 8888 of compG008 and accessible on port 8080 of rescomp, you would use the following tunneling: 124 | 125 | ```shell 126 | [LOCAL_COMPUTER]$ ssh -L 8080:localhost:8080 username@rescomp1.well.ox.ac.uk 127 | [RESCOMP_LOGIN_NODE]$ ssh -L 8080:localhost:8888 compG008 128 | [compG008]$ jupyter notebook --no-browser --port=8888 129 | ``` 130 | 131 | 132 | ## Run your script in cluster: 133 | 134 | [Here](https://github.com/BIMSBbioinfo/intro2UnixandSGE/blob/master/sun_grid_engine_for_beginners/how_to_submit_a_job_using_qsub.md) are instructions 135 | 136 | Few most commonly used: 137 | 138 | ```shell 139 | $ qsub myScript.sh 140 | 141 | $ qsub -l h_vmem=1G,h_rt=01:50:20 -N testCluster myScript.sh 142 | 143 | ``` 144 | 145 | To see the jobs in queue or all: 146 | 147 | ```shell 148 | $ qstat -u $USER 149 | 150 | $ qstat -s p/r 151 | 152 | $ qsum -u $USER (compact summary, use -h for help) 153 | 154 | ``` 155 | 156 | See jobs not in queue: 157 | 158 | ```shell 159 | $ qstat -f -ne 160 | 161 | ``` 162 | 163 | Kill your jobs: 164 | 165 | ```shell 166 | $ qdel $jobID 167 | 168 | $ qselect -u | xargs qdel 169 | 170 | ``` 171 | 172 | Check/monitor state of execution hosts: 173 | 174 | ```shell 175 | $ qhost -q 176 | 177 | $ qconf -sql 178 | 179 | $ qload -u $USER 180 | 181 | $ qload -nh -v 182 | 183 | ``` 184 | 185 | Assigning your job to a specific node (TODO: need to be checked!!): 186 | 187 | 188 | ```shell 189 | $ qsub -q gpu.q@compG002 -N testGPU myScript.sh (use qhost -q to check possibilities) 190 | 191 | $ qsub -q himem.qh@compH000 -N testHighMem myScript.sh 192 | 193 | ``` 194 | 195 | Check softwares on the server *before you install one*: 196 | 197 | ```shell 198 | $ /apps/well/ 199 | 200 | $ /mgmt/modules/eb/modules 201 | 202 | $ modules avail 203 | 204 | ``` 205 | Check GPU type and capacities: 206 | 207 | ```shell 208 | $ nvidia-smi 209 | $ print(torch.version.cuda) 210 | ``` 211 | 212 | -------------------------------------------------------------------------------- /conda_essentials.md: -------------------------------------------------------------------------------- 1 | ## Conda 2 | 3 | **Install Conda** 4 | 5 | 1. Download [link](https://conda.io/miniconda.html) 6 | 2. In terminal run: *bash* Miniconda3-latest-Linux-x86_64/macXXX.sh 7 | 8 | **Creating local virtual environment for softwares** 9 | 10 | conda create --name myenv 11 | 12 | conda create -n myenv python=3.4 13 | 14 | conda env create -f environment.yml 15 | 16 | 17 | **Activate** 18 | 19 | source activate myenv 20 | 21 | 22 | **Listing all envs** 23 | 24 | conda env list 25 | 26 | conda info --envs 27 | 28 | ```Example 29 | conda environments: 30 | myenv /home/username/miniconda/envs/myenv 31 | snowflakes /home/username/miniconda/envs/snowflakes 32 | bunnies /home/username/miniconda/envs/bunnies 33 | ``` 34 | 35 | **Single install** 36 | 37 | conda install -c conda-forge opencv=3.4.1 38 | 39 | **Sharing/export an environment** 40 | 41 | source activate myenv 42 | conda env export > environment.yml 43 | 44 | **Creating an environment file** 45 | 46 | name: stats2 47 | channels: 48 | - javascript/nodefaults 49 | dependencies: 50 | - python=3.4 # or 2.7 51 | - opencv=3.4.1 52 | - bokeh=0.9.2 53 | - numpy=1.9.* 54 | - nodejs=0.10.* 55 | - flask 56 | - pip: 57 | - Flask-Testing 58 | 59 | #Note: Identation is important 60 | 61 | **Removing environment** 62 | 63 | conda remove --name myenv --all 64 | conda env remove --name myenv 65 | 66 | #### Example with your CNN: includes Theano, keras and tensorflow [link](https://bitbucket.org/aliNepal/kerastiatf/src/211dc79df45f?at=master) 67 | 68 | name: kerasBackEndTFlow 69 | channels: 70 | - javascript 71 | dependencies: 72 | - pip 73 | - pip: 74 | - Theano 75 | - keras 76 | - https://storage.googleapis.com/tensorflow/mac/cpu/tensorflow-1.1.0-py2-none-any.whl 77 | - brew!!!: 78 | - 79 | 80 | ``` 81 | conda env create -f dependenciesInstall.yml 82 | ``` 83 | 84 | #### Alternatively, you can also install dependencies using pip 85 | 86 | pip install -r requirements.txt 87 | -------------------------------------------------------------------------------- /docker_essentials.md: -------------------------------------------------------------------------------- 1 | ## Docker 2 | 3 | **Install** 4 | 5 | 1. [Download](https://store.docker.com/search?type=edition&offering=community) 6 | 2. (Opt) Install visual studio code, bash-completion (can be installed with brew then link installing bash completion given in docker doc) 7 | 2. Dockers automated script to add repository and install all dependencies in Linux 8 | 9 | ```shell 10 | $ curl -sSL https://get.docker.com/ | sh 11 | $ sudo usermod -aG docker $UserName 12 | ``` 13 | 14 | 3. Install docker-machine, docker-compose 15 | 4. Install docker in visual studio code (in mac put the application/visual.../bin in your .bash_profile) 16 | 5. Do code . from the folder of docker user folder (i.e. place with your codes that you want to create an image file) 17 | 3. For windows 10 pro/ent you can use docker (go to 1, use powershell, use docker version) otherwise only Docker Toolbox available (see 5) 18 | 4. You can install posh-docker (see docker docs for details, this is just auto-completion for docker commands, extra tools: Cmder.net, download and extract to run (unblock before zipping)-->docker machine env default--> copy and paster last line) 19 | 5. Win7/8/10 home: Runs tiny linux VM in virtualBox via docker-machine (go to store and click on Dockertoolbox --> check all options) 20 | 6. You can shutdown VM and then change settings (like memory) or docker-machine ls (/start/stop etc...) 21 | 7. Make sure the code in C:\Users only (Bind Mounts) 22 | 2. docker run hello-world 23 | 24 | 25 | **Container and image** 26 | - Image: build in binaries that we want to run - 27 | - Container: is an instance of the image running as a process (can be many of the same image) 28 | - Docker Hub (hub.docker.com): default image registry (a remote image repository) 29 | - Docker is actually a process in host machine rather than a Virtual Machine 30 | 31 | *Example* 32 | 33 | Using image ``ngnix`` 34 | 35 | ```shell 36 | $ docker container run --publish 80:80 nginx 37 | ``` 38 | - once running, open your browser and type: localhost and enter 39 | - To run in background 40 | ```shell 41 | $ docker container run --publish 80:80 --detach --name $yourpreferredName nginx 42 | $ docker container logs $yourpreferredName` 43 | $ docker top $yourpreferredName 44 | ``` 45 | 46 | To stop: 47 | ```shell 48 | $ docker container stop first 3 nos of containerID (docker container ls -a should show exited) 49 | ``` 50 | 51 | To remove container: 52 | ```shell 53 | $ docker container rm -f 3digits 54 | ``` 55 | 56 | write your container file (**_Dockerfile_** [Link](/Users/shariba/Documents/deployment)) 57 | 58 | FROM ubuntu:14.04 59 | RUN apt-get update 60 | RUN apt-get install -y curl 61 | 62 | 63 | **build** 64 | ```shell 65 | $ docker build -t myfirstdocker:latest . 66 | ``` 67 | 68 | ---> Create image using this directory's Dockerfile 69 | 70 | See the list of containers 71 | 72 | ```shell 73 | $ docker container ls -a 74 | ``` 75 | 76 | **bind volumes** [link](https://docs.docker.com/storage/volumes/) 77 | 78 | $ docker run -d \ 79 | --name=nginxtest \ 80 | -v nginx-vol:/usr/share/nginx/html \ 81 | myFirstDocker:latest 82 | 83 | **run** 84 | 85 | ```shell 86 | $ docker run -p 4000:80 myfirstdocker:latest 87 | ``` 88 | 89 | --> Run "myfirstdocker:latest" mapping port 4000 to 80 90 | 91 | ```shell 92 | $ sudo docker run -ti --rm myfirstdocker:latest /bin/bash 93 | ``` 94 | 95 | **register and run** 96 | ` 97 | ```shell 98 | $ docker push username/repository:tag 99 | ``` 100 | 101 | - Upload tagged image to registry 102 | 103 | ```shell 104 | $ docker run username/repository:tag 105 | ``` 106 | 107 | - Run image from a registry 108 | 109 | **Docker push step-by-step** 110 | 111 | [see here](https://karlcode.owtelse.com/blog/2017/01/25/push-a-docker-image-to-personal-repository/) 112 | 113 | **remove/stop** 114 | 115 | ```shell 116 | $ docker container rm 117 | ``` 118 | 119 | ```shell 120 | $ docker image rm 121 | ``` 122 | 123 | - Removes all: 124 | 125 | ```shell 126 | $ docker container rm $(docker container ls -a -q) 127 | ``` 128 | 129 | ```shell 130 | $ docker image rm $(docker image ls -a -q) 131 | ``` 132 | 133 | - Forcefull: 134 | 135 | ```shell 136 | $ docker container rm -f $(docker container ls -a -q) 137 | ``` 138 | 139 | - stop 140 | 141 | ```shell 142 | $ docker stop $(docker ps -a -q) 143 | ``` 144 | 145 | ### GUI 146 | 147 | open -a XQuartz 148 | 149 | 150 | In the XQuartz preferences, go to the “Security” tab and make sure you’ve got “Allow connections from network clients” ticked: 151 | 152 | ip=$(ifconfig en0 | grep inet | awk '$1=="inet" {print $2}') 153 | xhost + $ip 154 | 155 | 156 | -e DISPLAY=$ip:0 -v /tmp/.X11-unix:/tmp/.X11-unix 157 | 158 | [plugins](http://www.open-terrain.org/index.php/Pong/August30th2016QMediaPlayerOnUbuntu16-04LTS) 159 | 160 | docker run -it -v /tmp/.X11-unix:/tmp/.X11-unix:rw -e DISPLAY=unix$DISPLAY --device /dev/dri --privileged ubuntu 161 | 162 | Then 163 | 164 | apt update && apt install --no-recommends -y nvidia-340 mesa-utils && glxgears 165 | 166 | 167 | ### Dockerhub 168 | 169 | ```shell 170 | $ docker tag firstimage YOUR_DOCKERHUB_NAME/firstimage 171 | 172 | ``` 173 | 174 | ```shell 175 | $ docker push YOUR_DOCKERHUB_NAME/firstimage 176 | ``` 177 | 178 | ```shell 179 | $ docker login 180 | username: xxx (e.g, sharibox) 181 | password: xxxxx 182 | ``` 183 | -------------------------------------------------------------------------------- /environment.yml: -------------------------------------------------------------------------------- 1 | name: kerasBackEndTFlow 2 | channels: 3 | - defaults 4 | dependencies: 5 | - pip 6 | - keras 7 | - numpy 8 | - scipy 9 | - matplotlib 10 | - pandas 11 | - scikit-learn 12 | - h5py 13 | - jupyter 14 | -------------------------------------------------------------------------------- /firstClusterUsage.md: -------------------------------------------------------------------------------- 1 | #### Getting started and basics on usages 2 | 3 | - Login 4 | 5 | ssh username@rescomp1.well.ox.ac.uk 6 | 7 | - Installations 8 | 9 | - create your own environment.yml(example [here](https://gitlab.com/sharibOx/tutorials/blob/master/environment.yml)) file depending upon what you want to install 10 | 11 | 12 | - Example to run keras with tensorFlow backend 13 | 14 | ``` 15 | module use -a /mgmt/modules/eb/modules/all #Add the path where the Anacoda module is located. 16 | module load Anaconda3/5.1.0 17 | module load cuda/9.0 18 | 19 | conda env create -f environment.yml 20 | 21 | source activate kerasBackEndTFlow 22 | 23 | ``` 24 | 25 | - To check all your installed packages 26 | 27 | ``` 28 | conda list (check if all packages that you want are installed correctly) 29 | pip list (if you have installed using pip) 30 | ``` 31 | 32 | - To test the gpu (will only show no error if GPU is available or see [here](https://www.tensorflow.org/programmers_guide/using_gpu)) 33 | 34 | ``` 35 | import tensorflow as tf 36 | print(tf.__version__) 37 | sess = tf.Session(config=tf.ConfigProto(log_device_placement=True)) 38 | 39 | or 40 | 41 | from tensorflow.python.client import device_lib 42 | print(device_lib.list_local_devices()) 43 | 44 | ``` 45 | 46 | If above prints you the name of the GPU then voila you are ready to go! 47 | 48 | 49 | - To use cluster and notebook (see [here](https://gitlab.com/sharibOx/tutorials/blob/master/clusterRescomp.md) ) 50 | 51 | About gpu see [here](https://rescomp.well.ox.ac.uk/wiki/Using_the_Rescomp_GPU_resources) <--- requires your credentials 52 | 53 | **Note if you go into different node then you will need to do activate your environment again (before that use or put in your .bashrc `module use -a /mgmt/modules/eb/modules/all` + `module load Anaconda3/5.1.0`)** 54 | 55 | - With torch for gpu 56 | 57 | torch.cuda.is_available() 58 | 59 | torch.cuda.init() 60 | 61 | torch.cuda.get_device_name(0) 62 | 63 | torch.cuda.device_count() 64 | 65 | -------------------------------------------------------------------------------- /githubTroubles.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | - cloning your private repo: 4 | 5 | 6 | git clone https://username@github.com/username/repository 7 | -------------------------------------------------------------------------------- /installations.md: -------------------------------------------------------------------------------- 1 | ## Installations 2 | 3 | ### Install qtlua 4 | 5 | luarocks install https://raw.githubusercontent.com/gluque/gluque.github.io/master/qtlua-scm-1.rockspec 6 | 7 | luarocks install qttorch 8 | -------------------------------------------------------------------------------- /putmyDataResComp.md: -------------------------------------------------------------------------------- 1 | #### Where to put your data? 2 | 3 | - Once you have ssh into the server, do below (this is private!): 4 | ```shell 5 | $ /well/rittscher/users/$yourName 6 | ``` 7 | 8 | (now from your local computer do below) 9 | ```shell 10 | $ scp -r fullPathofYourLocalDirectoryThatYouWant2Copy username@rescomp1.well.ox.ac.uk:/well/rittscher/users/yourAccountName 11 | ``` 12 | 'OR' 13 | ```shell 14 | $ rsync -aP fullPathofYourLocalDirectoryThatYouWant2Copy username@rescomp1.well.ox.ac.uk:/well/rittscher/users/yourAccountName 15 | ``` 16 | 17 | - Extra software builds can be done inside project (this can be accessed by everyone else) 18 | 19 | - You will find softwares that are built locally or using an environment 20 | 21 | - make sure you are in the proper environment to be able to use it 22 | 23 | - never build something which is already available on the server 24 | 25 | - To check available softwares: 26 | 27 | ```shell 28 | $ module avail 29 | ``` 30 | 31 | 32 | -------------------------------------------------------------------------------- /qsub2RescompScript.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | 4 | #$ -P rittscher.prjc -q gpu8.q -l gputype=p100 5 | #$ -l gpu=1 -pe shmem 2 6 | 7 | ## ssh compG005/compG006/compG007 8 | 9 | source /etc/profile.d/modules.sh 10 | module use -a /mgmt/modules/eb/modules/all 11 | module load Anaconda3/5.1.0 12 | module load gcc/5.4.0 13 | 14 | #source activate TFPytorchGPU 15 | module load cuda/9.0 16 | 17 | 18 | ROOT_FOLDER=/well/rittscher/projects/felix-Sharib/darknet 19 | 20 | # ./imageclassify detector train ${ROOT_FOLDER}/cfg/voc.data ${ROOT_FOLDER}/cfg/yolov3-voc.cfg ${ROOT_FOLDER}/darknet53.conv.74 -gpus 0,1,2,3 21 | 22 | from scratch... 23 | ${ROOT_FOLDER}/darknet detector train ${ROOT_FOLDER}/cfg/endo-anchor.data ${ROOT_FOLDER}/cfg/yolo-obj-anchor.cfg ${ROOT_FOLDER}/darknet53.conv.74 -gpus 0,1,2,3 24 | #${ROOT_FOLDER}/darknet detector train ${ROOT_FOLDER}/cfg/endo-anchor.data ${ROOT_FOLDER}/cfg/yolo-obj-anchor.cfg ${ROOT_FOLDER}/backup_anchor/yolo-obj-anchor.backup -gpus 1,2,3 25 | 26 | 27 | echo "Finished at :"`date` 28 | exit 0 29 | -------------------------------------------------------------------------------- /screen.md: -------------------------------------------------------------------------------- 1 | #### About Screen 2 | 3 | ``The screen program allows you to use multiple windows (virtual VT100 terminals) in Unix.`` 4 | 5 | See [here](https://kb.iu.edu/d/acuy) 6 | 7 | 8 | 1. To see the screen list 9 | 10 | screen -ls 11 | 12 | 2. To make a new screen 13 | 14 | screen -S $NAME 15 | 16 | 3. To detach 17 | 18 | crtl+A then d 19 | 20 | 4. To resume screen 21 | 22 | screen -R 23 | 24 | 25 | 5. To scroll 26 | 27 | ctrl + A then Esc and use the arrow keys to move up/down 28 | 29 | 6. To exit the "scrolling mode" 30 | 31 | q or Esc 32 | 33 | 7. To kill the screen 34 | 35 | ctrl + A (then 'K' to kill) 36 | 37 | *Note: you will have to be in the same terminal to be able to resume the screen.* 38 | 39 | 40 | -------------------------------------------------------------------------------- /scriptForRunningParallelJobs.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | #$ -P rittscher.prjc 4 | #$ -q long.qc 5 | #$ -t 7501-8000 6 | #$ -pe shmem 4 7 | 8 | ### -t 7501-8000 will take 500 separate cores and send each single task to different cores 9 | 10 | source /etc/profile.d/modules.sh 11 | module use -a /mgmt/modules/eb/modules/all 12 | module load Anaconda3/5.1.0 13 | 14 | ## this is my virtual environment with which my executable is built 15 | source activate tools 16 | 17 | # note your number of threads should be equal to no. of processes 18 | export OMP_NUM_THREADS=4 19 | 20 | export DISPLAY=:0.0 21 | 22 | execdir="/users/rittscher/sharib/base/aceColorEnhancement/scripts" 23 | cd $execdir 24 | printf -v sect "%06g" $SGE_TASK_ID 25 | 26 | EXE_FOLDER=/users/rittscher/sharib/base/aceColorEnhancement/build 27 | PARENTFOLDER=/well/rittscher/projects/dataset-NS/originalFiles 28 | SUB_DIR=47502_16_601169_RP_6M 29 | 30 | #_6N 31 | FILENAME='Tile'${sect}'.jpg' 32 | REUSLT_FOLDER="/well/rittscher/projects/dataset-NS/results/"${SUB_DIR} 33 | resultFileName=${REUSLT_FOLDER}/${FILENAME} 34 | ${EXE_FOLDER}/ace ${PARENTFOLDER}/${SUB_DIR}/${FILENAME} -a 8 -m interp:12 ${resultFileName} 35 | 36 | echo "me: $sect" 37 | -------------------------------------------------------------------------------- /sshToRemoteServer.md: -------------------------------------------------------------------------------- 1 | ## Adding your ssh key to remote server 2 | 3 | 1. ssh-keygen 4 | 5 | 2. To copy your generated keygen to remote server: 6 | 7 | ``ssh-copy-id -i ~/.ssh/mykey user@host`` 8 | 9 | 3. Test: ssh -i ~/.ssh/mykey user@host 10 | 11 | 12 | Details see here: 13 | 14 | ``https://www.digitalocean.com/community/tutorials/how-to-set-up-ssh-keys--2`` 15 | -------------------------------------------------------------------------------- /tensorboard.md: -------------------------------------------------------------------------------- 1 | ### Essentials only 2 | 3 | [1] connect 4 | 5 | <$> ssh -L 16007:127.0.0.1:6006 sharib@rescomp1.well.ox.ac.uk`` 6 | <$> source activate kerasTFCPU 7 | <$> tensorboard --logdir logs 8 | <$> Instead of: http://rescomp1:6007 use http://127.0.0.1:16007 (paste on your browser) 9 | 10 | [2] specific port 11 | 12 | <$> tensorboard --logdir logs --port 8080 13 | 14 | [2] Close the port 15 | 16 | <$> fuser 6006/tcp -k 17 | --------------------------------------------------------------------------------