FastQC Report![[OK]](Icons/tick.png)
Basic Statistics
222 | | Measure | 225 |Value | 226 |
|---|---|
| Filename | 229 |Mov10_oe_1.fastq | 230 |
| File type | 233 |Conventional base calls | 234 |
| Encoding | 237 |Sanger / Illumina 1.9 | 238 |
| Total Sequences | 241 |39971841 | 242 |
| Filtered Sequences | 245 |0 | 246 |
| Sequence length | 249 |100 | 250 |
| %GC | 253 |47 | 254 |
Per base sequence quality
258 | 
Per sequence quality scores
261 | 
![[FAIL]](Icons/error.png)
Per base sequence content
264 | 
Per base GC content
267 | 
Per sequence GC content
270 | 
Per base N content
273 | 
Sequence Length Distribution
276 | 
Sequence Duplication Levels
279 | 
Overrepresented sequences
282 | No overrepresented sequences
283 |![[WARN]](Icons/warning.png)
Kmer Content
285 | 
| Sequence | 289 |Count | 290 |Obs/Exp Overall | 291 |Obs/Exp Max | 292 |Max Obs/Exp Position | 293 |
|---|---|---|---|---|
| AAAAA | 296 |16795015 | 297 |4.1748657 | 298 |6.059911 | 299 |2 | 300 |
| CTGGG | 303 |8376590 | 304 |2.5479658 | 305 |6.4841547 | 306 |1 | 307 |
| TTCTT | 310 |12161990 | 311 |2.543816 | 312 |5.0711346 | 313 |6 | 314 |
| TCTTC | 317 |10938540 | 318 |2.529536 | 319 |5.185696 | 320 |7 | 321 |
| CTTCT | 324 |10885845 | 325 |2.5173504 | 326 |5.0255194 | 327 |1 | 328 |
| CTCCA | 331 |8804215 | 332 |2.377327 | 333 |8.132946 | 334 |1 | 335 |
| GGCAG | 338 |7360785 | 339 |2.3646495 | 340 |9.048611 | 341 |1 | 342 |
| TCCAG | 345 |8433860 | 346 |2.3336692 | 347 |5.7122536 | 348 |7 | 349 |
| CTCCT | 352 |8862290 | 353 |2.2658334 | 354 |6.7507205 | 355 |1 | 356 |
| CAGGA | 359 |7528755 | 360 |2.2545867 | 361 |5.9135337 | 362 |1 | 363 |
| CTTCA | 366 |9153500 | 367 |2.235553 | 368 |6.1804776 | 369 |1 | 370 |
| CCCAG | 373 |7216920 | 374 |2.207828 | 375 |6.119858 | 376 |1 | 377 |
| GCCAG | 380 |6455370 | 381 |2.023714 | 382 |6.2788043 | 383 |1 | 384 |
| CTGCA | 387 |7241750 | 388 |2.0038092 | 389 |5.1436768 | 390 |1 | 391 |
| CTTGG | 394 |6897085 | 395 |1.8517264 | 396 |5.505673 | 397 |1 | 398 |
| CTGGA | 401 |6511845 | 402 |1.8464246 | 403 |6.7122235 | 404 |1 | 405 |
| CTCAG | 408 |6449570 | 409 |1.7846115 | 410 |7.22948 | 411 |1 | 412 |
| CTTTT | 415 |8479045 | 416 |1.7734871 | 417 |5.9101095 | 418 |1 | 419 |
| TTTCA | 422 |7934210 | 423 |1.7526736 | 424 |5.187293 | 425 |6 | 426 |
| TTCAG | 429 |6830700 | 430 |1.7095337 | 431 |5.021524 | 432 |7 | 433 |
| CTTGA | 436 |5609765 | 437 |1.4039677 | 438 |5.2207584 | 439 |1 | 440 |
| CTCAT | 443 |4925100 | 444 |1.2028538 | 445 |5.1273108 | 446 |1 | 447 |
| CTCAA | 450 |4489260 | 451 |1.1579475 | 452 |5.334822 | 453 |1 | 454 |
25 |
26 | _Image aquired from the [Harvard Biomedical Data Management Website](https://datamanagement.hms.harvard.edu/hms-data-lifecycle)_
27 |
28 | We will cover some parts of this lifecycle by talking about best practices for the **Research** half of the above lifecycle. Later in this workshop we will talk a little more about the data storage. For more information about the full lifecycle and more guidelines for data management, please look at the resources linked below.
29 |
30 | **Resources**
31 |
32 | * The [HMS Data Management Working Group's website](https://datamanagement.hms.harvard.edu/)
33 | * A guide from the [Harvard library](http://guides.library.harvard.edu/dmp).
34 |
35 | ### Planning
36 |
37 | You should approach your sequencing project in a very similar way to how you do a biological experiment, and ideally, begins with **experimental design**. We're going to assume that you've already designed a beautiful sequencing experiment to address your biological question, collected appropriate samples, and that you have enough statistical power.
38 |
39 | During this stage it is important to keep track of how the experiment was performed and clearly tracking the source of starting materials and kits used. It is also best practice to include information about any small variations within the experiment or variation relative to standard experiments.
40 |
41 | ### Organization
42 |
43 | Every computational analysis you do is going to spawn many files, and inevitability you'll want to run some of those analyses again. For each experiment you work on and analyze data for, it is considered best practice to get organized by creating a planned storage space (directory structure).
44 |
45 | We will start by creating a directory that we can use for the rest of the RNA-seq session.
46 |
47 | First, make sure that you are in your home directory.
48 |
49 | ```bash
50 | $ cd
51 | $ pwd
52 | ```
53 |
54 | This should return `/home/username`.
55 |
56 | We will change into the `unix_lesson` directory:
57 |
58 | ```bash
59 | cd unix_lesson
60 | ```
61 |
62 | Next, we will create a project directory and set up the following structure within it to keep files organized.
63 |
64 | ```bash
65 | rnaseq
66 | ├── logs
67 | ├── meta
68 | ├── raw_data
69 | ├── results
70 | └── scripts
71 | ```
72 |
73 | *This is a generic structure and can be tweaked based on personal preference and analysis workflow.*
74 |
75 | - `logs`: to keep track of the commands run and the specific parameters used, but also to have a record of any standard output that is generated while running the command.
76 | - `meta`: for any information that describes the samples you are using, which we refer to as [metadata](https://datamanagement.hms.harvard.edu/metadata-overview). We will discuss this in more detail as it pertains to our example dataset, later in this lesson.
77 | - `raw_data`: for any **unmodified** (raw) data obtained prior to computational analysis here, e.g. FASTQ files from the sequencing center. We strongly recommend leaving this directory unmodified through the analysis.
78 | - `results`: for output from the different tools you implement in your workflow. Create sub-folders specific to each tool/step of the workflow within this folder.
79 | - `scripts`: for scripts that you write and use to run analyses/workflow.
80 |
81 | Here, you can use the parents flag (`-p` or `--parents`) with `mkdir` to complete the file path by creating any parent directories that do not exist. In our case, we have not yet created the `rnaseq` directory and so since it does not exist it will be created. This flag can be very useful when scripting workflows.
82 |
83 |
84 | ```bash
85 |
86 | $ mkdir -p rnaseq/logs rnaseq/meta rnaseq/raw_data rnaseq/results rnaseq/scripts
87 | ```
88 |
89 | Verify that the project directory and subdirectories now exist.
90 |
91 | ```bash
92 |
93 | $ cd rnaseq
94 | $ ls -l
95 |
96 | ```
97 |
98 | Let's populate the `rnaseq/` project with our example RNA-seq FASTQ data.
99 |
100 | The FASTQ files are located inside `~/unix_lesson/raw_fastq/`, and we need to copy them to `raw_data/`. We can match them by file extension with `*.fq`.
101 |
102 | ```bash
103 | $ cp ~/unix_lesson/raw_fastq/*.fq raw_data/
104 | ```
105 |
106 | Perfect, now the structure of `rnaseq/` should look like this:
107 |
108 | ```bash
109 | rnaseq
110 | ├── logs
111 | ├── meta
112 | ├── raw_data
113 | │ ├── Irrel_kd_1.subset.fq
114 | │ ├── Irrel_kd_2.subset.fq
115 | │ ├── Irrel_kd_3.subset.fq
116 | │ ├── Mov10_oe_1.subset.fq
117 | │ ├── Mov10_oe_2.subset.fq
118 | │ └── Mov10_oe_3.subset.fq
119 | ├── results
120 | └── scripts
121 | ```
122 |
123 | > #### File naming conventions
124 | >
125 | > Another aspect of staying organized is making sure that all the filenames in an analysis are as consistent as possible, and are not things like `alignment1.bam`, but more like `20170823_kd_rep1_gmap-1.4.bam`. [This link](https://datamanagement.hms.harvard.edu/file-naming-conventions) and [this slideshow](http://www2.stat.duke.edu/~rcs46/lectures_2015/01-markdown-git/slides/naming-slides/naming-slides.pdf) have some good guidelines for file naming dos and don'ts.
126 |
127 |
128 | ### Documentation
129 |
130 | **Documentation doesn't stop at the sequencer!** Keeping notes on what happened in what order, and what was done, is essential for reproducible research.
131 |
132 | #### Log files
133 |
134 | In your lab notebook, you likely keep track of the different reagents and kits used for a specific protocol. Similarly, recording information about the tools and parameters is important for documenting your computational experiments.
135 |
136 | - **Make note of the software you use.** Do your research and find out what tools are best for the data you are working with. Don't just work with tools that you are able to easily install.
137 | - **Keep track of software versions.** Keep up with the literature and make sure you are using the most up-to-date versions.
138 | - **Record information on parameters used and summary statistics** at every step (e.g., how many adapters were removed, how many reads did not align)
139 | - A general rule of thumb is to test on a single sample or a subset of the data before running your entire dataset through. This will allow you to debug quicker and give you a chance to also get a feel for the tool and the different parameters.
140 | - Different tools have different ways of reporting log messages and you might have to experiment a bit to figure out what output to capture. You can redirect standard output with the `>` symbol which is equivalent to `1> (standard out)`; other tools might require you to use `2>` to re-direct the `standard error` instead.
141 |
142 | #### README files
143 |
144 | After setting up the directory structure and when the analysis is running it is useful to have a **[README file](https://datamanagement.hms.harvard.edu/readme-files) within your project directory**. This file will usually contain a quick one line summary about the project and any other lines that follow will describe the files/directories found within it. An example README is shown below. Within each sub-directory you can also include README files to describe the analysis and the files that were generated.
145 |
146 | ```
147 | ## README ##
148 | ## This directory contains data generated during the Intro to RNA-seq course
149 | ## Date:
150 |
151 | There are six subdirectories in this directory:
152 |
153 | raw_data : contains raw data
154 | meta: contains...
155 | logs:
156 | results:
157 | scripts:
158 | ```
159 |
160 | ***
161 |
162 | ### Homework exercise
163 |
164 | - Take a moment to create a README for the `rnaseq/` folder (hint: use `vim` to create the file). Give a short description of the project and brief descriptions of the types of files you will be storing within each of the sub-directories.
165 |
166 | ***
167 |
168 | ## Exploring the example dataset
169 |
170 | The dataset we are using is part of a larger study described in [Kenny PJ et al., *Cell Rep* 2014](http://www.ncbi.nlm.nih.gov/pubmed/25464849). The authors are investigating interactions between various genes involved in Fragile X syndrome, a disease of aberrant protein production, which results in cognitive impairment and autistic-like features. **The authors sought to show that RNA helicase MOV10 regulates the translation of RNAs involved in Fragile X syndrome.**
171 |
172 | ### Raw data
173 |
174 | From this study we are using the [RNA-seq](http://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE50499) data which is publicly available in the [Sequence Read Archive (SRA)](https://www.ncbi.nlm.nih.gov/sra/?term=SRP029367).
175 |
176 | ### Metadata
177 |
178 | In addition to the raw sequence data we also need to collect **information about the data**, also known as **metadata**. We are usually quick to want to begin analysis of the sequence data (FASTQ files), but how useful is it if we know nothing about the samples that this sequence data originated from? Some relevant metadata for our dataset is provided below:
179 |
180 | * The RNA was extracted from **HEK293F cells** that were transfected with a **MOV10 transgene**, **MOV10 siRNA**, or an **irrelevant siRNA**. (*For this workshop we won't be using the MOV10 knock down samples.*)
181 | * The libraries for this dataset are **stranded** and were generated using the standard Tru-seq prep kit (using the dUTP method).
182 | * Sequencing was carried out on the **Illumina HiSeq-2500** and **100bp single end** reads were generated.
183 | * The full dataset was sequenced to **~40 million reads** per sample, but for this workshop we will be looking at a small subset on chr1 (~300,000 reads/sample).
184 | * For each group we have three replicates as described in the figure below.
185 |
186 | 
187 |
188 |
189 |
190 | ---
191 |
192 | *This lesson has been developed by members of the teaching team at the [Harvard Chan Bioinformatics Core (HBC)](http://bioinformatics.sph.harvard.edu/). These are open access materials distributed under the terms of the [Creative Commons Attribution license](https://creativecommons.org/licenses/by/4.0/) (CC BY 4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.*
193 |
194 | * *The materials used in this lesson were derived from work that is Copyright © Data Carpentry (http://datacarpentry.org/).
195 | All Data Carpentry instructional material is made available under the [Creative Commons Attribution license](https://creativecommons.org/licenses/by/4.0/) (CC BY 4.0).*
196 | * *Adapted from the lesson by Tracy Teal. Original contributors: Paul Wilson, Milad Fatenejad, Sasha Wood and Radhika Khetani for Software Carpentry (http://software-carpentry.org/)*
197 |
198 |
--------------------------------------------------------------------------------
/lessons/02_assessing_quality.md:
--------------------------------------------------------------------------------
1 | ---
2 | title: "Quality control using FASTQC"
3 | author: "Mary Piper, Radhika Khetani"
4 | date: Wednesday, September 20, 2017
5 | duration: 85 minutes
6 | ---
7 |
8 | ## Learning Objectives:
9 |
10 | * Evaluate the quality of your NGS data using FastQC
11 | * Create and run a job submission script to automate quality assessment
12 |
13 | ## Quality Control of FASTQ files
14 |
15 | 
16 |
17 | The first step in the RNA-Seq workflow is to take the FASTQ files received from the sequencing facility and assess the quality of the sequence reads.
18 |
19 | ### Unmapped read data (FASTQ)
20 |
21 | The [FASTQ](https://en.wikipedia.org/wiki/FASTQ_format) file format is the defacto file format for sequence reads generated from next-generation sequencing technologies. This file format evolved from FASTA in that it contains sequence data, but also contains quality information. Similar to FASTA, the FASTQ file begins with a header line. The difference is that the FASTQ header is denoted by a `@` character. For a single record (sequence read) there are four lines, each of which are described below:
22 |
23 | |Line|Description|
24 | |----|-----------|
25 | |1|Always begins with '@' and then information about the read|
26 | |2|The actual DNA sequence|
27 | |3|Always begins with a '+' and sometimes the same info in line 1|
28 | |4|Has a string of characters which represent the quality scores; must have same number of characters as line 2|
29 |
30 | Let's use the following read as an example:
31 |
32 | ```
33 | @HWI-ST330:304:H045HADXX:1:1101:1111:61397
34 | CACTTGTAAGGGCAGGCCCCCTTCACCCTCCCGCTCCTGGGGGANNNNNNNNNNANNNCGAGGCCCTGGGGTAGAGGGNNNNNNNNNNNNNNGATCTTGG
35 | +
36 | @?@DDDDDDHHH?GH:?FCBGGB@C?DBEGIIIIAEF;FCGGI#########################################################
37 | ```
38 |
39 | As mentioned previously, line 4 has characters encoding the quality of each nucleotide in the read. The legend below provides the mapping of quality scores (Phred-33) to the quality encoding characters. *Different quality encoding scales exist (differing by offset in the ASCII table), but note the most commonly used one is fastqsanger.*
40 |
41 | ```
42 | Quality encoding: !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHI
43 | | | | | |
44 | Quality score: 0........10........20........30........40
45 | ```
46 |
47 | Using the quality encoding character legend, the first nucelotide in the read (C) is called with a quality score of 31 and our Ns are called with a score of 2. **As you can tell by now, this is a bad read.**
48 |
49 | Each quality score represents the probability that the corresponding nucleotide call is incorrect. This quality score is logarithmically based and is calculated as:
50 |
51 | Q = -10 x log10(P), where P is the probability that a base call is erroneous
52 |
53 | These probabaility values are the results from the base calling algorithm and dependent on how much signal was captured for the base incorporation. The score values can be interpreted as follows:
54 |
55 | |Phred Quality Score |Probability of incorrect base call |Base call accuracy|
56 | |:-------------------|:---------------------------------:|-----------------:|
57 | |10 |1 in 10 | 90%|
58 | |20 |1 in 100| 99%|
59 | |30 |1 in 1000| 99.9%|
60 | |40 |1 in 10,000| 99.99%|
61 |
62 | Therefore, for the first nucleotide in the read (C), there is less than a 1 in 1000 chance that the base was called incorrectly. Whereas, for the the end of the read there is greater than 50% probabaility that the base is called incorrectly.
63 |
64 | ## Assessing quality with FastQC
65 |
66 | Now we understand what information is stored in a FASTQ file, the next step is to examine quality metrics for our data.
67 |
68 | [FastQC](http://www.bioinformatics.babraham.ac.uk/projects/fastqc/) provides a simple way to do some quality control checks on raw sequence data coming from high throughput sequencing pipelines. It provides a modular set of analyses which you can use to give a quick impression of whether your data has any problems of which you should be aware before doing any further analysis.
69 |
70 | The main functions of FastQC are:
71 |
72 | * Import of data from BAM, SAM or FastQ files (any variant)
73 | * Providing a quick overview to tell you in which areas there may be problems
74 | * Summary graphs and tables to quickly assess your data
75 | * Export of results to an HTML based permanent report
76 | * Offline operation to allow automated generation of reports without running the interactive application
77 |
78 | ### Run FastQC
79 |
80 | Before we run FastQC, let's start an interactive session on the cluster (if you don't already have one going):
81 |
82 | ```bash
83 | $ srun --pty -p short -t 0-12:00 --mem 8G --reservation=HBC /bin/bash
84 | ```
85 |
86 | ***An interactive session is very useful to test tools, workflows, run jobs that open new interactive windows (X11-forwarding) and so on.***
87 |
88 | Once your interactive job starts, notice that the command prompt has changed; this is because we are working on a compute node now, not on a login node. Change directories to `raw_data`.
89 |
90 | ```bash
91 | $ cd ~/unix_lesson/rnaseq/raw_data
92 | ```
93 |
94 | Before we start using software, we have to load the environments for each software package. On the O2 cluster, this is done using an **LMOD** system.
95 |
96 | If we check which modules we currently have loaded, we should not see FastQC.
97 |
98 | ```bash
99 | $ module list
100 | ```
101 |
102 | This is because the FastQC program is not in our $PATH (i.e. its not in a directory that unix will automatically check to run commands/programs).
103 |
104 | ```bash
105 | $ echo $PATH
106 | ```
107 |
108 | To run the FastQC program, we first need to load the appropriate module, so it puts the program into our path. To find the FastQC module to load we need to search the versions available:
109 |
110 | ```bash
111 | $ module spider
112 | ```
113 |
114 | Then we can load the FastQC module:
115 |
116 | ```bash
117 | $ module load fastqc/0.11.3
118 | ```
119 |
120 | Once a module for a tool is loaded, you have essentially made it directly available to you like any other basic UNIX command.
121 |
122 | ```bash
123 | $ module list
124 |
125 | $ echo $PATH
126 | ```
127 |
128 | FastQC will accept multiple file names as input, so we can use the `*.fq` wildcard.
129 |
130 | ```bash
131 | $ fastqc *.fq
132 | ```
133 |
134 | *Did you notice how each file was processed serially? How do we speed this up?*
135 |
136 | Exit the interactive session and start a new one with 6 cores, and use the multi-threading functionality of FastQC to run 6 jobs at once.
137 |
138 | ```bash
139 | $ exit #exit the current interactive session
140 |
141 | $ srun --pty -c 6 -p short -t 0-12:00 --mem 8G --reservation=HBC /bin/bash #start a new one with 6 cpus (-n 6) and 8G RAM (--mem 8G)
142 |
143 | $ module load fastqc/0.11.3 #reload the module for the new session
144 |
145 | $ cd ~/unix_lesson/rnaseq/raw_data
146 |
147 | $ fastqc -t 6 *.fq #note the extra parameter we specified for 6 threads
148 | ```
149 |
150 | How did I know about the -t argument for FastQC?
151 |
152 | ```bash
153 | $ fastqc --help
154 | ```
155 |
156 |
157 | Now, let's create a home for our results
158 |
159 | ```bash
160 | $ mkdir ~/unix_lesson/rnaseq/results/fastqc
161 | ```
162 |
163 | ...and move them there (recall, we are still in `~/unix_lesson/rnaseq/raw_data/`)
164 |
165 | ```bash
166 | $ mv *fastqc* ~/unix_lesson/rnaseq/results/fastqc/
167 | ```
168 |
169 | ### Performing quality assessment using job submission scripts
170 | So far in our FASTQC analysis, we have been directly submitting commands to O2 using an interactive session (ie. `srun --pty -n 6 -p short -t 0-12:00 --mem 8G bash`). However, there are many more partitions available on O2 than just the interactive partition. We can submit commands or series of commands to these partitions using job submission scripts.
171 |
172 | **Job submission scripts** for O2 are just regular scripts, but contain the O2 **options/directives** for job submission, such as *number of cores, name of partition, runtime limit, etc*. We can submit these scripts to whichever partition we specify in the script using the `sbatch` command as follows:
173 |
174 | ```bash
175 | # DO NOT RUN THIS
176 | $ sbatch job_submission_script.run
177 | ```
178 |
179 | Submission of the script using the `sbatch` command allows SLURM to run your job when its your turn. Let's create a job submission script to load the FASTQC module, run FASTQC on all of our fastq files, and move the files to the appropriate directory.
180 |
181 | Change directories to `~/unix_lesson/rnaseq/scripts`, and create a script named `mov10_fastqc.run` using `vim`.
182 |
183 | ```bash
184 | $ cd ~/unix_lesson/rnaseq/scripts
185 |
186 | $ vim mov10_fastqc.run
187 | ```
188 |
189 | The first thing we need in our script is the **shebang line**:
190 |
191 | ```bash
192 | #!/bin/bash
193 | ```
194 |
195 | Following the shebang line are the O2 options. For the script to run, we need to include options for **queue/partition (-p) and runtime limit (-t)**. To specify our options, we precede the option with `#SBATCH`, which tells O2 that the line contains options for job submission to SLURM.
196 |
197 | ```bash
198 | #SBATCH -p short # partition name
199 | #SBATCH -t 0-2:00 # hours:minutes runlimit after which job will be killed
200 | #SBATCH -c 6 # number of cores requested -- this needs to be greater than or equal to the number of cores you plan to use to run your job
201 | #SBATCH --job-name rnaseq_mov10_fastqc # Job name
202 | #SBATCH -o %j.out # File to which standard out will be written
203 | #SBATCH -e %j.err # File to which standard err will be written
204 | ```
205 | Now in the body of the script, we can include any commands we want run:
206 |
207 | ```bash
208 | ## Changing directories to where the fastq files are located
209 | cd ~/unix_lesson/rnaseq/raw_data
210 |
211 | ## Loading modules required for script commands
212 | module load fastqc/0.11.3
213 |
214 | ## Running FASTQC
215 | fastqc -t 6 *.fq
216 |
217 | ## Moving files to our results directory
218 | mv *fastqc* ../results/fastqc/
219 | ```
220 |
221 | Save and quit the script. Now, let's submit the job to the SLURM:
222 |
223 | ```bash
224 | $ sbatch mov10_fastqc.run
225 | ```
226 |
227 | You can check on the status of your job with:
228 |
229 | ```bash
230 | $ sacct
231 | ```
232 |
233 | ```bash
234 | $ ls -lh ../results/fastqc/
235 | ```
236 | There should also be standard error (`.err`) and standard out (`.out`) files from the job listed in `~/unix_lesson/rnaseq/scripts`. You can move these over to your `logs` directory and give them more intuitive names:
237 |
238 | ```bash
239 | $ mv *.err ../logs/fastqc.err
240 | $ mv *.out ../logs/fastqc.out
241 | ```
242 |
243 | ***
244 | **Exercise**
245 |
246 | How would you change the `mov10_fastqc.run` script if you had 9 fastq files you wanted to run in parallel.
247 |
248 | ***
249 |
250 | ### FastQC Results
251 |
252 | Let's take a closer look at the files generated by FastQC:
253 |
254 | `$ ls -lh ~/unix_lesson/rnaseq/results/fastqc/`
255 |
256 | #### HTML reports
257 | The .html files contain the final reports generated by fastqc, let's take a closer look at them. Transfer the file for `Mov10_oe_1.subset.fq` over to your laptop via *FileZilla*.
258 |
259 | ##### Filezilla - Step 1
260 |
261 | Open *FileZilla*, and click on the File tab. Choose 'Site Manager'.
262 |
263 | 
264 |
265 | ##### Filezilla - Step 2
266 |
267 | Within the 'Site Manager' window, do the following:
268 |
269 | 1. Click on 'New Site', and name it something intuitive (e.g. O2)
270 | 2. Host: transfer.rc.hms.harvard.edu
271 | 3. Protocol: SFTP - SSH File Transfer Protocol
272 | 4. Logon Type: Normal
273 | 5. User: ECommons ID
274 | 6. Password: ECommons password
275 | 7. Click 'Connect'
276 |
277 | 
278 |
279 | ***FastQC is just an indicator of what's going on with your data, don't take the "PASS"es and "FAIL"s too seriously.***
280 |
281 | FastQC has a really well documented [manual page](http://www.bioinformatics.babraham.ac.uk/projects/fastqc/) with [more details](http://www.bioinformatics.babraham.ac.uk/projects/fastqc/Help/) about all the plots in the report. We recommend looking at [this post](http://bioinfo-core.org/index.php/9th_Discussion-28_October_2010) for more information on what bad plots look like and what they mean for your data.
282 |
283 | > **We also have a [slidedeck](https://github.com/hbctraining/Intro-to-rnaseq-hpc-O2/raw/master/lectures/error_profiles_mm.pdf) of error profiles for Illumina sequencing, where we discuss specific FASTQC plots and possible sources of these types of errors.**
284 |
285 | Below are two of the most important analysis modules in FastQC, the **"Per base sequence quality"** plot and the **"Overrepresented sequences"** table.
286 |
287 | The **"Per base sequence quality"** plot provides the distribution of quality scores across all bases at each position in the reads.
288 |
289 | 
290 |
291 | The **"Overrepresented sequences"** table displays the sequences (at least 20 bp) that occur in more than 0.1% of the total number of sequences. This table aids in identifying contamination, such as vector or adapter sequences.
292 |
293 | 
294 |
295 | We will go over the remaining plots in class. Remember, our report only represents a subset of reads (chromosome 1) for `Mov10_oe_1.subset.fq`, which can skew the QC results. We encourage you to look at the [full set of reads](../fastqc/Mov10oe_1-fastqc_report.html) and note how the QC results differ when using the entire dataset.
296 |
297 | > **_NOTE:_**
298 | >The other output of FastQC is a .zip file. These .zip files need to be unpacked with the `unzip` program. If we try to `unzip` them all at once:
299 | >
300 | >```bash
301 | >$ cd ~/unix_lesson/rnaseq/results/fastqc/
302 | >$ unzip *.zip
303 | >```
304 | >
305 | >Did it work?
306 | >
307 | >No, because `unzip` expects to get only one zip file. Welcome to the real world.
308 | >We *could* do each file, one by one, but what if we have 500 files? There is a smarter way.
309 | >We can save time by using a simple shell `for loop` to iterate through the list of files in *.zip.
310 | >
311 | >After you type the first line, you will get a special '>' prompt to type next lines.
312 | >You start with 'do', then enter your commands, then end with 'done' to execute the loop.
313 | >
314 | >Note that in the first line, we create a variable named `zip`. After that, we call that variable with the syntax `$zip`. `$zip` is assigned the value of each item (file) in the list *.zip, once for each iteration of the loop.
315 | >
316 | >This loop is basically a simple program. When it runs
317 | >
318 | >```bash
319 | >$ for zip in *.zip
320 | > do
321 | > unzip $zip
322 | > done
323 | >```
324 | >This will run unzip once for each file (whose name is stored in the $zip variable). The contents of each file will be unpacked into a separate directory by the unzip program.
325 | >
326 | >The 'for loop' is interpreted as a multipart command. If you press the up arrow on your keyboard to recall the command, it will be shown like so:
327 | >
328 | >```bash
329 | >for zip in *.zip; do unzip $zip; done
330 | >```
331 | >
332 | >When you check your history later, it will help you remember what you did!
333 | >
334 | >What information is contained in the unzipped folder?
335 | >
336 | >```bash
337 | >$ ls -lh Mov10_oe_1.subset_fastqc
338 | >$ head Mov10_oe_1.subset_fastqc/summary.txt
339 | >```
340 | >
341 | >To save a record, let's `cat` all `fastqc summary.txt` files into one `full_report.txt` and move this to `~/unix_lesson/rnaseq/docs`.
342 | >You can use wildcards in paths as well as file names. Do you remember how we said `cat` is really meant for concatenating text files?
343 | >
344 | >```bash
345 | >$ cat */summary.txt > ~/unix_lesson/rnaseq/logs/fastqc_summaries.txt
346 | >```
347 |
348 | ## Quality Control (*Optional*) - Trimming
349 |
350 | We want to make sure that as many reads as possible map or align accurately to the genome. To ensure accuracy, only a small number of mismatches between the read sequence and the genome sequence are allowed, and any read with more than a few mismatches will be marked as being unaligned.
351 |
352 | Therefore, to make sure that all the reads in the dataset have a chance to map/align to the genome, unwanted information can be trimmed off from every read, one read at a time. The types of unwanted information can include one or more of the following:
353 | - leftover adapter sequences
354 | - known contaminants (strings of As/Ts, other sequences)
355 | - poor quality bases at read ends
356 |
357 | **We will not be performing this step** because:
358 | * our data does not have an appreciable amount of leftover adapter sequences or other contaminating sequences based on FastQC.
359 | * the alignment tool we have picked (STAR) is able to account for low-quality bases at the ends of reads when matching them to the genome.
360 |
361 | If you need to perform trimming on your fastq data to remove unwanted sequences/bases, the recommended tool is [cutadapt](http://cutadapt.readthedocs.io/en/stable/index.html).
362 |
363 | Example of cutadapt usage:
364 |
365 | ```bash
366 | $ cutadapt --adapter=AGATCGGAAGAG --minimum-length=25 -o myfile_trimmed.fastq.gz myfile.fastq.gz
367 | ```
368 |
369 | After trimming, cutadapt can remove any reads that are too short to ensure that you do not get spurious mapping of very short sequences to multiple locations on the genome. In addition to adapter trimming, cutadapt can trim off any low-quality bases too, but **please note that quality-based trimming is not considered best practice, since majority of the newer, recommended alignment tools can account for this.**
370 |
371 | ---
372 | *This lesson has been developed by members of the teaching team at the [Harvard Chan Bioinformatics Core (HBC)](http://bioinformatics.sph.harvard.edu/). These are open access materials distributed under the terms of the [Creative Commons Attribution license](https://creativecommons.org/licenses/by/4.0/) (CC BY 4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.*
373 |
374 | * *The materials used in this lesson was derived from work that is Copyright © Data Carpentry (http://datacarpentry.org/).
375 | All Data Carpentry instructional material is made available under the [Creative Commons Attribution license](https://creativecommons.org/licenses/by/4.0/) (CC BY 4.0).*
376 |
--------------------------------------------------------------------------------
/lessons/03_alignment.md:
--------------------------------------------------------------------------------
1 | ---
2 | title: "Alignment with STAR"
3 | author: "Meeta Mistry, Bob Freeman, Mary Piper"
4 | date: Wednesday, June 7, 2017
5 | ---
6 |
7 | Approximate time: 90 minutes
8 |
9 | ## Learning Objectives:
10 |
11 | * Understanding the alignment method STAR utilizes to align sequence reads to the reference genome
12 | * Identifying the intricacies of alignment tools used in NGS analysis (parameters, usage, etc)
13 | * Choosing appropriate STAR alignment parameters for our dataset
14 |
15 | ## Read Alignment
16 |
17 | 
17 |
18 | Once we have our reads aligned to the genome, the next step is to count how many reads have mapped to each gene. There are many tools that can use BAM files as input and output the number of reads (counts) associated with each feature of interest (genes, exons, transcripts, etc.). 2 commonly used counting tools are [featureCounts](http://bioinf.wehi.edu.au/featureCounts/) and [htseq-count](https://htseq.readthedocs.io/en/release_0.10.0/).
19 |
20 | * The above tools only report the "raw" counts of reads that **map to a single location** (uniquely mapping) and are best at counting at the **gene level**. Essentially, total read count associated with a gene (*meta-feature*) = the sum of reads associated with each of the exons (*feature*) that "belong" to that gene.
21 |
22 | * There are **other tools** available that are able to account for **multiple transcripts** for a given gene. In this case the counts are not whole numbers, but have fractions. In the simplest example case, if 1 read is associated with 2 transcripts, it can get counted as 0.5 and 0.5 and the resulting count for that transcript is not a whole number.
23 |
24 | * In addition there are **other tools that will count multimapping reads**, but this is a dangerous thing to do since you will be overcounting the total number of reads which can cause issues with normalization and eventually with accuracy of differential gene expression results.
25 |
26 | **Input for counting = multiple BAM files + 1 GTF file**
27 |
28 | Simply speaking, the genomic coordinates of where the read is mapped (BAM) are cross-referenced with the genomic coordinates of whichever feature you are interested in counting expression of (GTF), it can be exons, genes or transcripts.
29 |
30 | 
31 |
32 | **Output of counting = A count matrix, with genes as rows and samples are columns**
33 |
34 | These are the "raw" counts and will be used in statistical programs downstream for differential gene expression.
35 |
36 | 
37 |
38 | ### Counting using featureCounts
39 | Today, we will be using the [featureCounts](http://bioinf.wehi.edu.au/featureCounts/) tool to get the *gene* counts. We picked this tool because it is accurate, fast and is relatively easy to use. It counts reads that map to a single location (uniquely mapping) and follows the scheme in the figure below for assigning reads to a gene/exon.
40 |
41 | 
42 |
43 | featureCounts can also take into account whether your data are **stranded** or not. If strandedness is specified, then in addition to considering the genomic coordinates it will also take the strand into account for counting. If your data are stranded always specify it.
44 |
45 | #### Setting up to run featureCounts
46 | First things first, start an interactive session with 4 cores:
47 |
48 | ``` bash
49 | $ srun --pty -p short -t 0-12:00 -c 4 --mem 8G --reservation=HBC /bin/bash
50 | ```
51 |
52 | Now, change directories to your rnaseq directory and start by creating 2 directories, (1) a directory for the output and (2) a directory for the bam files:
53 |
54 | ``` bash
55 | $ cd ~/unix_lesson/rnaseq/
56 | $ mkdir results/counts results/STAR/bams
57 | ```
58 |
59 | Rather than using the BAM file we generated in the last lesson, let's copy over all of the BAM files that we have already generated for you:
60 |
61 | ``` bash
62 |
63 | $ cp /n/groups/hbctraining/intro_rnaseq_hpc/bam_STAR38/*bam ~/unix_lesson/rnaseq/results/STAR/bams
64 | ```
65 | featureCounts is not available as a module on O2, but we have already added the path for it to our `$PATH` variable last time.
66 |
67 | ``` bash
68 | $ echo $PATH # You should see /n/app/bcbio/tools/bin/ among other paths
69 | ```
70 |
71 | > ** If you don't see `/n/app/bcbio/tools/bin/` in your `$PATH` variable, add the following `export` command to your `~/.bashrc` file using vim: `export PATH=/n/app/bcbio/tools/bin/:$PATH`.**
72 |
73 |
74 | #### Running featureCounts
75 |
76 | How do we use this tool, what is the command and what options/parameters are available to us?
77 |
78 | ``` bash
79 | $ featureCounts
80 | ```
81 |
82 | So, it looks like the usage is `featureCounts [options] -a 
284 |
285 | Alternatively, you could remove featureCounts from the original script, and run it after all the jobs finish generating BAM files.
286 |
287 | ---
288 | *This lesson has been developed by members of the teaching team at the [Harvard Chan Bioinformatics Core (HBC)](http://bioinformatics.sph.harvard.edu/). These are open access materials distributed under the terms of the [Creative Commons Attribution license](https://creativecommons.org/licenses/by/4.0/) (CC BY 4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.*
289 |
290 | * *The materials used in this lesson was derived from work that is Copyright © Data Carpentry (http://datacarpentry.org/).
291 | All Data Carpentry instructional material is made available under the [Creative Commons Attribution license](https://creativecommons.org/licenses/by/4.0/) (CC BY 4.0).*
292 |
--------------------------------------------------------------------------------
/lessons/08_salmon.md:
--------------------------------------------------------------------------------
1 | ---
2 | title: "Quantification of transcript abundance using Salmon"
3 | author: "Mary Piper and Meeta Mistry"
4 | date: "Wednesday, September 20, 2017"
5 | ---
6 |
7 | Approximate time: 1.25 hours
8 |
9 | ## Learning Objectives
10 |
11 | * Explore using lightweight algorithms to quantify reads to pseudocounts
12 | * Understand how Salmon performs quasi-mapping and transcript abundance estimation
13 |
14 |
15 | ## Lightweight alignment and quantification of gene expression
16 |
17 | In the standard RNA-seq pipeline that we have presented so far, we have taken our reads post-QC and aligned them to the genome using our transcriptome annotation (GTF) as guidance. The goal is to identify the genomic location where these reads originated from. **Another strategy for quantification which has more recently been introduced involves transcriptome mapping**. Tools that fall in this category include [Kallisto](https://pachterlab.github.io/kallisto/about), [Sailfish](http://www.nature.com/nbt/journal/v32/n5/full/nbt.2862.html) and [Salmon](https://combine-lab.github.io/salmon/); each working slightly different from one another. (For this workshop we will explore Salmon in more detail.) Common to all of these tools is that **base-to-base alignment of the reads is avoided**, which is a time-consuming step, and these tools **provide quantification estimates much faster than do standard approaches** (typically more than 20 times faster) with **improvements in accuracy** at **the transcript level**.
18 |
19 | These transcript expression estimates, often referred to as 'pseudocounts', can be converted for use with DGE tools like DESeq2 (using [tximport](https://bioconductor.org/packages/release/bioc/html/tximport.html)) or the estimates can be used directly for isoform-level differential expression using a tool like [Sleuth](http://www.biorxiv.org/content/biorxiv/early/2016/06/10/058164.full.pdf).
20 |
21 | 
22 |
23 | The improvement in accuracy for lightweight alignment tools in comparison with the standard alignment/counting methods primarily relate to the ability of the lightweight alignment tools to quantify multimapping reads. This has been shown by Robert et. al by comparing the accuracy of 12 different alignment/quantification methods using simulated data to estimate the gene expression of 1000 perfect RNA-Seq read pairs from each of of the genes [[1](https://genomebiology.biomedcentral.com/articles/10.1186/s13059-015-0734-x)]. As shown in the figures below taken from the paper, the **standard alignment and counting methods such as STAR/htseq or Tophat2/htseq result in underestimates of many genes - particularly those genes comprised of multimapping reads** [[1](https://genomebiology.biomedcentral.com/articles/10.1186/s13059-015-0734-x)].
24 |
25 | 
26 |
27 | _**NOTE:** Scatter plots are comparing FPKM for each of the 12 methods against the known FPKM from simulated data. The red line indicates the y = x line. For histograms of read counts, we expect a single peak at 1000 therefore tails represent over and under estimates._
28 |
29 | While the STAR/htseq standard method of alignment and counting is a bit conservative and can result in false negatives, **Cufflinks tends to overestimate gene expression and results in many false positives**, which is why Cufflinks is generally not recommended for gene expression quantification.
30 |
31 | 
32 |
33 | Finally, the most accurate quantification of gene expression was achieved using the lightweight alignment tool Sailfish (if used without bias correction).
34 |
35 | 
36 |
37 | Lightweight alignment tools such as Sailfish, Kallisto, and Salmon have generally been found to yield the most accurate estimations of transcript/gene expression. Salmon is considered to have some improvements relative to Sailfish, and it is considered to give very similar results to Kallisto.
38 |
39 |
40 | ### What is Salmon?
41 |
42 | [Salmon](http://salmon.readthedocs.io/en/latest/salmon.html#using-salmon) is based on the philosophy of lightweight algorithms, which use the reference transcriptome (in FASTA format) and raw sequencing reads (in FASTQ format) as input, but do not align the full reads. These tools perform both mapping and quantification. Unlike most lightweight and standard alignment/quantification tools, **Salmon utilizes sample-specific bias models for transcriptome-wide abundance estimation**. Sample-specific bias models are helpful when needing to account for known biases present in RNA-Seq data including:
43 |
44 | - GC bias
45 | - positional coverage biases
46 | - sequence biases at 5' and 3' ends of the fragments
47 | - fragment length distribution
48 | - strand-specific methods
49 |
50 | If not accounted for, these biases can lead to unacceptable false positive rates in differential expression studies [[2](http://salmon.readthedocs.io/en/latest/salmon.html#quasi-mapping-based-mode-including-lightweight-alignment)]. The **Salmon algorithm can learn these sample-specific biases and account for them in the transcript abundance estimates**. Salmon is extremely fast at "mapping" reads to the transcriptome and often more accurate than standard approaches [[2](http://salmon.readthedocs.io/en/latest/salmon.html#quasi-mapping-based-mode-including-lightweight-alignment)].
51 |
52 |
53 | ### How does Salmon estimate transcript abundances?
54 |
55 | Similar to standard base-to-base alignment approaches, the quasi-mapping approach utilized by Salmon requires a reference index to determine the position and orientation information for where the fragments best map prior to quantification [[3](https://academic.oup.com/bioinformatics/article/32/12/i192/2288985/RapMap-a-rapid-sensitive-and-accurate-tool-for)].
56 |
57 | #### **Indexing**
58 |
59 | This step involves creating an index to evaluate the sequences for all possible unique sequences of length k (kmer) in the **transcriptome** (genes/transcripts).
60 |
61 | **The index helps creates a signature for each transcript in our reference transcriptome.** The Salmon index has two components:
62 |
63 | 1. a **suffix array** (SA) of the reference transcriptome sequence
64 | 2. a **hash table** to map each k-mer occurring in the reference transcriptome to it's location in the SA (is not required, but improves the speed of mapping drastically)
65 |
66 | #### **Quasi-mapping and quantification**
67 |
68 | The quasi-mapping approach estimates the numbers of reads mapping to each transcript, and is described below. We have details and a schematic provided by the Rapmap tool [[3](https://academic.oup.com/bioinformatics/article/32/12/i192/2288985/RapMap-a-rapid-sensitive-and-accurate-tool-for)], which provides the underlying algorithm for the quasi-mapping.
69 |
70 | - **Step 1: Quasi mapping and abundance estimation**
71 |
72 | 
73 |
74 | >RapMap: a rapid, sensitive and accurate tool for mapping RNA-seq reads to transcriptomes. A. Srivastava, H. Sarkar, N. Gupta, R. Patro. Bioinformatics (2016) 32 (12): i192-i200.
75 |
76 | The quasi-mapping procedure performs the following steps [[3](https://academic.oup.com/bioinformatics/article/32/12/i192/2288985/RapMap-a-rapid-sensitive-and-accurate-tool-for)]:
77 |
78 | 1. The read is scanned from left to right until a k-mer that appears in the hash table is discovered.
79 | 2. The k-mer is looked up in the hash table and the SA intervals are retrieved, giving all suffixes containing that k-mer
80 | 3. Similar to STAR, the maximal matching prefix (MMP) is identified by finding the longest read sequence that exactly matches the reference suffixes.
81 | 4. Instead of searching for the next MMP in the read as we do with STAR, Salmon uses a k-mer skipping approach to identify the next informative position (NIP). In this way the SA search is likely to return a different set of transcripts than those returned for the previous MMP. Often natural variation or a sequencing error in the read is the cause of the mismatch from the reference, so beginning the search at this position would likely return the same set of transcripts.
82 | 5. This process is repeated until the end of the read.
83 | 6. The final set of mappings is determined by a consensus mechanism. Specifically, the algorithm reports the intersection of transcripts appearing in all MMPs for that read. The transcripts, orientation and transcript location are output for each read.
84 |
85 | >
86 | > **NOTE:** If there are k-mers in the reads that are not in the index, they are not counted. As such, trimming is not required when using this method. Accordingly, if there are reads from transcripts not present in the reference transcriptome, they will not be quantified. Quantification of the reads is only as good as the quality of the reference transcriptome.
87 |
88 |
89 | - **Step 2: Improving abundance estimates**
90 | Using multiple complex modeling approaches, like Expectation Maximization (EM), Salmon can also correct the abundance estimates for any sample-specific biases/factors [[4](http://www.nature.com.ezp-prod1.hul.harvard.edu/nmeth/journal/v14/n4/full/nmeth.4197.html?WT.feed_name=subjects_software&foxtrotcallback=true)]. Generally, this step results in more accurate transcript abundance estimation.
91 |
92 | ## Running Salmon on O2
93 |
94 | First start an interactive session and create a new directory for our Salmon analysis:
95 |
96 | ```bash
97 | $ srun --pty -p short -t 0-12:00 --mem 8G --reservation=HBC /bin/bash
98 |
99 | $ mkdir ~/unix_lesson/rnaseq/salmon
100 |
101 | $ cd ~/unix_lesson/rnaseq/salmon
102 | ```
103 |
104 | > Salmon is not available as a module on O2, but it is installed as part of the bcbio pipeline. Since we already have the appropriate path (`/n/app/bcbio/tools/bin/`) in our `$PATH` variable we can use it by simply typing in `salmon`.
105 |
106 | As you can imagine from the description above, when running Salmon there are also two steps.
107 |
108 | **Step 1: Indexing**
109 | "Index" the transcriptome using the `index` command:
110 |
111 | ```bash
112 | ## DO NOT RUN THIS CODE
113 | $ salmon index -t transcripts.fa -i transcripts_index --type quasi -k 31
114 | ```
115 | > **NOTE:** Default for salmon is --type quasi and -k 31, so we do not need to include these parameters in the index command. The kmer default of 31 is optimized for 75bp or longer reads, so if your reads are shorter, you may want a smaller kmer to use with shorter reads (kmer size needs to be an odd number).
116 |
117 | >
118 | **We are not going to run this in class, but it only takes a few minutes.** We will be using an index we have generated from transcript sequences (all known transcripts/ splice isoforms with multiples for some genes) for human. The transcriptome data (FASTA) was obtained from the [Ensembl ftp site](ftp://ftp.ensembl.org/pub/current_fasta/homo_sapiens/cdna/Homo_sapiens.GRCh38.cdna.all.fa.gz).
119 |
120 |
121 | **Step 2: Quantification:**
122 | Get the transcript abundance estimates using the `quant` command and the parameters described below (more information on parameters can be found [here](http://salmon.readthedocs.io/en/latest/salmon.html#id5)):
123 |
124 | * `-i`: specify the location of the index directory; for us it is `n/groups/hbctraining/ngs-data-analysis-longcourse/rnaseq/salmon.ensembl38.idx`
125 | * `-l A`: auto-detect library type. *You can also specify stranded single-end reads, more info available [here](http://salmon.readthedocs.io/en/latest/salmon.html#what-s-this-libtype)*
126 | * `-r`: list of files for sample
127 | * `-o`: output quantification file name
128 | * `--seqBias` will enable Salmon to learn and correct for sequence-specific biases in the input data
129 | * `--useVBOpt`: use variational Bayesian EM algorithm rather than the ‘standard EM’ to optimize abundance estimates (more accurate)
130 | * `--writeMappings=salmon.out`: creates a SAM-like file of all of the mappings. **We won't add this parameter, as it creates large files that will take up too much space in your home directory.**
131 |
132 | To run the quantification step on a single sample we have the command provided below. Let's try running it on our subset sample for `Mov10_oe_1.subset.fq`:
133 |
134 | ```bash
135 | $ salmon quant -i /n/groups/hbctraining/ngs-data-analysis-longcourse/rnaseq/salmon.ensembl38.idx \
136 | -l A \
137 | -r ~/unix_lesson/rnaseq/raw_data/Mov10_oe_1.subset.fq \
138 | -o Mov10_oe_1.subset.salmon \
139 | --seqBias \
140 | --useVBOpt
141 | ```
142 |
143 | >**NOTE 1:** Paired-end reads require both sets of reads to be given in addition to a [paired-end specific library type](http://salmon.readthedocs.io/en/latest/salmon.html#what-s-this-libtype):
144 | `salmon quant -i transcripts_index -l 
17 | 
32 | 
40 | 
64 |
65 | Installing packages can be timely and particularly cumbersome when doing this on a cluster environment. So rather than installing packages we have instructions for you to use the libraries from our installation.
66 |
67 | > **NOTE:** Packages are bundles of code that perform functions and include detailed documentation on how to use those functions. Once installed, they are referred to as _libraries_.
68 |
69 | **To use the libraries we have created for you first exit R with:**
70 |
71 | ```R
72 | q()
73 | ```
74 | You should find yourself back at the shell command prompt. The next few lines will set the environment variable `R_LIBS_USER` to let R know where the R libraries directory resides.
75 |
76 | ```bash
77 | # check if the variable is already set
78 | $ echo $R_LIBS_USER
79 |
80 | # If the above command returns nothing, then run the command below
81 | $ export R_LIBS_USER="/n/groups/hbctraining/R/library/"
82 | ```
83 |
84 | To run differential expression analysis, we are going to run a script from the `results` directory, so let's navigate there and create a directory for the results of our analysis. We will call the directory `diffexpression`:
85 |
86 | ```bash
87 | $ cd ~/unix_lesson/rnaseq/results
88 | $ mkdir diffexpression
89 | ```
90 | First, let's copy over the script file:
91 |
92 | ```bash
93 | $ cp /n/groups/hbctraining/intro_rnaseq_hpc/DESeq2_script.R diffexpression/
94 | ```
95 |
96 | The DE script will require as input **1) your count matrix file** and **2) a metadata file**. The count matrix we generated in the last lesson and is in the `counts` directory. The metadata file is a tab-delimited file which contains any information associated with our samples. Each row corresponds to a sample and each column contains some information about each sample.
97 |
98 | ```bash
99 | $ cp ~/unix_lesson/other/Mov10_rnaseq_metadata.txt diffexpression
100 | ```
101 | > **NOTE:** If you _didn't generate this file in class_ we have a pre-computed count matrix generated that you can use:
102 | >
103 | > `$ cp /groups/hbctraining/intro_rnaseq_hpc/counts_STAR/Mov10_rnaseq_counts_complete.txt diffexpression`
104 | >
105 |
106 | Once you have the files copied, take a quick look at the metadata using `less`.
107 |
108 | Now we're all setup to run our R script! Let's run it from within our `diffexpression` directory,
109 | ```bash
110 | $ cd diffexpression
111 | $ Rscript DESeq2_script.R Mov10_rnaseq_counts_complete.txt Mov10_rnaseq_metadata.txt
112 | ```
113 |
114 | > **NOTE:** You will notice chunks of code in the text that correspond to plotting figures, and these chunks have been commented out. The reason for this is in order to generate figures on **O2 you require the X11 system**, which we are currently not setup to do with the training accounts. If you are interested in learning more about using X11 applications you can [find out more on the O2 wiki page](https://wiki.rc.hms.harvard.edu/display/O2/Using+X11+Applications+Remotely).
115 |
116 | ### Gene list exploration
117 |
118 | There are two results files generated from `DE_script.R`, a full table and significant genes table (at FDR < 0.05). Take a look at the significant results file and see what values have been reported:
119 |
120 | ```bash
121 | $ head DEresults_sig_table.txt
122 | ```
123 | You should have a table with 7 columns in it:
124 |
125 | 1. `Gene symbols` (this will not have a column name, due to the nature of the `write` function)
126 | 2. `baseMean`: the average normalized counts across all samples
127 | 3. `log2FoldChange`
128 | 4. `lfcse`: the standard error of the log2 FC
129 | 5. `stat`: the Wald test statistic
130 | 6. `pvalue`
131 | 7. `padj`: p-value adjusted for multiple test correction using the BH method
132 |
133 | Since we have the full table of results for all genes, we could apply a filter based on the `padj` column to keep only genes we consider significant. We could also increase the stringency by adding in a fold change criteria. Alternatively, the full table can be useful for investigating groups of interesting genes of that are co-regulated but did not appear in our significant list.
134 |
135 | Using `wc -l` find out how many genes are identified in the significant table. Keep in mind this is generated using the truncated dataset.
136 |
137 | ```bash
138 | $ wc -l DEresults_sig_table.txt
139 | ```
140 |
141 | For downstream analysis, the relevant information that we will require from this results table is the gene names and the FDR value. We can cut the columns to a new file and and use that as input to some functional analaysis tools.
142 |
143 | ```bash
144 | $ cut -f1,7 DEresults_sig_table.txt > Mov10_sig_genelist.txt
145 | ```
146 |
147 | Since the list we have is generated from analaysis on a small subset of chromosome 1, using these genes as input to downstream tools will not provide any meaningful results. As such, **we have generated a list using the full dataset for these samples and can be downloaded to your laptop via [this link]().**
148 |
149 |
150 | ## Differential expression analysis using pseudocounts
151 | ----------------------
152 |
153 | In the script described above, we used count data generated from the standard RNA-seq workflow as input. The instructions are below to **perform a similar analysis with the output from Salmon, but on your local laptop**. To perform this analysis, you will need to use R and Rstudio directly. We do not have a script available that works on O2.
154 |
155 | **The rest of this section assumes that you are comfortable with R and RStudio.**
156 |
157 | The output from Salmon is transcript counts, but DESeq2 works well only with gene counts. To bridge this gap, the developers of DESeq2 have developed a package makes the output of Salmon compatible with DESeq2. This package is called [`tximport`](https://bioconductor.org/packages/release/bioc/html/tximport.html) and is also available through Bioconductor. `tximport` imports transcript-level abundance, estimated counts and transcript lengths, and summarizes this into matrices for use with downstream gene-level analysis packages.
158 |
159 | First, you have to download the directory with the quant.sf files for the 8 full datasets using the link below. Once you have them downloaded continue to follow the rest of instructions:
160 |
161 | 1. [Download Salmon files](https://www.dropbox.com/s/aw170f8zge01jpq/salmon.zip?dl=0)
162 | 2. Decompress (unzip) the zip archive and move the folder to an appropriate location (i.e `~/Desktop`)
163 | 3. Open RStudio and select 'File' -> 'New Project' -> 'Existing Directory' and navigate to the `salmon` directory
164 | 4. Open up a new R script ('File' -> 'New File' -> 'Rscript'), and save it as `salmon_de.R`
165 |
166 | Your Rstudio interface should look something like the screenshot below:
167 |
168 | 
169 |
170 | To perform this analysis you will have to install the following libraries:
171 |
172 | `tximport`
173 |
174 | `readr`
175 |
176 | `DESeq2`
177 |
178 | `biomaRt`
179 |
180 | **Step 1:** Load the required libraries:
181 |
182 | ```R
183 | # Load libraries
184 | library(tximport)
185 | library(readr)
186 | library(DESeq2)
187 | library(biomaRt) # tximport requires gene symbols as row names
188 | ```
189 |
190 | **Step 2:** Load the quantification data that was output from Salmon:
191 |
192 | ```R
193 | ## List all directories containing data
194 | samples <- list.files(path = ".", full.names = F, pattern="\\.salmon$")
195 |
196 | ## Obtain a vector of all filenames including the path
197 | files <- file.path(samples, "quant.sf")
198 |
199 | ## Since all quant files have the same name it is useful to have names for each element
200 | names(files) <- samples
201 | ```
202 |
203 | The **main objective here is to add names to our quant files which will allow us to easily discriminate between samples in the final output matrix**.
204 |
205 | **Step 3.** Create a dataframe containing Ensembl Transcript IDs and Gene symbols
206 |
207 | Our Salmon index was generated with transcript sequences listed by Ensembl IDs, but `tximport` needs to know **which genes these transcripts came from**, so we need to use the `biomaRt` package to extract this information.
208 |
209 | > *NOTE:* Keep in mind that the Ensembl IDs listed in our Salmon output contained version numbers (i.e ENST00000632684.1). If we query Biomart with those IDs it will not return anything. Therefore, before querying Biomart in R do not forget to strip the version numbers from the Ensembl IDs.
210 |
211 | ```R
212 | ## DO NOT RUN
213 |
214 | # Create a character vector of Ensembl IDs
215 | ids <- read.delim(files[1], sep="\t", header=T) # extract the transcript ids from one of the files
216 | ids <- as.character(ids[,1])
217 | require(stringr)
218 | ids.strip <- str_replace(ids, "([.][0-9])", "")
219 |
220 | # Create a mart object
221 | # Note that we are using an archived host, since "www.ensembl.org" gave us an error
222 | mart <- useDataset("hsapiens_gene_ensembl", useMart("ENSEMBL_MART_ENSEMBL", host="mar2016.archive.ensembl.org"))
223 |
224 | # Get official gene symbol and Ensembl gene IDs
225 | tx2gene <- getBM(
226 | filters= "ensembl_transcript_id",
227 | attributes= c("ensembl_transcript_id", "external_gene_name"),
228 | values= ids.strip,
229 | mart= mart)
230 |
231 | ```
232 |
233 | **We have already run the above code for you and saved the output in a text file which is in the salmon directory.** Load it in using:
234 |
235 | ```R
236 | tx2gene <- read.delim("tx2gene.txt",sep="\t")
237 | ```
238 |
239 | **Step 4:** Run tximport to summarize gene-level information
240 | ```R
241 | ?tximport # let's take a look at the arguments for the tximport function
242 |
243 | txi <- tximport(files, type="salmon", txIn = TRUE, txOut = FALSE, tx2gene=tx2gene, reader=read_tsv, ignoreTxVersion=TRUE)
244 | ```
245 | ### Output from `tximport`
246 |
247 | The `txi` object is a simple list with three matrices: abundance, counts, length.
248 | ```R
249 | attributes(txi)
250 | ```
251 | A final element 'countsFromAbundance' carries through the character argument used in the tximport call. The length matrix contains the average transcript length for each gene which can be used as an offset for gene-level analysis.
252 |
253 | ### Using DESeq2 for DE analysis with pseudocounts
254 |
255 | ```R
256 | ## Create a sampletable/metadata
257 |
258 | # Before we create this metadata object, let's see what the sample (column) order of the counts matrix is:
259 | colnames(txi$counts)
260 |
261 | condition=factor(c(rep("Ctl",3), rep("KD", 2), rep("OE", 3)))
262 | sampleTable <- data.frame(condition, row.names = colnames(txi$counts))
263 |
264 | ## Create a DESeqDataSet object
265 | dds <- DESeqDataSetFromTximport(txi, sampleTable, ~ condition)
266 | ```
267 |
268 | Now you have created a DESeq object to proceed with DE analysis you can now complete the DE analysis using methods in the script we ran
269 | for the counts from STAR.
270 |
271 | ### Resources for R
272 |
273 | * https://www.datacamp.com/courses/free-introduction-to-r
274 | * Software Carpentry materials: http://swcarpentry.github.io/r-novice-inflammation/
275 | * Data Carpentry materials: http://tracykteal.github.io/R-genomics/
276 | * Materials from IQSS at Harvard: http://tutorials.iq.harvard.edu/R/Rintro/Rintro.html
277 | * [swirl](http://swirlstats.com/): learn R interactively from within the R console
278 | * The free "try R" class from [Code School](http://tryr.codeschool.com)
279 | * HarvardX course ["Statistics and R for the Life Sciences"](https://courses.edx.org/courses/HarvardX/PH525.1x/1T2015/info)
280 |
281 | ***
282 | *This lesson has been developed by members of the teaching team at the [Harvard Chan Bioinformatics Core (HBC)](http://bioinformatics.sph.harvard.edu/). These are open access materials distributed under the terms of the [Creative Commons Attribution license](https://creativecommons.org/licenses/by/4.0/) (CC BY 4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.*
283 |
284 |
--------------------------------------------------------------------------------
/lessons/advanced_bash.md:
--------------------------------------------------------------------------------
1 | ---
2 | title: "Advanced Shell"
3 | author: "Radhika Khetani, Meeta Mistry"
4 | date: "May 9, 2018"
5 | ---
6 |
7 | ## Learning Objectives
8 |
9 | * Increasing productivity when working on the command-line and in a cluster environment
10 | * Becoming familiar with advanced bash commands and utilities
11 |
12 | ## Advanced Bash Commands and Utilities
13 |
14 | As you begin working more with the Shell, you will discover that there are mountains of different utilities at your fingertips to help increase command-line productivity. So far we have introduced you to some of the basics to help get you started. In this lesson, we will touch on some more advanced topics that can be very useful as you carry out analyses in a cluster environment.
15 |
16 | ## O2-specific utilities
17 |
18 | * [Configuring your shell](#config)
19 | * [`.bashrc` versus `.bash_profile`](#bashrc)
20 | * [Aliases](#alias)
21 | * [Symbolic links](#symlinks)
22 | * [Transferring files with `rsync`](#rsync)
23 | * [Working on `/n/scratch2/`](#nscratch)
24 |
25 | ***
26 |
27 | ## Configuring your shell 
25 |
26 | Different biological processes, as well as mutations, can affect which genes are turned on and which are turned off, in addition to, *how much* specific genes are turned on/off.
27 |
28 | To make proteins, the DNA is transcribed into messenger RNA, or mRNA, which is translated by the ribosome into protein. However, some genes encode RNA that does not get translated into protein; these RNAs are called non-coding RNAs, or ncRNAs. Often these RNAs have a function in and of themselves and include rRNAs, tRNAs, and siRNAs, among others. All RNAs transcribed from genes are called transcripts.
29 |
30 | 
31 |
32 | To be translated into proteins, the RNA must undergo processing to generate the mRNA. In the figure below, the top strand in the image represents a gene in the DNA, comprised of the untranslated regions (UTRs) and the open read frame. Genes are transcribed into pre-mRNA, which still contains the intronic sequences. After post-transciptional processing, the introns are spliced out and a polyA tail and 5' cap are added to yield mature mRNA transcripts, which can be translated into proteins.
33 |
34 | 
35 |
36 | **While mRNA transcripts have a polyA tail, many of the non-coding RNA transcripts do not as the post-transcriptional processing is different for these transcripts.**
37 |
38 | RNA-seq data can be used to explore and/or quantify the RNA transcripts, which can be utilized for the following types of experiments:
39 |
40 | - Differential Gene Expression: *quantitative* evaluation and comparison of transcript levels
41 | - Transcriptome assembly: building the profile of transcribed regions of the genome, a *qualitative* evaluation.
42 | - Can be used to help build better gene models, and verify them using the assembly
43 | - Metatranscriptomics or community transcriptome analysis
44 |
45 |
46 | ## Illumina library preparation
47 |
48 | When starting an RNA-seq experiment, for every sample the RNA needs to be isolated and turned into a cDNA library for sequencing. Generally, ribosomal RNA represents the majority of the RNAs present in a cell, while messenger RNAs represent a small percentage of total RNA, ~2% in humans.
49 |
50 | 
51 |
52 | Therefore, if we want to study the protein-coding genes, we need to enrich for mRNA or deplete the rRNA. **For differential gene expression analysis, it is best to enrich for Poly(A)+, unless you are aiming to obtain information about long non-coding RNAs, then do a ribosomal RNA depletion.**
53 |
54 | The workflow for library preparation is detailed in the image below:
55 |
56 | 
57 |
58 | *Image credit: [Martin J.A. and Wang Z., Nat. Rev. Genet. (2011) 12:671–682](https://www.nature.com/articles/nrg3068)*
59 |
60 | Briefly, the RNA is isolated from the sample and contaminating DNA is removed, followed by either selection of the mRNA or depletion of the rRNA. The resulting RNA is fragmented then reverse transcribed into cDNA. Sequence adapters are added to the ends of the fragments and the fragments are PCR amplified if needed. Finally, the fragments are size selected (usually ~300-500bp) to finish the library.
61 |
62 | The cDNA libraries can be generated in a way to retain information about which strand of DNA the RNA was transcribed from. Libraries that retain this information are called stranded libraries, which are now standard with Illumina’s TruSeq stranded RNA-Seq kits. Stranded libraries should not be any more expensive than unstranded, so there is not really any reason not to acquire this additional information.
63 |
64 | There are 3 types of cDNA libraries available:
65 |
66 | - Forward (secondstrand) – reads resemble the gene sequence or the secondstrand cDNA sequence
67 | - Reverse (firststrand) – reads resemble the complement of the gene sequence or firststrand cDNA sequence (TruSeq)
68 | - Unstranded
69 |
70 | > **NOTE:** This workflow is specific to Illumina sequencing, which is currently the most utilized sequencing method. But there are other long-read methods worth noting, such as:
71 | >
72 | > - Pacific Biosciences: http://www.pacb.com/
73 | > - Oxford Nanopore (MinION): https://nanoporetech.com/
74 | > - 10X Genomics: https://www.10xgenomics.com/
75 | >
76 | > Advantages and disadvantages of these technologies can be explored in the table below:
77 | >
78 | > 
79 |
80 | ## Illumina Sequencing
81 |
82 | After preparation of the libraries, sequencing can be performed to generate the nucleotide sequences of the ends of the fragments, which are called **reads**. You will have the choice of sequencing a single end of the cDNA fragments (single-end reads) or both ends of the fragments (paired-end reads).
83 |
84 | 
85 |
86 | - SE - Single end dataset => Only Read1
87 | - PE - Paired-end dataset => Read1 + Read2
88 | - can be 2 separate FastQ files or just one with interleaved pairs
89 |
90 | Generally single-end sequencing is sufficient unless it is expected that the reads will match multiple locations on the genome (e.g. organisms with many paralogous genes), assemblies are being performed, or for splice isoform differentiation. Be aware that paired-end reads are generally 2x more expensive.
91 |
92 | There are a variety of Illumina platforms to choose from to sequence the cDNA libraries.
93 |
94 | 
95 |
96 | *Image credit: Adapted from [Illumina](www.illumina.com)*
97 |
98 | Differences in platform can alter the length of reads generated as well as the total number of reads sequenced per run and the amount of time required to sequence the libraries. The different platforms each use a different flow cell, which is a glass surface coated with an arrangement of paired oligos that are complementary to the adapters added to your template molecules. The flow cell is where the sequencing reactions take place.
99 |
100 | 
101 |
102 | *Image credit: Adapted from [Illumina](www.illumina.com)*
103 |
104 |
105 | Let's explore how Illumina sequencing is performed:
106 |
107 | [
](https://www.dropbox.com/s/f4t94tcw06f9stg/Illumina%20Sequencing%20by%20Synthesis-14840.mp4?dl=0)
108 |
109 | - Number of clusters ~= Number of reads
110 | - Number of sequencing cycles = Length of reads
111 |
112 | The number of cycles (length of the reads) will depend on sequencing platform used as well as your preferences.
113 |
114 | Charges for sequencing are usually per lane of the flow cell, and usually you don’t need one lane per sample. Multiplexing allows you to sequence multiple samples per lane with addition of indices (within the Illumina adapter) or special barcodes (outside the Illumina adapter).
115 |
116 | 
117 |
118 | ## Experimental planning considerations
119 |
120 | Understanding the steps in the experimental process of RNA extraction and preparation of RNA-Seq libraries is helpful for designing an RNA-Seq experiment, but there are special considerations that should be highlighted that can greatly affect the quality of a differential expression analysis.
121 |
122 | These important considerations include:
123 |
124 | 1. Number and type of **replicates**
125 | 2. Avoiding **confounding**
126 | 3. Addressing **batch effects**
127 |
128 | We will go over each of these considerations in detail, discussing best practice and optimal design.
129 |
130 | ## Replicates
131 |
132 | Experimental replicates can be performed as **technical replicates** or **biological replicates**.
133 |
134 | 
135 |
136 | *Image credit: [Klaus B., EMBO J (2015) **34**: 2727-2730](https://dx.doi.org/10.15252%2Fembj.201592958)*
137 |
138 | - **Technical replicates:** use the same biological sample to repeat the technical or experimental steps in order to accurately measure technical variation and remove it during analysis.
139 |
140 | - **Biological replicates** use different biological samples of the same condition to measure the biological variation between samples.
141 |
142 | For mice or rats, this might be easy to determine what constitutes a different biological sample, but it's a bit more difficult to determine for cell lines. When using cell lines it's best to include as much variation between samples as possible, and [this article](http://paasp.net/accurate-design-of-in-vitro-experiments-why-does-it-matter/) gives some great recommendations for cell line replicates.
143 |
144 | In the days of microarrays, technical replicates were considered a necessity; however, with the current RNA-Seq technologies, technical variation is much lower than biological variation and **technical replicates are unneccessary**.
145 |
146 | In contrast, **biological replicates are absolutely essential**. For differential expression analysis, the more biological replicates, the better the estimates of biological variation and the more precise our estimates of the mean expression levels. This leads to more accurate modeling of our data and identification of more differentially expressed genes.
147 |
148 | 
149 |
150 | *Image credit: [Liu, Y., et al., Bioinformatics (2014) **30**(3): 301–304](https://doi.org/10.1093/bioinformatics/btt688)*
151 |
152 | As the figure above illustrates, **biological replicates are of greater importance than sequencing depth**, which is the total number of reads sequenced per sample. The figure shows the relationship between sequencing depth and number of replicates on the number of differentially expressed genes identified [[1](https://academic.oup.com/bioinformatics/article/30/3/301/228651/RNA-seq-differential-expression-studies-more)]. Note that an **increase in the number of replicates tends to return more DE genes than increasing the sequencing depth**. Therefore, generally more replicates are better than higher sequencing depth, with the caveat that higher depth is required for detection of lowly expressed DE genes and for performing isoform-level differential expression.
153 |
154 | Replicates are almost always preferred to greater sequencing depth for bulk RNA-Seq. However, **guidelines depend on the experiment performed and the desired analysis**. Below we list some general guidelines for replicates and sequencing depth to help with experimental planning:
155 |
156 |
157 | - **General gene-level differential expression:**
158 |
159 | - ENCODE guidelines suggest 30 million SE reads per sample (stranded).
160 |
161 | - 15 million reads per sample is often sufficient, if there are a good number of replicates (>3).
162 |
163 | - Spend money on more biological replicates, if possible.
164 |
165 | - **Gene-level differential expression with detection of lowly-expressed genes:**
166 |
167 | - Similarly benefits from replicates more than sequencing depth.
168 |
169 | - Sequence deeper with at least 30-60 million reads depending on level of expression (start with 30 million with a good number of replicates).
170 |
171 | - **Isoform-level differential expression:**
172 |
173 | - Of known isoforms, suggested to have a depth of at least 30 million reads per sample and paired-end reads.
174 |
175 | - Of novel isoforms should have more depth (> 60 million reads per sample).
176 |
177 | - Choose biological replicates over paired/deeper sequencing.
178 |
179 | - Perform careful QC of RNA quality. Be careful to use high quality preparation methods and restrict analysis to high quality RIN # samples.
180 |
181 | - **Other types of RNA analyses (intron retention, small RNA-Seq, etc.):**
182 |
183 | - Different recommendations depending on the analysis.
184 |
185 | - Almost always more biological replicates are better!
186 |
187 | > **NOTE:** The factor used to estimate the depth of sequencing for genomes is "coverage" - how many times do the number nucleotides sequenced "cover" the genome. This metric is not exact for genomes, but it works okay. It **does not work for transcriptomes** because expression of the genes depend on the condition being studied.
188 |
189 | ## Confounding
190 |
191 | A confounded RNA-Seq experiment is one where you **cannot distinguish the separate effects of two different sources of variation** in the data.
192 |
193 | For example, we know that sex has large effects on gene expression, and if all of our *control* mice were female and all of the *treatment* mice were male, then our treatment effect would be confounded by sex. **We could not differentiate the effect of treatment from the effect of sex.**
194 |
195 | 
196 |
197 | **To AVOID confounding:**
198 |
199 | - Ensure animals in each condition are all the **same sex, age, litter, and batch**, if possible.
200 |
201 | - If not possible, then ensure to split the animals equally between conditions
202 |
203 | 
204 |
205 | ## Batch effects
206 |
207 | Batch effects are a significant issue for RNA-Seq analyses, since you can see significant differences in expression due solely to the batch effect.
208 |
209 | 
210 |
211 | *Image credit: [Hicks SC, et al., bioRxiv (2015)](https://www.biorxiv.org/content/early/2015/08/25/025528)*
212 |
213 | To explore the issues generated by poor batch study design, they are highlighted nicely in [this paper](https://f1000research.com/articles/4-121/v1).
214 |
215 | ### How to know whether you have batches?
216 |
217 | - Were all RNA isolations performed on the same day?
218 |
219 | - Were all library preparations performed on the same day?
220 |
221 | - Did the same person perform the RNA isolation/library preparation for all samples?
222 |
223 | - Did you use the same reagents for all samples?
224 |
225 | - Did you perform the RNA isolation/library preparation in the same location?
226 |
227 | If *any* of the answers is **‘No’**, then you have batches.
228 |
229 | ### Best practices regarding batches:
230 |
231 | - Design the experiment in a way to **avoid batches**, if possible.
232 |
233 | - If unable to avoid batches:
234 |
235 | - **Do NOT confound** your experiment by batch:
236 |
237 | 
238 |
239 | *Image credit: [Hicks SC, et al., bioRxiv (2015)](https://www.biorxiv.org/content/early/2015/08/25/025528)*
240 |
241 | - **DO** split replicates of the different sample groups across batches. The more replicates the better (definitely more than 2).
242 |
243 | 
244 |
245 | *Image credit: [Hicks SC, et al., bioRxiv (2015)](https://www.biorxiv.org/content/early/2015/08/25/025528)*
246 |
247 | - **DO** include batch information in your **experimental metadata**. During the analysis, we can regress out the variation due to batch so it doesn’t affect our results if we have that information.
248 |
249 | 
250 |
251 | ***
252 | **Exercise**
253 |
254 | Your experiment has three different treatment groups, A, B, and C. Due to the lengthy process of tissue extraction, you can only isolate the RNA from two samples at the same time. You plan to have 4 replicates per group.
255 |
256 | 1. Fill in the `RNA isolation` column of the metadata table. Since we can only prepare 2 samples at a time and we have 12 samples total, you will need to isolate RNA in 6 batches. In the `RNA isolation` column, enter one of the following values for each sample: `group1`, `group2`, `group3`, `group4`, `group5`, `group6`. Make sure to fill in the table so as to avoid confounding by batch of `RNA isolation`.
257 |
258 | 2. **BONUS:** To perform the RNA isolations more quickly, you devote two researchers to perform the RNA isolations. Fill in their initials to the `researcher` column for the samples they will prepare: use initials `AB` or `CD`.
259 |
260 | | sample | treatment | sex | replicate | RNA isolation |
261 | | --- | --- | --- | --- | --- |
262 | | sample1 | A | F | 1 |
263 | | sample2 | A | F | 2 |
264 | | sample3 | A | M | 3 |
265 | | sample4 | A | M | 4 |
266 | | sample5 | B | F | 1 |
267 | | sample6 | B | F | 2 |
268 | | sample7 | B | M | 3 |
269 | | sample8 | B | M | 4 |
270 | | sample9 | C | F | 1 |
271 | | sample10 | C | F | 2 |
272 | | sample11 | C | M | 3 |
273 | | sample12 | C | M | 4 |
274 |
275 | ***
276 |
--------------------------------------------------------------------------------
/sam.md:
--------------------------------------------------------------------------------
1 | ## samtools extras
2 |
3 | To play around with a few `samtools` commands, first change directories into the directory containing all BAM files.
4 |
5 | `$ cd ~/unix_workshop/rnaseq/results/STAR/bams`
6 |
7 | ### Write only mapped reads to file (filter out unmapped reads)
8 |
9 | `$ samtools view -b -h -F 4 Mov10_oe_1_Aligned.sortedByCoord.out.bam > Mov10_oe_1_Aligned.onlyAligned.bam`
10 |
11 | ### Create a FASTQ file containing only mapped reads
12 |
13 | `$ bamtofastq -o Mov10_oe_1_Mapped.fastq --no-unaligned Mov10_oe_1_Aligned.onlyMapped.bam`
14 |
15 | ### Index BAM file
16 |
17 | `$ samtools index Mov10_oe_1_Aligned.sortedByCoord.out.bam`
18 |
19 | ### Extract reads from a specific region of the chromosome
20 |
21 | `$samtools view Mov10_oe_1_Aligned.sortedByCoord.out.bam chr1:200000-500000`
22 |
23 | ### Randomly subsample half of the reads into a new BAM file
24 |
25 | `$ samtools view -s 0.5 -b Mov10_oe_1_Aligned.sortedByCoord.out.bam > Mov10_oe_1_subsample.bam`
26 |
27 | ### Simple stats for alignment file
28 |
29 | `$ samtools flagstat Mov10_oe_1_Aligned.sortedByCoord.out.bam`
30 |
31 | ### Visualizing mismatches
32 |
33 | `$ samtools view -h Mov10_oe_1_Aligned.sortedByCoord.out.bam | head -n 5 | samtools fillmd -e - ~/unix_workshop/rnaseq/reference_data/chr1.fa`
34 |
35 |
--------------------------------------------------------------------------------
/schedule/.DS_Store:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/hbctraining/Intro-to-rnaseq-hpc-O2/0feca5559bcbde27cbc7634085f07b3624a36f2f/schedule/.DS_Store
--------------------------------------------------------------------------------
/schedule/2-day/.DS_Store:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/hbctraining/Intro-to-rnaseq-hpc-O2/0feca5559bcbde27cbc7634085f07b3624a36f2f/schedule/2-day/.DS_Store
--------------------------------------------------------------------------------
/schedule/2-day/README.md:
--------------------------------------------------------------------------------
1 | # Workshop Schedule (2-day)
2 |
3 | ## Day 1
4 |
5 | | Time | Topic | Instructor |
6 | |:------------------------:|:------------------------------------------------:|:--------:|
7 | |9:00 - 9:40 | [Workshop Introduction] | Radhika |
8 | |9:40 - 10:40 | [Introduction to the Shell] | Radhika |
9 | |10:40 - 10:50 | Break | |
10 | |10:50 - 11:35 | [Introduction to the Shell (cont.)] | Meeta |
11 | |11:35 - 12:15 | [Searching and Redirection] | Mary |
12 | |12:15 - 13:15 | Lunch | |
13 | |13:15 - 13:35 | [Introduction to the Vim Text Editor] | Mary |
14 | |13:35 - 14:50 | [Loops and Shell Scripts] | Meeta |
15 | |14:50 - 15:00 | Break | |
16 | |15:00 - 15:30 | [Permissions and Environment Variables] | Radhika |
17 | |15:30 - 16:00 | [Project Organization and Best Practices in Data Management] | Meeta |
18 | |16:00 - 17:00 | [Introduction to RNA-seq and Library Prep] | Radhika |
19 |
20 | ## Day 2
21 |
22 | | Time | Topic | Instructor |
23 | |:------------------------:|:----------:|:--------:|
24 | |9:00 - 9:50 | [Introduction to High-Performance Computing] | Radhika |
25 | |9:50 - 10:30 | [RNA-seq analysis workshop - Quality Assessment] | Mary |
26 | |10:30 - 10:40 | Break | |
27 | |10:40 - 11:15 | [RNA-seq analysis workshop - Quality Assessment] | Mary |
28 | |11:15 - 12:00 | [RNA-seq analysis workshop - Alignment and Counting] | Meeta |
29 | |12:00 - 13:00 | Break | |
30 | |13:00 - 13:45 | [RNA-seq analysis workshop - Alignment and Counting] | Meeta |
31 | |13:45 - 14:45 | [Automating the RNA-seq workflow] | Radhika |
32 | |14:45 - 15:00 | Break | |
33 | |15:00 - 16:30 | [Advanced concepts in bash] | Meeta/Radhika |
34 | |16:30 - 17:00 | [Wrap up + Q & A] | Radhika |
35 |
--------------------------------------------------------------------------------
/schedule/README.md:
--------------------------------------------------------------------------------
1 | # Workshop Schedule
2 |
3 | ## Day 1
4 |
5 | | Time | Topic | Instructor |
6 | |:------------------------:|:------------------------------------------------:|:--------:|
7 | |9:00 - 9:40 | [Workshop Introduction](https://github.com/hbctraining/Intro-to-rnaseq-hpc-O2/raw/master/lectures/Intro_to_workshop.pdf) | Meeta |
8 | |9:40 - 10:30 | [Introduction to the Shell](https://hbctraining.github.io/Intro-to-Shell/lessons/01_the_filesystem.html) | Mary |
9 | |10:30 - 10:45 | Break | |
10 | |10:45 - 11:35 | [Introduction to the Shell (cont.)](https://hbctraining.github.io/Intro-to-Shell/lessons/01_the_filesystem.html) | Meeta |
11 | |11:35 - 12:15 | [Searching and Redirection](https://hbctraining.github.io/Intro-to-Shell/lessons/02_searching_files.html) | Mary |
12 | |12:15 - 13:15 | Lunch | |
13 | |13:15 - 13:45 | [Introduction to the Vim Text Editor](https://hbctraining.github.io/Intro-to-Shell/lessons/03_vim.html) | Mary |
14 | |13:45 - 15:00 | [Loops and Shell Scripts](https://hbctraining.github.io/Intro-to-Shell/lessons/04_loops_and_scripts.html) | Meeta |
15 | |15:00 - 15:15 | Break | |
16 | |15:15 - 15:45 | [Permissions and Environment Variables](https://hbctraining.github.io/Intro-to-Shell/lessons/05_permissions_and_environment_variables.html) | Mary |
17 | |15:45 - 17:00 | [Project Organization and Best Practices in Data Management](https://hbctraining.github.io/Intro-to-rnaseq-hpc-O2/lessons/01_data_organization.html) | Meeta |
18 |
19 | ## Day 2
20 |
21 | | Time | Topic | Instructor |
22 | |:------------------------:|:----------:|:--------:|
23 | |9:00 - 9:45 | [Introduction to High-Performance Computing for HMS-RC's O2](https://github.com/hbctraining/Intro-to-rnaseq-hpc-O2/raw/master/lectures/HPC_intro_O2.pdf) | Meeta |
24 | |9:45 - 10:00 | Break | |
25 | |10:00 - 11:15 | [Introduction to RNA-seq and Library Prep](https://github.com/hbctraining/Intro-to-rnaseq-hpc-O2/blob/master/lectures/rna-seq_design.pdf) | Mary |
26 | |11:15 - 11:55 | [NGS workflows and data standards](https://github.com/hbctraining/Intro-to-rnaseq-hpc-O2/blob/master/lectures/NGS_workflows.pdf) | Meeta |
27 | |11:55 - 12:55 | Lunch | |
28 | |12:55 - 13:50 | [Quality Assessment of Sequence Data](https://hbctraining.github.io/Intro-to-rnaseq-hpc-O2/lessons/02_assessing_quality.html) | Mary |
29 | |13:50 - 14:30 | [Sequence Alignment Theory](https://github.com/hbctraining/Intro-to-rnaseq-hpc-O2/blob/master/lectures/Sequence_alignment.pdf) | Meeta |
30 | |14:30 - 14:45 | Break | |
31 | |14:45 - 16:00 | [RNA-seq Alignment with STAR](https://hbctraining.github.io/Intro-to-rnaseq-hpc-O2/lessons/03_alignment.html) | Mary |
32 | |16:00 - 17:00 | [Assessing Alignment Quality](https://hbctraining.github.io/Intro-to-rnaseq-hpc-O2/lessons/04_alignment_quality.html) | Meeta |
33 |
34 | ## Day 3
35 |
36 | | Time | Topic | Instructor |
37 | |:------------------------:|:----------:|:--------:|
38 | |9:00 - 10:15 | [Generating a Count Matrix](https://hbctraining.github.io/Intro-to-rnaseq-hpc-O2/lessons/05_counting_reads.html) | Meeta |
39 | |10:15 - 10:45 | [Documenting Steps in the Workflow with MultiQC](https://hbctraining.github.io/Intro-to-rnaseq-hpc-O2/lessons/06_multiQC.html) | Meeta |
40 | |10:45 - 11:00 | Break | |
41 | |11:00 - 12:35 | [Automating the RNA-seq workflow](https://hbctraining.github.io/Intro-to-rnaseq-hpc-O2/lessons/07_automating_workflow.html) | Mary |
42 | |12:35 - 13:35 | Lunch | |
43 | |13:35 - 13:45 | [Alternative workflows for analyzing RNA-seq data](https://github.com/hbctraining/Intro-to-rnaseq-hpc-O2/blob/master/lectures/RNAseq-analysis-methods.pdf) | Mary |
44 | |13:45 - 15:20 | [Quantifying expression using alignment-free methods (Salmon)](https://hbctraining.github.io/Intro-to-rnaseq-hpc-O2/lessons/08_salmon.html) | Meeta |
45 | |15:20 - 15:35 | Break | |
46 | |15:35 - 15:45 | [Other Applications of RNA-seq](https://github.com/hbctraining/Intro-to-rnaseq-hpc-O2/blob/master/lectures/other%20rnaseq%20applications.pdf) | Mary |
47 | |15:45 - 16:25 | [Troubleshooting RNA-seq Data Analysis](https://github.com/hbctraining/Intro-to-rnaseq-hpc-O2/blob/master/lectures/RNA-seq_troubleshooting.pdf) | Mary |
48 | |16:25 - 17:00 | [Genome Builds and Accessing Data on GEO/SRA](https://github.com/hbctraining/Intro-to-rnaseq-hpc-O2/blob/master/lectures/Accessing_genomics_dataonline.pdf) | Meeta |
49 | | | [Wrap-up](https://www.dropbox.com/s/6diqq661xn3wgko/Wrap-up.pdf?dl=0) | Mary |
50 |
51 |
--------------------------------------------------------------------------------
/scripts/mov10_fastqc.run:
--------------------------------------------------------------------------------
1 | #!/bin/bash
2 |
3 | #SBATCH -p short # partition name
4 | #SBATCH -t 0-2:00 # hours:minutes runlimit after which job will be killed
5 | #SBATCH -n 6 # number of cores requested -- this needs to be greater than or equal to the number of cores you plan to use to run your job
6 | #SBATCH --job-name rnaseq_mov10_fastqc # Job name
7 | #SBATCH -o %j.out # File to which standard out will be written
8 | #SBATCH -e %j.err # File to which standard err will be written
9 |
10 | ## Changing directories to where the fastq files are located
11 | cd ~/unix_workshop/rnaseq/raw_data
12 |
13 | ## Loading modules required for script commands
14 | module load seq/fastqc/0.11.3
15 |
16 | ## Running FASTQC
17 | fastqc -t 6 *.fq
18 |
19 | ## Moving files to our results directory
20 | mv *fastqc* ../results/fastqc/
21 |
--------------------------------------------------------------------------------
/scripts/rnaseq_analysis_on_allfiles_for-slurm.sh:
--------------------------------------------------------------------------------
1 | #! /bin/bash
2 |
3 | for fq in ~/unix_lesson/rnaseq/raw_data/*.fq
4 | do
5 |
6 | sbatch -p short -t 0-2:00 -n 6 --job-name rnaseq-workflow --wrap="sh ~/unix_lesson/rnaseq/scripts/rnaseq_analysis_on_input_file.sh $fq"
7 | sleep 1 # wait 1 second between each job submission
8 |
9 | done
10 |
--------------------------------------------------------------------------------
/scripts/rnaseq_analysis_on_input_file.sh:
--------------------------------------------------------------------------------
1 | #!/bin/bash/
2 |
3 | # This script takes a fastq file of RNA-Seq data, runs FastQC and outputs a counts file for it.
4 | # USAGE: sh rnaseq_analysis_on_allfiles.sh