|
Introduction
|
Many bioinformatics tools can only process large data in the command line version not the GUI.
The shell makes your work less boring (same set of tasks with a large number of files)”
The shell makes your work less error-prone
The shell makes your work more reproducible.
Many bioinformatic tasks require large amounts of computing power
|
|
Navigating and Editing Files
|
The file system is responsible for managing information on the disk.
Information is stored in files, which are stored in directories (folders).
Directories can also store other directories, which then form a directory tree.
The command pwd prints the user’s current working directory.
The command ls [path] prints a listing of a specific file or directory; ls on its own lists the current working directory.
The command cd [path] changes the current working directory.
Most commands take options that begin with a single -.
Directory names in a path are separated with / on Unix.
Slash (/) on its own is the root directory of the whole file system.
An absolute path specifies a location from the root of the file system.
A relative path specifies a location starting from any location other than the root.
A ~ indicates your home directory
A - indicates the last directory that you were in
Dot (.) on its own means ‘the current directory’; .. means ‘the directory above the current one’.
|
|
Working With Files and Directories
|
|
|
Repeating yourself
|
wc counts lines, words, and characters in its inputs (see SWC primer).
cat displays the contents of its inputs.
sort sorts its inputs (see SWC primer).
head displays the first 10 lines of its input by default without additional arguments.
tail displays the last 10 lines of its input by default without additional arguments.
command > [file] redirects a command’s output to a file (overwriting any existing content!!).
command >> [file] appends a command’s output to a file.
first | second is a pipeline. The output of the first command is used as the input to the second.
The best way to use the shell is to use pipes to combine simple single-purpose programs (filters)
A for loop repeats commands once for every thing in a list.
Every for loop needs a variable to refer to the thing it is currently operating on.
Use $name to expand a variable (i.e., get its value). ${name} can also be used.
Do not use spaces, quotes, or wildcard characters such as * or ? in filenames, as it complicates variable expansion.
Give files consistent names that are easy to match with wildcard patterns to make it easy to select them for looping.
Use the ↑/↓ keys to scroll through previous commands to edit and repeat them.
Use Ctrl+R to search through the previously entered commands (see SWC primer).
Use history to display recent commands, and !number to repeat a command by number (see SWC primer).
Save commands in files (usually called shell scripts) for re-use.
bash filename runs the commands saved in a file.
$@ refers to all of a shell script’s command-line arguments (see SWC primer).
$1, $2, etc., refer to the first command-line argument, the second command-line argument, etc.
Place variables in quotes if the values might have spaces in them.
|
|
Starting with data in R
|
The dplyr package provides functions for manipulating data frames, such as select, filter, mutate, summarize, group_by, and count.
The tidyr package provides functions for reshaping data frames, such as gather and spread (see https://lessons.datacarpentry.org/R-ecology-lesson/03-dplyr.html).
|
|
Manipulating, analyzing and exporting data with tidyverse
|
The dplyr package provides functions for manipulating data frames, such as select, filter, mutate, summarize, group_by, and count.
The tidyr package provides functions for reshaping data frames, such as gather and spread (see https://lessons.datacarpentry.org/R-ecology-lesson/03-dplyr.html).
|
|
Read Alignment and Small Variant Calling
|
|
|
Structural Variation in short reads
|
Structural variants are more difficult to identify.
Discovery of SVs usually requires multiple types of read evidence.
There is often significant disagreement between SV callers.
|
|
Structural Variation in long reads
|
|
|
SV Exercises
|
|
|
Cancer Genomics
|
|
|
Bulk RNA differential expression
|
Normalization is necessary to account for different library sizes between samples.
Dimensional reduction (PCA) can help us understand the relationship between samples, and how it tracks with biological and technical variables.
We often want to include technical variables in the design formula, so that the differential expression comparison may include controlling for these as we solve for the effect of the biological variable.
Heatmaps are a useful visualization for understanding differential expression results, but must often be scaled by gene to become interpretable.
ORA and GSEA accept very different inputs (only DE genes versus all genes), and their outputs may be different as well as a result (though often they include similar categories).
|
|
Single-cell RNA
|
Single-cell RNA analysis often starts with a gene-cell count matrix.
QC metrics used for understanding and filtering data quality include mitochondrial rate and number of genes.
Processing stages in a scRNA workflow includes normalization, feature selection, scaling, and dimensional reduction.
Clustering is a key step in understanding biology.
We can use known marker genes or run differential expression between clusters to annotate cell types after clustering.
Annotation often requires starting with broad cell types, then moving to more specific.
|