Linux for Biological Data Science
Bioinformatics File Navigation
Navigating the Data Jungle
Bioinformatics often means working with massive datasets spread across many folders. The first step is learning how to move around. The command line, or shell, is your map and compass.
To find out where you are at any moment, use the pwd command, which stands for "print working directory." It tells you your current location in the file system's hierarchy.
To move to a different directory, you'll use cd, for "change directory." You can specify a full path from the root directory (e.g., cd /home/user/data/genomes) or a relative path from your current location (e.g., cd ../raw_reads).
# Check your current location
pwd
# Move to a directory named 'project_alpha'
cd project_alpha
# Move back up one level
cd ..
Sizing Up Your Files
Once you've navigated to the right directory, you'll want to see what's inside. The ls command lists the contents. By itself, it's not very informative for large data files. Adding flags gives you more details.
The ls -lh command is particularly useful. The -l flag provides a long list format, showing permissions, ownership, and modification times. The -h flag makes file sizes "human-readable," displaying them in kilobytes (K), megabytes (M), or gigabytes (G) instead of just bytes. This is essential when you're dealing with enormous FASTQ or BAM files and need a quick sense of their scale without trying to open them.
# List files with human-readable sizes
ls -lh
# Example output:
# -rw-r--r-- 1 user group 4.1G Jan 15 10:30 sample1_R1.fastq.gz
# -rw-r--r-- 1 user group 3.9G Jan 15 10:32 sample2_R1.fastq.gz
Peeking Inside Without Opening
Opening a 50 GB file just to see its format is a recipe for a frozen computer. Instead, you can peek at small portions of a file. The head command shows you the first ten lines of a file by default, while tail shows you the last ten. This is perfect for checking column headers in a data table or the structure of a sequence file.
You can specify how many lines you want to see with the -n flag. For example, head -n 4 will show you just the first four lines.
Bioinformatics files are almost always compressed to save space, usually indicated by a .gz extension. You can't use head or tail directly on them. Instead, you need tools that can handle compression.
The zcat command decompresses a .gz file and prints its contents to the screen without creating a new, uncompressed file. To view just the beginning of a compressed file, you can pipe the output of zcat into head.
For browsing larger files page by page, less is a powerful tool. It loads the file one screen at a time, so you can scroll through it without consuming all your computer's memory. The equivalent for compressed files is zless, which lets you scroll through a .gz file as if it were uncompressed.
# Look at the first 5 lines of a compressed file
zcat huge_file.fastq.gz | head -n 5
# Browse a compressed file page by page
zless another_file.vcf.gz
Ensuring Data Integrity
When you download a massive genome sequence from a database like , how do you know the file wasn't corrupted during the transfer? A single flipped bit could alter crucial data. This is where checksums come in.
A is a unique signature generated from the contents of a file. If even one character in the file changes, the checksum will change completely. Data providers often publish a checksum alongside their downloadable files. After you download the file, you can calculate your own checksum and compare it to the one provided.
The md5sum command is a common tool for this. You run it on your local file, and it produces a long string of characters. If that string matches the one from the database, you can be confident your data is intact.
| Command | Purpose |
|---|---|
pwd | Show current directory |
cd [directory] | Change to a new directory |
ls -lh | List files with human-readable sizes |
head [file] | View the beginning of a file |
tail [file] | View the end of a file |
zless [file.gz] | Browse a compressed file |
md5sum [file] | Calculate the MD5 checksum of a file |
Now, let's test your understanding of these essential navigation and inspection commands.
Which command would you use to find out your current location within the file system?
When using ls -l, what is the primary benefit of adding the -h flag (i.e., ls -lh) for a bioinformatician?
Mastering these commands allows you to handle the massive datasets of modern biology efficiently and safely.