Research. We make it happen.

www.msi.umn.edu

Unix Basics

Presented by: Patton Fast

User Support Group help-line: help@msi.umn.edu personal: pfast@msi.umn.edu 612-626-0802 612-625-6573

Outline
www.msi.umn.edu

I. Warnings! II. Basic Concepts III. Basic Commands IV. The Fun Stuff V. Resources

Please Ask Questions!

Warnings!
www.msi.umn.edu

If you delete a file, it is GONE! The file can only be retrieved from a backup. If you overwrite a file it has been changed forever. The original file can only be restored from a backup. Home directories are backed up nightly. The scratch spaces are NOT backed up!

Bottom line? Be Careful With Your Files!

Warnings!
scratch space
www.msi.umn.edu

Scratch Space Time Limits!

Regatta (/regscratch1 and /regscratch2) 7 days, 50 Gb SP (/scratch1 and /scratch2) 7 days, 50 Gb Altix (/scratch1 and /scratch2) 14 days Netfinity (/scratch1 and /scratch2) 14 days SDVL (/scratch) 14 days BSCL (/wrk) 14 days

Bottom line? Be Careful With Your Files!

Warnings!
service units
www.msi.umn.edu

Core Resource Service Units

IBM Power4 3.5 hours / su SP Winterhawk+ 5 hours / su SP Nighthawk 10 hours / su Altix 3.5 hours / su Netfinity 12 hours / su

Basic Concepts
core resources
www.msi.umn.edu

IBM Power4
364 Power4 processors 603 Gb of memory 6 Tb of storage

SGI Altix
432 Itanium2 processors 876 Gb of memory 4 Tb of storage

IBM Linux Cluster

160 Intel processors 226 Gb of memory 2.5 Tb of storage

IBM SP
400 Power3 processors 675 Gb of memory 4 Tb of storage

Basic Concepts
laboratory resources
www.msi.umn.edu

Basic Sciences Computing Lab

48 CPU SGI Altix 16 CPU SGI Onyx4 9 SGI Octanes 8 CPU Sun UltraSparcIII 2 Linux Workstations 1 SunBlade 2000 10 Tb of storage 64 x 80 stereo system

Med. Chem./Supercomputing Inst. Vis. Workstation Lab

3 SGI Octanes 3 SGI O2s 2 SunBlade 2000s

http://www.msi.umn.edu/vwl

http://www.msi.umn.edu/bscl

Scientific Dev. & Vis. Lab

9 SunBlade 2000s 7 SGI Octanes 1 SGI Onyx

Computational Genetics Lab

16 4 1 1 1 1 CPU SunFire 6800 Dell PCs Mac G5 SGI Fuel SGI Tezro IBM IntelliStation

http://www.msi.umn.edu/sdvl

http://www.msi.umn.edu/cgl

Basic Concepts
laboratory resources
www.msi.umn.edu

Basic Sciences Computing Lab Proteomics Structural Biology Computational Genetics Lab Bioinformatics Genomics Lab for Large Scale Data Analysis Data Mining Database Modeling Med. Chem./MSI Visualization Workstation Lab Drug Design Molecular Modeling Scientific Development & Visualization Lab Computational Fluid Dynamics Scientific Computation

Basic Concepts
changing passwords
www.msi.umn.edu

After your first login you should change your initial password to something that is easy for you to remember, but not easy for someone to guess. %> yppasswd %> passwd (Institute machines) (general)

You should NEVER give your password to anyone! If you forget your password, send a message to help@msi.umn.edu and your password will be reset.

Note: Never share accounts or passwords!

Basic Concepts
changing passwords
www.msi.umn.edu

Initial Passwords
Your initial password should be the same for all machines within a given lab, regardless of operating system.

Unix, Linux, Mac OS X

All of the Unix based machines share the same password file. When you change your password on one of the machines within a given lab, it is changed for all of the machines.

Windows
All of the Windows bases machines share the same password file across all labs. When you change your password on one machine it is changed for all machines regardless of the lab.

Basic Concepts
console

Console Login:

www.msi.umn.edu

Enter username Enter password

Using the keyboard and mouse physically attached to the machine. Console Logout: Logout procedures vary between consoles. Make sure you logout! Otherwise your files and account are open to the next person. You could be logged out by an administrator.

Basic Concepts
remote login
www.msi.umn.edu

ssh [options] machine

secure shell; a program for logging into a remote machine providing encrypted communications between machines login name sets environment variables for porting X-display

options: -l -X example:
type: ssh -l pfast sp type: ssh bi2

open a secure connection for the user pfast on the machine sp.msi.umn.edu open a secure connection to bi2.msi.umn.edu with current username

SSH clients: http://www.msi.umn.edu/user_support/ssh

Basic Concepts
forwarding email
www.msi.umn.edu

Why? So that you get all of your email sent to one account. Makes it easier for sys admins to communicate with you. How? 1) Create the file .forward in your home directory 2) On the first line: your.email@desired.location 3) Create this same file in all of your accounts that you want to forward.

Basic Concepts
command line help
www.msi.umn.edu

man [options] command displays manual pages about command

-k keyword search for a command

info command

displays Info document of command

When in doubt, check the man/info pages! Need extra details? Check the man/info pages!

Basic Concepts
file structure overview

/ bin@ user1/ file1 .cshrc mail/ source/ work/ user1/ proj1/ proj2/ input output directory ( / ) executable ( * ) link (@) dir1/ home/ user2/ lib@ scratch/ bin/

www.msi.umn.edu

usr/ lib/ local/ more* user2/ dir2/

file1/

file2/

Basic Concepts
metacharacters
www.msi.umn.edu

? * ~ ~name . ..

single character wild card wild card, arbitrary number of characters home directory of current user home directory of user name current directory parent directory (back one)

Basic Concepts
command anatomy
www.msi.umn.edu

command [options] argument example: [options] (usually preceded by a dash)

ls -al /scratch
command name argument (quite often a filename or directory)

Note: UNIX commands, options, and arguments are all CasE seNsiTive!

Basic Commands
Command ls [options] dirname pwd mkdir dirname cd dirname mv [options] source destination cp [options] source destination rm [options] source ssh [options] hostname scp [options] user@host1:file user@host2:file Description
www.msi.umn.edu

list the contents of dirname display full pathname create the directory dirname change to the directory dirname move a file or directory copy a file or directory remove a file or directory remote login remote copy; file transfer

skip basic commands

Listing Contents
ls [options] dirname options: -a -l examples:
type:

list the contents of dirname

www.msi.umn.edu

list all files including hidden files [hidden files are preceded by a .; eg .cshrc] long listing showing ownership, permissions, and links

ls /home/user/temp view the contents of a directory with absolute pathname /home/user/temp ls ../../temp list the contents of a directory using a relative path.

type:

Make Directory
www.msi.umn.edu

mkdir dirname examples:

type: mkdir work

create the directory dirname

create the directory work/ in the current working directory create the directory proj1/ in the work/ directory create the directory user2/ in the /wrk/ directory

type: mkdir work/proj1 type: mkdir /wrk/user2

Change Directory
www.msi.umn.edu

cd cd dirname examples:
type: cd ~tom type: cd /wrk/user2

change to your home directory change to the directory dirname

change to toms home directory change to the directory /wrk/user2

Moving or Renaming
www.msi.umn.edu

mv [options] presname newname mv [options] srcfile destdir options: -i example:

type: mv ~user1/file ./outputfile

rename a file move a file to another directory

confirm overwrites

moves file from user1s home directory to the current working directory and renames it output file

Note: Be careful when overwriting files!

Copying
www.msi.umn.edu

cp [options] srcfile destfile cp [options] srcfile destdir options: -i -R example:

type: cp -R proj1 proj2

copy a file to another filename copy a file to another directory

confirm overwrites recursively copy a directory and its contents, copies symbolic links

copy the directory proj1/ and name it proj2/

Note: Be careful when overwriting files!

Copying
www.msi.umn.edu

cp [options] srcfile destfile cp [options] srcfile destdir options: -i -R example:

copy a file to another filename copy a file to another directory

Tip:
If you have large files to confirm overwrites move from the scratch recursively copy a directories,copy (cp) themdirectory and its contents, copies rather than moving (mv) symbolic links them.
copy the directory proj1/ and name it proj2/

type: cp -R proj1 proj2

Note: Be careful when overwriting files!

Removing
www.msi.umn.edu

rm [options] filename options: -f -i -r example:

type: rm -rf /scratch/user2/dir1

remove a file

remove without prompting confirm removal recursively remove a directory and its contents

remove the directory /scratch2/user2/dir1 and its contents

Note: Be careful when removing files! Especially with wild cards (*,?)!

Remote Copy
www.msi.umn.edu

scp [options] host:file1 host:file2 secure copy; copy files between hosts on a network using the ssh protocol options: -r example:
type: scp pfast@wind:output/*.out . type: scp -r mail origin:mail_copy

recursive (used for directories)

copy the output files from host wind with username pfast to current directory copy the mail directory to the origin and rename the directory mail_copy

Note: You can use sftp if scp is not available.

Remote Copy
www.msi.umn.edu

scp [options] host:file1 host:file2 secure copy; copy files between hosts on a network using the ssh protocol options: -r

Tip:

recursive (used for directories) If you have large files to example: transfer, use sftp instead of scp. type: scp pfast@wind:output/*.out . copy the output files from host wind
with username pfast to current directory
type: scp -r mail origin:mail_copy

copy the mail directory to the origin and rename the directory mail_copy

Note: You can use sftp if scp is not available.

Changing Permissions
chmod [options] who ops permission filename change file permissions use ls -l to view permissions
www.msi.umn.edu

options R recursively change permissions

who can be any combination of: u (user) o (other/world) g (group) a (all or ugo)

ops adds or takes away permission, and can be: + (add permission) (remove permission) permission can be any combination of: r (read) w (write) x (execute) X (check user first)

Command Repetition
history !string up/down arrows left/right arrows Control-E Control-A examples:
type:

display list of most recent commands scroll up and down through most recent commands; csh and tcsh use for editing current line end of line beginning of line

www.msi.umn.edu

repeat command beginning with string

!ls

repeat last ls command

Command Completion
www.msi.umn.edu

TAB or Control-D

finish the current command, filename, or directory if possible, or show the possible completers (works with c and tc shells)

Tarring
www.msi.umn.edu

tar [options] files options: c x f v example:

type: tar xvf file.tar

collect or extract files or directories into or from an archive create a new archive extract files from archive store files in archive print function letter and name of files store to standard output
extract the files and or directories from the archive, file.tar

Compression
www.msi.umn.edu

compress file

compress file and rename it file.Z (the program automatically renames the file) uncompress file.Z and rename it file compress file and rename it file.gz (the program automatically renames the file) uncompress file.gz and rename it file

uncompress file.Z gzip file

gunzip file.gz

Process Control
www.msi.umn.edu

Control-C Control-Z jobs bg fg & example:

cancel a foreground job stop (interrupt) a foreground job list of background jobs run stopped job in the background run stopped job in the foreground appended to the end of a command will place that job in the background
run WU-BLAST job in the background

type: blastp swissprot test.seq > test.out &

Process Control
www.msi.umn.edu

ps [options] options: -a -u jobs

display the status of the current processes and the process id-number all processes display processes owned by a particular user shows any jobs that are currently running in the background or suspended

Process Control
www.msi.umn.edu

kill 9 id-number

terminate a process owned by you; idnumber (process id) can be found with the ps command

or kill %job-number terminate a process owned by you; jobnumber can be found with the jobs command

Other Commands
www.msi.umn.edu

du -sk file df stat file quota -v

display total disk usage of file or directory in kilobytes display the current disk usage for all connected file systems view attributes of a file or directory list your disk quota

Viewers
www.msi.umn.edu

cat [options] files more [options] files less file

dumps the contents of files to the screen a pager, that lets you view one page at a time similar to more, but more

Editors
www.msi.umn.edu

Graphical: nedit filename xemacs filename Command-line: pico filename vi filename full-screen text editor available on most UNIX systems full-screen text editor available on virtually every UNIX system if not available, ask to have it installed

The Fun Stuff

redirect
www.msi.umn.edu

> >> < examples:

how to: type:

command output redirection (create new, overwrites existing file) command output redirection (append) command input redirection (from file)

combine individual sequence files into one file cat seq1 seq2 > seq

how to: email a file to a collaborator type:

mail -s results collab@umn.edu < new_blast_results

The Fun Stuff

piping
www.msi.umn.edu

command1 | command2 |

directs standard output of one command into standard input for the next command

examples:
type: ls -al | more type: qstat -a | grep user1

look at the ls output one page at a time check the queue on the origin for any processes being run by user1

The Fun Stuff

pattern searching
www.msi.umn.edu

grep [options] string file options: -i examples:

find and print out lines containing the string in a file or multiple files

ignore case

how to: extract sequence information from protein structure files type:

grep SEQRES pdbfile | cut -c20-70 > seqs

The Fun Stuff

.cshrc file
www.msi.umn.edu

.cshrc Some things to add to your .cshrc file: alias alias-string com-string examples:
type: alias la ls -a type: alias q qstat -u pfast

shell startup file

abbreviate a command string with an alias string

create a shortcut for ls -a check if you have any jobs running in the queue (origin) load or unload the environment settings for a particular software runs the gcgstartup script to set the necessary env. variables

module load(unload) software source /usr/local/gcg/gcgstartup

Resources
www.msi.umn.edu

UNIX in a Nutshell by OReilly Developing Bioinformatics Computer Skills by OReilly http://www.iats.missouri.edu/servlets/knowledgebase/article/422 http://www.geek-girl.com/unix.html http://riceinfo.rice.edu/Computer/Documents/Unix/unix1.01.pdf

Questions?
www.msi.umn.edu

email: help@msi.umn.edu phone: 612-626-0802