MODULE 1:
INTRODUCTION
Linux & bash shell
I. Linux
2
What is Linux?
Linux is an operating system (OS) developed
by Linus Torvalds in 199
Based on UNIX - developed in response to Tux
closing legal loophole that made UNIX fre
Many “distributions” - Fedora, RedHat,
CentOS, Debian, Ubunt
Linus Torvalds
Typically free and open source GNU licensin
Command line interface (CLI) and graphical
desktop environments (GDE)
Richard Stallman 3
u
Why Linux?
Developed by Bell Labs in 1969, and initially free, UNIX
was quickly adopted as de facto scienti c computing O
Powerful CLI enables direct low level access
GDE provides simplicity and usabilit
Free and open source makes code development eas
Linux is everywhere
- 90% of supercomputers run Linux (incl. Blue Waters)
- Android OS is based on a Linux kernel
- Ubuntu distro is the most popular OS in the world
4
y
fi
y
Can’t I use Windows / Mac OS X?
Maybe
Some software have Mac OS X / Windows / Windows +
Cygwin versions to install on your local machin
Remote login via Mac OS X terminal / [Windows +
Cygwin / Putty] to SSH into EWS Linu
A key learning objectives of this course is to develop
familiarity and competence using Linux - Bon Courage!
5
.
What distro are we using?
EWS Linux machines run CentOS 7
6
II. bash shell
7
The command line
CLI and GDE offer alternatives to interact with a machin
Switching to a CLI can be very intimidating for new users
CLI interaction is powerful, concise, and ef cien
CLI scripting enables task automatio
e.g. Download of 1500 daily NASDAQ stock prices
GDE: Point and click le download extremely tedious!
CLI Trivially automated using CLI wget loop
8
:
fi
n
fi
t
bash shell
“Command line interpreters”
or “shells” convert text inputs
into OS command
Many avors: sh, bash, ksh, csh,
zsh, tcs
The bash shell (“Bourne-again
shell”) is one of the most
popular, and the default on
many Linux distros
9
fl
h
III. bash basics
10
bash: basics
Pop a bash terminal by clicking on
or navigating Applications è Accessories è Termina
pwd - show path to present working director
ls - list contents of current director
ls -alh - list all contents of cdir in long form with
human readable le size
ls /sw/q - list contents of directory /sw/
cd <path> - change directory into <path
cd .. - change directory up one leve
cd ../.. - change directory up two levels
11
fi
s
>
bash: basics
touch <file> - make new le < le> or update last
access of existing l
mkdir <dir> - make director
chmod 755 <file> - change le permissions to
r+w+x (user), r+x (group, world)
chmod 644 <file> - change le permissions to
r+w (user), r (group, world)
[N.B. r=4, w=2, x=1
var=ferrari42 - assign ferrari42 to var
echo $var - print $var
12
fi
fi
fi
y
fi
fi
e
bash: basics
./<execFile> - execute execFile in cdir
<path>/<execFile> - execute execFile in pat
which <cmd> - location of command cm
clear - clear termina
wget -O <file> <url> - download url data into le
e.g wget -O myProf.png http://bit.ly/2jt9NAl
13
.
l
fi
d
bash: basics
cp <source> <target> - copy le source to target
e.g. cp myFile /apps/doc/
cp -r <source> <target> - copy recursively
(copy source directory and everything in it)
e.g cp -r myDir ./dir1/dir2/
mv <source> <target> - move source to target
(same for les and directories
rm <file> - remove le
rm -r <directory> - remove recursively directory
14
.
fi
fi
fi
)
bash: safety!
cp / rm / mv - These do exactly what you ask
They do not ask for permission
Furthermore, there is no Trash/Recycling
Once you remove / overwrite a le, it’s gone
Standard “safety” choices: use alias in your .bashrc
alias cp=‘cp -i’
alias rm=‘rm -i’
alias mv=‘mv -i’
setopt noclobber
You don’t have to do this, but you may breathe a little
easier with some safety.
15
fi
.
bash: basics
whoami - show your login username
who - show everyone currently logged in
cat <file> - show le content
less <file> - show le contents (spacebar ê, b é)
head <file> - show head of le
tail <file> - show tail of le
tail -n <nLines> <file> - show tail nLines of le
tail -f <file> - show tail of le and follow
16
fi
fi
fi
fi
fi
s
fi
bash: basics
zip <archive> <file1 file2 ...>
- create zip le archive.zip containing le1, le2, ...
unzip <archive>
- unzip zip le archive.zi
tar cvzf <archive.tgz> <file1 file2 ...>
- create gzip compressed tape archive archive.tgz
containing le1, le2, ...
tar xvzf <archive.tgz>
- uncompress end extracted compressed tape
archive archive.tgz
17
fi
fi
fi
fi
p
fi
fi
bash: basics
top - show active processes
top -o cpu - show active processes ordered by cpu %
top -U <usr> - show active processes owned by us
grep <str> <file> - return lines in le containing
string st
find <path> -name <*str*> -print
- print all les in path containing str in their nam
<cmd> --help - help for cmd
man <cmd> - manual for cmd (spacebar ê, b é
Google is your friend for bash help! 18
fi
r
fi
e
bash: special symbols
~ - your home director
. - current director
.. - directory one level u
* - wildcard characte
\ - escape succeeding character
e.g mkdir My\ Directory
| - pipe
e.g. cat <file> | grep tungsten 19
.
y
bash: special symbols
> - redirect standard output and overwrite
>> - redirect standard output and append
e.g. echo “Today was great!” >> myDiary.txt
$var - dereference variable va
“ ” - enclose text string but expand $
‘ ’ - enclose text string but do not expand $
e.g. myVar=“My String With Spaces”
echo “This is $myVar”
`<stuff>` - execute stuff rst
e.g. echo `expr 1 + 1` 20
fi
r
IV. bash utilities
21
bash: integer arithmetic
expr - integer arithmetic engin
e.g
$ echo `expr 1 + 1`
2
$ var1=`expr 10 \* 2`
$ var2=`expr 21 / 7`
$ echo $var1 $var2 `expr $var1 / $var2`
20 3 6
22
.
bash: quick calculator
bc -l - arbitrary precision calculator (w/ math lib
$ bc -l
2/3
.66666666666666666666
2^3
8
e(1)
2.71828182845904523536
pi=a(1)*4
pi
3.14159265358979323844
s(pi/6)
.49999999999999999999
c(pi/6)
.86602540378443864676
23
bash: ssh & scp
SSH CLI remote login is supported by ssh (secure shell)
ssh <user>@<hostname> - login to host
ssh -Y <user>@<hostname> - login to host w/
secure X forwarding (use this to get graphics via SSH!
N.B. For EWS, hostname=remlnx.ews.illinois.edu
SCP CLI le transfers supported by scp (secure copy)
scp <src> <user>@<hostname>:<target>
- upload
scp <user>@<hostname>:<src> <target>
- download
24
fi
)
bash: ssh & scp
ssh and scp are prepackaged with Linux / Mac OS X and
are accessible directly from the bash termina
On Windows, you need to download a third party ssh
client in order to make a ssh connection with EWS
https://answers.uillinois.edu/
www.putty.org
illinois.engineering/page.php?id=81727
25
l
bash: sftp
SFTP more sophisticated alternative to scp
(secure file transfer protocol)
sftp <user>@<hostname> - login to host
ls - remote ls
lls - local ls
pwd - remote pwd
lpwd - local pwd
cd - remote cd
lcd - local cd
get <file> - download le
put <file> - upload le
quit - logout
26
fi
fi
bash: vi/vim
Two built-in CLI text editors: vi/vim & emacs
Seem slow and painful, but invaluable for on-the- y edit
Use whichever you prefer, I use both.
(It is very fashionable to argue over which is better...
vi/vim is fast for text manipulation, uses two mode
emacs is has lots of built-in modules, more “Word”-lik
Two-modes navigation for moving
insertion for editin
Nav mode is the default mode, and can be accessed by
hitting Esc
Ins mode is accessed by hitting i 27
:
g
fl
)
bash: vi/vim
Nav mod
éêçè - single char / single line movemen
gg - go to top of le
^ - go to beginning of lin
$ - go to end of lin
<n>G - go to line
w - skip forward one wor
b - skip backward one wor
yy or y$ - copy current lin
y<n>ê - copy next n lines
p - paste
28
e
fi
e
bash: vi/vim
Nav mod
x - delete characte
o - create new line below and enter insert mod
i - enter insert mode to left of current characte
I - enter insert mode at beginning of lin
a - enter insert mode to right of current characte
A - enter insert mode at end of lin
29
e
bash: vi/vim
Nav mod
dd or d$ - delete current lin
d<n>w - delete next n word
d<n>ê - delete next n line
u - und
Ctrl+r - red
30
e
bash: vi/vim
Nav mod
/<str><Enter> - search forward for st
?<str><Enter> - search backward for st
n - go to next matc
<N>n - go to Nth matc
31
e
bash: vi/vim
Nav mod
:w - writes le
:w! - writes le even if read only
:q - quit
:q! - quit and don't question me
(good way to mess things up)
:wq - write quit
:wq! - write quit and don't question me
(very good way to mess things up)
32
e
fi
fi
bash: vi/vim
Ins mod
Type normally - what you enter appears on screen
éêçè work as in nav mod
Hit Esc to get back to nav mode
33
e
bash: .bash_pro le & .bashrc
Hidden les start with .
~/.bashrc is executed for every new termina
~/.bash_profile is executed when you login
(~/.bash_profile calls ~/.bashrc
These les are useful to store aliases and modify PATH
N.B. On some systems ~/.bash_profile is replaced
by ~/.profile
34
fi
fi
fi
)
bash: .bash_pro le & .bashrc
(i) Use vi to add lls as alias for ls -al to .bashr
$ vi ~/.bashrc edit .bashrc
$ G go to end of le
$ o edit line belo
$ alias lls="ls -l" add alia
$ Esc escape to navigate mod
$ :wq write and quit
35
fi
s
fi
c
bash: .bash_pro le & .bashrc
36
fi
bash: .bash_pro le & .bashrc
(ii) Use vi to add ~/local/bin to your PATH in .bashr
$ vi ~/.bashrc edit .bashrc
$ G go to end of le
$ o edit line belo
$ export PATH=$PATH:~/local/bin add to PAT
$ Esc escape to navigate mod
$ :wq write and quit
37
fi
w
fi
c
bash: .bash_pro le & .bashrc
38
fi
bash: installing software
Typical anatomy of an installation from source
$ wget <app_url> download
$ tar xvzf <app.tgz> uncompress
$ cd ./app
$ ./configure --prefix=<location>
con gure and specify locatio
$ make compile
$ make install instal
39
fi
l
:
V. bash scripting
40
What is bash scripting?
A bash script is nothing more than a list of bash
commands in an executable text l
Exactly the same behavior could be achieved by copying
and pasting the script into the bash shel
Extremely powerful way to automate system tasks
e.g le downloads
system backups
job submission
le processing
41
.
fi
fi
fi
e
Anatomy of a script
A script is nothing more than a text le
- write using vi, emacs, Notepad, or favorite text editor
the “sha-bang” line
comments (start with #)
list of bash commands
42
fi
Script 1: hello world!
$ touch helloWorld new script le
$ chmod 755 helloWorld making executable
$ vi helloWorld edit line belo
$ i enter insert mode
$ #!/bin/bash <Enter>
$ # this is my hello world script <Enter>
$ echo “Hello World!”
$ Esc escape to navigate mod
$ :wq write and quit
43
fi
w
Script 2: backup
Passing variables $1, $2, $3, ...
Placing all les in current directory into a
compressed tape archive bkp.tg
Renaming bkp.tgz bkp_<arg>.tgz where
arg is the rst argument in the call to the
executable
44
fi
fi
z
Script 3: summer
while loo arithmetic comparisons
$# and shift
Initializing sum to
while loop - run loop while the variable
$# is greater than
- $# = number of parameters in exec cal
- shift = kick out $1 and shift rest down
(i.e. $1 ç $2, $2 ç $3, $3 ç $4, ...
- arithmetic comparisons:
-l
-g
-l <
-g >
-e =
-n !=
45
t
e
t
e
q
e
<
>
Script 4: oracle
if/else statemen
nesting
if loo
nested if loo
- can also use the construct
if [ ] ; the
elif [ ] ; the
elif [] ; the
els
46
f
e
Script 5: calculator
case conditional
exit
defensive programming
safeguard on usag
- exit terminates scrip
case conditiona
- starts case, ends esac
- ) terminates pattern matc
- ;; terminates each cas
- | is the “or” characte
- * is the wildcard “catch all”
47
l
Script 5: calculator
48
Script 6: stringer
array
$@
Create an array strArray from parameter
- $@ = all parameters passed to bash cal
- ${ARRAY[@]} = array content
Create empty array leArra
For all strings except “virus” append txt
and store in leArra
- ${#ARRAY[@]} = array siz
- “” terminates $ dereference strin
- str comparisons
equal
! not equa
greater tha
< less than
-n <str not empt
-z <str empt 49
=
>
=
s
>
>
fi
:
y
fi
y
l
y
n
Script 7: ler
in nite loo read user input
Ctrl + C
In nite loop (Ctrl + C to break
Read user input into st
Test if str is a regular le in the
present working director
- le comparison operators
- le exists (may be directory
- le exists (not directory
- directory exist
- le readabl
- le writabl
- le executable
50
fi
fi
e
f
d
r
w
x
fi
fi
fi
fi
fi
fi
e
fi
fi
r
Script 8: squarer
iteratin sleep
functions
Declaring a function at top of scrip
As for main function $1,$2,... are passed variable
Setting up the iterative loo
Performing square using our functio
sleep 0.5 = 0.5 s pause between print
incrementing loop variable
51
g
VI. awk
52
awk
awk is a programming language in its own righ
Developed at Bell Labs in 70’s by Aho, Weinberger, &
Kernigha
Powerful, simple, fast and exible languag
Standard part of most Linux distributions, used primarily
for rapid and ef cient line-by-line text processing
53
n
fi
fl
e
Why awk?
“Forget awk, I’ll just use vi / emacs / Notepad!
OK, good luck...
- extract the third column of this 50,000 line le
- divide the second eld of each line by the seventh, and
save results in csv format
- extract every 15th line of this le and invert the eld
ordering to run from last to rs
awk can do these things (and many others!) extremely
ef ciently and quickly using “one liner” commands
integrates seamlessly into bash shell cat <file> | awk ...
integrates seamlessly into bash script
great power using only a handful of command
is simply the “right tool” for many text processing jobs 54
fi
fi
fi
fi
t
fi
s
fi
awk basics
Rudimentary awk, comprehensive beginner’s tutorial at:
http://www.grymoire.com/Unix/Awk.html
Anatomy of an awk progra
awk 'BEGIN { ... Do stuff before starting [optional
{ ... Line-by-line processin
END { ... }' Do stuff after end-of- le [optional
inFile > outFil Read from inFile, write to outFile
Can place within a script, or enter directly into termina
White space doesn’t matte
55
}
fi
g
awk basics
Alternatively, can pipe input from termina
cat inFile > awk 'BEGIN { ...
{ ...
END { ... }
> outFil
Omit “> outFile” to output directly to termina
Use “>>” instead of “>” to append rather than overwrit
cat inFile > awk 'BEGIN { ...
{ ...
END { ... }
>> outFile
56
}
'
'
What goes in the { }?
Commands perform line-by-line text processin
Assignment of internal awk variables
Flow control and loops
Pulling in of bash variables from surrounding scrip
Printing to terminal or l
Basic arithmeti
57
c
fi
e
[bash + awk] script example
Extract the x,y,z coordinates of peptide atoms from pdb
formatted les peptide[1-3].pdb into coords[1-3].txt
Concatenate coords[1-3].txt into coords_concat.txt
Use bash to iterate over le
Use awk to perform text processing
58
fi
fi
s
[bash + awk] script example
setting up in and out le
setting up iteratio
initializing concat l
awking each le in tur
- printf is formatted print
- formatting like Matla
- $n are eld code
- NR is a special variable
for number of records
= number of line
cat each le into concat l
rm each coordsX.txt le
increment iterator
59
fi
fi
fi
fi
n
fi
fi
n
fi
e
[bash + awk] script example
Doing this [by hand / in Excel / in Matlab] at any signi cant
scale would be extremely tedious and error prone!
60
fi
