Downloaded 20 times
![Exercise 7: for loop
for [condition ]
do
command1
command2
commandN
done
1. for ((j=1; j<=10; j++)) Set j equal to 1 and as long as j is
less than or equal to 10 execute the
loop followed by increasing j by 1.
2. do Do the following…
3. echo $j echo the value of j…
4. done and exit the loop if condition is
false.
space is critical
for ((j=1; j<=10; j++)); do echo $j; done](https://image.slidesharecdn.com/unix-170502192833/85/Introduction-to-UNIX-34-320.jpg)
![Exercise 7: for loop
for [condition ]
do
command1
command2
commandN
done
1. for ((j=1; j<=10; j++)) Set j equal to 1 and as long as j is
less than or equal to 10 execute the
loop followed by increasing j by 1.
2. do Do the following…
3. echo $j echo the value of j…
4. done and exit the loop if condition is
false.
space is critical
for ((j=1; j<=10; j++)); do echo $j; done](https://image.slidesharecdn.com/unix-170502192833/75/Introduction-to-UNIX-34-2048.jpg)
Introduction to UNIX
The document provides an overview of Unix basics and scripting. It defines what an operating system and Unix are, describes the Unix philosophy and directory structure, and covers shells, commands, writing and executing scripts, variables, loops, and file permissions. The key topics covered include the Unix philosophy of small, modular programs; the hierarchical directory structure with / as the root; common shells like bash and commands like ls, grep, sort; and how to write simple shell scripts using variables, conditionals, and loops.
More Related Content
More from Bioinformatics and Computational Biosciences Branch
Introduction to UNIX
- 1. Unix Basics andScripting Michael A. Dolan, Ph.D.
- 2. Outline • What isan operating system? • What is UNIX? • UNIX philosophy • UNIX directory structure • Shells • Basic, important commands (hands-on exercises) • Writing and executing a script
- 3. What is anoperating system? - consists of a group of programs which make a computer work - stable, multi-user, multi-tasking system for servers, desktops and laptops. Source: hubpages.com
- 4.
- 5. What is UNIX? 5 •UNIX is an operating system developed in the late 1960s • derives from the original AT&T Unix, developed at the Bell Labs research center by Ken Thompson, Dennis Ritchie, and others. • originally meant to be used for developing software to be run on multiple platforms; used more and more to run application software • over time, grew larger as the operating system started spreading in the academic community • users added their own tools to the system and shared them with colleagues • many different versions of UNIX, although they share common similarities • most popular varieties of UNIX are Sun Solaris, GNU/Linux, and MacOS X
- 6. UNIX “philosophy” 6 • portable •multi-tasking • multi-user in a time-sharing configuration • use of plain text for storing data • treating devices and certain types of inter-process communication (IPC) as files • use of a large number of software tools/small programs that can be strung together instead of using a single large program that includes all of the same functionality • small utilities with master program called the ‘kernel’ • everything in UNIX is either a file or a process • a hierarchical file system
- 7. UNIX directory structure •also known as ‘UNIX filesystem’ • the filesystem appears as a single rooted tree of directories • root of tree denoted as / • disk partitions, removable media, and network shares are part of a single tree - items can be “mounted” to tree • MS-DOS (Windows) has multiple trees with each drive letter being the root
- 8. UNIX directory structure “root” Source:www-uxsup.csx.cam.ac.uk/pub/doc/suse/suse9.0/userguide-9.0/ch24s02.html
- 9. Typical UNIX directorystructure 9Source: Andrew Oler / “root” /bin essential binaries /etc system config /home user directories /home/USER1 USER1 home /home/USER2 USER2 home /mnt network drives /sbin system binaries /usr shared, read-only /usr/bin other binaries /usr/local installed packages /usr/local/bin installed binaries /var variable data /var/tmp program caches administrator permissions user permissions
- 10. Basic format ofa UNIX system 10 kernel computer hardware user shell
- 11. What is ashell? 11 • a that provides the user with an interface to access an operating system's services (kernel) • can be command-line interface or graphical user interface • shell referred to as: UNIX/LINUX systems “shell” Mac “terminal” Windows “Program manager” or “Windows shell” program
- 12. Types of shells Operatingsystem Shell program command/name UNIX/LINUX, OSX sh (Bourne shell) csh (C shell) tcsh (TENEX C shell) ksh (Korn shell) bash (Bourne Again shell) zsh (Zhong shell) ash (Almquist shell) Windows Windows shell
- 13. Commands are usedto interact with shell which in turn interacts with kernel 13
- 14. Command in contextof shell command argumentshell prompt cursor output “Terminal” or “window” in which one can interact with shell program prompt shell $ sh, bash % > csh, tcsh % zsh # root
- 15. Anatomy of aUNIX command 15Source: http://sc.tamu.edu/help/general/unix/unix.html
- 16. Ten essential UNIXcommands 16 pwd print working directory pwd grep search for text or text within file grep –i mike /usr/dolanmi
- 17. Other Unix commands CommandFunction cat read a file or combine files chmod change file permissions chown change file ownership date display the date echo display result to output (file or screen) ftp (file transfer protocol) connect to remote machine and move files man manual pages head display first part of file less, more, most use to read file sort sort files or text tail display last part of file tar create an archive, add or extract file wc count characters, words, lines
- 18. Exercise 1: Navigatingin a shell 1. Open a terminal and type: pwd 2. ls 3. ls -aFlt 4. cd .. 5. pwd 6. ls 7. cd Desktop 8. pwd 9. cd ~ 10. pwd 11. echo $SHELL 12. cd Desktop (for next exercise) Try these too – what is different with each one? ls ls -a ls -l ls -al ls -lh ls -lhS ls -lhSr
- 19. Need help? manpages! • online software documentation usually found on a Unix or Unix-like operating system • topics covered include computer programs (including library and system calls), formal standards and conventions, and even abstract concepts. Type: man ls To exit, type: :q
- 20. zsh shell • Zhongshell • Extended Bourne shell (sh) with a large number of improvements, including some features of bash, ksh, and tcsh • used by Bio-Linux as default login shell • prompt can be defined • echo $SHELL /bin/sh • echo $ZSH_VERSION https://en.wikipedia.org/wiki/Z_shell
- 21. vi • a screen-orientedtext editor • the de facto standard UNIX editor…. ubiquitous • written by Bill Joy for UNIX in 1976 • name vi is derived from the ex command ‘visual’ which switches the ex line editor to visual mode • relatively steep learning curve, but a short hill • other text editors: TextEdit (Mac), Notepad++ (Windows), nano (terminal), pico
- 22. Exercise 2: Filemanipulation 1. Type: mkdir temp 16. rm file3 2. ls 17. mv file file.script 3. cd temp 18. ls 4. pwd 5. ls 6. vi file 7. Press 'i' 8. Type: echo your_name 9. Press <esc> 10. Type: :wq 11. ls 12. cat file 13. cp file file_orig 14. cat file file.orig > file3 14. ls 15. cat file3
- 23. Sorting - the sortcommand arranges lines of text …. alphabetically… by default - many options - man sort -n sort numerically (example: 10 will sort after 2) -r reverse the order of sort. -f sort upper and lower case together +x ignore first x fields when sorting.
- 24. Exercise 3: Sorting 1.echo “apple 10” > file.txt Write ‘apple 10’ to a new file called ‘file.txt 2. ls A file called ‘file.txt’ is present. 3. cat file.txt File contents listed. 4. echo “banana 11” > file.txt 5. cat file.txt What happened? 6. echo “apple 10” >> file.txt File appended 7. cat file.txt 8. cat “orange 5” >> file.txt File appended 9. cat file.txt 10. sort file.txt 11. sort –r file.txt 12. sort -k 2n file.txt 13. sort –k 2 -r -n file.txt
- 25. grep • name comesfrom ed command g/re/p (globally search a regular expression and print) • allows one to search for a string in a file • very powerful, especially when combined with other commands "string" = expression, text, pattern
- 26. Exercise 4: grep-ing 1.grep apple file.txt All lines with ‘apple’ are returned. 2. grep an file.txt All lines with ‘an’ are returned. 3. cp file.txt file2.txt 4. grep apple *.txt 5. grep –v apple *.txt Get everything that does not contain string ‘apple’
- 27. Bonus exercise: grepexample 1. Try: curl -L -o seq.txt http://bit.ly/2gZkJSs or wget http://bit.ly/2gZkJSs -O seq.txt 2. grep hypothetical seq.txt
- 28. Pipelining 28Source: Andrew Oler •a pipeline is a set of processes chained by their standard streams, so that the output of each process (stdout) feeds directly as input (stdin) to the next one program 1 | program 2 | program 3
- 29. Exercise 5: Pipelining 1.ls | wc –l Count the number of files in a directory 2. grep an file.txt 3. grep an file.txt | sort -r 4. grep an file.txt | sort -r > file3.txt 5. cat file3.txt 6. ls *.txt | grep fi | sort -r List the files that end in “.txt” and grep this output for the string “fi” and then reverse sort 7. ls *.txt | grep fi | sort –r | wc -l Do above and then count the items in the list.
- 30. Variables • programming languagesuse variables to store information • user can define variables … • and UNIX has predefined variables (“environment variables”) echo $SHELL echo $HOST echo $PATH echo $HOME echo $RANDOM Many others…..
- 31. Exercise 6: Definingvariables User defined Type: banana=cherry echo $banana banana=apple Press PgUp key to show 'echo $banana' UNIX defined echo $HOME echo $PATH
- 32. A note about$PATH The PATH environment variable is a colon delimited list of directories that the shell searches through when a user enters a command. Commands/programs (“executables”) can be in many different places. A user can append to the $PATH variable. echo $PATH cd /usr/localbin ls -aFlt cd /usr/bin ls -aFlt
- 33. Loops 33 • powerful programmingtools that enable one to execute a set of commands repeatedly as long as a condition is “true” • Types: while for until select
- 34. Exercise 7: forloop for [condition ] do command1 command2 commandN done 1. for ((j=1; j<=10; j++)) Set j equal to 1 and as long as j is less than or equal to 10 execute the loop followed by increasing j by 1. 2. do Do the following… 3. echo $j echo the value of j… 4. done and exit the loop if condition is false. space is critical for ((j=1; j<=10; j++)); do echo $j; done
- 35. shell scripting 35 Q: Whatis a shell script? A: A shell script is a file containing a number of UNIX commands that can be executed upon invoking or calling the script
- 36. Elements of aShell Script 36 • Interpreter ("shebang" line) #!/bin/sh • Comments Lines added to a script starting with # that are not executed but that help to document Example: #Created this script 2015-03-30 • Set variables (e.g., environment, input files, output files, etc.) • Commands to execute These are different although they start with #
- 37. 37 1. Open atext editor: vi script.sh Defines file name called ‘script.sh’ 2. i insert 3. #!/bin/sh “she-bang” 4. #This script is .... comments 5. for((j=1; j<=10; j++)) do echo "$j" done for loop 6. <esc> Leave insert mode 7. :wq write to file and quit vi 8. ls –aFlt script.sh list file details 9. chmod 755 script.sh Make the script executable 10. ls –aFlt script.sh list file details 11. ./script.sh Run script Exercise 8: Writing a shell script
- 38. File permissions andownership chmod - “change mode” - sets access to files and directories - accepts four octal digits describing access - three rightmost digits refer to permissions for the file owner, the group, and world chmod 755 = rwxr-xr-x chmod a=rw=file.txt chown - “change owner” - sets file and directory ownership chown root tmpfile
- 39. File tranfer: sftp 39 1.Type: sftp demo@test.rebex.net 2. password = password 3. ls 4. get readme.txt 5. bye • used to transfer files between computers To upload a file, use ‘put’ • secure file transfer protocol
- 40. Resources 40 • http://linuxcommand.org/lc3_writing_shell_scripts.php • “UnixShells By Example.” Ellie Quigley. Prentice Hall. • “bash Cookbook” Carl Albing, JP Vossen, Cameron Newham. O’Reilly. • “Classic Shell Scripting” Arnold Robbins, Nelson H.F. • Beebe. O’Reilly.
- 41. Recommended Reading Linux ina Nutshell, 6th Edition Ellen Siever, Stephen Figgins, Robert Love, Arnold Robbins Running Linux, 5th Edition Matthias Kalle Dalheimer, Matt Welsh UNIX® Shells by Example, Fourth Edition Ellie Quigley From Andrew Oler
- 42. Summary 42 • Read “man”pages • Practice, practice, practice • Start with existing script when writing your own • Everything is on the internet • Reference books are worthwhile • Work on copies, make backups, and use “rm”, “mv”, and “>” carefully • Pick a text editor and master it (pico/nano, emacs, vi/vim, etc.)
- 43. File and systeminformation 43 wc file1 wc *.fastq “word count”; output is “lines”, “words”, characters” “word count” of all fastq files, including summary df –k “disk free” (size) top report for local machine on the processes using the most system resources (memory, CPU, etc.); “q” to exit Source: Andrew Oler
- 44. File compression gzip temp/*compress every file in “temp”; adds .gz extension gunzip temp/*.gz expand every “gzipped” file in “temp” temp/* create a single archive of every file in “temp” tar -zcvf myfiles.tar.gz tar –xvf test_data.tar.gz copy every file out of the archive 44
- 45. More file manipulation.... catfile1 file2 > file3 >> file2 write “file3”, containing first “file1”, then “file2” append “file1” onto “file2” will write file contents to shell terminal cat file1 cat file1 Careful ! 45 sort file1 alphabetize “file1.txt” sort -n file1 sort “file1” by number sort -n -r -k 2 file1 sort “file1” by the second word or column in reverse numerical order
- 46. shortcuts 46 ln -s /usr/localdata make a shortcut in current directory pointing to /usr/local cd data takes you to /usr/local Linking files: Creating aliases: alias fs="ftp anonymous@speedtest.tele2.net" Note: To have an alias defined each time a shell is opened, place the aliases in the $HOME/.bashrc file (for BASH shells) ‘fs’ is the alias that is created. Typing fs while invoke the command ftp anonymous@speedtest.tele2.net.
- 47. Tips to makelife easier! tab completion - press tab key to allow for completion of command or filename - recall the previous command(s) history - show every command issued during the session !ls !! - repeat the previous “ls” command - repeat the previous command Source: Andrew Oler
- 48. Biowulf @ NIH •https://hpc.nih.gov/docs/userguide.html
- 49. I/O 49 §STDIN • keyboard bydefault §STDOUT • terminal screen by default §STDERR • terminal screen by default §Redirecting STDOUT • > vs. >> §Redirecting STDERR §2> §Redirecting STDOUT + STDERR (depends on shell) • &> §Pipe – send STDOUT as STDIN to next command • | • ls | grep “.fasta” | wc
- 50. wget download multiplefiles from ftp or http address curl download single files from ftp, http, sftp, etc. http://curl.haxx.se/docs/comparison-table.html
- 51.