CSI223

Systems Programming

Shell programming (b)

(B. Gopolang)
 Outline
• Redirection and Piping
• String Manipulation
• File Manipulation

2
Redirection & piping
 Redirection
• Refers to allowing standard streams ( e.g. keyboard
and screen) to be diverted to user-specified
locations.
• Very common in most Unix shells

• Usually done by inserting certain characters

between commands

4
• Syntax:
a) command1 > file1
• Command1 executes, and its output sent to
file1 (instead of the screen)
• E.g: grep ‘bash’ infile > outfile.txt

Note:
• > : will override contents of outfile.txt
• >> : will append to outfile.txt

5
b) command1 < file1
• Command1 executes using input provided
in file1 (instead of using keyboard input)
• E.g: grep ‘bash’ infile > outfile.txt

c) command1 < infile > outfile

• Command 1 executes using data from infile
and sends output to outfile
• E.g: grep ‘bash’ infile > outfile.txt

6
 Piping
• Programs can be run together such that one program
reads input from another with no need for an explicit
intermediate file:
Syntax:
command1 | command2
• Command1 is executed, its output is passed to
command2 where it is used as input
• This is called piping
• "|" character is referred to as a "pipe“
• It allows us to chain commands in the script
7
• Examples of piping
• ls | grep “.sh”
• ls output passed to grep, which filters list of
files to display those ending with .sh only
• echo “Good morning friend”|wc –w
• Output of echo command is piped into wc –w
• Output?
• ls /csi223/tests | grep ‘.txt’
• Output?
8
 Chained Pipelines
• Complex commands can be created by using
redirection and piping
• Example:
• ls | grep '.sh' | sort > shlist
• Sends the list of files in current directory to
grep, grep filters it to only contain lines that
have .sh, output sorted alphabetically, and
sent to file called shlist.
• cat arrays.sh | grep ‘echo’ | wc –l >> out.txt
• Output ??
9
String Manipulation in Bash
• Can be done in two ways in Shell scripting:

a) Using Bash string manipulation features

• Features inbuilt into Bash for handling
string values

b) Using external commands

11
a) Using Bash features
i. Assigning a value to String variable
• Ex:
• dept=”Computer Science” # ‘Computer Science’
ii. Accessing string value:
• Like other variables, use $
• E.g.:
• echo $dept # or echo $”dept” or echo ${dept}

12
iii. Getting string’s length:
• add ‘#‘ before string variable
• Syntax:
${#variablename}

• Example:
echo “It has: ${#dept} characters
Output
It has: 16 characters
13
iv. String concatenation
• List string variables, one after the other
• Syntax:
• var=${var1}${var2}
• # OR var=$var1$var2 OR var="$var1""$var2"
• Example
• n1=‘Computer ’
• n2=‘Science’ Output
• dname= ${n1}${n2} ComputerScience
• echo $dname 14
• You can also concatenate strings using += operator
• Example
string=“CSI"
course+=“223"
echo “$course"

15
• You can add some characters between strings
• How?
• Add the character between concerned variables
• E.gs:
dname= ${n1}” “${n2}
echo $dname Output
Computer Science
• Try to omit “”
• var=${var1}..${var2} # adds “..”

16
v. Extracting a substring
• Syntax:
• ${string:pos}
• substring from $pos till the end
• ${string:pos:len}
• substring $len characters from $pos

17
• Important:
• len and pos MUST be valid
• >= 0 & within range
• Examples:
• echo ${dept:5}
Output
• echo ${dept:5:6} ?
• echo ${dept:29} ?
?
• Try –ve number

18
vi. Pattern matching
• Rem: command ls lists files and directories
• What does ls *.txt do?
• List items based on some pattern matching
• Using pattern matching, we can search for and
delete the shortest and longest possible match of a
substring from either front or back.

19
• String removal from front
• ${str#substr}
• Removes shortest match of $substr from
front of $str.
• ${str##substr}
• Removes longest match of $substr from front
of $str.

20
• Example
• dept=“Computer Science”
• echo ${dept#o*e} # shortest!
Computer Science Output
?

• echo ${dept##o*e} ## longest!

Computer Science Output
?

21
• String removal from back
• ${str%substr}
• Removes shortest match of $substr from the
back of $str.
• ${str%%substr}
• Removes longest match of $substr from back
of $str.

22
• Example
• dept=“Computer Science”
• echo ${dept%e*e} # shortest!
Computer Science Output
?

• echo ${dept%%e*e} ## longest!

Computer Science Output
?

23
 b) Using external commands
i. tr command
• Converts (translates) from upper to lowercase or visa versa
• Example
Must have space here!
str=“Computer Science”
lower_str=$(echo "$str" | tr ‘[:upper:]’ ‘[:lower:]’)
echo “Initial String : $str"
echo “In lowercase : $lower_str"
• Note: swap upper and lower to convert to uppercase

24
ii. eval command
• Has many uses
• One of them is to convert case of a string
• Usually used together with tr command
• Example
conv_comand='tr "[:upper:]“ "[:lower:]"'
eval "lower_str=\$(echo \ "$str\" | $conv_comand)"
echo “Initial String: $str"
echo “in lowercase: $lower_str"

25
Another use of eval
• To execute dynamic or complex commands
• These are usually stored in strings or variables.
• Example
comand="ls -l"
eval "$comand"
• 1st line: stores command ls –l in variable comand
• 2nd line: evaluates value of variable comand
• ls –l is a command, it will be executed!
• will display directory contents in long format
26
iii. sed command
a) Most commonly used for text replacement tasks
Replacing a specific character with another character
sed locates specified pattern in a file and acts accordingly
• Example:
str="Hello World"
str2=$(echo "$str" | sed -e 's/H/h/g' -e 's/W/w/g')
echo “Original string: $str”
echo “String after replacing some characters: $str2”
27
For replacing a substring in with another one
if [ "$#" -ne 3 ]; then
echo "Usage: $0 <str> <old-str> <new_str>
exit 1
fi
str="$1“; old_str="$2“; new_str="$3"
final_str=$(echo "$str" | sed "s/$old_str/$new_str/g"
echo “Original_string: $str"
echo “Substring to replace: $old_str"
echo “Substring to use: $new_str"
echo “Final string: $final_str"

28
• Command sed can be applied to files
• Example
• sed 's/Section/Chapter/' output.txt
• Will replace the word “Section” with the
word “Chapter” in file output.txt

29
b) Can be used to remove first character in a string
• Example:
echo “Programming" | sed 's/^.//
• ^. : from beginning of string
• take 1 character # ^.. : for first 2 chars
• //: remove it

30
c) Can be used to remove last character
• Differs slightly from previous example
• Just replace ^. with .$

• Example:
echo “Programming" | sed 's/.$//'

31
• Note: command sed has many other uses
• E.g.
• Deleting text from a file
• Assuming file tempo.txt has these contents:
Hello
People
Thanks

• sed ‘1d’ tempo.txt ➔ will lead to

People
- ‘1d’ is for deleting 1st line Thanks

32
iv. rev command
• Used to reverse a string
• Examples:
str="Hello, World!"
reversed_str=$(echo "$str" | rev)
echo "Original String is: $str"
echo “Reversed string: $reversed_str"

33
v. cut command
• Used to extract a substring from a string
• A fundamental text manipulation operation
• Syntax: cut option … [file] …

• Notes:
• Various options available (use man command)
• If file is not specified, cut uses std input
34
• Many possible options. Egs:
• -d: uses some delimiter
• -f: uses fields in specified list. Uses a delimeter
• Tab: default delimeter
• Examples:
• cut -d " " -f 1 meter.txt
• Will extract 1st field, using space as delimeter
• cat meter.txt | head -n 5 | cut -d ' ' -f 1 > first5.txt

35
• Another Example:
• echo “Today is $(date)" | cut -c 9-28
• Dissecting the above:
• output from echo is piped to cut command.
• cut :
• -c: extract using character position
• 9-28: means from 9th to 28th position

36
vi. grep command
• An acronym for General Regular Expression Parser
• Searches files for a specific pattern of text
• And send matching lines to the terminal
• Syntax
grep [options] '[search_pattern]' [filename]
• If no filename, it uses stdin

37
• Many possible options - Check man grep
• Some options:
• -c: lists number of matching lines for each give file
• -i: Ignore case
• -l: List names of files with matching lines
• -r: for recursive search
• -w: search whole words only
• -v: Invert matching pattern - selects non-matching
lines

38
• Examples:
• grep “local” in.txt # searches “local” in in.txt
• grep ‘local’ *.sh # searches “local” in all .sh files
• grep ‘local’ * # searches “local” in all files
• grep ”local” f1.sh f2.sh f4.sh
• # can take multiple files as arguments
• cat out.txt | grep ‘201900001‘
• # can work with piping
39
• What is the output?
a) grep –c CSI223 out1.txt out2.txt

b) grep –c –v CSI223 out1.txt out2.txt

40
REGEX
• Acronym for REGular Expression
• Regular Expression (RE)
• A sequence of characters used in pattern
matching
• Allows us to perform sophisticated matching
• In RE, certain characters have special
interpretation
• See next slide
41
Character Description
^ Anchor to beginning of line of text (1st char)
$ Anchor to end of line of text (last char)
[A-Z] For matching any character in the given range
“.” For matching any character

• Examples
• grep ^t in.txt # for all lines starting with ‘t’
• grep g$ in.txt # for all lines ending with ‘g’

42
• RE Options
• ?: Match optional but may be matched once
• *: Must be matched zero or more times
• +: Must be matched one or more times
• {n} : Must be matched n times
• {n,}: Must be matched n or more times
• {n,m}: Must be matched between and m times

43
• Examples
• Search for lines that end with letter “e” in out.txt
grep e$ out.txt
• Search for three-letter words that start with “Th”
in out.txt
grep Th. [[:space:]] out.txt
• Search for lowercase words that are exactly 10
chars long in out.txt
grep -E [a-z]\{10\} out.txt
44
vii. printf command • Supported escape sequence:
• Print output just like echo • \” :double quote
• BUT • \\ :Backslash char

• Formats output better • \a :Alert

• \b :Backspace char
• Example: • \c :Suppress output
• printf “Hello \n • \n : newline char
World” • \r : Carriage return
• \t : Tab char

45
printf conversion specifier
• Constitutes’%’ followed by a conversion character.
• Examples of conversion specifiers:
• D: Display a decimal number
• C: Display a character
• S: Display a string
• %: Display the % char
• Examples:
• printf “%s\n” Hello # Output: Hello
• printf “%s\t %d” ”They are ” 3 # They are 3
46
 Array of strings
• Array declaration example:
names=(“Tom" “Bonolo" “Jade")

• Looping through array elements

for i in ${!names[@]}; do … done

• Appending a new element to an existing array

names+=(“Moses")

47
• Removing an element from an array
• Syntax: unset names[index]
• Example:
echo “Array after adding Moses:”
echo ${names[@]}
unset names[2] # removes 3rd element!
echo –n “Array after removing the element:”
echo ${names[@]}
48
• Sorting an array
sorted=($(for i in "${names[@]}"; do
echo $i; done | sort))
echo “Array contents after sorting:”
echo ${sorted[@]}

49
Splitting a string into words using
arrays
• We can split a long string into substrings
• How to?
• Declare an empty array
• Using a loop
• extract each word
• append it to the array

50
• Example
str=”Hello Welcome Here”
arr=() # creating an empty array
for i in $str; do # iterate through words in str
arr+=($i) ; # append a word to arr
done
for j in ${arr[@]}; do # display contents of arr
echo $j;
done

51
• Next & last Bash scripting lesson:

a) Using awk command

b) File I/O

52