MODULE 1

Browsing directories with the Linux terminal

The Linux terminal

You can interact with the Linux shell by entering commands into the Linux terminal, which is also
commonly referred to as the command line or the command prompt.

In this case, the present working directory is /home/project, as indicated by the blue text. The dollar
sign $ following it is called the command prompt.

Recall that a terminal window is a simple user interface that allows you to run any commands that
you would like, simply by typing the command on your keyboard and hitting Enter. Many commands
will respond by returning some sort of output, which by default appears as text in your terminal
window.

Let's see how this works using a couple of the most common Linux commands,
the pwd and ls commands:
Here we've entered the pwd command, which prints the path name for our present working
directory on the next line. You can see that the command prints what we expected, which is the path
to the present working directory, /home/project. Notice also that the command prompt shows up
again on the following line, awaiting your next command.

Great! Now, how do you see what's inside your present working directory?

You can use the ls command to list the contents of the directory you are currently working in. At the
moment, the /home/project directory is brand new and doesn't contain anything yet, so entering
the ls command will return nothing:

Being a conservative program, ls won't bother with printing a blank line to express that there is
nothing to list.

Let's see if we can find a directory that already contains content. You can list the contents of any
directory with the ls command by specifying the directory name you'd like to explore.

For example, ls /home lists the contents of the /home directory:

You can see that the directory /home contains two objects, namely project and theia.

Tip: Think of a directory as a folder that contains files and subdirectories. In this
case, project and theia are subdirectories of /home. Subdirectories can contain additional files and
subdirectories. You'll learn more about exploring subdirectories in later labs.

Notice the naming convention for a directory's path: /home/project indicates that
the project directory is a subdirectory of /home. The path for the theia subdirectory would similarly
be /home/theia.

Like a tree, your Linux file system has a root directory (/, called "slash") from which your entire Linux
file system branches out.

One important subdirectory of your root directory is home. You can see this for yourself by
entering ls / to list the contents of /:
Summary
 You can interact with the Linux shell by entering commands into the Linux terminal

 The pwd command prints the path name to the present working directory

 The ls command lists the contents of a directory

Linux Terminal Tips - Tab completion, command history

Tab Completion
Many shells support a feature called tab completion.

Tab completion allows you to autocomplete a command you're typing on the command line.

Suppose you're in your home directory ~, which contains the directories:

 Pictures

 Videos

 Documents

 Downloads

And suppose your Documents folder only contains the folder:

 python-examples

Tab completion:

In this exercise, you will use tab completion to quickly enter a command to navigate to
the Pictures directory.

If you type:

1. ~ $ cd P

and press Tab, the shell will autocomplete the command to:

1. ~ $ cd Pictures/

because the Pictures directory is the only directory within your current folder that starts with a "P".

Tab completion for long paths:

You can also use tab completion to autocomplete longer paths.

If you type:

1. ~ $ cd Do
and press Tab, nothing will happen because your current directory contains more than one directory
that starts with "Do". The shell won't know whether to autocomplete
with Documents or Downloads.

On the other hand, if you type:

1. ~ $ cd Doc

and press Tab, the shell will autocomplete to:

1. ~ $ cd Documents/

because in this case, there's only one directory that starts with "Doc," Documents.

If you press Tab again, the shell will further autocomplete to:

1. ~ $ cd Documents/python-examples/

because the folder python-examples is the only existing file within the ~/Documents directory. With
just a few clicks, you can quickly autocomplete longer paths in your command line.

Command History
Command history allows you to navigate previous commands you've entered using the Up
Arrow and Down Arrow keys.

Let's say you've created a Python script called hello_world.py in your python-examples directory that
simply prints Hello, World! when you run it.
Consider you have just entered the following sequence of commands, and the shell is awaiting your
next input:

1. ~ $ cd ~/Documents/python-examples

2. ~/Documents/python-examples $ python3 myprogram.py

3. Hello, World!

4. ~/Documents/python-examples $ cd /

5. / $

In the following exercises, you will learn how to rerun a previous command withut having to retype it
by pressing the Up Arrow key.

Running the last command:

If you press the Up Arrow key once, the shell will automatically insert the last command you entered:

1. ~ $ cd ~/Documents/python-examples

1. ~/Documents/python-examples $ python3 myprogram.py

2. Hello, World!

3. ~/Documents/python-examples $ cd /

4. / $ cd /
Notice that the last command you entered was cd /, which has been automatically inserted onto the
command line for you.

Running previous command from session:

If you press the Up Arrow key two additional times (so, three times total), the shell will automatically
insert the command you ran three commands ago:

1. ~ $ cd ~/Documents/python-examples

2. ~/Documents/python-examples $ python3 myprogram.py

3. Hello, World!

4. ~/Documents/python-examples $ cd /

5. / $ cd ~/Documents/python-examples

In this case, the command you ran three commands ago was cd ~/Documents/python-examples.
Note that the line of printed output, Hello, World!, does not count as a command. Command history
will only take you through commands you entered, not every line that is visible in the terminal.

Tip: If you click the Up Arrow key too many times, you can use the Down Arrow key to cycle through
your command history in the opposite direction.

Press Enter to return to your ~/Documents/python-examples directory. You should see something
like the following:

1. ~ $ cd ~/Documents/python-examples

2. ~/Documents/python-examples $ python3 myprogram.py

3. Hello, World!

4. ~/Documents/python-examples $ cd /

5. / $ cd ~/Documents/python-examples

6. ~/Documents/python-examples $

Summary
In this reading, you learned how to:

 Use tab completion to autocomplete commands

 Use the command history to quickly navigate to previous commands

Opening a terminal
 When you first connect to the lab environment, there may already be an open terminal
for you to use. If not, open a new terminal by clicking the menu bar and
selecting Terminal > New Terminal.
 
 This will open a new terminal at the bottom of the screen.


 In your newly opened terminal, you will run the commands provided in the following
exercises. You can copy the code to your clipboard by clicking the copy button on the
bottom right of each codeblock, then pasting it to the command line. However, it will
be more beneficial to type the code yourself, so you can start building some
"command muscle memory".

Exercise 1 - Browsing Directories

ls

In this exercise, you will learn how to browse the content of directories using the ls command.

ls is a special command that the shell interprets and runs as a program.

1.1. Viewing files in the current working directory

Typing and entering ls will display all files and directories contained within your current working
directory:

1. ls

Your default current working directory in this lab environment, the /home/project directory, is
empty. Thus entering ls into the command line will not return anything yet.

1.2. Viewing files and directories within any directory

If you know the path to a directory, you can view its contents by passing the path name as
a command line argument to the ls command as follows:

ls [PATH TO DIRECTORY]

For example

1. ls /

will show the contents of 'slash', your Linux system's root directory.

Recall some of the standard subdirectories of 'slash' that you've learned about previously:

Directory Contains

System commands,
/bin
also called binaries

System administration
/sbin
binaries

User programs and

/usr
data

/home Home directory

Removable media
/media
device directories

For example, entering the following command:

1. ls /bin

will display the contents of the /bin directory.

Notice that one of the files in the /bin directory is called "ls". Indeed, when you enter
the ls command, Linux searches for and runs the ls command by executing the binary file /bin/ls.

Exercise 2 - Navigating Directories

cd

In this exercise, you will explore directories using the cd command.

Recall the symbols used to navigate to special paths:

Symbol Stands for path to

~ Home directory

/ Root directory
 Symbol Stands for path to

. Current directory

.. Parent directory

2.1. Changing your present working directory to your home directory

To change your current working directory to your default home directory, use the ~ symbol as
follows:

1. cd ~

Note that your lab environment's default home directory, ~, is /home/theia. After changing
directories to ~, you can check this yourself by entering the pwd command, which displays the path
to your current working directory.

2.2. Changing your present working directory to its parent directory

To change your current working directory to the parent directory of your present working directory,
enter the following command using ..:

1. cd ..

For example, if your working directory was /home/theia, then the new working directory will
be /home.

2.3. Changing working directory to root directory

To change your current working directory to the root directory, use the following command with
the / key:

1. cd /

2.4. Changing your present working directory to a child directory

Assuming you are still in your root directory, use the following command to change your present
working directory to the /bin directory:

1. cd bin

The /bin directory is called a child of the root directory, /, because it's a subdirectory within it.

The cd command understands implicitly that bin is the name of a subdirectory of your current
working directory. As a result, it deduces that its path is a combination of the current working
directory path (/) and the subdirectory name (bin).

Another method to change directories is to enter the command cd ./bin. In this case, . represents the
path to your current working directory.

Give it a try by entering cd ./bin in your command line! What directory did you end up in?

Tip: If cd ./bin doesn't take you to the /bin directory, double-check what directory you are currently
in. Remember, . represents the directory you are in. If you're already in /bin, the terminal will
attempt to direct you to /bin/bin.
2.5. Changing from your working directory back to your home directory
There are multiple ways to change your current working directory back to your default home
directory. One way is to return to your parent directory, .., and type the address of your home
directory /home/theia:

1. cd ../home/theia

A simpler way to do this would be to use the ~ symbol to quickly and directly navigate to your home
directory:

1. cd ~

Both methods will change your working directory back to your home directory.

2.6. Changing from your working directory to your project directory

Directories that are contained within the same parent directory are called siblings.

In this environment, we have provided a special empty project directory for your work.
This project directory is located at /home/project and is a sibling directory to /home/theia.

Change your current working directory to its sibling directory, your project directory, by entering the
following command:

1. cd ../project

Exercise 3 - Using tab completion and the command history

In this exercise, you will gain familiarity with tab completion and the command history.

3.1. Scrolling through your command history

At the command prompt, simply hit the Up Arrow and Down Arrow keys, ↑ and ↓, to scroll through
your command history.

Assuming you are still in the same session as you were when you completed the previous exercises in
this lab, you should see exactly what commands you have entered so far.

For example, suppose you want to use the cd command to navigate to /bin again. Rather than
retyping the command, simply press the Up Arrow key until you see the cd bin displayed at the
command prompt and press Enter.

Of course, this command will only work as expected if your current working directory is the correct
one. If the command fails, you can keep scrolling through your history until you find the cd command
to return to your root directory, (cd /). Enter that command first, then use the Up Arrow key to find
and enter the cd bin command from your history.

Your command history is very useful if you are doing repetitive tasks and want to avoid retyping
things repeatedly.

3.2. Using tab completion

Tab completion can save you a lot of typing, thinking, and time. Let's take a look at some examples.

Type cd /bi and press the Tab key. As you might expect, the command autocompletes to cd /bin. But
what happens if you just type cd /b and press Tab?
In this case, nothing happens. But if you hit Tab a second time, you'll see a list of all the valid
candidates for autocompletion, namely bin/ and boot/.

Notice that cd /b reappears in the command prompt. You cannot use tab completion while there are
multiple valid candidates for autocompletion. To narrow down your choices, add an i at the end of
your line, then press Tab again. This time the prompt will fill with cd /bin, the only valid option
remaining.

Tip: Tab completion only works when there is one valid candidate for autocompletion. When there
are multiple possible options, the command prompt will prompt you to enter more letters until there
is only one possible option.

You can even use tab completion repeatedly to "dig into" a directory tree.

Let's say you have used tab completion to get to the point where ls /home is displayed at the
command prompt. You can press Tab twice to get a listing of all the directories under /home. As you
did with cd /bin, use tab completion to complete the address of a subdirectory such as ls
/home/theia. You can repeat this process as many times as you like until you reach the end of a
chosen path.

As an exercise, practice using tab completion repeatedly to build up the command ls

/home/theia/dsdriver/bin.

Future labs will cover command options more in-depth. In this lab, you became familiar with the
terminal environment and some basic commands.

Practice Exercises

1. List the contents of the root directory.

Solution: ls /

2. Change directories to your default home directory.

Solution: cd ~

Note: Your home directory in this environment is /home/theia. The default home directory in Linux
is /home/{username}, where username is the name you would have been assigned had you created
a personal Linux user account.

3. Verify your current working directory is /home/theia.

Solution: pwd

4. Use tab completion to change directories to /bin.

Solution:

Because there are two directories in / that have names starting with a b, you need to
press Tab twice. Then the terminal displays the two available options: bin/ and boot/. To
disambiguate, enter an i and press the Tab key.

5. Use your terminal's command history to change directories back to your home directory.

Solution:

Use the Up Arrow key until you find

 1. cd ~

and press the Enter key.

Exercise 1 - Upgrading and installing packages

In your lab environment, we provide you access to a system admistration utility called "super-user
do", or sudo.

Although this sandbox environment will not require a password to run the sudo command, typical
Linux production-oriented environments will require a password to run sudo.

You will need to use the sudo command to activate the powerful apt command. You'll use apt to
upgrade nano to its latest version. You will also use apt to install Vim.

apt (Advanced Packaging Tool) is a powerful command line tool. You use it to perform system
administration operations such as installing software packages, upgrading existing packages, and
updating your system's package list index.

1.1 Updating your Linux sytem's package list index

Before installing or upgrading any packages on your Linux system, it's best practice to
first update your package list index. Go ahead and enter this command,

1. sudo apt update

to update (re-synchronize) your package index files from their sources. This will take a bit of time to
run. While you're waiting, if you're interested, go ahead and open another terminal and view the
locations apt uses to access those sources in the file /etc/apt/sources.list.

In short, running apt with the update option ensures all of your package dependencies are up-to-
date and correctly specified prior to making any changes to your system's packages.

1.2. Upgrading nano

nano is a simple command line editor that enables you to use the terminal window as a text editor.

nano is already installed on your system. Go ahead and upgrade to the latest supported version of
nano by entering:

1. sudo apt upgrade nano

You may be prompted: Do you want to continue? [Y/n]

Type Y and press Enter to continue. Updating nano will take some time to complete.

The capital Y in Y/n (yes or no) means "yes" is the default - if you press Enter without typing
anything, the terminal assumes you are choosing "yes".

Note: If you encounter a colon (:) similar to the one highlighted in the screenshot while upgrading
Nano, please press 'q' to exit and continue with the upgrade process.
You'll get to use nano soon, but first, let's take a look at the other main use case for apt, installing
packages.

1.3. Installing Vim

Another popular text-editing program is Vim. Vim is a highly configurable text editor built for
efficiency. It takes some practice to get good at using Vim, but the time investment is very
worthwhile.

Because Vim isn't preinstalled on your Linux system, you'll need to install it yourself. If you haven't
already done so in this session, ensure you run the command sudo apt update. Then to install Vim,
enter the following command:

1. sudo apt install vim

Similar to when you updated nano, you may again be prompted: Do you want to continue? [Y/n]

Type Y and press Enter to continue. Vim will begin installing on your system.

In a few exercises, you will use Vim to edit a text file.

Exercise 2 - Creating and editing files with nano

In this exercise, you will use the command line editor nano to create and edit a file.

nano is known for being a simple and easy-to-master text editor. Vim is harder to learn, but it has
many expert-level features that nano doesn't offer.

2.1 Navigating to the project directory

We provide you with an empty project directory at /home/project. Ensure you're working in this
folder by changing directories using the command:

1. cd /home/project

Try auto-completing the path by typing cd /home/pr and pressing the Tab key.

If you enter ls here, you shouldn't see any files or subdirectories listed.

2.2 Creating and editing a text file with nano

To create a new file, enter

1. nano hello_world.txt
in the terminal. This will simultaneously create a new file called hello_world.txt and enable you to
begin editing it using the nano text editor.

Double-check that your new file was created by opening another terminal window and running
the ls command on /home/project. You should see hello_world.txt listed.

In your nano terminal, whatever you type will be added to your text buffer, where text is stored until
you save it. Type the following text in your nano terminal:

1. Hello world!

This will create the text Hello world! in your text buffer.

To create another line of text, press Enter. In your new line, type

1. This is the second line of my first-ever text file created with nano.

to create a second line of text in your text buffer.

Now:

1. Press CTRL+X to exit nano.

2. You will be prompted as follows:

Save modified buffer (ANSWERING "No" WILL DESTROY CHANGES) ?

Y Yes

N No ^C Cancel

Press Y to save your new lines of text to your file.

3. Press Enter to confirm the file name.

At this point, nano should have exited and returned you to the command prompt.

2.3 Verifying your new text file

By entering a familiar command, such as

1. cat hello_world.txt

you should be able to inspect and verify that your new file contains the two lines you wrote to it with
nano. Cool!

Exercise 3 - Creating and editing files with Vim

3.1 Quick intro to Vim
Recall that Vim has two basic modes: Insert mode, where you enter text, and Command mode,
where you do everything else. You can start Vim simply by entering

1. vim

at the command prompt, which displays something like the following in your terminal window:
Notice that you can get help on Vim by entering :help, and you can quit Vim by entering :q.

Go ahead and enter :help. This brings up an informative help file you can scroll through.

When you are done reading, simply enter :q to quit Vim and return to the command prompt.

Vim is very powerful and takes some time to learn. We're just covering the very basics here and
leaving it to you to explore further. Check out the official Vim site at https://www.vim.org.

3.2 Creating and editing a text file with Vim

Begin by navigating back to your /home/project directory if you aren't already there.

Type the command,

1. vim hello_world_2.txt

to create a new file called hello_world_2.txt and edit it using Vim.

Once your Vim session has started, go ahead and press i to enter Insert mode. This is the mode
where you can enter and delete text in the text buffer.

Go ahead and type some text in the buffer, for example:

1. Hello World!
Just like in nano, press Enter to start a new line, and then type

1. This is the second line.

to create a second line of text in the buffer.

When you're done typing text in the buffer, press the Escape key, Esc, to exit the Insert mode. This
brings you to Command mode.

In Vim, it's easy to accidentally end up in a mode you didn't intend to be in. No worries - you can use
the Esc key to return to Command mode.

Now that you are back in Command mode, you can save your work by entering the command :w.
This writes the contents of the text buffer to your text file.

Finally, you can exit your Vim session by entering :q.

Practice Exercises

1. Using nano, edit your new hello_world.txt file to add a new line containing the following text:

1. This is line three of my new file.

Then, save your changes and exit nano.

Tip: Use the Up Arrow key to go through your command history until you see the right command.

Solution:

1.

2.

3.

2. Using Vim, create a file called done.txt that prints "I am done with the lab!" when you execute
the file with Bash.

In this exercise you will use Vim to create a file that contains the echo command, which you may not
have seen yet. You will also run the file using Bash, a shell scripting language that you will learn more
about later in this course. Basically, the file you create will contain a basic command that Bash can
interpret and do something with.

Solution: 1

Enter

1. vim done.txt

Solution: 2

Press i to enter Insert mode

Add the following content to the file:

1. echo "I am done with the lab!"

Then save your work and exit by:

1. Using Esc to exit out of Insert mode and return to Command mode

2. Typing :wq and presssing Enter

Solution: 3

Once you've saved and exited and are back at the command prompt, you can run the file with Bash
by entering:

1. bash done.txt

This command invokes the Bash shell to interpret the text contained in done.txt as a command. In
particular, Bash runs the echo command along with the quoted text as the input to echo,
while echo simply prints the input text to the terminal window.

When finished, you should see the following text echoed to your terminal:

1. I am done with the lab!

Summary

By now you are beginning to understand a lot more about how your Linux system functions. In this
lab, you learned how to:

 Perform some fundamental sys admin operations, such as updating your package list,
upgrading existing packages, and installing new packages

 Create and edit a few text files using some serious command-line text editors, nano and Vim

 Understand how the Bash scripting language can be used to interpret commands you include
within a text file

(Module 1 Summary and Highlights)

 In the 1980s, GNU was developed at MIT. GNU stands for “GNU’s not Unix” and was made as
a free, open source set of the existing Unix system tools. And in 1991, Linus Torvalds
developed a free, open source version of the Unix kernel called Linux.

 Linux is widely used today in mobile devices, desktops, supercomputers, data centers, and
cloud servers.

 Linux distributions (also known as distros) differ by their UIs, shell applications, and how the
OS is supported and built.

 The design of a distro is catered toward its specific audience and/or use case. Popular Linux
distributions include Red Hat Enterprise Linux (RHEL), Debian, Ubuntu, Suse (SLES, SLED,
OpenSuse), Fedora, Mint, and Arch.

 The Linux system consists of five key layers: the UI, application, OS, kernel, and hardware.
The user interface enables users to interact with applications. Applications enable users to
perform tasks within the system. The operating system runs on top of the kernel and is vital
for system health and stability, and the kernel is the lowest-level software that enables
applications to interact with hardware. Hardware includes all the physical or electronic
components of your PC.
  The Linux filesystem is a tree-like structure consisting of all directories and files on the
system.

 A Linux shell is an OS-level application that you can use to enter commands. You use a
terminal to send commands to the shell, and you can use the command to navigate around
your Linux filesystem.

 You can use a variety of command-line or GUI-based text editors such as GNU nano, vim, vi,
and gedit.

 .deb and .rpm are distinct file types used by package mangers in Linux operating systems.

 You can use GUI-based and command-line package managers to update and install software
on Linux systems.

Module 1 Cheat Sheet - Introduction to Linux

Linux terminal tips

Use tab completion to autocomplete pathnames and command names.

Scroll through your command history with the Up Arrow and Down Arrow keys to find and re-run a
command you already used.

Getting information

Display the reference manual for the ls command:

1. man ls

Browsing and navigating directories

Special paths

Symbol Represents path to

~ home directory

/ root directory

. present working directory

.. parent of present working directory

List files and directories in the current directory:

1. ls

List files and directories in a directory:

 1. ls path_to_directory

Return path to present working directory:

1. pwd

Change the current directory to a subdirectory:

1. cd child_directory_name

Tip: Because cd looks in the current directory for child_directory_name, you don’t need to type the
entire path.

Change the current directory:

Up one level: cd ../

To home: cd ~ or cd

To some other directory: cd path_to_directory

Change the current directory to another one at the same level:

Suppose you have two sibling directories within the same directory, dir_1 and dir_2, and your
present working directory is dir_1. To switch to dir_2, enter:

cd ../dir_2

Tip: Using .., you don't need to know the path to the parent directory to switch to a sibling.

Change the current directory back to the directory you were in previously:

cd -

Upgrading and installing packages

Fetch and display up-to-date information about all upgradable packages:

1. sudo apt update

Upgrade to the latest supported version of nano:

1. sudo apt upgrade nano

Install Vim:

1. sudo apt install vim

Creating and editing files

Create a new text file and open it with nano:

1. nano file_name.txt

Tip: If the file already exists, nano simply opens it for editing.
MODULE 2 – INTRODUCTION TO LINUX COMMANDS
Getting Help for Linux Commands

There is huge value in spending time exploring and experimenting with commands, but there are
many things you can't figure out just by experimenting. You need to see what's out there, see what's
possible, and learn where to look to get answers. Let's take a look at some great ways to get the
information you need to help you make progress.

In this reading, you may see links to external sources. You can open them by right-clicking and
pressing "Open in new tab."

1. Use the built-in man command

The man command, which stands for "manual", provides the standard way to access help for Unix-
like commands from the command prompt. It has been in development since 1971.

You can get a listing of all the commands on your system that have a manual page by entering:

1. man -k .

The resulting list includes a brief description of what each command does.

To see the man page for a command, simply enter:

1. man command_name

All man pages are divided into several sections, including:

NAME

The name of the command or feature and a brief description of what it does.

SYNOPSIS
A summary of the command syntax, including any options and arguments that can be used.

DESCRIPTION

A more detailed description of the command, including its function and behavior.

OPTIONS

All the available options and arguments that can be used with the command.

EXAMPLES

Some examples of how to use the command.

SEE ALSO

Related commands and documentation that may be helpful.

You may also see other sections, including: EXIT STATUS, RETURN VALUE, ENVIRONMENT, BUGS,
FILES, AUTHOR, REPORTING BUGS, HISTORY, and COPYRIGHT.

2. Install and use the tldr command

Similar to man pages, TLDR Pages is a free and open-source collaborative documentation effort. The
goal is to create documentation that is more accessible than the traditional man pages, which tend
to be quite verbose.

TLDR Pages, short for "Too Long; Didn't Read" and also known simply as tldr, provide examples for
common use cases of various commands. The format of TLDR pages is similar to that of a cheatsheet.

You can install a command-line tool to access TLDR Pages from your terminal. Install it using the
following command:

1. npm install -g tldr

Once you've installed the tool, you can use the tldr command to easily access the TLDR page of a
command.

1. tldr command_name

The tool will display a short, easy-to-understand summary of the command along with some
examples of how to use it.

3. Search Stack Overflow

Stack Overflow is a popular community-driven question and answer platform for programmers,
developers, and system administrators. It has a vast repository of questions and answers related to
various programming languages, tools, and operating systems, including Linux.

To search for information about commands on Stack Overflow, you can use the search bar on the
homepage and enter the name of the command you're looking for, along with any specific keywords
or parameters. You can also refine your search by adding relevant tags, such as "linux" or "command-
line".

Once you've entered your search query, Stack Overflow will display a list of relevant questions and
answers that match your query. You can browse through the results to find the information you
need, and even post your own question if you can't find an answer to your specific query.When
searching for information about commands on Stack Overflow, it's important to check the date of the
answers to ensure that the information is still current and relevant. You should also read through the
comments and discussion threads to get a better understanding of the context and any potential
issues or limitations related to the command you're researching.

Newest questions on Stack Overflow tagged

"Linux": https://stackoverflow.com/questions/tagged/linux

4. Search Stack Exchange

Stack Exchange is a network of question and answer communities, similar to Stack Overflow, but
covering a broader range of topics beyond just programming. There are several Stack Exchange
communities that specialize in topics related to Linux and open source software, such as Unix &
Linux, Ask Ubuntu, and Server Fault.

Visit the relevant community to search for information on Stack Exchange. Like Stack Overflow, you
can use the search bar to enter the name of the command you're looking for, along with any
keywords or parameters.

Unix and Linux community on Stack Exchange: https://unix.stackexchange.com/

5. Just google it!

Google is a powerful tool that can provide you the answer to almost any question. Learn how to
enter the right queries and filter your results, such as by including "Wikipedia", "Stack Overflow", or
"Linux" as part of your search. However, use at your own risk. Never blindly trust what you find on
the web - there's a lot of noise out there!

6. Use the cheat sheets from this course

Throughout this course, you will encounter "cheat sheets" that condense the information you've
learned into easy-to-reference guides. They are great for reviewing the material you've learned and
can also help you out with your graded assignments.

7. Refer to Wikipedia's list of Unix commands:

Finally, Wikipedia maintains a list of commands that can be found on Unix operating systems, along
with a short description. You can check the page to quickly reference a Unix
command: https://en.wikipedia.org/wiki/List_of_Unix_commands

Summary

This reading has provided an overview of how to find more information about Linux commands. As
you continue to use commands in this course and beyond, you'll likely find yourself using the same
command patterns over and over. Each time you use a command, you'll get a little more comfortable
and familiar with it. Your fingers will develop muscle memory until using commands becomes second
nature!

Exercise 1 - Informational Commands

In this exercise, you will familiarize yourself with useful commands for providing system and user
information.

1.1. Display the name of the current user

whoami

Enter the whoami command to return your current username.

1. whoami

It will display the user name as theia because you are logged into this lab as theia.

You can get a list of currently logged in users using the command who, but this command doesn't
work in the Theia environment yet.

1.2. Get basic information about the operating system

uname

By default the command prints the kernel name. The u in uname refers to "unix-like 0S".

1. uname

If you enter the command uname, you will see Linux printed in the output.

Using the -a option prints all the system information.

1. uname -a

You will see system information listed in the following order:

 Kernel name

 Network node hostname

 Kernel release date

 Kernel version

 Machine hardware name

 Hardware platform

 Operating system

1.3. Obtain the user and group identity information

id

This command displays the user id and group id information of the current user.

1. id

It will display the uid (user id) and gid (group id) for the user theia.

1.4 Get available disk space

df

The df command is used to display available disk space.

1. df

This command will display available disk space in 512-byte blocks. To get available disk space in a
"human-readable" format, enter:
 1. df -h

This will return the available disk space in units like gigabytes and terabytes.

1.5. View currently running processes

ps

The ps command lists each process that is currently running and its PID (process id).

1. ps

However, the output only contains the processes that are owned by you.

By using the -e option, you can display all of the processes running on the system. The includes
processes owned by other users.

1. ps -e

1.6. Get information on the running processes and system resources

top

The top or "table of processes" command provides a dynamic, real-time view of your system.

1. top

The top command displays a table of detailed information related to the processes or threads which
are currently running and managed by the kernel. Additionally, it also provides information related to
CPU and memory usage per process.

When you start top, you'll be presented with the following elements on the main top screen:

 Summary area - shows information like system uptime, number of users, load average, and
overall memory usage

 Column header - attribute names

 Task area - displays the data for each process, or PID

The output keeps refreshing until you press q or Ctrl + c.

If you want to exit automatically after a specified number of repetitions, use the -n option as follows:

1. top -n 10

You can press the following keys with Shift while top is running to sort the table:

Key Sorts by

M Memory Usage

P CPU Usage

N Process ID (PID)
 Key Sorts by

T Running Time

For example, you can find out which process is consuming the most memory by entering Shift + m.

1.7. Display Messages

echo

The echo command displays the given text on the screen. For example, entering:

1. echo "Welcome to the linux lab"

prints:

1. Welcome to the linux lab.

These special characters help you better format your output:

Special Character Effect

\n Start a new line

\t Insert a tab

Use the -e option of the echo command when working with special characters. For example:

1. echo -e "This will be printed \nin two lines"

will print:

1. This will be printed

2. in two lines

1.8. Display date and time

date

The date command displays the current date and time.

1. date

It has several options which allow you to display the current date and time in different formats.

For example, the following command displays the current date in mm/dd/yy format:

1. date "+%D"

Here are some popular format specifiers that you can try out:

Specifier Explanation

%d Displays the day of the month (01 to 31)

 Specifier Explanation

%h Displays the abbreviated month name (Jan to Dec)

%m Displays the month of year (01 to 12)

%Y Displays the four-digit year

%T Displays the time in 24 hour format as HH:MM:SS

%H Displays the hour

1.9. View the Reference Manual For a Command

man

The man command displays the user manual for any command that you provide as its argument.

For example, to see the manual page for the ls command, enter:

1. man ls

Scroll through the command's manual to find any info you may need. When you're done, press q to
quit.

You will sometimes encounter a command that does not have a man page avaible on your system. To
see all available man pages with a brief description of each command, enter:

1. man -k .

Practice exercises

1. Get basic information about the operating system.

Solution: uname

2. View all running processes on the system.

Solution: ps -e

3. Get the table of processes and sort by memory usage.

Solution: top

Then press Shift+m.

4. Display the current time.

Solution: date "+%T"

5. Using one command, display the messages "Hello!" and "Goodbye!" separated by a new line.

Solution: echo -e "Hello! \nGoodbye!"

Summary

In this lab, you learned that you can use the commands:

 whoami to return your username

 uname to print the kernel name

 id to display the user and group id

 df to print available disk space

 ps to list running processes and their process id

 top to view a real-time table of processes

 echo to print given text

 date to display the current time and date

 man to get the user manual for a command

Exercise 1 - Navigating Files and Directories

In these exercises, you will practice using commands for navigating and managing files and
directories.

1.1. Get the location of the present working directory

pwd

When working in a Linux terminal, you will always be working from a directory. By default, you will
start in your home directory. To get the absolute path of your present working directory, enter the
following:

1. pwd

This will print the name of the directory you are currently working in.

1.2. List the files and directories in a directory

ls

To list the files and directories in the current directory, enter the following:

1. ls

If your directory happens to be empty, ls will not return anything.

The following command will list the many binary and executable files which are present in
your /bin (binaries) directory.

1. ls /bin

The /bin directory happens to be where Linux commmands such as ls and pwd are stored. For
example, you can see that ls is present by entering the following:

1. ls /bin/ls

To list all files starting with b in the /bin directory, try entering the following:
 1. ls /bin/b*

Tip: The asterisk * is a special character called a wildcard. It is used to represent any string of
characters.

To list all files ending in r in the /bin directory, enter the following:

1. ls /bin/*r

To print a longer list of files with additional information, such as the last-modified date, enter the
following:

1. ls -l

Here are some common options that you can try with the ls command:

Option Description

-a list all files, including hidden files

-d list directories only, do not include files

-h with -l and -s, print sizes like 1K, 234M, 2G

-l include attributes like permissions, owner, size, and last-modified date

-S sort by file size, largest first

-t sort by last-modified date, newest first

-r reverse the sort order

To get a long list of all files in /etc, including any hidden files, enter the following:

1. ls -la /etc

Here we combined the options -l and -a by using the shorter notation, -la.

Exercise 2 - Creating Files and Directories

2.1. Create a directory

mkdir

The mkdir command is used to create a new directory.

To create a directory named scripts in your current directory, run the following command:

1. mkdir scripts

Use the ls command to verify whether the scripts directory was created:

1. ls
You should see a directory named scripts listed.

2.2. Change your current working directory

cd

To change your present working directory to the scripts directory, run the following command:

1. cd scripts

Now use the pwd command to verify whether your current working directory has changed as
expected:

1. pwd

You can enter cd without any directory name to move back to your home directory:

1. cd

Then, enter the pwd command to verify whether your current working directory has changed:

1. pwd

The syntax .. is a shortcut that refers to the parent directory of your current directory. Run the
following command to change dictories up one level:

1. cd ..

2.3. Create an empty file

touch

First, return to your home directory by entering:

1. cd

Next, use the touch command to create an empty file named myfile.txt:

1. touch myfile.txt

Now use the ls command to verify the creation of myfile.txt:

1. ls

If the file already exists, the touch command updates the access timestamp, or last-modified date of
the file. To see this, enter:

1. touch myfile.txt

And use use the date command to verify the date change:

1. date -r myfile.txt

Exercise 3 - Managing Files and Directories

3.1. Search for and locate files

find
The find command is used to search for files in a directory. You can search for files based on different
attributes, such as the file's name, type, owner, size, or timestamp.

The find command conducts a search of the entire directory tree starting from the given directory
name.

For example, the following command finds all .txt files in the /etc directory and all of its
subdirectories:

1. find /etc -name \'*.txt\'

Note: Along with listing all the .txt files, the terminal may return "Permission denied" errors.
These errors are normal, as you have limited access permissions on the lab machine.

3.2. Remove files

rm

The rm command is used to delete files, ideally with the -i option, which creates a prompt to ask for
confirmation before every deletion.

To remove the file myfile.txt, enter the following command and press y to confirm deletion, or n to
deny deletion:

1. rm -i myfile.txt

Use the ls command to verify removal:

1. ls

Tip: When you are only removing one file with the rm command, the -i option is redundant. But if
you want to remove multiple files, for example by using a wildcard to find all filenames matching a
pattern, it's best practice to confirm or deny each deletion by including the -i option.

Be careful when deleting files or directories! There is normally no way to restore a deleted file once it
is deleted, as there is no trash folder. This is why you should always back up, or archive, your
important files. You will learn more about archiving files soon.

3.3. Move and rename a file

mv

You can use the mv command to move files from one directory to another and/or rename them.

Before doing so, let's first create a new file called users.txt:

1. touch users.txt

You should always use caution when moving a file. If the target file already exists, it will be
overwritten, or replaced, by the source file.

Conveniently, however, when the source and target directories are the same, you can use mv to
rename a file.

To illustrate this, use mv to rename users.txt to user-info.txt by entering the following command:

1. mv users.txt user-info.txt
Because the source and target directories are the same (your present working directory),
the mv command will rename the file.

Now use the ls command to verify the name change:

1. ls

Now, you can move user-info.txt to the /tmp directory as follows:

1. mv user-info.txt /tmp

Use the ls command twice to verify the move:

1. ls

1. ls -l /tmp

3.4. Copy files

cp

You can use the cp command to copy user-info.txt, which is now in your /tmp directory, to your
current working directory:

1. cp /tmp/user-info.txt user-info.txt

Use the ls command to verify that the copy was successful:

1. ls

At times, you may want to copy the contents of an existing file into a new one.

The following command copies the content of /etc/passwd to a file named users.txt within the
current directory:

1. cp /etc/passwd users.txt

Again, use the ls command to verify if the copy was successful:

1. ls

Practice exercises

1. Display the contents of the /home directory.

Solution: ls /home

2. Ensure that you are in your home directory.

Solution: cd

pwd

3. Create a new directory called tmp and verify its creation.

Solution: mkdir tmp

ls

4. Create a new, empty file named display.sh in the tmp directory, and verify its creation.
Solution: cd tmp

touch display.sh

ls -l

5. Create a copy of display.sh, called report.sh, within the same directory.

Solution: cp display.sh report.sh

6. Move your copied file, report.sh, up one level in the directory tree to the parent directory. Verify
your changes.

Solution: mv report.sh ../

ls

ls ../

7. Delete the file display.sh.

Solution: rm -i display.sh

8. List the files in /etc directory in the ascending order of their access time.

Solution: ls -ltr /etc/

9. Copy the file /var/log/bootstrap.log to your current directory.

Solution: cp /var/log/bootstrap.log

Summary

In this lab, you learned that you can use the commands:

 pwd to get the location of your present working directory

 ls to list the files and directories within a directory

 mkdir to create a new directory

 cd to change your present working directory

 touch to create a new file

 find to search for and locate files

 rm to remove a file

 mv to rename or move a file

 cp to copy a file

Security: Managing File Permisions and Ownership

Why do we need file permissions and ownership?

Linux is a multi-user operating system. This means that by default, other users will be able to view
any files you store on the system. However, you may have some files - such as your personal tax
documents or your employer's intellectual property documents - that are private or confidential.
How can you protect these sensitive documents from being viewed or modified by others?

File ownership and permissions

There are three possible levels of file ownership in Linux: user, group, and other.
Whoever creates a file, namely the user at the time of creation, becomes the owner of that file by
default. A group of users can also share ownership of a file. The other category essentially refers
anyone in the universe with access to your Linux machine - careful when assigning ownership
permission to this level!

Only an official owner of a file is allowed to change its permissions. This means that only owners can
decide who can read the file, write to it, or execute it.

Viewing file permissions

Let's say you've entered the following lines of code:

1. $ echo "Who can read this file?" > my_new_file

2. $ more my_new_file

3. Who can read this file?

4. $ ls -l my_new_file

5. -rw-r--r-- 1 theia users 25 Dec 22 17:47 x

Here we've echoed the string "Who can read this file?" into a new file called my_new_file. The next
line uses the more command to print the contents of the new file. Finally, the ls command with the -
l option displays the file's (default) permissions: rw-r--r--

The first three characters (rw-) define the user permissions, the next three (r--)
the group pemissions, and the final three (r--) the other permissions.

So you, being the user, have the permission rw-, which means you have read and write permissions
by default, but do not have execution permissions. Otherwise there would be an x in place of the
last -.

Thus by looking at the entire line, rw-r--r--, you can see that anyone can read the file, nobody can
execute it, and you are the only user that can write to it.

Note: The - at the very beginning of the line in the terminal means that the permissions are referring
to a file. If you were getting the permissions to a directory, you would see a d in the front for
"directory".

Directory permissions

The permissions for directories are similar but distinct for files. Though directories use the
same rwx format, the symbols have slightly different meanings.

The following table illustrates the meanings of each permission for directories:
 Directory Permission Permissible action(s)

r List directory contents using ls command

w Add or remove files or directories

x Enter directory using cd command

Setting appropriate permissions on directories is a best practice for both security and stability
reasons. Though this reading focuses on security, you will learn more about other reasons for setting
file permissions and ownership later in this course.

Making a file private

You can revoke read permissions from your group and all other users by using the chmod command.
Ensure successful modification by using the ls -l command again:

1. chmod go-r my_new_file

2. ls -l my_new_file

3. -rw------- 1 theia users 24 Dec 22 18:49 my_new_file

In the chmod command, go-r is the permission change to be applied, which in this case means
removing for the group (g) and others (o) the read (r) permission. The chmod command can be used
with both files and directories.

Executable files - looking ahead

You've learned what it means to read or write to a file, but what does it mean to have permissions
to execute a file in Linux?

A Linux file is executable if it contains instructions that can be directly interpreted by the operating
system. Basically, an exectuable file is a ready-to-run program. They're also referred to
as binaries or executables.

In this course, you will become very familiar with a particular kind of executable called a script, which
is a program written in a scripting language. You'll learn all about shell scripting, or more
specifically Bash scripting, which is writing scripts in Bash (born-again shell), a very popular shell
scripting language. A shell script is a plain text file that can be interpreted by a shell.

Formally speaking, for a text file to be considered an executable shell script for a given user, it needs
to have two things:

1. Execute permissions set for that user

2. A directive, called a "shebang", in its first line to declare itself to the operating system as a
binary

All of this will become more clear to you soon when we get to the topic of shell scripting.

Summary

In this reading, you learned that:

  There are three possible levels of file ownership in Linux - user, group, and other - which
determine who can read, write to, and execute a file

 You can use the ls -l command to view file and directory permissions

 You can change permissions on a file by using the chmod command

Exercise 1 - Viewing and modifying file access permissions

1.1 View file access permissions

Required files:

Run the following code to download the required files for this exercise:

1. cd /home/project

2. wget https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBM-LX0117EN-
SkillsNetwork/labs/module%201/usdoi.txt

Each file and each directory in your Linux system has permissions set for three permission categories:
the 'user', the 'group', and 'all users' (or 'other').

The following permissions are set for each file and directory:

Permission Symbol

Read r

Write w

Execute x

To see the permissions currently set for a file, run the ls command with the -l option.

For example, to see the permissions for the file named usdoi.txt in your current directory, enter the
following:

1. ls -l usdoi.txt

A sample output looks like the following:

-rw-r--r-- 1 theia theia 8121 May 31 16:45 usdoi.txt

The permissions set here are rw-r--r--. The - preceeding these permissions indicates that usdoi.txt is a
file. If it were a directory, you would see a d instead of the -.

The first three entries correspond to the current user, the next three correspond to the group, and
the last three are for all others. You can see the user has read and write permissions, while the user
group only has read permission, and all other users have only read permission. No users have
execute permission, as indicated by the - instead of an x in the third position for each user category.
1.2 Change file access permissions

chmod

The chmod or change mode command lets you change the permissions set for a file.

Specify which permissions to change with a combination of the following characters:

Option Description

r, w, x Permissions: read, write, and execute

u,g, o User categories: user, group, and all others

+, - Operations: grant and revoke

The following command revokes read permissions for all users (user, group, and other) on the
file usdoi.txt:

1. chmod -r usdoi.txt

You can verify the changed permissions by entering:

1. ls -l usdoi.txt

To grant read access to all users on usdoi.txt, enter:

1. chmod +r usdoi.txt

Verify the changed permissions again with the following:

1. ls -l usdoi.txt

Now to remove the read permission only for 'other' category, enter the following:

1. chmod o-r usdoi.txt

Verify the changed permissions as follows:

1. ls -l usdoi.txt

Exercise 2 - Understanding directory access permissions

2.1 View default directory access permissions

Recall the following table, which illustrates the meanings of each permission for directories with
examples of allowable operations for a given directory.

Directory Permission Permissible action(s)

r list directory contents using ls command

w add/remove files or directories from directory

 Directory Permission Permissible action(s)

x enter directory using cd command

For this exercise, first move to your project directory and create a new directory called test:

1. cd /home/project

2. mkdir test

Check the default permissions that the system sets for your new test directory:

1. ls -l

As you can see from the resulting output:

1. total 12

2. drwxr-sr-x 2 theia users 4096 May 15 14:06 test

3. -rw-r----- 1 theia users 8121 Sep 28 2022 usdoi.txt

You, "theia", as the owner of test, have read, write, and execute permissions set by default. But all
others only have read and execute permissions set and cannot write to your test directory. This
means users outside your group can't add or remove files from test. They can, however, explore your
directory to see what files and directories exist there.

Note: You might be wondering what that s permission is in the execute slot for your group.
The s stands for "special permission". It means that any new files created within the directory will
have their group ownership set to be the same as the directory owner. We won't go into this level of
detail in this course, but you can learn more about advanced Linux permissions here: Linux
permissions: SUID, SGID, and sticky bit.

Go ahead and verify for yourself that you have permission to run the following commands. Change
the directory to your test directory, create a new directory within it, then return to your parent
directory:

1. cd test

2. mkdir test2

3. cd ../

2.2 Remove user execute permissions on your test directory

Remove your user execute permissions on test using the following command:

1. chmod u-x test

Now, what happens when you try to change directories to test?

1. cd test

You get an error message!

bash: cd: test: Permission denied

As you just removed execute permissions for yourself on your test directory, you can no longer make
it your present working directory. However, you can still "read" it with the ls command:

1. ls -l

Even though you have "write" permissions set, you can't actually create a new directory within test,
because removing execute permissions overrides write permissions. For example, entering,

1. mkdir test/test3

throws an error:

mkdir: cannot create directory ‘test/test’: Permission denied

This time, try restoring execute permissions on test and denying write permissions. Then verify your
changes:

1. chmod u+x test

1. chmod u-w test

2. ls -l

Now you can go into test, but you still can't write to it! Entering

1. cd test

2. mkdir test_again

throws the error:

mkdir: cannot create directory ‘test_again’: Permission denied

Practice exercises

1. List the permissions set for the file usdoi.txt that you downloaded to your project directory at
the beginning of the lab.

Solution: cd /home/project

ls -l usdoi.txt

2. Revoke the write permission on usdoi.txt for the user, and verify your result.

Solution: chmod u-w usdoi.txt

ls -l usdoi.txt

You should only see read permissions set for all three user categories.

3. What happens if you try to delete usdoi.txt after revoking write permissions for the user?

Use the rm command to find out.

Solution: If you run the following command:

1. rm usdoi.txt

You should see the following prompt:

 1. rm: remove write-protected regular file \'usdoi.txt\'?

Entering y will override the write protection and delete the file, which you can verify as follows:

1. ls usdoi.txt

4. Create a new directory called tmp_dir in your home directory.

Solution: mkdir tmp_dir

5. View the permissions of the newly created directory, tmp_dir.

Solution: ls -ld tmp_dir

6. Revoke the user write permission for tmp_dir.

Solution: chmod u-w tmp_dir

7. Check whether you can create a subdirectory of tmp_dir called sub_dir.

Solution: cd tmp_dir

mkdir sub_dir

You should be able to enter the directory with the cd command, but you should see a "permission
denied" error after entering the mkdir command.

Solution: 2. mkdir tmp_dir/sub_dir

Again, you should see a "permission denied" error for the mkdir command.

Summary

In this lab, you learned that:

 Files and directories can have read, write, and execute permissions for the user, group, and
others

 You can view the permissions for all files and directories within a directory by using ls -l

 You can modify permissions for a file by using chmod

 Changing permissions on a directory will change who can do what to the directory and its
files

Exercise 1 - Viewing file contents

cat
more
less

In this exercise, you will learn how to explore file contents using the cat, more, and less commands to
display the file contents in your terminal window.

Begin by changing directories to your default home directory, ~, or \home\theia:

1. cd ~
Using the ls command, you should see a file called entrypoint.sh. The .sh is a convention used to
identify a text file as being a shell script.

Next, let's take a look inside this file.

1.1. Viewing file content with the cat command

The cat command displays the contents of the file and exits back to the command prompt as follows:

1. cat entrypoint.sh

It only displays the tail end of the file, so if the file is too long to fit on the terminal, you won't be able
to see some of its contents.

Although the cat command may not be the best way to view the contents of a file, especially larger
files, it is quite useful for shell scripting applications. For example, it is often used to concatenate, or
append one file onto another.

1.2. Viewing file content with the more command

A better alternative to the cat command for viewing file contents is the more command. By entering
the following command:

1. more entrypoint.sh

you will see the top portion of the file first.

Tip: The first line of this particular file, #!/bin/bash, is called a shebang. Basically, this shebang line
makes the file a bash script by invoking the bash shell. You will learn more about shebang lines later
in this course.

When using the more command, you can see only as many lines as will fit on your terminal window
at once.

To see the next portion of the file, just press your spacebar. You can keep paging this way, tapping
the spacebar until you reach the end of the file. Once you reach the last page, you will exit back to
the command prompt.

Another way to exit is simply to type q, which quits and returns to the command prompt.

1.3. Scrolling through file content with the less command

What if you want to move up and down through the file, not just downward? In this case, you can
use the less command:

1. less entrypoint.sh

Just like more, the less command displays the first page of the file. What's useful about less is that
you can use it to move around the file, page by page, using the Page Up and Page Down keys.

You can also scroll up and down through the file line-by-line, using the Up Arrow and Down
Arrow keys, ↑ and ↓.

Unlike more, less does not automatically exit when you reach the end of a file, allowing you the
option to continue scrolling around. You can quit at any time by typing q.

Exercise 2 - Viewing text file contents

In this exercise, you will work with a few more commands for viewing the content of text files.

To begin, run the following commands:

1. cd /home/project

1. wget https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBM-DB0250EN-
SkillsNetwork/labs/Bash%20Scripting/usdoi.txt

The wget command downloads a text file called usdoi.txt from the provided URL. You'll see this
command again later in the context of networking commands. You can check to see if you
successfully downloaded the usdoi.txt by using the ls command.

2.1. Display the first N lines of a file

head

By default, head will print the first 10 lines of a file. To use it with usdoi.txt, enter the following:

1. head usdoi.txt

You can also specify the number of lines to be printed. Print only the first 3 lines of text from the
file usdoi.txt by entering:

1. head -3 usdoi.txt

2.2. Display the last N lines of a file

tail

By default, tail will print the last 10 lines of the file usdoi.txt:

1. tail usdoi.txt

Just like with head, you can specify the number of lines to be printed. Print the last 2 lines of the
file usdoi.txt by entering the following:

1. tail -2 usdoi.txt

Exercise 3 - Getting basic text file stats

3.1. Count lines, words, or characters from a text file

wc

If you want to find the number of lines, words, and characters in a file like usdoi.txt, enter the
following command:

1. wc usdoi.txt

The output contains the number of lines, followed by the number of words, followed by the number
of characters in the file.

To get just the count of lines in usdoi.txt, use the -l option:

1. wc -l usdoi.txt

Similarly, for the count of words in usdoi.txt, use the -w option:

 1. wc -w usdoi.txt

To print the number of characters in usdoi.txt, use the -c option:

1. wc -c usdoi.txt

Exercise 4 - Basic text wrangling: sorting lines and dropping duplicates

4.1. Sort and display lines of file alphanumerically

sort

You can use the sort command to display the lines of a file sorted alphanumerically.

To view the lines of usdoi.txt sorted alphanumerically, enter:

1. sort usdoi.txt

To view those lines sorted in reverse order, enter:

1. sort -r usdoi.txt

4.2. Drop consecutive duplicated lines and display result

uniq

First download the following file:

1. wget https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBM-LX0117EN-
SkillsNetwork/labs/module%201/zoo.txt

View the raw contents of zoo.txt with the cat command:

1. cat zoo.txt

View the contents of zoo.txt with identical, consecutive lines dropped using the uniq command:

1. uniq zoo.txt

The uniq line will drop any lines in the file that are identical and consecutive. This is similar to what is
known as "dropping duplicates". As you can see from this example, however, there can still be
duplicated lines left over if these lines are not repeated right after the other.

Exercise 5 - Basic text wrangling: extracting lines and fields

5.1. Extract lines matching a specified criterion

grep

The grep command allows you to specify a pattern and search for lines within a file that match that
pattern.

For example, the following command prints all lines in the file usdoi.txt which contain the
word people:

1. grep people usdoi.txt

Some frequently used options for grep include:

 Option Description

-n Along with the matching lines, also print the line numbers

-c Get the count of matching lines

-i Ignore the case of the text while matching

-v Print all lines which do not contain the pattern

-w Match only if the pattern matches whole words

You can use these options to print all the lines from the /etc/passwd file which do not contain the
pattern login:

1. grep -v login /etc/passwd

5.2. Extract fields from lines of text

cut

The cut command allows you to view only specific fields from each line of text in a file.

For example, you can use cut with the -c option to view only the first two characters of each line:

1. cut -c -2 zoo.txt

Or to view each line starting from the second character:

1. cut -c 2- zoo.txt

The cut command can also be used to extract a field from a delimited file.
To demonstrate this, start by downloading and taking a look at the following comma-separated file:

1. wget https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBM-LX0117EN-
SkillsNetwork/labs/v4_new_content/labs/names_and_numbers.csv

2. cat names_and_numbers.csv

Now you can extract just the phone numbers for each person listed in the file using the -d (delimiter)
and f (field) options as follows:

1. cut -d "," -f2 names_and_numbers.csv

-d "," tells the command that the delimiter is a comma, and -f2 tells it to extract the second field.

Exercise 6 - Basic text wrangling: merging lines as fields

6.1. Merge text files line-by-line, aligned as columns

paste

Use the paste command to merge lines of multiple files together.

Download the following file:

 1. wget https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBM-LX0117EN-
SkillsNetwork/labs/module%201/zoo_ages.txt

Then use the paste command to view the two files merged together, line-by-line, as columns
delimited by a Tab character:

1. paste zoo.txt zoo_ages.txt

Try changing the delimiter. Instead of the default Tab delimiter, you can specify a comma , as follows:

1. paste -d "," zoo.txt zoo_ages.txt

Practice Exercises

Before you begin, ensure you're working in your home directory by entering:

1. cd ~

2. pwd

1. Display the number of lines in the /etc/passwd file.

Solution: wc -l /etc/passwd

2. Display the lines that contain the string "not installed" in /var/log/bootstrap.log.

Solution:

1. grep "not installed" /var/log/bootstrap.log

3. The text file at https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBM-

DB0250EN-SkillsNetwork/labs/Bash%20Scripting/top-sites.txt contains a list of popular websites.
Find all the websites on the list that have the word "org" in them.

4. Print the first seven lines of top-sites.txt.

Solution: head -n 7 top-sites.txt

5. Print the last seven lines of top-sites.txt.

Solution: tail -n 7 top-sites.txt

6. Print the first three characters of each line from top-sites.txt.

Solution: cut -c -3 top-sites.txt

7. Extract and view only the names, without their phone numbers, from the
file names_and_numbers.csv.

Solution: cd /home/project

cut -d "," -f 1 names_and_numbers.csv

Summary

In this lab, you learned how to:

 View file contents with cat, more, and less

 See the first and last N lines of a file using head and tail
  Find the number of lines, words, and characters in a file with wc

 Sort lines and drop duplicates using sort and uniq

 Extract lines and fields from a file with grep and cut

 Merge text files using paste

Exercise 1 - View configuration info about your network

1.1. Display your system's hostname and IP address

hostname

A hostname is a name that is assigned to a computer or device on a network, and it is used to

identify and communicate with that device.

To view the current hostname, run the command below:

1. 1

1. hostname

An IP address (Internet Protocol address) is a numerical label assigned to each device connected to
a computer network that uses the Internet Protocol for communication.

You can use the -i option to view the IP address of the host:

1. 1

1. hostname -i

1.2. Display network interface configuration

ifconfig

The ifconfig command is used to configure or display network interface parameters for a network.

To display the configuration of all network interfaces of your system, enter:

1. 1

1. ifconfig

To display the configuration of a particular device, such as the ethernet adapter eth0, enter:

1. 1

1. ifconfig eth0

eth0 is usually the primary network interface that connects your server to the network.

You can see your server's IP address in line 2 after the word inet.
 

1Hands-on Lab: Working with Networking Commands2About Skills Network Cloud IDE3Exercise 1 -
View configuration info about your network4Exercise 2 - Test network connectivity5Exercise 3 -
View or download data from a server6Practice exercises7Summary

Exercise 2 - Test network connectivity

2.1. Test connectivity to a host

ping

Use the ping command to check if www.google.com is reachable. The command keeps pinging
data packets to server at www.google.com and prints the response it gets back. (Press Ctrl+c to
stop pinging.)

1. 1

1. ping www.google.com

If you want to ping only a limited number of times, use -c option.

1. 1

1. ping -c 5 www.google.com
 

1Hands-on Lab: Working with Networking Commands2About Skills Network Cloud IDE3Exercise 1 -
View configuration info about your network4Exercise 2 - Test network connectivity5Exercise 3 - View
or download data from a server6Practice exercises7Summary

Exercise 3 - View or download data from a server

3.1. Transfer data from a server

curl

You can use curl to access the file at the following URL and display the file's contents on your screen:

1. 1

1. curl https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBM-DB0250EN-
SkillsNetwork/labs/Bash%20Scripting/usdoi.txt

To access the file at the given URL and also save it in your current working directory, use the -
O option:

1. 1

1. curl -O https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBM-DB0250EN-
SkillsNetwork/labs/Bash%20Scripting/usdoi.txt

You can also use curl to view the HTML code for any web page if you know its URL.

3.2. Download file(s) from a URL

wget
The wget command is similar to curl, however its primary use is for file downloading. One unique
feature of wget is that it can recursively download files at a URL.

To see wget in action, first remove usdoi.txt from your current directory:

1. 1

1. rm usdoi.txt

then download it again using wget as follows:

1. 1

1. wget https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBM-DB0250EN-
SkillsNetwork/labs/Bash%20Scripting/usdoi.txt

1Hands-on Lab: Working with Networking Commands2About Skills Network Cloud IDE3Exercise 1 -
View configuration info about your network4Exercise 2 - Test network connectivity5Exercise 3 - View
or download data from a server6Practice exercises7Summary

Practice exercises

Before you begin, ensure you're in your /home/project directory by entering:

1. 1

2. 2
 1. cd `/home/project`

2. pwd

1. Display your host's IP address.

Solution:

1. 1

1. hostname -i

2. Get connectivity stats on your connection to www.google.com.

Solution:

1. 1

1. ping www.google.com

3. View info about your ethernet adapter eth0.

Solution:

1. 1

1. ifconfig eth0

4. View the HTML code for www.google.com's landing page.

Solution:

1. 1

1. curl www.google.com

5. Download the HTML code for www.google.com's landing page.

Solution:

1. 1
 1. wget www.google.com

Note: wget saves the HTML code as index.html. You can check this with:

1. 1

1. ls -l

1Hands-on Lab: Working with Networking Commands2About Skills Network Cloud IDE3Exercise 1 -
View configuration info about your network4Exercise 2 - Test network connectivity5Exercise 3 - View
or download data from a server6Practice exercises7Summary

Summary

In this lab, you learned how to:

 View your network configuration using the hostname and ifconfig commands

 Test a network connection using the ping command

 Transfer data using the curl and wget commands

Authors

Jeff Grossman
Ramesh Sannareddy
Sam Prokopchuk
Other contributors

Rav Ahuja

© IBM Corporation. All rights reserved.

Hands-on Lab: Archiving and Compressing Files

1Hands-on Lab: Archiving and Compressing Files2About Skills Network Cloud IDE3Exercise 1 - File
and folder archiving and compression4Summary

Exercise 1 - File and folder archiving and compression

1.1. Create and manage file archives

tar

The tar command allows you to pack multiple files and directories into a single archive file.

The following command creates an archive of the entire /bin directory and writes the archive to a
single file named bin.tar.

The options used are as follows:

Option Description

-c Create new archive file

-v Verbosely list files processed

-f Archive file name

1. 1

1. tar -cvf bin.tar /bin

To see the list of files in the archive, use the -t option:

 1. 1

1. tar -tvf bin.tar

To untar the archive or extract files from the archive, use the -x option:

1. 1

1. tar -xvf bin.tar

Use the ls command to verify that the folder bin is extracted.

1. 1

1. ls -l

1.2. Package and compress archive files

zip

The zip command allows you to compress files.

The following command creates a zip file named config.zip consisting of all the files with
extension .conf in the /etc directory.

1. 1

1. zip config.zip /etc/*.conf

The -r option can be used to zip an entire directory.

The following command creates an archive of the /bin directory.

1. 1

1. zip -r bin.zip /bin

1.3. Extract, list, or test compressed files in a ZIP archive

unzip

The unzip command allows you to extract files.

To list the files of the archive config.zip, enter the following:

1. 1

1. unzip -l config.zip
The following command extracts all the files in the archive bin.zip.

1. 1

1. unzip -o bin.zip

We added the -o option to force overwrite in case you run the command more than once.

You should see a folder named bin created in your directory.

Summary

In this lab, you learned that:

 tar allows you to pack multiple files and directories into a single archive file

 zip allows you to compress files

 unzip allows you to extract files

Module 2 Summary and Highlights

 A shell is an interactive user interface. You use shell commands to navigate and work with
files and directories.

 The curl and wget commands display and download files from URLs, and
the cat and tail commands display file contents.

 You can get user information with the whoami and id commands, and get operating system
information using the uname command. You can check system disk usage using
the df command and monitor processes and resource usage with ps and top. Print string or
variable value using echo, print and extract information about the date with
the date command, and read the manual for any command using man.

 ls lists all files and directories within a specified directory tree and cd navigates between
directories. The find command finds files in your directories.

 Relative paths are relative to your current working directory, while absolute paths stand
independently

 You can create files and directories with the touch and mkdir commands, delete them
with rm and rmdir, and copy and move them cp and mv.

 The cat, more, head, and tail commands allow you to sort and view file contents or view
only a certain number of lines. Determine line, word, and character counts with wc.

 You can use sort to view the lines of a file alphanumerically and uniq to remove repeated
lines from your view. grep gets the lines of a file that match your desired criteria,
and cut extracts slices and fields from lines. You can merge lines from different files
using paste.

 hostname and ifconfig allow you to view the network configuration. You can test a network
connection using ping and send and receive data using curl and wget.

 Compression preserves storage space, speeds data transfer, and reduces system load.
  zip compresses files and folders prior to archiving them. tar archives and compresses files
and directories into a tarball. unzip unpacks and decompresses a zipped archive, and tar can
also decompress and unpack a tar.gz archive.

Module 2 Cheat Sheet - Introduction to Linux Commands

Getting information

Return your user name:

1. whoami
Return your user and group id:
1. id
Return operating system name, username, and other info:
1. uname -a

Display reference manual for a command:

1. man top

List available man pages, including a brief description for each command:
1. man -k .

Get help on any command (for eg: curl):

1. curl --help

This provides a brief overview of the curl command's usage and options.
Return the current date and time:
1. date

Navigating and working with directories

List files and directories by date, newest to last:

1. ls -lrt
Find files in directory tree that end in .sh:
1. find -name \'\*.sh\'

Return path to present working directory:

1. pwd

Make a new directory:

1. mkdir new_folder

Change the current directory:

Up one level:
1. cd ../

To home:
1. cd ~` or `cd

To some other directory: cd path_to_directory

Remove directory verbosely:
1. rmdir temp_directory -v

Monitoring system performance and status

List selection of/all running processes and their PIDs:

1. ps

1. ps -e

Display resource usage:

1. top

List mounted file systems and usage:

1. df
Creating, copying, moving, and deleting files:

Create an empty file or update existing file's timestamp:

1. touch a_new_file.txt

Copy a file:
1. cp file.txt new_path/new_name.txt

Change file name or path:

1. mv this_file.txt that_path/that_file.txt

Remove a file verbosely:

1. rm this_old_file.txt -v

Working with file permissions

Change/modify file permissions to 'execute' for all users:

1. chmod +x my_script.sh

Change/modify file permissions to 'execute' only for you, the current user:
1. chmod u+x my_file.txt

Remove 'read' permissions from group and other users:

1. chmod go-r

Displaying file and string contents

Display file contents:

1. cat my_shell_script.sh

Display file contents page-by-page:

 1. more ReadMe.txt

Display first 10 lines of file:

1. head -10 data_table.csv

Display last 10 lines of file:

1. tail -10 data_table.csv

Display string or variable value:

1. echo "I am not a robot"
2. echo "I am $USERNAME"

Basic text wrangling

Sorting lines and dropping duplicates:

Sort and display lines of file alphanumerically:
1. sort text_file.txt

In reverse order:
1. sort -r text_file.txt

Drop consecutive duplicated lines and display result:

1. uniq list_with_duplicated_lines.txt

Displaying basic stats:

Display the count of lines, words, or characters in a file:
Lines:
1. wc -l table_of_data.csv

Words:
1. wc -w my_essay.txt
Characters:
1. wc -m some_document.txt

Extracting lines of text containing a pattern:

Some frequently used options for grep:

Option Description

-n Print line numbers along with matching lines

-c Get the count of matching lines

-i Ignore the case of the text while matching

-v Print all lines which do not contain the pattern

-w Match only if the pattern matches whole words

Extract lines containing the word "hello", case insensitive and whole words only:
1. grep -iw hello a_bunch_of_hellos.txt

Extract lines containing the pattern "hello" from all files in the current directory
ending in .txt:
1. grep -l hello *.txt

Merge two or more files line-by-line, aligned as columns:

Suppose you have three files containing the first and last names of your customers, plus their
phone numbers.
Use paste to align file contents into a Tab-delimited table, one row for each customer:
1. paste first_name.txt last_name.text phone_number.txt

Use a comma as a delimiter instead of the default Tab delimiter:

1. paste -d "," first_name.txt last_name.text phone_number.txt

Use the cut command to extract a column from a table-like file:

Suppose you have a text file whos rows consist of first and last names of customers,
delimited by a comma.
Extract first names, line-by-line:
1. cut -d "," -f 1 names.csv

Extract the second to fifth characters (bytes) from each line of a file:
1. cut -b 2-5 my_text_file.txt

Extract the characters (bytes) from each line of a file, starting from the 10th byte to the
end of the line:
1. cut -b 10- my_text_file.txt

Compression and archiving

Archive a set of files:

1. tar -cvf my_archive.tar.gz file1 file2 file3

Compress a set of files:

1. zip my_zipped_files.zip file1 file2
1. zip my_zipped_folders.zip directory1 directory2

Extract files from a compressed zip archive:

1. unzip my_zipped_file.zip
2. unzip my_zipped_file.zip -d extract_to_this_direcory

Working with networking commands

Print hostname:
1. hostname

Send packets to URL and print response:

1. ping www.google.com

Display or configure system network interfaces:

 1. ifconfig
2. ip

Display contents of file at a URL:

1. curl <url>

Download file from a URL:

1. wget <url>
Co

Reading: A Brief Introduction to Shell Variables

Learning Objectives

After completing this reading, you will be able to:

 Describe shell variables

 Create shell variables

What is a shell variable?

Shell variables offer a powerful way to store and later access or modify information such as numbers,
character strings, and other data structures by name. Let's look at some basic examples to get the
idea.

Consider the following example.

1. $ firstname=Jeff

2. $ echo $firstname

3. Jeff

The first line assigns the value Jeff to a new variable called firstname. The next line accesses and
displays the value of the variable, using the echo command along with the special character $ in
front of the variable name to extract its value, which is the string Jeff.

Thus, we have created a new shell variable called firstname for which the value is Jeff.

This is the most basic way to create a shell variable and assign it to a value all in one step.

Reading user input into a shell variable at the command line

Here's another way to create a shell variable, using the read command.
After entering

1. $ read lastname
on the command line, the shell waits for you to enter some text:

1. $ read lastname

2. Grossman

3. $

Now we can see that the value Grossman has just been stored in the variable lastname by
the read command:

1. $ read lastname

2. Grossman

3. $ echo $lastname

4. Grossman

By the way, notice that you can echo the values of multiple variables at once.

1. $ echo $firstname $lastname

2. Jeff Grossman

As you will soon see, the read command is particularly useful in shell scripting. You can use it within a
shell script to prompt users to input information, which is then stored in a shell variable and available
for use by the shell script while it is running. You will also learn about command line arguments,
which are values that can be passed to a script and automatically assigned to shell variables.

Summary

In this reading, you learned that:

 Shell variables store values and allow users to later access them by name

 You can create shell variables by declaring a shell variable and value or by using
the read command

Exercise 1 - Create and execute a basic shell script

In this exercise, you will create a simple script which will do the following:

 Accept a user name

 Print a welcome message to the user

You will also add comments to the script, which are lines starting with #. Comments are not executed
by the shell.

When used appropriately, comments can make a shell script more readable and help in debugging
the script.
1.1. Create a new script file

Step 1: On the menu on the lab screen, use File->New File to create a new file.

Step 2: Name it as greet.sh and click OK

Step 3: Copy and paste the following lines into the newly created file.

1. 1

2. 2

3. 3

4. 4

5. 5

6. 6

7. 7

8. 8

9. 9

10. 10

11. 11

12. 12
 13. 13

14. 14

15. 15

1. # This script accepts the user\'s name and prints

2. # a message greeting the user

3.

4. # Print the prompt message on screen

5. echo -n "Enter your name :"

6.

7. # Wait for user to enter a name, and save the entered name into the variable \'name\'

8. read name

9.

10. # Print the welcome message followed by the name

11. echo "Welcome $name"

12.

13. # The following message should print on a single line. Hence the usage of \'-n\'

14. echo -n "Congratulations! You just created and ran your first shell script "

15. echo "using Bash on IBM Skills Network"

Step 4: Save the file using the File->Save menu option.

1.2. Execute the script

Open a new terminal by clicking on the menu bar and selecting Terminal->New Terminal, as in the
image below.
This will open a new terminal at the bottom of the screen.
Run the commands below in the newly opened terminal.

Let's check the permissions for this new file by entering the following:

1. 1

1. ls -l greet.sh

If the file exists and has read permissions, run the following command to execute it:

1. 1

1. bash greet.sh

The message Enter your name : appears on screen.

Type your name and press the Enter key.

You should now see the welcome messages displayed on screen with your entered name.

Congratulations! You have succesfully executed your first Bash shell script.
Exercise 2 - Using a shebang line

In this exercise, you will edit the greet.sh script you created in the previous exercise by adding a
'shebang' and making it an executable file.

This is done to ensure that the name of the script can be used like a command. Adding this special
shebang line lets you specify the path to the interpreter of the script - in this case, the Bash shell.

Follow the steps below to learn how to add a shebang to your script.

2.1. Find the path to the interpreter

The which command helps you find out the path of the command bash.

1. 1

1. which bash

In this case, it returns the path /bin/bash.

2.2. Edit the script greet.sh and add the shebang line to the script

Open the file and add the following line at the beginning of the script:

1. 1

1. #! /bin/bash

The script should now look like the following:

2.3. Check the permissions of the script

One more step needs to be completed to make greet.sh completely executable by name.

To add the execute permission for the user on greet.sh, enter the following:

1. 1

1. chmod +x greet.sh
Verify whether the execute permission is granted.

Tip: Generally it's not a good idea to grant permissions to a script for all users, groups, and others.
It's more appropriate to limit the execute permission to only the owner, or the user who created the
file (you).

To change permissions for greet.sh to make the file executable for the user, run the command below:

1. 1

1. chmod u+x greet.sh

Verify the permissions using the command below:

1. 1

1. ls -l greet.sh

If you wish to grant execute permission to everyone, you need to run the command chmod +x
greet.sh.

2.4. Execute the script.

Enter the command given below to run the shell script.

1. 1

1. ./greet.sh

The . here refers to the current directory. You are telling Linux to execute the script greet.sh and that
it can be found in the current directory.

Practice exercise

1. Create a script named greetnew.sh that takes the first and last names of the user, saves them in
corresponding variables firstname and lastname, and prints a welcome message, such as "Hello
<firstname> <lastname>".

Use the read command and echo commands. Write comments. Make sure to add the shebang line.

Solution:

Step 1: Create a new file named greetnew.sh.

Step 2: Add the following lines to the file:

1. 1

2. 2
3. 3

4. 4

5. 5

6. 6

7. 7

8. 8

9. 9

10. 10

11. 11

12. 12

13. 13

14. 14

15. 15

16. 16

17. 17

18. 18

19. 19

1. #! /bin/bash

2.

3. # This script accepts the user\'s name and prints

4. # a message greeting the user

5.

6. # Print the prompt message on screen

7. echo -n "Enter your firstname :"

8.

9. # Wait for user to enter a name, and save the entered name into the variable \'name\'

10. read firstname

11.

12. # Print the prompt message on screen

13. echo -n "Enter your lastname :"

14.
 15. # Wait for user to enter a name, and save the entered name into the variable \'name\'

16. read lastname

17.

18. # Print the welcome message followed by the name

19. echo "Hello $firstname $lastname."

Step 3: Save the file.

Step 4: Add the execute permission to greetnew.sh for the owner:

1. 1

1. chmod u+x greetnew.sh

Step 5: Execute the file from the command prompt using the following command:

1. 1

1. ./greetnew.sh

Summary

In this lab, you learned how to:

 Create and execute a simple Bash shell script

 Implement the shebang directive #! /bin/bash in a Bash shell script

What are pipes?

Put simply, pipes are commands in Linux which allow you to use the output of one command as the
input of another.

Pipes | use the following format:

1. [command 1] | [command 2] | [command 3] ... | [command n]

There is no limit to the number of times you can chain pipes in a row!

In this lab, you'll take a closer look at how you can use pipes and filters to solve basic data processing
problems.

Pipe examples

Combining commands

Let's start with a commonly used example. Recall the following commands:

 sort - sorts the lines of text in a file and displays the result

 uniq - prints a text file with any consecutive, repeated lines collapsed to a single line

With the help of the pipe operator, you can combine these commands to print all the unique lines in
a file.

Suppose you have the file pets.txt with the following contents:

1. $ cat pets.txt

2. goldfish

3. dog

4. cat

5. parrot

6. dog

7. goldfish

8. goldfish

If you only use sort on pets.txt, you get:

1. $ sort pets.txt

2. cat

3. dog

4. dog

5. goldfish

6. goldfish

7. goldfish

8. parrot

The file is sorted, but there are duplicated lines of "dog" and "goldfish".
On the other hand, if you only use uniq, you get:

1. $ uniq pets.txt

2. goldfish

3. dog

4. cat

5. parrot

6. dog

7. goldfish

This time, you removed consecutive duplicates, but non-consecutive duplicates of "dog" and
"goldfish" remain.

But by combining the two commands in the correct order - by first using sort then uniq - you get
back:

1. $ sort pets.txt | uniq

2. cat

3. dog

4. goldfish

5. parrot

Since sort sorts all identical items consecutively, and uniq removes all consecutive duplicates,
combining the commands prints only the unique lines from pets.txt!

Applying a command to strings and files

Some commands such as tr only accept standard input - normally text entered from your keyboard -
but not strings or filenames.

 tr (translate) - replaces characters in input text

1. tr [OPTIONS] [target characters] [replacement characters]

In cases like this, you can use piping to apply the command to strings and file contents.

With strings, you can use echo in combination with tr to replace all the vowels in a string with
underscores _:

1. $ echo "Linux and shell scripting are awesome\!" | tr "aeiou" "_"

2. L_n_x _nd sh_ll scr_pt_ng _r_ _w_s_m_!

To perform the complement of the operation from the previous example - or to replace all
the consonants (any letter that is not a vowel) with an underscore - you can use the -c option:

1. $ echo "Linux and shell scripting are awesome\!" | tr -c "aeiou" "_"

2. _i_u__a_____e______i__i___a_e_a_e_o_e_
With files, you can use cat in combination with tr to change all of the text in a file to uppercase as
follows:

1. $ cat pets.txt | tr "[a-z]" "[A-Z]"

2. GOLDFISH

3. DOG

4. CAT

5. PARROT

6. DOG

7. GOLDFISH

8. GOLDFISH

The possibilities are endless! For example, you could add uniq to the above pipeline to only return
unique lines in the file, like so:

1. $ sort pets.txt | uniq | tr "[a-z]" "[A-Z]"

2. CAT

3. DOG

4. GOLDFISH

5. PARROT

Extracting information from JSON Files:

Let's see how you can use this json file to get the current price of Bitcoin (BTC) in USD, by using grep
command.

1. {

2. "coin": {

3. "id": "bitcoin",

4. "icon": "https://static.coinstats.app/coins/Bitcoin6l39t.png",

5. "name": "Bitcoin",

6. "symbol": "BTC",

7. "rank": 1,

8. "price": 57907.78008618953,

9. "priceBtc": 1,

10. "volume": 48430621052.9856,

11. "marketCap": 1093175428640.1146,

12. "availableSupply": 18877868,

 13. "totalSupply": 21000000,

14. "priceChange1h": -0.19,

15. "priceChange1d": -0.4,

16. "priceChange1w": -9.36,

17. "websiteUrl": "http://www.bitcoin.org",

18. "twitterUrl": "https://twitter.com/bitcoin",

19. "exp": [

20. "https://blockchair.com/bitcoin/",

21. "https://btc.com/",

22. "https://btc.tokenview.com/"

23. ]

24. }

25. }

Copy the above output in a file and name it as Bitcoinprice.txt.

The JSON field you want to grab here is "price": [numbers].[numbers]". To get this, you can use the
following grep command to extract it from the JSON text:

1. grep -oE "\"price\"\s*:\s*[0-9]*?\.[0-9]*"

Let's break down the details of this statement:

 -o tells grep to only return the matching portion

 -E tells grep to be able to use extended regex symbols such as ?

 \"price\" matches the string "price"

 \s* matches any number (including 0) of whitespace (\s) characters

 : matches :

 [0-9]* matches any number of digits (from 0 to 9)

 ?\. optionally matches a .

Use the cat command to get the output of the JSON file and pipe it with the grep command to get
the required output.

1. cat Bitcoinprice.txt | grep -oE "\"price\"\s*:\s*[0-9]*?\.[0-9]*"

You can also extract information directly from URLs and retreive any specific data using such grep
commands.

Click here to see the process of extracting information directly from URLs and retreiving specific data:
1.

2.

3.

4.

5.
6.

7.
 8. Utilize the ‘grep’ command, following the guidelines outlined earlier, to fetch
the necessary data.

Summary

In this reading, you learned that:

 Pipes are commands in Linux which allow you to use the output of one command as the
input of another

 You can combine commands such as sort and uniq to organize strings and text file contents

 You can pipe the output of a curl command to grep to extract components of URL data

Examples of Bash Shell Features

Learning Objectives

After completing this reading, you will be able to:

 List examples of metacharacters

 Use quoting to specify literal or special character meanings

 Implement input and output redirection

 Apply command substitution

 Describe applications for command line arguments

Metacharacters

Metacharacters are characters having special meaning that the shell interprets as instructions.

Metacharacter Meaning

# Precedes a comment

; Command separator

* Filename expansion wildcard

? Single character wildcard in filename expansion

Pound #

The pound # metacharacter is used to represent comments in shell scripts or configuration files. Any
text that appears after a # on a line is treated as a comment and is ignored by the shell.

1. 1

2. 2
 3. 3

4. 4

1. #!/bin/bash

2.

3. # This is a comment

4. echo "Hello, world!" # This is another comment

Comments are useful for documenting your code or configuration files, providing context, and
explaining the purpose of the code to other developers who may read it. It's a best practice to
include comments in your code or configuration files wherever necessary to make them more
readable and maintainable.

Semicolon ;

The semicolon ; metacharacter is used to separate multiple commands on a single command line.
When multiple commands are separated by a semicolon, they are executed sequentially in the order
they appear on the command line.

1. 1

2. 2

3. 3

1. $ echo "Hello, "; echo "world!"

2. Hello,

3. world!

As you can see from the example above, the output of each echo command is printed on separate
lines and follows the same sequence in which the commands were specified.

The semicolon metacharacter is useful when you need to run multiple commands sequentially on a
single command line.

Asterisk *

The asterisk * metacharacter is used as a wildcard character to represent any sequence of characters,
including none.

1. 1

1. ls *.txt

In this example, *.txt is a wildcard pattern that matches any file in the current directory with
a .txt extension. The ls command lists the names of all matching files.
Question mark ?

The question mark ? metacharacter is used as a wildcard character to represent any single character.

1. 1

1. ls file?.txt

In this example, file?.txt is a wildcard pattern that matches any file in the current directory with a
name starting with file, followed by any single character, and ending with the .txt extension.

Quoting

Quoting is a mechanism that allows you to remove the special meaning of characters, spaces, or
other metacharacters in a command argument or shell script. You use quoting when you want the
shell to interpret characters literally.

Symbol Meaning

\ Escape metacharacter interpretation

"" Interpret metacharacters within string

'' Escape all metacharacters within string

Backslash \

The backslash character is used as an escape character. It instructs the shell to preserve the literal
interpretation of special characters such as space, tab, and $. For example, if you have a file with
spaces in its name, you can use backslashes followed by a space to handle those spaces literally:

1. 1

1. touch file\ with\ space.txt

Double quotes " "

When a string is enclosed in double quotes, most characters are interpreted literally, but
metacharacters are interpreted according to their special meaning. For example, you can access
variable values using the dollar $ character:

1. 1

2. 2

1. $ echo "Hello $USER"

2. Hello <username>

Single quotes ' '

When a string is enclosed in single quotes, all characters and metacharacters enclosed within the
quotes are interpreted literally. Single quotes alter the above example to produce the following
output:

1. 1

2. 2

1. $ echo 'Hello $USER'

2. Hello $USER

Notice that instead of printing the value of $USER, single quotes cause the terminal to print the
string "$USER".

Input/Output redirection

Symbol Meaning

> Redirect output to file, overwrite

>> Redirect output to file, append

2> Redirect standard error to file, overwrite

2>> Redirect standard error to file, append

< Redirect file contents to standard input

Input/output (IO) redirection is the process of directing the flow of data between a program and its
input/output sources.

By default, a program reads input from standard input, the keyboard, and writes output to standard
output, the terminal. However, using IO redirection, you can redirect a program's input or output to
or from a file or another program.

Redirect output >

This symbol is used to redirect the standard output of a command to a specified file.

ls > files.txt will create a file called files.txt if it doesn't exist, and write the output of the ls command
to it.

Warning: When the file already exists, the output overwrites all of the file's contents!

Redirect and append output >>

This notation is used to redirect and append the output of a command to the end of a file. For
example,

ls >> files.txt appends the output of the ls command to the end of file files.txt, and preserves any
content that already existed in the file.
Redirect standard output 2>

This notation is used to redirect the standard error output of a command to a file. For example, if you
run the ls command on a non-existing directory as follows,

ls non-existent-directory 2> error.txt the shell will create a file called error.txt if it doesn't exist, and
redirect the error output of the ls command to the file.

Warning: When the file already exists, the error message overwrites all of the file's contents!

Append standard error 2>>

This symbol redirects the standard error output of a command and appends the error message to the
end of a file without overwriting its contents.

ls non-existent-directory 2>> error.txt will append the error output of the ls command to the end of
the error.txt file.

Redirect input <

This symbol is used to redirect the standard input of a command from a file or another command.
For example,

sort < data.txt will sort the contents of the data.txt file.

Command Substitution

Command substitution allows you to run command and use its output as a component of another
command's argument. Command substitution is denoted by enclosing a command in either backticks
(`command`) or using the $() syntax. When the encapsulate command is executed, its output is
substituted in place, and it can be used as an argument within another command. This is particularly
useful for automating tasks that require the use of a command's output as input for another
command.

For example, you could store the path to your current directory in a variable by applying command
substitution on the pwd command, then move to another directory, and finally return to your
original directory by invoking the cd command on the variable you stored, as follows:

1. 1

2. 2

3. 3

1. $ here=$(pwd)

2. $ cd path_to_some_other_directory

3. $ cd $here

Command Line Arguments

Command line arguments are additional inputs that can be passed to a program when the program
is run from a command line interface. These arguments are specified after the name of the program,
and they can be used to modify the behavior of the program, provide input data, or provide output
locations. Command line arguments are used to pass arguments to a shell script.

For example, the following command provides two arguments, arg1, and arg2, that can be accessed
from within your Bash script:

1. 1

1. $ ./MyBashScript.sh arg1 arg2

Summary

In this reading, you learned that:

 Metacharacters such as # ; * and ? are characters that the shell interprets with special
meanings

 Quoting allows you to ensure any special characters, spaces, or other metacharacters are
interpreted literally by the shell

 Input/output redirection redirects a program's input or output to/from a file

 Command substitution allows you to use the output of a command as an argument for
another command

 Command line arguments can be used to pass information to a shell script

Introduction to Advanced Bash Scripting

Estimated time needed: 5 minutes

In the hands-on lab portion of the final project, you will be using more advanced scripting commands
and concepts that the course has not covered yet. This reading will familiarize you with these more
advanced concepts, so you can complete the lab with confidence.

Objectives

After completing this reading, you will be able to create Bash scripts that:

 use conditional statements to run a set of commands only if a specified condition is true

 apply logical operators to create true/false comparisons

 perform basic arithmetic calculations

 create list-like arrays and access their elements

 implement for loops to execute operations repeatedly, based on a looping index

Conditionals

Conditionals, or if statements, are a way of telling a script to do something only under a specific
condition.
Bash script conditionals use the following if-then-else syntax:

1. 1

2. 2

3. 3

4. 4

5. 5

6. 6

1. if [ condition ]

2. then

3. statement_block_1

4. else

5. statement_block_2

6. fi

If the condition is true, then Bash executes the statements in statement_block_1 before exiting the
conditional block of code. After exiting, it will continue to run any commands after the closing fi.

Alternatively, if the condition is false, Bash instead runs the statements in statement_block_2 under
the else line, then exits the conditional block and continues to run commands after the closing fi.

Tips:

 You must always put spaces around your condition within the square brackets [ ].

 Every if condition block must be paired with a fi to tell Bash where the condition block ends.

 The else block is optional but recommended. If the condition evaluates to false without
an else block, then nothing happens within the if condition block. Consider options such as
echoing a comment in statement_block_2 to indicate that the condition was evaluated
as false.

In the following example, the condition is checking whether the number of command-line arguments
read by some Bash script, $#, is equal to 2.

1. 1

2. 2

3. 3

4. 4

5. 5

6. 6
 1. if [[ $# == 2 ]]

2. then

3. echo "number of arguments is equal to 2"

4. else

5. echo "number of arguments is not equal to 2"

6. fi

Notice the use of the double square brackets, which is the syntax required for making integer
comparisons in the condition [[ $# == 2 ]].

You can also make string comparisons. For example, assume you have a variable
called string_var that has the value "Yes" assigned to it. Then the following statement evaluates
to true:

1. 1

1. `[ $string_var == "Yes" ]`

Notice you only need single square brackets when making string comparisons.

You can also include multiple conditions to be satified by using the "and" operator && or the "or"
operator ||. For example:

1. 1

2. 2

3. 3

4. 4

5. 5

6. 6

1. if [ condition1 ] && [ condition2 ]

2. then

3. echo "conditions 1 and 2 are both true"

4. else

5. echo "one or both conditions are false"

6. fi

1. 1
 2. 2

3. 3

4. 4

5. 5

6. 6

1. if [ condition1 ] || [ condition2 ]

2. then

3. echo "conditions 1 or 2 are true"

4. else

5. echo "both conditions are false"

6. fi

Logical operators

The following logical operators can be used to compare integers within a condition in an if condition
block.

==: is equal to

If a variable a has a value of 2, the following condition evaluates to true; otherwise it evalutes
to false.

1. 1

1. $a == 2

!=: is not equal to

If a variable a has a value different from 2, the following statement evaluates to true. If its value is 2,
then it evalutes to false.

1. 1

1. a != 2

Tip: The ! logical negation operator changes true to false and false to true.

<=: is less than or equal to

If a variable a has a value of 2, then the following statement evaluates to true:

1. 1

1. a <= 3
and the following statement evalutates to false:

1. 1

1. a <= 1

Alternatively, you can use the equivalent notation -le in place of <=:

1. 1

2. 2

3. 3

4. 4

5. 5

6. 6

7. 7

8. 8

1. a=1

2. b=2

3. if [ $a -le $b ]

4. then

5. echo "a is less than or equal to b"

6. else

7. echo "a is not less than or equal to b"

8. fi

We've only provided a small sampling of logical operators here. You can explore resources such as
the Advanced Bash-Scripting Guide to find out more.

Arithmetic calculations

You can perform integer addition, subtraction, multiplication, and division using the notation $(()).
For example, the following two sets of commands both display the result of adding 3 and 2.

1. 1

1. echo $((3+2))

or
 1. 1

2. 2

3. 3

4. 4

1. a=3

2. b=2

3. c=$(($a+$b))

4. echo $c

Bash natively handles integer arithmetic but does not handle floating-point arithmetic. As a result, it
will always truncate the decimal portion of a calculation result.

For example:

1. 1

1. echo $((3/2))

prints the truncated integer result, 1, not the floating-point number, 1.5.

The following table summarizes the basic arithmetic operators:

Symbol Operation

+ addition

- subtraction

* multiplication

/ division

Table: Arithmetic operators

Arrays

The array is a Bash built-in data structure. An array is a space-delimited list contained in
parentheses.ʊTo create an array, declare its name and contents:

1. 1

1. my_array=(1 2 "three" "four" 5)

This statement creates and populates the array my_array with the items in the
parentheses: 1, 2, "three", "four", and 5.

You can also create an empty array by using:

1. 1

1. declare -a empty_array

If you want to add items to your array after creating it, you can add to your array by appending one
element at a time:

1. 1

2. 2

1. my_array+=("six")

2. my_array+=(7)

This adds elements "six" and 7 to the array my_array.

By using indexing, you can access individual or multiple elements of an array:

1. 1

2. 2

3. 3

4. 4

5. 5

6. 6

7. 7

8. 8

1. # print the first item of the array:

2. echo ${my_array[0]}

3.

4. # print the third item of the array:

5. echo ${my_array[2]}

6.

7. # print all array elements:

8. echo ${my_array[@]}
Tip: Note that array indexing starts from 0, not from 1.

for loops

You can use a construct called a for loop along with indexing to iterate over all elements of an array.

For example, the following for loops will continue to run over and over again until every element is
printed:

1. 1

2. 2

3. 3

1. for item in ${my_array[@]}; do

2. echo $item

3. done

or

1. 1

2. 2

3. 3

1. for i in ${!my_array[@]}; do

2. echo ${my_array[$i]}

3. done

The for loop requires a ; do component in order to cycle through the loop. Additionally, you need to
terminate the for loop block with a done statement.

Another way to implement a for loop when you know how many iterations you want is as follows.
For example, the following code prints the number 0 through 6.

1. 1

2. 2

3. 3

4. 4

1. N=6

2. for (( i=0; i<=$N; i++ )) ; do

3. echo $i

4. done
You can use for loops to accomplish all sorts of things. For example, you could count the number of
items in an array or sum up its elements, as the following Bash script does:

1. 1

2. 2

3. 3

4. 4

5. 5

6. 6

7. 7

8. 8

9. 9

10. 10

11. 11

12. 12

13. 13

14. 14

15. 15

1. #!/usr/bin/env bash

2. # initialize array, count, and sum

3. my_array=(1 2 3)

4. count=0

5. sum=0

6. for i in ${!my_array[@]}; do

7. # print the ith array element

8. echo ${my_array[$i]}

9. # increment the count by one

10. count=$(($count+1))

11. # add the current value of the array to the sum

12. sum=$(($sum+${my_array[$i]}))

13. done
 14. echo $count

15. echo $sum

Go ahead and try running this script, so you get a sense of how this loop works.

Summary

In this lab, you learned that:

 Conditional statements can be used to run commands based on whether a specified

condition is true

 Logical operators do true/false comparisons

 Arithmetic operators perform basic arithmetic calculations

 You can create list-like arrays and access their individual elements

 for loops execute operations repeatedly, based on a looping index

1Hands-on Lab: Scheduling Jobs using crontab2About Skills Network Cloud IDE3Exercise 1 -
Understand crontab file syntax4Exercise 2 - List cron jobs5Exercise 3 - Add a job in the crontab
file6Exercise 4 - Remove the current crontab7Practice exercises8Summary
Exercise 1 - Understand crontab file syntax

Cron is a system daemon used to execute desired tasks in the background at designated times.

A crontab file is a simple text file containing a list of commands meant to be run at specified times. It
is edited using the crontab command.

Each line in a crontab file has five time-and-date fields, followed by a command, followed by a
newline character (\n). The fields are separated by spaces.

The five time-and-date fields cannot contain spaces and their allowed values are as follows:

Field Allowed values

minute 0-59

hour 0-23, 0 = midnight

day 1-31

month 1-12

weekday 0-6, 0 = Sunday


 

1Hands-on Lab: Scheduling Jobs using crontab2About Skills Network Cloud IDE3Exercise 1 -
Understand crontab file syntax4Exercise 2 - List cron jobs5Exercise 3 - Add a job in the crontab
file6Exercise 4 - Remove the current crontab7Practice exercises8Summary

Exercise 2 - List cron jobs

Open a new terminal, by clicking on the menu bar and selecting Terminal->New Terminal, as in the
image below.

This will open a new terminal at the bottom of the screen as in the image below.
Run the commands below on the newly opened terminal.

The -l option of the crontab command prints the current crontab.

1. 1

1. crontab -l

You may get a message no crontab for theia if your crontab is empty.


 

1Hands-on Lab: Scheduling Jobs using crontab2About Skills Network Cloud IDE3Exercise 1 -
Understand crontab file syntax4Exercise 2 - List cron jobs5Exercise 3 - Add a job in the crontab
file6Exercise 4 - Remove the current crontab7Practice exercises8Summary

Exercise 3 - Add a job in the crontab file

3.1. Add a job to crontab

To add a cron job, run the command below:

1. 1

1. crontab -e

This will create a new crontab file for you (if you don't have one already). Now you are ready to add a
new cron job.

Your crontab file will be opened in an editor as shown in the image below:

Scroll down to the end of the file using the arrow keys:
Add the below line at the end of the crontab file:

1. 1

1. 0 21 * * * echo "Welcome to cron" >> /tmp/echo.txt

The above job specifies that the echo command should run when the minute is 0 and the hour is 21.
It effectively means the job runs at 9.00 p.m every day.

The output of the command should be sent to a file /tmp/echo.txt.

Press Ctrl + x to save the changes.

Press y to confirm.
Press Enter to come out of the editor.

Check if the job is added to the crontab by running the following command.

1. 1

1. crontab -l

You should see the newly added job in the output.

3.2. Schedule a shell script

Let us create a simple shell script that prints the current time and the current disk usage statistics.

Step 1: On the menu on the lab screen, use File->New File to create a new file:

Step 2: Give the file name as diskusage.sh and click 'OK'

Step 3: Save the following commands into the shell script:

1. 1

2. 2

3. 3

4. 4

5. 5

1. #! /bin/bash

2. # print the current date time

3. date

4. # print the disk free statistics

5. df -h

Step 4: Save the file using the File->Save menu option.

Step 5: Verify that the script is working:

1. 1

2. 2

1. chmod u+x diskusage.sh

2. ./diskusage.sh

The script should print the current timestamp and the disk usage stats.

Let us schedule this script to be run everyday at midnight 12:00 (when the hour is 0 on the 24 hour
clock).
We want the output of this script to be appended to /home/project/diskusage.log.

Edit the crontab:

1. 1

1. crontab -e

Add the following line to the end of the file:

1. 1

1. 0 0 * * * /home/project/diskusage.sh >>/home/project/diskusage.log

Press Ctrl + x to save the changes.

Press y to confirm.

Press Enter to come out of the editor.

Check if the job is added to the crontab by running the following command:

1. 1

1. crontab -l

You should see the newly added job in the output.

1Hands-on Lab: Scheduling Jobs using crontab2About Skills Network Cloud IDE3Exercise 1 -
Understand crontab file syntax4Exercise 2 - List cron jobs5Exercise 3 - Add a job in the crontab
file6Exercise 4 - Remove the current crontab7Practice exercises8Summary

Exercise 4 - Remove the current crontab

The -r option causes the current crontab to be removed.

Caution: This removes all your cron jobs. Be extra cautious when you use this command on a
production server.

1. 1
 1. crontab -r

Verify if your crontab is removed:

1. 1

1. crontab -l

1Hands-on Lab: Scheduling Jobs using crontab2About Skills Network Cloud IDE3Exercise 1 -
Understand crontab file syntax4Exercise 2 - List cron jobs5Exercise 3 - Add a job in the crontab
file6Exercise 4 - Remove the current crontab7Practice exercises8Summary

Practice exercises

1. Create a cron job that runs the task date >> /tmp/everymin.txt every minute.

Solution:

Edit the crontab file:

1. 1
 1. crontab -e

Add the following line at the end of the file:

1. 1

1. * * * * * date >> /tmp/everymin.txt

Save the file and quit the editor.

1Hands-on Lab: Scheduling Jobs using crontab2About Skills Network Cloud IDE3Exercise 1 -
Understand crontab file syntax4Exercise 2 - List cron jobs5Exercise 3 - Add a job in the crontab
file6Exercise 4 - Remove the current crontab7Practice exercises8Summary

Summary

In this lab, you learned how to:

 List cron jobs using crontab -l

 Add cron jobs using crontab -e

 Remove your current crontab using crontab -r

Module 3 Summary and Highlights
Level 2 headings may be created by course
providers in the future.
Bookmark this page

 A shell script is a program that begins with a ‘shebang’ directive and

is used to run commands and programs. Scripting languages are
interpreted rather than compiled.
 Filters are shell commands. The pipe operator allows you to chain
filter commands.
 Shell variables can be assigned values with and listed
using Environment variables are shell variables with extended
scope, and you can list them with
 Metacharacters are special characters that have meaning to the
shell.
 Quoting specifies whether the shell should interpret special
characters as metacharacters or 'escape' them.
 Input/Output, or I/O redirection, refers to a set of features used for
redirecting.
 You can use command substitution to replace a command with its
output.
 Command line arguments provide a way to pass arguments to a shell
script.
 In concurrent mode, multiple commands can run simultaneously.
 You can schedule cron jobs to run periodically at selected times. m h
dom mon dow command is the cron job syntax.
 You can edit cron jobs by running crontab-e, and crontab-l lists all
cron jobs in the cron table.

Module 3 Cheat Sheet - Introduction to Shell Scripting

Bash shebang

1. 1

1. #!/bin/bash
Get the path to a command

1. 1

1. which bash

Pipes, filters, and chaining

Chain filter commands together using the pipe operator:

1. 1

1. ls | sort -r

Pipe the output of manual page for ls to head to display the first 20 lines:

1. 1

1. man ls | head -20

Use a pipeline to extract a column of names from a csv and drop duplicate names:

1. 1

1. cut -d "," -f1 names.csv | sort | uniq

Working with shell and environment variables:

List all shell variables:

1. 1

1. set

Define a shell variable called my_planet and assign value Earth to it:

1. 1

1. my_planet=Earth

Display value of a shell variable:

 1. 1

1. echo $my_planet

Reading user input into a shell variable at the command line:

1. 1

1. read first_name

Tip: Whatever text string you enter after running this command gets stored as the value of the
variable first_name.

List all environment variables:

1. 1

1. env

Environment vars: define/extend variable scope to child processes:

1. 1

2. 2

1. export my_planet

2. export my_galaxy='Milky Way'

Metacharacters

Comments #:

1. 1

1. # The shell will not respond to this message

Command separator ;:

1. 1

1. echo 'here are some files and folders'; ls

File name expansion wildcard *:

1. 1
 1. ls *.json

Single character wildcard ?:

1. 1

1. ls file_2021-06-??.json

Quoting

Single quotes '' - interpret literally:

1. 1

1. echo 'My home directory can be accessed by entering: echo $HOME'

Double quotes "" - interpret literally, but evaluate metacharacters:

1. 1

1. echo "My home directory is $HOME"

Backslash \ - escape metacharacter interpretation:

1. 1

1. echo "This dollar sign should render: \$"

I/O Redirection

Redirect output to file and overwrite any existing content:

1. 1

1. echo 'Write this text to file x' > x

Append output to file:

1. 1

1. echo 'Add this line to file x' >> x

Redirect standard error to file:

 1. 1

1. bad_command_1 2> error.log

Append standard error to file:

1. 1

1. bad_command_2 2>> error.log

Redirect file contents to standard input:

1. 1

1. $ tr “[a-z]” “[A-Z]” < a_text_file.txt

The input redirection above is equivalent to:

1. 1

1. $cat a_text_file.txt | tr “[a-z]” “[A-Z]”

Command Substitution

Capture output of a command and echo its value:

1. 1

2. 2

1. THE_PRESENT=$(date)

2. echo "There is no time like $THE_PRESENT"

Capture output of a command and echo its value:

1. 1

1. echo "There is no time like $(date)"

Command line arguments

1. 1

1. ./My_Bash_Script.sh arg1 arg2 arg3

Batch vs. concurrent modes

Run commands sequentially:

1. 1

1. start=$(date); ./MyBigScript.sh ; end=$(date)

Run commands in parallel:

1. 1

1. ./ETL_chunk_one_on_these_nodes.sh & ./ETL_chunk_two_on_those_nodes.sh

Scheduling jobs with cron

Open crontab editor:

1. 1

1. crontab -e

Job scheduling syntax:

1. 1

1. m h dom mon dow command

(minute, hour, day of month, month, day of week)

Tip: You can use the * wildcard to mean "any".

Append the date/time to a file every Sunday at 6:15 pm:

1. 1

1. 15 18 * * 0 date >> sundays.txt

Run a shell script on the first minute of the first day of each month:

1. 1

1. 1 0 1 * * ./My_Shell_Script.sh
Back up your home directory every Monday at 3:00 am:

1. 1

1. 0 3 * * 1 tar -cvf my_backup_path\my_archive.tar.gz $HOME\

Deploy your cron job:

Close the crontab editor and save the file.

List all cron jobs:

1. crontab -l

Conditionals

if-then-else syntax:

1. if [[ $# == 2 ]]

2. then

3. echo "number of arguments is equal to 2"

4. else

5. echo "number of arguments is not equal to 2"

6. fi

'and' operator &&:

1. if [ condition1 ] && [ condition2 ]

'or' operator ||:

1. if [ condition1 ] || [ condition2 ]

Logical operators

Operator Definition

== is equal to

!= is not equal to

< is less than

> is greater than

<= is less than or equal to

 Operator Definition

>= is greater than or equal to

Arithmetic calculations

Integer arithmetic notation:

1. $(())

Basic arithmetic operators:

Symbol Operation

+ addition

- subtraction

* multiplication

/ division

Display the result of adding 3 and 2:

1. echo $((3+2))

Negate a number:

1. echo $((-1*-2))

Arrays

Declare an array that contains items 1, 2, "three", "four", and 5:

1. my_array=(1 2 "three" "four" 5)

Add an item to your array:

1. my_array+="six"

2. my_array+=7

Declare an array and load it with lines of text from a file:

1. my_array=($(echo $(cat column.txt)))

for loops

Use a for loop to iterate over values from 1 to 5:

1. for i in {0..5}; do
 2. echo "this is iteration number $i"

3. done

Use a for loop to print all items in an array:

1. for item in ${my_array[@]}; do

2. echo $item

3. done

Use array indexing within a for loop, assuming the array has seven elements:

1. for i in {0..6}; do

2. echo ${my_array[$i]}

3. done

Practice Project: Introduction

Scenario

You've been tasked by your team to create an automated Extract, Transform, Load (ETL) process to
extract daily weather forecast and observed weather data and load it into a live report to be used for
further analysis by the analytics team. As part of a larger prediction modelling project, the team
wants to use the report to monitor and measure the historical accuracy of temperature forecasts by
source and station.

As a proof-of-concept (POC), you are only required to do this for a single station and one source to
begin with. For each day at noon (local time), you will gather both the actual temperature and the
temperature forecasted for noon on the following day for Casablanca, Morocco.

At a later stage, the team anticipates extending the report to include lists of locations, different
forecasting sources, different update frequencies, and other weather metrics such as wind speed and
direction, precipitation, and visibility.

Data source

For this practice project, you'll use the weather data package provided by the open source
project wttr.in, a web service that provides weather forecast information in a simple and text-based
format. For further information, you can read more about the service on its GitHub Repo.

First, you'll use the curl command to scrape weather data via the wttr.in website. For example, to get
data for Casablanca, enter:

1. curl wttr.in/casablanca

which prints the following to stdout:

Learning Objectives

After completing this practice project, you will be able to apply your new shell scripting skills in a
real-world scenario to:

 Download raw weather data

 Extract data of interest from the raw data

 Transform the data as required

 Load the data into a log file using a tabular format

 Schedule the entire process to run automatically at a set time daily

Overview

Weather reporting tasks

You must extract and store the following data every day at noon, local time, for Casablanca,
Morocco:

 The actual temperature (in degrees Celsius)

 The forecasted temperature (in degrees Celsius) for the following day at noon

Here is an example of what the the resulting weather report should look like:
 year month day obs_tmp fc_temp

2023 1 1 10 11

2023 1 2 11 12

2023 1 3 12 10

2023 1 4 13 13

2023 1 5 10 9

2023 1 6 11 10

… … … … …

Table 1. Example of weather report

Feel free to try completing the entire project on your own, or follow the exercises below to guide you
through the process. Either way, check out the exercises, hints, and Solution:s.

Tip: At each step of the process, test your code to ensure it does what you intended. For more
complicated steps, break the task down into smaller, more managable steps that you can test
individually. You can test code on the command line or by running your script as you develop it. The
context will indicate the best approach.

Hands-on Lab: Historical Weather Forecast Comparison to Actuals

Exercise 1 - Initialize your weather report log file

1.1 Create a text file called rx_poc.log

rx_poc.log will be your POC weather report log file, or a text file which contains a growing history of
the daily weather data you will scrape. Each entry in the log file corresponds to a row as in Table 1.

Solution:

Create rx_poc.log by entering the following in your terminal:

1. touch rx_poc.log

1.2 Add a header to your weather report

Your header should consist of the column names from Table 1, delimited by tabs.
Write the header to your weather report.

Solution:

Use a shell variable and command substitution:

1. header=$(echo -e "year\tmonth\tday\tobs_temp\tfc_temp")
 2. echo $header>rx_poc.log

OR more directly, use echo and redirection:

1. 1

1. echo -e "year\tmonth\tday\thour\tobs_temp\tfc_temp">rx_poc.log

Tip: Although it might seem redundant, it is better practice to use variables in these cases. Variables
make for much cleaner code, which is easier to understand and safer to modify by others or even
yourself at a later date. Using meaningful names for your variables also makes the code more "self-
documenting."

Exercise 2 - Download the raw weather data

2.1. Create a text file called rx_poc.sh and make it an executable Bash script

Solution: 1

Create the file rx_poc.sh

1. touch rx_poc.sh

Include the Bash shebang on the first line of rx_poc.sh:

1. #! /bin/bash

Solution: 2

Make your script executable by running the following in the terminal:

1. 1

1. chmod u+x rx_poc.sh

Verify your changes using the ls command with the -l option.

2.2. Assign the city name to Casablanca for accessing the weather report

Solution:

1. city=Casablanca

2.3 Obtain the weather information for Casablanca

Solution:

Edit rx_poc.sh to include:

1. curl -s wttr.in/$city?T --output weather_report

Exercise 3 - Extract and load the required data

3.1. Edit rx_poc.sh to extract the required data from the raw data file and assign them to
variables obs_temp and fc_temp
Extracting the required data is a process that will take some trial and error until you get it right. Study
the weather report you obtained in Step 2.3, determine what you need to extract, and look for
patterns.

You are looking for ways to 'chip away' at the weather report by:

 Using shell commands to extract only the data you need (the signal)

 Filtering everything else out (the noise)

 Combining your filters into a pipeline (recall the use of pipes to chain filters together)

Click here for a Hint to get started

Extract only those lines that contain temperatures from the weather report, and save the result to
variables representing the temparature output.

3.1.1. Extract the current temperature, and store it in a shell variable called obs_temp

Remember to validate your results.

You may have noticed by now that the temperature values extracted from wttr.in are surrounded by
special formatting characters. These "hidden" characters cause the numbers to display in specific
color - for example, when you use the cat command to display your log file.

Unfortunately you cannot perform arithmetic calculations on such formatted text, so you will need to
extract the values from the surrounding formatting so you can make use of them later in this lab.

Solution:

While adding the following lines to rx_poc.sh, ensure you understand what each filter accomplishes
by using the command line. Try adding one filter at a time to see what the outcome is as you develop
the pipeline.

1. #To extract Current Temperature

2. obs_temp=$(curl -s wttr.in/$city?T | grep -m 1 '°.' | grep -Eo -e '-?[[:digit:]].*')

3. echo "The current Temperature of $city: $obs_temp"

 The first line uses the curl command to fetch weather information from wttr.in for the
specified city ($city). It then uses a combination of grep and grep -Eo to extract the current
temperature in degrees Celsius and assigns it to the variable obs_temp.

 The second line (echo $obs_temp) prints the current temperature to the console.

3.1.2. Extract tomorrow's temperature forecast for noon, and store it in a shell variable
called fc_temp

Solution:

Add to rx_poc.sh:

1. # To extract the forecast tempearature for noon tomorrow

 2. fc_temp=$(curl -s wttr.in/$city?T | head -23 | tail -1 | grep '°.' | cut -d 'C' -f2 | grep -Eo -e '-?
[[:digit:]].*')

3. echo "The forecasted temperature for noon tomorrow for $city : $fc_temp C"

 The first line uses a combination of head and tail to narrow down to the line containing
tomorrow’s noon forecast temperature. grep and cut are employed to isolate and format the
temperature information, and, the numeric part of the temperature is captured using grep -
Eo.

 The second line (echo $fc_temp) prints the forecast temperature for noon tomorrow to the
console.

3.2. Store the current hour, day, month, and year in corresponding shell variables

Solution:

Add to rx_poc.sh:

1. #Assign Country and City to variable TZ

2. TZ='Morocco/Casablanca'

3.

4. # Use command substitution to store the current day, month, and year in corresponding shell
variables:

5. hour=$(TZ='Morocco/Casablanca' date -u +%H)

6. day=$(TZ='Morocco/Casablanca' date -u +%d)

7. month=$(TZ='Morocco/Casablanca' date +%m)

8. year=$(TZ='Morocco/Casablanca' date +%Y)

Note: You might be wondering why we didn't just set hour to a value of 12, since we want to get the
time at noon.
However, if we did, we would lose the ability to verify that we are actually running the code at the
correct local time.You should think of the local time as a measurement rather than a set number.

3.3. Merge the fields into a tab-delimited record, corresponding to a single row in Table 1

Append the resulting record as a row of data to your weather log file.

Solution:

Add to rx_poc.sh:

1. record=$(echo -e "$year\t$month\t$day\t$obs_temp\t$fc_temp C")

2. echo $record>>rx_poc.log

Exercise 4 - Schedule your Bash script rx_poc.sh to run every day at noon local time
4.1. Determine what time of day to run your script

Recall that you want to load the weather data coresponding to noon, local time, in Casablanca every
day.

First, check the time difference between your system's default time zone and UTC.

Solution:

Running the following commands gives you the info you need to get the time difference between
your system and UTC.

1. $ date

2. Mon Feb 13 11:28:12 EST 2023

3. $ date -u

4. Mon Feb 13 16:28:16 UTC 2023

Now you can determine how many hours difference there are between your system's local time and
that of Casablanca. In the example above, we see that the system time relative to UTC is UTC+5 (i.e.
16 - 11 = 5).

We know Casablanca is UTC+1, so the system time relative to Casablanca is 4 hours earlier. Thus to
run your script at noon Casablanca time, you need to run it at 8 am.

4.2 Create a cron job that runs your script

Solution:

Edit the crontab file:

1. crontab -e

Add the following line at the end of the file:

1. 0 8 * * * /home/project/rx_poc.sh

Save the file and quit the editor.

4.3 Full Solution:

For reference, here is a Bash script that creates the raw weather report. Try to follow all the steps on
your own before looking!

Click here for Full Solution:

1. #! /bin/bash

2.

3. #Assign city name as Casablanca

4. city=Casablanca

5.
 6. #Obtain the weather report for Casablanca

7. curl -s wttr.in/$city?T --output weather_report

8.

9. #To extract Current Temperature

10. obs_temp=$(curl -s wttr.in/$city?T | grep -m 1 '°.' | grep -Eo -e '-?[[:digit:]].*')

11. echo "The current Temperature of $city: $obs_temp"

12.

13. # To extract the forecast tempearature for noon tomorrow

14. fc_temp=$(curl -s wttr.in/$city?T | head -23 | tail -1 | grep '°.' | cut -d 'C' -f2 | grep -Eo -e '-?
[[:digit:]].*')

15. echo "The forecasted temperature for noon tomorrow for $city : $fc_temp C"

16.

17. #Assign Country and City to variable TZ

18. TZ='Morocco/Casablanca'

19.

20.

21. # Use command substitution to store the current day, month, and year in corresponding shell
variables:

22. hour=$(TZ='Morocco/Casablanca' date -u +%H)

23. day=$(TZ='Morocco/Casablanca' date -u +%d)

24. month=$(TZ='Morocco/Casablanca' date +%m)

25. year=$(TZ='Morocco/Casablanca' date +%Y)

26.

27.

28. # Log the weather

29. record=$(echo -e "$year\t$month\t$day\t$obs_temp\t$fc_temp C")

30. echo $record>>rx_poc.log

Exercise 5 - Create a script to report historical forecasting accuracy

Now that you've created an ETL shell script to gather weather data into a report, let's create another
script to measure and report the accuracy of the forecasted temperatures against the actuals.

 To start, create a tab-delimited file named historical_fc_accuracy.tsv.

Insert the following code into the file to include a header with column names:
 1. echo -e "year\tmonth\tday\tobs_temp\tfc_temp\taccuracy\taccuracy_range" >
historical_fc_accuracy.tsv

One key difference between this report and the previous report you generated is that the forecast
temperature will now be aligned with the date the forecast is for. As a result, the date will be in the
same row as the observed temperature for that date, rather than the previous row on the day that
the forecast was made.

 Also create an executable Bash script called fc_accuracy.sh.

Rather than scheduling your new script to run periodically, think of it as a tool you can use to
generate the historical forecast accuracy on demand.

5.1. Determine the difference between today's forecasted and actual temperatures

Rather than writing the script to process all of the data at once, let's simplify by solving the problem
for just one instance. Later you can modify the script to handle the general case of mupltiple days.

5.1.1. Extract the forecasted and observed temperatures for today and store them in variables

Solution:

1. yesterday_fc=$(tail -2 rx_poc.log | head -1 | cut -d " " -f5)

5.1.2. Calculate the forecast accuracy

Solution:

1. today_temp=$(tail -1 rx_poc.log | cut -d " " -f4)

2. accuracy=$(($yesterday_fc-$today_temp))

3. echo "accuracy is $accuracy"

Tip: Your weather report must have at least two days of data for this calulation to make sense.
To test your code you can simply append some artificial data to your weather report, rx_poc.log.

5.2. Assign a label to each forecast based on its accuracy

Let's set the accuracy labels according to the range that the accuracy fits most tightly within,
according to the following table. Validate your result.

accuracy range accuracy label

+/- 1 deg excellent

+/- 2 deg good

+/- 3 deg fair

+/- 4 deg poor

 accuracy range accuracy label

Solution:

1. if [ -1 -le $accuracy ] && [ $accuracy -le 1 ]

2. then

3. accuracy_range=excellent

4. elif [ -2 -le $accuracy ] && [ $accuracy -le 2 ]

5. then

6. accuracy_range=good

7. elif [ -3 -le $accuracy ] && [ $accuracy -le 3 ]

8. then

9. accuracy_range=fair

10. else

11. accuracy_range=poor

12. fi

13.

14. echo "Forecast accuracy is $accuracy"

5.3. Append a record to your historical forecast accuracy file.

Solution:

1. row=$(tail -1 rx_poc.log)

2. year=$( echo $row | cut -d " " -f1)

3. month=$( echo $row | cut -d " " -f2)

4. day=$( echo $row | cut -d " " -f3)

5. echo -e "$year\t$month\t$day\t$today_temp\t$yesterday_fc\t$accuracy\t$accuracy_range"
>> historical_fc_accuracy.tsv

5.4. Full Solution: for handling a single day

Below is the final script of fc_accuracy.sh for handling the accuracy calculations based on just one
instance, or day.

Solution:

1. #! /bin/bash

2.

3. yesterday_fc=$(tail -2 rx_poc.log | head -1 | cut -d " " -f5)

4. today_temp=$(tail -1 rx_poc.log | cut -d " " -f4)

5. accuracy=$(($yesterday_fc-$today_temp))

6.

7. echo "accuracy is $accuracy"

8.

9. if [ -1 -le $accuracy ] && [ $accuracy -le 1 ]

10. then

11. accuracy_range=excellent

12. elif [ -2 -le $accuracy ] && [ $accuracy -le 2 ]

13. then

14. accuracy_range=good

15. elif [ -3 -le $accuracy ] && [ $accuracy -le 3 ]

16. then

17. accuracy_range=fair

18. else

19. accuracy_range=poor

20. fi

21.

22. echo "Forecast accuracy is $accuracy_range"

23.

24. row=$(tail -1 rx_poc.log)

25. year=$( echo $row | cut -d " " -f1)

26. month=$( echo $row | cut -d " " -f2)

27. day=$( echo $row | cut -d " " -f3)

28. echo -e "$year\t$month\t$day\t$today_temp\t$yesterday_fc\t$accuracy\t$accuracy_range"

>> historical_fc_accuracy.tsv
5.5. Generalization to all days

We leave it as an exercise for you to generalize your code to create the entire weather accuracy
history. In the next exercise, you will download and work with a synthetic version of this weather
accuracy history report.

Here are some suggestions to guide you should you wish to create the weather accuracy history
yourself:

 Iterate through your weather log file using a for loop. On each iteration:

o Use head and tail to extract consecutive pairs of lines on each iteration

 This provides you with the current and previous day's data

o Treat this pair of lines like you did in your code as yesterday and today's data

o Perform your accuracy calulations as before

o Use the correct row to extract date information

o Append your resulting data to your historical forecast accuracy report

Exercise 6 - Create a script to report weekly statistics of historical forecasting accuracy

In this exercise, you will download a synthetic historical forecasting accuracy report and calculate
some basic statistics based on the latest week of data.

Begin by creating an executable bash script called weekly_stats.sh.

6.1. Download the synthetic historical forecasting accuracy dataset

Run the following command in the terminal to download the dataset to your current working
directory.

1. 1

1. wget https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMSkillsNetwork-
LX0117EN-Coursera/labs/synthetic_historical_fc_accuracy.tsv

6.2. Load the historical accuracies into an array covering the last week of data

Remember to make your script executable. Also validate your result by printing the array to the
terminal.

Solution:Your shell script should look something like the following. There are many ways to
accomplish this task, so the Solution: provided is not unique.

1. #!/bin/bash

2.
 3. echo $(tail -7 synthetic_historical_fc_accuracy.tsv | cut -f6) > scratch.txt

4.

5. week_fc=($(echo $(cat scratch.txt)))

6.

7. # validate result:

8. for i in {0..6}; do

9. echo ${week_fc[$i]}

10. done

6.3. Display the minimum and maximum absolute forecasting errors for the week

Now use your array to calculate the minimum and maximum absolute errors over the last week. For
example, if you have a vaule of -1, change it to be 1. Echo the minimum and maximum absolute
errors to the terminal.

Solution:

Your final shell script for weekly_stats.sh should now look something like the following.

1. #!/bin/bash

2.

3. echo $(tail -7 synthetic_historical_fc_accuracy.tsv | cut -f6) > scratch.txt

4.

5. week_fc=($(echo $(cat scratch.txt)))

6.

7. # validate result:

8. for i in {0..6}; do

9. echo ${week_fc[$i]}

10. done

11.

12. for i in {0..6}; do

13. if [[ ${week_fc[$i]} < 0 ]]

14. then

15. week_fc[$i]=$(((-1)*week_fc[$i]))

16. fi

17. # validate result:

 18. echo ${week_fc[$i]}

19. done

20.

21. minimum=${week_fc[1]}

22. maximum=${week_fc[1]}

23. for item in ${week_fc[@]}; do

24. if [[ $minimum > $item ]]

25. then

26. minimum=$item

27. fi

28. if [[ $maximum < $item ]]

29. then

30. maximum=$item

31. fi

32. done

33.

34. echo "minimum ebsolute error = $minimum"

35. echo "maximum absolute error = $maximum"

Peer-Graded Final Project: Introduction

Congratulations! You have finished all the modules and completed a real-world practice project.

You will soon get to put your skills to the test by completing a peer-graded, hands-on lab as the main
component of your final project.

To help you complete this hands-on lab, we have provided:

 A cumulative cheat sheet that covers the entire course, including any advanced concepts that
you will encounter in the final project

 Sufficient hints and/or the required code snippets for you within each task as needed

Scenario

Imagine that you are a lead Linux developer at the top-tech company ABC International Inc. ABC
currently suffers from a huge bottleneck: each day, interns must painstakingly access encrypted
password files on core servers and back up any files that were updated within the last 24 hours. This
process introduces human error, lowers security, and takes an unreasonable amount of work.
As one of ABC Inc.'s most trusted Linux developers, you have been tasked with creating a script
called backup.sh which runs every day and automatically backs up any encrypted password files that
have been updated in the past 24 hours.

Learning Objectives

By completing this final project, you will:

 Demonstrate your advanced shell scripting skills in a real-world scenario

 Apply the knowledge you've gained to reviewing and grading technical work submtted by
your peers

Overview

Hands-on lab: Scheduled Backup Script

Instructions

Immediately following this reading, you will complete the hands-on lab portion of your final project
where you will create a scheduled backup script.

Your work will be graded by your peers who are also completing this assignment within the same
session. Likewise, as part of your final project requirements, you will also review the work done by
your peers.

Deliverables

You will need to submit the following items for peer review:

 Screenshots clearly displaying both the code and the output for each task

 Your completed script file

Full details are provided for each task within the hands-on lab.

Grading criteria

There are a total of 20 points to be earned for 17 tasks in this final project.
Your grade will be based on the following tasks within the hands-on lab:

 [Tasks 1-13]: Upload screenshot of sections from the backup.sh script displaying the correct
code (13 pts: 1 pt for each of the 13 tasks)

 [Task 14]: Submit your completed backup.sh file (1 pt)

 [Task 15]: Upload screenshot showing executable permissions (2 pts)

 [Task 16]: Upload screenshot showing file named backup-[TIMESTAMP].tar.gz (2 pts)

 [Task 17]: Upload screenshot showing crontab schedule of once every day (2 pts)

How to submit

IMPORTANT: You will be prompted to save screenshots throughout the lab and to save your
completed script at the end. These will be the files that you will upload during the Project
Submission and Peer Review steps.
Please note that you must submit your screenshots in either JPEG or PNG format.

Discussion forum tab

You can chat with other learners taking this course using the Discussion tab near the top of this page.

My Submission tab

Once you are ready to submit your work for peer review, simply click on the My Submission tab and
follow the instructions provided there. Good luck!

Summary

Congratulations! You've just completed a challenging, real-world practice project using many of the
concepts you've learned from this course. The knowledge you've gained has prepared you to solve
many practical real world problems. You're almost finished with this course now, and the final step in
your journey is to complete the peer-reviewed Final Project.

In this lab, you learned how to:

 Initialize your weather report log file

 Write a Bash script that downloads the raw weather data, and extracts and loads the
required data

 Schedule your Bash script rx_poc.sh to run every day at noon local time

 Apply advanced Bash scripting to produce reporting metrics

 Create a script to report historical forecasting accuracy

 Create a script to report the minimum and maximum absolute errors for the week

Reading: Cumulative Cheatsheet: Linux Commands and Shell Scripting

Cumulative Cheat Sheet - Hands-On Introduction to Linux Commands and Shell Scripting

Introduction to Linux

Comman Descriptio
Syntax Example
d n

List files
ls [OPTIONS] ls
and
List [FILE/DIRECTOR /home/user/docume
directorie
Y] nts
s at path

Print
Print
present
Working pwd pwd
working
Directory
directory
 Comman Descriptio
Syntax Example
d n

Change cd
Change
cd [DIRECTORY] current /home/user/docume
Directory
directory nts

Run
command
Super sudo
with sudo apt update
user do [COMMAND]
superuser
privileges

Open file
Text with Nano
nano [FILE] nano myfile.txt
Editor text
editor

Introduction to Linux Commands

Informational, Navigational, & Management Commands

Comman Descripti
Syntax Example
d on

Return
Who Am
whoami usernam whoami
I
e

Return
current
User ID id id
user or
group ID

Display
System
uname system
Informati uname -a
[OPTIONS] informati
on
on

Display
manual
Manual man page for
man ls
Pages [COMMAND] a
comman
d
Comman Descripti
Syntax Example
d on

Transfer
curl
curl [OPTIONS] data from
Curl https://some_website.
[URL] or to
com
server

Display
date current
Date date
[OPTIONS] date and
time

Find files
and
find
directorie find /home/user -
Find [DIRECTORY]
s at name '*.txt'
[OPTIONS]
specified
path

Create
Make mkdir
new mkdir myfolder
Directory [DIRECTORY]
directory

Remove
Remove rmdir
empty rmdir myfolder
Directory [DIRECTORY]
directory

Display
process
Process
ps [OPTIONS] status ps -ef
Status
informati
on

Display
live
Table of
top system top
Processes
resource
usage

Display
Disk df [OPTIONS] disk
df -h
Usage [FILESYSTEM] space
usage
 Comman Descripti
Syntax Example
d on

Create
Create new file
Empty touch [FILE] or update touch myfile.txt
File timestam
p

Copy files
or
cp [OPTIONS]
directorie cp myfile.txt
[SOURCE]
Copy s from /home/user/documen
[DESTINATION
source to ts
]
destinati
on

Move or
mv [OPTIONS]
rename mv myfile.txt
[SOURCE]
Move files and /home/user/documen
[DESTINATION
directorie ts
]
s

rm [OPTIONS]
Remove
Remove [FILE/DIRECTO rm my_scratch_file.txt
files
RY]

Remove
rm -r
nonempt
path_to_temp_directo
y
ry
directory

rmdir Remove
rmdir
[OPTIONS] empty
path_to_my_directory
[DIRECTORY] directory

Change
chmod file or
Change
[OPTIONS] directory chmod u+x myfile.txt
Mode
[MODE] [FILE] permissio
ns

Working with Text Files, Networking & Archiving Commands

Comma Descriptio
Syntax Example
nd n

Display the
Concat
cat [FILE] contents of cat myfile.txt
enate
a file

Concatenta
te and
display
cat file1 file2
contents of
multiple
files

Display file
more
More one screen more myfile.txt
[FILE]
at a time

head Display
Head [OPTION first N lines head -5 myfile.txt
S] [FILE] of file

tail Display last

Tail [OPTION N lines of tail -5 myfile.txt
S] [FILE] file

echo Display
Echo [ARGUM arguments echo Hello, World!
ENTS] in console

Alphanum
sort
erically
Sort [OPTION sort file.txt
sort file
S] [FILE]
contents

Report or
remove
uniq
consecutiv
Unique [OPTION uniq file.txt
ely
S] [FILE]
repeated
lines in file

Word wc Print the wc file.txt

Count [OPTION number of
S] [FILE] lines,
Comma Descriptio
Syntax Example
nd n

words, and
characters
in a file

grep
Search for
[OPTION
a specified
Grep S] grep "hello" file.txt
pattern in a
PATTERN
file
[FILE]

paste Merge
[OPTION lines of
Paste paste file1.txt file2.txt
S] [FILE1] files side
[FILE2] by side

Remove
cut sections
Cut [OPTION from each cut -d":" -f1 /etc/passwd
S] [FILE] line of a
file

Archive
tar files
tar -czvf archive.tar.gz
Tar [OPTION together
/directory
S] [FILE] into a
single file

zip Compress
zip archive.zip file1.txt
Zip [OPTION files into a
file2.txt
S] [FILE] zip archive

Uncompres
unzip
s files from
Unzip [OPTION unzip archive.zip
a zip
S] [FILE]
archive

Print the
name of
Hostna hostnam
the current hostname
me e
host
system
 Comma Descriptio
Syntax Example
nd n

Send ICMP
ping
ECHO_REQ
[OPTION
UEST
Ping S] ping google.com
packets to
HOSTNA
a network
ME/IP
host

Display or
ifconfig configure
Ifconfig [INTERFA network ifconfig
CE] interface
parameters

Show or
manipulate
routing,
ip
devices,
IP [OPTION ip addr
policy
S]
routing,
and
tunnels

Transfer
curl
data from curl
Curl [OPTION
or to a https://some_website.com
S] URL
server

wget Download wget

Wget [OPTION files from https://some_website.com/
S] URL the web some_file.txt

Introduction to Shell Scripting

Com Descriptio
Syntax Example
mand n

Sheb First line of

#!/bin/[shell] #!/bin/bash
ang shell script

Chain any
Pipe filter1 | filter2 number of ls | sort -r
filters
Com Descriptio
Syntax Example
mand n

Locat Display
e which location
which bash
execu [EXECUTABLE] of bash exe
table cutable

Interpret
and run
Bash bash [SCRIPT] script bash script.txt
using Bash
shell

List all
Set set [OPTION] shell set
variables

Define
Defin shell
e [VARIABLE_NAM variable by
name="John"
varia E]=[VALUE] name and
ble assign
value

Read from
standard
Read read [VARIABLE] input and read name
store result
in variable

Print all
environme
nt
Env env env
variables
and their
values

Extend
scope of
Expor export local
export name
t [VARIABLE] variable to
all child
processes
 Com Descriptio
Syntax Example
mand n

Open
Cront crontab crontab
crontab -e
ab [OPTIONS] default
editor

List all cron

crontab -l
jobs

Sched
ule
tasks
to Append
run at date/time
speci m h dom mon to file 15 18 * * 0 date >>
fied dow command every sundays.txt
times Sunday at
using 6:15 pm
cron
daem
on

Back up
home 0 3 * * 1 tar -cvf
directory my_backup_path\
every my_archive.tar.gz
Monday at $HOME\
3:00 am

Run shell
script first
minute of 101**
first day of ./My_Script.sh
each
month

Hands-on Lab: Peer-Graded Final Project

1Linux Commands and Shell Scripting - Final Project2

Linux Commands and Shell Scripting - Final Project

Estimated time needed: 90 minutes

Welcome to the hands-on lab for the final project!

In this scenario, you are a lead Linux developer at the top-tech company ABC International Inc. As
one of ABC Inc.'s most trusted Linux developers, you have been tasked with creating a script
called backup.sh which runs every day and automatically backs up any encrypted password files
that have been updated in the past 24 hours.

Please complete the following tasks, and be sure to follow the directions as you go. Don't forget to
save your work.

1Linux Commands and Shell Scripting - Final Project2Getting started34Checkpoint56Note on

arrays78Checkpoint91011Congratulations!

Getting started

Task 0

1. Open a new terminal by clicking on the menu bar and selecting Terminal->New Terminal:

2. Download the template file backup.sh by running the command below:

1. 1

1. wget https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBM-LX0117EN-
SkillsNetwork/labs/Final%20Project/backup.sh

Copied!

3. Open the file in the IDE by clicking File->Open as seen below:

then click on the file, which should have been downloaded to your project directory:
About the template script backup.sh

1. You will notice the template script contains comments (lines starting with the # symbol).
Do not delete these.

The ones that look like # [TASK {number}] will be used by your grader:
 2. Also, please do not modify any existing code above # [TASK 1] in the script.

Saving your progress

Your work will not be saved if you exit your session.

In order to save your progress:

1. Save the current working file (backup.sh) with CTRL+s [Windows/Linux], CMD+s [MAC], or
navigate to File->Save as seen below:
2. Download the file to your local computer by navigating to File->Download as seen below:
3. Unfortunately, our editor does not currently support file uploading, so you will need to
copy and paste your work as follows:

o To "upload" your in-progress backup.sh file and continue working on it:

a. Open a terminal and type touch backup.sh

b. Open the empty backup.sh file in the editor

c. Copy-paste the contents of your locally-saved backup.sh file into the

empty backup.sh file in the editor


 

1Linux Commands and Shell Scripting - Final Project2Getting started34Checkpoint56Note on

arrays78Checkpoint91011Congratulations!

Task 1

Navigate to # [TASK 1] in the code.

Set two variables equal to the values of the first and second command line arguments, as follows:

1. Set targetDirectory to the first command line argument

2. Set destinationDirectory to the second command line argument

This task is meant to help with code readability.

Click here for Hint

The command line arguments interpreted by the script can be accessed via $1 (first argument)
and $2 (second argument).

Take a screenshot of the code above and save it as 01-Set_Variables.jpg or .png.

Task 2
 1. Display the values of the two command line arguments in the terminal.

Click here for Hint

Remember, you can use the command echo as a print command.

 Example: echo "The year is $year"

2. Take a screenshot of the code above and save it as 02-Display_Values.jpg or .png.

Task 3

1. Define a variable called currentTS as the current timestamp, expressed in seconds.

Click here for Hint

Remember you can customize the output format of the date command.

To set a variable equal to the output of a command you can use command substitution: $() or ` `

 For example: currentYear=$(date +%Y)

2. Take a screenshot of the code above and save it as 03-CurrentTS.jpg or .png.

Task 4

1. Define a variable called backupFileName to store the name of the archived and
compressed backup file that the script will create.

The variable backupFileName should have the value "backup-[$currentTS].tar.gz"

 For example, if currentTS has the value 1634571345, then backupFileName should have
the value backup-1634571345.tar.gz.

2. Take a screenshot of the code above and save it as 04-Set_Value.jpg or .png.

Task 5

1. Define a variable called origAbsPath with the absolute path of the current directory as the
variable's value.

Click here for Hint

You can get the absolute path of the current directory using the pwd command.

2. Take a screenshot of the code above and save it as 05-Define_Variable.jpg or .png.

Task 6

1. Define a variable called destAbsPath whose value equals the absolute path of the
destination directory.

Click here for Hint

First use cd to go to destinationDirectory, then use the same method you used in Task 5.

2. Take a screenshot of the code above and save it as 06-Define_Variable.jpg or .png.

 

1Linux Commands and Shell Scripting - Final Project2Getting started34Checkpoint56Note on

arrays78Checkpoint91011Congratulations!

Task 7

1. Change directories from the current working directory to the target

directory targetDirectory.

Click here for Hint

cd into the original directory origAbsPath and then cd into targetDirectory.

2. Take a screenshot of the code above and save it as 07-Change_Directory.jpg or .png.

Task 8

You need to find files that have been updated within the past 24 hours. This means you need to
find all files whose last-modified date was 24 hours ago or less.
To do make this easier:

1. Define a numerical variable called yesterdayTS as the timestamp (in seconds) 24 hours
prior to the current timestamp, currentTS.

Click here for Hint

Math can be done using $(()), for example:

 zero=$((3 * 5 - 6 - 9))

Thus, to get the timestamp in seconds of 24 hours in the future, you would use:

 tomorrowTS=$(($currentTS + 24 * 60 * 60))

2. Take a screenshot of the code above and save it as 08-YesterdayTS.jpg or .png.


1Linux Commands and Shell Scripting - Final Project2Getting started34Checkpoint56Note on
arrays78Checkpoint91011Congratulations!

Note on arrays

In the script, you will notice the line:

1. 1

1. declare -a toBackup

Copied!

This line declares a variable called toBackup, which is an array. An array contains a list of values,
and you can append items to arrays using the following syntax:

1. 1

1. myArray+=($myVariable)

Copied!

When you print or echo an array, you will see its string representation, which is simply all of its
values separated by spaces:

1. 1

2. 2

3. 3

4. 4

5. 5

6. 6

1. $ declare -a myArray

2. $ myArray+=("Linux")

3. $ myArray+=("is")

4. $ myArray+=("cool!")

5. $ echo ${myArray[@]}

6. Linux is cool!

Copied!

This will be useful later in the script where you will pass the array $toBackup, consisting of the
names of all files that need to be backed up, to the tar command. This will archive all files at once!


 

1Linux Commands and Shell Scripting - Final Project2Getting started34Checkpoint56Note on

arrays78Checkpoint91011Congratulations!

Task 9

1. Within the $() expression inside the for loop, write a command that will return all files and
directories in the current folder.

Click here for Hint

There is a very clean way of doing this using ls.

2. Take a screenshot of the code above and save it as 09-

List_AllFilesandDirectoriess.jpg or .png.

Task 10

1. Inside the for loop, you want to check whether the $file was modified within the last 24
hours.

To get the last-modified date of a file in seconds, use date -r $file +%s then compare the value
to yesterdayTS.
if [[ $file_last_modified_date > $yesterdayTS ]] then the file was updated within the last 24 hours!

2. Since much of this wasn’t covered in the course, for this task you may copy the code below
and paste it into the double round brackets (()):

1. 1

1. `date -r $file +%s` > $yesterdayTS

Copied!

3. Take a screenshot of the code above and save it as 10-IF_Statement.jpg or .png.

Task 11

1. In the if-then statement, add the $file that was updated in the past 24-hours to
the toBackup array.

2. Since much of this wasn’t covered in the course, you may copy the code below and place
after the then statement for this task:

1. 1

1. toBackup+=($file)

Copied!

3. Take a screenshot of the code above and save it as 11-Add_File.jpg or .png.


 

1Linux Commands and Shell Scripting - Final Project2Getting started34Checkpoint56Note on

arrays78Checkpoint91011Congratulations!

Task 12

1. After the for loop, compress and archive the files, using the $toBackup array of filenames,
to a file with the name backupFileName.

Click here for Hint

Use tar -czvf $backupFileName ${toBackup[@]}.

2. Take a screenshot of the code above and save it as 12-Create_Backup.jpg or .png.

Task 13

Now the file $backupFileName is created in the current working directory.

Click here for Hint

Move the file backupFileName to the destination directory located at destAbsPath.

2. Take a screenshot of the code above and save it as 13-Move_Backup.jpg or .png.

Congratulations! You have now done the coding portion of the lab!


 

1Linux Commands and Shell Scripting - Final Project2Getting started34Checkpoint56Note on

arrays78Checkpoint91011Congratulations!

Task 14

1. Save the current working file backup.sh with CTRL+s [Windows/Linux], CMD+s [MAC] or by
navigating to File->Save as seen below:
2. Download the file to your local computer by navigating to File->Download as seen below:
You may save the file as backup.sh

3. You will later submit this file will for peer-grading.

Task 15

1. Open a new terminal by clicking on the menu bar and selecting Terminal->New Terminal,
as in the image below:
This will open a new terminal at the bottom of the screen as seen below:

2. Save the backup.sh file you're working on and make it executable.

Click here for Hint

Use the chmod command with the correct options.

 3. Verify the file is executable using the ls command with the -l option:

1. 1

1. ls -l backup.sh

Copied!

4. Take a screenshot of the output of the command above and save it as 15-
executable.jpg or .png.

Task 16

1. Download the following .zip file with the wget command:

1. 1

1. wget https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBM-LX0117EN-
SkillsNetwork/labs/Final%20Project/important-documents.zip

Copied!

2. Unzip the archive file:

1. 1

1. unzip -DDo important-documents.zip

Copied!

Note: -DDo overwrites without restoring original modified date.

3. Update the file’s last-modified date to now:

1. 1

1. touch important-documents/*

Copied!

4. Test your script using the following command:

1. 1

1. ./backup.sh important-documents .

Copied!

This should have created a file called backup-[CURRENT_TIMESTAMP].tar.gz in your current

directory.

5. Take a screenshot of the output of ls -l and save it as 16-backup-complete.jpg or .png.

Task 17

1. Copy the backup.sh script into the /usr/local/bin/ directory. (Do not use mv.)

Note: You may need to use sudo cp in order to create a file in /usr/local/bin/.
 2. Test the cronjob to see if the backup script is getting triggered by scheduling it for every 1
minute.

Click here for Hint

1. 1

1. */1 * * * * /usr/local/bin/backup.sh /home/project/important-documents /home/project

Copied!

3. Please note that since the Theia Lab is a virtual environment, we need to explicitly start
the cron service using the below command:

1. 1

1. sudo service cron start

Copied!

4. Once the cron service is started, check in the directory /home/project to see if the .tar files
are being created.

5. If they are, then stop the cron service using the below command, otherwise it will continue
to create .tar files every minute:

1. 1

1. sudo service cron stop

Copied!

6. Using crontab, schedule your /usr/local/bin/backup.sh script to backup the important-

documents folder every 24 hours to the directory /home/project.

7. Take a screenshot of the output of crontab -l and save as 17-crontab.jpg or .png.

Tip: When you are setting up cron jobs in a real-life scenario, ensure the cron service is running, or
start the cron service if needed.