Jenkins Administration - Fundamentals

COURSE STRUCTURE

COURSE SECTIONS

● This course covers the following topics:

1. Install, set up, and upgrade Jenkins
2. Explore the "Manage Jenkins" screens
3. Manage plugins
4. Configure notifications
5. Implement distributed builds with Masters, Nodes, and Agents
6. Manage security on your Jenkins instance
7. Use folders to segregate builds by project and team
8. Monitor Jenkins
9. Back up your Jenkins instance
10. Use the Jenkins command line interface (CLI) to automate tasks

SCOPE OF COURSE

● Understand and practice:

■ How to efficiently administer Jenkins
■ Best practices
■ Where to find additional information
■ This course covers open source features available in Jenkins

AUDIENCE

● DevOps Engineers
● System Administrators
● Build and Release Engineers
● QA Engineers
● Technical Project Managers
● Architects
PREREQUISITES

● Students should understand the principles of Continuous Integration and Continuous

Delivery (CI/CD), Source Code Management, and modern testing theories that are used in
this course:
● Students should also understand ancillary technologies that are mentioned in this class:
■ Docker; Git; Java; Apache Maven, Gradle, Apache Ant or npm; Apache
Groovy
■ The class has been structured so you can do the exercises even if you are not
familiar with these tools but learning them will help you implement your
Pipelines when you go back to work.
AUDIENCE

● DevOps Engineers
● System Administrators
● Build and Release Engineers
● QA Engineers
 ● Technical Project Managers
● Architects

PREREQUISITES

● Students should understand the principles of Continuous Integration and Continuous

Delivery (CI/CD), Source Code Management, and modern testing theories that are used in
this course:
● Students should also understand ancillary technologies that are mentioned in this class:
■ Docker; Git; Java; Apache Maven, Gradle, Apache Ant or npm; Apache
Groovy
■ The class has been structured so you can do the exercises even if you are not
familiar with these tools but learning them will help you implement your
Pipelines when you go back to work.

Introduction

This document explains how to install and start your CloudBees Lab Environment.

Please follow all the steps carefully, before running any Lab exercise.

Local VM: Vagrant + Virtualbox

A Virtual Machine (VM) will be used for hosting your ​Lab Environment​:

● It does not break any local environment

● It does not depend on the host OS you are using
● It is portable with the same behavior for everyone

This VM runs using the VirtualBox hypervisor, and is managed and automated by Vagrant (see
requirements below).

Both of those tools are Open Source, free and multi-platforms, but they require:

● Having admin rights on your computer for installation only

● Your computer must not be already a Virtual Machine. Nested virtualization is not supported.

Common Requirements

● An HTML5 compliant Web Browser is required: ​Mozilla Firefox​, G

​ oogle Chrome​, ​Microsoft
Edge​, ​Apple Safari​, ​Opera
 Internet Explorer is not supported
IMPORT
ANT

●
The following ports must be allowed access to your instance’s domain (which is ​localhost​):
○ 5000
○ 5001
○ 5002
○ 5003
○ 20000
○ 30001
○ 30002
● The following protocols must be allowed by any antivirus/firewall software:
○ HTTP
○ HTTPS
○ Websockets
■ Some antivirus software like ​Kasperky​ and ​McAfee​ might block websocket
silently
■ You can test websockets from this page: ​WebSockets Test
■ For more about security and websockets: ​Testing WebSockets
● Even if the training lab is running in offline mode, an open Internet access is ​recommended
○ HTTP Proxy can ​only​ be configured for Jenkins operations

Hardware Requirements

Your machine must meet the following ​hardware​ requirements:

● Intel 64 Bits Dual-Core CPU compliant (Intel Pentium/i3 at least)

● 6GB of RAM (the VM will allocate 4GB for itself)
● 20GB of free space on your local hard drive
● One OS from this list:
○ Windows >= 8.1
○ Mac OS >= 10.10
○ Linux "classic" distribution (Ubuntu >= 12.04, Debian >= Jessie, RHEL>= 6)
● The "Virtualization instructions" of your CPU must be enabled (Intel VT-x or AMD SVM)
○ More information here: ​https://forums.virtualbox.org/viewtopic.php?f=6&t=58820
 ○ Intel official VT page:
http://www.intel.com/content/www/us/en/virtualization/virtualization-technology/int
el-virtualization-technology.html

Software Requirements

Your machine must meet the following ​software​ requirements:

● For ​All​ OSes, download and install the latest (64 Bits) versions of:
○ VirtualBox​ (An Open Source Hypervisor from Oracle):
■ Downloads page: ​https://www.virtualbox.org/wiki/Downloads
■ Make sure to download the appropriate binary for your OS
We encourage you to download the latest available version of
VirtualBox. However, it is worth noting that the last version we
tested with this courseware was 6.0.12. So, if you run into trouble
with the latest version, please try using this one.

■
Windows users:
If you have HyperV installed, VirtualBox may throw some errors with
the code ​VERR_VMX_NO_VMX​.

In this case (​Stack Overflow - Vagrant up - VBoxManage.exe

error: VT-x is not available (VERR_VMX_NO_VMX)​), please
disable HyperV temporarily : (​Disable HyperV​)

bcdedit /​set​ hypervisorlaunchtype off

 and reboot

○
Vagrant​ (An Open Source VM manager):
■ Downloads page: ​https://www.vagrantup.com/downloads.html
■ Make sure to download the appropriate binary for your OS
We encourage you to download the latest available version of Vagrant.
However, it is worth noting that the last version we tested with this
courseware was 2.2.5. So, if you run into trouble with the latest
version, please try using this one.

●
For ​Windows​ only, download latest version of ​Git for Windows
Git for Windows​ provides a bash-compliant shell and OpenSSH client
TI

Getting Lab Resources

After installing the software prerequisites:

● Right click this ​link to the virtual machine’s ​ZIP​ archive​ to open it in a new tab or window
○ The archive will download to your local disc
● Extract the virtual machine ​ZIP​ archive to your local disc
○ This archive contains your virtual machine image and automated settings in a folder
named ​training-admin-fundamentals

Starting the Lab Environment

 ● Open a Command Line on your host OS:
○ On Mac OS, open ​Applications​ , ​Utilities​ , ​Terminal
○ On Windows, open ​Start Menu​ , ​Git Bash
○ On Linux, this can be named ​Command Line​ or ​Terminal

The command line is required to start the Virtual Machine without having to care to any
TIP
specific configuration.

●
Using the command line ​cd​, navigate to the un-archived folder that should be located on your
Desktop:
● cd​ ~/Desktop/training-admin-fundamentals/

●
TIP
The ​~​ special character means "Full path to the user home folder"
● Desktop​ may vary depending on your operating system: can be lower case, or
localized in your operating system’s language.

●
Use the command line ​ls​ to check the content of this directory. We need to have a file named
Vagrantfile​ here:

ls -1
● Vagrantfile
● Now you are able to start the Virtual Machine, using the ​vagrant​ command:
● vagrant up

The VM should start without a GUI, and without any ​error​:

If some warnings about VirtualBox version appears, ignore them as long as everything is
TIP
working well.
Figure 1. Vagrant starting the VM for you

● You need to be able to stop and start the Virtual Machine whenever you want. Let’s do it now:
○ From the training-admin-fundamentals folder that contains a ​Vagrantfile​:
○ Stop the VM "gracefully" with the vagrant "halt" command:

REQUIREMENTS AND PLANNING

MINIMAL REQUIREMENTS: JENKINS

● Jenkins runs on current versions of:

■ Linux
■ Windows
■ macOS
■ Unix/BSD
● Java 8 or Java 11 is required
■ Jenkins Long Term Support (LTS) runs on Java 11 beginning in Release
2.164.1
■ Java 9 and 10 are not supported
RUNTIME REQUIREMENTS: JVM
● Java Virtual Machine (JVM) must be tuned:
■ Memory Heap size: -Xms1G -Xmx2G
■ G1 garbage collector for heap > 4GB : -XX:+UseG1GC
● Check your Java Memory documentation for details
RUNTIME REQUIREMENTS: OPERATING SYSTEM

● Beware of operating system Limits

■ Maximum number of open files
■ Maximum number of forked processes
■ Network tuning (packet size, TCP timeouts)
JENKINS IS DISK I/O INTENSIVE

● Jenkins writes a lot of files as it processes builds

■ Some of these can be offloaded to external repositories such as Nexus and
Artifactory
● Disk usage grows over time
■ Especially when you start to host jobs from people who are not close to you
■ Be sure you add more disks when necessary
DISK REQUIREMENTS

● No need to waste money on 15000rpm SCSI disks

■ But bigger disk is nice
■ Low latency is the big winner (SSD)
● Can be network mounted
■ NFS
■ SAN
DISK SPACE

● Artifacts, logs and other files written by builds can be very large
■ Consider external storage such as Nexus or Artifactory for storing
these large files off the Jenkins master
■ Retention policy affects the amount of space these consume
○ If you need to retain all (or most) artifacts for a long time, the
disk consumption issues are more serious
DISK SPACE FOR BACKUPS

● Be sure to configure enough disk space for backups

■ Can configure a separate disk on the master just to hold backups
■ Can back up Jenkins files to a local directory then copy those files to an
external location
○ Must have enough space on the disk to accommodate the
backed up files
○ Recommend creating a separate filesystem (mountpoint) for the
backup directory
EXPANDABLE VOLUME

● “Spanned volume” on Windows

● LVM on Linux
■ This needs the most upfront planning
● ZFS
■ There’s no viable free Solaris flavor
 ● The Jenkins home directory ($JENKINS_HOME) should be its own filesystem
■ If it’s too late, symlink is your friend

INSTALL JENKINS

DOWNLOAD JENKINS

● Jenkins is distributed on many channels:

■ OS native packages (RPM, DEB…​)
■ WAR file
■ Docker image
■ Cloud templates (AWS, Azure…​)
● Start from the ​Jenkins download page

INSTALL JENKINS: LINUX PACKAGES

● Native packages available for major Linux distributions. The major ones are:
■ RPM for Red Hat family
■ Deb for Debian/Ubuntu
■ Many others: Gentoo, OpenSuse…​
● Pattern with package managers:
■ Add the Jenkins Package Repository
■ Install and start Jenkins
● Example for Red Hat family:
# Add the Jenkins Yum Package Repository
$ wget -O /etc/yum.repos.d/cje.repo ​"http://<...>/rpm/jenkins.repo"
$ rpm --import ​"http://<...>/rpm/jenkins.io.key"
# Install it
$ yum install jenkins
# Start it
● $ service jenkins start
MORE ABOUT LINUX NATIVE PACKAGES

● Based on ​stand-alone jenkins.war​ (with embedded Jetty Application Server)

● Benefits:
■ Creates jenkins user
■ Sets up service scripts (init.d, upstart or systemd)
■ Configuration files follow native conventions
■ Provides log rotation out of the box
● Where are files?
■ Settings in /etc/default/jenkins or /etc/sysconfig/jenkins
■ $JENKINS_HOME defaults to /var/lib/jenkins
INSTALLING JENKINS: NATIVE WINDOWS

● Run either:
■ setup.exe
■ jenkins.msi if .NET 2.0 runtime is already available
● Install Jenkins as a Windows service
■ Files are installed into %JENKINS_HOME%
 RUNNING JENKINS: WAR

● The Web Application Archive (WAR) distribution of Jenkins

can be run:
■ as a stand-alone application
■ in a servlet container

RUNNING JENKINS: WAR AS A STAND-ALONE APP

● Run from the command line

● Uses an embedded application server (Jetty)
● Provides some extra features (restart from the web interface,…​)
● $ java ​${JAVA_OPTS}​ -jar jenkins.war ​${JENKINS_OPTS}

JENKINS AS A STAND-ALONE APP: COMMAND LINE OPTIONS

● Options configured by startup flags (JENKINS_OPTS):

--prefix $PREFIX (default: /)
Runs Jenkins to include the $PREFIX at the end of the URL
--httpPort $PORT (default: 8080)
Jenkins listens on $PORT port.
--httpListenAddress $HTTP_HOST (default: 0.0.0.0)
Binds Jenkins to the IP address represented by $HTTP_HOST.
--logfile $LOGFILE
write to $LOGFILE instead of stdout
MORE ABOUT COMMAND LINE OPTIONS

● These "options" are actually flags

■ Passed as members of the JENKINS_OPTS variable to the "java -jar"
command line
mentioned above
● See ​Starting and Accessing Jenkins​ for a complete list of start-up flags that are available
● Caveat: If you misspell an option, Jenkins ignores it rather than generate an error
JENKINS AS A STAND-ALONE APP: SIGNAL

● Signal management:
■ Reacts to SIGTERM and SIGINT to initiate proper shutdown
■ When receiving SIGALRM, log file gets reopened; allow Log rotation.

JENKINS AS A STAND-ALONE APP: EXAMPLE

● Configuration Example:
● java -jar jenkins.war --httpPort=8081 --prefix=/ci --httpListenAddress=127.0.0.1
● The Jenkins instance will be reachable only on ​http://127.0.0.1:8081/ci​ .
● More information in the ​documentation

RUNNING JENKINS: ON APPLICATION SERVER

● Deploy the Jenkins WAR file to an existing application server

● May fit in better with existing infrastructures
● Deploy the jenkins.war file in the usual manner
 INSTALLING JENKINS: CONTAINERS

● Jenkins can be installed in a container

■ Docker
■ Kubernetes
● Both container types provide:
■ Weekly and Long Term Support (LTS) Jenkins versions
■ Extensible; can be used to build your own Docker or Kubernetes Image
■ Based on standalone Jenkins WAR
INSTALLING JENKINS: DOCKER CONTAINER

● Official Docker image on DockerHub: ​https://hub.docker.com/r/jenkins/jenkins/

■ Docker infrastructure ​native​ integration

INSTALLING JENKINS: KUBERNETES CONTAINER

● Official Helm Chart: ​https://github.com/kubernetes/charts/tree/master/stable/jenkins

■ Helm is the package manager used to find, share and use software built for
Kubernetes
● Jenkins-X project: ​https://jenkins-x.io
 ■ Jenkins X is an opinionated platform for providing CI / CD on top of
Kubernetes.
POST-INSTALLATION WIZARD

● The post-installation wizard takes you through a few quick initial configuration steps
that are required before you can view the Jenkins Home Page:
■ Bring up a web browser and go to the ​http://<myServer>:8080​ URL
○ Replace <myServer> with the name of the system running
Jenkins
■ Follow the instructions on the screen to:
○ Unlock Jenkins
○ Install essential plugins
○ Create first admin user
● See ​Setup Wizard​ for more details

UNLOCK JENKINS

● Follow the instructions on the screen to access the administrative password that was set

● Note that you may need to issue the sudo chmod 0755 name_of_dir command
to access the file that contains the initial password

INSTALL ESSENTIAL PLUGINS

● Many of the features of Jenkins are implemented as plugins

● Before opening Jenkins, you need to install a set of essential plugins that are required:

● You can install additional plugins later

CREATE FIRST ADMIN USER

● Create the first admin user by filling out the form on this screen, then click "Save and
Finish":

● You can create additional users after Jenkins is running.

 ACCESSING HOMEPAGE

● You can now access the Jenkins Home Page, also called
the Jenkins Dashboard or the Classic Web UI

JENKINS HOME DIRECTORY

● This is where all the important data goes

■ Need to be backed up, etc.
● Can be set in several ways
■ JENKINS_HOME system property
■ JENKINS_HOME environment variable
■ Falls back to ~/.jenkins
● Can be verified from system config page

JENKINS HOME DIRECTORY STRUCTURE

 WHAT DID WE LEARN ?

● Jenkins is distributed in many channels, even Docker

● Try to use the standalone version
● Tune your JVM and operating system and keep an eye on this
● Know your Jenkins Configuration and Jenkins Home locations

GOING FURTHER

Some recommended readings on this subject:

● Jenkins Home Page
● Installing and Starting Jenkins
● Oracle, ​Understanding Memory Management
● limits.conf(5) man page
● Docker Repository for Jenkins

JENKINS UPGRADES

ABOUT JENKINS UPGRADES

● Upgrade packages for Jenkins are published regularly

■ Long-term Support (LTS) packages are released every 12 weeks
■ Interim packages are released weekly to make critical bug-fixes
and features available quickly to users who need them
● Upgrade to the latest LTS release to stay up-to-date
■ Always check the Release Notes and changelog before you upgrade
● Security releases are published as soon as possible

UPGRADE ALERTS

● The "Manage Jenkins" page tells you when upgrades are available:
 INSTALL JENKINS UPGRADES

● Right-click on "changelog" in the alert panel to open the changelog for this release
■ Read this to be aware of changes this upgrade may make to your system
● Click on "download" to download the upgrade software
● Use the "Prepare for Shutdown" screen under "Manage Jenkins" to initiate
a graceful shutdown of Jenkins
● Replace the jenkins.war file on your system with the new file
■ On Linux systems, this is located in the /usr/share/jenkins directory by default
● Restart Jenkins to apply the upgrade

MANAGE JENKINS

OVERVIEW

● From the Jenkins Dashboard, click ​Manage Jenkins

■ This displays the screen from which most administrative tasks are done
● We are going to take a quick tour of this page now
■ We will discuss major sections in detail later

"MANAGE JENKINS" SCREEN

 NOTIFICATIONS ABOUT UPDATES

● The top of the screen may contain "Monitors" that alert you when a new version of the
Jenkins software or a security update is available.
● You should apply all updates as soon as possible, according to your processes (staging
environment, validation, backup/restore). We will cover this later.
■ New features, bug fixes and security updates
■ Read the changelog for information about what is in the package
■ Click the link to download and install the system

SYSTEM CONFIGURATION

● You can configure many aspects of your Jenkins server:

■ Security
■ JDK installations
■ Build tools - Ant and Maven installations
■ Version control tools
■ Email configuration
■ Plugin-specific configuration
■ …​ and a lot more …​

CONFIGURE SYSTEM

● Use the "Configure System" page to check and modify some basic system parameters
 ABOUT SYSTEM CONFIGURATION

● Sets most of the fundamental tools that Jenkins uses

● Plugins may add sections to this page

GENERAL SYSTEM-WIDE CONFIGURATIONS

● Home Directory​ — Display the Jenkins home directory

■ To change this location, modify the value of the JENKINS_HOME environment
variable
● System Message​ — Text that is displayed at the top of your Jenkins home page
■ Can include HTML tags
■ Use this page to display the name of the server and a short description of its
purpose
■ May include a pointer to a Wiki page and information about who to contact
■ Display announcements about scheduled down time or other information of
interest to all Jenkins users

INLINE HELP

● Click on the "?" icon on the right of each field for information about this field.
● For example, the help screen for the "System Message" box is:

● Click on the "?" icon again to hide the help text

SYSTEM CONFIGURATION - # OF EXECUTORS

● What is an ‘Executor’
■ Unit of task execution on a computer
■ Defines upper bound of concurrent build execution
● Typically, you set this to # of CPU cores
■ Unless your builds are highly parallel in itself
● This configures the number of executors on the master
■ Set this to 0 to prevent builds from running on the master
■ We will talk later about configuring executors for builds on remote nodes
● Watch out for memory requirement

SYSTEM CONFIGURATION - QUIET PERIOD

● Waits for this many seconds before actually starting the build
● Groups adjacent commits
■ This can collapse multiple changes into one build

SYSTEM CONFIGURATION - SCM TOOLS

● Git is supported out-of-the-box

● Plug-in support for others:
■ Subversion, Mercurial, Perforce, Team Foundation Server, …
● Most require the tools to be installed separately

SCM AND AUTHENTICATION

● Most SCM tools support/require authentication when talking to servers

■ How to configure Jenkins depends on SCM
■ Most use Credentials plugin (Git, SVN, Mercurial, Team Foundation Server,
…​)
● Perforce
■ Each project needs you to enter credentials separately

TIMESTAMPER

● Scroll down the page to the "Timestamper" area

● Use these fields to define the format used to display time stamp information

JENKINS LOCATION

● Scroll down further to the "Jenkins Location" area

● Replace the localhost:8080 string with the actual URL that people use to access Jenkins
● Define the email address to which email about general Jenkins issues are sent
■ This can be an individual’s email address or an email alias that contains
multiple email addresses

INVEST IN GOOD URL

● If your users cannot see Jenkins, much of the benefit is lost

■ Make the URL easier to remember:
○ Bad: ​http://tjek-02.int.rev.example.org
○ Good: ​https://jenkins.example.org

USE NAME SERVICE

● Get the host alias rather than use the primary machine name
■ If your IT cannot provide this, use dynamic DNS
■ This also makes your service relocatable

MORE ABOUT SYSTEM CONFIGURATION

● Jenkins is very flexible and powerful

■ Default configurations can be used while you are starting
■ Settings on this page give you fine control of how Jenkins executes
○ As you learn more about Jenkins and your applications, use
these settings to fine-tune the behavior of your Jenkins instance
● When you are done modifying this screen, scroll to the bottom and click the Save button to
apply your configuration changes
GLOBAL TOOLS CONFIGURATION

GLOBAL TOOLS CONFIGURATION

● Use the "Global Tools Configuration" page to configure the tools used for Pipeline
development:

WHAT TOOLS ARE CONFIGURED?

● Some tools are listed here by default; others only if the appropriate plugins are installed
■ JDK and other languages
■ Build tools: Maven, Gradle, Ant and others
■ SCM: Git, Mecurial and others
■ Containers: Docker and Kubernetes (when installed)
● When you select a tool to install, Jenkins provides boxes for you to supply information that is
required to install that tool
● Many support multiple versions of the tool
● Some support auto installation

GLOBAL TOOLS CONFIGURATION - JDK

● Define the JDK versions you need for your projects

● You can define multiple versions for different projects
 GLOBAL TOOLS CONFIGURATION - JDK

● Install JDK versions automatically

GLOBAL TOOLS CONFIGURATION - MAVEN

● Install Maven automatically

RELOAD CONFIGURATION FROM DISK

● Click on "Reload Configuration from Disk" to refresh the Jenkins configuration files and
directory structure without restarting Jenkins

● Use this when you modify Jenkins or its environment outside of the UI
■ For example, when you edit config files from the command line or move jobs
between folders using the command line
● Not necessary when you modify the configuration from the UI or restart Jenkins after
modifying

SECURITY

CONFIGURE GLOBAL SECURITY

● The "Configure Global Security" screen is used to:

■ Define how users are authenticated and what they are authorized to do
■ Control other security settings

MANAGE USERS SCREEN

● The "Manage Users" screen is used to add user to the Jenkins user database and lists all
users who are in that database

CONFIGURE CREDENTIALS

● Jenkins credentials control access to third-party sites and applications such as artifact
repositories and cloud-based storage systems.
● Use the "Configure Credentials" screen to manage credentials:

● We will discuss credentials at more detail later

SCRIPT CONSOLE

● The Script Console allows you to type in and execute an arbitrary Groovy script on the
server.
 ■ Apache Groovy is the foundation of the DSL used for Jenkins Pipelines

● This is useful for troubleshooting and diagnostics

● Scripts execute in a Groovy Sandbox that limits the internal APIs that are accessible
● With the Script Security plugin, administrators can use these results to manage which
unsafe
PROVIDE BUILD ENVIRONMENTS

● Administrators must configure build environments that Pipeline developers can use.
● Two main elements:
■ Configure the tools that Pipeline developers can use
■ Configure the nodes and agents on which Pipelines execute

MANAGE NODES

● Use the "Manage Nodes" screen to create, configure and view nodes:

RECAP OF NODES, AGENTS AND EXECUTORS

● The "Jenkins - Fundamentals" class discusses these terms. To recap:

■ A ​node​ is a server where Jenkins runs jobs on ​executors
■ The ​agent​ is the tool that manages the ​executors​ on a remote ​node​, on
behalf of Jenkins.
■ The Jenkins master also runs on a ​node​.
 CREATE, CONFIGURE AND VIEW NODES

● The "Manage Nodes" screen opens to display information about nodes that are currently
configured:

● Initially, this just shows the node where the Jenkins master runs.
● Click "New Node" in the left frame to create a new node
● Click the flywheel to the right of the node line to pop a screen where you can define the
number of executors and other characteristics of this node
● Click the flywheel in the left frame to configure the items Jenkins monitors for all nodes it
runs
MONITORING NODE USAGE

● Use the "Load Statistics" page to monitor node utilization for your Jenkins instance:

NODE USAGE INFORMATION

● The "Load Statistics" reports the following information about the nodes configured on your
Jenkins instance:
■ Number of online executors
■ Number of busy executors
■ Number of available executors
■ Queue length (number of jobs that are waiting for an available executor)
● Use this information to determine when you need to add more nodes and/or executors to
your instance to improve throughput
INFORMATION PAGES

● Some of the "Manage Jenkins" pages provide reference information that will be useful as
you administer your Jenkins cluster:
■ System Log
■ About Jenkins
 ■ System Information
■ Jenkins CLI

SYSTEM LOG

● The "System Log" page includes a link to the log about activities on the Jenkins instance itself:

● Note that this information is independent of the build logs that are presented for each job or
Pipeline that runs
ABOUT JENKINS

● The "About Jenkins" page shows the current release of Jenkins on your system plus
information about licenses for all components:

ABOUT JENKINS INFORMATION

● Release and version of Jenkins that is running

● List of all third-party libraries used for this release of Jenkins
■ Links to licensing details about each library
● List of installed plugins
■ Includes a link that shows all third-party dependencies for each plugin
■ That list includes a link to licensing details about each library

SYSTEM INFORMATION
● The "System Information" page provides detailed information what is available on this
Jenkins instance:

WHAT IS ON SYSTEM INFORMATION PAGE

● This page contains the following lists:

■ System Properties​ that can be used as arguments to the command line used
to start Jenkins
■ Environment Variables​ with current values
■ List of ​Plugins​ installed on the system

JENKINS CLI

● The Jenkins CLI page lists supported CLI commands:

WHAT IS JENKINS CLI

● The Jenkins Command Line Interface (CLI) provides commands that perform functions that
are usually executed using the UI
● These can be used in scripts that automate administrative tasks
● Can be accessed in either of the following ways:
■ Over SSH
■ With the Jenkins CLI client, which is a .jar file distributed with Jenkins
 ● See ​Command Line Interface​ in the Jenkins Handbook for more information
MANAGEMENT: SERVICE LIFECYCLE

Managing Options used to Start/Stop/Reload Jenkins service

● Reload Configuration From Disk:
■ Read configuration again from $JENKINS_HOME without Jenkins restart
● Prepare for shutdown:
■ Prevents new jobs from being started
■ Waits for running jobs to complete (clean job termination)

JENKINS UPGRADE

● Management page provides Jenkins Upgrade management

■ With standalone server: 1-click upgrade
■ Without: direct link to latest WAR
● "Manage Old Datas" helps to take care of dropped configurations items after an upgrade

WHAT DID WE LEARN ?

● Jenkins has a dedicated Management page with many sections, providing:

■ System, Tool configurations
■ Security options
■ Metrics, Logs, Configuration Item overview
■ Lifecycle Management utilities
● Jenkins can be managed with Apache Groovy Code
■ Helps integrating with Configuration Management
■ Automate large scale Jenkins deployment

LAB EXERCISE

Configuring Global Tools

Configuring Global Tools

The goal of this exercise is to install two different versions of Maven

● Maven 3.5.4
● Maven 2.2.1

Install Maven

Login to ​http://localhost:5000/jenkins​ using the following credentials:

● Username: ​butler
● Password: ​butler
Start by going to ​Jenkins → Manage Jenkins → Global Tools Configuration

● Scroll down to Maven

● Under Maven Installations, click ​Add Maven​ to install Maven
● Enter ​mvn3​ for ​Name
● Make sure the toggle for ​Install automatically​ is checked
● Enter ​3.5.4​ for ​Version

Figure 1. Install Maven Versions

● Click ​Add Maven​ to install another Maven version
● Enter ​mvn2​ for ​Name
● Make sure the toggle for ​Install automatically​ is checked
● Enter ​2.2.1​ for ​version
● Click ​Save​ at the bottom of the screen to save the changes

You have now installed two versions of Maven

Create a Maven Build Project

The goal of this lab exercise is to configure a simple Maven build job on your Jenkins server.

Step1. Create a new build job

Start by clicking on the ​New Item​ menu on the home page, and choose ​Maven project

● Name this project ​pipeline-job​ and click ​ok

Figure 2. Create Pipeline Maven Job

Step2. Configure the SCM details

● For this build job, you need to configure those SCM settings:
● Choose ​Git​ in the ​Source Code Management​ section
● In the ​Repository URL​ field, enter your GIT repository address:
http://localhost:5000/gitserver/butler/pipeline-lab.git
● Since this repository needs authentication to be accessed, you must add a new credential with
those properties:
● Kind: ​Username with password
● Scope: ​Global​ (to allow reusing in other jobs)
● Username is ​butler
● Password is the same value as the username
● ID is ​butler-git-password
● Description is ​Git User and Password for butler

Figure 3. Add credentials for authenticating to SCM

 Figure 4. Configure SCM

Step3. Configure the build goals

● Make sure to select ​mvn3​ for M

​ aven Version
 ● Finally, you need to configure the Maven build goals, in the ​Goals and options​ field: ​clean install

Figure 5. Specify Maven Goals

● Now ​save​ this project.
● Click ​Build Now​ to kick off a new build

Go back to the dashboard to watch the build in progress

That is all for this exercise

PLUGINS

JENKINS PLUGINS

● Jenkins uses plugins to provide much of its functionality

 ■ Many "classical" Jenkins functions are implemented as plugins
■ More than 1700 plugins are available
● This "modular" architecture means that your Jenkins installation includes the features and
functionality you need without being bloated with functionality you do not need
● Managing plugins is a critical part of managing Jenkins
■ Many features are implemented with multiple plugins
■ Many plugins have dependencies on other plugins

FEATURES IMPLEMENTED WITH PLUGINS

● Source code management tools

● Build Tools
● Reporting tools
■ Code coverage, static code analysis
● Online source code browsers
● Issue tracker
● Notification tools
● Views and UI customizations
● Distributed builds

WHAT ARE PLUGINS ?

● A plugin is a JAR file with some special conventions (e.g. no web.xml)

● Contained in a file with an hpi or jpi extension
● Stored in the ${JENKINS_HOME}/plugins directory
■ unless you use --pluginroot to change where the binary of the plugins are
"exploded"
● May have dependencies (mandatory or optional)
● Plugins are versioned artifacts that can be upgraded
■ If necessary, they can be downgraded but this is not generally recommended

SUGGESTED PLUGINS

● The Installation Wizard provides a list of "Suggested Plugins" that everyone should install
■ These are plugins that provide commonly-used functionality
■ Are widely used, well-maintained, and have been examined for security issues
● This list is a feature of the Installation Wizard
■ When you upgrade to later Jenkins releases, it is up to you to identify new
plugins that you may want to install

LIST OF INSTALLED PLUGINS

● Click on ​Manage Jenkins → System Information​ to see a list of installed plugins; for
example:

● A list of installed plugins is also available from the ​Manage Plugins​ page we will discuss
next

MANAGE PLUGINS

● Use the "Manage Plugins" page to add, remove, update, enable or disable plugins:

MANAGE PLUGINS — DETAILS

● Tabs at the top of the page define what is displayed:

● Use the "Filter Box" to search for specific plugins

INSTALLED TAB — COLUMN HEADERS

● Name​ — Name of the plugin with a brief description

■ Click on the ​Name​ column header to toggle between alphabetical order and
reversed alphabetical order
■ Right-click on the plugin name to view the official documentation for the plugin
● Version​ — Version of each installed plugin
● Previously Installed Version​ — Links used to "roll back" a plugin to the previously
installed version
■ Do not try to roll back a plugin unless absolutely necessary
■ Additional cleanup is required after a rollback
● Enabled​ — Check mark indicates that the plugin is enabled
■ Greyed-out check marks indicate plugins that are required by other plugins
● Uninstall​ — Links used to uninstall plugins
■ Lines with no "Uninstall" button are for plugins that are required by other
plugins

UNINSTALLING A PLUGIN

● Removes the plugin binary (jpi or hpi file) from the disk
■ Plugin continues to function until you restart Jenkins
■ After a restart, plugin does not appear in the UI and all extensions it
contributed disappear
● Does not remove the configuration that the plugin may have created
■ If the plugin contributed extensions to any jobs/agents/views/builds/etc,
Jenkins reports unrecognized fragments in configuration files then ignores
such fragments
■ Until you remove those configuration files, you can re-install the plugin and
restart Jenkins to restore the configurations
■ Use the "Manage Old Data" screen to scrap configurations left by an
uninstalled plugin

DISABLING A PLUGIN

● Disabling a plugin is a softer way to retire a plugin

■ The extension for the plugin binary is changed to jpi.disabled or hpi.disabled
■ Jenkins does not start the plugin
■ Extensions contributed by this plugin are not visible
■ The plugin still appears in the "Installed" list
○ The box under the ​Enabled​ column is not checked
● Fragments contributed to configuration files by a disabled plugin are handled like those for
uninstalled plugins
● Reenable the plugin by checking the box in the ​Enabled​ column
■ You may need to restart the instance to make the plugin operational

MANAGE OLD DATA

● Use the "Manage Old Data" page to clean up configurations left by uninstalled and disabled
plugins:

● The page lists any "old data" on your Jenkins instance and discusses they types of old data
and how to handle each

UPDATES TAB

● The Updates tab lists installed plugins for which an update is available:

UPDATE PLUGINS REGULARLY

● Many plugins are updated frequently

■ Enhanced functionality, bug fixes, security fixes
● These should be applied as soon as is feasible
■ Always read the changelog information about new plugins before installing
 ■ Use a staging environment to validate updated plugins before they are applied
to the production environment
■ Take a backup of your instance before updating a plugin
● Many updates take effect only after Jenkins is restarted

JENKINS UPDATE CENTER

● The full set of open source plugins are available at the ​Plugins Center
■ Also called the "Update Center"
● When you download a plugin:
■ Jenkins downloads the plugin index from the configured Update Center
■ Jenkins filters the index by the core version
■ If you install the plugin, the proper binary is downloaded

AVAILABLE TAB

● The "Available" tab lists plugins that are available for installation
● Click the "Install" column header to determine whether the plugins are listed alphabetically
or sorted by category:

WHAT PLUGINS ARE LISTED AS AVAILABLE?

● The "Available" tab contains a filtered list of links to the Update Center
● This list is NOT curated
■ Inclusion on this list means only that the plugin can be installed
■ Look for the number of installations and how frequently and recently updates appear
to evaluate the reliability of each plugin before you download it.
● The list of available plugins is updated for each Jenkins release but is not updated between
releases
INSTALL NEW PLUGIN

● To illustrate how to install a new plugin, we are going to install the beer plugin
 ■This is a harmless plugin that adds a page on the Jenkins root to print some jokes
about beer
■ It has no dependencies and is lightweight.
● You MUST have Internet access (HTTP proxy supported) to download a new plugin from this
page
CONFIGURE HTTP PROXY ACCESS

● If you are running under an HTTP proxy, select the ​Advanced​ tab to configure HTTP proxy
access:

● Click the ​Advanced​ button on the bottom right to enable the ​Validate Proxy​ configuration
fields before clicking "Submit" to submit the configuration
INSTALL NEW PLUGIN

● To illustrate how to install a new plugin, we are going to install the beer plugin
■ This is a harmless plugin that adds a page on the Jenkins root to print some
jokes about beer
■ It has no dependencies and is lightweight.
● You MUST have Internet access (HTTP proxy supported) to download a new plugin from
this page
CONFIGURE HTTP PROXY ACCESS
● If you are running under an HTTP proxy, select the ​Advanced​ tab to configure HTTP proxy
access:

INSTALL BEER PLUGIN

● Type "beer" in the filter box to locate the plugin:

● Click the box in the "Install" column

● Click "Install without restart"

INSTALLING PLUGINS/UPGRADES SCREEN

● When you click "Install without restart", the following screen is displayed:

● Note the checkbox "Restart Jenkins when no job running"

RUN THE PLUGIN

● Return to the Jenkins dashboard

● Note the new item in the left frame:
 LAST REMARKS ABOUT PLUGINS

● You can install plugins manually, without using the "Manage Plugins" UI; this is not covered
here
● Plugins may have dependencies — sometimes a lot of them
■ The documentation for each plugin lists all dependencies
■ The plugin manager installs all ​required​ dependencies
■ You must manually install ​optional​ dependencies if you want them
● Some plugins require a Jenkins restart before they can be run.
■ Plan accordingly when installing plugins in a production environment

LAB EXERCISE

Installing necessary plugins

MANAGE PLUGINS — INSTALLED TAB

● The "Installed" tab lists the plugins that are installed on this instance:

Installing necessary plugins

The goal of this exercise is to install 2 plugins using 2 different methods:

● Recommended: Using the Update Center

● Manual method
 Please note that Internet access (HTTP proxy supported) is ​required​.
IMPORTANT

If you do not have one, you can safely skip the the ​Update Center Method
section and will only be able to run the ​Manual Method​.

Update Center Method

We are going to install a harmless plugin: ​the Beer plugin

This plugin just adds a page on Jenkins root to print some jokes about beer. It
TIP
has no dependencies and is lightweight.

Login to ​http://localhost:5000/jenkins​ using the following credentials:

● Username: ​butler
● Password: ​butler

Start by going to the ​Plugin Management Page​:

● From the ​Manage Jenkins Page​, click on the ​Manage Plugins​ link

Figure 1. Accessing the Plugin Management

 If you are running under an HTTP proxy, select the ​Advanced​ tab:
TIP

● Use the ​HTTP Proxy Configuration​ section to configure

the HTTP proxy access
● Click the ​Advanced​ button to enable the ​Validate Proxy
configuration fields before submitting the configuration

●
Install the plugin with the "GUI Update Center":
○ Select the ​Available​ tab:
○ Use the ​Search Filter​ to search for the ​beer​ keyword
The search is done on the fly: do ​NOT​ hit the
IM
"Return" Key, or you will be redirected you to
P
the plugins installation’s log page.
O
R
T
A
N
T

○
From there, select the beer plugin:

Figure 2. Searching for the Beer plugin

 ● Click the ​Install without restart​ button and you’ll land on the "waiting installation" page:

Figure 3. Waiting for the Plugin Installation

Please note that some plugins require a Jenkins Restart and/or downloading
IMPORTANT
dependencies (other plugins). Plan this carefully in production !

Note the checkbox "Restart Jenkins when no job running"

TIP

Once everything terminates, go back to the ​Jenkins Homepage​. You should see a refreshing new Link in
the left menu, to use when waiting for builds to complete:
Figure 4. Beer Plugin is installed !

Manual Method

We are going to ​manually​ install another harmless plugin: ​the Chuck Norris plugin

This plugin provides "Chuck Norris Facts" (read: jokes) on the status build
TIP
screens. It is also lightweight and has no dependencies.

To download the plugin, open the following link on your web browser:
http://localhost:5000/chucknorris.hpi

By default, the download is done from the Lab VM to avoid Internet connectivity
TIP
issues. However, if you have internet access, you can also download the plugin
from the plugins.jenkins.io website: ​https://plugins.jenkins.io/chucknorris

●
Go back to the ​Manage Plugin​ page and select the ​Advanced​ tab:
○ We’re going to use the ​Upload Plugin​ section
This section requires you to have the ​*.hpi​ file of the
T
plugin.

○
Select the previously downloaded plugin by using the file upload option
○ Click the ​Upload​ button
● You’ll then land again on the same ​Waiting for installation​ page:
○ Wait for the end of the plugin’s installation as you did on the previous section
○ You should restart the Jenkins instance, by selecting the related checkbox
 After a restart, you can check the installation state by browsing the ​Installed​ tab
TIP
on the ​Manage Plugin​ tab. Use the plugin documentation to see how to use it
(Clue: Post-Build Action)

Going further

● There are also other methods to install plugins:

○ Jenkins Official Docker Image​ builds a Jenkins Docker Image with the plugins you
specify.
■ This Jenkins Docker Image provides a shell command to fetch the exhaustive list
of plugins (+ versions) for a given Jenkins instance; see ​Preinstalling plugins​]
○ You can package your own Jenkins and plugins by using the ​Custom WAR Packager
○ You can copy plugins directly to the ​${JENKINS_HOME}/plugins​ folder and restart
Jenkins
● But ​outside​ the ​Update Center​ method, the dependency management is ​YOUR​ responsibility.
You have to manage your upgrade policy carefully, in order to have ​stability​ across versions!

That’s all for this exercise !

CONFIGURE NOTIFICATIONS

NOTIFICATIONS FOR CI/CD

● Primary feedback mechanism

● Can be delivered through email, Slack, etc
● Pipeline code defines the content of notifications for that build
● Administrator must configure and manage the notifications
PREPARE FOR NOTIFICATIONS

● Install required plugin(s)

● Configure globally in Jenkins Management:
■ Notification Service (Email Extension, Slack, etc)
■ Credentials (generally an API token)
■ Other provider-specific options (rooms, team domains)

DEFINE A NOTIFICATION

● Per Project, specify:

■ When do you want to trigger a message ?
○ Status change, warning, other event..
■ Recipients for the messages ?
 ○ Can depend on trigger (notify only the developer who broke the
build)
■ What do you want to say ?
○ This is about the content of the message.
○ Use templates and environment variables to have the "right"
information; for example, you can use the Jenkins
BUILD_NUMBER environment variable to include the build
number in the subject and body of the message; if you are using
Git as your SCM, you can use the GIT_COMMIT environment
variable to specify the Git commit

BUILT-IN EMAIL NOTIFICATIONS

● Main configuration is about SMTP (sending) server

■ Hostname, port, encryption, authentication, SPAM configuration
● Built-in Jenkins Mailer:
■ 1. Every failed build triggers a new e-mail
○ Never fails a build
■ 2. A successful build after a failed (or unstable) build triggers a new e-mail
○ Crisis is over
■ 3. An unstable build after a successful build triggers a new e-mail
○ A wild regression appears
■ 4. Unless configured, every unstable build triggers a new e-mail
○ Regression still there
● Project configuration level: assign recipients to the whole project

EMAIL-EXT PLUGIN

● E-mail Ext plugin extends the built-in Jenkins Mailer capability

● Better customization for:
■ Triggers add more granularity: 2nd failure, aborted build, Not built, Test
Improvement…​
■ Content has many more templates and variables. Can attach scripts.
■ Recipients can change depending on notifications. Managed at both global
and jobs levels.

INSTALLING EMAIL-EXT PLUGIN

CONFIGURING EMAIL-EXT GLOBALLY

CONFIGURE EMAIL-EXT PER PROJECT
 SLACK NOTIFICATION PLUGIN

● Slack Notification plugin allows to post build notifications to a Slack channel

● Better customization for:
■ Notifications can be be configured to be sent for any build status:
○ Start, Aborted, Failure, Unstable…
■ Custom content message can be included in the notifications:
○ Jenkins Environment Variables, Commit list, Authors
■ Notifications can be sent via bot users
● Managed at both global and job levels

INSTALLING SLACK NOTIFICATION PLUGIN

 CONFIGURING SLACK NOTIFICATIONS - GLOBAL

CONFIGURING SLACK NOTIFICATIONS - PER PROJECT

 WHAT DID WE LEARN ?

● Jenkins allows you to send notifications

● Configure the appropriate plugins for how you receive notifications
■ Emails, Slack etc
● Goals of notifying:
■ Notify Developers when a Build Fails
■ Failure Notification to Developers Who Break a Build
● Take in account reduction of the Notification Spam
■ "Right" message, to "right" recipient at the "right" time

GOING FURTHER

Some recommended readings on this subject:

● Jenkins: Notifications
● Cleaning up and notifications
● Notification Plugin
● Mailer Plugin
● Slack Plugin
 Masters, Nodes, and Agents

DISTRIBUTED BUILD ARCHITECTURE

WHAT IS DISTRIBUTED BUILD ARCHITECTURE

● Distributed Builds run on nodes other than the master node.

■ Additional nodes are configured and the master node manages these nodes to
do the actual buildwork
● Distributed build architecture is recommended for all production systems
■ For demonstration purposes, you can run builds on the master node

JENKINS COMPONENTS

● The ​Jenkins Master​ is the Jenkins service itself.

■ It is a webserver that also acts as a "brain" for deciding how, when and where
to run tasks.
● A ​node​ is a server where Jenkins runs jobs on ​executors​.
● The ​agent​ is the tool that manages the ​executors​ on a remote ​node​, on behalf of Jenkins.
● Note that the Jenkins master also runs on a ​node​.

JENKINS MASTER

● This is where Jenkins is installed

● Management tasks (configuration, authorization/authentication)
are executed on the master
● Files written when a Pipeline executes are written to the filesystem
on the master unless they are off-loaded to an artifact repository
such as Nexus or Artifactory

PROPER USE OF JENKINS MASTER

● Do not run the workload of building projects on the Master

in a production environment
■ You can do this for demonstration and study purposes
● Some administrative tasks (such as running a Backup)
are executed on the node that hosts the Jenkins Master
■ Add an executor to the master to perform the task
○ You can label this executor (for example, backup) and only allow
it to be used by jobs that specify it
○ You can also create the executor, use it, then delete the
executor to prevent build jobs from running on the master

NODES

● Provide the tools and packages to be used during your builds:

■ Application SDKs such as the JDK or C++ build tools
■ Apache Maven, Gradle, npm, shells, Docker, etc.
● Jenkins monitors each attached node for:
■ Disk space, free temp space, free swap
■ Clock time/sync and response time
● A node is taken offline if any of these values go outside the configured threshold

WHAT ARE DISTRIBUTED BUILDS ?

● Distributed Builds are builds that run on nodes other than the master node.
■ Master
○ Serves HTTP requests
○ Stores all important information
■ Agents
○ 170KB single jar
○ Assumed to be unreliable
○ We know of clusters with 500+ nodes online at the same time
○ Memory, network bandwidth, disk I/O speed, etc on
the master usually limit the number of builds that
can be run simultaneously on the cluster more than
the number of agents does

ARCHITECTURE OF DISTRIBUTED BUILDS

ADVANTAGES OF DISTRIBUTED BUILDS

● Your ${JENKINS_HOME} is protected from malicious builds
● Makes the Jenkins instance more scalable
■ If you do not have enough resources to run your builds, just add more agents
■ Distributes the build load for better resource utilization
● You can define specialized nodes
■ Run different operating systems, use different CPU architectures, or different
JDK versions

WHAT ARE DISTRIBUTED BUILDS: AGENTS

● Build agent is a small Java program

■ Runs locally on the ​Node​ (machine) that it controls
■ Accesses files, forks processes, etc., on behalf of master
■ Uses the ​slave.jar​ file
○ Fetchable from the Jenkins Master
○ Point your browser to jnlpJars/slave.jar under
$JENKINS_URL

MASTER AND AGENT COMMUNICATION

● Needs a bi-directional byte stream

■ No shared file system, no network topology constraints, etc
● Agent-to-Master Access Control is enabled by default
■ Protects the master from malicious agents
● Very flexible in how this gets set up
■ See ​Master/Agent​ Connectors

MASTER-AGENT CONNECTORS

● An Agent can be launched using any of these master/agent connectors:

■ SSH issued from master
○ Supported on all Linux and MacOS systems
○ Windows 10, Windows Server 2012, and Windows Server 2016
offer an optional SSH server
■ Agent Managed as Windows Service
■ JNLP (Java Web Start)
■ Custom script run on Master

MASTER-AGENT CONNECTORS: SSH

● Communication between Jenkins and Unix agents can use SSH

● SSHD only needs to be installed on agents
■ Jenkins can install Java and other tools
■ Agent’s JAR file is managed by Jenkins Master
● Just need a hostname and credentials to connect
■ Master’s SSH Public Key must be in remote node’s authorized keys
 MASTER-AGENT CONNECTORS: WINDOWS SERVICE

● To connect Jenkins to Windows agent, DCOM is an option

■ Just need admin user name and password
■ No manual intervention
■ Works even from Unix masters

MASTER-AGENT CONNECTORS: JNLP

● Agent is launched as a single command with the slave.jar file and signals itself to the master
■ For cases when the master cannot “see” agents (firewall, subnets)
■ A separate socket connection is made (TCP or HTTP-tunneling)
■ The TCP port must be defined on the ​Manage Jenkins → Configure Global
Security​ page
● Once started, can be installed as a service (Unix Daemon, systemd, …​)
■ Can also be run ​headlessly​ via jar (instead of Java Web Start with GUI)
 MASTER-AGENT CONNECTORS: SCRIPT ON MASTER

● Launch agent with script on master:​ Customizable way of launching agent.

■ Responsibility of the script to :
○ Fetch slave.jar
○ Establish connection by launching remote agent
○ Do anything else related to your custom need

HOW TO: CREATE AGENT

● Access the ​Node​ Management page:

■ From Jenkins ​Management Page​:
■ Browse to ​Manage Nodes
● Jenkins Master is already (and ​always​) registered as a "Node"
● You can create, delete, edit ​nodes​ from there

BENEFITS: SCALABILITY

● Masters can use resources to orchestrate, allowing one master

to orchestrate 100s to 1,000s of builds a day.
● When build demand increases, scale up by merely pointing Jenkins
at another agent node, rather than creating an entirely new master from scratch.

BENEFITS: SECURITY
● Jobs that run on a master have full permissions to all Jenkins resources on the master
node, potentially allowing a malicious user to access private information from other builds or
the system itself.
■ Running jobs on separate and isolated machines mitigates this risk.
● For even greater security, use Cloud Agents that are created for each build then destroyed
when that build is over.
■ If a malicious agent taints the node/executor, no other build is affected.

INSTALLED TOOLS PROBLEM

● Tools location problem: Are JDK, Maven, Gradle and other CI tools installed on ​all​ your
agents?
■ Are you really that good at provisioning machines?
■ Huge​ Nodes in that case
■ With versioning problems (JDKs 6, 7, 8, 9 …​)
● From this, we consider 3 patterns of Agents:
■ Dedicated Agents
■ Fungible Agents
■ Cloud Agents

SOLUTION 1: DEDICATED AGENT

● Default case; having dedicated agents is ​required

■ Specialized hardware / operating systems
■ Segregation by teams / groups
■ Specialized software
● Jobs can be tied to specific Agents using ​labels
■ Use the ​Restrict where this project can run​ Project Configuration Option
■ Pattern and logical combination can be applied
■ Agent Usage must be tuned:
○ Use it as much as possible?
○ Restrict only when requested by a label restriction?

SOLUTION 2: FUNGIBLE AGENTS

● A ​Fungible​ Agent can be substituted by any other Agent

■ Notice, when creating Node, how it is called ​"Dumb"​ ? :o)
● Builds know they are running in "empty" environment:
■ Consider installing required tools themselves
■ Use Pre-Build Step scripts, other Jobs, Pipeline Libraries
■ A plugin is available for this: ​Custom Tools Installation

SOLUTION 3: CLOUD AGENTS

● Throwable​ Agent in a Cloud, to ​efficiently use​ resources

■ Start a Cloud-based Agent
■ Run the build inside this agent
■ Tear-down the agent
● Cloud​ can be a ​public​ cloud like Amazon EC2, MS Azure.
 ■ Can also be any private platform supported by ​Apache jclouds​ (Apache
CloudStack, OpenStack, VMware…​)
● Can be combined with the "Bring you own Agent" pattern
■ Jenkins Users can specify their own cloud templates with their ​own​ tools
● This is ​easy​ to do with container technologies like ​Docker:​
■ Quick start of container and immutable templates; no waiting!

MONITORING NODES

● Jenkins has a tool named ​NodeMonitor

● It runs healthchecks against ​Nodes
■ CPU and memory usage
■ Checking disk space
■ More can be added by plugins
● When a node is down or in bad shape, Jenkins stops launching builds until it comes up
again
■ Alerting can be triggered for admins
WHAT DID WE LEARN ?

● Jenkins can distribute its build workload across a farm of ​Agents

● A Jenkins good practice is: ​Never build on master
■ Simple Security
■ Scale build workloads
■ Separate concerns
■ Specialized builds
● Be careful: Distributing builds adds some overhead for sysadmin
■ Road to DevOps: Jenkins users will efficiently learn their tooling
● Highly tunable concepts:
■ Scheduling, monitoring, restrictions, resource usages, provisioning

GOING FURTHER

● Some recommended readings on this subject. Note that some of these documents use the
obsolete "slave" terminology to refer to an agent.
■ Distributed builds​ is an updated version of Kosuke Kawaguchi’s seminal and
comprehensive paper
■ Distributed Builds Architecture
■ Step by step guide to set up master and agent machines on Windows
■ Swarm Plugin
■ Custom Tools Plugin
■ Job Restrictions Plugin
■ Docker Slaves Plugin
● If you are interested in really large Jenkins clusters, you may enjoy the ​So you want to build
the world’s largest Jenkins cluster?​ presentation from the 2016 Jenkins World.

LAB EXERCISE

Distributed Jenkins Builds

Distributed Jenkins Builds

The goal of this exercise is to create nodes and agents to create a distributed Jenkins instance that is
scalable and secure.

The target is to have:

● 2 ssh agents that handle builds

● Each agent has 1 executor
○ We have 2 CPUs available; rule of thumb says ~2 executors
● The master does not have any executors, so no builds run on master
● Security must follow up

Start by logging into ​http://localhost:5000/jenkins​ using the following credentials:

● Username: ​butler
● Password: ​butler

Please note that the lab instance is already configured with two agents. We will first go ahead and delete
those.

● Go to ​Manage Jenkins → Managed Nodes

● You will see two agents, ​jdk7-node​ and ​jdk-8-node

Figure 1. Manage Nodes

● Click the down arrow next to ​jdk7-node​ and select D
​ elete Agent
 ● Confirm that you really want to delete the agent by clicking ​yes​ button

Figure 2. Delete jdk7-node from nodes list

● Repeat the steps above for deleting ​jdk8-node​ agent.

Figure 3. Delete jdk8-node from nodes list

Adding SSH agents

The Lab instance has an SSH "nodes" already running with these properties:

● Hostname: ​jdk8-ssh-agent​ and j​ dk7-ssh-agent

● Port: ​22
● Username: ​jenkins
● User authentication: RSA private Key
○ Key Passphrase: ​No Passphrase
○ Private Key location: In ​Devbox​, in the file /​ ssh-keys/vagrant_insecure_key

A Passphrase is different than a password ! Passphrases are typically longer

TIP
than passwords (for added security) and are most often used to control access
to cryptographic systems.
 Validating SSH

We are going to "validate" the SSH connection first:

● Spawn a shell in the ​Devbox

● Use ssh to connect to the SSH node using previous settings:

cloudbees-devbox $ ssh -i /ssh-keys/vagrant_insecure_key -p22 \

jenkins@jdk8-ssh-agent \
​echo​ ​"-- Hello from jdk8-ssh-agent"
...
Are you sure you want to ​continue​ connecting (yes/no)? yes
...
-- Hello from jdk8-ssh-agent

cloudbees-devbox $ ssh -i /ssh-keys/vagrant_insecure_key -p22 \

jenkins@jdk7-ssh-agent \
​echo​ ​"-- Hello from jdk7-ssh-agent"
...
Are you sure you want to ​continue​ connecting (yes/no)? yes
...
● -- Hello from jdk7-ssh-agent
● Copy the content of ​/ssh-keys/vagrant_insecure_key​ somewhere; we will use it in Jenkins

Adding the two SSH Agents in Jenkins

● Log in to Jenkins as an administrator account

● Browse to the ​Node Management Page​:
○ Use ​Manage Jenkins​ and then ​Manage Node​ links
○ or use direct URL: ​http://localhost:5000/jenkins/computer/
You see that your Jenkins instance, by default, has a
T
node created. This is the node used for the
Jenkins master itself.
 ●

Create a ​New Node​ using the left-menu:

○ Name: ​ssh-agent-1
○ Type: ​Permanent Agent
● Configure it with these properties:
○ # of executors: ​2
○ Remote root directory: ​/home/jenkins
○ No Labels
○ Usage: ​Use this node as much as possible
○ Launch method: ​Launch slave agents via SSH​ and configure it with previous properties
■ Host is ​jdk8-ssh-agent
■ Set the Credentials field dropdown to "Jenkins (SSH Key for the Agent)"
■ Click the button ​Advanced​ to access the ​Port​ field, and set it to ​22
○ ​ on verifying Verification Strategy
Host Key Verification Strategy​: N
○ Under the Node Properties
■ Check the toggle for ​Environment Variables
■ Click ​Add​ to add a new environment variable
■ Enter ​JAVA_HOME​ for ​Name​ and
■ /usr/lib/jvm/java-1.8-openjdk​ for ​Value
In case of error and retry, use the ​Trust
SSH Host Key​ button if it has
changed
 ■

Figure 4. SSH Agent 1 Configuration

● You can see that your Node now appears in the Node list.
○ However it may have a little ​red cross:​ the agent is not yet started on this Node or is
experiencing troubles
● Browse to the Log Console of the Node to see what is happening:
○ Click on the Node ​ssh-agent-1​ on the list
○ On the left-menu, click on the ​*Log
○ Notice​ that the Agent is now online, by reading the log
Agent took a few seconds to start. This is why you may
T
have experienced the red cross
 ○

Figure 5. Jenkins SSH Agent 1 Online

● Browse back to the Node list
○ The Agent is fully online:

Figure 6. Jenkins SSH Agent 1 Online

● Repeat the steps above to add the second ssh agent
● Create a ​New Node​ using the left-menu:
○ Name: ​ssh-agent-2
○ Type: ​Permanent Agent
● Configure it with these properties:
○ # of executors: ​2
○ Remote root directory: ​/home/jenkins
○ No Labels
○ Usage: ​Utilize as much as possible
○ Launch method: ​Launch slave agents via SSH​ and configure it with previous properties
■ Host is ​jdk7-ssh-agent
■ Click the button ​Advanced​ to access the ​Port​ field, and set it to ​22
○ ​ on verifying Verification Strategy
Host Key Verification Strategy​: N
○ Under the Node Properties
■ Check the toggle for ​Environment Variables
 ■ Click ​Add​ to add a new environment variable
■ Enter ​JAVA_HOME​ for ​Name​ and
■ /usr/lib/jvm/java-1.8-openjdk​ for ​Value
● You should now see the two agents when you browse the Node list:

Figure 7. Jenkins SSH Agents

Reconfiguring Master

It is now time to reconfigure the Jenkins master, to stop executing builds and enforce security to the
Nodes:

● Browse the Jenkins ​Configure System​ page

○ Set the ​# of executors​ to ​0
It is a good practice to avoid having executors on the
T
Jenkins master. Any job running on the master can
access the JENKINS_HOME where it could leak
data (Credentials, etc.) or delete things
accidentally.

○
Add a ​label​ m
​ aster
○ Set the ​Usage​ to "Only build jobs with label restrictions matching this node"
 ○ Don’t forget to save the configuration !

Figure 8. Master Global Configuration

● Now, browse to the ​Configure Global Security​ page
○ Ensure that the toggle for ​Enable Agent → Master Access Control​ is checked
○ Save the configuration
● Go back to the Jenkins dashboard,
○ You should now see a total of 4 executors, two per agent.

Figure 9. Jenkins Executors Count

Reconfiguring Jobs

● Reconfigure ​pipeline-job​ created in previous exercise

● Direct URL: ​http://localhost:5000/jenkins/job/pipeline-job/
● Click​ Configure to make changes to this job
● Select the checkbox ​Restrict where this project can be run
● Enter ​ssh-agent-1​ to the ​Label Expression field
● Save the configuration

Figure 10. Restricting jobs to agent

Running Distributed Builds

● It is time to run our builds.

● Kick off a new build for`pipeline-job`
● Check ​Console Output​ to confirm that your build job ran on ​ssh-agent-1

Figure 11. Build job console output

● The Node Management system has a lot of admin utilities.
○ Browse to a Node page, using ​Manage Node​ ⇒ < Click a Node > ​ssh-agent-1​.
○ Check the ​Load statistics​ graph to view the node activity after a few builds
Figure 12. Example of Load Statistics

That’s all for this exercise!

SECURITY OVERVIEW

WHY DO SECURITY ?

● Protect your information

● Avoid executing malicious code

SECURE YOUR INFORMATION

● Your organization has ​information​ that is used to create value

● Information is valuable and you must ensure the following:
■ Confidentiality
■ Integrity
■ Availability
● Implementing security practices protects your information

AVOID EXECUTING MALICIOUS CODE

● Your Jenkins environment is also a fully distributed build system

■ Each network connection is a potential point of entry
● A bad actor could access your environment to launch a DDoS attack or a bot or other
mischief
■ The code that runs your builds can be perverted to run anything
● Users could inadvertently download malware from a web site if the site is not protected
 ■ This malware could then infect Jenkins in some way

WHAT IS SECURITY ?

● Security is the set of practices and tools to fight and prevent ​threats
● Major principles for security
■ Know the system:​ The more you understand about how your system works,
the more you are prepared to protect the integrity of your system
■ Least privilege:
○ Give people the privileges required to do their jobs but do not
give everyone
permission to do everything
○ Do not open ports that are not required for your work
■ Defense in Depth:​ Systems are layered. Put security on all layers.
■ Prevention is good. Detection is better:​ Monitor your Jenkins installation
constantly to detect signs of a security breech quickly

HOW TO DO SECURITY? LEAST PRIVILEGE

● Handling "Least privileges" concepts helps you manage the ​AAA​ concepts:
■ Authentication — Control who can access the system
■ Authorization — Control what each user can do on the system
■ Accounting — Monitor your system to ensure that only valid processes are
executing
HOW JENKINS EXECUTES A PIPELINE

● A simple overview of how Jenkins executes a Pipeline helps us understand the security
considerations:
■ By default, the Pipeline executes with the full privileges of the Jenkins
administrator
○ You can configure Jenkins to execute Pipelines with fewer
privileges
■ All of the Pipeline logic, the Groovy conditionals, loops, etc execute on the
master
■ Creates a workspace for each build that runs
○ Stores files created for that build
● The Pipeline calls steps that run on agents

WHAT AGENTS DO

● The Pipeline calls a series of steps, each of which is a script or command that does the real
work and mostly executes using an executor on an agent
■ Agent writes some files to the local node
■ Agent sends data back to the master
■ Agent can also request information from the master

HOW JENKINS CAN DO MISCHIEF

● Jenkins and the Pipelines it runs must be able to do almost anything

■ A malicious Pipeline could reconfigure the Jenkins instance, delete files,
launch malware attacks
● A trusted user could visit an infected web site and accidentally introduce malicious code into
the Jenkins instance

AGENTS AND SECURITY

● Agents that execute on the Jenkins master may be able to access Jenkins
configuration files and the workspaces of other builds
● An agent may be able to request information that belongs to other teams
or organizations from the master
■ This is not generally a concern for small Jenkins environments where
all agents are "trusted" to the same degree that the master is trusted
and all users have the same level of access to all configured processes
● An agent can write malicious code to the local disk so that the node is tainted
■ For maximum security, run all builds on ephemeral agents in the cloud
that are destroyed at the end of the build job

IMPLICATIONS OF BUILDING ON MASTER

● Do not run builds on the Jenkins master in production environments

■ A Pipeline that performs administrative tasks such as backup may run on the
master
○ Be sure to label the executor and only allow it to be used by jobs
that use that label
■ Build jobs can run on the master node for demonstration purposes
in non-production environments
● A build job that uses an agent that runs on the master node has access to Jenkins
master files and configuration, which poses a security risk

SECURE ACCESS TO EXTERNAL RESOURCES

● Pipelines may need trusted access to external resources such as a Nexus, Artifactory or
Elasticsearch database
● To avoid having to code usernames and passwords in each Pipeline and share the
information with everyone each time you change them, you can set up Credentials
■ Defined by the administrator
■ Called from the Pipeline code when accessing the external resource
● The administrator can easily change the password frequently to reduce the time that
"stolen" credential information can be used

AUTHENTICATION

AUTHENTICATION

● Authentication is the set of tools and procedures that ​identify​ a user with ​enough
confidence
■ When police check your ID, it is authentication
■ Using a login and a password is an authentication
■ Biometrics are also a type of authentication
 ■ 2FA, which stands for "Two Factor Authentication", is a stronger authentication
scheme

SET UP AUTHENTICATION

● Setting Security Realms tells Jenkins which "referential" to use for ​authentication
■ Security Realm is a dedicated ​database​ for user and passwords
● 4 kinds of ​Realms​ supported out of the box, covered below:
■ Jenkins User Database
■ Unix user/group Database
■ Servlet Container
■ External LDAP
● New Realms types can be added by other plugins
■ E.g. ActiveDirectory
● Users who are not authenticated are always bound to the ​"anonymous"​ special user
SECURITY REALM

● Defines the security implementation used to establish the identity of users

■ Jenkins supports many different authentication systems through security
realms
● Only one security realm can be active at a time
● Additional security implementations can be added with plugins
■ For example, the SAML 2.0 Single Sign On facility
● By default, users and groups come from the Jenkins internal user database
■ Smaller, more informal installations can use this internal database
■ Enterprise installations usually use a corporate service (LDAP, AD, or Unix),
which allows users to log into Jenkins with their usual username and
password.

CONFIGURE SECURITY REALM

● Choose the security realm used to authenticate users:

MANAGE USERS IN JENKINS USER DATABASE

● Use the "Manage Users" screen to create new users in the Jenkins User Database and to
view and modify information about existing users:
● It is not necessary to populate this database if you are using another security realm
 ■ Users can log in with their regular user ID and password

CREATE NEW USER

● Click the "Create User" link in the left frame to add a new user:

USER INFORMATION
 ● Fill in the form that is displayed

● Click the "Create User" button to create the new user

NEW USER IS CREATED

● Jenkins displays the revised list of all users in the Jenkins user database, which now
includes the new user you created:

● Click the flywheel to the right of each user to view or modify configuration information
for that user
MODIFY USER INFORMATION

● Click the flywheel to the right of each user to view details about that user. On that screen,
you can:
■ Provide a "Description" that tells other users more about who you are and
what you do
■ Supply SSH Public Keys
■ Modify the configured email address and password
1.8
User InformationFill in the form that is displayedClick the "Create User" button to create the new user
 MODIFY USER INFORMATION

● Click the flywheel to the right of each user to view details about that user. On that screen,
you can:
■ Provide a "Description" that tells other users more about who you are and
what you do
■ Supply SSH Public Keys
■ Modify the configured email address and password

AUTHORIZATION

WHAT IS AUTHORIZATION?

● Once a user is ​authenticated​, need to determine the actions this user should have the right
to do
● Authorization​ always occurs in the context of ​authentication
● Authorization grants access rights to:
■ Resource:​ Task, object and/or action to manipulate
■ Role:​ A set of rights grouped together by commodity
■ Requester:​ User or group that has roles assigned and wants to manipulate
resources

SECURITY MATRIX

● Use a Security Matrix to populate and visualize the authorizations:

■ Matrix-based security​ assigns global privileges to users and groups
■ Project-based Matrix Authorization Strategy​ assigns permissions
to users and groups based on the project(s) they access
■ Role strategy​ assigns permissions to specific roles;
○ It is available as an open source plugin but not discussed in this
class

CONFIGURE AUTHORIZATION

● Configure authorization — who is allowed to do what on Jenkins

● The default "Logged-in users can do anything" is seldom a good idea!

● One of the matrix based schemes is typically used
 MATRIX-BASED SCHEMES

● Choose one of the following schemes:

■ Matrix-Based Security (Per User)
○ Define actions allowed for each user or group globally
■ Project-based Matrix Authorization Strategy
○ Define actions allowed by project

MATRIX-BASED SECURITY PER USER

● The following screen is displayed when you choose "Matrix-based security":

ABOUT THE SECURITY MATRIX

● You can add users and/or groups to this matrix

● Hover the cursor over each "Privilege" to display information about what that privilege
entails
● Click the appropriate boxes to define the access that is allowed for each defined user or
group

ROWS

● Each row in the table represents a user or group

● Two rows are defined by default:
■ anonymous​ — permissions granted to all ​unauthenticated​ users
who access the Jenkins environment
■ authenticated​ — permissions granted to all ​authenticated​ users
who access the environment
● To add a user or group to this matrix, type the name into
the "User/group to add:" box + and click "Add"
● Permissions granted are additive

PERMISSION GROUPINGS

● Permissions are grouped:

■ Overall
■ Credentials
■ Agent
 ■ Job
■ Run
■ View
■ SCM
● Other plugins you install could add groupings
● The ​Matrix-based security​ wiki gives more detailed descriptions of many of these
permissions

OVERALL PERMISSIONS

● The first group of permissions in the matrix are the "Overall" permissions
■ Several of these are at least as powerful as "administer"
○ For historical reasons, "administer" can also perform the actions
associated
with these permissions
■ Users with overall permissions have the ability to do great harm to your
cluster,
either accidentally or deliberately
● Consider carefully how you allocate these permissions

UNDERSTAND OVERALL PERMISSIONS

● Administer​ - Make system-wide configuration changes

■ Perform highly sensitive operations that amount to full local system access
(within the scope granted by the underlying operating system)
■ In versions 5.25 (November 2018) and later, Overall/Administrator does ​not
imply the Overall/RunScripts, Overall/UploadPlugins, and Overall/Configure
UpdateCenter permissions
● Read​ - View almost all pages within Jenkins
● RunScripts​ - Run Groovy scripts using the Groovy console or Groovy CLI command
● UploadPlugins​ - Upload arbitrary plugins
■ Can execute arbitrary code in the context of any Jenkins internal user
● ConfigureUpdateCenter​ - Configure update sites and proxy settings
■ Can control the update site metadata and plugin files downloaded y the
Jenkins plugin manager
■ Can be used to execute arbitrary code

MORE ABOUT OVERALL/RUNSCRIPTS

● Users with Overall/RunScripts permissions can use the ​Script Console​ to run
a Groovy script in the Groovy Sandbox without getting administrator approval
■ Is useful for troubleshooting and diagnostics
■ Each method call, object construction, and field access is checked against
a whitelist of approved operations
■ If the script attempts to use an unapproved operation, it is killed
■ The unapproved operation is added to an approval queue which an
administrator
can review and, if appropriate, can add the operation to the whitelist

CAVEATS FOR OVERALL/RUNSCRIPTS PERMISSIONS

● Users with the "Overall/RunsScripts" permission can execute arbitrary code
in the context of any Jenkins internal user, including the internal SYSTEM user
● Jenkins Script Console​ includes valuable security guidelines

CONSIDERATIONS FOR SETTING PERMISSIONS

● Security principle of ​Least Privilege

● Trade-off between maximum security and avoiding bottlenecks in your CI/CD flow
■ The fewer people who have a permission, the less chance of a security breach
■ Ensure that someone is always available to do critical tasks such as take an
agent offline
○ A particular challenge with groups spread across different
timezones
■ Are tasks covered adequately if someone is sick or takes vacation?

DOING SOME WORK

● Grant appropriate permissions to "Anonymous Users" and "Authenticated Users"

● Add the other users that are in the "Authentication" section
■ Discuss what sorts of permissions each needs
■ Skip Credentials for now; we’ll do that in the Credentials section that follows
■ Set up those permissions
● Add a user who is not in the Jenkins User Database
■ Discuss what happens
● Show how to define a group
■ Add a group and set permissions

PROJECT-BASED MATRIX AUTHORIZATION STRATEGY

● Use the project-based matrix to define authorization by project rather than users:

IMPLEMENTING PROJECT-BASED MATRIX AUTHORIZATION STRATEGY

● Additional access control lists (ACLs) can be defined separately for each project
■ Users and groups can be granted access only to the specified projects
● ACLs defined for project-based matrix authorization are additive
■ Access grants defined here are combined with project-specific ACLs
■ Managing inheritance options is complicated
 IMPLEMENTING PROJECT-BASED MATRIX AUTHORIZATION STRATEGY

● Additional access control lists (ACLs) can be defined separately for each project
■ Users and groups can be granted access only to the specified projects
● ACLs defined for project-based matrix authorization are additive
■ Access grants defined here are combined with project-specific ACLs
■ Managing inheritance options is complicated

ACCOUNTING

ACCOUNTING

● Occurs in the context of a user who is authenticated and authorized.

● Measures resources used or consumed by the user during access.
■ This can be amount of data, compute resources or system time.
● Enforces limits when they are defined to protect system.
● Related to system measurement, capacity planning and feedback loops

LOGS ON THE SYSTEM

● When running jenkins.war manually with java -jar jenkins.war,

all logging information is output to stdout by default.

LOGS ON LINUX SYSTEMS

● By default, logs are available in /var/log/jenkins/jenkins.log,

■ Log location can be customized
○ In /etc/default/jenkins (for *.deb)
○ In /etc/sysconfig/jenkins (for *.rpm)

LOGS ON WINDOWS SYSTEMS

● By default, logs are available in:

■ %JENKINS_HOME%/jenkins.out
■ %JENKINS_HOME%/jenkins.err
● The log location is customized in %JENKINS_HOME%/jenkins.xml

LOGS ON DOCKER

● If you run Jenkins inside Docker as a detached container:

■ Use docker logs containerId to collect Jenkins logs

LOGS ON JENKINS GUI

● Go to ​Manage Jenkins → System Log

● Create a custom log recorder
■ Helps you group relevant logs together while filtering out the noise
 MONITORING SERVER LOAD

● Jenkins provides built-in monitoring of server activity

● Go to ​Manage Jenkins → Load Statistics
■ Displays a graph of the server load time for the master node
■ Graph keeps track of four metrics:
○ Number of online executors
○ Number of busy executors
○ Number of available executors
○ Queue length

MONITORING

● Many open source plugins are available for monitoring

● Some of the popular ones are:
■ Monitoring
■ Disk Usage
■ Build Monitor

MONITORING PLUGIN

● The Monitoring plugin provides monitoring of Jenkins with JavaMelody

■ Charts of CPU, memory, system load average, HTTP response time
■ Details of HTTP sessions, errors and logs, actions for GC, heap dump, invalid
session(s)
● The Disk Usage plugin shows project-wide details for all jobs and all workspaces
■ It also displays Disk Usage Trend
● The Build Monitor Plugin provides a detailed view of the status of selected Jenkins jobs
■ It also shows the names of people who might be responsible for "breaking the
build".

AUDITING

● Many open source plugins are available for Auditing

● Some of the popular ones are:
■ Audit Trail
■ Job Configuration History

AUDIT TRAIL PLUGIN

● Keeps a log of users who performed particular Jenkins operations, such as configuring jobs
● This plugin adds an Audit Trail section in the main Jenkins configuration page
● Many configuration options are supported
■ Save output audit logs in rolling files
■ Send audit logs to a Syslog server
 ■ Output audit logs in stdout or stderr. Primarily intended for debugging
purposes

VIEW JOB CONFIGURATION HISTORY PLUGIN

● Saves a copy of the configuration file of a job ​(config.xml)​ for each change made and of the
system configuration
● Provides an overview page of all changes
● The overview page only lists changes made to the system configuration (for performance
reasons)
■ Use links to view either all job configuration histories or just the deleted jobs or
all kinds of configuration history entries together
○ This may take some time to load, If your instance is running
many jobs

WHAT DID WE LEARN

● Security on Jenkins is a vast topic, but many resources are available

● Start by enabling authentication and authorization
■ Numerous authentication realms, extensible by plugins
■ Few authorization strategies, but powerful ones
■ Many threats can be leveraged by accounting and Jenkins OOB settings

GLOBAL SECURITY SETTINGS

ABOUT SECURITY SETTINGS

● Global Security Settings close off intrusion paths for the Jenkins instance
● By default, Jenkins is installed with restrictive settings
■ This reduces the attack surface of the Jenkins cluster
■ You may need to relax these settings to run your jobs
● Recommend that you reduce protection only when necessary
ACCESS CONTROL FOR BUILDS

● By default, all Pipelines/jobs run with administrator privileges

■ Any Pipeline can reconfigure Jenkins, delete/modify files, etc
● Set "Access Control for Builds" to limit the permissions used for builds:

OTHER GLOBAL SECURITY SETTINGS

● The "Global Security Settings" screen also includes fields to configure or enable various
features that have security ramifications
● By default, Jenkins is installed in "locked-down" mode
■ Features that can have security ramifications are turned off
 ■ You may need to enable some of these features to get the functionality you
require
■ In most cases, plugins and practices are available to reduce the security
vulnerability of these features

CSRF (CROSS-SITE REQUEST FORGERY)

● CSRF is an exploit that enables an unauthorized third party to perform requests

against a web application by impersonating another, authenticated user
● In a Jenkins environment, a CSRF attack could allow a malicious actor to delete projects,
alter builds, or modify the system configuration of the Jenkins instance
● Also sometimes called XSRF
● See the OWASP Foundation’s ​Cross-Site Request Forgery​ document for more information
about CSRF exploits

CSRF PROTECTION

● Jenkins enables CSRF Protection by default; we strongly recommend that you keep it
enabled:

● When enabled, Jenkins checks for a CSRF token (or "crumb") on any request that might
change data in the Jenkins environment
■ This includes any form submission and calls to the remote API, including those
that use "Basic" authentication

ISSUES WITH CSRF PROTECTION

● CSRF protection ​might​ result in challenges for your Jenkins builds:

■ Accessing Jenkins through a poorly-configured reverse proxy may result in the
CSRF HTTP header being stripped from requests, causing protected actions
to fail
○ See ​Jenkins behind an NGINX reverse proxy​ for information
about correcting the proxy configuration.
○ Clicking "Enable proxy compatibility" may allow you to run your
build without disabling CSRF protection.
■ Outdated plugins that have not been tested with CSRF protection enabled
may not function properly
■ In Jenkins 2.95 and earlier, CSRF protection made use of the remote API
difficult. Since 2.96, using the remote API with username/API token
authentication no longer requires that a valid CSRF protection crumb be
provided.
○ You must use an API token rather than a password
● It may be necessary to disable CSRF protection if any of these interfere with your builds

JNLP (JAVA NETWORK LAUNCH PROTOCOL)

● JNLP (Java Network Launch Protocol) can be used to launch an application on a client
desktop by using resources that are hosted on a remote web server
■ Used primarily by Windows based agents
■ These agents could instead be run using SSH
● Jenkins uses a TCP port to communicate with agents launched via JNLP.
■ By default, this port is chosen randomly to avoid collisions

CONFIGURING JNLP ACCESS

● Use the "Agents" field to configure JNLP access

● Recommend leaving this disabled unless you have JNLP agents

■ If your Windows agents use SSH rather than JNLP, leave the TCP port for
JNLP agents disabled
■ You can then configure your Firewall appropriately
● To enable JNLP access, click either the "Fixed" or "Random" buttons
■ The more secure option is to choose "Fixed" and define a TCP port to use for
JNLP

AGENT-TO-MASTER ACCESS CONTROL

● Agent-to-Master Access Control filters the commands that agents can send to the Jenkins
master
■ It is enabled by default:

● Strongly recommend​ that this protection be enabled

■ Especially for a large enterprise deployment that takes agents from other
teams
■ Click on the link to tweak the rules if necessary

TWEAK THE AGENT-TO-MASTER RULES

● If builds are failing because of the Agent-to-Master Access Control:

■ First, be sure that you have upgraded to the latest version of all plugins
○ Older versions of plugins may not work correctly with this feature
■ Whitelist the specific commands that you need
 ■ Add allow/deny rules to refine the list of files on the master and the type of
access that is allowed for the agent.
■ See the ​Slave To Master Access Control​ documentation for more information.

MARKUP FORMATTING

● Jenkins allows user input in some text areas and configuration fields
■ Examples include job descriptions, user descriptions, view descriptions and
build descriptions
● HTML formatting in these text areas can lead to users inadvertently (or maliciously) inserting
unsafe HTML and/or JavaScript
■ This could be used for Cross-site Scripting (XSS) attacks on web pages
■ XSS enables attackers to inject client-side scripts into web pages that are
viewed by others

CONTROL HTML FORMATTING

● Use the "Markup Formatter" field to control how HTML code is rendered on your Jenkins
instance:

● By default, this is set to "Plain text", which causes all HTML formatting to be ignored. This is
the safest option but users are deprived of the aesthetic advantages of formatted text
● You can choose the "Safe HTML" option, which allows you to use a subset of HTML markup
such as hyperlinks, bold and italic characters.
● Other plugins can be installed to support additional options for handling HTML markup
■ If such plugins are installed, they are included on the dropdown menu for this
field

CONTENT SECURITY POLICY (CSP)

The Content Security Policy (CSP) header is applied to static files served by Jenkins

Set to a very restrictive default set of permissions to protect Jenkins users from malicious
HTML and JavaScript files in workspaces, /userContent or archived artifacts

May need to relax the default rules for some popular, useful plugins

Details in Configuring Content Security Policy

MANAGING CREDENTIALS

ABOUT CREDENTIALS

● Credentials are used to get trusted access to other resources without having to share the
actual passwords, et cetera, with all users.
 ■ Jenkins itself uses credentials to secure internal communication
■ Administrators can define credentials that Pipelines use to access secured
resources
● The canonical documentation about credentials is ​Credentials API Plugin documentation​,
written and maintained by Stephen Connolly, the author of the Credentials plugin.

EXAMPLES OF CREDENTIAL USE

● Jenkins uses credentials to get trusted access to resources such as:

■ SCM services for getting and pushing code
■ Remote Secured services (such as secured LDAP plugins or the Elasticsearch
analytics engine)
● Pipelines use Credentials to access resources such as
■ Artifact servers (such as Nexus and Artifactory) for fetching dependencies (or
deploying new ones)
■ Secured environments to which it is deploying code

WHY USE CREDENTIALS

● Using Credentials:
■ Provides centralized credentials management
■ Treats credentials as ​Jenkins Resources
■ Provides an API consumable by plugins and other Jenkins Resources
■ Plugins may add additional credential types:
○ GitHub/Google/Bitbucket OAuth
○ "SSH agents" plugin
● Authorization matrix defines who can create, delete, update, and view credentials

HOW CREDENTIALS ARE RELATED TO RESOURCES

● Implemented through plugins for a type of external resource

■ The plugin defines the credentials that are supported for that resource
■ Examples include the plugins to access SCMs and artifact servers like Nexus
and Artifactory
● Each plugin defines the credential types it supports and may not support all credential types
■ For example, the Git Plugin (and Git Client Plugin) can only use
username/password credentials and private key credentials
■ This means that secret text, certificates, or other credential types are not
proposed when using the Git plugin

HOW CREDENTIALS ARE USED IN A PIPELINE

● Pipeline needs to deploy results to an external resource

such as Nexus, Artifactory, or Elasticsearch.
● Access to that external resource is protected by a username and password
● Sharing that username/password with 20 or 100 users is undesireable
■ It is a security risk to have multiple people share a password
 ■ If the password is changed, all those people must be informed, which is a
hassle
● Instead, the administrator can define a credential that knows the username and password
for the external resource
■ Defined under "Configure → Credentials" on the Jenkins dashboard
● Pipelines can then use either the environment or withCredentials directive to supply the
username and password and access the secured resource
■ Users who run the Pipeline do not know what the password is

LOWEST LEVEL PRINCIPLE

● Limiting the number of projects that can access a Credential enhances its security
● Credentials should be defined at the lowest possible level
■ Credentials defined for the master are available to all Pipelines run by that
master
■ Credentials defined for a Folder are only available to Pipelines run from that
Folder.
● The screens used to add and manage credentials for masters​HOW CREDENTIALS ARE
STORED
● Credentials are stored in an encrypted form on the Jenkins Master
■ Credentials are encrypted using a key derived from the master key
■ All credentials that you create are stored as type Secret
○ For users who have appropriate (configure) permission, the
secret fields are round-tripped in their encrypted form
○ For other users, only a placeholder is used
● Keys are tied to and encrypted by the instance ID
■ This means they can not be migrated or copied to a new instance
● and folders are identical except for their location

PROTECTING SECRETS

● Keys to decrypt secrets are stored in the ​$JENKINS_HOME/secrets/​ directory

■ This directory has restrictive Linux file system permissions and should be
excluded from backups
■ Keys should not be stored in the SCM where their contents could be breached
■ A disk crash or data corruption for your application could cause the loss of all
the encrypted information for your instance
■ You should ​store the secret in some other location​ with restricted access; it
is small and never changes after it is created; you can manually re-add it when
restoring from backup

ADD CREDENTIALS

● Credentials can be added by any authorized user who has the ​Credentials → Create
permission
● To create a credential, fill in the following form under "Configure" on the Web UI
 NOTES ABOUT ADDING CREDENTIALS (1/2)

● Assign an ID that can be used to access the credential. The Pipeline uses this ID to apply
these credentials
■ We recommend that you use a string that is reasonably easy to remember and
type
■ If you do not assign an ID, Jenkins assigns an ID that is a GUID
■ IDs should be 40 characters or less
■ Only use alpha-numerical characters plus the separator characters - , _ .
■ IDs are case sensitive

NOTES ABOUT ADDING CREDENTIALS (2/2)

● Scope​ defines the places where this credential can be used

■ System:​ Credential is available only to the object with which the credential is
associated. Examples include email authorization or node connection, where
the Jenkins instance itself is using the credential
■ Global:​ Credential is available to the object with which the credential is
associated plus ​all objects that are children of that object.​ Most Credentials
that are called in Pipelines are defined as ​Global.​

CREDENTIALS TYPE

● The Jenkins Credentials Type defines the behavior. It must match what the target object
expects:
■ Secret Text​ type:
○ MYVARNAME contains the path of the file that contains the
Secret Text
■ Standard username and password​ type:
 ○ MYVARNAME is set to <username>:<password> (conjoined
name and password string)
○ MYVARNAME_USR is set to just the username
○ MYVARNAME_PSW is set to just the password

MORE ABOUT CREDENTIAL TYPES

● Each credential type supported for the environment directive:

■ Must be supported in the credentials binding plugin
■ Must have a special handler for Declarative Pipeline
● Standard Declarative syntax currently supports the following credentials:
■ Username and Password (plugin type
StandardUsernamePasswordCredentials)
■ Secret file (FileCredentials)
■ Secret text (StringCredentials)
● Use the sshagent step for SSH credentials, which withCredentials does not support

EXAMPLE CODE

...
stage(​'Deploy Reports'​)
steps {
...
withCredentials(​bindings:​ [string(​credentialsId:​ ​'elastic-access-key'​, ​variable:
'ELASTIC_ACCESS_KEY'​)]) {
​// Environment Variable available in the remote shell
sh ​'env | grep ELASTIC_ACCESS_KEY'
sh ​'echo ${ELASTIC_ACCESS_KEY} > secret-file.txt'

​// Groovy Variable

sh ​"echo ${ELASTIC_ACCESS_KEY} > secret-file.txt"
}
...
}

NOTES ABOUT EXAMPLE CODE

● Credentials are implemented in a Pipeline using the withCredentials() method

● Uses the credential that is defined with the ID of elastic-access-key
● withCredentials binds that to a local Pipeline variable called ELASTIC_ACCESS_KEY
● The deploy-token step uses the ELASTIC_ACCESS_KEY local Pipeline variable to establish
the credentials that it needs
● Anyone who can run this Pipeline gets the necessary credentials to run deploy-token
FOR FURTHER READING

● Using Credentials
● Storing Secrets

SECURITY BEST PRACTICES

 JENKINS SECURITY TEAM

● The Jenkins security team works constantly to improve the general security of Jenkins
■ See ​Jenkins Security
● Security advisories and updates are released as needed
● Notifications are posted to the jenkinsci-advisories Google group
■ We recommend that you subscribe to this group to receive timely notifications
■ When an applicable security update is released, install it as soon as you can!

GOING FURTHER

● Some recommended readings on this subject:

■ Quick and Simple Security
■ Script Security Plugin
■ Matrix-based security
■ Best Practices for Setting up Jenkins Auditing and Compliance
■ Audit Trail Plugin
■ JobConfigHistory Plugin
■ Job Restrictions Plugin
■ Disable security
■ PAM Authentication Plugin

LAB EXERCISE

Enable and Configure Jenkins Security

Enable and Configure Jenkins Security

The goal of this exercise is to set up a basic security model on your Jenkins instance and manipulate
Credentials.

This Jenkins instance is not wide open due to security concerns over the
NOTE
internet. Basic Authentication using the Jenkins embedded user database has
been enabled. You can use ​butler:butler​ credentials for logging into
http://localhost:5000/jenkins​.

Internal LDAP overview

The Lab instance contains a Directory Service (OpenLDAP) that we will use as the authentication source
for Jenkins.
 OpenLDAP is not the only Directory Service system that Jenkins can use.
TIP

This LDAP instance comes ​pre-populated​ with some data:

● It contains:
○ 3 users: butler, colleague and contributor
○ 3 groups: jenkins-admin, jenkins-developer, jenkins-users

This LDAP is ​only​ for demonstration purposes:

NOTE

● No LDAP*S* (TLS) support

● Anonymous binding to the LDAP server
● No Replicas
● It ​tries​ to follow ​RFC2798

Enabling Authentication

First, we will enable Security and Authentication on Jenkins, based on the LDAP.

● Start by browsing to the ​Configure Global Security​ page of ​Jenkins​:

○ Make sure the toggle for ​Enable security​ is checked
○ Select LDAP as ​Security Realm
■ Use this value for the ​Server​ field: ​ldap://ldap-service:389
○ Click on the ​Advanced Server Configuration…​​ button in the LDAP configuration
section, and set the following settings:
■ root DN:​ ​dc=ldap,dc=cloudbees,dc=training,dc=local
■ ​ u=people
User search base:​ o
■ User search filter:​ ​uid={0}
■ Group search base:​ ​ou=groups
■ Group search filter:​ ​cn={0}
■ Group membership:​ Search for LDAP groups containing user
■ Group membership filter:​ ​member={0}
■ Set ​Manager Password​ empty if your web browser inserted a random password
■ Display Name LDAP attribute:​ displayname
 ■ Email Address LDAP attribute:​ mail

Figure 1. LDAP Security Realm configuration

○ Before saving, we want to test the LDAP settings:
■ Click on the button ​Test LDAP settings​ in the LDAP configuration section
■ A panel named ​Test LDAP settings​ should open. Set both ​User​ and ​Password
field to the value ​butler​:

Figure 2. Test LDAP settings

 ■ Click on the button ​Test​ to validate your settings. You should see the following
output (if not the case, check your settings based on the error):

Figure 3. LDAP settings validated

● You can now ​Save​ this configuration to enable Authentication and go back to the ​Jenkins
dashboard​:
○ Login to Jenkins as butler (password is same as username):
● You are now logged as the ​Butler​ user, authentication works.
○ Verify that the buttons in the upper right of the screen print your "displayname" and
logout​ link.
○ Browse to your user configuration page to check your settings:
Some fields have been populated using LDAP data.
T
Updating ​only​ changes it inside the Jenkins
database, not in the LDAP Directory Service.
 ○

Figure 4. Log In as Butler

Trying to set up Authorization policy

● Log out from "Butler" user.

Current authorization policy for lab instance is set to ​Logged-in
NOTE
users can do anything​.

●
If you have an error message (in red) on the ​LDAP​ section, click
IMPORTA
on the ​Advanced Server Configuration…​​ button, and set
NT
the ​Manager DN​ and M
​ anager password​ fields to empty
value

These fields can be populated if you are using the

"autosave/autofill" functionality of your web-browser.

Auto-filling forms on the web browser without requiring any action

from the user is not recommended.
 ●
Time to login again as butler
○ The Jenkins Management page is now allowed.
● It is fine, but this policy is only for a small set of users in a trusted environment.
○ We want to separate rights based on group membership so that Jenkins can be scaled
for the organization’s growth.
● Browse again to the ​Global Security Configuration:​ page.
○ Enable the ​Project-based Matrix Authorization Strategy​ policy:

Figure 5. Enabling Project-Matrix Policy with default settings

● Save the configuration using the default settings
● Logout and log back in as ​butler
 ●
TIP
If you get locked out of Jenkins:
○ Login to your Jenkins server
○ Browse to JENKINS_HOME directory
○ Verify the content of config.xml file
○ Use the ​sed​ command shown below to disable
security
○ Verify ​again​ the config.xml
○ Restart Jenkins

# Spawn a shell in the Jenkins machine

cloudbees-devbox $ {spawn-docker-jenkins-shell}

# Browse to the JENKINS_HOME directory

bash-4.3$ cd /var/jenkins_home

# Show content of the config.xml file

bash-4.3$ cat config.xml
...

# Show status of the "Enable Security" setting

bash-4.3$ grep useSecurity config.xml
...

# Show status of the "Authorization Strategy" setting

bash-4.3$ grep authorizationStrategy config.xml
...

# Set the XML attribute useSecurity to false in config.xml

bash-4.3$ sed -i \
's#<useSecurity>true#<useSecurity>false#g' \
config.xml

# Disable security by deleting the lines containing "authorizationStrategy"

# in the config.xml file
bash-4.3$ sed -i \
'/authorizationStrategy/d' \
config.xml

# Validate our changes

bash-4.3$ cat config.xml
...
 # Exit the Jenkins machine
bash-4.3$ exit
exit

# Restart Jenkins to apply the configuration change

cloudbees-devbox $ docker restart jenkins

We want to apply the following rights:

IMPORTANT

● Non authenticated users (Jenkins system’s ​anonymous​ user):

○ No access to Jenkins
● Authenticated users, members of ​jenkins-users​ LDAP group:
○ Can view Jenkins dashboard and non-private jobs
● Authenticated users, members of ​jenkins-developers​ LDAP group:
○ Can Launch, Create/View/Update/Delete (CRUD),
Configure, Move jobs
○ Can CRUD Credentials
● Authenticated users, members of ​jenkins-admin​ LDAP group:
○ Can administer Jenkins

●
Select the ​Project-based Matrix Authorization Strategy​ policy
○ Use the ​User/group to add​ field add your 3 groups
 A tiny icon will appear to tell you that Jenkins has a
T
mapping to the configured LDAP’s groups:

○
Select the right permissions using the permissions table below
Drag your cursor over each permission to display a
T
description popup. Read ​Job:Discover​,
Job:Cancel​ and ​Run:Update​ as mandatory
examples.

Matrix Group/User Members Permissions (section, checkbox)

anonymous None (unauthenticated users) None None

jenkins-users butler, colleague, contributor Overall Read

Job Read

View Read
jenkins-developers butler, colleague Overall Read

Credentials Create

Credentials Delete

Credentials Update

Credentials View

Job Build

Job Cancel

Job Configure

Job Create

Job Delete

Job Discover

Job Read

Job Workspace

Run Delete

Run Replay

Run Update

View Configure

View Create

View Delete

View Read
jenkins-admin butler Overall Administer

●
Check your settings with the following screenshot, and ​Save​ the configuration:

Figure 6. Project Matrix Settings

● Once saved, test this new configuration by:
○ Accessing as non-authenticated user: not allowed to do anything.
○ Logging in as butler still lets you have rights
○ Logging in as ​colleague​, ​contributor​ and checking rights:

Figure 7. Expected dashboard for Contributor

Figure 8. Expected dashboard for Colleague

Accounting: Privacy and Credentials

The goal of this exercise is to create an administrator ​private​ job. It will be hidden for ​ALL​ users excepts
administrators. Credentials are also used to access the related ​private​ repository.

● Start by logging into the ​GitServer​ as butler

● From the butler "homepage", create a new repository with these settings:
○ Owner:​ ​butler
○ Repository Name:​ ​admin-scripts
○ Visibility:​ Select "This repository is Private"
○ ​ set of admin scripts that must remain private
Description:​ A
○ Use the ​default​ values for the other options:

Figure 9. New Private Repository in Gitea

 ● Once created, verify that it is ​empty​ and ​private​, and copy the HTTP URL:

Figure 10. A private and empty Repository in Gitea

● Open the ​WebIDE​ in a new tab, and create a new project based on this repository:
○ ​ dmin-scripts
Project Name:​ a
○ ​ dmin-scripts
Folder Name:​ a
○ …​from Git Repo​ ​›​ ​Git Repository​: h
​ ttp://localhost:5000/gitserver/butler/admin-scripts
○ It will be cloned ​empty
● In the ​WebIDE​, create a new file named ​run-admin-task.sh​ at the root, with the following content:

#!/bin/bash

set​ -e
set​ -u

● echo​ ​"Some secret admin task to run there"

● Add it to git tracking, commit and push (providing credentials)
Use butler, for the Git authentication when pushing.
TIP
●

Figure 11. New script in private repository

Figure 12. Git-tracking the new script

Figure 13. Git-committing the new script

● Go back to Jenkins, log in as butler
● Create a new ​Freestyle​ job named ​admin-tasks​ and configure it as following:
○ Select the checkbox ​Enable project-based security
■ Select the ​Do not inherit permission grants from the other ACLs​ checkbox
 ■ Ensure that ​only​ the ​jenkins-admin​ group has all rights:

Figure 14. Correct rights for admin project

● Build​:
○ Execute a shell script​ with this content:
○ bash -x run-admin-task.sh
● SCM:​ Git repository, from the previously created ​private​ repository
○ Notice the ​Red​ error message that should appear: username and password are required
to access this repository

Figure 15. No access to this repository without authentication

● To add Credentials, click the ​Add​ button
○ Select ​Jenkins​ in the drop-down
○ Generate the Credential with these properties:
■ Domain:​ Global credentials (unrestricted)
■ Kind:​ Username with password
■ Username:​ butler
■ Password:​ butler
■ ID:​ butler-gogs-creds
 ■ Description:​ Butler’s credentials for Gitea

Figure 16. Adding a Credentials for the private Repository

● Make the ​Red​ message go away by selecting the ​newly​ created ​Credential
● Save the job and build it
○ It should end in success, thanks to this credential
● Finally, log-in as ​colleague​ in Jenkins:
○ The job should not be visible

That’s all for this exercise !

Folders

ORGANIZE BUILDS

WHY ORGANIZE YOUR BUILDS

● As the number of build projects grows, you’ll need some organization

FOLDERS

● Use ​Folders​ to organize a large number of builds around taxonomies such as projects or
departments
● Folders can be nested like file system folders
■ Folder 1
○ Folder 2
○ Folder 3 … Folder n
● Creates namespace
 ■ "​build"​ job in Folder ​A​ is different than "​build"​ job in Folder ​B

ABOUT FOLDERS
● One can create a new folder and move existing Pipelines into that folder
● You can clone an existing folder, leaving the children intact
■ Useful to seed new projects
● Folders can define properties that are only visible to jobs inside them
■ For example, assign Credentials to a Folder
○ Only Pipelines in that Folder can use those Credentials
○ This improves security
■ Simplifies branch and Pipeline management

CREATE A FOLDER

● Click on ​New Item​, then click on ​Folder​ and type the name for the folder in the box:

CLONE A FOLDER
● To clone an existing folder, fill in the "Copy from" box at the bottom of the screen:

CONFIGURE A FOLDER

● You can (optionally) supply a "Display Name" for this folder and a description:

● Other configuration options are available here but are beyond the scope of this class

MOVE A JOB TO A FOLDER

● Go back to the Jenkins dashboard and click on a Pipeline project you want to move to this
folder; click "Move" in the left frame:
●

GOING FURTHER

● Some recommended readings on this subject:

■ CloudBees Folders Plugin
 ■ User’s Guide for the ​Folders Plugin

LAB EXERCISE

Organize your jobs with Folders and Views

Organize your jobs with Folders and Views

The goal of this exercise is to use Folders on the Jenkins instance.

Your group of developers will be split in 2 teams: ​TeamA​ and ​TeamB​. They will work on the same
application, but adapt it for different customers.

Sadly, the staging environment is shared. We suppose that project

IMPORTANT
managers of TeamA and TeamB are well synchronized :)

Creating Folder

● Login as user butler

● Create a new folder:
○ Use the ​New Item​ button on the left-menu
○ Name:​ ​TeamA
○ Kind: ​Folder
● Take time to review the Folder settings:
○ Enable Project security to only allow access to members of the ​jenkins-admin​ and
jenkins-developers​ groups:
Figure 1. New Folder for Team A
● The newly created folder is empty:

Figure 2. Team A Folder is empty

● Browse to main Dashboard (use the ​Up​ button in left-menu)
● For ​pipeline-job​:
○ Go the the Job page
○ Use the ​Move​ button (on the left-menu)
○ Move to the ​TeamA​ folder

Figure 3. Move to the Team A folder

 Copy Folder

We now want to initialize the Team B folder, with the same settings as Team A.

● Create a ​New Item​:

○ Name:​ ​TeamB
○ Type: ​Copy existing Item
○ Copy from​: ​TeamA
● Review the configuration: same as Team A !
● You now have a ​TeamB​ folder with a ​copy​ of the original content
○ Jobs have no ​Build​ button (yet) and their status is ​Grey
You should review the Job configurations ​before
IM
you try to launch a build from them.
P
O
R
T
A
N
T

Figure 4. TeamB folder with newly created jobs

● Review and ​Save​ the new job configurations and kick off builds of the ​pipeline-job​ for ​both
folders
○ Observe that they behave as independent jobs
 Notice the pipeline-job in TeamB folder is greyed out. This is because any job
TIP
you copy is disabled or rather, not buildable by default. You usually don’t
recognize it because the form opens and you change things and upon saving the
job becomes buildable. You can disable the project and enable it back to see the
Build Now option on the left hand side menu

Folders on Filesystem

Folders provide ​namespacing​ for Jenkins objects. As you have seen, jobs have same names, but can
still co-exist within their own folders.

● How is it behaving in the Jenkins home folder ?

● Open a Command Line on ​Devbox
● Spawn a ​bash​ shell on the Jenkins machine using ​docker exec (…​)
● Browse to the ​${JENKINS_HOME}/jobs​ directory and check the file-system tree:

cloudbees-devbox $ docker ​exec​ -ti jenkins /bin/bash

bash-4.3$ ​cd​ /var/jenkins_home/​jobs​/
bash-4.3$ ls -1
# 1 Filesystem Dir for EACH TeamX object
TeamA
TeamB
admin-tasks
bash-4.3$ ls -1 TeamA
# Each contains a config file and a sub `jobs` folder
config.xml
​jobs
bash-4.3$ ls -1 TeamA/​jobs​/
# Jobs are stored inside the sub-folder `jobs`
pipeline-job
bash-4.3$ diff -u TeamA/config.xml TeamB/config.xml
● # No differences at configuration level or the one you may have introduced

That’s all for this exercise!

MONITOR JENKINS

WHY MONITOR?
 ● Ensure that the Jenkins instance is healthy
● Must understand how resources are being used before one can take actions to improve the
health of the instance

LOGS

● Define and differentiate:

■ Build Logs
■ System Logs
METRICS

● Out-of-the-box metrics from inside Jenkins

● Other Metrics types:
■ build time
■ agent related
METRIC AGGREGATORS

● Metrics exist but must be collected

● A Metric Aggregator service is recommended
■ Not covered in this class

BACKUP AND RESTORE

BACKUPS ARE IMPORTANT!

● Disaster recovery
● Recover an older configuration
■ An accidental configuration change may not be discovered until long after it
was made
● Recover a file that is corrupted or deleted accidentally
● Your backup strategy should include validation of each backup
■ You do not want to learn that your backup is no good when you need it!

BACKUP TOOLS: FILESYSTEM SNAPSHOTS

● Filesystem snapshots provide maximum consistency for backups

■ Run faster than live backups, reducing the possibility of copying different data
at different time points
■ Supported by Linux Logical Volume Manager (LVM), Solaris ZFS (which also
supports incremental backups), and other file system architecture
■ Some separate storage devices also let you create snapshots at the storage
level

BACKUP TOOLS: PLUGINS

● Several plugins are available for backup

■ Go to ​Manage Jenkins → Manage Plugins → Available​ and search for
"backup"
■ None of the open source plugins are being maintained
● You can try these plugins but you may have problems with them
 BACKUP TOOLS: WRITE A SHELL SCRIPT

● You can write your own shell script that copies the appropriate files and directories to a
backup location
● Use cron to schedule when the backup script runs
● Create a directory such as ​/mnt/backup​; be sure that you have write permissions
■ Consider creating ​/mnt/backups​ as a separate filesystem with its own mount
point
○ A subdirectory to ​/var​ is another option; you may need to use
sudo​ for the restore operation
○ Backing up to ​/tmp​ is not advised because ​/tmp​ may be cleaned
on reboot
■ Create a unique identifier for each backup (use a timestamp, for example) to
ensure that today’s backup does not overwrite yesterday’s backup
● Writing files to a local file system is the fastest way to take the backup
■ Consider copying the completed backup to a rem​BACK UP
$JENKINS_HOME
● All files are located under the ​$JENKINS_HOME​ directory
● You can back up the entire ​$JENKINS_HOME​ directory
■ This preserves everything
■ To restore the system, just copy the entire backup to the new system
■ Includes a number of files that do not really need to be backed up
● Selecting specific directories and files to back up yields smaller backups
■ May require a greater effort to restore a system
● You may want to back up different directories on different schedules
■ ote backup server or device for long term storage

BACKUP CONFIGURATION FILES

● Configuration files are stored directly in the ​$JENKINS_HOME​ directory

■ ./config.xml​ is the main Jenkins configuration files
■ Other configuration files have the ​.xml​ suffix
● Specify ​$JENKINS_HOME/*.xml​ to back up all configuration files
● Configuration files can also be stored in an SCM repository
■ This keeps copies of all previous versions of each file that can be retrieved
using standard SCM facilities

BACK UP ./JOBS SUBDIRECTORY

● $JENKINS_HOME/jobs​ contains information related to all the jobs you create in Jenkins
■ ./builds​ — Contains build records
■ ./builds/archive​ — Contains archived artifacts
○ Back this up if it is important to retain these artifacts long term
○ These can be very large and may make your backups very large
■ ./workspace​ — Contains files checked out from the SCM
○ Usually not necessary to back them up; you can perform a clean
checkout after restoring the system
 ■ ./plugins/*.hpi​ — Plugin packages with specific versions used on your
system
■ ./plugins/*.jpi​ — Plugin packages with specific versions used on your system

WHAT DOES NOT NEED TO BE BACKED UP

● The following files and directories do not usually need to be backed up:
■ ./war​ — Exploded ​war​ file
○ To restore a system, download the latest ​war​ file
■ ./cache​ — Downloaded tools
○ To restore a system, download the current version of the tools
■ ./tools​ — Extracted tools
○ To restore a system, extract the tools again
■ ./plugins/xxx​ — Subdirectories of installed plugins
○ Will be automatically populated on next restart
● Alternatively, you can make an infrequent backup of just these files if you want to be able to
easily restore the same versions of the system and all downloaded tools

SIMPLE VALIDATION OF A BACKUP

● A simple way to validate a full backup is to restore it to a temporary location:

■ Create a directory for the test validation (such as ​/mnt/backup-test​) and
restore the backup to that directory
■ Set $JENKINS_HOME to point to this directory, specifying a random HTTP
port so you do not collide with the real Jenkins instance:
■ export​ JENKINS_HOME=/mnt/backup-test
■ Now execute the restored Jenkins instance:
■ java-jar jenkins.war ---httpPort=9999
WHAT DID WE LEARN

● Backups are critical for Disaster Recovery

● Always set up a backup policy
■ What configurations and records need to be saved from the master?
■ How often should backups be taken?
■ Where should backups be stored?
● Validate your backups
■ you should periodically check whether your backups are intact and can be
used to meet your recovery objectives
● Taking backups is a ​Jenkins best practice
GOING FURTHER

● Some recommended readings on this subject:

■ Why Smart, Efficient Backup and Restore Techniques are Essential with
Jenkins Production Server
■ Backup Plugin
■ thinBackup Plugin
 Automate Tasks with CLI and API

AUTOMATE JENKINS

WHY AUTOMATE JENKINS

● Scripting allows you to automate and standardize tasks:

■ Routine tasks
■ Bulk updates
■ Troubleshooting
■ and more
● Provides consistency and efficiency
■ For different staff members
■ At different times

WHAT IS JENKINS CLI ?

● Jenkins Command Line Interface (CLI) is a Java application

■ Provided as jenkins-cli.jar file, downloadable from Jenkins
■ Executed in a JVM
● This application is a client that reaches your Jenkins instance
■ Provides sub-commands to execute remote operations
■ Does not have to run on the same machine as Jenkins

HOW IT WORKS ?

● CLI makes an HTTP call to Jenkins, discovers the port

■ This is the port used for the JNLP Agent
■ CLI attempts to open TCP/IP connection
● If it fails, it falls back to HTTP-based connection
■ Slow and wastes effort, but works with reverse proxy

● Command execution mostly happens on master

■ jenkins-cli.jar file needs to be downloaded locally
■ Therefore, one CLI jar can be used to talk to different masters

HOW TO: CLI AUTHENTICATION

● By SSH public key

■ Register your public key to Jenkins under your user account
■ Hide your private key in ~/.ssh
 HOW TO: ACCESS CLI HOMEPAGE

● Access CLI home page from the Manage Jenkins page:

HOW TO: USING CLI

● -s​ option: specifies the URL to connect to Jenkins

● JENKINS_URL environment variable to use instead
 EXISTING CLI COMMANDS

CLI SHOW CASE - BUILD COMMAND

● Equivalent of clicking “build now”

● Can take parameters:
■ -c​: run polling first and build only if there is a change
■ -s​: do not just schedule, but wait until the build is completed
■ -p​: specify build parameters
■ -v​: report console output as well

CLI SHOW CASE - CRUD OF JOBS

● create-job, get-job, update-job, delete-job

● Reads/writes XML representation of the job
■ use stdin/stdout of CLI process
● Can be used to script job creation / update

CLI SHOW CASE - GROOVYSH

● Use the interactive Apache Groovy shell to manipulate the in-memory state of Jenkins
 ● Equivalent to the script console, but interactive

MORE ABOUT APACHE GROOVY IN CLI

● Use ​groovy​ command to run a pre-authored script file

● See ​https://github.com/jenkinsci/jenkins-scripts​ for scripts other people wrote
JENKINS REMOTE ACCESS API

WHY USE JENKINS API ?

● You may need to integrate Jenkins with another application

● Machine-to-machine communications is required when a service talks to the Jenkins instance
■ Not GUI

WHAT IS THE JENKINS API ?

● A Remote access API that is machine consumable

● A REST-like API, allowing remote resources manipulation
● No single entrypoint, but per resource /api URL
● Projects API: http://<JENKINS_URL>/job/api
● A multi-flavored data serialization:
■ XML
■ JSON with JSONP support
■ Python

HOW TO: SIMPLE JSON API EXAMPLE

● This example will launch a parameterized build using the API

● We are using the JSON flavor
● This is done using a command-line tool named ​curl​, that can issue HTTP requests
● An example Jenkins configuration:
■ Main URL is http://${JENKINS_URL}
■ Project name is ${MY_PROJECT}
■ The project is protected with a TOKEN: ${TOKEN}
■ Build requires 2 parameters:
○ parameter name ​id​ which will be exported as ${ID} (or %ID on
windows) when the build starts
○ parameter name ​verbosity​ which will be exported as
${verbosity} (or %verbosity% on windows) when the build starts
 MORE ABOUT THE JENKINS API

● Jenkins API clients are available for various programming languages:

■ Ruby: ​https://rubygems.org/gems/jenkins_api_client
■ Python: ​https://pypi.python.org/pypi/python-jenkins/
■ JavaScript: ​https://www.npmjs.com/package/jenkins-api

WHAT DID WE LEARN ?

● Automating Jenkins can be done using its CLI

● An alternative to running Groovy with the GUI
● The CLI is a JAR file that can be executed remotely to run Jenkins operations
● The API is REST-like and can be used to integrate Jenkins with other services

GOING FURTHER

● Some recommended readings on this subject:

■ Jenkins CLI
■ Remote access API

LAB EXERCISE

Jenkins Command Line Interface

Jenkins Command Line Interface

The goal of this exercise is to experiment with the Jenkins Command Line Interface (CLI) and its REST
API.

Requirements

The TCP port for inbound agents port must be correctly configured to ensure that Jenkins CLI will work:

● Check this on the ​Jenkins Global Security​:

○ Enable security​ must be enabled
○ The ​TCP port for inbound agents​ field must be set to ​Fixed​, using port ​50000

Our Jenkins instance is secured. Thus the ​JENKINS CLI plugin​ requires that you configure
authentication for each user.
 the flags ​--username​ and ​--password​ are still there for backward compatibility
TIP
but do not work on Jenkins versions >=1.419

●
Browse to the User configuration page for butler:
○ Click on the username in the top-right of the screen
○ Click on the left-menu ​Configure button
○ OR use the direct URL: ​http://localhost:5000/jenkins/user/butler/configure
○ Click "Add new Token" under ​API Token
■ Click "Generate" to generate a new token

Figure 1. Add new Token

○ Copy the API Token

The API token should be protected like your password, since it allows other
TIP
people to access Jenkins as you.

Step 1. Download the client jar

● Start by browsing to the Jenkins Master, click ​Manage Jenkins​, and then click ​Jenkins CLI​ to
get to the Jenkins CLI web page.

Alternatively, you can go directly to this page by navigating to

TIP
http://localhost:5000/jenkins/cli

●
You will find there a ​quickstart​ example, link to documentation and a comprehensive list of
commands available
● It also gives you the URL for ​jenkins-cli.jar​:
Figure 2. Link to download Jenkins CLI

● Open a ​Devbox​ command line, where you will download the ​jenkins-cli.jar​ file into ​/tmp​:

We will use the Jenkins private URL, ​http://jenkins:8080/jenkins​, instead of

IMPORTANT
the public one that Jenkins shows you. This allows us to reach Jenkins
directly rather than using the reverse-proxy.

cloudbees-devbox $ cd /tmp
cloudbees-devbox $ curl -LO \
http://jenkins:8080/jenkins/jnlpJars/jenkins-cli.jar
...
cloudbees-devbox $ ls jenkins-cli.jar
jenkins-cli.jar

Step 2. Run the help command

● Run the proposed command (​help​).

○ It should ask you for authentication against the Jenkins master
cloudbees-devbox $ java -jar ./jenkins-cli.jar \
-s http://jenkins:8080/jenkins help
You must authenticate to access this Jenkins.
Use --username/--password/--password-file parameters or login command.
...
cloudbees-devbox $ java -jar jenkins-cli.jar \
-s http://jenkins:8080/jenkins \
help
...

● Set Jenkins URL as the ​${JENKINS_URL}​ environment variable so that you don’t have to keep
retyping the long URL.

It should produce the same output:

cloudbees-devbox $ export JENKINS_URL=http://jenkins:8080/jenkins

cloudbees-devbox $ java -jar jenkins-cli.jar \
-auth butler:{API Token} help

Step 3. Run a build

● Now, let’s launch a new "our CD pipeline" run by kicking off a new build of the job demoapp-build:

The ​-s​ flag makes the command wait for the completion of the build.
TIP

cloudbees-devbox $ java -jar jenkins-cli.jar \

-auth butler:{API Token} build -s \
TeamA/pipeline-job -v

For extra credit, try the command again, but this time manually abort the build while it is running; CLI will
report a failure.

Also try running this command with the ​-c​ option and verify that the build is skipped.

That’s all for this exercise !