UNIT I

Introduction to DevOps and

The Culture
 What is DevOps?
• DevOps is a combination of two words: "Development" and
"Operations."
• It’s a modern approach where software developers and software
operations teams work together throughout the entire software life
cycle, from planning and coding to testing, deploying, and monitoring.
• The main idea of DevOps is to improve the software delivery process
which can be achieved by software release automation, continuous
monitoring and testing.
 Goals of DevOps
• The goals of DevOps are:
1. Faster and continuous software releases.
2. Reduces manual errors through automation.
3. Built-in Monitoring, detect failures or issues during delivery or in
production.
4. Automate testing throughout the software delivery process.
 DevOps Pipeline / Devops Chain
• A DevOps pipeline is a set of automated processes and tools that
allows developers and operations professionals to collaborate on
building and deploying code to a production environment.
• A DevOps pipeline typically includes build automation/continuous
integration, automation testing, validation, and reporting but it can
differ by organization.
• It may also include one or more manual gates that require human
intervention before code is allowed to proceed.
• “ Continuous ” is a differentiated characteristic of a DevOps pipeline.
• This includes continuous integration, continuous delivery/deployment
(CI/CD), continuous feedback, and continuous operations.
• Since there isn’t one standard DevOps pipeline, an organization’s
design and implementation of a DevOps pipeline depends on its
technology stack, a DevOps engineer’s level of experience, budget,
and more.
• Also, each organization has a different technology stack that can
impact the process.
• Example:
• If codebase is node.js, factors include whether you use a local proxy
npm registry, whether you download the source code and run `npm
install` at every stage in the pipeline, or do it once and generate an
artifact that moves through the pipeline.
• Or, if an application is container-based, you need to decide to use a
local or remote container registry, build the container once and move
it through the pipeline, or rebuild it at every stage.
 Trunk based development
• In DevOps, the "trunk" (or "main" or "master") branch in a version
control system is the central, shared branch where all developers
integrate their code changes.
• It represents the main line of development and is kept in a releasable
state, meaning it's always ready for deployment.
• Trunk-based development, a common practice in DevOps, involves
developers merging their code frequently into this single branch,
typically multiple times a day.
• This contrasts with feature-branching strategies where developers
work on separate branches before merging.
 Components of a DevOps Pipeline
• 1. Continuous integration/Continuous delivery/deployment (CI/CD)
• Continuous integration is the practice of making frequent commits to
a common source code repository.
• It’s continuously integrating code changes into existing code base so
that any conflicts between different developer’s code changes are
quickly identified and relatively easy to make them right.
• This practice is critically important to increasing deployment
efficiency.
• We believe that trunk-based development is a requirement of
continuous integration.
• If you are not making frequent commits to a common branch in a
shared source code repository, you are not doing continuous
integration.
• If your build and test processes are automated but your developers
are working on isolated, long-living feature branches that are
infrequently integrated into a shared branch, you are also not doing
continuous integration.
• Developers practicing continuous integration merge their changes
back to the main branch as often as possible.
• The developer's changes are validated by creating a build and running
automated tests against the build.
• By doing so, you avoid integration challenges that can happen when
waiting for release day to merge changes into the release branch.
• Continuous integration puts a great emphasis on testing automation
to check that the application is not broken whenever new commits
are integrated into the main branch.
• Continuous delivery ensures that the “main” or “trunk” branch of an
application's source code is always in a releasable state.
• In other words, if management came to your desk at 4:30 PM on a
Friday and said, “We need the latest version released right now,” that
version could be deployed with the push of a button and without fear
of failure.
• This means having a pre-production environment that is as close to
identical to the production environment as possible and ensuring that
automated tests are executed, so that every variable that might cause
a failure is identified before code is merged into the main or trunk
branch.
• Continuous delivery is an extension of continuous integration since it
automatically deploys all code changes to a testing and/or production
environment after the build stage.
• This means that on top of automated testing, you have an automated
release process and you can deploy your application any time by
clicking a button.
• In theory, with continuous delivery, you can decide to release daily,
weekly, fortnightly, or whatever suits your business requirements.
• However, if you truly want to get the benefits of continuous delivery,
you should deploy to production as early as possible to make sure
that you release small batches that are easy to troubleshoot in case of
a problem.
• Continuous deployment goes one step further than continuous
delivery.
• Continuous deployment is a more advanced stage of the DevOps
pipeline that completely automates the release and deployment of
software into production environments.
• With this practice, every change that passes all stages of your
production pipeline is released to your customers.
• The new versions of software are validated and deployed into a
production environment without requiring any human intervention.
• Using continuous integration and continuous delivery as its basis it
takes automation to the next level by eliminating the need for manual
intervention and deployment.
• This approach delivers new features and improvements to end users
faster and more efficiently.
• Continuous deployment is an excellent way to accelerate the
feedback loop with your customers and take pressure off the team as
there isn't a "release day" anymore.
• Developers can focus on building software, and they see their work
go live minutes after they've finished working on it.
• This is rare and in most cases unnecessary.
• Some of the businesses who have hundreds or thousands of
developers and have many releases each day that require, or even
want to have, this level of automation use continuous deployment.
• It’s necessary to carefully evaluate and plan for your organization’s
specific needs before adopting this practice.
• In some cases, a more traditional approach, such as continuous
delivery, that retains human control over deployments might be more
suitable.
 Continuous integration vs. delivery vs.
deployment
• CI and CD are two acronyms frequently used in modern development
practices and DevOps.
• CI stands for continuous integration, a fundamental DevOps best
practice where developers frequently merge code changes into a
central repository where automated builds and tests run.
• But CD can either mean continuous delivery or continuous
deployment.
 How the practices relate to each other?
• To put it simply continuous integration is part of both continuous
delivery and continuous deployment.
• And continuous deployment is like continuous delivery, except that
releases happen automatically.
• Continuous delivery automates the build, testing, and preparation for
deployment, but requires a manual step to trigger the actual
deployment to production.
• Continuous deployment, on the other hand, automates the entire
process, including the deployment to production, without requiring
manual intervention.
 Continuous Testing
• Continuous testing involves continuous and automated testing
activities into the software release process rather than treating
testing as an isolated phase.
• This approach enables applications to be always in a deployable state
by using the ongoing automated and manual testing to validate code
quality, functionality, and security, providing constant feedback to
development teams.
• Test automation allows faster execution of tests, reduces manual
errors, allows QA teams to focus on building robust tests, and
shortens the end-to-end duration of software release time.
• The tests should include performance and security testing apart from
the standard functional testing.
• Continuous operations incorporate various methodologies and
strategies that allow teams to automate tasks that were traditionally
performed manually.
• For example, adopting Infrastructure as Code principles, performing
systematic Configuration Management, setting up automation
security and compliance guardrails, and optimizing application and
infrastructure performance and cost are all core components of an
effective continuous operations practice.
• Continuous operations is the practice of DevOps that emphasizes
stability, performance, and high availability of applications and
infrastructure environments.
• Minimizing downtime and reducing the number and impact of
incidents while ensuring environments stay operational at all times is
the core of this practice.
 DevOps Engineer
• A DevOps engineer is an IT generalist who should have a wide-ranging
knowledge of both development and operations, including the
following:
 coding,
 infrastructure management,
 system administration,
 DevOps toolchains.
• DevOps engineers need to have a strong understanding of the
following:
 common system architecture,
 provisioning,
 experience with the traditional developer toolset,
 practices such as using source control,
 giving and receiving code reviews,
 writing unit tests
 familiarity with agile principles.
 Role of DevOps Engineer
• The technical skills required of a DevOps engineer will vary depending
on the
team structure,
technologies,
toolsets in use.
• It’s also important for a DevOps engineer to have a solid
understanding of all the components of a delivery pipeline, and to
know the pros and cons of available tools and services.
1. Communication and collaboration:
• It’s important for a DevOps engineer to communicate and collaborate
effectively with teams, managers, and customers.
• These so-called “soft-skills” are often overlooked and undervalued,
but the success of DevOps relies heavily on the quality and quantity
of feedback across the entire value stream.
2. System administration:
• A DevOps engineer will have experience with system administration, such
as provisioning and managing servers, deploying databases, security
monitoring, system patching, and managing internal and external network
connectivity.
3. Experience with DevOps tools:
• Since using the right tools are essential to DevOps practices, the DevOps
engineer must understand, and be able to use, a variety of tools.
• These tools span the DevOps lifecycle from infrastructure and building, to
monitoring and operating a product or service.
4. Configuration management:
• DevOps engineers will often be expected to have experience with one
or more configuration management tools such as Chef, Puppet, or
Ansible.
• Many organizations have adopted these or similar tools to automate
system administration tasks such as deploying new systems or
applying security patches to systems already running.
5. Containers and container orchestration:
• In DevOps, a container is a lightweight, standalone, and executable
package of software that includes everything needed to run an application:
code, runtime, system tools, system libraries, and settings.
• With containerization, a technology popularized by Docker, the code for
the application and its runtime environment are bundled in the same
image.
• This makes traditional configuration management tools less necessary.
• At the same time managing containers brings its own challenges, and
experience with the class of tools known as “container orchestrators” (e.g.
Docker or Kubernetes) becomes a necessary skill for the DevOps engineer.
6. Continuous integration and continuous deployment:
• Continuous integration and continuous Delivery (CI/CD) are core
practices of a DevOps approach to software development, and
enabled by a host of available tools.
• The most fundamental function of any CI/CD tool or set of tools is to
automate the process of building, testing, and deploying software.
• DevOps engineers will usually need experience with configuring and
deploying one or more CI/CD tools, and will usually need to work
closely with the rest of the development organization to ensure that
these tools are used effectively.
7. System architecture and provisioning:
• A DevOps engineer should have the ability to design, provision, and
manage computer ecosystems, whether on-premise or in the cloud.
• It’s important to understand Infrastructure as Code (IaC), an IT
management process that applies best practices from DevOps
software development to the management of cloud infrastructure
resources.
• A DevOps engineer should understand how to model system
infrastructure in the cloud with Amazon Web Services (AWS), AWS
CloudFormation, or Terraform.
8. Familiarity with coding and scripting:
• Many traditional system administrators have experience writing shell
scripts to automate repetitive tasks.
• A DevOps engineer should go beyond writing automation scripts and
understand advanced software development practices and how to
implement agile development practices such as code reviews and
using source control.
9. Collaborative management skills:
• Cross-team collaboration is a fundamental component of an effective
DevOps strategy, regardless of the specific organizational structure.
• Whether the engineering team is a group divided only by roles or
there are separate teams for feature development, quality assurance,
DevOps, etc., the DevOps engineer should work across the
organization as coach and co-worker, with many different people.
• Example:
• One of the most valuable returns on a DevOps investment is the
ability to deliver faster feedback to developers.
• A DevOps engineer will often have to work with QA (whether they be
manual testers or developers who write test automation) to improve
the speed, efficacy, and output of testing methodologies.
• At the same time, developers may need support from DevOps
engineers when working to improve the process of building and
deploying application code.
 Centralizing the Building Server
• When we adopt a build server, we centralize everything in one server.
• This means that we use a different software to release our new
software.
• Having only one server, or cluster, for building new software means
that there is only one point of failure.
• Any problem is centralized at only one point.
• This reduces the cost of maintaining the software and speeds up the
release process.
• The building server is normally connected with an artifact repository
server.
• The repository server is where the build is stored and saved. It is
associated with continuous release.
• Essentially, with this practice, we build the software every time and
release it to a server.
• This server essentially maintains the different versions of the software
we build on the server.
• Normally, we establish a naming policy to maintain the different
software versions.
• This is because we want to identify every version of the software
uniquely.
 Monitoring Best Practices
• To be effective, monitoring must be combined with some other practice.
• Monitoring is the backbone for every DevOps practice, and if we want a
very successful journey, we must be sure to have a good monitoring
system.
• At the same time, we must start to monitor not only the production but
possibly the server.
• A log analysis is the most important practice for preventing errors and
understanding how the system functions.
• Some software is required for analyzing the log and making related
predictions.
• The software most commonly used is ELK (Elasticsearch, Logstash,
and Kibana).
• This is helpful because it gives a complete log analysis system, not
only providing alerts, but also a graphical representation of the error
and the logs.
• Having a graphical representation of the errors is important for
providing visible feedback about the software, without the necessity
of reading a log, in order to understand the status of the software.
 Forms of Monitoring
1. Black-box monitoring tests a piece of code as if it were in a black
box. It reveals only the status of the system, to determine whether
it is alive. It doesn’t really indicate what is happening inside,
because the monitoring is external. An example of black-box
software monitoring is Nagios.
2. White-box monitoring is a type of monitoring provides a clear
picture of the inside of the system, such as, for example, the
number of HTTP connections open, the number of errors, etc.
Prometheus is an example of white-box monitoring software.
 Best Practices for Operations
• The operations team is delegated to maintain the software in the
production environment.
• The operations team is the first face of the company.
• The operations team can help to identify common issues and provide
documentation to solve them faster.
• The documentation must indicate common errors in the log and show
how to solve the root cause(s) of the problem.
• This documentation should be written with people who don’t know
the system in mind and, based on that, must provide specific details
about the appropriate steps to be taken.
• Another operations practice we can put in place is developer on-call.
• The developer on-call is essentially a software engineer, working with
the operations professionals to resolve errors in production.
• This has two principal advantages.
• The first is a reduction in the time it takes to identify and fix an issue.
• The second advantage is improving the level of responsibility.
• Because the developer works to fix a live issue, he/she understands
better what’s wrong with the software and thus can improve the way
he/she writes the software and the log, because a bad log can result
in more work for him/her in future.