What is DevOps?
INTRODUCTION TO DEVOPS
Cem Sakarya
DevOps Risk Advisor
INTRODUCTION TO DEVOPS
Organizational structure
Online products are complex
Expensive to develop and maintain
Thousands of engineers
How to collaborate efficiently?
INTRODUCTION TO DEVOPS
INTRODUCTION TO DEVOPS
DevOps
DevOps is a combination of
methods,
tools, and
cultural behavior
that improves how software is developed and maintained.
It helps organizations deliver higher-quality online products faster.
INTRODUCTION TO DEVOPS
Traditional Change Management DevOps
Independent teams Different teams work together
Different teams have different goals Software Development + IT Operations =
Slow development DevOps
Different teams have similar goals
High speed development
INTRODUCTION TO DEVOPS
Traditional release
INTRODUCTION TO DEVOPS
Minimum Viable Product
A minimum viable product (MVP) is an
early version of a product with limited
functionality.
Cheaper to build
High speed time-to-market
INTRODUCTION TO DEVOPS
Product improvements
INTRODUCTION TO DEVOPS
DevOps benefits
INTRODUCTION TO DEVOPS
Let's practice!
INTRODUCTION TO DEVOPS
Use Cases for
DevOps
INTRODUCTION TO DEVOPS
Cem Sakarya
DevOps Risk Advisor
DevOps
DevOps: Software Development + IT
Operations
Collaborative and Shared Responsibilities
MVP Releases
Adaptable to various use cases
INTRODUCTION TO DEVOPS
Infrastructure Engineering
Design, Develop, and Maintain the IT
infrastructure
This infrastructure requires power from the
cloud or from hardware the company owns
Infrastructure engineers take care of the
hardware, network, and cloud components
INTRODUCTION TO DEVOPS
Product Engineering
Design, develop, and maintain the Software
Products
Customer serving components
INTRODUCTION TO DEVOPS
Data Engineering
Data engineering refers to the building of systems to enable the collection and usage of
data.
1 https://en.wikipedia.org/wiki/Data_engineering
INTRODUCTION TO DEVOPS
INTRODUCTION TO DEVOPS
DataOps MLOps
Software is powered by data Predicting the future using the past data
Data is moved in data pipelines Data Preparation (Historical data)
Ensuring data moves smoothly Model Training
Data is not lost while moving Model Testing
Move the data for the use of data scientists Deployment
and ML Engineers
Maintenance
INTRODUCTION TO DEVOPS
No competition
INTRODUCTION TO DEVOPS
Let's practice!
INTRODUCTION TO DEVOPS
Project
Management
Methodologies for
DevOps
INTRODUCTION TO DEVOPS
Cem Sakarya
DevOps Risk Advisor
Why is project management important to What is project management?
DevOps?
Project management is the use of
Defines how the team will operate
specific knowledge,
Drives change
skills,
Timeline/Resource Management
tools, and
Collaboration within/across teams
techniques
to deliver something of value to people.
INTRODUCTION TO DEVOPS
Waterfall
An old methodology
No return back to earlier stage
Develop all of the product at once
INTRODUCTION TO DEVOPS
Agile
Series of cycles
Achieving one small goal at a time
First MVP, then improvements
Going through the cycle each time
INTRODUCTION TO DEVOPS
Choosing the right project management model
The best methodology depends on:
Team's skill set
Budget
Complexity of the project
Expectations
Agile is the de facto standard for most
software development teams.
INTRODUCTION TO DEVOPS
Scrum Kanban
A sprint in Scrum is a two-to-four week No sprints in Kanban, instead tracking
timeframe with a light development goal for improvements continuously.
the team.
INTRODUCTION TO DEVOPS
Scrum Kanban
Regular, fixed-length sprints (i.e., two Continuous flow
weeks)
Use visuals to improve work-in-progress
Learn through experiences
Visualize the flow of work, limit work-in-
Sprint planning, sprint, daily standup, sprint progress, manage flow, incorporate
review, sprint retrospective feedback loops
Product owner, scrum master, development No defined roles
team
1 https://www.atlassian.com/agile/kanban/kanban-vs-scrum
INTRODUCTION TO DEVOPS
Both Scrum and Kanban are under Agile methodology
Both Scrum and Kanban improves collaboration
Both powerful methodologies when applied to correct use case
INTRODUCTION TO DEVOPS
Let's practice!
INTRODUCTION TO DEVOPS
The DevOps Change
Management Model
INTRODUCTION TO DEVOPS
Cem Sakarya
DevOps Risk Advisor
Requirements
Developing a new product
Leadership: Sets business expectations
Product Experts: Define user experience
requirements
INTRODUCTION TO DEVOPS
Design
Product Engineers: Designing the customer
experience
Infra Engineers: Designing the parts that
product engineers will use
Data Engineers: Designing the data
pipelines for the new product
INTRODUCTION TO DEVOPS
Develop and test
Product Engineers
Develop for the customer experience
Customer interaction
Infra Engineers
Supporting the product
Providing testing environment
Data Engineers
Providing test data
INTRODUCTION TO DEVOPS
Testing
Testing and developing at the same time
Testing before customer interaction
Otherwise poor user experience and
security issues
INTRODUCTION TO DEVOPS
Experimentation
Limited set of users
Early user interactions
Observe if the users like the product
Real user feedback
INTRODUCTION TO DEVOPS
Cyclical development
INTRODUCTION TO DEVOPS
Let's practice!
INTRODUCTION TO DEVOPS
Main Software
Architecture
Systems
INTRODUCTION TO DEVOPS
Cem Sakarya
DevOps Risk Advisor
Software architecture
Software architecture refers to the
fundamental structures of a software
system and the discipline of creating such
structures and systems. Each structure
comprises software elements, relations
among them, and properties of both
elements and relations.
1 https://en.wikipedia.org/wiki/Software_architecture
INTRODUCTION TO DEVOPS
Application Programming Interfaces (APIs)
Data flows through APIs
They are essential for software and architecture
INTRODUCTION TO DEVOPS
Various architectural systems
Factors to consider
Type of software
In machine or in the browser
Complexity of the software
Simpler software parts can be treated as a single unit
Separate the pieces for more complex software
INTRODUCTION TO DEVOPS
Microservices
Most important architectural system for
complex software
Common in modern organizations
Rather than a big single unit, have many
smaller independent units
Smaller units are called the microservices
They communicate to each other via APIs
INTRODUCTION TO DEVOPS
INTRODUCTION TO DEVOPS
INTRODUCTION TO DEVOPS
INTRODUCTION TO DEVOPS
DevOps for microservices architecture
INTRODUCTION TO DEVOPS
Benefits of DevOps for microservices
Software is changed constantly in large organizations
Change them one at a time
Microservices are more secure and reliable
If anything goes bad with one microservice, the rest will still function well
INTRODUCTION TO DEVOPS
Let's practice!
INTRODUCTION TO DEVOPS
Modern IT
Infrastructure
Concepts
INTRODUCTION TO DEVOPS
Cem Sakarya
DevOps Risk Advisor
Main components of IT infrastructure
Combination of various components
Customized for organizations specific
needs
INTRODUCTION TO DEVOPS
Hardware
Can be hosted on organization's own
premises
Or it can be accessed through a cloud
service provider
INTRODUCTION TO DEVOPS
Network
Communication with other parts of the
software
Communication with the users through the
Internet
INTRODUCTION TO DEVOPS
Software
We need software to: The goal of infrastructure software:
Manage the hardware and network enable the product teams to develop and
serve customers.
Configure them for the organization's use
INTRODUCTION TO DEVOPS
Change management
Software needs constant maintenance and improvement
These changes could be very expensive and disruptive to the services of the organization
Infrastructure Engineering provides the tools that's necessary to build products
INTRODUCTION TO DEVOPS
Developer platform
INTRODUCTION TO DEVOPS
Codebase & version control
Version Control: Git
INTRODUCTION TO DEVOPS
DevOps CI/CD pipelines
DevOps in narrow meaning - CI/CD
Code needs to be built
Turn code file into machine executable file
INTRODUCTION TO DEVOPS
Deployment platform
INTRODUCTION TO DEVOPS
Change management in microservices
INTRODUCTION TO DEVOPS
Let's practice!
INTRODUCTION TO DEVOPS
Elements of DevOps
INTRODUCTION TO DEVOPS
Cem Sakarya
DevOps Risk Advisor
DevOps change management model
Continuous Integration & Continuous Delivery
INTRODUCTION TO DEVOPS
INTRODUCTION TO DEVOPS
Principles of CI
1. Version Control Software
2. Frequent integrations
INTRODUCTION TO DEVOPS
Version control
Main branch as the source-of-truth for the codebase
Multiple teams working on the same software
Peer Review
Testing in every step
INTRODUCTION TO DEVOPS
Principles of CD
1. Automated builds
2. Automated tests
3. Predictable and short change times
INTRODUCTION TO DEVOPS
Experimentation
Limited set of users
Collect their feedback
Happens once a change is deployed
INTRODUCTION TO DEVOPS
Feedback loops
Continuously improving the product
Use the review stage for the feedback loop
Turn the feedback into action
Change the software if necessary
INTRODUCTION TO DEVOPS
Let's practice!
INTRODUCTION TO DEVOPS
Data Engineering
Implications of the
DevOps Architecture
INTRODUCTION TO DEVOPS
Cem Sakarya
DevOps Risk Advisor
Microservices architecture
Microservices are small-scale software
programs
Microservices deployed separately
Each microservice take care of a different
functionality
Each microservice has its own data and
logic
They store the data in private databases
INTRODUCTION TO DEVOPS
Monolithic architecture
Opposite of the microservices architecture
A big single unit
Much simpler compared to microservices
Maintenance and change is very hard and
risky
INTRODUCTION TO DEVOPS
Monolithic Architecture Microservices Architecture
Limited number of databases A database for each microservice
All application uses the same databases Microservices must do API calls to reach
another services database
Could be a viable option for small scale
applications Microservices are effective in large
organizations and complex products
INTRODUCTION TO DEVOPS
Microservices private databases
Some product functionalities require collaboration from multiple microservices
Microservices can not access each others databases freely, so they do API calls
Microservices always communicate with each other.
INTRODUCTION TO DEVOPS
Data Engineering applications in microservices
INTRODUCTION TO DEVOPS
Data Engineering applications in microservices
INTRODUCTION TO DEVOPS
Let's practice!
INTRODUCTION TO DEVOPS
Data Pipelines
INTRODUCTION TO DEVOPS
Cem Sakarya
DevOps Risk Advisor
Data pipelines
INTRODUCTION TO DEVOPS
ETL
1 https://www.istockphoto.com/nl/vector/data-extract-transform-load-gm1361894912-434102842?phrase=ETL
INTRODUCTION TO DEVOPS
Batch processing
Move offline data in batches
Batch Processing: A large amount of data
INTRODUCTION TO DEVOPS
INTRODUCTION TO DEVOPS
INTRODUCTION TO DEVOPS
INTRODUCTION TO DEVOPS
Recap
Data Engineering is an integral part of
Infrastructure Engineering
Data pipelines are robust tools important
for handling data
Batch processing works on regular
schedules, stream processing works
continuously
Batch processing is useful for handling the
accumulated data
Stream processing handles the real-time
d t
INTRODUCTION TO DEVOPS
Let's practice!
INTRODUCTION TO DEVOPS
Analytics &
Reporting with
DevOps
INTRODUCTION TO DEVOPS
Cem Sakarya
DevOps Risk Advisor
DevOps benefits for reporting
Handling Multiple Data Sources in harmony
Producing many logs during the change management process
INTRODUCTION TO DEVOPS
INTRODUCTION TO DEVOPS
INTRODUCTION TO DEVOPS
INTRODUCTION TO DEVOPS
INTRODUCTION TO DEVOPS
Reporting architecture For DevOps
DevOps produces a lot of data
Data sources are spread out
Data needs to be collected and moved to a database
Data Scientists can use the data for insights, reports, and dashboards
INTRODUCTION TO DEVOPS
Let's practice!
INTRODUCTION TO DEVOPS
Tools: Ecosystem
INTRODUCTION TO DEVOPS
Cem Sakarya
DevOps Risk Advisor
Requirements
Business and engineering interaction
Used throughout the software change
management process
Project Management Tools
Communication Tools
1 https://atlassian.design/foundations/logos 2 https://slack.com/media-kit 3 https://discord.com/branding
INTRODUCTION TO DEVOPS
Version Control Software
Main change management technology is
git
Git is a protocol used for version control
Git ensures multiple developers can work
on the same software
Git has many implementations, most
common ones are GitHub and GitLab
1 https://git-scm.com/downloads/logos 2 https://github.com/logos 3 https://about.gitlab.com/press/press-kit/
INTRODUCTION TO DEVOPS
Build Tools
Software needs to be built to become
executable
Main build tools are: Maven and Gradle
1 https://maven.apache.org/ 2 https://gradle.com/brand/
INTRODUCTION TO DEVOPS
CI/CD Tools
CI/CD pipelines are the main principles of
DevOps
CI/CD pipelines ensure automated building,
testing, and deployment of software
Main CI/CD tools are Jenkins and CircleCI
1 https://www.jenkins.io/press/ 2 https://circleci.com/legal/trademark-guidelines/
INTRODUCTION TO DEVOPS
Deployment
Microservices are developed and deployed
independently from each other
Containers imitate separate machines
Microservices are deployed on separate
containers
Containers: Docker and Podman
Container orchestration: Kubernetes
1 https://www.docker.com/company/newsroom/media-resources/ 2 https://podman.io/ 3 https://kubernetes.io/
INTRODUCTION TO DEVOPS
Monitoring Tools
Products need to be closely monitored and
observed for quality and reliability issues
Monitor the DevOps health and change
management metrics
Example tools used for monitoring are
SignalFX and AppDynamics
1 https://www.splunk.com/en_us/products/observability.html?301=/en_us/devops.html
INTRODUCTION TO DEVOPS
Data management tools: Kafka
Kafka is a message publishing system
Kafka is heavily used in microservices
architecture
Microservices keep a journal of the work
they do on Kafka
1 https://kafka.apache.org/trademark
INTRODUCTION TO DEVOPS
Data pipeline management tools
Most tools used for both batch and
streaming processing
Main tools used for data pipeline
management are: Apache Airflow, Hevo
Data, and Prefect
1https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow%20logos 2 https://design.hevodata.com/ 3
https://www.prefect.io/newsroom/logos/
INTRODUCTION TO DEVOPS
Recap
INTRODUCTION TO DEVOPS
Let's practice!
INTRODUCTION TO DEVOPS
Ensuring Data
Quality Standards
with DevOps
INTRODUCTION TO DEVOPS
Cem Sakarya
DevOps Risk Advisor
DevOps ensures good code
Most software handle data
Good software drives high-quality data
DevOps helps maintain high-quality in software
INTRODUCTION TO DEVOPS
Data quality
How trusted is the information?
Most recent and accurate information
Can be hard and costly to achieve
Not all data requires the same quality
There are many elements to Data Quality
INTRODUCTION TO DEVOPS
Elements of data quality
Elements breaks down and define Data
Quality
They should be traced and standardized
INTRODUCTION TO DEVOPS
Is the data correct in all details?
Should be closely monitored
INTRODUCTION TO DEVOPS
How comprehensive is the data?
Check if data is lost when it is stored in a
database
Check if data is lost when it is moved to
different databases
INTRODUCTION TO DEVOPS
Consistency is about the reliability of the
information
It is also referred to as Data Integrity
Should be closely monitored and tested
INTRODUCTION TO DEVOPS
Keeping and handling only the relevant
information
Irrelevant information wastes time and
resources
INTRODUCTION TO DEVOPS
How up to date is the information?
Old data could be replaced by more recent
information
Raw data within microservices is not used
in data analysis
Microservices talk to each other via API
calls to get the most recent information
The data within microservices should be
replicated at an acceptable frequency
INTRODUCTION TO DEVOPS
Let's practice!
INTRODUCTION TO DEVOPS
Observability and
Reliability
INTRODUCTION TO DEVOPS
Cem Sakarya
DevOps Risk Advisor
Observability
Observing inner components of software
Components might consist of:
Microservices
APIs
Databases
Other components
Check they are working correctly.
1https://www.istockphoto.com/nl/vector/het-lijnpictogram-van-de-verrekijker-oceaanconcept-verrekijkerteken-
op-witte-gm1245624425-363092800?phrase=field%20glass
INTRODUCTION TO DEVOPS
Why is observability needed?
Many different parts tied to each other
Spot specific problems and their root cause
Monitoring systems effectively
INTRODUCTION TO DEVOPS
Alerting
Alerts help detect problems early
Set alerts if software fails to run
Setting up alerts enable teams to react fast
when problems happen
1 https://www.istockphoto.com/nl/vector/notification-bells-icon-gm1364077374-435399123?phrase=alerting
INTRODUCTION TO DEVOPS
Why is reliability important?
Examples:
Government services
Defense systems
Banking software
Many other software also require high
reliability
INTRODUCTION TO DEVOPS
How to increase reliability?
Two main principles:
Testing
Observability
DevOps automates testing, and enforce
observability.
INTRODUCTION TO DEVOPS
Testing Observability
Testing is used to detect the problems before
Observability is used to detect the problems
they go to market.
as they happen in user interaction.
DevOps ensures automated testing.
Observability is needed even when the
software is tested enough.
INTRODUCTION TO DEVOPS
Reliability in microservices
INTRODUCTION TO DEVOPS
Let's practice!
INTRODUCTION TO DEVOPS
DevOps Cultural
Concepts
INTRODUCTION TO DEVOPS
Cem Sakarya
DevOps Risk Advisor
INTRODUCTION TO DEVOPS
Concepts
Collaboration
Autonomous Teams
Shared Responsibilities
Automation
Post-Mortem
INTRODUCTION TO DEVOPS
Collaboration
Heart of DevOps
Collaboration between Development and Operations
INTRODUCTION TO DEVOPS
Autonomous teams
Teams are diverse
Self sufficient teams
Teams have members who specialized in:
Product Engineering
Operations
Data Engineering, etc.
INTRODUCTION TO DEVOPS
Shared responsibilities
Autonomous teams take care of specific
parts of the software
They are responsible for developing and
running
You build it, you run it
INTRODUCTION TO DEVOPS
Automation
CI/CD pipelines helps automate change
management, testing, and deployment
Automation is a time-saver
Developers focus on creativity, not the
simple tasks
Without CI/CD pipelines, testing and
deployment must be done manually
INTRODUCTION TO DEVOPS
Post-mortem
All software is prone to errors
Something can always go wrong
When a software problem is identified:
First solve it
Then review what went wrong
This review is called Post-Mortem
No one is blamed in the Post-Mortem
INTRODUCTION TO DEVOPS
How this issue could have been How this issue could have been
prevented? better handled?
Common Answers More observability helps with:
More testing Earlier detection of issues
More code review Earlier detection of the root cause
INTRODUCTION TO DEVOPS
Let's practice!
INTRODUCTION TO DEVOPS
Tying it all together
INTRODUCTION TO DEVOPS
Cem Sakarya
DevOps Risk Advisor
What is DevOps after all?
DevOps is a combination of: It helps ensure high productivity, high quality,
and managed costs.
tools
technologies
cultural elements
Software Development -> DevOps
Data Engineering -> DataOps
Machine Learning Engineering -> MLOps
INTRODUCTION TO DEVOPS
What is DevOps about?
Reorganizes into autonomous teams
Collaboration of teammates with different
backgrounds
Teams being responsible for different parts
of software
Automating change management
Testing
Experimentation
INTRODUCTION TO DEVOPS
Why is DevOps necessary?
It is very efficient compared to other systems.
Automation and collaborations boost productivity
Testing and observability is necessary for reliability
Experimentation helps with product quality
INTRODUCTION TO DEVOPS
When is DevOps applicable?
DevOps principles are applicable most of
the time.
It is necessary for large organizations.
INTRODUCTION TO DEVOPS
What are the components of DevOps?
INTRODUCTION TO DEVOPS
INTRODUCTION TO DEVOPS
Let's practice!
INTRODUCTION TO DEVOPS
CONGRATS!
INTRODUCTION TO DEVOPS
Cem Sakarya
DevOps Risk Advisor
What did we learn?
1. Introducing ourselves to DevOps and its
main components
2. Looking deeper into modern software
architecture and infrastructure
3. Exploring the data engineering
applications of DevOps and DevOps
reporting
4. Ensuring reliability and data quality
INTRODUCTION TO DEVOPS
You can use your skills to...
navigate through software development
for software design, development, and testing
apply data engineering implications of DevOps
INTRODUCTION TO DEVOPS
Time to celebrate!
INTRODUCTION TO DEVOPS
