B.Sc.
Data Science
Academic Year – 2024-25
Semester -V
[BSCDS30]: Machine Learning Operations (ML Ops)
UNIT- VI
Kubeflow Pipelines on AI Platform
Prof. Mugdha Dharmadhikari
What is Kubeflow ?
• Kubeflow is an open source Kubernetes-native platform for developing, orchestrating,
deploying, and running scalable and portable ML workloads.
• Kubeflow is a community and ecosystem of open-source projects to address each stage in
the machine learning (ML) lifecycle with support for best-in-class open source tools and
frameworks.
• Kubeflow makes AI/ML on Kubernetes simple, portable, and scalable.
• Kubeflow offers modular and scalable tools that cater to all aspects of the ML lifecycle:
from building ML models to deploying them to production for AI applications
What is Kubeflow Platform ?
• The Kubeflow Platform refers to the full suite of Kubeflow components bundled together
with additional integration and management tools.
• Using Kubeflow as a platform means deploying a comprehensive ML toolkit for the entire
ML lifecycle.
The Kubeflow Platform includes:
• Kubeflow Notebooks for interactive data exploration and model development.
• Central Dashboard for easy navigation and management with Kubeflow Profiles for
access control.
• Additional tooling for data management (PVC Viewer), visualization (TensorBoards), and
more.
Kubeflow Overview Diagram :
Kubeflow’s Components:
1. Central Dashboard
2. Kubeflow Notebooks
3. Kubeflow Pipelines
4. Katib
5. Model Registry
6. Spark Operator
7. Training Operator
1. Kubeflow Central Dashboard :
• The Kubeflow Central Dashboard provides an authenticated web interface for
Kubeflow and ecosystem components.
• It acts as a hub for your machine learning platform and tools by exposing the UIs of
components running in the cluster.
2. Kubeflow Notebooks :
• Kubeflow Notebooks provides a way to run web-based development environments
inside your Kubernetes cluster by running them inside Pods.
Some key features include:
• Native support for JupyterLab, RStudio, and Visual Studio Code (code-server).
• Users can create notebook containers directly in the cluster, rather than locally on their
workstations.
• Admins can provide standard notebook images for their organization with required
packages pre-installed.
3. Kubeflow Pipelines :
• Kubeflow Pipelines (KFP) is a platform for building and deploying portable and scalable
machine learning (ML) workflows using Docker containers.
• With KFP you can author components and pipelines using the KFP Python SDK
4. Katib:
• Katib is a Kubernetes-native project for automated machine learning (AutoML). Katib
supports hyperparameter tuning, early stopping and neural architecture search (NAS)
• Katib is the project which is agnostic to machine learning (ML) frameworks. It can tune
hyperparameters of applications written in any language of the users’ choice and
natively supports many ML frameworks, such as TensorFlow, MXNet, PyTorch,
XGBoost, and others.
5. Model Registry
A model registry is an important component in the lifecycle of AI/ML models, an integral
component for any MLOps platform and for ML workflows.
A model registry provides a central index for ML model developers to index and manage
models, versions, and ML artifacts metadata. It fills a gap between model experimentation
and production activities. It provides a central interface for all stakeholders in the ML
lifecycle to collaborate on ML models.
▪ Kubeflow Pipelines :
• Kubeflow Pipelines is an open source machine learning platform designed to orchestrate
and automate the building, deployment, and management of machine learning workflows
on Kubernetes, with the ability to implement every step of operations on development of
machine learning models
• By leveraging Kubeflow Pipelines, organizations can streamline their machine learning
operations and enable collaboration between team members, data scientists and machine
learning engineers.
▪ CI/CD :
• This powerful approach automates key stages of the development process, from code
changes to deployments, allowing teams to release high-quality software faster and with
fewer errors.
What is Continuous Integration(CI) ?
Continuous integration (CI) is a software development practice that automates the process of integrating
code changes from multiple developers into a central repository. In MLOps, continuous integration (CI)
focuses on automating the building, testing, and packaging of machine learning pipelines whenever
there are changes to the codebase in the version control system.
What is Continuous Delivery/Deployment (CD) ?
Continuous deployment is a strategy in software development where code changes to an application are
released automatically into the production environment.
CD extends CI by automatically delivering or deploying the integrated code to production. In ML, this
could involve deploying updated models or datasets to a production environment.
