Openshift/Kubernetes integration with Apache YARN

Openshift/Kubernetes integration with Apache YARN

• 1.
  Page1 © HortonworksInc. 2014 OpenShift scheduling docker containers in YARN via Kubernetes
• 2.
  Page2 © HortonworksInc. 2014 Static website Web frontendUser DB Queue Analytics DB Development VM QA server Public Cloud Contributor’ s laptop Docker: a shipping container system for codeMultiplicityofStacks Multiplicityof hardware environments Production Cluster Customer Data Center Doservicesandapps interact appropriately? CanImigrate smoothlyandquickly …that can be manipulated using standard operations and run consistently on virtually any hardware platform An engine that enables any payload to be encapsulated as a lightweight, portable, self-sufficient container…
• 3.
  Page3 © HortonworksInc. 2014 Why are Docker containers lightweight? HDP 2.1 Hortonworks Data Platform I/O performance comparable to Bare Metal
• 4.
  Page4 © HortonworksInc. 2014 Kubernetes – container orchestrator • Service for container cluster management • Allows deploying and managing applications running on multiple hosts using docker • Open sourced by Google • Supports GCE, CoreOS, Azure, vSphere • Used to manage Docker containers as a default implementation • Master – maintain state of Kubernetes Server runtime • Scheduler, API server, registries, storage • Minions – represent the host were containers created • Kubelet – manage pod and container lifecycle
• 5.
  Page5 © HortonworksInc. 2014 OpenShift • Red Hat’s platform as a service for applications in the cloud that supports both public and private cloud • Provides high level abstraction for applications on top of containers allowing easy scaling, service discovery, and deployment • Enable Docker image authors to easily deliver reusable application components, including highly available databases, monitoring and log aggregation tools, service discovery platforms, and prepackaged web applications • Allow developers to deeply customize their runtime environments while preserving operational support at scale for those applications
• 6.
  Page6 © HortonworksInc. 2014 6 Kubernetes/YARN Integration YARN Node Manager YARN Node Manager Kubernetes AppMaster
• 7.
  Page7 © HortonworksInc. 2014 Demo: OpenShift scheduling docker container in YARN Running in docker
• 8.
  Page8 © HortonworksInc. 2014 Demo: 1. Launch Docker container on YARN via OpenShift #openshift kube create pods –c demo.json #cat demo.json
• 9.
  Page9 © HortonworksInc. 2014 Demo: 2. Enter into container and confirm web server started #openshift kube create pods –c demo.json
• 10.
  Page10 © HortonworksInc. 2014 Demo: 3. Webapp on Openshift interacting with Hadoop
• 11.
  Page11 © HortonworksInc. 2014 Demo: 4. Launch other workloads on YARN YARN allows Big Data and other workloads to run side by side on same cluster
• 12.
  Page12 © HortonworksInc. 2014 Questions?