Downloaded 137 times
Uploaded byJonathan Weiss
Build your own clouds with Chef and MCollective
The document discusses building cloud infrastructure using tools like Chef and MCollective, highlighting their roles in configuration, deployment, and management. It covers the components involved in cloud computing, including automation, host configuration, and communication responsibilities. The content also provides insights into Chef's workflow, configuration elements, and MCollective's architecture for orchestrating server tasks.
More Related Content
![[AWS Dev Day] 앱 현대화 | AWS Fargate를 사용한 서버리스 컨테이너 활용 하기 - 삼성전자 개발자 포털 사례 - 정영준...](https://cdn.slidesharecdn.com/ss_thumbnails/appmodernizationawsfargate-190930044252-thumbnail.jpg?width=600ounds&width=560&fit=bounds)
Similar to Build your own clouds with Chef and MCollective
Build your own clouds with Chef and MCollective
- 1. Build your ownclouds with Chef & MCollective Jonathan Weiss
- 2. Who am I? Workingfor Peritor in Berlin, Germany Written, maintain, or involved in Webistrano Capistrano SimplyStored Happening The great fire of London http://github.com/jweiss @jweiss 2
- 3. Scalarium Amazon EC2 ClusterManagement Auto-Config Self-Healing Auto-Scaling One-click-deployment www.scalarium.com 3
- 4. Cloud Computing Cloud Computing 4
- 5.
- 6.
- 7. What you get Open Solaris Oracle Suse Windows Linux Ubuntu CentOS Red IBM Hat Infrastructure as a Service
- 8. What you reallywant Load balancer App server Database Fully configured stack of servers, dependencies, firewalls, … + your own application deployed
- 9. Automation is needed for • Configuration • Healing • Scaling • Deployment flickr.com/photos/torek/3788181603
- 10.
- 11. Reasons to buildyour own In-house capacity Governance: Security and regulations Flexibility (or lack of…) 11
- 12. Ingredients 12
- 13. Components Communication Host configuration Command & Control
- 14. Command & Control Responsibilities: Communication with IaaS API User Interface (Web, CLI, API): Query, deploy, start, stop Repository of projects &configuration Controls which hosts do what&when 14
- 15. Command & Control Responsibilities: Implementation: Communication with IaaS API Fog & other API implemenations User Interface (Web, CLI, API): Rails Query, deploy, start, stop Repository of projects &configuration CouchDB Controls which hosts do what&when Async wokers & agents 15
- 16. Communication Responsibilities: Distributecommands and responses in a scalable & fault tolerant way 16
- 17. Communication Responsibilities: Implementation: Distribute commands and responses MCollective / ActiveMQ in a scalable & fault tolerant way 17
- 18. Host Configuration Responsibilities: Actually executes commands Responds to cluster changes Ensures local host is configured according to commands: Install software, start/stop services, ... 18
- 19. Host Configuration Responsibilities: Implementation: Actually executes commands Chef Responds to cluster changes MCollective Agents Ensures local host is configured according to commands: Install software, start/stop services, ... 19
- 20. Host Configuration Bootstrapping newservers 20
- 21. Chef Open Source ProvisioningTool Configures your servers Cookbooks & recipes Ruby DSL 21
- 22. Chef 22
- 23. Chef Server 23
- 24. Chef Solo 24
- 25. Chef-Solo vs. Chef-Server Chef-Server Complex No concept of different environments, e.g. staging vs. production Didn‘t want to run one Chef server per project (and env) No lifecycle events, e.g. deployment vs. bootstrap Pull vs. push Chef-Solo MCollective agents to push cookbooks and Chef runs 25
- 26. Cookbooks Cookbooks apache2 memcached mysql rails 26
- 27. Memcached Cookbook Cookbooks memcachd attributes recipes templates 27
- 28. Memcached Cookbook memcached attributes default.rb recipes service.rb install.rb templates default memcached.conf.erb 28
- 29. Attributes cookbooks/memcached/attributes/default.rb Set default valuesfor variables to use in recipes & templates 29
- 30.
- 31.
- 32.
- 33. Chef Solo Run Configuration JSON: /etc/chef/solo.json Run Chef Solo 33
- 34. Available Resources CookbookFile Ifconfig SCM Cron Link Script Deploy Log Service Directory Mdadm Subversion Erlang Call Mount Template Execute Package User File Remote Directory Git Remote File Group Route HTTP Request Ruby Block 34
- 35. Deployment with Chef Deployresource Compatible with Capistrano layout Supports Capistrano hooks via deploy/hook_name.rb Handles source checkout & callbacks Call custom script Arbitrary options, e.g: Download WAR file Compile Source … Automatically wrapped in Chef error handling & notifications 35
- 36.
- 37. Communication Relaying messages toservers 37
- 38. Marionette Collective „a frameworkto build server orchestration or parallel job execution systems“ 38
- 39.
- 40. RPC Calls Service:chef-run Service: update-cookbooks Client Discover agents Service: chef-run Service: update-cookbooks Message Routing Agents
- 41. RPC Calls Service:chef-run Service: update-cookbooks Client Service: chef-run Service: update-cookbooks Call agents Message Routing Agents
- 42.
- 43.
- 44.
- 45. Filters Facts Limitresponding agents by filters of facts/classes: Classes Arbitary used defined settings role Can be set by userdata Facts Information about local machine Plugins for Chef/Ohai, Puppet/Facter Examples: Linux version, installed packages, .... 45
- 46. Filters on theCommand Line 46
- 47.
- 48. MCollective Chef CallChef action on remote agents/machines 48
- 49. MCollective Chef Agentrunning Chef 49
- 50. Security Clients Stomp/ ActiveMQ user AES+RSA plugins: encrypt sign every message SSL plugin: sign every message TLS for connection encryption Middleware Topic permissions and subclusters Agents All client security measures available RPC authorization and auditing 50
- 51. Scalability Redundancy ActiveMQoffers different means Network of brokers Broker clusters Master/Slave 51
- 52. Command Control Controllingthe cloud 52
- 53. Command Software General capabilities Stores cluster model Interacts with IaaS API (EC2, VMWare friends) Handles recovery presence Integrates communication solution Generates host configuration / Chef events Custom requirements User management authorization Business rules processes Deployment recipes scripts 53
- 54. Server Roles 54
- 55.
- 56. Experiences Chef Flexibleand powerful Easy to learn first steps - easy to write spaghetti recipes Some annoyances: pseudo idempotent and 2 phases “Write once, test everywhere” MCollective Simple and robust Missing presences and events (Nanite has them but has different problems) Integrated with Puppet in the future? 56
- 57. QA Peritor GmbH Blücherstr. 22,Hof III Aufgang 6 10961 Berlin Tel.: +49 (0)30 69 20 09 84 0 Fax: +49 (0)30 69 20 09 84 9 Internet: www.peritor.com E-Mail: info@peritor.com © Peritor GmbH - Alle Rechte vorbehalten
- 58.
- 59. Will never happen,even if /data/logs/redhat will be created! 59
- 60. Two Phases ofChef 1) Compile Load all attributesrecipes and build list of actions/dependencies Compute attribute tree 2) Run Random Ruby Code runs here Actually execute the resources 60
- 61. Two Phases ofChef Use only_if and not_if Tell Chef to execute your ressource during compile phase 61