KEMBAR78
Open Source Systems Administration | PDF
Open Source Systems 
Administration 
Elizabeth K. Joseph 
@pleia2 
HP 
OpenStack Infrastructure Team 
All Things Open, 22 October 2014
Elizabeth K. Joseph 
● Linux Systems Administrator 
● Ubuntu Community Council Member & Co-author 
of The Official Ubuntu Book, 8th Edition 
● Director for the non-profit Partimus.org 
● Really likes open source software
The OpenStack Infrastructure team maintains the 
servers OpenStack developers use worldwide to 
contribute to OpenStack 
Elizabeth K. Joseph | @pleia2 3
The OpenStack Infrastructure is also an Open 
Source project 
(Other projects and companies even use it!) 
Elizabeth K. Joseph | @pleia2 4
All code repositories are available online: 
https://git.openstack.org/cgit/openstack-infra 
Elizabeth K. Joseph | @pleia2 5
Changes are proposed by anyone through a 
public code review system (Gerrit): 
https://review.openstack.org/ 
Elizabeth K. Joseph | @pleia2 6
We use OpenStack's CI 
Infrastructure 
● Launchpad (coming soon: StoryBoard) 
Elizabeth K. Joseph | @pleia2 7 
● Git 
● Gerrit 
● Zuul* 
● Gearman 
● Jenkins (with jenkins-job-builder*) 
● Nodepool* (with devstack-gate*) 
* Started by the OpenStack Infrastructure team
OpenStack CI Workflow 
Gerrit code review Zuul 
pipy mirrors & other 
semi-privileged servers 
Elizabeth K. Joseph | @pleia2 8 
Local changes 
submitted via 
git-review 
Jenkins (gearman-plugin) 
Git repository 
Jenkins01,02... ubuntu, fedora, 
centos & 
devstack slaves 
Github mirror git.openstack.org 
mirror 
Gearman Server
As a user: git-review and Gerrit 
Elizabeth K. Joseph | @pleia2 9 
elizabeth@r2d2$:~/git/openstack­infra$ 
git clone git://git.openstack.org/openstack­infra/ 
publications 
elizabeth@r2d2$:~/git/openstack­infra/ 
publications$ vim index.html 
elizabeth@r2d2$:~/git/openstack­infra/ 
publications$ git commit ­a 
[toolsconfigs/updates 0c63f4a] Update tools and review purposes. 
1 file changed, 4 insertions(+) 
elizabeth@r2d2$:~/git/openstack­infra/ 
publications$ git review 
Amending the commit to add a gerrit change id. 
remote: Resolving deltas: 100% (2/2) 
remote: Processing changes: new: 1, refs: 1, done 
remote: 
remote: New Changes: 
remote: https://review.openstack.org/128722 
remote: 
To ssh://lyz@review.openstack.org:29418/openstack­infra/ 
publications.git 
* [new branch] HEAD ­> 
refs/publish/sysadmin­codereview/ 
toolsconfigs/updates
Elizabeth K. Joseph | @pleia2 10
Elizabeth K. Joseph | @pleia2 11
What uses this CI system 
All core OpenStack projects 
StackForge projects 
Documentation 
Translations imports 
Project specifications 
Systems Administration 
Elizabeth K. Joseph | @pleia2 12
What we do as OpenStack Infrastructure Systems 
Administrators 
Elizabeth K. Joseph | @pleia2 13
Elizabeth K. Joseph | @pleia2 14
Elizabeth K. Joseph | @pleia2 15
Maintain a bunch of stuff 
● Continuous Integration systems 
Elizabeth K. Joseph | @pleia2 16 
● Cacti 
● Elasticsearch, Logstash and Kibana 
● IRC Bots 
● Etherpad 
● Git 
● Paste 
● Planet 
● Puppetboard 
● Mailing Lists 
● Various smaller web services 
● Wiki
Test our patches 
● flake8 (pep 8 and pyflakes) 
Elizabeth K. Joseph | @pleia2 17 
● bashate 
● puppet parser validate 
● puppet-lint 
● Beaker 
● XML 
● Alphabetized project files
Code review benefits: Peer review 
● Multiple eyes on changes prior to merging 
● Good infrastructure for developing new 
solutions (particularly for a distributed team) 
● No special process to go through for commit 
access 
● Trains us to be collaborative by default 
Elizabeth K. Joseph | @pleia2 18
How do you know what's going on with your 
servers? 
Elizabeth K. Joseph | @pleia2 19
Cacti: http://cacti.openstack.org/ 
Elizabeth K. Joseph | @pleia2 20
Puppetboard: http://puppetboard.openstack.org/ 
Elizabeth K. Joseph | @pleia2 21
Documentation: http://ci.openstack.org/ 
Elizabeth K. Joseph | @pleia2 22
Security 
The whole world knows what we run and how we run it. 
We need to be diligent about locking down services that believe they live behind 
an authentication wall. 
And when we're not? Be aware of the risks and prepared to deal with the 
consequences. 
Elizabeth K. Joseph | @pleia2 23
We need to fully trust some people 
We can't give access to everything to everyone on the Internet and 
we can't do everything through code review 
Some shell access is still required 
(viewing logs, migrations, unusual service failures) 
Private management of sensitive data 
(keys, certificates, passwords) 
Core reviewers have the power to push anything that passes tests through 
(self-approval, patch ninja-ing) 
Elizabeth K. Joseph | @pleia2 24
Questions? 
Documentation: http://ci.openstack.org/ 
Contact me: lyz@princessleia.com 
Elizabeth K. Joseph | @pleia2 25
Credits 
CC BY-SA slide background source: "The gate's 
unlocked!!! by Tripp" 
https://www.flickr.com/photos/trippchicago/3769904793 
The whole OpenStack Infrastructure team for the 
extraordinary work being done! 
Elizabeth K. Joseph | @pleia2 26

Open Source Systems Administration

  • 1.
    Open Source Systems Administration Elizabeth K. Joseph @pleia2 HP OpenStack Infrastructure Team All Things Open, 22 October 2014
  • 2.
    Elizabeth K. Joseph ● Linux Systems Administrator ● Ubuntu Community Council Member & Co-author of The Official Ubuntu Book, 8th Edition ● Director for the non-profit Partimus.org ● Really likes open source software
  • 3.
    The OpenStack Infrastructureteam maintains the servers OpenStack developers use worldwide to contribute to OpenStack Elizabeth K. Joseph | @pleia2 3
  • 4.
    The OpenStack Infrastructureis also an Open Source project (Other projects and companies even use it!) Elizabeth K. Joseph | @pleia2 4
  • 5.
    All code repositoriesare available online: https://git.openstack.org/cgit/openstack-infra Elizabeth K. Joseph | @pleia2 5
  • 6.
    Changes are proposedby anyone through a public code review system (Gerrit): https://review.openstack.org/ Elizabeth K. Joseph | @pleia2 6
  • 7.
    We use OpenStack'sCI Infrastructure ● Launchpad (coming soon: StoryBoard) Elizabeth K. Joseph | @pleia2 7 ● Git ● Gerrit ● Zuul* ● Gearman ● Jenkins (with jenkins-job-builder*) ● Nodepool* (with devstack-gate*) * Started by the OpenStack Infrastructure team
  • 8.
    OpenStack CI Workflow Gerrit code review Zuul pipy mirrors & other semi-privileged servers Elizabeth K. Joseph | @pleia2 8 Local changes submitted via git-review Jenkins (gearman-plugin) Git repository Jenkins01,02... ubuntu, fedora, centos & devstack slaves Github mirror git.openstack.org mirror Gearman Server
  • 9.
    As a user:git-review and Gerrit Elizabeth K. Joseph | @pleia2 9 elizabeth@r2d2$:~/git/openstack­infra$ git clone git://git.openstack.org/openstack­infra/ publications elizabeth@r2d2$:~/git/openstack­infra/ publications$ vim index.html elizabeth@r2d2$:~/git/openstack­infra/ publications$ git commit ­a [toolsconfigs/updates 0c63f4a] Update tools and review purposes. 1 file changed, 4 insertions(+) elizabeth@r2d2$:~/git/openstack­infra/ publications$ git review Amending the commit to add a gerrit change id. remote: Resolving deltas: 100% (2/2) remote: Processing changes: new: 1, refs: 1, done remote: remote: New Changes: remote: https://review.openstack.org/128722 remote: To ssh://lyz@review.openstack.org:29418/openstack­infra/ publications.git * [new branch] HEAD ­> refs/publish/sysadmin­codereview/ toolsconfigs/updates
  • 10.
    Elizabeth K. Joseph| @pleia2 10
  • 11.
    Elizabeth K. Joseph| @pleia2 11
  • 12.
    What uses thisCI system All core OpenStack projects StackForge projects Documentation Translations imports Project specifications Systems Administration Elizabeth K. Joseph | @pleia2 12
  • 13.
    What we doas OpenStack Infrastructure Systems Administrators Elizabeth K. Joseph | @pleia2 13
  • 14.
    Elizabeth K. Joseph| @pleia2 14
  • 15.
    Elizabeth K. Joseph| @pleia2 15
  • 16.
    Maintain a bunchof stuff ● Continuous Integration systems Elizabeth K. Joseph | @pleia2 16 ● Cacti ● Elasticsearch, Logstash and Kibana ● IRC Bots ● Etherpad ● Git ● Paste ● Planet ● Puppetboard ● Mailing Lists ● Various smaller web services ● Wiki
  • 17.
    Test our patches ● flake8 (pep 8 and pyflakes) Elizabeth K. Joseph | @pleia2 17 ● bashate ● puppet parser validate ● puppet-lint ● Beaker ● XML ● Alphabetized project files
  • 18.
    Code review benefits:Peer review ● Multiple eyes on changes prior to merging ● Good infrastructure for developing new solutions (particularly for a distributed team) ● No special process to go through for commit access ● Trains us to be collaborative by default Elizabeth K. Joseph | @pleia2 18
  • 19.
    How do youknow what's going on with your servers? Elizabeth K. Joseph | @pleia2 19
  • 20.
  • 21.
  • 22.
  • 23.
    Security The wholeworld knows what we run and how we run it. We need to be diligent about locking down services that believe they live behind an authentication wall. And when we're not? Be aware of the risks and prepared to deal with the consequences. Elizabeth K. Joseph | @pleia2 23
  • 24.
    We need tofully trust some people We can't give access to everything to everyone on the Internet and we can't do everything through code review Some shell access is still required (viewing logs, migrations, unusual service failures) Private management of sensitive data (keys, certificates, passwords) Core reviewers have the power to push anything that passes tests through (self-approval, patch ninja-ing) Elizabeth K. Joseph | @pleia2 24
  • 25.
    Questions? Documentation: http://ci.openstack.org/ Contact me: lyz@princessleia.com Elizabeth K. Joseph | @pleia2 25
  • 26.
    Credits CC BY-SAslide background source: "The gate's unlocked!!! by Tripp" https://www.flickr.com/photos/trippchicago/3769904793 The whole OpenStack Infrastructure team for the extraordinary work being done! Elizabeth K. Joseph | @pleia2 26