Python Deployment with Fabric

Prodution Architecture and Deployment with Fabric - Andy McCurdy - @andymccurdy

Basic Config (web) Apache mod_wsgi use daemon mode threads more efficient processes if you're unsure of thread safety WSGIDaemonProcess my-site python-path=/home/code/ processes=2 threads=150 maximum-requests=5000 WSGIProcessGroup my-site WSGIScriptAlias / /home/code/my-site/deploy/wsgi/my-site.wsgi

Basic Config (media) Nginx Use Nginx to proxy traffic to Apache Meanwhile Nginx serves media upstream my-site { server 127.0.0.1:8000; } server { listen 80; location ~ ^/media/ { root /home/code/my-site; expires 30d; } location / { proxy_pass http://my-site; proxy_set_header X-Real-IP $remote_addr; } }

Basic Config (db) Databases PostgreSQL Frank Wiles @ www.revsys.com MySQL Percona @ www.mysqlperformanceblog.com

Basic Config (cache) Use Memcached! Even 16mb will significantly help against a Digg or being Slashdot'ed It's incredibly easy... # settings.py MIDDLEWARE_CLASSES = ( 'django.middleware.cache.UpdateCacheMiddleware', 'django.middleware.common.CommonMiddleware', 'django.middleware.cache.FetchFromCacheMiddleware') CACHE_BACKEND = 'memcached://127.0.0.1:11211/' CACHE_MIDDLEWARE_SECONDS = 60*5 # 5 minutes CACHE_MIDDLEWARE_ANONYMOUS_ONLY = True

Basic Deployment Copy Code & Media (rsync or scp) Run DB migrations / syncdb Bounce wsgi daemons

Managing Growth (db) Move DB to its own server As much RAM as possible Write-heavy (>10%)? Get fast disks Tune your config file

Managing Growth (web) Add more web servers Use a resource monitoring tool like Munin to understand if your app is CPU or memory bound

Even More Growth Replicated or sharded Databases Multiple load balancers for redundancy Message queues Crons Search daemons (Solr, Sphinx) etc...

(not so) Basic Deployment image credit: Brad Fitzpatrick

Deployment Reqs Copy media to CDN Maintenance splash page Run DB migrations Install/Upgrade Python dependencies Add a new web server to the cluster Execute arbritrary commands for sysadmin maintenance tasks

Deployment Options Capistrano + Out of box support for common use cases + Hooks to customize tasks + Source control integration + Threaded deployment to multiple hosts - Ruby :(

Deployment Options Fabric + Very simple, tasks are just Python functions + Easy to chain together tasks to create complex scripts out of bite size pieces - No source control integration - No out of box support - Some bugs, although fairly easy to work around, and new maintainer is working on fixes

Fabric Basics sudo easy_install fabric need a fabfile.py from fabric.api import * be mindful of tasks that may fail each remote command starts fresh changing directories

Core Functionality local() - Run a command locally run() - Run a command remotely sudo() - Run a command remotely as another user put() - Copy a file from local to remote get() - Copy a file from remote to local many more helper-ish commands

Authentication Relies on SSH model Use SSH keys Control access to root user via sudoers When you have to revoke access, you just turn off their SSH account

Configuration Fabric environment (env) -- it's just a dictionary Hosts and Roles Code Repositories Whatever you need ~/fabricrc Global settings or all Fabric deployments SSH username

Example Config # fabfile.py from fabric.api import * env.roledefs = { 'web' : ['10.1.1.1', '10.1.1.2'], 'db' : ['10.1.1.3'], 'lb' : ['10.1.1.4'], } env.repositories = {...}

Tasks # fabfile.py def uptime(): run('uptime') $> fab uptime -H 10.1.1.3 [10.1.1.3] run: uptime [10.1.1.3] out: 05:20:39 up 88 days, 12:00, 0 users, load average: 0.03, 0.03, 0.00

Mapping Roles to Tasks # fabfile.py @roles('web') def uptime(): run('uptime') $> fab uptime [10.1.1.1] run: uptime [10.1.1.1] out: 05:20:39 up 88 days... [10.1.1.2] run: uptime [10.1.1.2] out: 05:20:39 up 88 days...

Decorator Problems Some problems with Fabric's role management Can't override decorated tasks at command line as docs suggest def default_roles(*role_list): def selectively_attach(func): if not env.roles and not env.hosts: return roles(*role_list)(func) else: if env.hosts: func = hosts(*env.hosts)(func) if env.roles: func = roles(*env.roles)(func) return func return selectively_attach

All better now #fabfile.py @default_roles('web', 'db') def uptime(): run('uptime') $> fab uptime # runs on all hosts in the 'web' and 'db' roles $> fab uptime --roles lb # runs only on hosts in the 'lb' role

Dealing with Failures By default Fabric dies if a task fails Use a context manager when failures are anticipated # fabfile.py from __future__ import with_statement # py2.5 def symlink_me(): with settings(warn_only=True): run('rm /path/to/symlink') run('ln -s /home/andy /path/to/symlink')

Easy sys-admin Make an "invoke" command Great for sys-admin and one-off tasks # fabfile.py @default_roles('all') def invoke(command): "Invoke an arbritrary command" sudo(command) # install new packages on all hosts in one command $> fab invoke:"apt-get install git-core"

Real World Tasks $> fab --list Available commands: bounce_wsgi_procs Bounce the WSGI procs by touching the files deploy Full deployment deploy_media Push media to S3 invoke Invoke an arbritrary command migrate Run any migrations via South reload_nginx Update Nginx's running config splash_off Configure Nginx to serve the site splash_on Configure Nginx to serve a downed-site page update_repositories Push code to servers update_dependencies Update dependencies to third party libs

Whiskey's Deployment def deploy(splash='no'): "Full deployment" deploy_media() update_cached_repositories() update_dependencies() generate_releases() if splash == 'yes': splash_on() _symlink_code() migrate() bounce_wsgi_procs() if splash == 'yes': splash_off() $> fab deploy:splash=yes

Python Deployment with Fabric

More Related Content

What's hot

Viewers also liked

Similar to Python Deployment with Fabric

Python Deployment with Fabric

Editor's Notes