Bridging the Developer and the Datacenter

Bridging the Developer and the Datacenter

• 1.
  Bridging
the
Developer
 and
the
Datacenter
 June
2009

• 2.
  Brief
History
 •  8
years
in
managed
services
industry
 •  Founded
first
company
in
2001,
sold
in
2004
 • Founded
Redux
in
2004,
sole
owner
 •  Managing
200
servers,
1200
VMs
across
two
faciliCes
in
 Portland
 •  Strong
OSS
Advocates
–
Adium,
ImageMagick,
Pidgin,
 SimpleMachines,
Camino
Browser
 Property
of
Network
Redux,
LLC

• 3.
  Why
should
we
build
a
bridge?
 •  Hosted
applicaCons
require
communicaCon
between
 datacenters
and
developers
 •  Provide
developers
with
the
tools
to
power
decision
making
 •  Become
less
reliant
on
hosCng
providers
for
troubleshooCng
 •  Understand
the
hosCng
service
stack
 Property
of
Network
Redux,
LLC

• 4.
  The
Network
 Data Center / Bandwidth •  Ethernet
is
the
standard
 •  10Mb,
100Mb,
1Gb,
10Gb
and
now
 100Gb
 •  1Mbps
sustained
equals
~320GB
 •  BursCng
is
important,
know
your
limits
 •  >
70%
sustained
usage
is
dangerous
 Property
of
Network
Redux,
LLC

• 5.
  BursCng
Example:
Twi]er
Vote
Report
 Property
of
Network
Redux,
LLC

• 6.
  Data
Transfer
 •  Core,
DistribuCon,
Access
 •  Border
Gateway
Protocol
(BGP)
 • Virtual
Router
Redundancy
 Protocol
(VRRP)
 •  Bandwidth
is
a
commodity
 •  Redundancy
has
to
happen
at
 all
layers
 •  Quality
over
quanCty
 •  Carrier
neutral
is
key
 •  ethtool,
traceroute,
tcpdump
 Property
of
Network
Redux,
LLC

• 7.
  Data
Backups
 •  Do
not
rely
on
the
host
for
disaster
recovery
 •  SLA’s
are
careful
to
indemnify
your
provider
 •  If
you
are
storing
client
data,
you
need
to
back
it
up
 and
indemnify
yourself
 •  RAID
is
NOT
a
backup
strategy
 •  Understand
the
retrieval
process
 •  Verify
the
integrity
of
your
backup
archives
 Property
of
Network
Redux,
LLC

• 8.
  Two
Common
Approaches
 Pull – Agent/ Host Push – server to server •  Agent
resides
on
server
 • 

Server
pushes
data
 •  Master
pull
servers
 • 

Rsync
/
rsnapshot
 •  Expensive
 • 

Inexpensive
 •  Proprietary
 • 

Easy
to
deploy
 •  Very
Reliable
 • 

As
reliable
as
you
design
it
 •  Many
commercial
 • 

More
home
grown
soluCons
 vendors
 Property
of
Network
Redux,
LLC

• 9.
  The
Agent
/
Host
Model
 •  Commonly
deployed
in
hosted
environments
 •  Works
at
the
chassis
level
 •  More
oden
than
not
a
kernel
module
 •  Can
provide
CDP
(ConCnuous
Data
ProtecCon)
 •  EncrypCon,
Disk
VerificaCon
 •  CDP
from
R1Sod
is
hieng
the
sweet
spot
 Property
of
Network
Redux,
LLC

• 10.
  Common
Issues
 •  Don’t
assume
your
provider
is
backing
up
 •  Don’t
assume
the
backup
data
has
been
verified
 •  Data
is
ulCmately
the
developers
responsibility
 •  Be
careful
with
providers
not
willing
to
backup
your
data
 •  Have
your
own
disaster
recovery
plan
 Property
of
Network
Redux,
LLC

• 11.
  Power
 •  Unbelievably
criCcal,
most
 common
point
of
failure
in
a
 facility
 •  Power
creates
heat,
and
 heat
causes
hardware
to
fail
 •  System
Administrators
are
 not
Electricians
 Property
of
Network
Redux,
LLC

• 12.
  Key
Points
on
Power
 •  Redundant
power
supplies
 need
to
route
through
 diverse
power
paths
 •  Automated
Transfer
 Switches
(ATS)
can
be
a
 single
point
of
failure
 Property
of
Network
Redux,
LLC

• 13.
  Servers
–
Resources
and
Metrics
 •  CPU,
Memory
and
Disk
IO
 •  Disk
IO
is
the
most
commonly
 overlooked
and
least
 understood
bo]leneck
 •  We
rarely
see
CPU’s
as
the
 bo]lenecks
for
applicaCon
 hosCng
 •  Memory
is
an
easy
to
 understand,
easy
to
inflate/ deflate
commodity
 •  Things
oden
find
their
way
back
 to
slow
disk
access
Cmes
 Property
of
Network
Redux,
LLC

• 14.
  Narrowing
in
on
Disk
I/O
 •  Disks
are
the
slowest
component
 •  SATA
and
SAS
are
widely
in
use
 •  RAID
–
Balancing
performance
and
cost.
 •  Don’t
confuse
memory
shorialls
with
disk
access
problems
 •  Disks
are
the
most
common
component
to
fail
 •  sar
and
iostat
are
valuable
tools
 •  Profile
your
environment
to
best
gauge
your
requirements
 Property
of
Network
Redux,
LLC

• 15.
  Profiling
your
requirements
 •  System
metrics
are
criCcal
 Sometimes performance tuning must be “out of the box.” Analysis on the •  Each
service
layer
is
its
own
 microscopic level should regularly be retired to more macroscopic views. This discussion
 multiresolutioned problem analysis can turn a question such as “How can I •  Caching
will
save
the
day
 merge all these log files faster?” into “Why do I have all these log files to •  Log
files
are
underrated
 merge and is there a better way?” Or a question such as “How can I make this set of problematic database queries •  sar
and
vmstat
are
fantasCc
 faster?” becomes “Why am I putting this information in a database?” •  Profiling
your
applicaCon
 Theo Schlossnagle requirements
will
ensure
 Scalable Internet Architectures efficient
spending
for
services
 Property
of
Network
Redux,
LLC

• 16.
  networkredux.com