KEMBAR78
Operating Docker | PDF
operating docker securely
Jen  Andre
about me
@fun_cuddles  /  jenpire.com  /  organizer  @BostonGoLang  
co-­‐founder  @threatstack,    formely  Mandiant  /  Symantec  
**  I  HAVE  NO  AFFILIATION  WITH  THE  SMART  PEOPLE  AT  DOCKER  **
REALITY  CHECK
*  I  stole  this  shamelessly  from  @petecheslock
houston,  we  have  concerns
managing misinformation
VS
http://iamondemand.com/blog/5-key-benefits-of-docker-ci-version-control-portability-isolation-and-security/
http://www.securityweek.com/disrupting-disruptor-security-docker-containers
-­‐  your  security  engineer
“Docker needs to
fix all of these
problems.”
“It’s probably
fine.”
-­‐ -­‐your  hipster  devops    
-­‐ gal  or  guy
is docker secure?
this is the wrong question.
How  can  we  enable  organizaOons  (security  
+  dev  +  ops)  operate  Docker  safely?
1. Understand  your  threat  model  
2. Understand  Docker’s  limitaOons  
3. Understand  the  tools  available  to  us    
4. Evolve  the  Docker  security  ecosystem  :)
1.  How  do  I  trust  the  code?  
2.  How  do  I  know  if  I’ve  configured  my  Docker  host  +  
containers  in  a  secure  way?  
3.  How  do  containers  change  my  security  pracOces,  
e.g.  monitoring?
empathy: understanding the security
engineer’s concerns
but also consider
the  consistency  of  applicaOon  environments  in  Docker  
containers  provides  for  interesOng  opportuniHes  for  
new  automaOon  around  security  hardening,  audiOng,  
and  tesOng.
issues with trust
docker  images  are  binaries  (opaque)  
who  am  I  trusOng?  
who  is  updaOng  these  things  when  there  is  a  criOcal  
security  flaw?  
The problem of patch management is a
real thing.
http://www.banyanops.com/blog/analyzing-docker-hub/
confusing advice
http://serverfault.com/questions/611082/how-to-handle-security-updates-within-docker-containers
always be updating!
• Do  perform  security  upgrades  (debian  example  
below)  
• sudo	
  docker	
  exec	
  -­‐it	
  <container>	
  apt-­‐get	
  update	
  	
  
• sudo	
  docker	
  exec	
  -­‐it	
  <container>	
  apt-­‐get	
  upgrade	
  
-­‐s	
  |	
  grep	
  -­‐i	
  security	
  #	
  dry	
  run	
  
• sudo	
  docker	
  exec	
  -­‐it	
  <container>	
  apt-­‐get	
  upgrade	
  
#	
  commit	
  changes	
  when	
  done	
  
who are you
trusting?
what  if  someone  
replaced  libc  with  a  
backdoored  version?
community  
addressing trust
automate  policy  audiOng  +  
enforcement
for  a  given  container,  tell  me  who/what  
I  am  trusOng
build  from  a  trusted  base  image
be  aware  of  who  you  are  trusOng
don’t  overrely  on  Docker  hub
tooling  to  apply  and    
validate  security  updates
more advice
• “The  best  opOon  is  to  block  index.docker.io  locally,  and  
download  and  verify  images  manually  before  imporOng  
them  into  Docker  using  docker  load.”  
• hcps://Otanous.com/posts/docker-­‐insecurity#fn:4  
• Use  a  private  docker  registry  
• hcps://www.digitalocean.com/community/tutorials/how-­‐to-­‐set-­‐up-­‐a-­‐private-­‐docker-­‐registry-­‐
on-­‐ubuntu-­‐14-­‐04  ,  hcps://quay.io        
• Use  RedHat  cerOfied  containers  
•   hcp://www.redhat.com/en/about/press-­‐releases/red-­‐hat-­‐announces-­‐cerOficaOon-­‐for-­‐
containerized-­‐applicaOons-­‐extends-­‐customer-­‐confidence-­‐and-­‐trust-­‐to-­‐the-­‐cloud  
opportunities
• trust,  but  verify:  build  an  binary  audiOng  tool  for  Docker  
images!  
• e.g.,  scan  images,  validate  installed  libraries  and  binaries    do  
not  have  criOcal  security  issues  and  align  with  signed  package  
manifests.  
• hcps://github.com/banyanops/collector    +`cruh’  but  for  containers?  
• hcps://github.com/OpenSCAP/container-­‐compliance  -­‐  RHEL  only  
• contribute  to  the  packaging/distribuOon  trust  conversaOon!  
• hcps://github.com/docker/distribuOon/pull/179  
• references:  hcp://theupdateframework.com/  
best practices,
hardening, &
secure
configurations
]
security empathy
How  do  I  know  if  I’ve  
configured  my  Docker  host  +  
containers  in  a  secure  way?  
the good!
Docker  released  a  
comprehensive  
security  benchmark.
hcps://blog.docker.com/2015/05/understanding-­‐docker-­‐security-­‐and-­‐best-­‐pracOces/
the bad
…it’s  118  pages  of  material!
the good!: can we automate these
checks?
for  most  of  them,  yes!  
github.com/dockersecuritytools/bacen  -­‐  IN  PROGRESS
serverspec example
the problem of isolation
container hardening: the good
there’s actually a lot of knobs to turn!
• SELinux  /  AppArmor  policies  (—security-­‐opt)    (more  
about  this  later)  
• capabiliHes  (—cap-­‐add,  —cap-­‐drop)    
• “give  root  without  all  of  root”
• cgroups  (resource  allocaOon,  many  flags)  
• Example:  $ docker run -it --rm -m
128m fedora bash
• hcps://goldmann.pl/blog/2014/09/11/resource-­‐management-­‐in-­‐docker/  
• user  namespaces  (soon!)  so  you  don’t  have  to  run  
id=0  processes  as  root!  
• seccomp  filtering  to  permit  or  block  individual  
system  calls  (soon!)  
• hcp://opensource.com/business/15/3/docker-­‐security-­‐future
the bad
there’s a lot of knobs to turn :(
we can do better.
what  problems  are  they  trying  to  solve?
AppArmor  +  SELinux
a question
if  engineers  can’t  figure  out  how  to  build  and  apply  SELinux  policies  now,    
how  will  Docker  change  things?
apparmor
a  gentler  mandatory  access  control  system  
hcps://wiki.ubuntu.com/AppArmor  
introducing  
#include <tunables/global>
/usr/sbin/tcpdump {
#include <abstractions/base>
#include <abstractions/nameservice>
#include <abstractions/user-tmp>
capability net_raw,
capability setuid,
capability setgid,
capability dac_override,
network raw,
network packet,
# for -D
capability sys_module,
@{PROC}/bus/usb/ r,
@{PROC}/bus/usb/** r,
# for -F and -w
audit deny @{HOME}/.* mrwkl,
audit deny @{HOME}/.*/ rw,
audit deny @{HOME}/.*/** mrwkl,
audit deny @{HOME}/bin/ rw,
audit deny @{HOME}/bin/** mrwkl,
@{HOME}/ r,
@{HOME}/** rw,
/usr/sbin/tcpdump r,
}
name of profile — generally the
binary.
the permitted (or denied)
capabilities
the files it can access (or is denied
access to).
• Docker’s  default  capabiliOes  are  
set  by  app  armor!  (and  turned  
off  when  you  docker	
  run	
  —
privileged=true)    
• be  careful:  when  you  supply  
your  own  apparmor  profile,  
your  are  essenOally  resetng  
the  capabiliOes.  
• copy  or  inherit  these  when  you  create  a  
new  profile  for  your  containers.      
this looks familiar…
using apparmor with
1. Create  the  custom  profile:	
  vim	
  my_container_profile	
  
2. Load  it  into  app  armor:  cat	
  my_container_profile	
  |	
  
sudo	
  apparmor_parser	
  -­‐r  
3. `
4. Run  it  with  your  docker  container:  docker	
  run	
  —
security-­‐opt=“apparmor:my_container_profile”	
  
5. $$$  Profit?
ok but what if I break things?
tip:
make  a  permissive  profile,  and  run  it  in  complain  mode
sudo	
  aa-­‐complain	
  my_container_profile
it  will  log  to  auditd
type=AVC	
  msg=audit(1390936201.188:15647):	
  apparmor="AUDIT"	
  
operation="file_lock"	
  parent=7873	
  profile="my_container_profile"	
  f	
  
name="/tmp/pam_krb5_tmp_FqhNDa"	
  pid=7875	
  comm="sshd"	
  
requested_mask="k"	
  denied_mask="k"	
  fsuid=0	
  ouid=0	
  
iterate  unOl  right.
“Great! but this is still hard
and annoying.”
what if you could automate it?
tip:
use  aa-­‐logprof  to  generate  the  apparmor  profiles  
automagically?
aa-­‐logprof	
  [	
  -­‐d	
  /path/to/profiles	
  ]	
  [	
  -­‐f	
  /path/to/logfile	
  ]	
  
!!  don’t  use  these  without  manual  review  and  ediHng  !!
http://manpages.ubuntu.com/manpages/natty/man8/aa-­‐logprof.8.html
(ok: we still need better tooling)
a vision
• IF  in  the  future…  
• DockerHub  registry  becomes  your  go-­‐to  trusted  
distribuOon  source  for  applicaOons.…  
•   Why  not    
• Have  a  registry  for  apparmor  and  SELinux  profiles  
geared  for  official  dockerized  app  containers?  
• …Include  seccomp  filters  and  other  security  configs?  
• Share  your  polices  &  reduce  the  burden  of  having  to  
harden  your  own  apps/containers.
#	
  fetch	
  apparmor	
  security	
  profile	
  for	
  wordpress	
  
image	
  
docker	
  security-­‐profile	
  fetch	
  wordpress:latest	
  
	
  	
  
#	
  you	
  can	
  even	
  fetch	
  by	
  image	
  /	
  tag	
  
docker	
  security-­‐profile	
  fetch	
  
0cc6ffbf1a0cd78ab244c4b3b5cef13618bf4c8bcd229ec2673
1a951c33df72e	
  	
  
#	
  allow	
  users	
  to	
  submit/push	
  their	
  own	
  app	
  armor	
  
profiles	
  	
  
docker	
  security-­‐profile	
  push	
  —-­‐profile=“apparmor:/
etc/apparmor/wordpress.profile”	
  jandre/
wordpress:custom
in conclusion
• we  need  more  automaOon  around  security  audiOng,  
hardening,  tesOng,  and  monitoring  
• InnovaOon  here  should  come  not  just  from  the  
Docker  folks.  
• The  consistency  of  Docker  containers  enables  us  to  
be  innovaOve  in  how  we  automate  the  above  ^^  
is this interesting to you?
• contact  me!  jandre@gmail.com  
• follow  @dockersecurity  for  news

Operating Docker

  • 1.
  • 2.
    about me @fun_cuddles  / jenpire.com  /  organizer  @BostonGoLang   co-­‐founder  @threatstack,    formely  Mandiant  /  Symantec   **  I  HAVE  NO  AFFILIATION  WITH  THE  SMART  PEOPLE  AT  DOCKER  **
  • 4.
    REALITY  CHECK *  I stole  this  shamelessly  from  @petecheslock
  • 5.
  • 6.
  • 7.
    -­‐  your  security engineer “Docker needs to fix all of these problems.” “It’s probably fine.” -­‐ -­‐your  hipster  devops     -­‐ gal  or  guy
  • 8.
  • 9.
    this is thewrong question.
  • 11.
    How  can  we enable  organizaOons  (security   +  dev  +  ops)  operate  Docker  safely? 1. Understand  your  threat  model   2. Understand  Docker’s  limitaOons   3. Understand  the  tools  available  to  us     4. Evolve  the  Docker  security  ecosystem  :)
  • 12.
    1.  How  do I  trust  the  code?   2.  How  do  I  know  if  I’ve  configured  my  Docker  host  +   containers  in  a  secure  way?   3.  How  do  containers  change  my  security  pracOces,   e.g.  monitoring? empathy: understanding the security engineer’s concerns
  • 13.
    but also consider the consistency  of  applicaOon  environments  in  Docker   containers  provides  for  interesOng  opportuniHes  for   new  automaOon  around  security  hardening,  audiOng,   and  tesOng.
  • 14.
    issues with trust docker images  are  binaries  (opaque)   who  am  I  trusOng?   who  is  updaOng  these  things  when  there  is  a  criOcal   security  flaw?  
  • 15.
    The problem ofpatch management is a real thing. http://www.banyanops.com/blog/analyzing-docker-hub/
  • 16.
  • 17.
    always be updating! •Do  perform  security  upgrades  (debian  example   below)   • sudo  docker  exec  -­‐it  <container>  apt-­‐get  update     • sudo  docker  exec  -­‐it  <container>  apt-­‐get  upgrade   -­‐s  |  grep  -­‐i  security  #  dry  run   • sudo  docker  exec  -­‐it  <container>  apt-­‐get  upgrade   #  commit  changes  when  done  
  • 18.
  • 19.
    what  if  someone  replaced  libc  with  a   backdoored  version?
  • 20.
    community   addressing trust automate policy  audiOng  +   enforcement for  a  given  container,  tell  me  who/what   I  am  trusOng build  from  a  trusted  base  image be  aware  of  who  you  are  trusOng don’t  overrely  on  Docker  hub tooling  to  apply  and     validate  security  updates
  • 21.
    more advice • “The best  opOon  is  to  block  index.docker.io  locally,  and   download  and  verify  images  manually  before  imporOng   them  into  Docker  using  docker  load.”   • hcps://Otanous.com/posts/docker-­‐insecurity#fn:4   • Use  a  private  docker  registry   • hcps://www.digitalocean.com/community/tutorials/how-­‐to-­‐set-­‐up-­‐a-­‐private-­‐docker-­‐registry-­‐ on-­‐ubuntu-­‐14-­‐04  ,  hcps://quay.io         • Use  RedHat  cerOfied  containers   •  hcp://www.redhat.com/en/about/press-­‐releases/red-­‐hat-­‐announces-­‐cerOficaOon-­‐for-­‐ containerized-­‐applicaOons-­‐extends-­‐customer-­‐confidence-­‐and-­‐trust-­‐to-­‐the-­‐cloud  
  • 22.
    opportunities • trust,  but verify:  build  an  binary  audiOng  tool  for  Docker   images!   • e.g.,  scan  images,  validate  installed  libraries  and  binaries    do   not  have  criOcal  security  issues  and  align  with  signed  package   manifests.   • hcps://github.com/banyanops/collector    +`cruh’  but  for  containers?   • hcps://github.com/OpenSCAP/container-­‐compliance  -­‐  RHEL  only   • contribute  to  the  packaging/distribuOon  trust  conversaOon!   • hcps://github.com/docker/distribuOon/pull/179   • references:  hcp://theupdateframework.com/  
  • 23.
  • 24.
    security empathy How  do I  know  if  I’ve   configured  my  Docker  host  +   containers  in  a  secure  way?  
  • 25.
    the good! Docker  released a   comprehensive   security  benchmark. hcps://blog.docker.com/2015/05/understanding-­‐docker-­‐security-­‐and-­‐best-­‐pracOces/
  • 26.
    the bad …it’s  118 pages  of  material!
  • 27.
    the good!: canwe automate these checks? for  most  of  them,  yes!   github.com/dockersecuritytools/bacen  -­‐  IN  PROGRESS
  • 28.
  • 29.
    the problem ofisolation
  • 30.
    container hardening: thegood there’s actually a lot of knobs to turn!
  • 31.
    • SELinux  / AppArmor  policies  (—security-­‐opt)    (more   about  this  later)   • capabiliHes  (—cap-­‐add,  —cap-­‐drop)     • “give  root  without  all  of  root”
  • 32.
    • cgroups  (resource allocaOon,  many  flags)   • Example:  $ docker run -it --rm -m 128m fedora bash • hcps://goldmann.pl/blog/2014/09/11/resource-­‐management-­‐in-­‐docker/   • user  namespaces  (soon!)  so  you  don’t  have  to  run   id=0  processes  as  root!   • seccomp  filtering  to  permit  or  block  individual   system  calls  (soon!)   • hcp://opensource.com/business/15/3/docker-­‐security-­‐future
  • 33.
    the bad there’s alot of knobs to turn :(
  • 34.
    we can dobetter.
  • 35.
    what  problems  are they  trying  to  solve? AppArmor  +  SELinux
  • 36.
    a question if  engineers can’t  figure  out  how  to  build  and  apply  SELinux  policies  now,     how  will  Docker  change  things?
  • 37.
    apparmor a  gentler  mandatory access  control  system   hcps://wiki.ubuntu.com/AppArmor   introducing  
  • 38.
    #include <tunables/global> /usr/sbin/tcpdump { #include<abstractions/base> #include <abstractions/nameservice> #include <abstractions/user-tmp> capability net_raw, capability setuid, capability setgid, capability dac_override, network raw, network packet, # for -D capability sys_module, @{PROC}/bus/usb/ r, @{PROC}/bus/usb/** r, # for -F and -w audit deny @{HOME}/.* mrwkl, audit deny @{HOME}/.*/ rw, audit deny @{HOME}/.*/** mrwkl, audit deny @{HOME}/bin/ rw, audit deny @{HOME}/bin/** mrwkl, @{HOME}/ r, @{HOME}/** rw, /usr/sbin/tcpdump r, } name of profile — generally the binary. the permitted (or denied) capabilities the files it can access (or is denied access to).
  • 39.
    • Docker’s  default capabiliOes  are   set  by  app  armor!  (and  turned   off  when  you  docker  run  — privileged=true)     • be  careful:  when  you  supply   your  own  apparmor  profile,   your  are  essenOally  resetng   the  capabiliOes.   • copy  or  inherit  these  when  you  create  a   new  profile  for  your  containers.       this looks familiar…
  • 40.
    using apparmor with 1.Create  the  custom  profile:  vim  my_container_profile   2. Load  it  into  app  armor:  cat  my_container_profile  |   sudo  apparmor_parser  -­‐r   3. ` 4. Run  it  with  your  docker  container:  docker  run  — security-­‐opt=“apparmor:my_container_profile”   5. $$$  Profit?
  • 41.
    ok but whatif I break things?
  • 42.
    tip: make  a  permissive profile,  and  run  it  in  complain  mode sudo  aa-­‐complain  my_container_profile it  will  log  to  auditd type=AVC  msg=audit(1390936201.188:15647):  apparmor="AUDIT"   operation="file_lock"  parent=7873  profile="my_container_profile"  f   name="/tmp/pam_krb5_tmp_FqhNDa"  pid=7875  comm="sshd"   requested_mask="k"  denied_mask="k"  fsuid=0  ouid=0   iterate  unOl  right.
  • 43.
    “Great! but thisis still hard and annoying.”
  • 44.
    what if youcould automate it?
  • 45.
    tip: use  aa-­‐logprof  to generate  the  apparmor  profiles   automagically? aa-­‐logprof  [  -­‐d  /path/to/profiles  ]  [  -­‐f  /path/to/logfile  ]   !!  don’t  use  these  without  manual  review  and  ediHng  !! http://manpages.ubuntu.com/manpages/natty/man8/aa-­‐logprof.8.html
  • 46.
    (ok: we stillneed better tooling)
  • 47.
  • 48.
    • IF  in the  future…   • DockerHub  registry  becomes  your  go-­‐to  trusted   distribuOon  source  for  applicaOons.…   •  Why  not     • Have  a  registry  for  apparmor  and  SELinux  profiles   geared  for  official  dockerized  app  containers?   • …Include  seccomp  filters  and  other  security  configs?   • Share  your  polices  &  reduce  the  burden  of  having  to   harden  your  own  apps/containers.
  • 49.
    #  fetch  apparmor  security  profile  for  wordpress   image   docker  security-­‐profile  fetch  wordpress:latest       #  you  can  even  fetch  by  image  /  tag   docker  security-­‐profile  fetch   0cc6ffbf1a0cd78ab244c4b3b5cef13618bf4c8bcd229ec2673 1a951c33df72e     #  allow  users  to  submit/push  their  own  app  armor   profiles     docker  security-­‐profile  push  —-­‐profile=“apparmor:/ etc/apparmor/wordpress.profile”  jandre/ wordpress:custom
  • 50.
    in conclusion • we need  more  automaOon  around  security  audiOng,   hardening,  tesOng,  and  monitoring   • InnovaOon  here  should  come  not  just  from  the   Docker  folks.   • The  consistency  of  Docker  containers  enables  us  to   be  innovaOve  in  how  we  automate  the  above  ^^  
  • 52.
    is this interestingto you? • contact  me!  jandre@gmail.com   • follow  @dockersecurity  for  news