Download as PDF, PPTX
Uploaded byBruce Wong
Chaos Driven Development
The document discusses chaos-driven development, emphasizing the importance of intentionally introducing failure into systems to validate their resilience. It outlines principles for fault-tolerant design and contrasts controlled chaos testing with unpredictable outages. The talk advocates for a culture of chaos engineering within organizations to enhance system reliability and response to potential failures.
Chaos Driven Development
- 1. CHAOS DRIVEN DEVELOPMENT FutureInsights Live 2015, LasVegas Bruce Wong
- 2. A LITTLE ABOUTME • Founder of Chaos Engineering @ Netflix • Computer Science Background • Multiple roles scaling Netflix from 8m to 60m+ subs • CurrentlyTaking a Break @bruce_m_wong
- 3. Most enterprises hirepeople to fix things. Netflix hires people to break things…. …we should embrace Netflix's culture of "chaos engineering" throughout organizations of all shapes and sizes. http://readwrite.com/2014/09/17/netflix-chaos-engineering-for-everyone @bruce_m_wong
- 4.
- 5.
- 6.
- 7. CHAOS DEFINED “If itain’t broke don’t fix it” -Bert Lance, Nation’s Business 1977 If it ain’t broke, try harder -chaos philosophy @bruce_m_wong
- 8. CHAOS DEFINED Intentionally introducingfailure into a system with the purpose of validating resilience design. @bruce_m_wong
- 9.
- 10. WHY CHAOS? •Hardware fails •Poweroutages •Software has bugs •Human error •Natural disasters @bruce_m_wong
- 11.
- 12.
- 14. BLUE MOONS Once ina blue moon will eventually happen @bruce_m_wong
- 15. FAULT-TOLERANT DESIGN PRINCIPLES •Eliminate Single Points of Failure • Allow parts of the system to fail independently (Failure Isolation) • Prevent propagation (Failure Containment) @bruce_m_wong
- 16. START WITH CONSEQUENCES Chaos DrivenDevelopment @bruce_m_wong
- 17. MINIMUMVIABLE PRODUCT • Understandyour users • Understand your value proposition • Understand your business @bruce_m_wong
- 18. PRIORITIZE • Many aspectsand features are important • Each have different consequences for not working • A product’s value proposition is what drives your business @bruce_m_wong
- 19. DESIGN FOR FAILURE What failureisolation might look like @bruce_m_wong
- 28.
- 29. BREAKINGTHE CONNECTION How Confidentare you? -Next week? -Next month? -After that “quick patch”
- 30. WHAT DOES CHAOSLOOK LIKE? • Types - errors, latency • Duration - how long? • Intensity - how much? @bruce_m_wong
- 31. WHAT DOES CHAOSLOOK LIKE? • Return errors a % of requests • i.e. return HTTP500 for 1% of requests for 1 minute @bruce_m_wong
- 32. WHAT DOES CHAOSLOOK LIKE? • Make it slow(er) - Introduce Latency • i.e. sleep for 10ms on every request for 1 minute @bruce_m_wong
- 33. WHAT DOES CHAOSLOOK LIKE? Gradually increase • i.e. sleep for 10ms on every request for 1 minute • sleep for 100ms on every request for 3 minutes @bruce_m_wong
- 34. WHAT DOES CHAOSLOOK LIKE? The design/implementation worked! • microscopic impact, high confidence What if it didn’t work? • smaller impact than an outage • proactively fix it and try again @bruce_m_wong
- 35. WHAT AN OUTAGELOOKS LIKE? • Detection takes time (TTD) • Analysis takes time • Resolution takes time (TTR) • Inconvenient times @bruce_m_wong
- 36. CHAOSVS OUTAGE Chaos • Controlled •Planned • Intentional • Microscopic user impact Outages • Uncontrolled • Unpredictable • Unintended • Large impact @bruce_m_wong
- 37. WHAT ABOUTTESTING? • Testingis good - do it, automate it • While great testing disciplines can find most functional bugs… • scale, traffic and capacity • System misconfiguration and design limitations @bruce_m_wong
- 38. LESSONS LEARNED • Learnmore from chaos exercises than outages • Fixing a failure mode will uncover new ones • Configuration is often overlooked • Tools can break @bruce_m_wong
- 39.
- 40. WHAT MAKES RESILIENCE DESIGNHARD? • Product and Engineering Decision • Tradeoffs are difficult • Organizational Silos @bruce_m_wong
- 41. ORGANIZATIONAL SILOS • Servicesby Domain • Dev/Ops/Product • Incomplete context @bruce_m_wong
- 42. WHAT MAKES CHAOSHARD? In addition to the technical challenges • Organizations rarely incentivize people to try and break production • Misconceptions about complex systems and scale @bruce_m_wong
- 43. TAKE AWAYS • Whatare the consequences? • Start small, start early • Work together - share context • Validate don’t assume @bruce_m_wong
- 44.