Downloaded 23 times
More Related Content
Similar to Test Design for Fully Automated Build Architecture
Test Design for Fully Automated Build Architecture
- 1. MG Half-day Tutorials Monday, April30th, 2018 8:30 AM Test Design for Fully Automated Build Architecture Presented by: Melissa Benua mParticle Brought to you by: 350 Corporate Way, Suite 400, Orange Park, FL 32073 888---268---8770 ·· 904---278---0524 - info@techwell.com - http://www.stareast.techwell.com/
- 2. Melissa Benua mParticle Melissa Benuahas worked in nearly every software development role—dev, test, DevOps, and program management—at companies big and small. She's created and run high-availability, high-quality services on products such as mParticle, PlayFab, Bing, Cortana, and Xbox One. Melissa discovered her love of massively- scaled systems while growing the Bing backend, where she honed the art of keeping highly-available complex systems up while undergoing massive code churn. Melissa isn’t afraid to mix traditional approaches with bold new ideas to make her products better, faster, and more reliable. She’s passionate not only about maximizing efficiency both in her product code and in her developer tools but also about sharing best practices among colleagues and the tech world at large.
- 3. 3/13/2018 1 1 Test Design forFully Automated Build Architectures Melissa Benua Senior Technical Lead, mParticle mbenua@gmail.com @queenofcode STAREAST 2018 2 About Me Boeing Microsoft PlayFab mParticle
- 4. 3/13/2018 2 3 About The Tutorial Source Control Track+ Code Review Build + Test Deploy + Monitor 4 Guiding Principles What are we doing here?
- 5. 3/13/2018 3 5 G U ID I N G P R I N C I P L E S Key Test Features Important Reliable Specific 6 G U I D I N G P R I N C I P L E S Importance Triage scenarios based on priority IMPOSSIBLE to cover every scenario Understand what failures can be tolerated Run the most important tests first
- 6. 3/13/2018 4 7 G U ID I N G P R I N C I P L E S Importance Build Failure? UI Test Failure? 8 G U I D I N G P R I N C I P L E S Reliability No flakiness No false-negatives or false-positives Repeatable without human intervention Cleans up after itself
- 7. 3/13/2018 5 9 G U ID I N G P R I N C I P L E S Reliability 10 G U I D I N G P R I N C I P L E S Specificity Clear answer to a clear question Have one main goal Don’t ‘boil the ocean’ Minimal overlapping coverage
- 8. 3/13/2018 6 11 G U ID I N G P R I N C I P L E S Specificity 12 G U I D I N G P R I N C I P L E S Proceed With Caution Shared resource reliance 1 Single- threading 2 Long duration test time 3 Caching 4
- 9. 3/13/2018 7 13 G U ID I N G P R I N C I P L E S Exercise: Test Cases Photo Gallery Site UX FrontEnd (HTML5 + JS) Logic BackEnd (Java) Data Store (NoSQL) • Most important? • Easiest? • Most reliable? 14 Automated Pipeline Structure What does it even look like?
- 10. 3/13/2018 8 15 A U TO M A T E D P I P E L I N E S T R U C T U R E CI + CD Pipeline Source Control Track + Code Review Build + Test Deploy + Monitor 16 A U T O M A T E D P I P E L I N E S T R U C T U R E Beyond Unit Tests Develop: Diff Build • Compile change against mainline • Execute unit tests Build: Continuous Integration • Compile change as a part of mainline submit • Execute functional tests Deploy: Continuous Deployment • Start staging environment • Deploy staging environment • Execute UI + load tests • Project structure? • Moving parts? • Functional boundaries? • Shared resources? • Mocking potential?
- 11. 3/13/2018 9 17 A U TO M A T E D P I P E L I N E S T R U C T U R E Example Service Architecture UI Layer RESTful API Admin UI Web Frontend Backend Layer Auth Service Logic Service Cache Service Data Layer Database File Storage 18
- 12. 3/13/2018 10 19 A U TO M A T E D P I P E L I N E S T R U C T U R E The Matrix Component • Compile • Unit Package • Functional • Integration Release • Acceptance • UX • Load Product BE Machine Function Auth Logic FE Machine Function Web UI Cache 20 A U T O M A T E D End to End
- 13. 3/13/2018 11 21 A U TO M A T E D P I P E L I N E S T R U C T U R E Exercise: Mapping Categories to Stages Photo Gallery Site UX FrontEnd (HTML5 + JS) Logic BackEnd (Java) Data Store (NoSQL) • What should we run? • When should we run it? • How long should we wait? 22 A U T O M A T E D P I P E L I N E S T R U C T U R E Key Takeaways Easy to conditionally run different categories 1 Find the balance between too many stages and too few 2 Machine time is MUCH cheaper than human time 3 Fail fast and fail often 4
- 14. 3/13/2018 12 23 Monitoring and Reporting Whatjust happened? 24 M O N I T O R I N G A N D R E P O R T I N G Code Coverage 0 10 20 30 40 50 60 70 80 90 100 0 25 50 75 100 %ofBugsFound Code Coverage % Bugs vs Effort % of Bugs Found • Line vs branch coverage • Reports and readability • Data-driven decisions
- 15. 3/13/2018 13 25 M O NI T O R I N G A N D R E P O R T I N G Flaky Test Handling Cost vs Value Net gain? Test Failure Fatigue Ignoring failures? Ease of Detection How do we know? 26 M O N I T O R I N G A N D R E P O R T I N G Logging vs Counters Text Log File? Graph?
- 16. 3/13/2018 14 27 M O NI T O R I N G A N D R E P O R T I N G Exercise: What Goes Where? Photo Gallery Site UX FrontEnd (HTML5 + JS) Logic BackEnd (Java) Data Store (NoSQL) 28 P U T T I N G I T T O G E T H E R Overall Summary Follow natural system boundaries 1 Know where your cutline is and respect your time 2 Don’t try to boil the ocean 3 Use your data wisely 4
- 17.