Download to read offline
More Related Content
Similar to Holistic testing in DevOps
![[DPE Summit] How Improving the Testing Experience Goes Beyond Quality: A Deve...](https://cdn.slidesharecdn.com/ss_thumbnails/dpesummitseptember20-212023-howimprovingthetestingexperiencegoesbeyondqualityadeveloperproductivityp-230924000523-453e8af8-thumbnail.jpg?width=600ounds&width=560&fit=bounds)
![[QE 2018] Paul Gerrard – Automating Assurance: Tools, Collaboration and DevOps](https://cdn.slidesharecdn.com/ss_thumbnails/automatingassurancegerrard-180629130855-thumbnail.jpg?width=600ounds&width=560&fit=bounds)
Holistic testing in DevOps
- 1. @janetgregoryca Holistic Testing inDevOps NovaTech – January 2022 Janet Gregory Twitter: @janetgregoryca Agile Testing Fellowship DragonFire Inc. With material from Lisa Crispin Copyright 2022
- 2. @janetgregoryca Focusing on…. - Whatdo I mean by holistic testing in DevOps - Testing early - Testing and your deployment pipeline - Testing your releases - Applying testing activities to observing and learning 2
- 3.
- 4. @janetgregoryca What we hearabout DevOps is … •Continuous delivery •Continuous deployment •Testing in production •The build pipeline •Tools and more tools 4
- 5. @janetgregoryca • Small batchesto delivery • Builds quality into your product • Improves time to market • Supports continuous testing and delivery • Provides for continuous learning and improvement by observing production use holistic
- 6. @janetgregoryca 6 We test here! DanAshby -https://danashby.co.uk/2016/10/19/continuous-testing-in-devops/
- 7. @janetgregoryca My version ofthe DevOps cycle
- 8.
- 9.
- 10. @janetgregoryca Testing early ….. •Isabout collaboration •Helps the customers get what they want •Helps prevent defects in code
- 11. @janetgregoryca How are we goingto test this? and working through examples What are the accessibility standards? What is the biggest risk? Let’s prioritize quality? Testing early is asking questions
- 12. @janetgregoryca Questions to askabout testing •What kinds of testing is carried out? •Who is responsible for creating and executing the different types of tests? •How does the team decide what to automate and at what levels?
- 13. @janetgregoryca To encourage wholeteam testing,
- 14. @janetgregoryca During development • Guide developmentwith examples (BDD/ATDD) 14 • Teams decide what to automate • Exploratory testing for unknowns
- 15. @janetgregoryca Let’s move tothe right side of the loop •Deployment pipeline •Testing in production •Observing and monitoring •Learning
- 16. @janetgregoryca Compile and run unit tests locally Deployto test env. Check-in and run unit tests Run API test suite Deploy to dev env. Run smoke tests Requires human intervention Automated steps Exploratory testing Accessibility testing …… UI test suite …….. Testing is the of DevOps
- 17. @janetgregoryca Where can youstart? Draw the pipeline – on a white board or a remote tool Look for bottlenecks • Ex. Wait times for someone to review test results Look for merging issues and long standing branches • Testing becomes more complicated • Automation often breaks Aim for fast feedback!
- 18. @janetgregoryca Questions to ask aboutthe pipeline • What tests run on what environment? • Which are human-centric tests? Which are automated? • Who is responsible for following up on failures? • How fast is the response to failures?
- 19. @janetgregoryca Local Unit tests Static analysis Testdoubles Run unit tests Run api auto tests Observability Monitoring Test in Prod Run API auto tests Security, load, performance, recovery…. Run workflow auto tests UAT Your deployment pipeline & tests
- 20. @janetgregoryca The release Image from:https://dzone.com/articles/blue-green-deployment-for-cloud-native-application
- 21. @janetgregoryca Observing Early in thecycle, • help to define what you want to measure • create queries and alerts for monitoring (helps testing) • think about good metrics to use with feature flags for A/B testing • Then we observe, and monitor
- 22.
- 23. @janetgregoryca 23 Experiment Set a goal,craft a hypothesis, and find ways to measure progress. For example: ● Goal: reduce cycle time ● Hypothesis: “We believe that automating more tests at the API level will shorten our cycle time by 10% over the next month.” If you don’t see progress, try another model or goal
- 24. @janetgregoryca Optimize the whole- reduce cycle time, improve quality - from concept to in the customer's’ hands Work identified Work started Work completed Process / Cycle Time
- 25. @janetgregoryca Cultural aspects • Silos/ turfs • Looking at individual optimization • It’s not my job (to test, to code, to ….) • Separate DevOps team • Do teams feel safe to experiment? • How is information funneled up / down?
- 26. @janetgregoryca Simple / obvious ComplexComplicated Chaotic Cynefin Model – Dave Snowden
- 27. @janetgregoryca No one rightway to build TRUST ---- different contexts different guardrails
- 28. @janetgregoryca What makes aholistic testing strategy work? ● Commitment to a level of quality ○ Understand what’s valuable to customers ○ Focus on bug prevention vs. bug detection ● Fast feedback loops ● Diverse perspectives, skill sets, biases
- 29. @janetgregoryca References • https://agiletester.ca • https://janetgregory.ca/blog •Donkeys & Dragons YouTube channel https://tinyurl.com/agiletfellow • https://lisacrispin.com • A Practical Guide to Testing in DevOps (Katrina Clokie) • https://medium.com/@copyconstruct/testing-in-production-the-safe- way-18ca102d0ef1 • Continuous Delivery: Reliable Software Releases Through Build, Test, and Deployment Automation (Jez Humble and David Farley) • Hendrickson, Elisabeth, Explore It!
- 30. @janetgregoryca Agile Testing: APractical Guide for Testers and Agile Teams More Agile Testing: Learning Journeys for the Whole Team Agile Testing Condensed: A Brief Introduction By Janet Gregory and Lisa Crispin Agile Testing Fellowship https://agiletestingfellow.com Contact info https://janetgregory.ca Email: janet@agiletester.ca Twitter: janetgregoryca