Downloaded 23 times
Uploaded byDavid Hart
Continuous testing for continuous delivery
This document discusses smarter approaches to testing in continuous delivery environments. It recommends starting testing earlier in the development process through techniques like test-driven development and behavior-driven development. It also recommends minimizing duplicated testing efforts across stories and teams through strategies like reducing unnecessary documentation, prioritizing common scenarios, and understanding what parts of the system are tested at each level. The document also emphasizes the importance of test automation, building testability into the system design, and taking a one team approach to delivering working software frequently.
More Related Content
Similar to Continuous testing for continuous delivery
![UiPath Automation Suite Installation (Hands-On) [2/3]](https://cdn.slidesharecdn.com/ss_thumbnails/automationsuitecommunitysession2-251015095633-a6d862f1-thumbnail.jpg?width=600ounds&width=560&fit=bounds)
Continuous testing for continuous delivery
- 1. Continuous Delivery withSmarter Testing
- 2. Me Dave Hart –Lead Developer in Test @ IBM david.hart@uk.ibm.com @DeveloperInTest http://developerintest.blogspot.com/
- 3. Agenda • Continuous delivery •Smarter story level testing • Smarter product level testing • Automation • One team • Stories + • Conclusion • Questions
- 4.
- 5. The Agile ideal Thetwelve principles of agile software: 1. Our highest priority is to satisfy the customer through early and continuous delivery of valuable software. 3. Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale. 7. Working software is the primary measure of progress.
- 6. Where we canfall short Test in agile teams – Can still be silo’d – Let test lag behind development Story focused – Our week by week work is blinkered – strong focus on individual stories – Lots of testing not necessarily story related gets neglected • System Quality Testing (Installers, performance, deployment, compatibility, stress, release hardening, globalisation etc etc)
- 7. What can wedo? • Develop a ‘Test Runway’ • Be a team
- 8.
- 9. Smarter testing –story level • Start testing earlier – Requirements and design – TDD, BDD – Alongside development
- 10. Smarter testing –story level • Minimise test documentation – Guidelines over precise steps – Completed stories and running tests primary measure of testing performed • Know your features – Understand the purpose of the story deliverable – Know how it fits in the wider product – Be involved in the design • Prioritise the scenarios – Attempt to understand how a feature will be used – Break it down into common, occasional and possibly never scenarios, and prioritise accordingly
- 11. Smarter testing –story level • Reduce test duplication – Know who is testing what – What is tested at the unit level, integration system etc • Utilise test metrics – Coverage statistics – Defect route cause analysis – In sprint defects raised
- 12.
- 13. Smarter testing –product level • Minimise Test documentation – Approach and strategy over detailed plan – Guidelines for common areas – The backlog state is our measure of progress • Know your customers – Understand how and why they use your software – Understand the day to day routes through your software – Prioritise the bits that are important over the rarely used (test for the 80%)
- 14. Smarter testing –product level • Have an idea of all the parts of your release • Know what is tested where – Reduce duplication – Identify bits that fall between teams/stories – Define a test strategy / approach so we have an idea of the confidence we can derive from the various testing activities • Choose your test technologies carefully
- 15. Smarter testing –an example Web Service Application Client Unit Unit Unit Unit Unit Unit Component Component Unit Unit Unit Component
- 16. Smarter Testing -Automation
- 17. Automation • Regression, regression,regression – Continuous feedback of continued operation – Run often (continuous integration builds ideal) • Build in testability – At code level – At UI level – Architect and model for code re-use • Automate what you can
- 18. Automation Diagram from AgileTesting – Lisa Crispin & Janet Gregory
- 19. One Team toRule them All
- 20. One team • Commitas a team, deliver as a team • Support specialism's • Develop a quality culture
- 21.
- 22. Stories + -infrastructure Infrastructure is Key! Build systems – Fast feedback – Track failures back to source easily – Configurable triggers
- 23. Stories+ - DevOps •Automate your deployments – Link it into your build system – Make it available – Consider build frequency Web Services Application Custom Code Web Server Hardware Configuration Environment Customer Configuration Deployment
- 24. Stories + -DevOps • Enables – Cutting edge software for demonstration – Fast, usable feedback – at all levels
- 25. Stories + -DevOps Testability Code Code Tests Code Tests Deployment Configuration Testing Performance Testing User testingAPI testing Globalisation Testing Enables
- 26. Smarter Testing –An example Web Service Application Client Unit Unit Unit Unit Unit Unit Component Component Unit Unit Unit Component Deployed Smoke test Deploy Performance Fully System UI test
- 27.
- 28. Conclusions • Start atthe design • Automation and regression • Testing Runway – Step back – Inspect – Plan • Know your product • Invest
- 29. Conclusions Recommended Reading • AgileSoftware Requirements – Dean Leffingwell • Agile Testing – Lisa Crispin, Janet Gregory • Continuous Delivery – Jez Humble, David Farley • Cartoon tester: http://cartoontester.blogspot.com/
- 30.
Editor's Notes
- #3 Some joke about being smarter
- #5 Lets start by defining what I mean by continuous delivery..
- #6 Like all good guidelines, I have twisted them to my own needs…. Principals suggest we ought to be aiming for production ready and quality all the time. The principals are loose enough to allow room for working in a way that is agreeable But wouldn’t it be beneficial to consider ourselves always ready for delivery (or as near as damn it)
- #7 Test is often still viewed as a separate team It can often lag behind development (a genuine need in some teams depending on the nature of the story and testability of the code) Tend to build in hardening sprints at the end, or compatibility test runs outside of sprints. Fixed delivery dates something lots of us have to live with Whenever you have a testing ‘phase’ – it will get squeezed, just like in the waterfall days. Inevitably sprint estimation under estimates, stories slip eating up any contingency and eventually into those hardening and ‘other testing’ sprints
- #8 I would like to introduce the concept of a test runway Much like an architectural runway, a test runway is a big picture view of your practices and infrastructure that you can turn into stories and tasks, or mould into your processes. Everything I talk about in this presentation sits underneath the umbrella of a test runway.
- #9 Smarter testing is all about examining your current testing activities and making them as lean and efficient as possible – its not a technique or wonder tool, but an investment in your current testing activities to enable you to fit in all that needs doing. Its also unique to a team, the points I am going to share with you are processes and techniques we have found useful in teams I have worked in. Some might seem obvious, and you may have already considered them yourselves. But its surprising how easy it is to carry on as before – if you listen to nothing else I say but go away and examine your teams testing activities with a view to making them leaner and efficient, my job is done (that’s not to say you can leave now – I have lots of interesting things to say, honest).
- #10 Start testing at the design – does it stand up to the customers requirement. Has the UX been considered. Have the usage scenarios been understood
- #11 Know your features – pretty bloody obvious, but it is surprising how often you start to look at something and be surprised at some of what it does. Scenarios – test for the 80% not the 20 - as testers we like to find the hard bugs, but that isn’t necessarily where the money is best spent
- #12 The list isn’t exhaustive Reduce test duplication – may seem obvious but it is surprising just how much testing we duplicate. In fact it is often actively encouraged – two pairs of eyes are better than one etc. When time is tight – test duplication is a luxury we might not be able to afford You may be surprised to see metrics up there. They are generally the bain of a testers life (who want to be measured by the number of bugs they raise). But they can be really useful. We measure in sprint defect rate - gives some indication of the stability, Bugs found after story sign off - examining the cause of defects found after sign off can help identify any holes on your testing strategy Coverage - and looking at coverage can identify parts of the system that aren’t being tested. We actually found bugs in our system just by looking at the coverage figures – wee have a feature that has two implementations – we do some manual processing under certain circumstances, and defer to a third party library in other circumstances. We had passing integration and api style tests for the feature, but because the behaviour was the same in both cases (it was a performance feature) the tests passed. It wasn’t until we noticed one path was completely uncovered that we realised there was a bug in the code. Metrics have their uses.
- #14 I mention documentation again intentionally
- #15 If you have multiple teams working on a delivery, or multiple products making a release, try and know how it all fits together – will give you a better idea of the testing activities needed
- #16 So at first glance, the obvious test route here is from the client through the server and back again – we can automate some tests here and have some manual testing. But its inefficient, to rely on manual testing to fully test all parts of this system is slow and can only be done when all parts are in place. Tried to reduce the need for end to end tests to test our system fully So lets have see if we can half the problem, lets try and have tests that test the server meaning client tests are only needed to test client functionality. We start with unit and integration tests at the code level to increase confidence and coverage in the individual code units and collections of – these tests can be written before any implementation code is run – are generally fast to run and comparatively fast to write. It would be really useful to test at the service boundaries with some API tests – lets assume we have a framework that can simulate the service parts of the stack so we can execute tests in a framework without having to fully deploy, but using everything else as if it were deployed – so any DB’s or local resources. With knowledge of what is tested at unit and integration, and some idea of how the code underneath is structured, we can write some API tests that test the routes through those already tested integration blocks. These tests use more of the system, are a bit slower – but because of the testing already undertaken, can be fewer in number. Lets add some unit and integration tests to the client too – all of a sudden the emphasis on the exploratory manual testing has shifted from a test everything to a lets test the Client, and as a consequence some scenario based testing of the server. So what's missing? Well, the server tests simulate being deployed so some of the processing that happens at the transport layer doesn’t get tested – the communication between the two technologies is well defined, and pretty stable – so lets create some tests that just test that.
- #18 Contributing tests to a (or a number of) regression suites is essential in order to ensure the continued operation of completed features and stories. If we don’t we have to manually duplicate a whole lot of effort at various stages and play a much bigger risk game, weighing already tested features against known code changes to determine if the testing needs to be replayed. Frequent runs are also essential to give an indication of failure as soon as possible. Keep ‘em green! Red tests invalidate the effort spent – and lead to an apathy about test state (the whole broken windows syndrome) Automate what you can – will go into a little more detail when I talk about some of the dev ops practices in a few slides time
- #21 Commit as a team – avoid mini waterfalls where dev complete software is passed off to test
- #22 So all that process introspection and streamlining has meant you are testing as smartly as possible. With any luck your sprint by sprint testing activities are all in sync – giving you the best possible chance of being delivery ready at any moment. But what about everything else…
- #23 Before I talk a little about some of the more dev-op type practices we can employ, it is worth quickly mentioning that Infrastructure really is key to nearly everything I have talked about. And that starts with the build system. There is absolutely no point trying so hard to make your testing fit to enable sprint end releasability if you then have to wait a week for the developers to build their code, then another week while you manually run through all your test runs. A good Continuous integration build system is worth its weight in gold, and definitely worth some investment. The features I find most useful are fast feedback – at all levels (goes back to the point earlier about the sooner a bug is found, the cheaper it is to fix) The ability to track build / test failures back to particular people or check ins is also essential in ensuring the route cause is quickly found, and fixed The ability to configure what starts builds is also pretty useful – the ability to have test builds not run until it has been confirmed the code actually compiles – or if you have a componentised application, with down stream dependencies, making sure builds are triggered to push a change at the bottom all the way through is hugely beneficial.
- #24 So what do I mean by Dev-ops? If you have looked into this at all you will find a plethora of off the shelf offerings to manage your deployments or installation, but it doesn’t have to be that complicated It is most often talked about in web application scenarios – where delivery means making your changes available to consumers via the internet (companies like facebook, flickr, twitter etc etc) but it isn't limited to that Devops is about taking your development bits and making them available as a product that can be immediately consumed
- #26 Automating your deployment enables you then to turn the manual process of setting up test environments into an automated process – once that happens, you can expand the amount and types of testing you automate and drive from your build system
- #27 Back to my earlier diagram – we now have the ability to automate our deployments – other possibilities have opened up Now we are venturing outside of the functional. What about deployment? In this instance we use some scripts to configure our application server – which is something we can wrap and automate. But how do we know the deployment has worked? Some smoke tests just to poke the services and make sure they are alive sound like a good idea. Now we have deployment automated – the possibilities are endless – we have an environment we can automatically deploy for things like full system ui testing, performance testing, we can go as far as we like creating automated localised builds, use them for compatibility testing etc etc All those things that you would have to make sure you did via manual testing have a home in one of the testing stages of your product.
- #29 So the take away point from this presentation are: Start testing early – right back at the design Automate what you can, and add to regressions suites Create a Testing Runway – being introspective about what you test and where, creating frameworks to fully test and planning in tasks to accomplish you runway Know your product and your customer Invest – in your runway, in your infrastructure and in your testing