Blunders in Test Automation

Blunders in Test Automation

• 1.
  info@dorothygraham.co.uk © Dorothy Graham2015 www.DorothyGraham.co.uk 1 Test Automation Blunders Prepared and presented by Dorothy Graham email: info@dorothygraham.co.uk Twitter: @DorothyGraham www.DorothyGraham.co.uk www.TestAutomationPatterns.org © Dorothy Graham 2015 2 Blunder •  from old Norse word “blundra” – meaning “to shut one’s eyes” •  now means – mistake caused by ignorance, carelessness or not thinking things through – a foolish or tactless remark – to move clumsily – to mismanage •  people blunder when they don’t see or understand
• 2.
  info@dorothygraham.co.uk © Dorothy Graham2015 www.DorothyGraham.co.uk 3 Contents •  Test automation blunders –  Testing-Tools-Test –  Who needs GPS? –  Silver Bullet –  The Wrong Thing –  How hard can it be? –  Stable Application Myth –  Project / Not Project –  Inside the Box –  Isolationism •  Conclusion Twitter: @DorothyGraham 4 Testing-Tools-Test •  blunder: thinking that tools actually do testing – most important, at the root of other blunders •  should not be called “testing tools” •  checking vs testing (Michael Bolton) – a check is automatable – a test requires sapience, intelligence •  better names: – “tester assistance tools” – “check-running tools”
• 3.
  info@dorothygraham.co.uk © Dorothy Graham2015 www.DorothyGraham.co.uk 5 People (testers) vs tools •  what do people do? – think, evaluate, assess, decide, observe, interpret – recognize patterns, have new ideas, find bugs – make mistakes •  what do tools do? – they just run stuff - whatever they’ve been programmed to execute (including bad tests) – intelligence level: zero Get tools to do what computers do best, get testers to do what people do best 6 Contents •  Test automation blunders –  Testing-Tools-Test –  Who needs GPS? –  Silver Bullet –  The Wrong Thing –  How hard can it be? –  Stable Application Myth –  Project / Not Project –  Inside the Box –  Isolationism •  Conclusion Twitter: @DorothyGraham
• 4.
  info@dorothygraham.co.uk © Dorothy Graham2015 www.DorothyGraham.co.uk 7 Who needs GPS? •  if you don’t know where you are going, any road will do (Lewis Caroll) •  where are you going with your automation? – testing and automation are different activities – different activities require different objectives •  good objectives for testing? – find bugs, gain confidence, investigate •  good objectives for automation? – Hint: they shouldn’t be the same! 8 What finds most bugs? regression testing exploratory testing likelihood of finding bugs most often automated What is usually automated?
• 5.
  info@dorothygraham.co.uk © Dorothy Graham2015 www.DorothyGraham.co.uk 9 Automation success = find lots of bugs? •  tests find bugs, not automation •  automation is a mechanism for running tests •  the bug-finding ability of a single test is not affected by the manner in which it is executed •  “find bugs” can be a dangerous objective – especially for regression automation! Automated tests Manual Scripted Exploratory Fix Verification 9.3% 24.0% 58.2% 8.4% Experiences of Test Automation, Ch 27, p 503, Ed Allen & Brian Newman 10 fast testing slow testing Effectiveness Low High EfficiencyManual testing Automated Efficiency and effectiveness poor fast testing poor slow testing goodgood greatest benefit not good but common worst better
• 6.
  info@dorothygraham.co.uk © Dorothy Graham2015 www.DorothyGraham.co.uk 11 Contents •  Test automation blunders –  Testing-Tools-Test –  Who needs GPS? –  Silver Bullet –  The Wrong Thing –  How hard can it be? –  Stable Application Myth –  Project / Not Project –  Inside the Box –  Isolationism •  Conclusion Twitter: @DorothyGraham 12 Silver bullet solution: get the right tool •  no such thing as “the right tool” or “best tool” – what’s “the best car”?
• 7.
  info@dorothygraham.co.uk © Dorothy Graham2015 www.DorothyGraham.co.uk 13 Silver bullet solution: get the right tool •  no such thing as “the right tool” or “best tool” – what’s “the best car”? poor benefits low cost good benefits high cost good benefits low cost poor benefits high cost benefits cost budget investment in good automation good benefits moderate cost commercial tools? open source tools? 14 Silver bullet: success is automatic •  automation is (much) more than just a tool •  it takes time and effort to succeed – building good automation is a learning process •  management support is critical – high level managers need to understand automation capability & limitations, and have realistic expectations and budget – “people issues” – people use the automation, people develop the automation
• 8.
  info@dorothygraham.co.uk © Dorothy Graham2015 www.DorothyGraham.co.uk 15 Contents •  Test automation blunders –  Testing-Tools-Test –  Who needs GPS? –  Silver Bullet –  The Wrong Thing –  How hard can it be? –  Stable Application Myth –  Project / Not Project –  Inside the Box –  Isolationism •  Conclusion Twitter: @DorothyGraham 16 Automate manual tests? manual tests automated tests new approaches, e.g. monkey testing manual tests automated (% manual) tests (& verification) not possible to do manually tests not automated yet tests not worth automating, (captcha, colours, too long)
• 9.
  info@dorothygraham.co.uk © Dorothy Graham2015 www.DorothyGraham.co.uk 17 Testware architecture – poor architecture gives high maintenance cost •  most frequent cause of abandoned automation / shelfware – two layers of abstraction •  technical: for long life •  human: for wide use – using the tool’s architecture ties you to that tool (version) Testers     Test  Execu+on  Tool   runs  scripts   HL Keywords Structured Scripts testware   architecture   write  tests  (in  DSTL)   Ch 5 18 Contents •  Test automation blunders –  Testing-Tools-Test –  Who needs GPS? –  Silver Bullet –  The Wrong Thing –  How hard can it be? –  Stable Application Myth –  Project / Not Project –  Inside the Box –  Isolationism •  Conclusion Twitter: @DorothyGraham
• 10.
  info@dorothygraham.co.uk © Dorothy Graham2015 www.DorothyGraham.co.uk 19 How hard can it be? •  different activities require different skills •  classic blunder: let the testers automate – automating without automation skills? •  newer blunder: let the automators write tests – testing without testing skills? •  if testers are automators  a conflict of interest – do you run tests or do you automate tests? – automation is better long-term, BUT – deadline pressure pushes you back into manual testing 20 Tools will replace testers? •  “we can reduce the number of testers once we have the tool” – what are your testers like? •  mindless morons, or •  intelligent investigators? – need more skills, not fewer – automation can free testers to do more test design, exploratory testing •  and find more bugs – tools don’t replace testers, they support them
• 11.
  info@dorothygraham.co.uk © Dorothy Graham2015 www.DorothyGraham.co.uk 21 Contents •  Test automation blunders –  Testing-Tools-Test –  Who needs GPS? –  Silver Bullet –  The Wrong Thing –  How hard can it be? –  Stable Application Myth –  Project / Not Project –  Inside the Box –  Isolationism •  Conclusion Twitter: @DorothyGraham 22 Stable application myth •  can’t start automating until the application (or the GUI) is stable – throw-back to testing attitudes 30 years ago? •  testing comes at the end? •  testing is (only) execution? •  testing is more than testing [execution] •  automation is more than execution •  design your automation early, ready to run when anything is ready to test
• 12.
  info@dorothygraham.co.uk © Dorothy Graham2015 www.DorothyGraham.co.uk 23 •  test automation pyramid – Mike Cohn, Lisa Crispin (Ch 1) – more unit/component tests – fewer GUI tests •  instability an opportunity rather than a problem – build flexibility & robustness in your automation against common types of changes •  which aspects are stable? (GUI often late) – execute tests for stable parts when ready – build (GUI) tests now but with detail abstracted out When to write automated tests 24 Contents •  Test automation blunders –  Testing-Tools-Test –  Who needs GPS? –  Silver Bullet –  The Wrong Thing –  How hard can it be? –  Stable Application Myth –  Project / Not Project –  Inside the Box –  Isolationism •  Conclusion Twitter: @DorothyGraham
• 13.
  info@dorothygraham.co.uk © Dorothy Graham2015 www.DorothyGraham.co.uk 25 Project / not project •  not thinking of automation as a project – doesn’t need funding, resourcing – just do it in your “spare time” •  thinking that automation is (just) a project – when will automation be finished? (wrong question!) – needs on-going continuous improvement – refactoring at regular intervals •  project to start automation, then continued support 26 Contents •  Test automation blunders –  Testing-Tools-Test –  Who needs GPS? –  Silver Bullet –  The Wrong Thing –  How hard can it be? –  Stable Application Myth –  Project / Not Project –  Inside the Box –  Isolationism •  Conclusion Twitter: @DorothyGraham
• 14.
  info@dorothygraham.co.uk © Dorothy Graham2015 www.DorothyGraham.co.uk 27 Automated tests/automated testing Select / identify test cases to run Set-up test environment: •  create test environment •  load test data Repeat for each test case: •  set-up test pre-requisites •  execute •  compare results •  log results •  analyse test failures •  report defect(s) •  clear-up after test case Clear-up test environment: •  delete unwanted data •  save important data Summarise results Automated tests Select / identify test cases to run Set-up test environment: •  create test environment •  load test data Repeat for each test case: •  set-up test pre-requisites •  execute •  compare results •  log results •  clear-up after test case Clear-up test environment: •  delete unwanted data •  save important data Summarise results Analyse test failures Report defects Automated testing Automated processManual process 28 Contents •  Test automation blunders –  Testing-Tools-Test –  Who needs GPS? –  Silver Bullet –  The Wrong Thing –  How hard can it be? –  Stable Application Myth –  Project / Not Project –  Inside the Box –  Isolationism •  Conclusion Twitter: @DorothyGraham
• 15.
  info@dorothygraham.co.uk © Dorothy Graham2015 www.DorothyGraham.co.uk 29 Isolationism: isolated from •  change: encasing your first efforts in stone –  automated tests benefit from review and refactoring •  realism: optimism as a test strategy for the automation –  automated tests need to be tested and have bugs –  realistic expectations •  managers: not making automation benefits visible –  to the people who matter, in a way that communicates •  developers: not collaborating –  design for automated testability, help developers •  the wider world: not seeking existing knowledge – books, wiki*, articles *TestAutomationPatterns.wikispaces.com 30 Conclusion: see how these are wrong •  Summary of test automation blunders –  Testing-Tools- DON’T –Test They just run stuff –  Who needs GPS? Automation needs direction –  NO Silver Bullet Needs time and effort –  The Wrong Thing Testware architecture, manual tests? –  How hard can it be? Needs different skills –  Stable Application Myth Opportunity for flexibility –  Project at times / Not just a Project Ongoing, refactor –  Inside the Box Automation is more than automation –  Isolationism Collaborate, experiment, read
• 16.
  info@dorothygraham.co.uk © Dorothy Graham2015 www.DorothyGraham.co.uk 31 Thank you! •  More information: •  downloads www.DorothyGraham.co.uk –  articles and papers •  email info@DorothyGraham.co.uk •  blog http://dorothygraham.blogspot.com –  including automation, DDP, certification •  twitter –  @DorothyGraham •  TestAutomationPatterns.org –  free wiki of automation advice, with Seretta Gamba eBook available at informit.com/ swtesting Save 35% with discount code SWTESTING