Download to read offline
More Related Content
Model-Driven Testing For Agile Teams
- 1. Model-Driven Testing for Agile Teams Kerry Kimbrough Cornutum Project
- 2. This Workshop IsAbout... • A design technique... – Input space modeling • Using a tool... – Tcases, a model-driven test case generator • To tackle test design issues...
- 3. Test Design Issues •How do I pick test cases? – Define an input model. Then generate test cases. • How many test cases do I need? – Define a coverage model. Then generate a minimal test set. • Are my tests good enough? – Is my input model good enough? – Is my coverage model good enough?
- 4. Why Does ItMatter? • Defects are a drag! – The friction that slows delivery • The Agile way – Test early – Test continuously • And yet...
- 5. The Curse ofthe Undone Sprint’s done! Except for fixing Umm, sure Stories finished? the defects! Product Dev QA
- 6. The Busted Sprint We planned 3 stories, but only We had a lot of stuff to fix finished 2? up in the 1st story. Product Dev QA
- 7. The Sprint ThatNever Was We planned 2 We had to work on those stories, but only production problems! finished 1?? Product Dev QA
- 8. The Hardening (?!)Sprint Stories done! Can Almost. Just have to finish we release now? the hardening sprint! Product Dev QA
- 9. Why Model-Driven Testing? •Much more powerful tests • For little or no extra effort • At every step of the way – Before dev – During dev – During system test
- 10. Tcases: How ToGet It www.cornutum.org See “Installing Tcases” Download it here
- 11. Example: find <pattern><file> Locates one or more instances of a given pattern in a text file. All lines in the file that contain the pattern are written to standard output. A line containing the pattern is written only once, regardless of the number of times the pattern occurs in it. The pattern is any sequence of characters whose length does not exceed the maximum length of a line in the file. To include a blank in the pattern, the entire pattern must be enclosed in quotes. To include a quotation mark in the pattern, two quotes in a row ("") must be used.
- 12. Modeling The InputSpace • All combinations of values • For each dimension of variation
- 13. Defining System Functions <Systemname=“Examples” <!-- All system function definitions go here --> <Function name=“find” <!-- All input definitions for “find” go here --> </Function> </System>
- 14. Defining Input Variables <Functionname="find“> <Input type="arg"> <!--arg: Direct input arguments (the default)--> <Var name="fileName"> ... </Var> ... </Input> <Input type="env"> <!--env: Environment state variables--> ... </Input> </Function>
- 15. Defining Input Values <Functionname=“find” <Input type=“arg”> <Var name=“fileName” <!-- The required file name is defined --> <Value name=“defined”/> <!-- The required file name is missing: an error --> <Value name=“missing” failure=“true”/> </Var> ... </Input> </Function>
- 16. To Define InputValues... • Enumerate all values • Or identify “equivalence classes” – A subset of values that are “test-equivalent” – No need to test all -- any one will do – Examples • Integer: Negative, Zero, Positive • Bounded: Below-Min, In-Range, Above-Max • List: Empty, One, Many
- 17. Modeling Complex Inputs find<pattern> <file> All lines in the file that contain the pattern are written to standard output. A line containing the pattern is written only once, regardless of the number of times the pattern occurs in it.
- 18. Defining Variable Sets <VarSetname="file"> <!-- Does the file exist? --> <Var name="exists"> ... </Var> <!-- Does the file contain... --> <VarSet name="contents"> <!-- ... any matching lines? --> <Var name="patterns"> ... </Var> <!-- ... multiple matches in a line? --> <Var name="patternsInLine"> ... </Var> </VarSet> </VarSet>
- 19. Modeling Complex Inputs find<pattern> <file> The pattern is any sequence of characters whose length does not exceed the maximum length of a line in the file. To include a blank in the pattern, the entire pattern must be enclosed in quotes. To include a quotation mark in the pattern, two quotes in a row ("") must be used.
- 20. Defining Variable Sets <VarSetname="pattern"> <!-- How many chars in pattern? --> <Var name="size"> ... </Var> <!-- Is it quoted? --> <Var name="quoted"> ... </Var> <!-- Does it contain blanks? --> <Var name="blanks"> ... </Var> <!-- Does it contain embedded quotes? --> <Var name="embeddedQuotes"> ... </Var> </VarSet>
- 21. System Input ModelBasics <System ...> <!-- A system has... --> <Function ...> <!-- functions, having... --> <Input ...> <!-- different types of inputs: --> <Var ...> <!-- simple variables, having... --> <Value ...> ...</Value> <!-- (classes of) values --> </Var> <VarSet ...> <!-- or complex inputs, composed of...--> <Var ...> ...</Var> <!-- many variables... --> <VarSet ...> ...</VarSet> <!-- and many levels --> </VarSet> </Input> </Function> </System>
- 22. Defining Constraints • Valuescan have properties – Assumed by any test case that includes this value • Values can have conditions – Properties that must appear in any test case that includes this value. – Properties that must not appear in any test case that includes this value.
- 23. When Variables AreIrrelevant <Function name="find"> <Input type="arg"> <VarSet name="pattern" when="fileExists"> <Var name="blanks" whenNot="empty"> ... </Var> ... </VarSet> ... </Input> ... </Function>
- 24. Input Model vs.Coverage Model • Input model – Defines the input space – Defines the input combinations possible • Coverage model – Defines the input combinations tested
- 25. What Is InputSpace Coverage? • Measures input combinations tested • A “black box” coverage metric – Specification-based • Complement to “white box” coverage – Code-based – Measures lines/branches/conditions tested • Basic unit: an N-tuple – Combination of values from N input variables
- 26. Default: 1-Tuple Coverage <TestCaseid="2"> <Input type="arg"> <Var name="pattern.size" value="manyChars"/> <Var name="pattern.quoted" value="yes"/> <Var name="pattern.blanks" value="many"/> <Var name="pattern.embeddedQuotes" value="none"/> <Var name="fileName" value="defined"/> </Input> <Input type="env"> <Var name="file.exists" value="yes"/> <Var name="file.contents.linesLongerThanPattern" value="many"/> <Var name="file.contents.patterns" value="one"/> <Var name="file.contents.patternsInLine" value="one"/> </Input> </TestCase>
- 27. 2-Tuple (Pairwise) Coverage <TestCaseid="2"> <Input type="arg"> <Var name="pattern.size" value="manyChars"/> <Var name="pattern.quoted" value="yes"/> <Var name="pattern.blanks" value="many"/> <Var name="pattern.embeddedQuotes" value="none"/> <Var name="fileName" value="defined"/> </Input> <Input type="env"> <Var name="file.exists" value="yes"/> <Var name="file.contents.linesLongerThanPattern" value="many"/> <Var name="file.contents.patterns" value="one"/> <Var name="file.contents.patternsInLine" value="one"/> </Input> </TestCase>
- 28. Failures From MultipleFactors Source: NIST http://csrc.nist.gov/groups/SNS/acts
- 29. Example: Web-Client.Config <Function name="Config"> <!-- Set up the configuration for a Web client test --> <Input> <Var name="Browser"> <Value name="Firefox"/> <Value name="Chrome"/> </Var> <Var name="OS"> <Value name="Linux"/> <Value name="Windows"/> <Value name="Mac-OS-X"/> </Var> <Var name="Flash"> <Value name="Yes"/> <Value name="No"/> </Var> </Input> </Function>
- 30. Example: Web-Client.Config • Defaultcoverage (1-tuple) – 7 tuples – 3 test cases • 2-tuple coverage – 16 tuples – 6 test cases (minimum) • All permutations – 12 tuples – 12 test cases
- 31. Tool Tips: UseTcases To... • Define multiple coverage levels • Reuse and extend previous test cases • Create random combinations • Simplify tests with once=“true” • Transform test case output
- 32. Model-Driven Testing Workflow Define Build Verify
- 33. Model-Driven Testing Workflow Define Example use cases: Explore and clarify with high-level system input model
- 34. Model-Driven Testing Workflow Build Unit tests: Generate and improve with detailed unit input models
- 35. Model-Driven Testing Workflow System tests: Generate coverage for system configuration space Verify
- 36. The End Done? Done! Hell, yeah! Product Dev QA