KEMBAR78
Quality at Speed: More API Testing, Less UI Testing | PDF
Quality at Speed:
More API Testing, Less UI Testing
January 21, 2021
Why API Testing
What “The Money” Wants
$
@titusfortner
January 21, 2021
Continuous Delivery Really Means…
4
● Deliver
● Quality Software
● More Frequently
@titusfortner
January 21, 2021
Digital Confidence
5
Continuous Delivery Requires Continuous Testing
@titusfortner
January 21, 2021
DOM to Database Testing is Difficult
6
@titusfortner
January 21, 2021
“Magic” Solutions
7
Manual Testing doesn’t scale
Test Automation Code is hard
Magic doesn’t work:
● Record & Playback
● Codeless Test Generation
● Machine Learning
● Artificial Intelligence
@titusfortner
January 21, 2021
Humans vs Computers
8
© Alexey Litvinenko
@titusfortner
January 21, 2021
Automating Things
9
@titusfortner
January 21, 2021
More is Not Always Better
10
Positive
Value
Negative
Value
Tests/
Resources
Requirements
Exceeding
Resources
Successful
Testing
Building
Coverage
Lost Cause
@titusfortner
January 21, 2021
Classic Test Pyramid
11
@titusfortner
January 21, 2021
Pyramid in Practice
12
@titusfortner
January 21, 2021
Re-thinking the Pyramid
13
Why You Should Use API Testing?
To create a world of digital confidence.
SAUCE LABS PURPOSE
CONFIDENTIAL
PRODUCTION TEST
• Smoke testing
• User acceptance
testing
• Visual Testing
• Production testing
• Debugging
• Error reporting
DEVELOPMENT TEST
• Static code analysis
• Unit testing
• Component testing
• Component Testing
• Sanity Checks
• Security Testing
• Visual testing
INTEGRATION TEST
• Regression testing
• Visual testing
• Exploratory testing
• Integration testing
• UI Tests
• API testing
• Performance testing
You Can Build a DevOps Test Toolchain
Three Silos of Quality Signal Makes Risk Mitigation Difficult to Interpret
CONFIDENTIAL
… or Adopt a Solution
An Integrated DevOps Test Toolchain
Development
Test
Integration
Test
Production
Test
Apply Insight Data Insights
CONTINUOUS IMPROVEMENT
Single view of all quality signals
across the pipeline
Framework agnostic to support
various personas and languages
Flexible test execution options
(cloud or local)
Rich data insights to test better,
and continuously improve
Ability to collaborate, share and
view results across teams
Code Analysis
Team
Analytics
Unit Tests
API Tests
Browser/Mobile
Functional Tests
Visual Tests
Error Tracing
Security Scans
Non-Technical
& LowCode
Automation
Live Testing
Error Reporting
CI/CD
@mmerrell
© Sauce Labs, Inc. 18
FEBRUARY 22, 2020
We must use data, driven by user signals, to
generate and validate requirements
THE JOURNEY
© Sauce Labs, Inc. 19
Requirements
Real-world user Behavior is now
available before requirements are
gathered, better informing focus group
and research
Monitor Analytics
Look at both overall traffic (compared
to Model) and qualitative user behavior
patterns
Development
Development understands User Traffic
Model, and has more trust &
confidence in Requirements
Testing
New tests are added to automation suite
based on analytics gathered after
previous release cycle
Release
Feature flags, quick-rollback, and New User Traffic
Model is in-place before release
THE JOURNEY
Planning UAT Maintaining
Monitoring
Releasing
Coding Testing
© Sauce Labs, Inc. 20
@mmerrell
SHIFT LEFT SHIFT RIGHT
@titusfortner
January 21, 2021
The Test Diamond
21
How Should You Test APIs
Best Practices: Basics
CONFIDENTIAL 24
Test Endpoints
Test Everything
(including Response Code & Content-Type)
Test Objects
March 5, 2020
Best Practices: Advanced
CONFIDENTIAL 26
• Variabilize for Test Reuse
• Confirm Business Logic & Datatypes
• Use Dynamic Data
• Integration / End-to-End Test
• Confirm Response Matches Request
January 21, 2021
Show Me How
28
API Monitoring – A False Sense of Security |
[
{
quantity: 5,
color: ["white","red"],
price: 29.99,
imageURL: http://apif.com/baseball_cap.jpg,
name: "Baseball Cap",
description: "Boston Red Sox Baseball Cap",
id: 1,
category: "head",
},
{
quantity: 7,
color: ["blue","yellow","red"],
price: 39.99,
imageURL: http://apif.com/long_sleev_shirt.jpg,
name: "Long Sleeve Shirt",
description: "A wonderful long sleeve shirt",
id: 2,
category: "body",
},
{
quantity: 50,
color: ["red","gray"],
price: 49.99,
imageURL: http://apif.com/earmuffs.jpg,
name: "Earmuffs",
description: "Keep those ears warm in the winter!",
id: 3,
category: "ears",
},
]
http://demoapi.apifortress.com/api/retail/product?q=red
January 21, 2021
29
API Monitoring – A False Sense of Security |
[
{
quantity: 50,
color: ["red","gray"],
price: 49.99,
imageURL:
http://apif.com/earmuffs.jpg,
name: "Earmuffs",
description: "Keep those ears warm in
the winter!",
id: 3,
category: "ears",
},
]
http://demoapi.apifortress.com/api/retail/product?q=red
http://demoapi.apifortress.com/api/retail/product/3
January 21, 2021
CONFIDENTIAL
January 21, 2021

Quality at Speed: More API Testing, Less UI Testing

  • 1.
    Quality at Speed: MoreAPI Testing, Less UI Testing January 21, 2021
  • 2.
  • 3.
  • 4.
    @titusfortner January 21, 2021 ContinuousDelivery Really Means… 4 ● Deliver ● Quality Software ● More Frequently
  • 5.
    @titusfortner January 21, 2021 DigitalConfidence 5 Continuous Delivery Requires Continuous Testing
  • 6.
    @titusfortner January 21, 2021 DOMto Database Testing is Difficult 6
  • 7.
    @titusfortner January 21, 2021 “Magic”Solutions 7 Manual Testing doesn’t scale Test Automation Code is hard Magic doesn’t work: ● Record & Playback ● Codeless Test Generation ● Machine Learning ● Artificial Intelligence
  • 8.
    @titusfortner January 21, 2021 Humansvs Computers 8 © Alexey Litvinenko
  • 9.
  • 10.
    @titusfortner January 21, 2021 Moreis Not Always Better 10 Positive Value Negative Value Tests/ Resources Requirements Exceeding Resources Successful Testing Building Coverage Lost Cause
  • 11.
  • 12.
  • 13.
  • 14.
    Why You ShouldUse API Testing?
  • 15.
    To create aworld of digital confidence. SAUCE LABS PURPOSE
  • 16.
    CONFIDENTIAL PRODUCTION TEST • Smoketesting • User acceptance testing • Visual Testing • Production testing • Debugging • Error reporting DEVELOPMENT TEST • Static code analysis • Unit testing • Component testing • Component Testing • Sanity Checks • Security Testing • Visual testing INTEGRATION TEST • Regression testing • Visual testing • Exploratory testing • Integration testing • UI Tests • API testing • Performance testing You Can Build a DevOps Test Toolchain Three Silos of Quality Signal Makes Risk Mitigation Difficult to Interpret
  • 17.
    CONFIDENTIAL … or Adopta Solution An Integrated DevOps Test Toolchain Development Test Integration Test Production Test Apply Insight Data Insights CONTINUOUS IMPROVEMENT Single view of all quality signals across the pipeline Framework agnostic to support various personas and languages Flexible test execution options (cloud or local) Rich data insights to test better, and continuously improve Ability to collaborate, share and view results across teams Code Analysis Team Analytics Unit Tests API Tests Browser/Mobile Functional Tests Visual Tests Error Tracing Security Scans Non-Technical & LowCode Automation Live Testing Error Reporting CI/CD
  • 18.
    @mmerrell © Sauce Labs,Inc. 18 FEBRUARY 22, 2020 We must use data, driven by user signals, to generate and validate requirements
  • 19.
    THE JOURNEY © SauceLabs, Inc. 19 Requirements Real-world user Behavior is now available before requirements are gathered, better informing focus group and research Monitor Analytics Look at both overall traffic (compared to Model) and qualitative user behavior patterns Development Development understands User Traffic Model, and has more trust & confidence in Requirements Testing New tests are added to automation suite based on analytics gathered after previous release cycle Release Feature flags, quick-rollback, and New User Traffic Model is in-place before release
  • 20.
    THE JOURNEY Planning UATMaintaining Monitoring Releasing Coding Testing © Sauce Labs, Inc. 20 @mmerrell SHIFT LEFT SHIFT RIGHT
  • 21.
  • 22.
    How Should YouTest APIs
  • 23.
  • 24.
    CONFIDENTIAL 24 Test Endpoints TestEverything (including Response Code & Content-Type) Test Objects March 5, 2020
  • 25.
  • 26.
    CONFIDENTIAL 26 • Variabilizefor Test Reuse • Confirm Business Logic & Datatypes • Use Dynamic Data • Integration / End-to-End Test • Confirm Response Matches Request January 21, 2021
  • 27.
  • 28.
    28 API Monitoring –A False Sense of Security | [ { quantity: 5, color: ["white","red"], price: 29.99, imageURL: http://apif.com/baseball_cap.jpg, name: "Baseball Cap", description: "Boston Red Sox Baseball Cap", id: 1, category: "head", }, { quantity: 7, color: ["blue","yellow","red"], price: 39.99, imageURL: http://apif.com/long_sleev_shirt.jpg, name: "Long Sleeve Shirt", description: "A wonderful long sleeve shirt", id: 2, category: "body", }, { quantity: 50, color: ["red","gray"], price: 49.99, imageURL: http://apif.com/earmuffs.jpg, name: "Earmuffs", description: "Keep those ears warm in the winter!", id: 3, category: "ears", }, ] http://demoapi.apifortress.com/api/retail/product?q=red January 21, 2021
  • 29.
    29 API Monitoring –A False Sense of Security | [ { quantity: 50, color: ["red","gray"], price: 49.99, imageURL: http://apif.com/earmuffs.jpg, name: "Earmuffs", description: "Keep those ears warm in the winter!", id: 3, category: "ears", }, ] http://demoapi.apifortress.com/api/retail/product?q=red http://demoapi.apifortress.com/api/retail/product/3 January 21, 2021
  • 30.