How to Optimise Continuous Testing

Achieving Continuous Testing through Test
Optimization and Automation
June 28th, 2018

Gerie Owen
VP, Knowledge and Innovation US
Giovanni Rago
Solutions Engineer, EMEA
Webinar Presenters

Agenda
Continuous Testing Test Optimization
Introduction
SummaryTest Automation Q&A

About
QualiTest
QA and testing is
all that we do!
World’s largest
pure play
testing
company
Over 3,000 testing
professionals
worldwide
Business
Assurance
QA &
Testing
Focus
Outcome
Based
Testing
(OBT)
Contractual guarantees for
quality improvement &
cost reduction
Managed
Testing
Solutions
(MTS)
Long term, SLA-based testing
outsourcing
Industry & Technology
Expert
Global
Service
Delivery
Model
On-Shore, Near-Shore,
Off-Shore and Crowd
Testing

Sauce Labs
Analytics
Comprehensive
Coverage
Secure
Most
Reliable
Experience
DebuggingHighly Scalable

Agenda
Introduction
SummaryTest Automation Q & A

ContinuousTesting
| ContinuousTesting is the basis for Continuous Delivery
| Continuous Delivery requires not only increased velocity, but more importantly
increased quality.
| Testing Cycles must be shortened yet quality must be increased.
| Increases in microservices architectures require high-quality functional testing including
automated API testing.
| Builds, integrations and deployments are automated; testing must be integrated into the
pipeline.

ContinuousTesting
| A core capability in Agile and DevOps where all testing activities run continuously in
an integrated fashion with development and delivery. Immediate bug fixing is
enabled, test environments are provisioned instantly, and unit, functional, and
nonfunctional tests are run in an automated way, orchestrated by Continuous
Integration and Continuous Delivery tools. The ForresterWave™: ModernApplication
FunctionalTest AutomationTools, Q4 2016

ContinuousTesting
| ContinuousTesting requires risk analysis and process improvement as well as
implementation of automation throughout the entire software development process.
| “Shift Left” i.e., moving automated testing to the lowest levels of the testing pyramid
is an important part of the approach.

ContinuousTesting vsTest Automation
| ContinuousTesting requires automation, however, it encompasses much more.
| ContinuousTesting is an approach to managing risk by focusing not only on improving
testing efficiency but more importantly, increasing the effectiveness of our test
processes.

| Assesses business risk coverage as its primary goal
| Establishes a safety net that helps the team protect the user experience
| Requires a stable test environment to be available on demand
| Integrates seamlessly into the software delivery pipeline and DevOps toolchain
| Delivers actionable feedback appropriate for each stage of the delivery pipeline
Attributes of ContinuousTesting

ContinuousTesting
| ContinuousTesting Minimizes Business RiskThrough:
| Test Optimization
| Automation throughout the Continuous Delivery Pipeline

Test Optimization
| Test Optimization encompasses risk assessment of the entire test scope:
| Analyze test case for quality, effectiveness, automation potential
| Evaluating the test suites to determine if they are providing the maximum amount of coverage
with the fewest number of test cases.

OptimizeTest Scope
| Select end-to-end tests that focus on the critical and high-risk features of the
application
| Understand all integrations involved in critical user workflows including the
technologies employed in those applications (web, mobile, message/API-layer etc.)
| Implement service virtualization for integrations and components that aren’t readily
available in order to run end-to-end test continuously
| Use the most realistic data possible for each test, include synthetic data generation
where needed
| Include exploratory testing to find user-experience and other bugs not easily found by
automated tests

OptimizingTest Cases
| Tests should be focused on the functionality they are validating.
| Tests should be as simple as possible; the more complex the test, the more validation
is required for the test code itself.
| Even though service virtualization can be used to fill in for missing components in
integration testing, tests should as independent as possible.

OptimizingTest Suites
| Include test cases that validate high risk areas of code
| Include test cases that validate customer-centric features
| Include test cases that validate critical business workflows
| Use FunctionalAutomation tools that create multi-purpose scripts
| Cross-PlatformTesting
• Scripts should run on Desktop,Web, Mobile & Server
| Cross-BrowserTesting
• Scripts should run on multiple browsers including Chrome, Edge, Safari and any others required
| PerformanceTesting
• Scripts should collect performance statistics including page loads etc.

Agenda
Summary
Introduction
Q & ATest Automation

Test Automation
| Test automation is critical for Continuous Integration and Continuous Delivery
| Fast feedback is not just a nice-to-have
| Automated tests require a strategy
| Which cases to automate?
| How to automate them?
| When to run automated tests? (At which point in the delivery pipeline?)
| An automation framework can be a software project of its own
| Specific technical know-how is needed
| Maintenance needs to be accounted for
| Good software engineering practices apply

Dependable, Consistent, ReproducibleTests
| Test environment should mimic production to the highest degree possible
| Having a system free of artifacts or stale data is paramount for reproducible results
| Every session should be “fresh”: clean system cache, browser cache, cookies, etc.
| Make sure your execution workstation and system under test have enough bandwidth
and memory to support the number of simultaneous testing threads

Enabling BetterTesting
| Use the right tool for the job
| Keep functional tests separate from performance tests
| If reliable testing can be achieved faster with a unit test, use a unit test
| Place functional tests in the appropriate place in the build
| Create shortest and most succinct tests possible by creating a state closest to the
single feature being tested
| API Calls to insert data
| Deep linking
| Setup andTeardown Methods
| …enable atomic and autonomous tests!

KeepingTests Atomic
| Atomic
| Test a single feature
| Short and succinct

KeepingTests Atomic
| Example:

KeepingTests Autonomous
| Autonomous
| Tests should run independently from other tests
| One test’s actions should not drive another test’s acceptance criteria

| (Bad) Example:

| (Good) Example:

Benefits of Atomic and AutonomousTests
| Easier to debug and maintain
| Atomic -> Concise tests tend to be less complex
| Autonomous -> Low coupling ensures tests are “sandboxed”
| Easier to parallelize
| Atomic + Autonomous -> Independent tests testing independent features

Why Run in Parallel
| High test coverage across multiple key platforms is highly beneficial
| …but the resulting number of tests can be high, and execution time can explode
| Running tests sequentially is not an option anymore: parallelization is needed

Agenda
Summary
Introduction
Test Automation Q & A

How to Optimise Continuous Testing

In this document