Guide to UI Testing with

Continuous Integration
Guide to UI Testing with Continuous Integration | 1
Contents

3 | Why you should automate UI tests

6 | How to write robust UI tests

9 | How UI tests fit into CI/CD pipelines

12 | How to generate helpful bug reports

15 | Cross-platform testing strategies

17 | How TestComplete streamlines UI testing

18 | Questions and answers about TestComplete

Guide to UI Testing with Continuous Integration | 2

Why you should Functional user interface tests, or UI tests, are
one component of the testing process, and the
For example, an Agile team may come up with
user stories to describe certain pieces of function-

automate UI tests testing process is just one component of the ality, prioritize them in the order of importance,
software development process. Before diving and deploy new functionality in two-week cycles.
into the details of UI testing, it's important to Stakeholders can collect user or customer feedback
understand how it fits into the bigger picture of from these smaller iterations and use it to inform
Agile development and the DevOps process. future development work to avoid working on the
wrong things.
Let's take a look at Agile development, testing
pyramids and how DevOps plays a role in both.
UI Tests & the Testing Pyramid
An Agile Development Primer
Test-driven development, or TDD, has become
Agile development is an iterative approach to one of the most important components of Agile.
software delivery focused on building incrementally According to a growing body of research, these
from the beginning rather than delivering it all at practices greatly reduce the time it takes to
the end. By breaking projects down into small deliver a software project and enhance product
pieces, prioritizing them, and continuously delivering quality by reducing the number of bugs that
them in short cycles, the process helps keep software reach production and ensuring the business
projects on time and on budget. requirements are met on a continuous basis.

Guide to UI Testing with Continuous Integration | 3

 There are many different types of tests:

| Unit tests ensure that individual methods or

Testing pyramid functions return the correct results. These
tests are usually written by developers before
writing any code, tested in isolation (e.g. without
dependencies), and constitute the majority of the
tests (e.g. the base of the testing pyramid).

| Integration tests ensure that different parts of an

application work together. These tests are usually
written by test engineers before merging any code
and constitute the middle portion of the testing
pyramid.

| Functional UI tests are similar to integration

tests in that they test different parts of an
application, but they ensure that the high-level
functionality described in the system specification
passes. These tests are usually written by test
engineers prior to deployment and constitute the
top portion of the testing pyramid.

The idea behind the testing pyramid is that the

majority of the tests are unit tests, while the fewest
tests are functional UI tests. In most cases, unit tests
are quick and inexpensive to produce and maintain,

Guide to UI Testing with Continuous Integration | 4

but provide a high return on investment by ensuring merge. If something doesn't pass, they create a bug
that key functions work. While there are fewer report and send it back to developers to address.
functional UI tests than unit tests, they are critical
Functional UI tests are unique in that they may be
for ensuring that an application works correctly and
manual or automated. Manual UI testers go through
delivers business value. key user workflows to determine if an error occurs
before each deploy. While manual testing is ideal for
How DevOps Can Streamline It
exploratory tests, automated tests are a great way to
Most Quality Assurance (QA) teams would agree that cut down on the time it takes for regression testing

unit and integration tests should be automated with of UI workflows.

continuous integration (CI) build server. With each Looking Ahead

commit and merge, the CI server builds the appli-
UI tests can be challenging to automate without
cation and runs the associated tests. Any failures
the right tools and processes in place. A failure to
prevent the merge from occurring and generate
overcome these challenges may lead some orga-
relevant error reports for follow-up by the team.
nizations to abandon UI testing altogether, which

For example, a developer might write a unit test could compromise product quality by removing
key safeguards.
for a given piece of functionality and won't commit
it until they write the code to make it pass. Test In the next chapter, we take a look at how to write
engineers might run integration tests based on more robust functional UI tests before moving on to
business requirements and run them with every CI/CD integrations and other more advanced topics.

Guide to UI Testing with Continuous Integration | 5

How to write robust UI tests can be challenging to create and auto-
mate. For example, suppose that your applica-
continuous delivery. Since there are more people
committing code than maintaining tests, it's

UI tests tion relies on a third-party API that experiences

a temporary outage — your tests may fail even
impossible to fix each flaky test one-by-one. These
flaky tests would be worse than no tests at all
though there's nothing wrong with the underly- since at least the lack of tests doesn't hold up the
ing application code! development process.

Let's take a look at how to write more robust UI How to Make UI Tests Scale
tests and how TestComplete can help simplify the
process and enable more complete test coverage. There are several steps that you can take to make
functional UI tests more robust and scalable. The
How to Improve Your UI Tests key is isolating them from dependencies, accu-
rately recognizing elements, and taking steps to
Functional UI tests are challenging because the
improve maintainability over time. By investing
user interfaces tend to evolve more often than
more time at the start and using the right tools,
the rest of the codebase. For example, a small
you can save a lot of time over the long-term and
change to the class or ID of an element on a web
avoid brittle tests.
page could cause an entire test to fail if the script
used it as an identifier. It's hard for developers to Some important steps to consider include:
understand the ripple effect of these small changes.
Mocking backend servers to isolate front-end code
There are some cases where tests can be nearly and eliminate any issues with unreliable API calls
impossible to run in an automated fashion. For or other backend requests that can throw errors.
instance, you may want to run tests in a virtual
desktop (e.g. Citrix) environment with a single large Avoiding any trivial UI tests that aren't important.
image. Automating these tests may require using Every test should be either an important user
XY coordinates to identify and interact with user workflow for the business or a highly trafficked
interface elements — a time-consuming and error- user workflow.
prone approach.
Building tests that are easy to refactor by using
Unreliable UI tests are a significant obstacle to don't-repeat-yourself (DRY) principals, sensible
Agile development because it blocks high-speed variable names, and other techniques.

Guide to UI Testing with Continuous Integration | 6

Including any edge cases and negative cases for writing the tests for non-developers. GUI-based tion engine, the platform accurately finds dynamic
each test to ensure complete coverage and avoid UI testing tools are easier to use, but they may be user interface elements — even if they are user
unexpected errors despite passing tests. less flexible and adaptable to changing user inter- interface elements appearing on a virtual desktop.
face elements over time.
Improving reliability by adding wait times for async You can store object information separate from
automated UI tests in one central object repository
processes, data loading, or other cases where TestComplete Simplifies UI Tests
that can be stored locally or shared across teams.
you're asserting a state change.
TestComplete is an automated UI testing tool that If any property changes, the platform's 'Intelligent
It's also important to choose a good testing makes it fast and easy to create, maintain and Fix' enables you to update them dynamically during
framework. While many UI testing frameworks are execute functional tests across desktop, web and runtime or with a single click, which makes mainte-
accurate, there's a learning curve involved with mobile. With an AI-powered hybrid object recogni- nance a lot easier over time.

TestComplete's GUI Object Recognition

Guide to UI Testing with Continuous Integration | 7

Finally, TestComplete record and replay capabilities
mean that you can create new UI tests in minutes
rather than hours with minimal scripting. If there’s a
test that needs rewriting, it’s a relatively simple task
that doesn’t have to go on the back burner until a
test engineer has the time to follow-up and rewrite
an entire script.

Looking Ahead

The biggest challenge after creating functional

tests is automating them as part of a continuous
integration and deployment pipeline. For example,
how often should the test run and what should
trigger them? If there are any issues, who is
responsible for maintaining them? Without the
answers to these questions, UI tests can quickly
become unmaintainable.

In the next chapter, we take a look at these issues in-

depth before diving into generating bug reports and
addressing cross-platform challenges.

Guide to UI Testing with Continuous Integration | 8

How UI tests fit into Most Agile development teams use continuous
integration and deployment (CI/CD) processes. For
time and a nice break in-between to fix any bugs and
improve the automation. Other teams run UI tests

CI/CD pipelines example, they may have a Jenkins build server that
automatically compiles an application after each
before each deploy to a new environment.

commit and run the associated tests to check for The good news is that new technologies are mak-

errors. If there is an error, the server may reject ing it easier and faster to run functional UI tests.
the commit until the code passes the test or the For example, TestComplete dramatically cuts down
test is removed. on the time that it takes to create end-to-end tests,
while distributed testing capabilities make the tests
Let's take a look at how UI tests fit into these CI/
a lot more scalable than other testing options on
CD workflows and some best practices to keep in
the market. This means that you can run them
mind when implementing them in practice.
more frequently.

When to Run Functional UI Tests

Example of Integration
Functional UI tests take longer and consume more
There are many different functional UI test frame-
resources than unit tests. For example, testers
works, but the most popular continuous integra-
may need to spin up a headless browser and tion framework is Jenkins. It's a widely used and
execute a series of actions to test a user interface. well-documented tool that is free, open-source,
These requirements mean that it's limited in terms and available across many major operating sys-
of how many tests that it can run at once and how tems. It also integrates well with a wide range of
fast it can run them based on how many hub/node different tools and technologies.
configurations exist.
Assuming that Jenkins is already configured and
Given the greater overhead, functional UI tests are you have a UI test written, you can add the test to
run less frequently than unit tests that are run with Jenkins by converting the test into a batch file and
each merge. Most quality assurance teams run them adding it as a new job in Jenkins. You can then cre-
over the weekend every week to provide enough run ate a schedule for the job (e.g. weekends or based

Guide to UI Testing with Continuous Integration | 9

 on a trigger), as well as configure an email or other
notifications with the results.

In addition to the learning curve for setting up

TestComplete's Integration with Jenkins this integration, it can be challenging to manage
a large suite of UI tests that have been integrated
into a single build server. Many organizations use
external test management suites, such as Jira, to
manage complexity, streamline reporting, and
ensure that everyone is on the same page.

TestComplete simplifies the process via integra-

tions with Jenkins and other DevOps tools. Using a
lightweight execution engine and distributed testing,
the framework makes it easy to automate tests on
virtual computers and integrate with top CI systems,
like Jenkins, and SCMs, like Git, with minimal configu-
ration — you just have to install the plugin.

How to Address Any Issues

A continuous integration server records any
errors or failures based on its configuration. For
example, a UI test could be configured to take a
screenshot every time a failure or error occurs to
help diagnose the problem. These screenshots can
be generated and saved by the CI server or even
automatically posted into bug trackers.

Guide to UI Testing with Continuous Integration | 10

The biggest difference between unit tests and UI
tests is that there are more false positives. For
instance, a slow third-party API could lead to a tim-
eout, which isn't necessarily caused by a problem
with the application code. It's important to weed
out these false positives before classifying them as
bugs and placing them in front of developers to fix.

Valid bugs are typically evaluated by a test engineer,

augmented with screenshots or videos, and then
placed in a bug tracking system for developers to
fix. After the bugs are fixed by developers, the func-
tional UI test is re-run to confirm that there are no
remaining issues before the changes are deployed
to a test, staging, or production server.

Looking Ahead

A key challenge with addressing these issues is

clearly communicating any bugs with developers.
Without the right information, developers may be
unable to replicate the error and ultimately find
a solution. Test and quality assurance engineers
should take measures to ensure that they properly
communicate this information to everyone.

In the next chapter, we look at how to generate

helpful bug reports from functional UI tests before
getting into cross-platform testing strategies.

Guide to UI Testing with Continuous Integration | 11

How to generate Bug reports can be challenging for functional
UI tests since there are many different factors
3. Once the developer fixes the bug, it's reassigned
to a tester for verification that it's squashed.

helpful bug reports involved. For example, developers might need to

4. The tester verifies that the bug is fixed or re-opens
know the browser version, screen resolution, or
the bug if it still exists and the process repeats.
other factors in order to troubleshoot the problem.
This means that testers must carefully document is- The key to success is writing clear bug reports that
sues with screenshots and ensure it's reproducible. are well-organized in a bug tracker. That way, de-
velopers can immediately see what bugs they're re-
Let's take a look at some best practices for devel-
sponsible for, reproduce them with minimal effort,
oping helpful bug reports for UI tests.
and quickly dive into the problem to find a solution
Define & Stick to a Process without wasting time. The result is more bugs
closed per unit of time and higher quality software.
The process of creating a bug report may seem
straightforward, but it can quickly grow complex in Essential Bug Report Components
a fast-paced team environment. After all, the num-
Bug reports should follow a standardized format
ber of bugs almost always exceeds the resources
to streamline the report creation process for test
available to fix them, making prioritization of bugs
engineers and make it easy for developers to find in-
a challenge, and duplicate bug reports are a fre-
formation. While the specific requirements may vary
quent problem in large projects.
between companies, there are common components

A typical bug reporting process involves that most UI-focused bug reports contain, which are

several steps: a good starting point for developing a template.

1. A new bug is discovered by a user or tester and The most important components include:
the tester writes up a bug report. | Title: A concise summary of the bug and its category.

2. The bug is assigned to the appropriate developer by | Summary: A short summary of when and how
the project manager with an appropriate priority. the bug occurred.

Guide to UI Testing with Continuous Integration | 12

| Visual Proof: A screenshot or video to help
developers understand the problem faster.

| Expected vs. Actual Results: A clear summary TestComplete makes bug reporting simple
of what result was expected and how what
happened differed.

| Steps to Reproduce: A set of well-defined steps

to reproduce the bug with no ambiguity.

| Environment: The browser, operating

system, screen size, zoom level, pixel ratio,
and other information.

| Console Logs: All of the web page's errors in the

console, which can help identify the root problem.

| Source URL: The URL for the bug.

| Severity & Priority: The impact on the user expe-

rience and how quickly it should be investigated.

TestComplete makes it easy to collect many of

these elements without having to go through a
manual process. Test engineers can easily share
images, videos, or detailed logs captured during

Guide to UI Testing with Continuous Integration | 13

automated UI test execution in various formats — | Take the time to reproduce the bug before
even if they don't have TestComplete installed. You creating a report — don't take a user's word
can even access video recordings to find issues for it that a bug exists. You can then show the

visually without readings logs. developer how to reproduce the error.

| Keep feature requests separate from bug re-

Best Practices to Remember
ports. Since feature requests are more desir-
There are a handful of best practices that test engi- able for developers, keeping them in the same
neers should keep in mind when writing bug reports. place could lead to ignored bugs
While these best practices may not apply to every
situation, there represent some common practices Looking Ahead
that your organization may want to consider when
Clear bug reports and a well-defined bug reporting
developing your bug reporting processes.
process goes a long way in improving functional UI
The most important best practices include: tests. Using modern tools like TestComplete, you

| Only include one bug per issue. Don't combine can quickly collect relevant screenshots, videos,

multiple related bugs into a single issue for and console data to share with developers to help

developers to solve since they may be unrelated them reproduce and quickly diagnose bugs with-

under the surface. out having to spend as much time collecting the
material by hand.
| Search to make sure that the bug hasn't already
been reported. If it has been, check the existing In the next chapter, we take a look at cross-plat-
bug report to see if you can add any new infor- form testing strategies for applications that need
mation based on your findings. to work across mobile, web and desktop.

Guide to UI Testing with Continuous Integration | 14

Cross-platform Maintaining functional UI tests across different plat-
forms can be a challenge. For example, you may
tend to be longer. Other companies may run their
own proprietary versions of certain browsers.

testing strategies need to create tests that ensure a web application

Google Analytics offers a precise way to discover the
works across various desktop and mobile browsers
browsers, operating systems and devices that your
with different screen resolutions. With so many
existing visitors use. After setting up Google Analytics,
different combinations, it's hard to know where to
you can find this data by going to the "Audience" sec-
start and how to maintain such a test suite.
tion. The "Mobile" subsection shows mobile device
Let's take a look at how cross-platform UI tests are usage data, while the "Technology" subsection shows
normally handled and how TestComplete makes you browser and operating system breakdowns.
the process easier. You should consider testing any configurations with
significant usage.
What Platforms to Test
In addition to browsers, you may have companion
The ideal functional UI test suite would test every
mobile and desktop applications that require UI
combination of browser, operating system and
tests. The good news is that it's easier to determine
device, but with tens of thousands of combinations,
the devices and operating systems to test (in most
that's impossible. In addition, it doesn't make much
cases), but the bad news is that these tests typically
sense to spend hours building and maintaining a test
run using different platforms than web-based user
suite for — and updating an application to work with
interface tests, making it challenging to integrate.
— an old browser version with a handful of users.
Building Cross-Platform Tests
On the other hand, you cannot necessarily only
with TestComplete
support the latest browsers. For instance, a software-
as-a-service application targeting enterprise custom- TestComplete simplifies the cross-platform testing
ers may have a lot of users with older browsers and process with a robust set of capabilities. With min-
operating systems since enterprise upgrade cycles imal scripting, you can easily create cross-platform

Guide to UI Testing with Continuous Integration | 15

tests that leverage artificial intelligence to recognize | Web: You can create reusable cross-browser tests across all of these platforms with the same
objects on a screen rather than searching for them in for over 1,500 real browsers, operating systems, amount of resources to devote to testing efforts.

code. You can even customize mobile gestures and devices and resolution configurations.
Looking Ahead
other advanced UI interactions. The platform works
| Mobile: You can create and automate functional Cross-platform UI tests can be a challenge due to the
with nearly any popular software or device, eliminat-
wide range of potential configurations. While many
UI tests on physical for virtual iOS and Android
ing the need for patchwork solutions. quality assurance teams use a patchwork of solu-
devices without jailbreaking your phone.
tions, TestComplete makes it easy to keep everything
TestComplete supports most
| By supporting multiple platforms, TestComplete under one roof and integrate with external testing
major platforms:
tools, including Jenkins and Git. It’s a great way to
enables you to keep all of your functional UI tests
| Desktop: You can automate UI tests for desktop ensure maximum coverage with minimal investment.
under the same roof, making it easy to standard-
applications across a wide range of tools and con- ize testing and reporting. The ease of use also In the next chapter, we take a look at why you should
trols, including .NET, Windows 10 and Java. means that you can create more test coverage try TestComplete and answer some common questions.

Guide to UI Testing with Continuous Integration | 16

How TestComplete Functional UI tests can be challenging for a variety
of different reasons. Since user interfaces fre-
includes support for 1,500 different combinations
of browsers and operating systems, native desktop

streamlines UI quently change, UI tests tend to be brittle and applications, and native mobile applications.
throw a lot of false positives. There are also a large
testing number of potential browsers, operating systems,
“TestComplete improved application testing time
between 40-50%. It has helped us reduce resource
devices and platforms to test, which makes it chal-
costs by as much as $350K.” - Sandor Albert, Se-
lenging to reach complete test coverage.
nior Software Engineer, Philips Healthcare.
TestComplete alleviates these issues by handling
every aspect of end-to-end functional UI tests with- “We've been using TestComplete heavily for the last

in a single platform. Using AI-based object recogni- 3 years. Since that time we've been catching defects
tion, the platform can easily recognize UI elements earlier in the release cycle, releasing quicker and
based on their appearance rather than looking at the number of defects found post-release has dra-
classes or IDs that might change. The platform also matically dropped.” - G2 Reviewer

Download a free trial to learn how TestComplete

can help you today!

Try TestComplete for Free

Guide to UI Testing with Continuous Integration | 17

Questions &
1
Which browsers does TestComplete support
answers about for testing web pages?
TestComplete TestComplete supports all major browsers, including Chrome, Firefox, Edge and Inter-
net Explorer. Beginning in version 14.4, TestComplete also supports Safari on MacOS
and browsers on iOS and Android devices.

The platform also provides access to web pages and their elements displayed in
WebBrowser and Chromium Embedded Framework controls that reside in any
application. In other words, TestComplete supports any browser created based on
the WebBrowser and Chromium Embedded Framework controls.

2
How can I take test results from TestComplete and send
them to developers or to my boss?

| Select the desired log node in the tree on the left of the test log page (if you select
the root node, TestComplete sends the results of the whole run, if you select a
subnode, the item contains results shown by this subnode and its child nodes).

| Right-click the selected node and choose "Send via E-mail" from the context menu.

TestComplete packs the results to a Multipart Hypertext Storage file (.mht), attach
it to the email message and initiate the sending process. In addition you can also
export TestComplete results as JUnit XML that can be imported into any reporting
tool of your choice.

Guide to UI Testing with Continuous Integration | 18

3
Are there checkpoints available in TestComplete?

Currently, you can use the following checkpoint types:

| Property checkpoints - verify an object's property value.

| Table checkpoints - verify data in tabular controls (grids, list views, list boxes, etc.)

| File checkpoints - compare files.

| Mobile checkpoints - verifies that the desired image is displayed on the screen of
a mobile device.

| XML checkpoints - compare XML documents.

| Database checkpoints - verify data stored in a database.

| Region checkpoints - compare images, or searches for an image within another image.

| Web audit checkpoints - perform a quick performance, accessibility and SEO audit
of your web application or website

| Web accessibility checkpoints - perform various checks of web pages (for example,
checks for broken links).

| Web comparison checkpoints - compare web pages.

4
Does TestComplete integrate with Continuous Integration
(CI) systems?
Yes, TestComplete integrates with all common Continuous Integration systems today,
including Jenkins.

Guide to UI Testing with Continuous Integration | 19

5
Does TestComplete allow multiple users to work on a test
project simultaneously?

There are several ways to allow multiple users to work on a test project simultaneously:

| Work with test projects in shared mode - Store your TestComplete project in a
network drive, and give all users access to it so that they are able to open, edit, and
run these test projects on their computers.

| Bind test projects to a source control - Add your TestComplete test projects to
your source control system such as Git so that all the needed users can access
the repository, get the needed test project files, commit changes they made to
the project files to the repository, and so on. In addition, TestComplete provides
advanced support for a number of source control systems allowing you to work
with them directly from TestComplete IDE.

6
Does TestComplete support BDD and Gherkin-style syntax?
Yes, TestComplete has a native Gherkin IDE and Cucumber engine to help you
accelerate your Behavior-Driven Development initiatives.

7 Does TestComplete integrate with Jira?

Yes, TestComplete integrates with Jira natively or via our test management solution
called Zephyr for Jira.

Guide to UI Testing with Continuous Integration | 20

 Get started with automation quickly.
Whatever your team’s skill level, TestComplete is built to seamlessly integrate with
your process and get you more complete test coverage.

Start Your Free Trial Today

Guide to UI Testing with Continuous Integration | 21

Guide to UI Testing with Continuous Integration | 22