BCS Bristol Autumn School
Testing your App
Jim Thomas
Director of Software Testing
TVS background
TVS Germany
TVS UK
(2011)
(2008)
TVS France
(2012)
T&VS India
(2011)
Delivering tailored solutions for software testing and
hardware verification
What we’ll cover
• Why test?
• Mobile testing challenges
• Developing a test strategy
• Testing different aspects of your App
Why test?
• Identifies bugs introduced in development
• Ensures there is no unwanted functionality
• Reduces development and schedule risk
• Reduces the cost of failure
• Confirms that the software works as desired
• Provides confidence in the software
• Measures quality, helping to decide when to
release software to the market
And if you don’t …
Fail to do a good job of testing and your
customers will end up doing it for you
Customers expect an App to work …..
and when it doesn’t there are other options just
a couple of clicks away
No hiding place …..
Testing is not just about finding bugs
it’s about gaining confidence in
the quality of your App
Mobile Testing Challenges
Device diversity and more
Web testing is a challenge
Mobile testing is even more of a challenge
And add to that …
Web App
servers servers
AUT
DB
servers
With more challenges
• Rapidly changing landscape – new devices
and OS versions
• Extreme time to market pressure for new apps
and versions
• Frequent updates to 3 rd party Apps
• Greater performance concerns – memory,
CPU, power
• Lack of mature testing tools
Developing a Test Strategy
We all know this ….
>150X
Relative Cost of Defect Fix
50X
20X
10X
1X 5X
Requirements Design Code Dev Test Accept Test Operation
Source: Boehm – Equity Keynote Address 2007
So shift ‘Testing’ left
High Level Low Level
Requirements Code Dev Test Accept Test
Design Design
Defects
High Level Low Level
Requirements Code Dev Test Accept Test
Design Design
Defect Detection Point
Elements of a test strategy
• Take account of development flow and release phases
• Levels of testing
– Unit
– Integration
– Application
– End-to-end system
• Test exit criteria
• Test environment
– Devices/emulators etc
• Test automation tools
• Resourcing
– In-house/Outsourced/Crowdsourced
Test environment – devices
• Time/cost constraints and device availability
limit devices to test on
• Master list of ‘must have’ and ‘good to have’
devices
• Group based on manufacturer/OS
version/screen size etc
• Identify candidate device in each group
• Test on emulators until App stable
• Full test on real candidate devices
• Sanity test on other devices
Test environment – remote devices
• Subscription based SaaS
• Remote access to hundreds of real devices
• Connected to real networks around the world
• Record & play/keyword based scripting
• Detailed reports
Perfecto mobile
DeviceAnywhere
Test environment - emulators
• Access to a large number of device
types/browsers
• Easily switch devices types
• Fully instrumented
• Lack quirks, faults
and characteristics of
real devices
Resourcing - crowdsourcing
• On-demand extension of testing resources
• For ‘in the wild’ testing
• Coverage across devices, networks, locations
and languages
• Can be cost effective and quick
• Feedback analysis required
uTest
Testing Different Aspects of Your App
Unit testing
• Should be first priority
• Separate UI and app code logic to make unit
tests more granular and faster
• Decouple unit tests from emulator to make
faster
Roboelectric, JUnit (Android)
OCUnit, GHUnit, OCMock (iOS)
Application testing
• Functional
– Does App perform intended functions?
• UI
– Does it meet platform requirements etc?
• Usability
– Is it easy to use/intuitive?
• Localisation
• Performance/load
• Security
– Authentication, authorization, privacy etc
Manual testing
• Traditional method of mobile testing
• Manual execution of test cases
• Visual verification of results
• Repetitive and time consuming
• Prone to human error
• Can be costly and slow when frequent
releases
• Still main option for UI, Usability,
Performance/load and Localisation testing
Automated testing
• Primarily used for functional testing
• Define tests once & execute multiple times
• Useful for regression and cross-platform tests
• Can reduce effort/time to market
• No generic automation framework available
for all mobile devices
• Some features can’t be simulated so will
require manual testing
Test automation tools
• Android • iOS
Robotium (free) MonkeyTalk (free)
MonkeyTalk (free) QTP (paid)
QTP (paid) Ranorex (paid)
Ranorex (paid) eggPlant (paid)
eggPlant (paid) iOS Instrumentation
DDMS (monitor CPU (monitor CPU and
and memory) memory)
Testing – the network challenge
• Multiple network technologies
• Protocol handling differences
• Mobile web proxies imposing restrictions
• Test over internet or LAN (network bypass) –
emulators will do this
• Also some emulators can internet access
mobile web proxy for testing
• Use real devices on location where possible
Security testing
• Threat Modelling
– identifying possible attacks and defining controls to
prevent them
– prevent/detect/mitigate
– defensive v adversarial
• OWASP Mobile App Threat Model (beta)
• Open Web Application Security Project
• Top Ten Mobile Risks
Threat modelling
Security testing tools
• Static analysis
– Clang static analyser (Objective C)
– FindBugs (Java)
– Veracode (all platforms)
• Dynamic testing
– OWASP Zed Attack Proxy
• Forensic analysis
– Android Debug Bridge
– iPad File Explorer
– SQLite Database Browser
And finally …
You are not alone
65%
Do not have the right tools to test
52%
Do not have the devices readily available
Do not have the right testing process/method
34%
29%
No mobile testing experts available
19%18%
Do not have in-house test environment
Not enough time to test
World Quality Report 2012-13
Capgemini, Sogeti, HP Software
Testing your App
o Mobile App testing is challenging
o Focus on building quality and security in
o Develop a clear test strategy
o Make use of test automation
o Manual testing and real devices are part of
every test strategy
Don’t leave testing to your customers
Thank you
Jim Thomas
jim.t@testandverification.com
TVS Test and Verification Solutions
SETsquared Business Acceleration Centre
Engine Shed
Station Approach
Temple Meads
Bristol
BS1 6QH
Checklists
• Android app testing check-list
• http://testeropine.blogspot.in/2012/10/testing-checklist-for-
android-app.html
• iOS app testing check-list
• http://testeropine.blogspot.in/2012/10/testing-checklist-for-ios-
app.html
• Scripts that automate many static security testing preparation
tasks for both iOS and Android
www.smartphonesdumbapps.com
