www.MyYesM.

com
A Course in
Mobile Applications Testing
Presented by: Shailesh Joshi
Agenda
>> Complexity of Mobile domain
>> Traditional vs. Mobile testing
>> Strategies for testing mobile software
>> Types of Mobile Apps
>> Baseline Testing
>> Mobile Platforms
>> Platform specific testing tips
>> Testing on Emulators
>> Testing Automation
The Complexity
 Diversity of Platforms: Today's mobile devices differ not just at the firmware level, but
also at the operating system, browser and the application runtime layer (like Java or
BREW) levels. Testing for compatibility between these layers, as well as with the
network interface, is important to deploying trouble-free mobile services.

 Multiple interface and rendering standards: Handset manufacturers are moving toward
dual- or tri-mode phones capable of supporting multiple network standards. Also
applications rendered in HDML, WML, WAP, cHTML, xHTML, CSS, AJAX will continue
to require focused testing at the presentation level.

 With the advent of highly functional smart-phones and handsets that have rich client
execution environments, persistent data store and high speed network connectivity,
testing becomes much more complex.

 Security issues: With the ability to download apps and execute code on the device
comes the risk associated with user authentication, enterprise data and transactional
security as well as viruses and other malicious code. The newfound power of the mobile
desktop increases the need for testing and verification of security mechanisms, both in
the devices themselves and in the network infrastructure.

 Wireless Networks: 2G, 3G, GPRS/EDGE, Bluetooth, Wi-Fi/Wi-Max, etc.

The Basics
The number and type of mobile applications for mass market users (and enterprise
users) is exploding (250000+ i-phone apps and counting). With each of these types of
applications, there are a number of factors that must be considered when testing for
conformance to functional, user and performance requirements. Any good tester
knows to pay attention to the basics, which can include:

 Verifying the baseline functionality and features

 Checking the design and proof-of-concept solutions against user requirements early in the
development cycle
 Testing under tightly controlled conditions to validate code against design
during later stages of the development lifecycle
 Compatibility testing all known and planned variations in the software and hardware
configurations where the application will run
 Exposing the entire system or app to unexpected events, faults in dependent
databases, networks or other apps, or unpredictable user behavior
 Subjecting the software to volume, load and stress conditions to gauge performance at
the boundaries of its designed capacity and measure actual limitations of that
performance
 Determining if the system or app not only meets formal design requirements, but
also whether it will be usable and meet the (undocumented) needs of its users
Testing Strategy
Matrix
 Compatibility Testing

 Non-functional testing to evaluate application's compatibility with

the computing environment

 Browser compatibility
 Carrier compatibility (Verizon, AT&T, Sprint, etc.)
 Device/hardware compatibility
 Phone models

 Memory size, processors

 Operating system compatibility

 Backward compatibility
 Penetration Testing

 Non-functional testing to uncover security vulnerabilities resulting

from poor software design, hardware issues, improper system
configuration, malicious code, etc.

 Buffer overflows / poor memory management

 Third party unsigned apps on jail-broken phones
 Cross Site Scripting / Cross Site Request Forgery
 Unprotected data
 Keyboard cache
 Snapshots
 Clipboard data
 On-device database
 Log files
 Testing Mobile Devices
Strategies

 Must pay more attention to usability and form factors than with traditional
desktop applications. Smaller screens, slower processors, limited memory and network
bandwidth all require different testing methods. Testers must be aware of specific
methods developers use to compensate for these factors and develop testing methods
accordingly.

 Need to develop expertise in testing multiple layers of the mobile app's

software model. The diversity of device hardware platforms, operating systems, micro-
browsers and middleware require test experts to be aware of the
compatibility issues impacting functionality and performance of mobile apps.

 Be prepared to modify test strategies to treat the handheld as a computing

system itself, not as a simple client as has been the case with mobile devices in the past.
Mobile apps require testers to increase the focus on end-to-end testing as
interaction between handheld applications and enterprise data stores increases data
processing complexity.

 Continued …
 Testing Mobile Devices
Strategies

 Utilize established test beds to test under known network service quality conditions, e.g.
testing application survivability during network crossovers, drop back to slower/older
networks, roaming, etc. The technology and network implementations are not mature
enough to rely solely on lab testing of mobile applications. Testing on a target device,
rather than a prototype or emulator, will enable proper testing of security schemes and
performance under realistic loads (web, client/server, background loads, incremental
wireless traffic, variations in message size).

 Focus on key business features of mobile middleware. If the application allows m-

commerce transactions and depends on automated billing, then placing a test emphasis
on the real-time mobile billing features of your middleware and interfaces to corporate
systems is critical. Likewise, apps that use location-based technology should get special
test attention to both the device side (e.g., a GPS chip) and network side components.
Similar concerns should accompany test strategies for data synchronization (between
mobile devices and enterprise data stores), mobile to non-mobile user collaboration,
enterprise messaging, content formatting/trans-coding/internationalization.

 Continued …
 Testing Mobile Devices
Strategies

 Testing conducted through emulators or simulators cannot accomplish the same level
of quality verification of the user experience as field-based testing will. Emulators and
simulators are useful for validating functionality and compatibility under controlled
conditions, particularly during the development cycle and for white-box testing.
However, field verification is necessary to ensure proper validation of services in a real-
world environment. Each unique configuration instance of a mobile application should
be field tested as a separate solution (device, browser, data, network, carrier, gateway
and enterprise components).
Mobile Apps
 Web apps Native Apps
- accessed via - Pre-installed
Browser - Downloadable
Web Apps
 Web sites/apps built for mobile browsers.
 Accessed by entering a specific URL in mobile browser.
 Optimized for viewing on mobile phone.
 No download/installation required.
 Web-apps expect internet connectivity
 Internet speed and coverage become an

important test case

 Offer nearly all the functionality of traditional websites.
 Technology
 Markup (HTML variants) – Server generated content
 AJAX (rich apps) – Javascript on client + Server
Native Apps
 Apps that are shipped as in-built software with the mobile device or
apps that can be downloaded and installed on the phone.
 Native apps use/enhance the built-in features of the mobile phone
such as motion sensor, voice detection, camera and GPS.
 Native apps can be used offline.
 A native app is developed for a specific mobile platform, e.g. i-phone,
so it isn’t easily portable to other platforms such as Android or
Blackberry.
 Apps can be downloaded from app stores or Internet. They can also
be transferred to the phone via USB or wireless media like bluetooth,
infra-red, etc.
 Technology
 C/Objective-C for iPhone
 Java based SDK for Android
Web Apps vs.
Native Apps
Benefits Benfits
 Standard platform (WWW)  Richer/faster user
 Single app to build/maintain experience
 Cost effective  App store distribution
 From use of device specific
features
Limitations
 On user experience Limitations
 Multi-platform
 Due to access speed and
browser  Lack of standards across
 On using device specific platforms *
features  More expensive
 Steeper learning curve

* WAC (Wholesale Apps Community) intends

to enable development of single mobile
app that would work across carriers,
devices and operating systems.
Mobile Platforms
 iPhone (Apple)
 Android (Google)
 Blackberry (RIM)
 Windows Mobile
(Microsoft)
Functional Testing
 The most essential testing procedure is to verify the
basic functionality of the app. This can be done either
via a mobile simulator(desktop browser) or by field
testing.
 If you discover non functional URL/s during field
testing, then test in desktop browser. If reproducible,
then it isn’t a device specific issue, but a basic
application bug.
 Prioritize functional features early on in the
development phase. Perform quick tests on
devices/simulators based on priority.
Sample test case
 Assert preconditions
 Is the home screen displayed?
 Is the search button focused?

 Fire a RightKey event

 Wait for the dialog

 Assert the new state

 Is the dialog correct?

 Fire a Down and a RightKey event

 Wait while loading the results

 Assert the response

 Is the result screen displayed?
 Is the content correct?
Baseline Testing
 Test in various signal strengths.
 No Network
 Low
 Medium
 High
 Testing during change of signal strength from:
 No Network >> Low to High

 High to Low >> No Network

 Test in different network speeds

 Low
 High
 Ultra (gigabit)

 Generate a metrics to report number of defects in different strengths and

network speeds.
Baseline Testing
 Test in various network types.
 2G
 GPRS

 CDMA

 EDGE

 3G
 Wi-Fi/Wi-Max

 Test in different service plans by MSPs (Mobile Service Providers).

 Test in various battery strengths

 Critical
 Low
 While charging
 High

 Generate a metrics to report number of defects in different network types,

service plans and battery strengths.
 Baseline Testing
 Monitor Battery consumption patterns.
 Battery consumption rate while
 the app is run in the background

 the apps is run in the foreground

 multiple instances of the app are run

 the app is running for long durations.

 Monitor memory consumption patterns.

 Observe memory usage while
 the app is launching and closing

 the app is run in background

 the app is run in foreground

 the app is running for long durations

 Monitor app performance for above conditions when the mobile phone’s
free memory is
 low
 high
Baseline Testing
 Interruptions/Interactions
Activities that occur simultaneously in the mobile device while the app is
installing/launching/running/closing/updating/uninstalling. Interruptions
can happen in the form of
 Incoming call/message/notifications from calendars, alarms, clocks
 Receiving incoming call/message/notifications
 Losing wireless connection/device switched off
 Removing battery
 Activating camera
 Daylight time changes

 Device Settings
Look for configurable device settings on the phone and determine which
may affect the application. Also explore ways in which
you can force error messages in the apps.
Input Modes
Ways to interact with and control the mobile device

 Keyboard/Keypad
 Touch screen
 Virtual keypad
 Trackball/Trackwheel
 Peripherals that can be plugged in to the device

Try out all the different ways inputs can be performed.

 For a touch device, test with single touch and multi-touch inputs. Try different
combinations with multiple fingers, like keeping thumb on the edge of the screen while
typing, or tapping fingers on the device while using it.

 Test in portrait and landscape modes when the app is running in background.

 Test with touch gestures and typing on virtual keypad.

 Even the way devices are held can impact their connectivity and how well they
respond to inputs. Case in point, i-Phone 4 antenna issue. Does holding the device
in a certain way and interacting with it causes the app to crash or freeze up?
Usability Testing
 Should occur as soon as a stable build is available.
 Consider various input/scrolling/clicking/navigation
modes on multiple devices.
 Data intensive apps may feel better on devices with
physical keyboards.
 Touch screens may feel better for some apps than
non-touch screen.
 Report back issues to developers, who may consider
tweaking the UI to reduce severity of problem.
Testing Tips
iPhone
 Human Interface Guidelines from Apple need to be
followed.
 Backward OS compatibility
 4 major releases of iOS so far.
 Possible to debug via USB connection
 Utilities
 Screenshots: Home + Lock keys
 Memory sweep app: Check memory status or free up
memory
Testing Tips
iPhone
 Test under low/very low memory conditions
 Background apps: Safari, Mail, “interrupts”
 Memory sweep app
 Attach crash logs and screenshots when reporting
bugs
 Retrieve .crash file from iPhone
 Built-in screen capture
 Debug via console using iPhone config tool
 Check warnings/errors
 Boundary conditions for text input
 Special/localized/Unicode characters
Testing Tips
Android
 Allows multiple apps to run in background
 During certain changes to device configurations
(screen orientation, language, keyboard availability),
running apps may be automatically reloaded. Test if
app is able to handle shutdown/restart gracefully
without loss of user data or state.
 Possible to debug via USB connection
 Utilities
 Monkey: runs on emulator or device and generates
pseudo-random streams of user events such as clicks,
touches, or gestures. Useful for stress testing apps.
Android:
What to Test
From Android Documentation
http://developer.android.com/guide/topics/testing/testing_android.html

In addition to the functional areas you would normally test, here are some areas of
Android application testing that you should consider:
 Activity lifecycle events: You should test that your activities handle lifecycle events
correctly. For example an activity should respond to pause or destroy events by
saving its state. Remember that even a change in screen orientation causes the
current activity to be destroyed, so you should test that accidental device movements
don't accidentally lose the application state.
 Database operations: You should ensure that database operations correctly handle
changes to the application's state. To do this, use mock objects from the package
android.test.mock.
 Screen sizes and resolutions: Before you publish your application, make sure to test
it on all of the screen sizes and densities on which you want it to run. You can test
the application on multiple sizes and densities using AVDs, or you can test your
application directly on the devices that you are targeting. For more information, see
the topic Supporting Multiple Screens.
 When possible, you should run these tests on an actual device. If this is not possible,
you can use the Android Emulator with Android Virtual Devices configured for the
hardware, screens, and versions you want to test.
Testing Web Apps
Before testing mobile website to determine how it looks on
handsets, make sure the functionality of the site is as expected. At
some point, web site must be tested on a physical device, however
physical testing is expensive and time consuming. So ensure that
time spent on handsets is used to identify device-specific
problems;
functional issues should be handled before device testing begins.

Optimum Testing Process

 Test in desktop browser first to ensure functionality.
 After functionally complete, test with emulators.
 When site is tested successfully in required emulators, start
testing on handsets.
Mobile Web Testing
with Firefox
 Install Firefox Add ons.
The Modify Headers add on
 https://addons.mozilla.org/en-US/firefox/addon/967/

 The User Agent (UA) Switcher add on

 https://addons.mozilla.org/en-US/firefox/addon/59/

 Most mobile web sites look for HTTP header called “x-wap-
profile” to identify a mobile device. Below is a User Agent Profile
(UAProf) for Nokia N97 phone.
http://nds1.nds.nokia.com/uaprof/NN97r100-2G.xml
 Add a new header using Modify Headers add on.
Mobile Web Testing
with Firefox
 Pick a device from www.uaprof.com as shown below

 Type UAProf URL (ending with .xml) in Text Box 2 of Modify Headers
window. Type “x-wap-profile” in the Text Box 1. Type a suitable device
description in the Text Box 3. Click Add button on the right.
Mobile Web Testing
with Firefox
 The new device will be displayed in the list. The green dot at the end
indicates that it is enabled. Disable all other devices by double clicking.
Keep only one entry in the list enabled.
Mobile Web Testing
with Firefox
 Open User Agent Switcher add on
Mobile Web Testing
with Firefox
 Click New >> New User Agent
Mobile Web Testing
with Firefox
 Paste the user agent String into User Agent
field. Also enter a suitable Description. Then
click OK in two windows.
Mobile Web Testing
with Firefox
 Set the newly created User Agent as Default
User Agent.
Mobile Web Testing
with Firefox
 Test by browsing to
www.bbc.co.uk
 The browser will be
automatically redirected
to bbc’s mobile web
site.
Testing with
Emulators
 Traditional web apps testing is relatively easy – test for
compatibility issues in four or five browsers, and the job’s done.

 Not so in the mobile world. Browse to Wikipedia’s Mobile

Browser page, it lists about 20 major mobile browsers. In truth,
there are probably many more. Take a look at the list of devices
supported by DeviceAtlas, it supports over 5000 mobile devices
and continues to add more. With these kinds of numbers, how
are we supposed to test a web site across all devices?

 Test the site on at least one device from all popular

manufacturers, and on as many of the mobile browsers as
possible. This will give you a good idea of how the site works on
different devices and will also help you to resolve any issues.
Testing with
Emulators
 The catch is making sure you have access to all of the
necessary devices and mobile browsers. You may only be able
to use one or two physical devices – not enough to adequately
test on. But emulators can be used instead to allow you to
simulate testing on the real device or browser.
 Benefits
 No data browsing charges.
 Quicker access to devices.
 Access to a large number of devices via emulators.
 Most emulators are available for free.
 Limitations
 Emulators differ in many subtle ways from devices.
 Never be assured that app will look and feel same on
physical device. (Input modes, screen resolutions, etc.)
Testing with
Emulators
Types of Emulators
 Device : Provided by device manufacturers.
Good for testing web site or application
 Browser : Simulate mobile browsers. Not
useful for device specific testing.

Web based Device Emulators

 http://www.testiphone.com
 http://www.iphonetester.com
 http://emulator.mtld.mobi/emulator.php
Testing with
Emulators
 Popular Device Emulators
 Research in Motion
(BlackBerry)
http://na.blackberry.com/eng/developers/resources/simulators.jsp
 Start server first (MDS program)

which runs in a command window

 Then start the device simulator from

Windows Start menu.

 Click Browser button on simulator

and browse to a web site.

 You can type using computer

keyboard or simulator keyboard.

 Testing with
Android Emulator
 Download AppInventor: For Windows:
http://dl.google.com/dl/appinventor/installers/windows/appinventor_setup
_installer_v_1_2.exe
For Mac: http://dl.google.com/appinventor/installers/mac/AppInventor_Setup_v_1.1.dmg
 Install AppInventor
– Locate the file AppInventor_Setup_Installer_v_1_2.exe (~92
MB) in your Downloads or your Desktop. The location of the
downloads on PC depends on how your browser is configured.
– Double click to open the file.
– Click through the steps of the installer. Do not change the
installation location but record the installation directory. The
directory will differ depending on your version of Windows
– On 32-bit Windows machine, go to C:\Program
Files\Appinventor\commands-for-Appinventor
– On 64-bit windows machine, go to C:\Program Files
 Testing with
Android Emulator
 Start Emulator
– To start Emulator: double click run-emulator
– The emulator will initially appear with an empty black screen (#1). Wait until
the emulator is ready, with a colored screen background (#2). You also need
to use your mouse on the emulated phone screen to unlock the device by
dragging the green lock button to the right (#3).
Testing with
Android Emulator
 Download a free android app “Tippy Tipper”
https://tippytipper.googlecode.com/files/Tippy%20Tipper%20v_1_2.apk
 Note down location of the file Tippy Tipper v_1_2.apk
 Open CMD (Command Line Window) and run cd command
> cd C:\Program Files\AppInventor\commands-for-Appinventor
> cd C:\Program Files (x86)\AppInventor\commands-for-Appinventor
 Install the app on emulator using “adb” tool in CMD
> adb install “<Location of Tippy Tipper>\Tippy Tipper v_1_2.apk”
 Observe output of adb in CMD
xxx KB/s (0 bytes in 196024.001s)
pkg: /data/local/tmp/Tippy Tipper v_1_2.apk
Success
Testing with
Android Emulator
 Open Apps screen
 Use Arrow keys, locate Tippy Tipper, Click or Enter.
Testing with
Android Emulator
 Using Tippy Tipper
– Enter a number using number pad, click OK.
Testing with
Physical Devices
 Identify the devices you need to support
based on target customer, e.g. Blackberry for
enterprise apps, iPhone for casual apps, etc.
 Identify the devices released in the recent
past that are popular and decide which of
those you wish to test on.
Testing with
Physical Devices
 On-line Device Testing Providers
 Samsung RTL
 It’s a virtual device laboratory that incorporates remote phone
management service. You can test your developed apps in the same
environment of real device through RTL
http://developer.samsung.com/remotetestlab/rtlAboutRTL.action
 Register for Samsung Developer account

http://developer.samsung.com/sa/signUp.do
 Verify email and then sign into the account

http://developer.samsung.com/sa/signIn.do
 After login, Click Test Lab at the top of the page

 Click on Android Tab

 Pick any device.

 From drop-down, select OS version, a model that is on a server nearest

to you and reservation time of at least 1 hour. Click Start to verify.

 If asked, allow Java WebStart to run the client on your computer

 Click near top right for complete documentation.

Testing with
Physical Devices
 Other Online Device Testing Providers
 DeviceAnywhere.com
 Paid-for online device provider
 Large number of physical devices incl. iPhone, Android,
etc.
 Devices on a specific network in a specific country

 PerfectoMobile.com
 Paid-for online device provider
What is
Calabash-Android
 Open Source Automated Testing Tool for Android and iOS
native apps.
 Uses Cucumber Framework
 Cucumber is a BDD (Behavior Driven Development) tool to write
acceptance tests in a language like plain English.
 Acceptance tests describe behaviour of the application from end
user's point of view.
 Feature: It describes a single feature of the system.
 Scenario: A concrete example that illustrates a business rule. It
consists of a list of steps.
 Step: A step typically starts with Given, When or Then
– Given steps describe the initial context
– When steps describe an event, or an action
– Then steps describe an expected outcome, or result.
Run Calabash-Android
Testcases
 Signup for free Basic Plan at http://testmunk.com/signup
Run Calabash-Android
Testcases
 Login to your account at http://testmunk.com/login
 Click on New Testrun

 Drag-Drop TippyTipper.apk in the browser window or click to

select the file from your computer. Then click Next.
Run Calabash-Android
Testcases
 Drag-Drop features-TIppyTipper.zip to browser window and click
to select file from your computer. Click Next.

 Click Next.
 Click Next.
 Click Start Testrun.
Run Calabash-Android
Testcases
 Allow some time (5 min) for the testcase to complete.

 Refresh the browser screen.

 Click View Results button if available, otherwise wait for some
more time and refresh screen.

 Click on More button to see snapshots for all test steps.

Resources
 Cucumber Framework
https://cucumber.io/docs/reference
 Calabash-Android
https://github.com/calabash/calabash-android
 Calabash Predefined Test Steps
https://github.com/calabash/calabash-android/blob/master/ruby-
gem/lib/calabash-android/canned_steps.md
Resources
 A Practical Guide to Testing Wireless
Smartphone Applications
– by Julian Harty
 Hands on Mobile App Testing
– by Daniel Knott