Determining how to
carry out accurate
monitoring of E2E
data performance
Mobile Network Performance
Management
London June 2014
Martin Harris
Orange Corporate Services Ltd
martin.harris@orange.com
Contents
Why measure data service performance and what
is the benefit?
examining strategies to benchmark and monitor
end to end data performance against competitors
or other countries
ensuring performance is in line with promises
Monitoring end-to-end service performance
understanding the benefits and trade-offs between
the use of network KPIs, probes, robots, drive
tests, network conters
How do we assess performance?
looking at how and where to set performance
targets, what impacts the choice, and how these
evolve for LTE
examining the customer experience by using
speed tests (crowd sourced data) which can give
a specific view of QoS
How can Big Data help us?
or is this just another complexity?
2
Why measure service performance?
To verify performance is in line with promises given
to our customers
business offers and customer expectations
To understand how our networks (and competitor
networks) are performing
best network and meeting customer expectations
identify weaknesses (bottlenecks, slow speed,
packet loss, capacity, interconnect, outages)
ensure we meet SLAs with business partners
optimise expenditure, and radio spectrum use
To provide a good quality of experience
keeping customers, preventing churning
capturing new customers by good performance
ensuring services perform as required
To bridge the gap between network performance
and quality of experience
ensure that good network performance means
good QoE for customers
3
What is service performance?
Traditional data KPI measurements
network KPIs: drop session rate, session setup
success rate, congested cell ratio, cell loading
E2E KPIs (robots): data rates, web page, latency
used to ensure good network performance
insufficient to reflect quality of experience
Services run on top of the networks
monitor the real service performance
what the customers are doing
web browsing, YouTube, video, messaging,gaming
where they are doing it
indoors, at home, in the office, on the tube
challenges to measure in these locations
while respecting privacy and confidentiality
E2E performance depends on the other end and
third part connectivity
often outside the operators network
expectation of network operator responsibility
4
Performance has to be measurable
Network counters, passive probes
large volume of data from the complete network
high level overview or focus on specific areas
modern probes enable us to drill down to:
specific service streams
individuals or identified user groups
device types
What changes for LTE?
basic principles remain the same
upgrade of tools, robots, probes
extended to cover eNodeB, EPC, IMS etc
How to assess network and service performance?
against targets or objectives
against customer expectations?
against local competitors?
against other operators in the same group?
Traditional data KPI measurements
Monitor KPIs or a
regular basis, look at
trends, and keep them
under control
Variable
performance
Coming under
control here
Further investment
may be needed to
reduce congested
cells
Define congested
cells to meet your
own requirements
Track performance
by region to identify
any poorly
performing areas
Ensure good network performance = good quality of experience
6
Traditional data rate measurements
Measure where the customer is located
business areas, transport hubs
hotels, residential, indoors, etc
major challenge
Ensure consistent measurement method
drive tests, robots, stationary or mobility
understand the effect of TCP slow start
Ensure measurement tools up to date
capable of maximum data rates
ensure all tools are LTE capable
Data rates compared by operator, country,
region, vendor
Average data rate at
85% of instantaneous
data rate
TIP: larger file sizes are generally used for measuring LTE data rates, but these can take
excessive time if the terminal connects to 3G or 2G network; to avoid this measure the
data rate over a fixed time, e.g. 10-20 seconds and stop the transfer after this time
Data rate evolution over two years
7
Crowd sourcing an alternative solution
What is crowd sourcing?
a fresh way to look at network performance
from the customers perspective
Collected from apps/agents on handsets
different locations, users, handsets
tens of thousands of measurements
Very consistent data month on month
but do customers only perform these tests
when they have a network problem ?
Currently limited to traditional KPIs
data rate and latency
More advanced applications available
passive (silent) or active (on-click) monitoring
web page and video performance
friendly users
benchmarking - alternative to drive tests
capability to test indoors
How can we assess service performance?
Assessment of performance
What can we measure?
do we assess against targets?
how do we establish targets?
against aspirations, customer
expectations, local competitors,
other operators?
should targets be different for LTE?
can we give performance a number?
e.g. 25% = half target speed
50% = on target
100% = twice as good as target
data rate and latency
video services: access success rate,
speed to start playout, image quality
(pauses, lost frames)
web services: download success
rate / time; DNS response
other services: instant messaging,
specific OTT services
over a mix of locations
Example of weighted service performance
Service
Major service
weighing
Web page download
performance
40%
Download data rate
20%
Upload data rate
10%
Video performance
DNS access time
Latency (Round Trip Time)
20%
5%
5%
Individual services
Reference web page (Kepler)
Local web pages
International web pages
Indoors
Large cities (drive test)
Small cities (drive test)
Interconnecting routes
Indoors
Large cities (drive test)
Small cities (drive test)
Interconnecting routes
25%
37.5%
37.5%
40%
30%
20%
10%
40%
30%
20%
10%
20%
5%
5%
Overall service
weighting
10%
15%
15%
8%
6%
4%
2%
8%
6%
4%
2%
20%
5%
5%
9
Service performance versus customer experience?
We must not forget to measure actual customer experience
regular surveys of randomly selected customers
performed by third party, unbiased, covering all operators
telling us what the customer feels about our network performance
Results of recent (4Q-2013)
customer experience surveys for
five European operators show a
high level of correlation against
weighted service performance
However there is often a lag in
customer perception, with
customers remembering bad
experiences
Can we use this to better effect to improve customer experience?
Can Big Data help us to ?
10
How can Big Data help us?
Big data - a consolidated tool for customer experience management
making better use of all network performance indicators and correlating these with
other information, e.g. from customer services, billing
to improve network performance
data capture
to better understand the customer experience
QoS
internet
opportunities:
to provide customers with new offers
LTE
processing
use
probes
cases
services
storage
KPIs
latency
Big Data
benefits:
new value propositions
tools
data rates
IMS
processes
database
consolidation
social networks
QoE
customer retention and base expansion
analysis
3G
customer service optimization
QoS/QoE management and network optimisation
Portal and presentation layer
Data processing and storage
Mediation layer
Probes
Probes
Mobile usage
CRM data
Service description
Network description
IS sources (customer data, service description, network description)
11
Big Data use cases
Already proven benefits for both network operations and marketing
multiple use cases; each country can focus on those important to the business
examples:
15 complaints in one area highlighted that 600 customers had problems,
corrective action was put in place to improve performance
devices with high signalling volume, again able to take corrective action
12
Big Data network optimization - traffic
Traffic by application
highest traffic volume is web,
followed by streaming, file
download, mail, P2P
Traffic by service
provider
highest single traffic provider is
YouTube, with Facebook,
Apple, Orange portal, Skype
also providing high levels of
traffic
13
Big Data network optimization traffic by app & user
Top 10 users per day
7% of traffic from 10 users
can also identify usage by
application, e.g. web,
download, streaming
used to understand cause of
high load and better manage
the traffic
allows marketing to target high
users with upgrades
14
Big Data traffic by terminal type
Traffic volume by terminal brand
Traffic figures show almost half the traffic is on Samsung
or Apple devices. Further drill down could be performed if
necessary to show the precise terminal types.
Traffic by terminal type (smartphone, USB,mobile,
tablet)
Figures show 78% of mobile data usage is now on
smartphone, with less than 10% on USB modems.
Note the low volume of mobile data traffic on tablets as
these probably use Wi-Fi by preference.
15
Big Data QoS/QoE management IP sessions
IP session QoS
time for session
establishment (between
first user request and
first downlink packet)
where the time for
session establishment is
long, there is a need to
identify any single cause
Video performance
(RTSP)
packet desynchronisations
lost packets
16
Big Data QoS/QoE management - throughput
Throughput
distribution
note that the low data
rate sessions can be
caused by a the number
of low volume transfers
Percentage of
retransmitted packets
monitor performance,
review trends, identify
source of retransmissions
17
Summary
Traditional measurements of QoS obtained
from network performance
still applicable today, but not the whole story
LTE
increased data and service usage
Services not networks
emphasis on service measurement
greater need to focus on customer
New tools available to the operator
better understanding of the customer
experience of our data services
Crowd sourcing
measuring the service performance where the
customer is located
Big data
making better use of the data available
providing a high level overview or drilling down
to target specific issues
18
THANK YOU
Martin Harris
Orange Corporate Services Ltd
martin.harris@orange.com
19
