Marrying Web Analytics and User Experience

1.
Marrying Web Analytics andUser Experience Louis Rosenfeld • 5 August 2009 Delve NYC • Brooklyn 1

2.
Web Analytics? User Experience? 2

3.
Code “DELVE” for25% off at rosenfeldmedia.com 3

4.
My recent struggle 4

5.
CONTRASTING WEB ANALYTICSAND USER EXPERIENCE 5

6.
Who we are How we do our work What data we use How we use that data CONTRASTING WEB ANALYTICS AND USER EXPERIENCE 5

7.
WHO WE ARE ARETHE STEREOTYPES TRUE? 6

8.
VIVE LA DIFFÉRENCE!(FROM MARKO HURST) 7

9.
!"#$%&"'()*+),%(-).(%("-&/)0(1/*$%) Behavioral / Eyetracking Data Mining/Analysis A/B (Live) Testing Usability Benchmarking (in lab) / Data Source Usability Lab Studies Online User Experience Assessments (“Vividence-like” studies) Ethnographic Field Studies mix Diary/Camera Study Message Board Mining Participatory Design Customer feedback via email Focus Groups Desirability studies Intercept Surveys Attitudinal Phone Interviews Cardsorting Email Surveys mix Qualitative (direct) Approach Quantitative (indirect) Key for Context of Product Use during data collection Natural use of product De-contextualized / not using product © 2008 Christian Rohrer Scripted (often lab-based) use of product Combination / hybrid 20 HOW USER EXPERIENCE PEOPLE SEE THEIR WORK (FROM CHRISTIAN ROHRER) 8

10.
!"#$%&"'()*+),%(-).(%("-&/)0(1/*$%) Behavioral / Eyetracking Data Mining/Analysis A/B (Live) Testing Usability Benchmarking (in lab) / Data Source Usability Lab Studies Online User Experience Assessments (“Vividence-like” studies) Ethnographic Field Studies mix Diary/Camera Study Message Board Mining Participatory Design Customer feedback via email Focus Groups Desirability studies Intercept Surveys Attitudinal Phone Interviews Cardsorting Email Surveys mix Qualitative (direct) Approach Quantitative (indirect) Key for Context of Product Use during data collection Natural use of product De-contextualized / not using product © 2008 Christian Rohrer Scripted (often lab-based) use of product Combination / hybrid 20 HOW USER EXPERIENCE PEOPLE SEE THEIR WORK (FROM CHRISTIAN ROHRER) 8

11.
HOW WEB ANALYTICSPEOPLE SEE THEIR WORK (FROM AVINASH KAUSHIK) 9

12.
HOW WEB ANALYTICSPEOPLE SEE THEIR WORK (FROM AVINASH KAUSHIK) 9

13.
The data that drivesour decisions 10

14.
The data that drivesour decisions Web Analytics User Experience behavioral attitudinal quantitative qualitative high ﬁdelity artiﬁcial high volume high quality This data is about WHAT This data is about WHY 10

15.

16.

17.

18.

19.

20.
Not much useto know what is happening if you don’t know why 11

21.
Not much useto know what is happening if you don’t know why Hard to know why things are happening if you don’t know what is happening 11

22.
The ways weanalyze our data 12

23.

24.

25.
XXX.XXX.X.104 - -[10/Jul/2006:10:25:46 -0800] "GET /search? access=p&entqr=0&output=xml_no_dtd&sort=date%3AD%3AL %3Ad1&ud=1&site=AllSites&ie=UTF-8&client=www&oe=UTF-8&proxysty lesheet=www&q=lincense+plate&ip=XXX.XXX.X.104 HTTP/1.1" 200 971 0 0.02 XXX.XXX.X.104 - - [10/Jul/2006:10:25:48 -0800] "GET /search? access=p&entqr=0&output=xml_no_dtd&sort=date%3AD%3AL %3Ad1&ie=UTF-8&client=www&q=license+plate &ud=1&site=AllSites&spell=1&oe=UTF-8&proxystylesheet=www&ip=XX X.XXX.X.104 HTTP/1.1" 200 8283 146 0.16 XXX.XXX.XX.130 - - [10/Jul/2006:10:24:38 -0800] "GET /search? access=p&entqr=0&output=xml_no_dtd&sort=date%3AD%3AL %3Ad1&ud=1&site=AllSites&ie=UTF-8&client=www&oe=UTF-8&proxysty lesheet=www&q=regional+transportation+governance +commission&ip=XXX.XXX.X.130 HTTP/1.1" 200 9718 62 0.17 13

26.

27.
XXX.XXX.X.104 - -[10/Jul/2006:10:25:46 -0800] "GET /search? access=p&entqr=0&output=xml_no_dtd&sort=date%3AD%3AL %3Ad1&ud=1&site=AllSites&ie=UTF-8&client=www&oe=UTF-8&proxysty lesheet=www&q=lincense+plate&ip=XXX.XXX.X.104 HTTP/1.1" 200 971 0 0.02 XXX.XXX.X.104 - - [10/Jul/2006:10:25:48 -0800] "GET /search? access=p&entqr=0&output=xml_no_dtd&sort=date%3AD%3AL %3Ad1&ie=UTF-8&client=www&q=license+plate &ud=1&site=AllSites&spell=1&oe=UTF-8&proxystylesheet=www&ip=XX X.XXX.X.104 HTTP/1.1" 200 8283 146 0.16 XXX.XXX.XX.130 - - [10/Jul/2006:10:24:38 -0800] "GET /search? access=p&entqr=0&output=xml_no_dtd&sort=date%3AD%3AL %3Ad1&ud=1&site=AllSites&ie=UTF-8&client=www&oe=UTF-8&proxysty lesheet=www&q=regional+transportation+governance +commission&ip=XXX.XXX.X.130 HTTP/1.1" 200 9718 62 0.17 Q “What were the most common searches?” 14

28.
XXX.XXX.X.104 - -[10/Jul/2006:10:25:46 -0800] "GET /search? access=p&entqr=0&output=xml_no_dtd&sort=date%3AD%3AL %3Ad1&ud=1&site=AllSites&ie=UTF-8&client=www&oe=UTF-8&proxysty lesheet=www&q=lincense+plate&ip=XXX.XXX.X.104 HTTP/1.1" 200 971 0 0.02 XXX.XXX.X.104 - - [10/Jul/2006:10:25:48 -0800] "GET /search? access=p&entqr=0&output=xml_no_dtd&sort=date%3AD%3AL %3Ad1&ie=UTF-8&client=www&q=license+plate &ud=1&site=AllSites&spell=1&oe=UTF-8&proxystylesheet=www&ip=XX X.XXX.X.104 HTTP/1.1" 200 8283 146 0.16 XXX.XXX.XX.130 - - [10/Jul/2006:10:24:38 -0800] "GET /search? access=p&entqr=0&output=xml_no_dtd&sort=date%3AD%3AL %3Ad1&ud=1&site=AllSites&ie=UTF-8&client=www&oe=UTF-8&proxysty lesheet=www&q=regional+transportation+governance +commission&ip=XXX.XXX.X.130 HTTP/1.1" 200 9718 62 0.17 Q “What were the most common searches?” 14

29.
Analyzing data the UXway: play with the data, look for patterns, trends, and outliers

30.
Analyzing data the UXway: play with the data, look for patterns, trends, and outliers So what’s being measured?

31.

32.
XXX.XXX.X.104 - -[10/Jul/2006:10:25:46 -0800] "GET /search? access=p&entqr=0&output=xml_no_dtd&sort=date%3AD%3AL %3Ad1&ud=1&site=AllSites&ie=UTF-8&client=www&oe=UTF-8&proxysty lesheet=www&q=lincense+plate&ip=XXX.XXX.X.104 HTTP/1.1" 200 971 0 0.02 XXX.XXX.X.104 - - [10/Jul/2006:10:25:48 -0800] "GET /search? access=p&entqr=0&output=xml_no_dtd&sort=date%3AD%3AL %3Ad1&ie=UTF-8&client=www&q=license+plate &ud=1&site=AllSites&spell=1&oe=UTF-8&proxystylesheet=www&ip=XX X.XXX.X.104 HTTP/1.1" 200 8283 146 0.16 XXX.XXX.XX.130 - - [10/Jul/2006:10:24:38 -0800] "GET /search? access=p&entqr=0&output=xml_no_dtd&sort=date%3AD%3AL %3Ad1&ud=1&site=AllSites&ie=UTF-8&client=www&oe=UTF-8&proxysty lesheet=www&q=regional+transportation+governance +commission&ip=XXX.XXX.X.130 HTTP/1.1" 200 9718 62 0.17 Q “Are we converting license plate renewals?” 16

33.
Before data analysis: whyare we here? ★ Commerce ★ Lead Generation ★ Content/Media ★ Support/Self-Service 17

34.
Before data analysis: whyare we here? ★ Commerce ★ Lead Generation ★ Content/Media ★ Support/Self-Service Data supports metrics 17

35.
Analyzing data the WAway: start with metrics, benchmark and measure performance

36.
Analyzing data the WAway: start with metrics, benchmark and measure performance But you can’t measure what you don’t know

37.
WA: Top-down analysis UX:Bottom-up analysis 19

38.
what WA: Top-down analysis UX:Bottom-up analysis 19

39.
what WA: Top-down analysis UX:Bottom-up analysis why 19

40.
INTEGRATING WEB ANALYTICSAND USER EXPERIENCE 20

41.
Integrating methodologies: What, thenwhy 21

42.
Common queries can drivetask analysis 22

43.
Common queries can drivetask analysis “Can you ﬁnd a map of the campus?” “What study abroad options are available to students?” “When is the last home football game of the season?” 22

44.
Query data can augment personas 23

45.
Query data can augment personas “What Steven Searches” added to existing persona (from Adaptive Path) 23

46.
Looking ahead ★ Howdo we improve other qualitative methods with data? ★ How do qualitative data impact quantitative analyses? 24

47.
Methodology takeaways: ★ Qualitativeresearch is expensive ★ Start with quantitative research to identify where/when to use qualitative methods 25

48.
Changing how weanalyze: Moving away from the middle 26

49.
27

50.
28

51.
What’s in the middle? 28

52.
What’s in the middle? Youranalytics app’s canned reports 28

53.
Netﬂix moved away fromthe middle 29

54.

55.

56.

57.

58.
Analysis takeaways ★ Cannedreports are only a starting point ★ Move up, move down ★ Be prepared to “roll your own” ★ Demand better ad hoc reporting from analytics apps 30

59.
Changing our thinking: Gettingcomfortable with the other 31

60.
UX people needto get comfortable with measuring the unmeasurable 32

61.
Can you measure yourcontent’s quality? Systems can help us objectify the subjective 33

62.
Subjective evaluations... Can you measure your content’s quality? Systems can help us objectify the subjective 33

63.
Subjective evaluations... ...lead to Can you measure objective decisions your content’s quality? Systems can help us objectify the subjective 33

64.
UX people needto get comfortable with numbers (but just a little) 34

65.
This is notstatistics 35

66.
This is notstatistics This is not difﬁcult 35

67.
This is notstatistics This is not difﬁcult This is very useful 35

68.
This is notstatistics This is not difﬁcult This is very useful (and this is in MS Excel) 35

69.
WA people needto get comfortable with stories 36

72.
WA people needto understand the value of intuition and mistakes 38

73.
C# '&DE#F <=>>?@A=B !""#$%&'()*+*(,%+-'.()/-0%'(12*3(4+'(5"6()%'-#7%/ ! !"#$%&'#()*+,-.#/0/1#23#*'#456#3,5#17#1778.#17.19:#;6&%-# !"#$%#&%'(&)%*#+,")#$%%-.#/(01#2($3,4.#5((1+%6"#)('#*%"714%&.#8,"#9(4:7&3%*#)8,)#8%6"#+%,;741# <$%#8%,&#)(#=$7))%&>?#2($3,46"#&%,"(4"#:(&#@07))741#,&%#:,"974,)741#A#,4*#)8%B#"8($#$8B# 5((1+%6"#+("741#7)"#9((+? 2($3,4#C(74%*#5((1+%#)8&%%#B%,&"#,1(#A#)((#+,)%.#8%#4($#",B"?#=8%#9(3',4B6"#%4174%%&"D:7&")# 90+)0&%#$,"#:7&3+B#74#'+,9%.#3%,4741#%;%&B#*%97"7(4#8,*#)(#E%#'&(;%4#)8&(018#%F8,0")7;%#)%")741.# &,)8%&#)8,4#,#&%+7,49%#(4#,#9+%,&#;7"7(4#(:#5((1+%6"#*%"714?#!4*#74#,#E,9-8,4*%*#"+,3#,)#5((1+%#GH# I,&7"",#I,B%&.#)8%#8%,*#(:#J0"%&#%F'%&7%49%.J#8%#4()%"#)8,)#)('#3,4,1%3%4)#74#98,&1%#(:#*%"714# 8,*#4()#E,9-1&(04*#74#)8%#:7%+*K L8%4#,#9(3',4B#7"#:7++%*#$7)8#%4174%%&".#7)#)0&4"#)(#%4174%%&741#)(#"(+;%#'&(E+%3"?#M%*09%#%,98#*%97"7(4#)(#,#"73'+%#+(179# '&(E+%3?#M%3(;%#,++#"0EC%9)7;7)B#,4*#C0")#+((-#,)#)8%#*,),?#/,),#74#B(0&#:,;(&N#O-.#+,0498#7)?#/,),#"8($"#4%1,)7;%#%::%9)"N#2,9-#)(# )8%#*&,$741#E(,&*?#!4*#)8,)#*,),#%;%4)0,++B#E%9(3%"#,#9&0)98#:(&#%;%&B#*%97"7(4.#',&,+BP741#)8%#9(3',4B#,4*#'&%;%4)741#7)# :&(3#3,-741#,4B#*,&741#*%"714#*%97"7(4"? # # # Q%".#7)6"#)&0%#)8,)#,#)%,3#,)#5((1+%#9(0+*46)#*%97*%#E%)$%%4#)$(#E+0%".#"(#)8%B6&%#)%")741#RS#"8,*%"#E%)$%%4#%,98#E+0%#)(#"%%#$8798# (4%#'%&:(&3"#E%))%&?#T#8,*#,#&%9%4)#*%E,)%#(;%&#$8%)8%&#,#E(&*%&#"8(0+*#E%#U.#R#(&#V#'7F%+"#$7*%.#,4*#$,"#,"-%*#)(#'&(;%#3B# 9,"%?#T#9,46)#('%&,)%#74#,4#%4;7&(43%4)#+7-%#)8,)?#T6;%#1&($4#)7&%*#(:#*%E,)741#"098#3747"90+%#*%"714#*%97"7(4"?#=8%&%#,&%#3(&%# %F97)741#*%"714#'&(E+%3"#74#)87"#$(&+*#)(#),9-+%? WF97)741#*%"714#'&(E+%3".#+7-%#)8("%#,)#=$7))%&N#!#"(0&9%#)%++"#0"#)8,)6"#$8%&%#8%6"#1(741.#E0)#2($3,4#8,"46)#9(4:7&3%*#)8,)# B%)?#<X%#'&(37"%"#)(#*7"9+("%#87"#4%$#%3'+(B%&#74#,#:(++($0'#E+(1#'(")?>

74.

75.

76.
Tom Chi: “Think ofyour designer as a guide in this multi-variate optimization process. A good designer has been all over parts of the territory a dozen times on various projects and has studied the design patterns and techniques that help in different problems/situations. Because of this, he or she has intuition on how to approach a problem, just as an experienced software architect has intuition on software design approaches that provide different beneﬁts/drawbacks.” 40

77.
UX and WApeople need to talk together about project goals 41

78.
42

79.
Vanguard and the quantiﬁcationof search Target Oct 3 Oct 10 Oct 16 Mean distance from 1st 3 13 7 5 Median distance from 1st 2 7 3 1 Count: Below 1st 47% 84% 62% 58% Count: Below 5th 12% 58% 38% 14% Count: Below 10th 7% 38% 10% 7% Precision – Strict 42% 15% 36% 39% Precision – Loose 71% 38% 53% 65% Precision – Permissive 96% 55% 72% 92% Note: quantiﬁcation, not monetization

80.
Changing thinking takeaways ★ Mostthings can be quantiﬁed ★ Stories and emotions can make stronger cases than data, and for data ★ We need more talking, and more listening 44

81.
Challenges: how dowe... ★ Bridge cultural gaps? ★ Get different groups to speak the same language? ★ Design and manage integrated teams? ★ Find better, more open tools? ★ Develop a uniﬁed methodology? 45

82.
Do we havea choice? An individual often uses only half their brain Effective teams and organizations use both halves 46

84.
Some day mybook will come... Search Analytics for Your Site: Conversations with Your Customers Louis Rosenfeld & Marko Hurst Rosenfeld Media, 2009. rosenfeldmedia.com/books/searchanalytics 48

85.
Until then... Louis Rosenfeld 457Third Street, #4R Brooklyn, NY 11215 USA lou@louisrosenfeld.com www.louisrosenfeld.com www.rosenfeldmedia.com Twitter: @louisrosenfeld @rosenfeldmedia This presentation @ http://www.slideshare.net/lrosenfeld

Marrying Web Analytics and User Experience

More Related Content

What's hot

Viewers also liked

Similar to Marrying Web Analytics and User Experience

More from Louis Rosenfeld

Recently uploaded

Marrying Web Analytics and User Experience

Editor's Notes