KEMBAR78
Advanced Card Sorting and Mixed Method IA Validation | PDF
UXPA International , profile picture
Uploaded byUXPA International
PDF, PPTX1,523 views

Advanced Card Sorting and Mixed Method IA Validation

The document summarizes different methods for validating information architecture (IA), including card sorting techniques. It then presents a case study where multiple validation methods were used to evaluate the IA of a healthcare website. An open card sort revealed how users naturally grouped content and identified difficult items. A closed card sort showed better alignment with the proposed IA structure. The combination of methods provided valuable insights to improve the site's organization and findability.

Download as PDF, PPTX
www.nicecompany.comCOMPANY NAME Caron Garstka UXPA 2018 Card Sorting and Mixed Method IA Validation
www.nicecompany.comCOMPANY NAME A little about me Information Architecture (IA): A quick review IA validation methods Mixed method IA validation Case Study 01. 02. 03. 04. 05. C ON TEN TS Conclusion
www.nicecompany.comCOMPANY NAME A little about me 01
www.nicecompany.comCOMPANY NAME A l i ttl e a bo u t m e … Who I am… • Caron Garstka • Senior UX Researcher at Express Scripts • Masters of Science in Information from University of Texas at Austin What I do… • Conduct research with patients who have chronic, complex, or life-threatening conditions • Help shape their digital experiences using their feedback My goal is to make patient lives easier
www.nicecompany.comCOMPANY NAME Information Architecture (IA): A quick review 02
• It is the creation of a structure for a website, intranet, application, or other project • Allows users to understand where they are, and where the desired information is in relation to their position • Results in the creation of site maps, hierarchies, categorizations, navigation, and metadata W h a t i s In fo r m a ti o n Ar c h i te c tu r e ( IA) ?
Information architecture is about helping people understand their surroundings and find what they’re looking for, in the real world as well as online. – Information Architecture Institute W h a t i s In fo r m a ti o n Ar c h i te c tu r e ( IA) ?
A Ph a se d Appr o a c h to IA D e ve l o pm e n t There are multiple steps to creating and maintaining an IA Research Strategy Design Implementation Administration Card Sorting lives here!
A Ba l a n c e d IA Re se a r c h Appr o a c h Documents Data types Content objects Metadata Volume Existing structure Business goals Funding Politics Culture Technology Resources Constraints Audience Tasks Needs Information seeking behavior Experience Vocabularies C o n t e n t C o n t e x t U s e r s
How is it categorized? How is it displayed? Where should it be stored? What information is relevant at a given time? Is the information necessary? What are they looking for? What’s important to them? What are they familiar with? C o n t e n t C o n t e x t U s e r s A Ba l a n c e d IA Re se a r c h Appr o a c h
A Ba l a n c e d IA Re se a r c h Appr o a c h Heuristic evaluation Metadata and content analysis Content mapping Benchmarking Background research Presentations and meetings Stakeholder interviews Technology assessments Search log and clickstream analysis Use cases and personas Contextual inquiry User interviews and user testing C o n t e n t C o n t e x t U s e r s
IA Context UsersContent A Ba l a n c e d IA Re se a r c h Appr o a c h
www.nicecompany.comCOMPANY NAME IA Validation Methods 03
Content audits Establishing starting point Surveys Gathering opinions Field studies, diaries, contextual inquiry, interviews, focus groups Learning about users Web traffic statistics, clickstreams Online behavior Be fo r e do i n g IA va l i da ti o n … Do the following methods before starting validation research.
IA Va l i da ti o n R e s e a r c h M e th o ds Tree Testing Looks at IA as working hierarchy Closed Card Sorting Looks at viability of defined categories Click Testing Looks at first-level IA in context of UI Hybrid Card Sorting Combination of open and closed Skews open or closed Open Card Sorting Looks at user-generated groupings and names Delphi Card Sorting Looks at user-generated groupings and names
IA Va l i da ti o n R e s e a r c h M e th o ds Tree Testing Looks at IA as working hierarchy Closed Card Sorting Looks at viability of defined categories Click Testing Looks at first-level IA in context of UI Hybrid Card Sorting Combination of open and closed Skews open or closed Open Card Sorting Looks at user-generated groupings and names Delphi Card Sorting Looks at user-generated groupings and names • Users sort items into groups and give groups a name; typically done prior to an architecture being put in place • Allows the researcher to document a user’s mental model and inform the IA
IA Va l i da ti o n R e s e a r c h M e th o ds Tree Testing Looks at IA as working hierarchy Closed Card Sorting Looks at viability of defined categories Click Testing Looks at first-level IA in context of UI Hybrid Card Sorting Combination of open and closed Skews open or closed Open Card Sorting Looks at user-generated groupings and names Delphi Card Sorting Looks at user-generated groupings and names • Users sort items into defined categories • Allows the researcher to document a user’s mental model and understand how well the proposed IA matches their model
IA Va l i da ti o n R e s e a r c h M e th o ds Tree Testing Looks at IA as working hierarchy Closed Card Sorting Looks at viability of defined categories Click Testing Looks at first-level IA in context of UI Hybrid Card Sorting Combination of open and closed Skews open or closed Open Card Sorting Looks at user-generated groupings and names Delphi Card Sorting Looks at user-generated groupings and names • Users sort items into researcher-defined categories but have the option to create their own • Allows the researcher to document a user’s mental model and allows users to participate in building the IA
IA Va l i da ti o n R e s e a r c h M e th o ds Tree Testing Looks at IA as working hierarchy Closed Card Sorting Looks at viability of defined categories Click Testing Looks at first-level IA in context of UI Hybrid Card Sorting Combination of open and closed Skews open or closed Open Card Sorting Looks at user-generated groupings and names Delphi Card Sorting Looks at user-generated groupings and names • Users work one after another to complete card sort • Users modify as needed and pass modifications to next user • Allows the researcher to rapidly arrive at an organizational model
IA Va l i da ti o n R e s e a r c h M e th o ds Tree Testing Looks at IA as working hierarchy Closed Card Sorting Looks at viability of defined categories Click Testing Looks at first-level IA in context of UI Hybrid Card Sorting Combination of open and closed Skews open or closed Open Card Sorting Looks at user-generated groupings and names Delphi Card Sorting Looks at user-generated groupings and names • Users navigate a hierarchy using only top-level link names. • Allows the researcher to see how findable items are within the hierarchy
I A V a l i d a t i o n R e s e a rc h Me t h o d s Tree Testing Looks at IA as working hierarchy Closed Card Sorting Looks at viability of defined categories Click Testing Looks at first-level IA in context of UI Hybrid Card Sorting Combination of open and closed Skews open or closed Open Card Sorting Looks at user-generated groupings and names Delphi Card Sorting Looks at user-generated groupings and names • Users locate items within a static design • Allows the researcher to see if the top level IA is easily findable within the context of the design
C ompari son of IA Research Methods Method Research Type Activity Learn About Project Stage Open card sorting Generative Organizing Mental models Organization Labeling Relationships between items Early user research Closed card sorting Evaluative Organizing Mental models Organization Labeling Information architecture design Hybrid card sorting Generative Evaluative Organizing Mental models Organization Labeling Relationships between items Early user research Delphi card sorting Generative Organizing Mental models Organization Labeling Relationships between items Early user research Reverse card sorting (Tree testing) Evaluative Finding Organization Labeling Findability Information architecture design Click testing (or usability testing with low- fidelity prototypes) Evaluative Finding Organization Labeling Findability Navigation design Interface design Interface design
Pr a c ti c a l T r i a n gu l a ti o n [Triangulation is an] attempt to map out, or explain more fully, the richness and complexity of human behavior by studying it from more than one standpoint. - Cohen and Manion (1986)
www.nicecompany.comCOMPANY NAME Case Study: Accredo.com 04
www.nicecompany.comCOMPANY NAME Accredo.com Accredo.com 1. Stakeholders wanted to validate a proposed IA on the Accredo.com website 2. Conduct a closed card sort to catch any glaring issues The Perceived Need
www.nicecompany.comCOMPANY NAME The Real Need Expanded scope… 1. How well the proposed IA matches the user’s mental model 2. How findable items are within the proposed IA 3. How UI helps or obscures IA 4. Areas for improvement 5. Validate the proposed patient IA 6. Learn how potential users group content 7. Evaluate the findability of topics within the website structure 8. Evaluate the homepage design and whether potential users can fine the expected entry point for specific information or functionality within the design And there wasn’t much time… Accredo.com Accredo.com
www.nicecompany.comCOMPANY NAME • Site caters to multiple audiences • Users are distributed • Site IA is different pre and post login Focus on one side of the experience first. Conduct study remotely, asynchronously, and without moderation. Allowed for fast data capture without having to moderate each session. Accredo.com Accredo.com Other Challenges
Appr o a c h Tree Testing Closed Card Sorting Click Testing Hybrid Card SortingOpen Card Sorting Delphi Card Sorting One method alone would not get at what the team needed.
Appr o a c h Used UserZoom to conduct the study. Tree Test N= 43 Median Completion Time: 25 min Click Test N= 44 Median Completion Time: 20 min Open/Closed Card Sort N= 87
Appr o a c h • Get at users’ mental model • Note alternative categories, item placement, and category names • Pin point hard to place items Step 01 Open Card Sort • See how users group items within given categories • See how well groupings match the proposed IA • Note alternative categories, item placement, and category names • Pin point hard to place items Step 02 Closed Card Sort • Look at where users drill into the tree • Note findability of key items Tree Test Step 03A • Look at where users drill into the design • Note expectations of finding items in design Click Test Step 03B
A p p ro a c h Tree Test N= 43 Median Completion Time: 25 min Click Test N= 44 Median Completion Time: 20 min Open/Closed Card Sort N= 87
C ard Sort Setup 1. Selected content that could be grouped 2. Selected content that was at the same level 3. Selected content that was representative 4. Selected content that did not lead the activity 5. Selected content that participants would understand Maintained consistency across the open and closed card sorts.
Ope n C a r d So r t • Presented participants with a list of Accredo proposed IA items and asked them to sort those items into groups • Participants sorted items into groups that made sense to them • Participants created their own names for the categories the created • Some labels were modified for the card sort to add clarity UserZoom interface for open card sort
Ope n C a r d So r t F i n di n gs • Dendrogram: A tree diagram of hierarchical clustering to learn how users group items. • Dendorgram scale: 0 to 1; the lowest level of clusters with the highest agreement. • Euclidian distance the average distance between items. Interpreting dendrogram clusters is an art. Data Output: Dendrogram Overview
Ope n C a r d So r t F i n di n gs Interpretations Cluster 2: Had a runt: “Communication Information for digital support team” Cluster 1: Fairly cohesive, aligned with the hypothesized IA, and participants came up with similar names Cluster 3: High agreement for items related to prescriptions, not matching hypothesized IA, but logically made sense
Ope n C a r d So r t F i n di n gs Cluster 4: Cohesive group related to a Welcome or introductory section Cluster 5: Mixed assortment of forms and administration items. These items had strong agreement but did not match IA. Interpretations
Ope n C a r d So r t F i n di n gs Finding: Participants grouped items according to Disease FAQ, Prescriptions and Refills, and Account functions Participants were less certain on how to handle “Communication information for digital support team”
Ope n C a r d So r t F i n di n gs Difficult Items to Sort Items Participants Wanted to Sort into Multiple Categories Post hoc questions about the sorting experience provides additional insights
Ope n C a r d So r t F i n di n gs Difficult Items to Sort Items Participants Wanted to Sort into Multiple Categories Post hoc questions about the sorting experience provides additional insights Finding: “Communication information for digital support team” and “Register” were both highly ranked for difficulty to sort and participants thought the items could belong in multiple categories. Perhaps they would be easier to sort if given a structure. Sounds like a job for the Closed Card Sort!
C l osed C ard Sort • Participants were first presented with a list of Accredo IA items and were asked to sort those items into pre-named groups UserZoom interface for closed card sort
C l osed C ard Sort Wanted to see how well participants’ sorts aligned with proposed closed groups. Home Getting Started Conditions Find Medication Contact Us Manage your account Create an Accredo account information Symptoms Search to view Accredo drugs Communication information for digital support team Download Accredo mobile app Authorization to use and disclose health information PDF Disorder overview Phone numbers for disorder- specific assistance Order status Authorization to use and disclose health information PDF Find disease-specific support Billing call information Register Patient consent agreement and acknowledgement document Specific disease/disorder care team Phone number to refill medicine How to begin with Accredo Welcome to Accredo information Financial resources for a disorder Fax number for new prescription Refill an Rx Filling Rx for the first time Medicines for a particular disease Secure email form New patient refill process Payment mail-in address Patient forms What is a specialty pharmacy? video Refill procedure for existing patients
C l osed C ard Sort Fi ndi ngs Methodology • For this analysis I chose to use the average linkage dendrogram for analysis. • I set the agreement minimum to 50% - this is the minimum percentage of agreement the selected items that participants included into the same category.
C l osed C ard Sort Fi ndi ngs Finding: Participants seemed to have an easier time grouping “Communication information for digital support team” – this was the runt in the open sort – grouping it in with other “contact” related items. Interestingly, “Medicines for a particular disease” and “Search to view Accredo drugs” hung together, where as in the open card sort they did not.
C l osed C ard Sort Fi ndi ngs Items Home Getting Started Conditions Find a Medication Contact Us Highest % Register #% #% #% #% #% #% Refill an Rx #% #% #% #% #% #% Order status #% #% #% #% #% #% How to begin with Accredo #% #% #% #% #% #% Welcome to Accredo information #% #% #% #% #% #% New patient refill process #% #% #% #% #% #% Refill procedure for existing patients #% #% #% #% #% #% Manage your account #% #% #% #% #% #% Disorder overview #% #% #% #% #% #% Symptoms #% #% #% #% #% #% Medicines for a particular disease #% #% #% #% #% #% Financial resources for a disorder #% #% #% #% #% #% Find disease-specific support #% #% #% #% #% #% Specific disease/disorder care team #% #% #% #% #% #% Search to view Accredo drugs #% #% #% #% #% #% What is a specialty pharmacy? video #% #% #% #% #% #% Authorization to use and disclose health information PDF #% #% #% #% #% #% Patient consent agreement and acknowledgement document #% #% #% #% #% #% Phone number to refill medicine #% #% #% #% #% #% Fax number for new prescription #% #% #% #% #% #% Patient forms #% #% #% #% #% #% Filling Rx for the first time #% #% #% #% #% #% Create an Accredo account information #% #% #% #% #% #% Phone numbers for disorder-specific assistance #% #% #% #% #% #% Secure email form #% #% #% #% #% #% Payment mail-in address #% #% #% #% #% #% Billing call information #% #% #% #% #% #% Communication information for digital support team #% #% #% #% #% #% Download Accredo mobile app #% #% #% #% #% #%
C l osed C ard Sort Fi ndi ngs Items Home Getting Started Conditions Find a Medication Contact Us Highest % Register #% #% #% #% #% #% Refill an Rx #% #% #% #% #% #% Order status #% #% #% #% #% #% How to begin with Accredo #% #% #% #% #% #% Welcome to Accredo information #% #% #% #% #% #% New patient refill process #% #% #% #% #% #% Refill procedure for existing patients #% #% #% #% #% #% Manage your account #% #% #% #% #% #% Disorder overview #% #% #% #% #% #% Symptoms #% #% #% #% #% #% Medicines for a particular disease #% #% #% #% #% #% Financial resources for a disorder #% #% #% #% #% #% Find disease-specific support #% #% #% #% #% #% Specific disease/disorder care team #% #% #% #% #% #% Search to view Accredo drugs #% #% #% #% #% #% What is a specialty pharmacy? video #% #% #% #% #% #% Authorization to use and disclose health information PDF #% #% #% #% #% #% Patient consent agreement and acknowledgement document #% #% #% #% #% #% Phone number to refill medicine #% #% #% #% #% #% Fax number for new prescription #% #% #% #% #% #% Patient forms #% #% #% #% #% #% Filling Rx for the first time #% #% #% #% #% #% Create an Accredo account information #% #% #% #% #% #% Phone numbers for disorder-specific assistance #% #% #% #% #% #% Secure email form #% #% #% #% #% #% Payment mail-in address #% #% #% #% #% #% Billing call information #% #% #% #% #% #% Communication information for digital support team #% #% #% #% #% #% Download Accredo mobile app #% #% #% #% #% #% Finding: “Communication information for digital support team” and “Register” were both highly ranked for difficulty sorting and belonging in multiple categories in the open card sort. However, once given a structure, participants had higher agreement for placement. “Patient forms” has a split grouping for where participants think it belongs.
C l osed C ard Sort Fi ndi ngs Difficult Items to Sort Items Participants Wanted to Sort into Multiple Categories Post hoc questions about the sorting experience provides additional insights
C l osed C ard Sort Fi ndi ngs Difficult to Sort Patient consent agreement and acknowledgement document What is a specialty pharmacy? video Authorization to use and disclose health information PDF Patient forms Refill an Rx Communication information for digital support team Download Accredo mobile app Billing call information Fax number for new prescription New patient refill process Payment mail-in address Search to view Accredo drugs Create an Accredo account information Phone number to refill medicine Refill procedure for existing patients Financial resources for a disorder Find disease-specific support How to begin with Accredo Order status Disorder overview Manage your account Medicines for a particular disease Register Secure email form Welcome to Accredo information Filling Rx for the first time Phone numbers for disorder-specific assistance Specific disease/disorder care team Top “Difficult to Sort” Items Difficult to Sort Finding: “Communication information for digital support team” and “Register” were both highly ranked for participants thinking they were difficult to sort in the open card sort. Both dropped overall in the closed card sort. But now Refilling Rx is one of the top 5 most difficult to sort.
Items Participants felt belonged in Multiple Categories Communication information for digital support team Order status Patient consent agreement and acknowledgement document Create an Accredo account information Phone number to refill medicine Authorization to use and disclose health information PDF Billing call information Fax number for new prescription New patient refill process Patient forms Find disease-specific support Manage your account Refill procedure for existing patients What is a specialty pharmacy? video Financial resources for a disorder Payment mail-in address Refill an Rx Register Search to view Accredo drugs Specific disease/disorder care team Welcome to Accredo information Filling Rx for the first time Medicines for a particular disease Phone numbers for disorder-specific assistance Secure email form Disorder overview Download Accredo mobile app How to begin with Accredo Symptoms Top “Multiple Categories” Items Multiple Categories Finding: Even though Refill Rx was top five most difficult to sort, it was not as highly ranked for belonging in multiple categories. Order status has also jumped up in the list. This would be a prime candidate to look at in the tree test as well to see where people start looking for it. C l osed C ard Sort Fi ndi ngs
C l osed C ard Sort Fi ndi ngs Home Getting Started Conditions Find Medication Contact Us Hypothesis Correct Hypothesis Correct Hypothesis Correct Hypothesis Correct Hypothesis Correct Manage your account Green Create an Accredo account information Green Symptoms Green Search to view Accredo drugs Green Communication information for digital support team Green Download Accredo mobile app Yellow Authorization to use and disclose health information PDF Yellow Disorder overview Green Phone numbers for disorder-specific assistance Green Order status Red Authorization to use and disclose health information PDF Yellow Find disease-specific support Green Billing call information Green Register Red Patient consent agreement and acknowledgement document Yellow Specific disease/disorder care team Yellow Phone number to refill medicine Yellow How to begin with Accredo Red Welcome to Accredo information Yellow Financial resources for a disorder Red Fax number for new prescription Yellow Refill an Rx Red Filling Rx for the first time Yellow Medicines for a particular disease Red Secure email form Yellow New patient refill process Red Payment mail-in address Yellow Patient forms Red What is a specialty pharmacy? video Red Refill procedure for existing patients Red
C l osed C ard Sort Fi ndi ngs Home Getting Started Conditions Find Medication Contact Us Accredo Hypothesis Correct Accredo Hypothesis Correct Accredo Hypothesis Correct Accredo Hypothesis Correct Accredo Hypothesis Correct Manage your account Green Create an Accredo account information Green Symptoms Green Search to view Accredo drugs Green Communication information for digital support team Green Download Accredo mobile app Yellow Authorization to use and disclose health information PDF Yellow Disorder overview Green Phone numbers for disorder-specific assistance Green Order status Red Authorization to use and disclose health information PDF Yellow Find disease-specific support Green Billing call information Green Register Red Patient consent agreement and acknowledgement document Yellow Specific disease/disorder care team Yellow Phone number to refill medicine Yellow How to begin with Accredo Red Welcome to Accredo information Yellow Financial resources for a disorder Red Fax number for new prescription Yellow Refill an Rx Red Filling Rx for the first time Yellow Medicines for a particular disease Red Secure email form Yellow New patient refill process Red Payment mail-in address Yellow Patient forms Red What is a specialty pharmacy? video Red Refill procedure for existing patients Red Since the hypothesis works for some items but not others, we need to know how easily users can find these items given the current hierarchy. Card sorting is exploratory, tree testing is confirmatory. Time to look at the tree test…
T r e e T e s t Se tu p The setup for the tree test activity needed to align with the card sort and then be carried over into the click test. 1. Select items that may be difficult to sort, items you want more data on, or items we had additional questions about 2. Select easy and hard items based on location in the tree 3. Create tasks for those items
• Participants were presented with the top-level navigation • Participants then were asked to perform tasks using the tree • When a participant believed they found the desired information, they would select it and continue. If they were unsure where to find the information, they could select “I don’t know”. T r e e T e s t
Clicks Task Home Getting Started Conditions Find Medication Contact Us Overall Success Success Path Time N Sign up for Accredo Specialty Pharmacy #% #% #% #% #% #% Home > Register #%; Create an Account > Register Now #%; Filling for the First Time > Call patient services to get started #%; Home > How to get started #%; Welcome > Register Now #% 0:00 # Reorder a drug your doctor prescribed you #% #% #% #% #% #% Home > Refill an Rx #%; Contact Us > Call patient services – Refills and Orders #%; Refilling Prescriptions > Phone number to refill prescription #% 0:00 # Check where your drug is in the fulfillment and delivery process #% #% #% #% #% #% Contact Us > Call patient services – Refills and Orders #%; Home > Order Status #% 0:00 # Find information about the process for getting set up with Accredo Specialty Pharmacy #% #% #% #% #% #% Home > How to get started #%; Create an Account > Create an Accredo account information #%; Welcome > Welcome to Accredo! Information #% 0:00 # Learn about the purpose of a specialty pharmacy #% #% #% #% #% #% Welcome > “What is a specialty pharmacy?” video #%; Welcome > What Accredo provides the patient #% 0:00 # Learn how to order your drug with Accredo Specialty Pharmacy as a new patient #% #% #% #% #% #% Filling for the First Time > How to fill your prescription with Accredo for the first time #%; Home > How to get started #%; Filling for the First Time > Call patient services to get started #% 0:00 # Learn how to order your drug with Accredo Specialty Pharmacy as a returning patient #% #% #% #% #% #% Refilling Prescriptions > Existing Patient – How to refill your prescription #%; Contact Us > Call patient services – Refills and Orders #%; Home > Refill an Rx #% 0:00 # Find general information about Bladder Cancer #% #% #% #% #% #% Bladder Cancer > Overview #% 0:00 # Find out the signs of having Epilepsy #% #% #% #% #% #% Epilepsy > Symptoms #% 0:00 # Learn about drug treatments associated with Hemophilia B #% #% #% #% #% #% Hemophilia B > Medications #%; Find a Medication > Search Medications Offered #% 0:00 # Learn more about manufacturer and community programs that help reduce the drug costs for Growth Hormone Replacement #% #% #% #% #% #% Growth Hormone Replacement > Financing Your Care #% 0:00 # Learn more about Leukemia assistance programs and advocacy organizations #% #% #% #% #% #% Leukemia > Find Support #% 0:00 # Learn more about Hepatitis C's Accredo care team #% #% #% #% #% #% Hepatitis C > Accredo Team #% 0:00 # Find a drug by it's manufacturer's name #% #% #% #% #% #% Find a Medication > Search Medications Offered #% 0:00 # Find number to call to ask about a balance on your account #% #% #% #% #% #% Contact Us > Call patient services – Billing and Payments #% 0:00 # Find the mailing address to send in money to pay off an account balance #% #% #% #% #% #% Contact Us > Mail your payments address #% 0:00 # Find phone number for a specialized condition-assistance team #% #% #% #% #% #% Contact Us > Call patient services – Condition-specific assistance #% 0:00 # Reach out to the Digital Support Team to get help with the website #% #% #% #% #% #% Contact Us > Call patient services – Digital Support Team #% 0:00 # Find the information for sending Accredo Specialty Pharmacy a fax #% #% #% #% #% #% Contact Us > Send us a fax – General Patient Fax #% 0:00 # Send Accredo Specialty Pharmacy an email from the website #% #% #% #% #% #% Contact Us > Send us a secure online message – Email us #% 0:00 # T r e e T e s t F i n di n gs
T r e e T e s t F i n di n gs Tasks Overall Success Success First Attempt Success Multiple Attempts Non Success I don't know Percent Time Percent Time Percent Time Percent Time Percent Time Sign up for Accredo Specialty Pharmacy #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Reorder a drug your doctor prescribed you #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Check where your drug is in the fulfillment and delivery process #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Find information about the process for getting set up with Accredo Specialty Pharmacy #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Learn about the purpose of a specialty pharmacy #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Learn how to order your drug with Accredo Specialty Pharmacy as a new patient #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Learn how to order your drug with Accredo Specialty Pharmacy as a returning patient #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Find general information about Bladder Cancer #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Find out the signs of having Epilepsy #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Learn about drug treatments associated with Hemophilia B #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Learn more about manufacturer and community programs that help reduce the drug costs for Growth Hormone Replacement #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Learn more about Leukemia assistance programs and advocacy organizations #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Learn more about Hepatitis C's Accredo care team #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Find a drug by it's manufacturer's name #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Find number to call to ask about a balance on your account #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Find the mailing address to send in money to pay off an account balance #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Find phone number for a specialized condition-assistance team #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Reach out to the Digital Support Team to get help with the website #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Find the information for sending Accredo Specialty Pharmacy a fax #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Send Accredo Specialty Pharmacy an email from the website #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00
T r e e T e s t F i n di n gs Tasks Overall Success Success First Attempt Success Multiple Attempts Non Success I don't know Percent Time Percent Time Percent Time Percent Time Percent Time Sign up for Accredo Specialty Pharmacy #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Reorder a drug your doctor prescribed you #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Check where your drug is in the fulfillment and delivery process #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Find information about the process for getting set up with Accredo Specialty Pharmacy #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Learn about the purpose of a specialty pharmacy #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Learn how to order your drug with Accredo Specialty Pharmacy as a new patient #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Learn how to order your drug with Accredo Specialty Pharmacy as a returning patient #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Find general information about Bladder Cancer #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Find out the signs of having Epilepsy #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Learn about drug treatments associated with Hemophilia B #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Learn more about manufacturer and community programs that help reduce the drug costs for Growth Hormone Replacement #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Learn more about Leukemia assistance programs and advocacy organizations #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Learn more about Hepatitis C's Accredo care team #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Find a drug by it's manufacturer's name #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Find number to call to ask about a balance on your account #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Find the mailing address to send in money to pay off an account balance #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Find phone number for a specialized condition-assistance team #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Reach out to the Digital Support Team to get help with the website #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Find the information for sending Accredo Specialty Pharmacy a fax #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Send Accredo Specialty Pharmacy an email from the website #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Finding: “Check where your drug is in the fulfillment and delivery process” (Order Status) placement in the card sorts did not match our hypothesis. Participants also looked in a variety of categories trying to find this item in the tree test which made for longer search times. Where would they look on a homepage image? Time for the click test.
C l i ck Test Setup The setup for the click test activity needed to align with the tree test so the same tasks were used. 1. Select items that may be difficult to sort, items you want more data on, or items we had additional questions about. 2. Selected easy and hard items based on location in the tree. 3. Create tasks for those items.
C l i ck Test • Participants were presented with static image of home screen • Participants then were asked to perform tasks and click where they think they would find information
C l i ck Test Fi ndi ngs “Check where your drug is in the fulfillment and delivery process” (Order Status) placement did not match IA hypothesis. Participants also looked in a variety of categories trying to find this item in the tree test which made for longer search times. Where would they look on a homepage image?
www.nicecompany.comCOMPANY NAME Conclusion 05
An a l ysi s Appr o a c h Step 01 Open Card Sort Tree Test Step 03A Click Test Step 03BStep 02 Closed Card Sort To recap how we approached analysis…
An a l ysi s Appr o a c h Step 01 Open Card Sort Tree Test Step 03A Click Test Step 03B • Examine the groups • Analyze card placement • Analyze labels • Analyze organizational schemes • Analyze how accurately participants have grouped content • Analyze participant comments • Analyze a closed card sort • Look at items classified as difficult to sort or belonging in multiple categories Step 02 Closed Card Sort To recap how we approached analysis…
An a l ysi s Appr o a c h Step 01 Open Card Sort Click Test Step 03B • Analyze success rate • Analyze directness of success • Analyze time on task • Look at card sort items classified as difficult to sort or belonging in multiple categories – how do participants approach looking for those items? • Look at runts or categories with two items – are they more easily found in the tree? Where are users looking to find them? Step 02 Closed Card Sort Tree Test Step 03A To recap how we approached analysis…
An a l ysi s Appr o a c h Step 01 Open Card Sort Click Test Step 03BStep 02 Closed Card Sort Tree Test Step 03A • Compare where users drill into the tree v. where they click into the UI • Look at initial entry points for the tree and first click data for the click test – are there similarities? • Note if success changes from the tree test to the click test To recap how we approached analysis…
T i ps fo r M i x e d M e th o d IA Stu dy Se tu p • Select items and tasks that are crucial or controversial • Ordering of method matters • Be consistent across the study • Participant fatigue is a thing –scope your work!
T i ps fo r M i x e d M e th o d IA An a l y s i s 1. Determine if new categories should be created or consolidated using the mental model put forth in open card sort 2. Look to confirm or deny hypotheses about categories using the closed card sort 3. See what issues of findability there are in the tree test 4. Note where people would place items in the UI in the click study
• Allows for data comparisons – useful in understanding convergence and divergence of findings • Collects rich, comprehensive data – provides a more complete story than a single method would alone • Combining two or more of these methods worth the cost. • Time consuming to create • Participants can easily grow tired • Data might be messy • Can be difficult to successfully integrate data Some challenges with mixed method IA validation… Adva n ta ge sC hal l enges Some advantages of mixed method IA validation…
C oncl usi on By spending the time constructing a more robust study design, you can create a holistic view of how the user is thinking and behaving toward your IA.
www.nicecompany.comCOMPANY NAME GET IN T OU C H cegarstka@express-scripts.com caron.garstka@gmail.com E-mail Thanks!
www.nicecompany.comCOMPANY NAME Questions!

More Related Content

PDF
UXPA2019 Optimal AR UX for Complex Purchases — How immersive technology boost...
byUXPA International
 
PPTX
UX in the Age of AI: Leading with Design
byUXPA International
 
PPTX
UXPA2019 How to (Build and) Test Conversational Interfaces
byUXPA International
 
PDF
UXPA2019 Designing Technology for an Aging Population
byUXPA International
 
PPTX
Product Design in Agile Environments: Making it Work at ProductCamp Pittsburgh
byCarol Smith
 
PPTX
Prototyping for Beginners - Pittsburgh Inclusive Innovation Summit 2019
byCarol Smith
 
PPTX
Making Great User Experiences, Pittsburgh Scrum MeetUp, Oct 17, 2017
byCarol Smith
 
PPTX
UXPA2019 Not Your Average Chatbot: Using Cognitive Intercept to Improve Infor...
byUXPA International
 
UXPA2019 Optimal AR UX for Complex Purchases — How immersive technology boost...
byUXPA International
 
UX in the Age of AI: Leading with Design
byUXPA International
 
UXPA2019 How to (Build and) Test Conversational Interfaces
byUXPA International
 
UXPA2019 Designing Technology for an Aging Population
byUXPA International
 
Product Design in Agile Environments: Making it Work at ProductCamp Pittsburgh
byCarol Smith
 
Prototyping for Beginners - Pittsburgh Inclusive Innovation Summit 2019
byCarol Smith
 
Making Great User Experiences, Pittsburgh Scrum MeetUp, Oct 17, 2017
byCarol Smith
 
UXPA2019 Not Your Average Chatbot: Using Cognitive Intercept to Improve Infor...
byUXPA International
 

What's hot

PPTX
Making Great User Experiences at Cleveland C# .Net Meetup July 27 2017
byCarol Smith
 
PPTX
Users are Losers! They’ll Like Whatever we Make! and Other Fallacies.
byCarol Smith
 
PDF
10 Truths to Great Product Experiences
byJeremy Johnson
 
PPTX
Selling UX in Your Organization at Cleveland World Usability Day (WUD)
byCarol Smith
 
PPTX
Faster Usability Testing in an Agile World - Agile UX Virtual Summit 2017 by ...
byCarol Smith
 
PPTX
UX in the Age of AI: Where Does Design Fit In? Fluxible 2017
byCarol Smith
 
PPTX
Making Faster UX in an Agile World - HOAPitt 2017
byCarol Smith
 
PPT
UI For Alien Cowboys
byMatt Jones
 
PPTX
The Mobile Frontier
byRachel Hinman
 
PPTX
Introduction to User Experience Design 10/07/17
byRobert Stribley
 
PDF
Design Stories Are The New User Stories
byLuis Daniel Rodriguez
 
PDF
ai-ux-ui
byGiselle Herrera
 
PPT
Rulespace
byMatt Jones
 
PPT
World Usability day: Bridging Research-Practice Gap
byKeith Instone
 
PDF
Why UX is Important
byChris Becker
 
PDF
Prototyping - 4 Strategic Factors for Designers - UXPA 2016
byLyle Kantrovich
 
PPTX
The User Experience Design of Everything
bySiyabonga Africa
 
PDF
#1NWebinar: Digital on the Runway
byOne North
 
PDF
UX for Business Analysts
byTarik (Rick) Dzekman
 
KEY
Closing the Findability Gap: 8 better practices from information architecture
byLouis Rosenfeld
 
Making Great User Experiences at Cleveland C# .Net Meetup July 27 2017
byCarol Smith
 
Users are Losers! They’ll Like Whatever we Make! and Other Fallacies.
byCarol Smith
 
10 Truths to Great Product Experiences
byJeremy Johnson
 
Selling UX in Your Organization at Cleveland World Usability Day (WUD)
byCarol Smith
 
Faster Usability Testing in an Agile World - Agile UX Virtual Summit 2017 by ...
byCarol Smith
 
UX in the Age of AI: Where Does Design Fit In? Fluxible 2017
byCarol Smith
 
Making Faster UX in an Agile World - HOAPitt 2017
byCarol Smith
 
UI For Alien Cowboys
byMatt Jones
 
The Mobile Frontier
byRachel Hinman
 
Introduction to User Experience Design 10/07/17
byRobert Stribley
 
Design Stories Are The New User Stories
byLuis Daniel Rodriguez
 
ai-ux-ui
byGiselle Herrera
 
Rulespace
byMatt Jones
 
World Usability day: Bridging Research-Practice Gap
byKeith Instone
 
Why UX is Important
byChris Becker
 
Prototyping - 4 Strategic Factors for Designers - UXPA 2016
byLyle Kantrovich
 
The User Experience Design of Everything
bySiyabonga Africa
 
#1NWebinar: Digital on the Runway
byOne North
 
UX for Business Analysts
byTarik (Rick) Dzekman
 
Closing the Findability Gap: 8 better practices from information architecture
byLouis Rosenfeld
 

Similar to Advanced Card Sorting and Mixed Method IA Validation

PPTX
IA breakfast briefing apr12 upload
byRoss Philip
 
PPTX
Testing Taxonomies: Beyond Card Sorting
byDave Cooksey
 
PDF
Taxonomy Validation
byDave Cooksey
 
PPTX
Taxonomy Validation
byDave Cooksey
 
PDF
SPTechCon Austin 2015 - Perfecting Information Architecture
byJill Hannemann
 
PPTX
DC SPUG Feb 2015 The Secret Sauce to Information Architecture
byJill Hannemann
 
PDF
IA basics
byZdeněk Lanc
 
PPTX
Card Sort - UXRA Education Series_2022-06-16l.pptx
byJasonTelner
 
PDF
Testing Taxonomies: Beyond Card Sorting
byAlberta Soranzo
 
PPTX
Card Sorting Your Way to Meaningful Metadata
byRob Bogue
 
PDF
Designing an effective information architecture
byoptimalworkshop
 
PDF
How to Build a Customer-Focused Information Architecture
byCenterline Digital
 
PPT
Process And Methodology Research
byMiles Price
 
PDF
Information Architecture Exposing the Secret Sauce for Success
byBaltimore SharePoint (BSPUG)
 
PPS
Making IA Real: Planning an Information Architecture Strategy
byChiara Fox Ogan
 
PDF
AgileMidwest2018-Cooley-AllTheAI
byJason Tice
 
PDF
Managing your Metadata w/ SharePoint 2010
byvman916
 
PDF
Building a User-Focused Information Architecture
byDevin Asaro
 
PDF
Building a User-Focused Information Architecture
byGreg Harron
 
PPT
Information Architecture and the Distributed User Experience
byJason Ryan
 
IA breakfast briefing apr12 upload
byRoss Philip
 
Testing Taxonomies: Beyond Card Sorting
byDave Cooksey
 
Taxonomy Validation
byDave Cooksey
 
Taxonomy Validation
byDave Cooksey
 
SPTechCon Austin 2015 - Perfecting Information Architecture
byJill Hannemann
 
DC SPUG Feb 2015 The Secret Sauce to Information Architecture
byJill Hannemann
 
IA basics
byZdeněk Lanc
 
Card Sort - UXRA Education Series_2022-06-16l.pptx
byJasonTelner
 
Testing Taxonomies: Beyond Card Sorting
byAlberta Soranzo
 
Card Sorting Your Way to Meaningful Metadata
byRob Bogue
 
Designing an effective information architecture
byoptimalworkshop
 
How to Build a Customer-Focused Information Architecture
byCenterline Digital
 
Process And Methodology Research
byMiles Price
 
Information Architecture Exposing the Secret Sauce for Success
byBaltimore SharePoint (BSPUG)
 
Making IA Real: Planning an Information Architecture Strategy
byChiara Fox Ogan
 
AgileMidwest2018-Cooley-AllTheAI
byJason Tice
 
Managing your Metadata w/ SharePoint 2010
byvman916
 
Building a User-Focused Information Architecture
byDevin Asaro
 
Building a User-Focused Information Architecture
byGreg Harron
 
Information Architecture and the Distributed User Experience
byJason Ryan
 

More from UXPA International

PPTX
UXPA 2023: Start Strong - Lessons learned from associate programs to platform...
byUXPA International
 
PDF
UXPA 2023: Disrupting Inaccessibility: Applying A11Y-Focused Discovery & Idea...
byUXPA International
 
PDF
UXPA 2023 Poster: ESG & Sustainable UX
byUXPA International
 
PDF
UXPA 2023 Poster: The Two Tracks of UX Under Agile: Tactical and Strategic
byUXPA International
 
PDF
UXPA 2023: Data science and UX: Smarter together
byUXPA International
 
PDF
UXPA 2023: UX Fracking: Using Mixed Methods to Extract Hidden Insights
byUXPA International
 
PDF
UXPA 2023 Poster: Are virtual spaces the future of video conferencing?
byUXPA International
 
PDF
UXPA 2023: Learn how to get over personas by swiping right on user roles
byUXPA International
 
PDF
UXPA 2023: F@#$ User Personas
byUXPA International
 
PDF
UXPA 2023 Poster: Pocket Research Guide - Empower your Solution and Foster Cu...
byUXPA International
 
PDF
UXPA 2023: Experience Maps - A designer's framework for working in Agile team...
byUXPA International
 
PDF
UXPA 2023 Poster: Atomic Research in Practice: Using a Feedback Repository to...
byUXPA International
 
PDF
UXPA 2023 Poster: Leveraging Dial Testing To Measure Real-Time User Frustrati...
byUXPA International
 
PDF
UXPA 2023: UX Enterprise Story: How to apply a UX process to a company withou...
byUXPA International
 
PPTX
UXPA 2023: High-Fives over Zoom: Creating a Remote-First Creative Team
byUXPA International
 
PPTX
UXPA 2023: Behind the Bias: Dissecting human shortcuts for better research & ...
byUXPA International
 
PDF
UXPA 2023 Poster: Improving the Internal and External User Experience of a Fe...
byUXPA International
 
PDF
UXPA 2023 Poster: 5 Key Findings from Moderated Accessibility Testing with Sc...
byUXPA International
 
PDF
UXPA 2023: Lessons for new managers
byUXPA International
 
PDF
UXPA 2023: Redesigning An Automotive Feature from Gasoline to Electric Vehicl...
byUXPA International
 
UXPA 2023: Start Strong - Lessons learned from associate programs to platform...
byUXPA International
 
UXPA 2023: Disrupting Inaccessibility: Applying A11Y-Focused Discovery & Idea...
byUXPA International
 
UXPA 2023 Poster: ESG & Sustainable UX
byUXPA International
 
UXPA 2023 Poster: The Two Tracks of UX Under Agile: Tactical and Strategic
byUXPA International
 
UXPA 2023: Data science and UX: Smarter together
byUXPA International
 
UXPA 2023: UX Fracking: Using Mixed Methods to Extract Hidden Insights
byUXPA International
 
UXPA 2023 Poster: Are virtual spaces the future of video conferencing?
byUXPA International
 
UXPA 2023: Learn how to get over personas by swiping right on user roles
byUXPA International
 
UXPA 2023: F@#$ User Personas
byUXPA International
 
UXPA 2023 Poster: Pocket Research Guide - Empower your Solution and Foster Cu...
byUXPA International
 
UXPA 2023: Experience Maps - A designer's framework for working in Agile team...
byUXPA International
 
UXPA 2023 Poster: Atomic Research in Practice: Using a Feedback Repository to...
byUXPA International
 
UXPA 2023 Poster: Leveraging Dial Testing To Measure Real-Time User Frustrati...
byUXPA International
 
UXPA 2023: UX Enterprise Story: How to apply a UX process to a company withou...
byUXPA International
 
UXPA 2023: High-Fives over Zoom: Creating a Remote-First Creative Team
byUXPA International
 
UXPA 2023: Behind the Bias: Dissecting human shortcuts for better research & ...
byUXPA International
 
UXPA 2023 Poster: Improving the Internal and External User Experience of a Fe...
byUXPA International
 
UXPA 2023 Poster: 5 Key Findings from Moderated Accessibility Testing with Sc...
byUXPA International
 
UXPA 2023: Lessons for new managers
byUXPA International
 
UXPA 2023: Redesigning An Automotive Feature from Gasoline to Electric Vehicl...
byUXPA International
 

Recently uploaded

PDF
Seven C's for the effective communication
byamanbushra53
 
PPTX
cirrhosis best presentation about hepatitis
bynesrushukre34
 
PPTX
Taguchi method in Factorial experiment ppt
byVENUGOPALK61
 
PDF
(01) Crazy or REAL Conspiracy About Global Power? Let's find out
byetherealunfold
 
PDF
sangfor-security-cyber-resilience-challenge-outsmarting-threats-with-mdr.pdf
byvergiehadiana2
 
PPTX
AF6F92A7-PPT-Chapter1-discrete math.pptx
bynastaranEmamjomeh1
 
PPTX
Intestinal obstruction is one of the most important part of long case in surg...
bynesrushukre34
 
PDF
(02) The Globalist Elite Aren't Fiction.pdf
byetherealunfold
 
PPTX
TTE WINDOWS AND VIEWS . The heart is surrounded by lung tissue and is hidden ...
byDr:SANA IHSAN
 
PPTX
LIBRA.I. Libraries for AI Literacy: Guiding Citizens in the Digital World
byVlaamse Vereniging voor Bibliotheek, Archief & Documentatie vzw (VVBAD)
 
PDF
Admire Girls — Celebrating Women Who Inspire Change By Jessa Bariuan
byjessabariuan2
 
PDF
The American Public Health Association.pdf
bybarejapriyanka1
 
PPTX
GERMANY EINHEIT OCTOBER 1990 A MEMORIZABLE DAY IN WORLD HISTORY
bywk6346353
 
PDF
Dutch Power Presentatie – Arjan Wargers - interoperabiliteit
byDutch Power
 
PDF
What is a real ecosystem? And how can you respect nature?
byGeorge Torok
 
PPTX
Placket Burman Design PPT (Design of Experiments)
byVENUGOPALK61
 
PDF
BRAZA PPT PPT PPT PPT PPT P FOR DEMO.pdf
byBeaYonsonBraza1
 
PDF
Learn More About Digital Marketing in detail
bygetafreen123
 
PDF
Why you must stop saying the word "thing".
byGeorge Torok
 
PPTX
developing results and discussion in qualitative research.pptx
bybereniceandriano1985
 
Seven C's for the effective communication
byamanbushra53
 
cirrhosis best presentation about hepatitis
bynesrushukre34
 
Taguchi method in Factorial experiment ppt
byVENUGOPALK61
 
(01) Crazy or REAL Conspiracy About Global Power? Let's find out
byetherealunfold
 
sangfor-security-cyber-resilience-challenge-outsmarting-threats-with-mdr.pdf
byvergiehadiana2
 
AF6F92A7-PPT-Chapter1-discrete math.pptx
bynastaranEmamjomeh1
 
Intestinal obstruction is one of the most important part of long case in surg...
bynesrushukre34
 
(02) The Globalist Elite Aren't Fiction.pdf
byetherealunfold
 
TTE WINDOWS AND VIEWS . The heart is surrounded by lung tissue and is hidden ...
byDr:SANA IHSAN
 
LIBRA.I. Libraries for AI Literacy: Guiding Citizens in the Digital World
byVlaamse Vereniging voor Bibliotheek, Archief & Documentatie vzw (VVBAD)
 
Admire Girls — Celebrating Women Who Inspire Change By Jessa Bariuan
byjessabariuan2
 
The American Public Health Association.pdf
bybarejapriyanka1
 
GERMANY EINHEIT OCTOBER 1990 A MEMORIZABLE DAY IN WORLD HISTORY
bywk6346353
 
Dutch Power Presentatie – Arjan Wargers - interoperabiliteit
byDutch Power
 
What is a real ecosystem? And how can you respect nature?
byGeorge Torok
 
Placket Burman Design PPT (Design of Experiments)
byVENUGOPALK61
 
BRAZA PPT PPT PPT PPT PPT P FOR DEMO.pdf
byBeaYonsonBraza1
 
Learn More About Digital Marketing in detail
bygetafreen123
 
Why you must stop saying the word "thing".
byGeorge Torok
 
developing results and discussion in qualitative research.pptx
bybereniceandriano1985
 

Advanced Card Sorting and Mixed Method IA Validation

  • 1.
    www.nicecompany.comCOMPANY NAME Caron Garstka UXPA2018 Card Sorting and Mixed Method IA Validation
  • 2.
    www.nicecompany.comCOMPANY NAME A littleabout me Information Architecture (IA): A quick review IA validation methods Mixed method IA validation Case Study 01. 02. 03. 04. 05. C ON TEN TS Conclusion
  • 3.
  • 4.
    www.nicecompany.comCOMPANY NAME A li ttl e a bo u t m e … Who I am… • Caron Garstka • Senior UX Researcher at Express Scripts • Masters of Science in Information from University of Texas at Austin What I do… • Conduct research with patients who have chronic, complex, or life-threatening conditions • Help shape their digital experiences using their feedback My goal is to make patient lives easier
  • 5.
  • 6.
    • It isthe creation of a structure for a website, intranet, application, or other project • Allows users to understand where they are, and where the desired information is in relation to their position • Results in the creation of site maps, hierarchies, categorizations, navigation, and metadata W h a t i s In fo r m a ti o n Ar c h i te c tu r e ( IA) ?
  • 7.
    Information architecture isabout helping people understand their surroundings and find what they’re looking for, in the real world as well as online. – Information Architecture Institute W h a t i s In fo r m a ti o n Ar c h i te c tu r e ( IA) ?
  • 8.
    A Ph ase d Appr o a c h to IA D e ve l o pm e n t There are multiple steps to creating and maintaining an IA Research Strategy Design Implementation Administration Card Sorting lives here!
  • 9.
    A Ba la n c e d IA Re se a r c h Appr o a c h Documents Data types Content objects Metadata Volume Existing structure Business goals Funding Politics Culture Technology Resources Constraints Audience Tasks Needs Information seeking behavior Experience Vocabularies C o n t e n t C o n t e x t U s e r s
  • 10.
    How is itcategorized? How is it displayed? Where should it be stored? What information is relevant at a given time? Is the information necessary? What are they looking for? What’s important to them? What are they familiar with? C o n t e n t C o n t e x t U s e r s A Ba l a n c e d IA Re se a r c h Appr o a c h
  • 11.
    A Ba la n c e d IA Re se a r c h Appr o a c h Heuristic evaluation Metadata and content analysis Content mapping Benchmarking Background research Presentations and meetings Stakeholder interviews Technology assessments Search log and clickstream analysis Use cases and personas Contextual inquiry User interviews and user testing C o n t e n t C o n t e x t U s e r s
  • 12.
    IA Context UsersContent A Ba la n c e d IA Re se a r c h Appr o a c h
  • 13.
  • 14.
    Content audits Establishing startingpoint Surveys Gathering opinions Field studies, diaries, contextual inquiry, interviews, focus groups Learning about users Web traffic statistics, clickstreams Online behavior Be fo r e do i n g IA va l i da ti o n … Do the following methods before starting validation research.
  • 15.
    IA Va li da ti o n R e s e a r c h M e th o ds Tree Testing Looks at IA as working hierarchy Closed Card Sorting Looks at viability of defined categories Click Testing Looks at first-level IA in context of UI Hybrid Card Sorting Combination of open and closed Skews open or closed Open Card Sorting Looks at user-generated groupings and names Delphi Card Sorting Looks at user-generated groupings and names
  • 16.
    IA Va li da ti o n R e s e a r c h M e th o ds Tree Testing Looks at IA as working hierarchy Closed Card Sorting Looks at viability of defined categories Click Testing Looks at first-level IA in context of UI Hybrid Card Sorting Combination of open and closed Skews open or closed Open Card Sorting Looks at user-generated groupings and names Delphi Card Sorting Looks at user-generated groupings and names • Users sort items into groups and give groups a name; typically done prior to an architecture being put in place • Allows the researcher to document a user’s mental model and inform the IA
  • 17.
    IA Va li da ti o n R e s e a r c h M e th o ds Tree Testing Looks at IA as working hierarchy Closed Card Sorting Looks at viability of defined categories Click Testing Looks at first-level IA in context of UI Hybrid Card Sorting Combination of open and closed Skews open or closed Open Card Sorting Looks at user-generated groupings and names Delphi Card Sorting Looks at user-generated groupings and names • Users sort items into defined categories • Allows the researcher to document a user’s mental model and understand how well the proposed IA matches their model
  • 18.
    IA Va li da ti o n R e s e a r c h M e th o ds Tree Testing Looks at IA as working hierarchy Closed Card Sorting Looks at viability of defined categories Click Testing Looks at first-level IA in context of UI Hybrid Card Sorting Combination of open and closed Skews open or closed Open Card Sorting Looks at user-generated groupings and names Delphi Card Sorting Looks at user-generated groupings and names • Users sort items into researcher-defined categories but have the option to create their own • Allows the researcher to document a user’s mental model and allows users to participate in building the IA
  • 19.
    IA Va li da ti o n R e s e a r c h M e th o ds Tree Testing Looks at IA as working hierarchy Closed Card Sorting Looks at viability of defined categories Click Testing Looks at first-level IA in context of UI Hybrid Card Sorting Combination of open and closed Skews open or closed Open Card Sorting Looks at user-generated groupings and names Delphi Card Sorting Looks at user-generated groupings and names • Users work one after another to complete card sort • Users modify as needed and pass modifications to next user • Allows the researcher to rapidly arrive at an organizational model
  • 20.
    IA Va li da ti o n R e s e a r c h M e th o ds Tree Testing Looks at IA as working hierarchy Closed Card Sorting Looks at viability of defined categories Click Testing Looks at first-level IA in context of UI Hybrid Card Sorting Combination of open and closed Skews open or closed Open Card Sorting Looks at user-generated groupings and names Delphi Card Sorting Looks at user-generated groupings and names • Users navigate a hierarchy using only top-level link names. • Allows the researcher to see how findable items are within the hierarchy
  • 21.
    I A Va l i d a t i o n R e s e a rc h Me t h o d s Tree Testing Looks at IA as working hierarchy Closed Card Sorting Looks at viability of defined categories Click Testing Looks at first-level IA in context of UI Hybrid Card Sorting Combination of open and closed Skews open or closed Open Card Sorting Looks at user-generated groupings and names Delphi Card Sorting Looks at user-generated groupings and names • Users locate items within a static design • Allows the researcher to see if the top level IA is easily findable within the context of the design
  • 22.
    C ompari sonof IA Research Methods Method Research Type Activity Learn About Project Stage Open card sorting Generative Organizing Mental models Organization Labeling Relationships between items Early user research Closed card sorting Evaluative Organizing Mental models Organization Labeling Information architecture design Hybrid card sorting Generative Evaluative Organizing Mental models Organization Labeling Relationships between items Early user research Delphi card sorting Generative Organizing Mental models Organization Labeling Relationships between items Early user research Reverse card sorting (Tree testing) Evaluative Finding Organization Labeling Findability Information architecture design Click testing (or usability testing with low- fidelity prototypes) Evaluative Finding Organization Labeling Findability Navigation design Interface design Interface design
  • 23.
    Pr a cti c a l T r i a n gu l a ti o n [Triangulation is an] attempt to map out, or explain more fully, the richness and complexity of human behavior by studying it from more than one standpoint. - Cohen and Manion (1986)
  • 24.
  • 25.
    www.nicecompany.comCOMPANY NAME Accredo.com Accredo.com 1. Stakeholderswanted to validate a proposed IA on the Accredo.com website 2. Conduct a closed card sort to catch any glaring issues The Perceived Need
  • 26.
    www.nicecompany.comCOMPANY NAME The RealNeed Expanded scope… 1. How well the proposed IA matches the user’s mental model 2. How findable items are within the proposed IA 3. How UI helps or obscures IA 4. Areas for improvement 5. Validate the proposed patient IA 6. Learn how potential users group content 7. Evaluate the findability of topics within the website structure 8. Evaluate the homepage design and whether potential users can fine the expected entry point for specific information or functionality within the design And there wasn’t much time… Accredo.com Accredo.com
  • 27.
    www.nicecompany.comCOMPANY NAME • Sitecaters to multiple audiences • Users are distributed • Site IA is different pre and post login Focus on one side of the experience first. Conduct study remotely, asynchronously, and without moderation. Allowed for fast data capture without having to moderate each session. Accredo.com Accredo.com Other Challenges
  • 28.
    Appr o ac h Tree Testing Closed Card Sorting Click Testing Hybrid Card SortingOpen Card Sorting Delphi Card Sorting One method alone would not get at what the team needed.
  • 29.
    Appr o ac h Used UserZoom to conduct the study. Tree Test N= 43 Median Completion Time: 25 min Click Test N= 44 Median Completion Time: 20 min Open/Closed Card Sort N= 87
  • 30.
    Appr o ac h • Get at users’ mental model • Note alternative categories, item placement, and category names • Pin point hard to place items Step 01 Open Card Sort • See how users group items within given categories • See how well groupings match the proposed IA • Note alternative categories, item placement, and category names • Pin point hard to place items Step 02 Closed Card Sort • Look at where users drill into the tree • Note findability of key items Tree Test Step 03A • Look at where users drill into the design • Note expectations of finding items in design Click Test Step 03B
  • 31.
    A p pro a c h Tree Test N= 43 Median Completion Time: 25 min Click Test N= 44 Median Completion Time: 20 min Open/Closed Card Sort N= 87
  • 32.
    C ard SortSetup 1. Selected content that could be grouped 2. Selected content that was at the same level 3. Selected content that was representative 4. Selected content that did not lead the activity 5. Selected content that participants would understand Maintained consistency across the open and closed card sorts.
  • 33.
    Ope n Ca r d So r t • Presented participants with a list of Accredo proposed IA items and asked them to sort those items into groups • Participants sorted items into groups that made sense to them • Participants created their own names for the categories the created • Some labels were modified for the card sort to add clarity UserZoom interface for open card sort
  • 34.
    Ope n Ca r d So r t F i n di n gs • Dendrogram: A tree diagram of hierarchical clustering to learn how users group items. • Dendorgram scale: 0 to 1; the lowest level of clusters with the highest agreement. • Euclidian distance the average distance between items. Interpreting dendrogram clusters is an art. Data Output: Dendrogram Overview
  • 35.
    Ope n Ca r d So r t F i n di n gs Interpretations Cluster 2: Had a runt: “Communication Information for digital support team” Cluster 1: Fairly cohesive, aligned with the hypothesized IA, and participants came up with similar names Cluster 3: High agreement for items related to prescriptions, not matching hypothesized IA, but logically made sense
  • 36.
    Ope n Ca r d So r t F i n di n gs Cluster 4: Cohesive group related to a Welcome or introductory section Cluster 5: Mixed assortment of forms and administration items. These items had strong agreement but did not match IA. Interpretations
  • 37.
    Ope n Ca r d So r t F i n di n gs Finding: Participants grouped items according to Disease FAQ, Prescriptions and Refills, and Account functions Participants were less certain on how to handle “Communication information for digital support team”
  • 38.
    Ope n Ca r d So r t F i n di n gs Difficult Items to Sort Items Participants Wanted to Sort into Multiple Categories Post hoc questions about the sorting experience provides additional insights
  • 39.
    Ope n Ca r d So r t F i n di n gs Difficult Items to Sort Items Participants Wanted to Sort into Multiple Categories Post hoc questions about the sorting experience provides additional insights Finding: “Communication information for digital support team” and “Register” were both highly ranked for difficulty to sort and participants thought the items could belong in multiple categories. Perhaps they would be easier to sort if given a structure. Sounds like a job for the Closed Card Sort!
  • 40.
    C l osedC ard Sort • Participants were first presented with a list of Accredo IA items and were asked to sort those items into pre-named groups UserZoom interface for closed card sort
  • 41.
    C l osedC ard Sort Wanted to see how well participants’ sorts aligned with proposed closed groups. Home Getting Started Conditions Find Medication Contact Us Manage your account Create an Accredo account information Symptoms Search to view Accredo drugs Communication information for digital support team Download Accredo mobile app Authorization to use and disclose health information PDF Disorder overview Phone numbers for disorder- specific assistance Order status Authorization to use and disclose health information PDF Find disease-specific support Billing call information Register Patient consent agreement and acknowledgement document Specific disease/disorder care team Phone number to refill medicine How to begin with Accredo Welcome to Accredo information Financial resources for a disorder Fax number for new prescription Refill an Rx Filling Rx for the first time Medicines for a particular disease Secure email form New patient refill process Payment mail-in address Patient forms What is a specialty pharmacy? video Refill procedure for existing patients
  • 42.
    C l osedC ard Sort Fi ndi ngs Methodology • For this analysis I chose to use the average linkage dendrogram for analysis. • I set the agreement minimum to 50% - this is the minimum percentage of agreement the selected items that participants included into the same category.
  • 43.
    C l osedC ard Sort Fi ndi ngs Finding: Participants seemed to have an easier time grouping “Communication information for digital support team” – this was the runt in the open sort – grouping it in with other “contact” related items. Interestingly, “Medicines for a particular disease” and “Search to view Accredo drugs” hung together, where as in the open card sort they did not.
  • 44.
    C l osedC ard Sort Fi ndi ngs Items Home Getting Started Conditions Find a Medication Contact Us Highest % Register #% #% #% #% #% #% Refill an Rx #% #% #% #% #% #% Order status #% #% #% #% #% #% How to begin with Accredo #% #% #% #% #% #% Welcome to Accredo information #% #% #% #% #% #% New patient refill process #% #% #% #% #% #% Refill procedure for existing patients #% #% #% #% #% #% Manage your account #% #% #% #% #% #% Disorder overview #% #% #% #% #% #% Symptoms #% #% #% #% #% #% Medicines for a particular disease #% #% #% #% #% #% Financial resources for a disorder #% #% #% #% #% #% Find disease-specific support #% #% #% #% #% #% Specific disease/disorder care team #% #% #% #% #% #% Search to view Accredo drugs #% #% #% #% #% #% What is a specialty pharmacy? video #% #% #% #% #% #% Authorization to use and disclose health information PDF #% #% #% #% #% #% Patient consent agreement and acknowledgement document #% #% #% #% #% #% Phone number to refill medicine #% #% #% #% #% #% Fax number for new prescription #% #% #% #% #% #% Patient forms #% #% #% #% #% #% Filling Rx for the first time #% #% #% #% #% #% Create an Accredo account information #% #% #% #% #% #% Phone numbers for disorder-specific assistance #% #% #% #% #% #% Secure email form #% #% #% #% #% #% Payment mail-in address #% #% #% #% #% #% Billing call information #% #% #% #% #% #% Communication information for digital support team #% #% #% #% #% #% Download Accredo mobile app #% #% #% #% #% #%
  • 45.
    C l osedC ard Sort Fi ndi ngs Items Home Getting Started Conditions Find a Medication Contact Us Highest % Register #% #% #% #% #% #% Refill an Rx #% #% #% #% #% #% Order status #% #% #% #% #% #% How to begin with Accredo #% #% #% #% #% #% Welcome to Accredo information #% #% #% #% #% #% New patient refill process #% #% #% #% #% #% Refill procedure for existing patients #% #% #% #% #% #% Manage your account #% #% #% #% #% #% Disorder overview #% #% #% #% #% #% Symptoms #% #% #% #% #% #% Medicines for a particular disease #% #% #% #% #% #% Financial resources for a disorder #% #% #% #% #% #% Find disease-specific support #% #% #% #% #% #% Specific disease/disorder care team #% #% #% #% #% #% Search to view Accredo drugs #% #% #% #% #% #% What is a specialty pharmacy? video #% #% #% #% #% #% Authorization to use and disclose health information PDF #% #% #% #% #% #% Patient consent agreement and acknowledgement document #% #% #% #% #% #% Phone number to refill medicine #% #% #% #% #% #% Fax number for new prescription #% #% #% #% #% #% Patient forms #% #% #% #% #% #% Filling Rx for the first time #% #% #% #% #% #% Create an Accredo account information #% #% #% #% #% #% Phone numbers for disorder-specific assistance #% #% #% #% #% #% Secure email form #% #% #% #% #% #% Payment mail-in address #% #% #% #% #% #% Billing call information #% #% #% #% #% #% Communication information for digital support team #% #% #% #% #% #% Download Accredo mobile app #% #% #% #% #% #% Finding: “Communication information for digital support team” and “Register” were both highly ranked for difficulty sorting and belonging in multiple categories in the open card sort. However, once given a structure, participants had higher agreement for placement. “Patient forms” has a split grouping for where participants think it belongs.
  • 46.
    C l osedC ard Sort Fi ndi ngs Difficult Items to Sort Items Participants Wanted to Sort into Multiple Categories Post hoc questions about the sorting experience provides additional insights
  • 47.
    C l osedC ard Sort Fi ndi ngs Difficult to Sort Patient consent agreement and acknowledgement document What is a specialty pharmacy? video Authorization to use and disclose health information PDF Patient forms Refill an Rx Communication information for digital support team Download Accredo mobile app Billing call information Fax number for new prescription New patient refill process Payment mail-in address Search to view Accredo drugs Create an Accredo account information Phone number to refill medicine Refill procedure for existing patients Financial resources for a disorder Find disease-specific support How to begin with Accredo Order status Disorder overview Manage your account Medicines for a particular disease Register Secure email form Welcome to Accredo information Filling Rx for the first time Phone numbers for disorder-specific assistance Specific disease/disorder care team Top “Difficult to Sort” Items Difficult to Sort Finding: “Communication information for digital support team” and “Register” were both highly ranked for participants thinking they were difficult to sort in the open card sort. Both dropped overall in the closed card sort. But now Refilling Rx is one of the top 5 most difficult to sort.
  • 48.
    Items Participants feltbelonged in Multiple Categories Communication information for digital support team Order status Patient consent agreement and acknowledgement document Create an Accredo account information Phone number to refill medicine Authorization to use and disclose health information PDF Billing call information Fax number for new prescription New patient refill process Patient forms Find disease-specific support Manage your account Refill procedure for existing patients What is a specialty pharmacy? video Financial resources for a disorder Payment mail-in address Refill an Rx Register Search to view Accredo drugs Specific disease/disorder care team Welcome to Accredo information Filling Rx for the first time Medicines for a particular disease Phone numbers for disorder-specific assistance Secure email form Disorder overview Download Accredo mobile app How to begin with Accredo Symptoms Top “Multiple Categories” Items Multiple Categories Finding: Even though Refill Rx was top five most difficult to sort, it was not as highly ranked for belonging in multiple categories. Order status has also jumped up in the list. This would be a prime candidate to look at in the tree test as well to see where people start looking for it. C l osed C ard Sort Fi ndi ngs
  • 49.
    C l osedC ard Sort Fi ndi ngs Home Getting Started Conditions Find Medication Contact Us Hypothesis Correct Hypothesis Correct Hypothesis Correct Hypothesis Correct Hypothesis Correct Manage your account Green Create an Accredo account information Green Symptoms Green Search to view Accredo drugs Green Communication information for digital support team Green Download Accredo mobile app Yellow Authorization to use and disclose health information PDF Yellow Disorder overview Green Phone numbers for disorder-specific assistance Green Order status Red Authorization to use and disclose health information PDF Yellow Find disease-specific support Green Billing call information Green Register Red Patient consent agreement and acknowledgement document Yellow Specific disease/disorder care team Yellow Phone number to refill medicine Yellow How to begin with Accredo Red Welcome to Accredo information Yellow Financial resources for a disorder Red Fax number for new prescription Yellow Refill an Rx Red Filling Rx for the first time Yellow Medicines for a particular disease Red Secure email form Yellow New patient refill process Red Payment mail-in address Yellow Patient forms Red What is a specialty pharmacy? video Red Refill procedure for existing patients Red
  • 50.
    C l osedC ard Sort Fi ndi ngs Home Getting Started Conditions Find Medication Contact Us Accredo Hypothesis Correct Accredo Hypothesis Correct Accredo Hypothesis Correct Accredo Hypothesis Correct Accredo Hypothesis Correct Manage your account Green Create an Accredo account information Green Symptoms Green Search to view Accredo drugs Green Communication information for digital support team Green Download Accredo mobile app Yellow Authorization to use and disclose health information PDF Yellow Disorder overview Green Phone numbers for disorder-specific assistance Green Order status Red Authorization to use and disclose health information PDF Yellow Find disease-specific support Green Billing call information Green Register Red Patient consent agreement and acknowledgement document Yellow Specific disease/disorder care team Yellow Phone number to refill medicine Yellow How to begin with Accredo Red Welcome to Accredo information Yellow Financial resources for a disorder Red Fax number for new prescription Yellow Refill an Rx Red Filling Rx for the first time Yellow Medicines for a particular disease Red Secure email form Yellow New patient refill process Red Payment mail-in address Yellow Patient forms Red What is a specialty pharmacy? video Red Refill procedure for existing patients Red Since the hypothesis works for some items but not others, we need to know how easily users can find these items given the current hierarchy. Card sorting is exploratory, tree testing is confirmatory. Time to look at the tree test…
  • 51.
    T r ee T e s t Se tu p The setup for the tree test activity needed to align with the card sort and then be carried over into the click test. 1. Select items that may be difficult to sort, items you want more data on, or items we had additional questions about 2. Select easy and hard items based on location in the tree 3. Create tasks for those items
  • 52.
    • Participants werepresented with the top-level navigation • Participants then were asked to perform tasks using the tree • When a participant believed they found the desired information, they would select it and continue. If they were unsure where to find the information, they could select “I don’t know”. T r e e T e s t
  • 53.
    Clicks Task Home Getting Started Conditions Find Medication Contact Us Overall Success SuccessPath Time N Sign up for Accredo Specialty Pharmacy #% #% #% #% #% #% Home > Register #%; Create an Account > Register Now #%; Filling for the First Time > Call patient services to get started #%; Home > How to get started #%; Welcome > Register Now #% 0:00 # Reorder a drug your doctor prescribed you #% #% #% #% #% #% Home > Refill an Rx #%; Contact Us > Call patient services – Refills and Orders #%; Refilling Prescriptions > Phone number to refill prescription #% 0:00 # Check where your drug is in the fulfillment and delivery process #% #% #% #% #% #% Contact Us > Call patient services – Refills and Orders #%; Home > Order Status #% 0:00 # Find information about the process for getting set up with Accredo Specialty Pharmacy #% #% #% #% #% #% Home > How to get started #%; Create an Account > Create an Accredo account information #%; Welcome > Welcome to Accredo! Information #% 0:00 # Learn about the purpose of a specialty pharmacy #% #% #% #% #% #% Welcome > “What is a specialty pharmacy?” video #%; Welcome > What Accredo provides the patient #% 0:00 # Learn how to order your drug with Accredo Specialty Pharmacy as a new patient #% #% #% #% #% #% Filling for the First Time > How to fill your prescription with Accredo for the first time #%; Home > How to get started #%; Filling for the First Time > Call patient services to get started #% 0:00 # Learn how to order your drug with Accredo Specialty Pharmacy as a returning patient #% #% #% #% #% #% Refilling Prescriptions > Existing Patient – How to refill your prescription #%; Contact Us > Call patient services – Refills and Orders #%; Home > Refill an Rx #% 0:00 # Find general information about Bladder Cancer #% #% #% #% #% #% Bladder Cancer > Overview #% 0:00 # Find out the signs of having Epilepsy #% #% #% #% #% #% Epilepsy > Symptoms #% 0:00 # Learn about drug treatments associated with Hemophilia B #% #% #% #% #% #% Hemophilia B > Medications #%; Find a Medication > Search Medications Offered #% 0:00 # Learn more about manufacturer and community programs that help reduce the drug costs for Growth Hormone Replacement #% #% #% #% #% #% Growth Hormone Replacement > Financing Your Care #% 0:00 # Learn more about Leukemia assistance programs and advocacy organizations #% #% #% #% #% #% Leukemia > Find Support #% 0:00 # Learn more about Hepatitis C's Accredo care team #% #% #% #% #% #% Hepatitis C > Accredo Team #% 0:00 # Find a drug by it's manufacturer's name #% #% #% #% #% #% Find a Medication > Search Medications Offered #% 0:00 # Find number to call to ask about a balance on your account #% #% #% #% #% #% Contact Us > Call patient services – Billing and Payments #% 0:00 # Find the mailing address to send in money to pay off an account balance #% #% #% #% #% #% Contact Us > Mail your payments address #% 0:00 # Find phone number for a specialized condition-assistance team #% #% #% #% #% #% Contact Us > Call patient services – Condition-specific assistance #% 0:00 # Reach out to the Digital Support Team to get help with the website #% #% #% #% #% #% Contact Us > Call patient services – Digital Support Team #% 0:00 # Find the information for sending Accredo Specialty Pharmacy a fax #% #% #% #% #% #% Contact Us > Send us a fax – General Patient Fax #% 0:00 # Send Accredo Specialty Pharmacy an email from the website #% #% #% #% #% #% Contact Us > Send us a secure online message – Email us #% 0:00 # T r e e T e s t F i n di n gs
  • 54.
    T r ee T e s t F i n di n gs Tasks Overall Success Success First Attempt Success Multiple Attempts Non Success I don't know Percent Time Percent Time Percent Time Percent Time Percent Time Sign up for Accredo Specialty Pharmacy #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Reorder a drug your doctor prescribed you #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Check where your drug is in the fulfillment and delivery process #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Find information about the process for getting set up with Accredo Specialty Pharmacy #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Learn about the purpose of a specialty pharmacy #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Learn how to order your drug with Accredo Specialty Pharmacy as a new patient #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Learn how to order your drug with Accredo Specialty Pharmacy as a returning patient #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Find general information about Bladder Cancer #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Find out the signs of having Epilepsy #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Learn about drug treatments associated with Hemophilia B #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Learn more about manufacturer and community programs that help reduce the drug costs for Growth Hormone Replacement #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Learn more about Leukemia assistance programs and advocacy organizations #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Learn more about Hepatitis C's Accredo care team #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Find a drug by it's manufacturer's name #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Find number to call to ask about a balance on your account #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Find the mailing address to send in money to pay off an account balance #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Find phone number for a specialized condition-assistance team #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Reach out to the Digital Support Team to get help with the website #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Find the information for sending Accredo Specialty Pharmacy a fax #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Send Accredo Specialty Pharmacy an email from the website #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00
  • 55.
    T r ee T e s t F i n di n gs Tasks Overall Success Success First Attempt Success Multiple Attempts Non Success I don't know Percent Time Percent Time Percent Time Percent Time Percent Time Sign up for Accredo Specialty Pharmacy #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Reorder a drug your doctor prescribed you #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Check where your drug is in the fulfillment and delivery process #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Find information about the process for getting set up with Accredo Specialty Pharmacy #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Learn about the purpose of a specialty pharmacy #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Learn how to order your drug with Accredo Specialty Pharmacy as a new patient #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Learn how to order your drug with Accredo Specialty Pharmacy as a returning patient #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Find general information about Bladder Cancer #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Find out the signs of having Epilepsy #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Learn about drug treatments associated with Hemophilia B #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Learn more about manufacturer and community programs that help reduce the drug costs for Growth Hormone Replacement #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Learn more about Leukemia assistance programs and advocacy organizations #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Learn more about Hepatitis C's Accredo care team #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Find a drug by it's manufacturer's name #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Find number to call to ask about a balance on your account #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Find the mailing address to send in money to pay off an account balance #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Find phone number for a specialized condition-assistance team #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Reach out to the Digital Support Team to get help with the website #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Find the information for sending Accredo Specialty Pharmacy a fax #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Send Accredo Specialty Pharmacy an email from the website #% 0:00 #% 0:00 #% 0:00 #% 0:00 #% 0:00 Finding: “Check where your drug is in the fulfillment and delivery process” (Order Status) placement in the card sorts did not match our hypothesis. Participants also looked in a variety of categories trying to find this item in the tree test which made for longer search times. Where would they look on a homepage image? Time for the click test.
  • 56.
    C l ick Test Setup The setup for the click test activity needed to align with the tree test so the same tasks were used. 1. Select items that may be difficult to sort, items you want more data on, or items we had additional questions about. 2. Selected easy and hard items based on location in the tree. 3. Create tasks for those items.
  • 57.
    C l ick Test • Participants were presented with static image of home screen • Participants then were asked to perform tasks and click where they think they would find information
  • 58.
    C l ick Test Fi ndi ngs “Check where your drug is in the fulfillment and delivery process” (Order Status) placement did not match IA hypothesis. Participants also looked in a variety of categories trying to find this item in the tree test which made for longer search times. Where would they look on a homepage image?
  • 59.
  • 60.
    An a lysi s Appr o a c h Step 01 Open Card Sort Tree Test Step 03A Click Test Step 03BStep 02 Closed Card Sort To recap how we approached analysis…
  • 61.
    An a lysi s Appr o a c h Step 01 Open Card Sort Tree Test Step 03A Click Test Step 03B • Examine the groups • Analyze card placement • Analyze labels • Analyze organizational schemes • Analyze how accurately participants have grouped content • Analyze participant comments • Analyze a closed card sort • Look at items classified as difficult to sort or belonging in multiple categories Step 02 Closed Card Sort To recap how we approached analysis…
  • 62.
    An a lysi s Appr o a c h Step 01 Open Card Sort Click Test Step 03B • Analyze success rate • Analyze directness of success • Analyze time on task • Look at card sort items classified as difficult to sort or belonging in multiple categories – how do participants approach looking for those items? • Look at runts or categories with two items – are they more easily found in the tree? Where are users looking to find them? Step 02 Closed Card Sort Tree Test Step 03A To recap how we approached analysis…
  • 63.
    An a lysi s Appr o a c h Step 01 Open Card Sort Click Test Step 03BStep 02 Closed Card Sort Tree Test Step 03A • Compare where users drill into the tree v. where they click into the UI • Look at initial entry points for the tree and first click data for the click test – are there similarities? • Note if success changes from the tree test to the click test To recap how we approached analysis…
  • 64.
    T i psfo r M i x e d M e th o d IA Stu dy Se tu p • Select items and tasks that are crucial or controversial • Ordering of method matters • Be consistent across the study • Participant fatigue is a thing –scope your work!
  • 65.
    T i psfo r M i x e d M e th o d IA An a l y s i s 1. Determine if new categories should be created or consolidated using the mental model put forth in open card sort 2. Look to confirm or deny hypotheses about categories using the closed card sort 3. See what issues of findability there are in the tree test 4. Note where people would place items in the UI in the click study
  • 66.
    • Allows fordata comparisons – useful in understanding convergence and divergence of findings • Collects rich, comprehensive data – provides a more complete story than a single method would alone • Combining two or more of these methods worth the cost. • Time consuming to create • Participants can easily grow tired • Data might be messy • Can be difficult to successfully integrate data Some challenges with mixed method IA validation… Adva n ta ge sC hal l enges Some advantages of mixed method IA validation…
  • 67.
    C oncl usion By spending the time constructing a more robust study design, you can create a holistic view of how the user is thinking and behaving toward your IA.
  • 68.
    www.nicecompany.comCOMPANY NAME GET INT OU C H cegarstka@express-scripts.com caron.garstka@gmail.com E-mail Thanks!
  • 69.