Moderated Usability Study
Website: papajohns.com (Kent State University project)
Role: Moderated Usability Study/Research
Timeline: 4 weeks
Tools: Reflector, Keynote
This short-term usability study project was team-based.
Summary of the Study
Project Goal:
The goal of this usability research study was to understand the ease or difficulty with which customers are able to access, use, find information on, and order from the Pap John’s Pizza website.
Project Logistics:
Each moderated research session was done one-on-one with one moderator and one user participant. The sessions were conducted in front of a computer screen and the moderator gave the participant tasks then watched as they completed them. The screen, audio, and video was recorded for each session.
Top Results & Implications:
All four of the study participants had ordered pizza online before, so it was expected that there would not be many problems or frustrations in regards to the act of ordering food online even if it was through a new site. This was proven correct as of the participants were able to place their orders without any frustration.
A few design recommendations were made by study participants for more intuitive experiences maneuvering through the ordering section of the site. However, little to no frustration was shown while placing orders by the participants through this study.
Customer service forms and contact information was easily accessed as well, though some issues came when the participants were tasked with signing up for specials and coupons emails without registering for an actual account through the site. All four participants had difficulty finding where to access the email sign-up for specials and coupons, and only three were able to accomplish this task after much scrolling.
Study Logistics
Methodology:
Research was done one-on-one with one moderator and one research participant. The study was conducted in front of a computer running screen capture and audio/video recording software. The environments for each of the four subjects seemed quiet and relaxed.
All four of the research participants were placed next to the moderator in front of a computer screen. Data was collected in two ways. Audio and video of each session in their entirety were recorded to aid in note-taking and follow-up observations done after the fact, and notes were taken throughout each of the four research session recordings.
Participant Information:
Three of the participants were male, one was female. Though specific ages were not known, all four participants seemed to be between the ages of 30–50. After reviewing the screen recordings for the four sessions, it was obvious that all participants were very comfortable with technology and computers is general, and had little to no frustration during the sessions. All four participants had ordered pizza online in the past.
Required Tasks:
The following three tasks were performed and assessed during the research session:
- There are a bunch of people at this party and you need to order 3 pizzas. They are: 1. Mushroom and pepperoni, 2. Half-onion and half-sausage with light sauce, and 3. A specialty pizza because you want some variety.
- You want to sign up for deals and coupons but you don’t want to register for an account. You just want to give them your email. Can you do this and what do you think you will receive by email?
- Your pizza arrives and it is terrible. Your driver was rude and you are really upset about what just happened. You call the local store but get nowhere. You need to contact the corporate office. How would you do that?
Expectations & Synopsis of Findings
In general, the expectations of this study were met. Since all four of our participants had ordered pizza online before, it was expected that there would not be many problems or frustrations in regards to the act of ordering food online even if it was through a new site. This was proven correct as of the participants were able to place their orders without any frustration.
Customer service forms and contact information was easily accessed as well, though some issues came when the participants were tasked with signing up for specials and coupons emails without registering for an actual account through the site. Specifics and more details can be found below in the task analysis section.
Task Analysis & Findings
Task 1:
Is the interface successful in helping people complete this task?
Yes. All three participants quickly ordered their three required pizzas with relative ease. Participant 3 began her order by scrolling down the landing page and finding the “create your own pizza” link. Participant 2 started his order through the ‘Menu’ link at the top, and the other two participants started their order by selecting the “start your order” button located in the top toolbar. Participant 4 said, “In looking at the site, right along the top in the banner it says start your order, so it’s very simple and easy. I don’t have to create a log-in because I wouldn’t want to do that.” Participant 1 agreed and said, “The start your order button is easily visible and right at the top.”
Within the ordering process a few minor annoyances became apparent. Participant 2 was frustrated that after selecting the ‘customize’ button to build his first pizza, he was then taken to a page where he was required to fill in his address information. He said, “I think it breaks the flow a little bit Because I come in and click into the menu and see what I expect — the options of what I want to select. Then I chose what I want but then it pulls me out of that experience to enter all of my information before putting me back into that experience. I would almost rather it ask me for my address before I even got into the menu. I would prefer them cutting out a step and just starting me with entering my information.”
When adding toppings to her first pizza, Participant 3 was initially thrown off when she reached the end of the scrolling list for the pizza “base” category. It took her a minute to see the category options at the top, though she did not voice any frustration or complaints.
Participant 2 had the same issue. He said that the process of ordering was “straight forward with an asterisk. Because it puts me here creating my own pizza and I’m modifying the base. But when I scroll down there aren’t any more options and entering in that first pizza I was a little confused until I looked back at the top to see additional selection options for cheese, meats, etc. I think even if there was something as small as having another navigation button down at the bottom of the base options saying ‘continue to cheeses’ I think that would have alleviated that minor confusion I had before I realized where I needed to go next.”
Both participants 1 and 4 really loved the graphics of the ordering site with Participant 4 saying “I really like how It has the 1/2 moons and it tells you which side you’re putting toppings on. That’s very clear and I like that.”
Did participants complete this task in a similar or different way?
Though the participants initially accessed the pizza ordering page through different links (as mentioned above), they all then completed this task of ordering the three pizzas in the same way.
Task 2:
Is the interface successful in helping people complete this task?
No, the interface is not clear enough for this task. All four participants had problems completing this task quickly and without frustration. Participant 1 had difficulty knowing if he was able to sign-up for deals without inputting all of his information. He thought he had the right sign-up link at first, but then wasn’t so sure. He looked around the site a bit more and tried to find a search window to aid in his tasks, but wasn’t able to find what he needed. Then, after a prompt from the moderator, he maneuvered back to his original finding and completed the task. Throughout the entire process though, he was unsure and questioning about every step.
Participant 2 never actually found the correct link to sign-up for deals with just an email address and instead, mistakenly signed up for an account. Participant 3 was able to find the correct link for specials sign-up relatively quickly by scrolling to the very bottom of the page and selecting the option to ‘sign-up for emails and texts’. However, she wasn’t 100% sure this was correct as she was afraid entering this email would somehow still sign her up for an account. Participant 4 wasn’t sure where to start. He looked around the page for a minute or so before selecting the ‘Papa Rewards’ link near the top as he thought that’s where this information should be. However, he could only find how to create an account by providing all of your information. Eventually, after a prompt from the moderator, he found he correct sign-up link at the bottom. This participant noted that the sign-up link for coupons was “very nondescript. They really want you to sign up and create an account. It’s not something I would normally be looking for but it’s obvious they want you to create an account - that’s what they really want you to do. This was not easy to find, it was all the way at the bottom so I don’t think this is what the website is pushing you towards.”
Did participants complete this task in a similar or different way?
The participants approached this task from different avenues though all of them did a fair amount of scrolling before finding their solution, 3 of which were correct.
Task 3:
Is the interface successful in helping people complete this task?
Yes, if they know where to look. When given this task, all four research participants immediately scrolled to the very bottom of the page that they were on and located the link to customer service. Participant 1 noted that he thought “most people would not automatically scroll to the bottom to fi nd customer service information” though it would seem that customers who are familiarity with computers and technology quickly do scroll to the bottom.
Did participants complete this task in a similar or different way?
Yes. All four of the participants completed this task in the same way.
Follow-up Research & Questions
Short-Term:
Are there small changes that can be made design-wise to expedite the ordering process? Specifically with a slightly more intuitive “build your custom pizza” ordering and selection form.
Long-Term:
- Additional research done specifically into the access of specials and coupons through the Papa John’s website. This would include questions such as: How quickly does it take a customer on the website to find and sign-up for coupon access?
- Additional research into the sign-up preferences of customers. This would include questions such as: Does signing up for an account before ordering cause more or less frustration than signing up after readying an order?
