INTRODUCTION

User research allows us to interact with key users of a product or process, and shift our design thinking to solve problems that directly affect core customers. Often times, users help us highlight issues and potential fixes, that we otherwise couldn’t determine. The website being assessed in the following research is papajohns.com, and focused in on the usability of the online ordering process, the ability to register for deals by email, and the availability of access to corporate customer service.

METHODOLOGY

ChadCo. Studio scheduled and moderated in-person research sessions. Four users were selected based on screening criteria, and invited to participate in this portion of the study. Observation of three predetermined tasks were coupled with contextual and post-task interview questions, allowing us to understand the deeper “whys” of a user’s actions and comprehension.

Interview Protocol

We asked participants to complete three tasks, focusing on submitting an online order, signing up for email offers and promotions, and contacting customer support. We also asked participants a series of organically derived questions to gauge their reactions to the experiences they just completed.

The sessions were recorded and uploaded to our proprietary video data center for reference, and future viewing opportunities, by both researcher and the Papa John’s organization. Each session lasted roughly 30 minutes, with buffer for extended conversation where necessary. Task analysis and interviews took place at ChadCo. Studio’s Design Research and Usability Lab.

The researcher followed the attached predetermined script for Ordering Pizza Online, but may have deviate from the script, in order to appropriately ask or address any clarifying questions.

• • Participants were offered water or coffee to drink.
  • Each participant was comfortably sat at a table, with a computer in front of them.
  • The computer had a web browser open with http://www.google.com and http://www.papajohns.com open to the homepage, on separate tabs.
  • The researcher also sat in a position to observe tasks, take notes, ask follow up questions, and  record the session with documented permission from the participant.
  • The researcher took notes on items of concerning, interesting, or surprising nature.

Analysis Protocol

For the Task Analysis portion of the research, we observed users’ ability to complete the entire task from start to finish. This information was coupled in a table including analysis on the user’s ability to complete the requested task, and any quotes or sentiments related to the task. Pertinent notes have been paired to the appropriate task and time within the table.

To optimize the data collected during our usability test, ChadCo. Studio performed an exercise known as affinity mapping to derive our key insights, and created an informed action plan for our design recommendations. Below you will find a description and visualization of that process, and how we developed the items we are highlighting to you (fig. 1).

Affinity Mapping Process (fig. 1):

Tools

The data collection instruments we used for this research method included:

• • Video Recordings of Interviews
  • Task Analysis & Interview Questions Moderator’s Guide
  • Moderator Notes
  • Timing and Sentiment Notation for Charting Task Time on Spreadsheet

Participant Information

Participants of this study were considered to be more advanced users, and had previous experience ordering pizza online. These users were selected through screening criteria to fit both demographic and business needs. Four participants from across the U.S. were selected for partnership in this research effort. Participants were compensated for their time and input in accordance with industry standards and procedures.

List of Tasks

The research plan consisted of three main tasks, preceded by an introduction and an “ice-breaker” task that eased each user into the more difficult tasks.

Those tasks were as follows:

• • Task 1: Create an Order for 3 Pizzas: Mushroom & Pepperoni, Half Onion & Half Sausage with Light Sauce, and A Specialty Pizza
  • Task 2: Sign Up for Deals & Coupons Using Only Your Email Address
  • Task 3: You Have a Poor Service Experience, and Wish to the Contact Corporate Office

The full moderator’s guide, with suggested language and the entirety of tasks, is available in the appendix of this report.

FINDINGS & RECOMMENDATIONS

From our affinity mapping, we have derived four key insights and corresponding recommendations that we believe would facilitate a better experience for Papa John’s users. We will break these insights and recommendations down into the various tasks that were observed, along with some additional findings that were highlighted within each of the various tasks. Each of these smaller findings have been listed within their parent insights below.

Task 1 – Create an Order for 3 Pizzas: Mushroom & Pepperoni, Half Onion & Half Sausage with Light Sauce, and a Specialty Pizza.

• Insight 1 – Ordering pizza is a non-memorable experience:

Placing an online order for pizza is not viewed as a difficult or time-consuming process. The mental model for users is that ordering online saves time by not having to call, and perhaps not having to even leave your location, in the case of delivery. This was observed in the fact that even though all observed users had at least some difficulty within this initial task, they each stated that the process was either: “simple,”  “pretty standard,”  “clear cut,” or “easy,” once their order had been placed.

Unexpected moments, also known as delighters (a-la-pizza animation) are highly effective, and can assist in “righting” any “wrongs” that may have been missed in the experience. In Video 1, the user was clearly delighted by the animation and in Video 4, the participant mentions, “I like the graphic, I want to do that again!”  Our recommendation is to look for additional opportunities to surprise and delight a user throughout each experience and process flow within the site.

• Insight 2 – In the absence of clear signals, users determine their own approach, and remain loyal to their initial process.

Users approached this initial task in various ways, all resulting in a satisfactory outcome and all were able to complete the task as presented.

Several users had difficulty with various terminology used within the order process. (This was also observed in the subsequent tasks.) Specifically, the terms “normal cut” vs. “clean cut” and “base” gave users pause when customizing their pizzas, as shown below.

Confusing Terminology in Order Process (fig. 2 – Customize Pizza Page):

Some users had difficulty locating the option to add an ingredient to only one side of a pizza. One participant even stated, “I would never have guessed that,” in reference to the half-pizza option displaying after adding sausage to their pie. Our recommendation is to consider displaying this feature sooner in the customization flow, alongside the listed ingredients.

Regardless of the process chosen for adding an initial pizza to the order, all users returned to the same process for adding their second pizza. This was a clear and observable phenomena within Task 1. Our recommendation would be to clarify, but keep, the two potential paths (Customizing a Band New Pizza and customizing a Menu Listed Pizza), as users exhibit different mental models when customizing their order.

Task 2 – Sign Up for Deals and Coupons Using Only Your Email Address.

• Insight 3 – Personal info and time are highly valued by your user. Only require what is minimally necessary for any given process.

Users expect to receive deals and coupons in exchange for their personal information, however, these discounts are viewed as table-stakes when it comes to ordering pizza. All users expressed that they expected some type of coupon or deal, by giving their email address, but none of them found that to be especially appealing or differentiating from other pizza companies. We hypothesize that this stems from the fact that customers are used to easily finding and utilizing online coupons or promotion codes, and are also becoming more and more aware of the value of their personal data. Our recommendation would be to, again, seek opportunities to delight users in larger ways than they are expecting. A practical example of this might be to give users a free pizza, just for entering their email address.

Furthermore, the majority of users (all but the one who happened to see it right in front of them at the time of presenting) had difficulty completing this task. Users were observed first checking under Papa Rewards, then upon realizing they were unable to complete the task, returning to the homepage to continue their search. Users who were successfully able to locate Text & Email Offers, even stated “That was hard to find,” and mentioned that they still weren’t sure what pieces of their information were considered mandatory. Our recommendation is to call out what is minimally required and eliminate confusion between these types of offers and the Papa Rewards program. Additional consideration may be taken to combine the programs where possible and appropriate.

Similar Programs Led to Confusion (fig. 3 – Papa Rewards vs. Text & Email Offers):

Task 3 – You Have a Poor Service Experience, and Wish to the Contact Corporate Office.

• Insight 4 – Users Follow Standard Web Patterns. Use this to your advantage.

All users were seen following standard web patterns for Contact Information and are trained to look at the footer/bottom of a web page. Each participant followed the same pattern. Scroll to the Bottom of the Page > Click on Customer Service > Click on Feedback Form. There is opportunity to use this to your advantage by following standard web practices in other ways. An area where this becomes an issue, is using common web patterns (such as terminology and visual cues) in an inappropriate way. An example of this is the location of the Menu button, as seen in fig. 4 below. In its current placement, the button could be mistaken for a navigation menu, as was observed in Video 4, where the participant is seen clicking the menu to find her way through the site, and even states, “Nope, that’s not it.”

Menu Button (fig. 4 – Papa John’s Homepage):

Insights 2 & 3 were also derived from observances in Task 3.

Insight 2 (In the absence of clear signals, users determine their own approach, and remain loyal to their initial process.) was reflected, in that, some users were again confused by terminology “Feedback Form” vs. “Click here to provide feedback.” In Video 1, the participant states “I wonder if that takes me to the same place?” (fig. 5) Other users were confused by a shift in design style (fig. 6); one user even mentioning “this looks more like a survey.” Again, our recommendation is to simplify all terminology used and remove redundant entry points. Additionally, we recommend aligning design language across all experiences, even when different platforms or services are being utilized.

Similarities In Feedback Options (fig. 5 – https://www.papajohns.com/contact-us/):

Change in Styling (fig. 6 – https://contactus.papatalksurvey.smg.com/):

Insight 3 (Personal info and time are highly valued by your user. Only require what is minimally necessary for any given process.) was reflected, in that, some users mentioned that the amount of work required to contact the corporate office with their customer service complaint, didn’t seem proportional to the payoff/outcome they expected to receive in return. Users also mentioned not even being sure that this would get to corporate at all, and may be handled by someone else. When presented with the screen in fig. 7, a user stated “I’m only at 30%? How much more could there be?!” We would again recommend simplifying the terms used to contact corporate customer care, and the process required to do so. Further considerations may be taken to offer additional methods in which a customer might contact the corporate office (to include online chat  or a direct phone number) and phrasing used to encourage and console the user along the customer care experience flow.

Work Required for Customer Care (fig. 7 – https://contactus.papatalksurvey.smg.com/Survey.aspx?c=520921):

FOLLOW-UP RESEARCH QUESTIONS

Based on the derived insights and observable behaviors during usability testing, ChadCo. Studio recommends the following additional research questions be considered for continuous improvement to papajohns.com:

• • Understanding Terminology and Definitions – Audit papajohns.com for terms that stand out to users as unclear. Additional efforts may be needed to identify and solution for terms in addition to those mentioned in this research report. Simplifying and universalizing phrasing can positively impact the user experience by adding clarity and ease-of-use, which reduces friction in time-to-order and the necessity for involving customer care.
  • Identify Standard Web Practices – Users follow established patterns. Knowing and utilizing these patters across papajohns.com can lead to increased ease-of-use and a sense of familiarity. Additional research into determining these standards and practices, and how they might be used is recommended at this time.
  • Delighter Expansion Opportunity – Unexpected surprises are highly effective, and can lead to more memorable experiences. Understanding the threshold between what delighters are most valued, where they come into play, and what amount may be too many, are all categories that require further research and exploration.

RESEARCH DISCLAIMER

The data collected and presented is qualitative in nature and is not statistically projectable across the Papa John’s user population. Findings should be considered directional and indicative of user attitudes, perceptions and motivations. Business decisions requiring statistical validation should be supported by supplemental quantitative research; although findings of this research were consistent across participants, and with projected hypotheses.

APPENDIX

Usability Test Script & Tasks (PDF)
User Sentiment Spreadsheet (PDF)