The Effectiveness of Virtual Achievement Feature in Online Shopping Process
Lead UX Researcher
Collaborated with Lingcan Ding, Jingwen Sun, Ryan Lamarche
October-December 2019, 2 months
This project is exploring the effectiveness of virtual achievement system in E-commerce website. We designed a task-based in-lab experiment. The task is to add items in cart on website with/without virtual achievement system.
We didn't find significant difference between websites with and without virtual achievement system. Read the final report here.
As e-commerce growing more and more rapidly nowadays, online shopping websites face a bottleneck where their sales barely growing anymore. With thousands of competitors out there in the market, not making improvement is dangerous.
There should be a new method implemented to increase customers’ experience, further to boost or grow the sales.
To figure out a more specific path to make the improvement, we asked five whys that lead us to form a deeper understanding of the problem and help us identifying the pain points.
According to this, we realized that the reason why online sale has entered a bottleneck is that there doesn't have an effective trigger that drives customer behavior.
Therefore, to make up for the missing trigger is exactly the problem we need to solve.
Generally speaking, we didn't find significant difference between websites with and without virtual achievement system, users don't show preference on website with virtual achievement system. Compared with virtual achievement, they care more about actual benefits they can get from e-commerce website. Please find the full data analysis here.
The following are key numbers of the UX experiment:
We got the inspiration from Hook Model, and combined the model with current E-commerce dilemma.
Objective: Identify the UX research problem we want to explore
Detail: Pre-stage research, Affinity Diagram, Brainstorming, Proto-Persona,
- How to Build Habit-Forming Products -
(Hook Model by Nir Eyal)
Brainstorming: The most optimal scenario is that we could figure out an owned trigger, which should serve as a specific element that can be embedded in the website.
In this case, we chose to create our own online virtual achievement system as the Trigger during our UX Research process
for the following reasons:
Easy to develop and to add as add-ons on Webpage. No extra cost for CRM, unlike traditional loyal programs.
Interesting function to interact with during online shopping process, may facilitate and generate higher volume per order.
Achievement system combine with gaming feature is not pervasive online yet. It serves as a medium to help marketers understand more about consumer behaviors.
Proto-persona: According our pre-stage research, we developed our intuitive proto-persona.
User Journey: We crafted the current user journey in online shopping process to uncover touch points.
Participants: We recruited 12 participants with a wide variety demographic.
We planned to recruit 12 participants whose ages are from 22 to 30 in total and we decided to do A/B testing with 6 participants for each prototype. We would also require pre and post experiment questionnaires. As for gender, even though the lipstick webpage would be more attractive by female customers, we still think male opinions are equally important. After all, we are testing the virtual achievement system itself not the product category. Therefore, we planned to recruit 5 male participants and 7 female participants. We mainly used snowball sampling to recruit participants.
Presentation: "This research is with the strong spirit of exploration".
Task Flow: Before sketching, we mapped out the task flow with and without virtual achievement system to address the difference.
Objective: Design an accessible approach that can research the problem
Detail: Task Flow, Sketch, Build Prototype
Sketch & Lo-fi: Ideated our design idea from sketch to low-fidelity prototype
Objective: Design specific procedure and measurements
Detail: Procedure, Measurements
Procedure: Our team developed the unanimous criteria to conduct a usability test and collect metrics. The experiment was designed into two independent tests: one participant only interacts with one prototype without knowing the existence of the other one. We assigned the task is to add items to cart.
The following flow shows our lab experiment procedure:
Measurements: We decided to measure our experience from two perspectives: Subjective and Objective. We collected data from pre-test questionnaire and post-test survey, including any comments, feedback, motion, or quotes during the test.
Satisfaction Rate: We designed two 5-scale questions of willingness of continue using the tested approach in the future and how satisfied are toward the tested prototype. Those two questions worth 10 points in total, which defines the range for the overall satisfaction rate.
Think out loud Protocol: We encourage users to speak out their thoughts and emotions during the process. We noted keywords and quoted their comments in our report.
Task Completion Time: This is the physical time elapsing recorded from the beginning to the time when participants finish adding their first three items to the cart.
Validate Time Range: This is a time indicator that evaluates usability of experiment result. Specifically, interviewees’ task completion time mostly distributes within a specific time range. If someone’s task completion time went too far beyond the normal range, we consider this participant’s result as unusable.
Laboratory Experiment Process:
Study Introduction --> Consent Form --> Pre-test Form --> Experiment --> Post-test Form
Objective: Conduct lab research with participants and collect insights
Detail: User Experiment, Mind Mapping, Storyboard, Empathy Map, Synthesized Insights
Example of Finished Forms
Example of Note-taking
Mind Mapping: Based on Note-taking, we created the mind mapping of our participants.
Storyboard: Exploring different scenarios to bring us closer to our users.
Empathy Map: We grouped what did users say, think, do, and feel during the experiments.
Objective: Translate finds to data and deliver findings
Detail: Data Analysis, Report, Presentation
Data Analysis: We conducted statistical analysis of the collected metrics. The data come from mainly in two perspectives, the one is subjective preference to virtual achievement system. The other is objective number of products in shopping cart. The results are shown below:
No significant difference between two prototypes
About how many products did participants add into their shopping cart, there is no significant difference. They answered in the post survey that they care more about product itself, the virtual achievement feature wouldn't influence their choices.
Users show interest in virtual achievement system
Even the virtual achievement feature didn't alter
the number of product they added in the shopping
cart, but most participants showed great interest in
it because they thought the design was fun.
Use easy understanding words. In this research, we wrote add 3 items on the pop-up window, we were intended to incentivize participants to add 3 items to unlock trophy. But in the interview after experiment, some users said they though the 3 items is a limitation, therefore, they considered more when adding products to cart. Understandable words are essential during the research and design process.
Highlight the difference. In the group who are testing with virtual achievement feature, some users said they even didn't notice the pop-up window because it stayed too short. if the pop-up window showed longer time, the result might be different.
Users words and users actions sometimes are inverse. Most users showed interest in virtual achievement feature, but their behaviors didn't change because of this feature. In another word, only testing their actions works, don't believe what they said.
Continue research. According to this research problem, we found that users behaviors didn't change significantly because of embedded virtual achievement system. But all of them indicated that if this feature can work together with some monetary promotion, they were willing to complete the achievement. That might manifest two directions of virtual achievement system, the one is continuing research this feature combined with some discount, sample, or promo code on E-commerce website. The other is that we can applied the virtual achievement system on some "virtual website", like e-learning website to stimulate students to unlock achievement.