My team and I worked with our client, Expedia, to improve their voice user interface (VUI) for travel. The VUI is a skill for Amazon Alexa which allows users to get information about their trips and ask about or search for flights, hotels, or car reservations through voice commands.
Understanding the Voice UI
To understand the Expedia VUI, we first developed an interaction flow diagram for its existing functions. We tested each possible command and examined the flow of the conversation. In some cases, the user reached a dead end or continuous loop, and they chose to end the interaction. The diagram displays three common statuses that a user might have when interacting with the Expedia skill: 1) not signed in to their Expedia account, 2) signed in, without upcoming trips, and 3) signed in, with upcoming trips.
We developed the following research questions to guide our study:
1. Does the Alexa Skill provide appropriate detail in its search results for users to make an informed booking? If not, why not?
2. Are users satisfied with the conversation flow?
3. What barriers do users encounter while communicating through the Expedia skill?
4. Does the voice system provide actionable error messages that are helpful to the user?
We recruited 8 fluent English speakers who were planning to travel within the next six months. All participants were familiar with voice UIs and owned a device such as the Google Home or Amazon Alexa in their home. By request of our client, we matched the gender balance to prior research for travel planning at Expedia, which demonstrated that more females tended to do the planning.
Participants were recruited for a 60-minute study. At the beginning of the study, we briefed the participant on the topic of the study and the types of tasks they would be asked to do. After obtaining consent, we began video recordings of the participant with the Alexa. Next, we provided a brief training to the Expedia Alexa skill by informing them of a few possible commands within the skill and by giving the participant a training task to test the skill. Participants were also given a list of possible commands as a reference.
Participants were then given written prompts for each task. They were asked to communicate with the Expedia Alexa skill until they believed they completed the task. If they became too frustrated with a task, they could choose not to complete it. At the end of each task, video recordings were played back to participants for a retrospective think-aloud session. We chose this method because we found during pilot tests that thinking aloud while using a voice UI could cancel or confuse a voice command. In addition to the retrospective think-aloud session, participants also answered post-task questionnaires and a post-test interview.
Participants were asked to complete the following tasks:
1. Review Reservation: Review the details of an existing trip reservation.
2. Book a Flight: Search for a one-way flight to a specified destination and book it.
3. Book a Hotel: Search for a hotel at a specific destination and book it.
4. Book a Car Reservation: Search for a rental car at a specific destination and book it.
We collected data from the usability sessions in three ways:
1. Metrics collected during tasks: These metrics focused on efficiency, effectiveness, and satisfaction. They included quantitative measurements that were primarily analyzed through descriptive statistics and content analysis.
2. Post-task questionnaires: These questionnaires focused on the challenges and high points of each task. They included Likert scale questions aimed to supplement findings from the metrics.
3. Post-test interview - The interviews were qualitative reflections about the participants' overall perception of the voice UI at the end of the study.
We reported the results of this study to our client at the Expedia headquarters. We hope that our findings were helpful for understanding how users interact with their voice UI and for understanding the potential of voice UIs for travel purposes.