What does Alexa know? What doesn’t she know? Why did Google Assistant share that answer? And most importantly, where is my brand in all of this?
As more consumer queries occur via Voice, these are the questions marketers are struggling to understand. Voice agents can feel like black boxes – murky, mysterious, and always changing. For those of us who’ve been around this industry, it’s similar to the early days of Search, when brands wanted to understand Yahoo, Google and Ask Jeeves (remember that one?).
VSM, “Voice Search Monitor” is the first-of-its-kind proprietary software that reverse engineers the algorithms behind voice technologies such as Alexa, Google Home and others to better understand how these intelligent agents are communicating back to consumers.
Our Voice Search Monitor (VSM) has been studying the responses on leading Voice assistant platforms for nearly a year, reverse engineering the algorithms behind voice technologies such as Alexa, Google Home and others to better understand how these intelligent agents are communicating back to consumers. We’ve posed to the platforms thousands of questions and commands consumers are asking of these devices around the clock. The questions range from the informational (How tall is Mount Everest?) to command-based (Buy paper towels), and VSM actively catalogues how each platform responds, tracking data sourcing and response rate.
Among the things we’re looking for:
- Did the assistant have an answer or not?
- Was the answer good, bad, or incomplete?
- How do the assistants respond to commands vs. informational questions?
- Does performance and relevance differ by topic or industry vertical?
- Does the language used to express the same intent create different results?
- How does personalization and history impact the experience and results?
- How does this change over time for the same questions?
Our findings – and the changes we’ve seen over time – demonstrate the importance of monitoring and learning about what each intelligent assistant does or doesn’t know.
We’ve learned that Google knows a lot more than Alexa—five times more on average. Google Assistant prefers to draw from location-based data for retail queries, whereas Alexa relies on her own top-matching product recommendations. And the most popular brand for responses in the automotive field is Cars.com. In Voice, sometimes it pays to be obvious.
Our research to date has covered four industry verticals: Travel, Finance, Retail, Automotive. Here’s an overview of VSM’s primary findings:
Travel
Questions asked: 5,000
Top brand: Royal Caribbean
Google Assistant answered (April 2017): 72% of questions
Amazon Alexa answered (April 2017): 13.3% of questions
Top travel sources mentioned in Google and Amazon responses:
- USAToday (489 mentions)
- Cruise Critic (181 mentions)
- U.S. Department of State (146)
- Wikipedia (135 mentions)
- Passports and Visas (68 mentions)
We re-asked the same question set seven months later and both agents answered 8% more questions from the original test.
Finance
Questions asked: 3,000
Top brand: Discover
Google Assistant answered: 68.9% of questions
Amazon Alexa answered: 13.6% of questions
Top finance sources mentioned in Google and Amazon responses:
- bankrate.com (99 mentions)
- Wikipedia (94 mentions)
- Investopedia (64 mentions)
- nerdwallet (48 mentions)
- Wiki how (38 mentions)
Automotive
Questions asked: 3,000
Top brand: PepBoys
Google Assistant answered: 67% of questions
Amazon Alexa answered: 22% of questions
Top finance sources mentioned in Google and Amazon responses:
- Cars.com (85 mentions)
- Angie’s List (85mentions)
- Your Mechanic (75 mentions)
- Wikihow (49 mentions)
- Wikipedia (48 mentions)
Retail
Questions asked: 5,000
Google Assistant answered: 72% of questions
Amazon Alexa answered: 53% of questions (clearly Amazon’s best vertical)
- Google Assistant was able to answer 67% of the retail questions Alexa couldn’t. To field those queries, Google relied on two primary data sources:
- Web results (46% of the time)
- Local places and listings (49% of the time)
- Alexa was able to answer 45% of the retail questions Google couldn’t, each time using Amazon’s top matching product recommendation.
- Google answers nearly 100% of “where” questions with a list of local stores and directed users to make an in-store purchase.
- Alexa meanwhile answered 61% of “where” questions with a product recommendation from Amazon.com that best matched the original query.
- Google tends to provide multiple matching results including source/store, location, product, size, price, shipping, while Alexa likes to provide only one matching result including source/vendor, location, product, and price.
- Google mentions other partner’s name/source/brand more often than Alexa.
- When the purchase intent is explicit (buy, order, purchase, product xyz), both devices answer these inventory specific questions 99% of the time.
- Overwhelmingly, devices do not have answers for questions that are not product specific.
For more on what we know about leading Voice technologies download 360i’s Voice Playbook.
