Alexa Seems to Be Getting Worse With Time
I loved my Amazon Echo when they first came out. I love to cook, and I love to listen to music, so my Amazon Echo naturally won a spot in my kitchen when I first got it. Later on, it got upgraded to a Sonos One. Both speakers had voice control features that I loved, and used all the time. Hands-free timer setting and music control? Sign me up! However, Alexa’s shine has started to wear off - especially in the last six months. What’s changed?
Buggy Timers
Just in the last week or two, I’ve noticed a pretty serious bug in the Alexa software: its timers are not playing alarms when they expire! I can set a timer on my Sonos One using Alexa, open the Alexa app, and watch the timer count down to “00:00:00”. I expected hitting “00:00:00” to result in a tone from the speaker, like it’s done reliably for years. Instead: silence. As if to add insult to injury, the timer stays in the Alexa app, stuck forever at “00:00:00”. I’m not certain if this is a Sonos problem, or an Alexa problem - I see this behavior on my Sonos One, but not on my Sonos Move.
Whatever the case, it’s very literally burnt my biscuits. I do, primarily, use it as a kitchen timer, and it’s hard not to notice a bug that ruins your food!
Ignoring My Wife
Alexa unquestionably has a much lower success rate understanding my wife’s voice than mine. At first, I chalked this up to a matter of training, and understanding how the voice recognition technology works. I’ve participated in the development of a few voice speakers. I know that the best results come when you face the speaker, and speak slowly, clearly, and without pausing. Even with coaching to do this, Alexa either fails to understand my wife, or ignores her completely, at a rate of at least twice my own.
Unfortunately, I think I know the reason for this, and it’s not a pretty one. Voice assistant software naturally has a better success rate when tested against speech signals with lower frequency ranges. As a result, they naturally have better chance of understanding deeper voices, which predisposes them to understand men better than women. I thought this was just paranoid suspicion on my part, but I’m not the only one who has noticed this phenomenon. Not to mention that it’s hard to argue with a plot of Voice Command Success Rate against average input frequency. The trend just goes down, down, down as the average input frequency increases.
It’s a disappointing bias set by technology.
Perception, or Reality?
I also get the impression that the quality of voice interactions has gone downhill over time. No longer does Alexa magically find the thing I want to do - whether that’s set a timer, add an item to a grocery list, or start playing a particular album I want to hear. I find myself leaning, more and more, on my smartphone to queue up music. Alexa is just not up to the task on any given day.
I have to admit that this might not be a real problem - or, rather, not a change in the voice recognition software. The only thing I know for sure has changed in the last six months is that I’m home a lot more. Working from home has given me many more opportunities to use my voice speaker. I’d estimate that, pre-COVID, I used the voice control functionality maybe five times a day, in two discrete blocks: in the morning, before leaving for work, and in the evening, after getting back home. Working from home has blown that schedule out of the water. Instead of a two-hour window at the start and end of each day, I’m at home _all the time_. This gives me more opportunities to use my voice speaker. Correspondingly, it gives more opportunities for Alexa to fail to meet my expectations.
Having previously worked at a company that makes voice speakers, I understand that there’s such a thing as an acceptable failure rate of voice transactions. (The terms of art are “False Accepts” and “False Rejects”, if you’re curious.) This leads me to wonder: is Alexa’s voice assistant technology getting worse, or am I just using it enough to expose its warts?
For example: let’s say I use the speaker an average of once per hour. If I’m home for four hours a day (like I was pre-pandemic), that averages out to four voice commands per day. If a voice command fails, on average, one time in twenty, this is likely an acceptable failure rate. It’s easy to forget a failed voice command if it only happens, on average, once a week or so.
Now, though, I’m always home. That gives me sixteen waking hours in which to issue voice commands. If we’re holding constant the once-per-hour rate of voice commands, and the one-in-twenty rate of failed voice transactions, then failures of the voice assistant start becoming a daily occurrence, rather than a weekly one. I suspect that this passes some sort of psychological threshold: even though the overall failure rate has not changed, I notice more failures just because the time between failures decreases. Since it’s now a daily occurrence instead of a weekly one, I have less other minutiae going on in my life to help me forget Alexa’s shortcomings.
I’d imagine the folks at Amazon would consider this a pretty serious problem, even though their underlying metrics for Alexa’s accept/reject rates likely haven’t changed at all. Being a consumer good, perception of its quality is almost as important as the reality of it!