Get ready for Alexa to wave at you.
Well, not literally. Amazon hasn't built a humanoid smart home robot just yet, but the company's latest Echo Show 10 smart display is a small step in that direction. Equipped with a display that can rotate 360 degrees around its speaker base, the Echo Show 10 is the first device of its kind that can physically move. When you say "Alexa, goodbye," it not only responds with audible and visual feedback, but also performs a kind of sweeping motion with its display, resembling a slow farewell wave.
Originally conceived as a way to keep the display in view while you move around the kitchen, Amazon's hardware team has turned this motion feature into a novel way to interact with the device. For instance, if you profess your love to Alexa, the Echo Show 10 does a little happy dance, with the display bobbing back and forth. "It seemed appropriate for Alexa to look cheerful when you say you love her," Amazon's Alexa devices and developer technologies VP, Nedim Fresko, said.
For Fresko and his team, adding motion to the Echo Show 10 was just the latest step toward a more responsive and multimodal future of ambient computing. The company also opened the feature up to developers, and began to choreograph a set of predefined motions to make Alexa's skills more expressive. In an exclusive conversation with Protocol, Fresko explained why Amazon is betting on motion, how his team conceives new features for smart home devices and why Amazon executives regularly spend hours reading customer reviews.
Working backward from an idea
Out of all the things Amazon could have added to its smart displays, giving developers the ability to make it wave and shake may not have been the most obvious choice. Fresko said that it all started with an idea for solving a common customer problem with regular smart displays. "Every year, we have new ideas for working-backwards documents, as we call them, where we consider the customer need, and then we try to look at which technologies can achieve that customer need," he explained.
"Motion was described in exactly the same way a couple of years ago, where we described how it would behave before any hardware or software was built," Fresko said. "We turned that around until we really loved it. Then we built the technology."
That early idea was all about making screens a key part of the ambient computing experience. "Sometimes, when you're multitasking in the kitchen, or walking around the home, you have to face a choice: You're either facing the screen or you have to make do with voice," Fresko said. "So we thought: What if we could build a system where the screen always faces you? No matter where you are, you'll find the screen facing you."
A screen that moves with you, aided by the built-in camera and its ability to track your movement across the room, seemed like a good solution for cooking in the kitchen. It also lent itself naturally to Amazon's push into video chat. "Person-to-person communication, where you're always in the frame no matter where you are, was kind of consistent with the overall vision," Fresko said.
Opening up the feature to developers
Once Amazon's hardware team began working on the technology, it had to keep it under the lid for a while. "It's hard to go out there and do a survey about motion," he said. "That kind of thing is an internal idea." However, the company did reach out to a few trusted developers before it officially announced the Echo Show 10 with motion last October. "If we find that the experience is useful for consumers, we automatically assume that developers will also find it useful for similar reasons," Fresko said.
Some of the third-party developers to make use of motion include Comedy Central, Sony and Universal Games. One of the few smaller companies that got early access is Volley, a San Francisco-based startup that builds casual games for smart speakers. "It's fun," Volley CEO Max Child said. "It's like a cute little robot from a movie."
Volley has added the Echo 10's ability to have the display follow a user to its Song Quiz Alexa skill, and Child is already thinking about additional applications that could add to the company's multiplayer games. "Secret information is interesting, and also just cues around whose turn it is, or who should be engaging, that kind of thing," he said.
In addition to giving developers access to the basic motion that keeps the display in view, Amazon has also begun to release a few choreographed motions that can be used to make skills more dynamic. "We added a few preprogrammed choreos to express emotion, and we also added something called 'entity sensing' that allows you to act as input yourself," Fresko said. "As you move, the UI could react with updates based on your motion."
"We'll add more of these personality and reaction features," Child said. "There's a lot of opportunity to make these devices more friendly and more endearing."
On Wednesday, Amazon released three new such "choreos," including one aptly called "MildExpressiveShakes." To develop these and other choreographed motions, Amazon's designers spitballed a bunch of different ideas on what Fresko described as a "mild to wild" scale. "They basically disregarded all constraints and tried to see what would feel delightful, what would evoke emotion," he said. "Then they thought about what experiences they could attach those motions to, and eventually narrowed the focus to a few that we felt were in good taste, delightful, useful."
As part of that process, they also had to throw out a bunch of different motions that were simply too exhausting or dizzying. "It's very easy to go overboard with these things," Fresko said. "Sometimes, you just have to restrain yourself."
Getting it wrong to ultimately get it right
It's too early to say whether consumers are willing to pay a premium for motion. The Echo Show 10 became available for $250 at the end of February; early reviews on Amazon.com are generally positive, but some reviewers have taken issue with the feature. "The first thing I turned off was motion," one customer noted, who complained about a lack of accuracy when the display follows you. A few others had similar gripes, with another consumer expressing hope that things could improve: "The follow-mode feels more gimmicky than useful right now but that could change as they update."
"Once we launch, our customers publicly grade us," Fresko said. "Very publicly grade us, and tell us exactly what they like and don't like." Amazon keeps close tabs on those comments, and has developed automated ways to pay attention to thousands of reviews. But Fresko is also going through them manually. "In the few months that are after a device's launch, I read every single review, without exception," he said. "They're very interesting. We're very grateful to our customers for being frank about that feedback."
One reason these reviews are so valuable is that ambient computing is still so novel. The first Echo smart speaker was released in 2014, and the company's first smart display only became available in 2017. Both products ultimately laid the foundation for entire product categories, and turned Amazon into the market leader in the smart speaker space. But they began as unproven bets, just as the Echo Show 10's motion feature.
"We keep on placing bets in areas where we have a really good theory that customers will find it useful," Fresko said. "But there's only one way to find out, which is to build it."
Some bets ultimately don't pan out, as Fresko knows all too well. Before working on Alexa devices, he was director of engineering for Amazon's Fire phone. Just like the Echo Show 10, that device had a very unique and interesting feature that set it apart from the competition: a 3D display that gave consumers a perception of depth by tracking their viewing angle. The device nonetheless failed miserably, and Amazon pulled the plug on the project after just one year.
"You imagine customer utility, you write about it, you convince yourself, and then you realize the only way to find out if it's going to work is to actually build it," Fresko said. "Sometimes there is awesome customer interest, and sometimes there isn't. But there's no way to find out unless you try. So I think of the phone as exactly that. There was a very good theory behind its utility. It didn't find the reception we expected, and therefore we moved to the next thing. And guess what? The next thing was incredibly successful."
