Bose Design Challenge: How might sound-based augmented reality improve our lives?

By Jessie Ying

From Pokémon Go to the Ikea Place app, Augmented Reality (AR) technology has made a firm place in all of our lives. But all of these applications are in the visual aspect. What about augmenting sound into the real world? On Tuesday, October 16, 2018, 45 Berkeley students took on a two-week design challenge to come up with ideas of using sound-based AR to improve our lives.

This challenge was a partnership between Bose, the Fung Institute for Engineering Leadership and Real Industry, a non-profit that creates programs for university students to work with industry mentors on real-world problems. Students formed five-person interdisciplinary teams that encompassed both undergraduate and graduate students from a wide range of majors. Engineers, artists, psychologists, designers and business leaders came together to imagine the possibilities that superimposing sound on top of the real world would create. The teams received Bose headphones and wearable speakers to work with. After two weeks of effort, nine teams submitted their design ideas and the Nod ‘n Talk team was declared the winner. There were five students on the team: Jovin Foo (MEng ME), Beth Lu (Undergrad ME), Kristjan Theodor Sigurdsson (MEng ME), John Young (MEng ME) and Zining Wang (MEng CEE). When brainstorming for ideas, they focused on issues people encounter on the move because they wanted to take advantage of the wireless character of the Bose hardware.

“We, especially as college students, understand the struggles of living in an active and fast-paced world where everyone is constantly trying to make the most use of their time,” Beth said.

They quickly decided to focus on the issue of communication.

“There were many times where we would see someone and want to say hi or ask a quick question. But we are either too far apart or it’d be inconvenient to go up to them, like when we are across a busy street or in a crowded hallway,” Zining shared.

The noise cancelling feature on Bose headphones made it even harder for users to hear each other. Therefore, the team decided to use AR technology to turn headphones from something that might impede communication into a powerful tool that facilitates it. Just like their team name suggested, their idea was: Bose users can start a phone conversation by simply nodding to each other, without having to touch their phones or headsets at all. They designed a demo iOS app that would gather users’ phone numbers and connect users if they are in close proximity. Sensing a nod from the user, a call will be automatically made to the other user and will be dropped if the user shakes his or her head. In the future, they hope to use GPS data from users’ phones and motion data from sensors on the headsets to determine whether to initiate a call based on if users are facing someone within their Bose social network and if they nod in the direction of that person. They also envision making the call through VoIP (Voice over Internet Protocol) rather than cellular.

Behind this simple design was the team’s relentless effort to overcome obstacles. The biggest challenge for them was that they were restricted to the development of an iPhone application. None of them had any programming experience for iOS and they only had one Macbook to run XCode, the integrated development environment (IDE) needed to build iPhone applications. In the two weeks of the design challenge, they had to learn the programming language for making iPhone apps while coordinating the only Macbook among the five of them. As the deadline loomed, they were skeptical about finishing their solution. At one point, they weren’t even sure if they would submit anything. But eventually, they pulled through. “We were able to take advantage of online resources and the resources Bose offered us. With a lot of hours and forum scrolling, we were able to successfully create a working prototype,” Beth said.

Kristjan gave his takeaway: “Even when things look pretty bad, pull through and don’t give up. Use it as motivation to work even harder.”

The other teams that made it into the top three also presented their ideas at the wrap-up event. The Soundscape team designed a music therapy app that would automatically play music when users go past designated areas, which is especially helpful for people suffering from dementia and Alzheimer’s. The Marco Polo team designed a tool that could guide Bose users through space using ambisonics (full-sphere surround sound) which is particularly useful for the visually-impaired. All three teams received Bose hardware as prizes. The Nod ‘n Talk team will also get Bose AR glasses once they are released in January next year and have the opportunity to present their idea to the Bose leadership. They hope to continue with the idea and see how far they can take it.

We asked the students how they felt about the challenge:

Kristjan: One thing I really liked about the challenge was the opportunity to practice my presentation skills in the final event. Pitching to people in the industry is different than in the safe setting in the classroom. It was a good experience.

Zining: It was really cool to see what we can accomplish once we put our minds to it. It was definitely a great experience and I’m very glad to have participated in this challenge, especially with this team.

Jovin: I think I definitely learned a lot with and from my group mates, such opportunities are great cause there is nothing to lose (maybe except sleep) but so much to gain (not only in terms of prizes, but also learning experiences.)

Beth: This was a great opportunity to be introduced to the industry, but what made the experience so valuable was being able to work with a team of incredibly hardworking individuals. They pushed me to be resilient, to think outside of the box, to put in the extra hours, and to also take a break. Working in this team, on this project, has taught me more than I can in class, so I am incredibly grateful.

John: It felt amazing. It’s a great opportunity to work among the brilliant minds here at Berkeley. I’ve only been here a few months, and I can already tell how much the Berkeley staff care about their engineering students. I am grateful to have the chance to gain exposure to design challenges like these, and it feels great to be able to have some success.

---

Augmenting sound into reality: A Bose design challenge was originally published in Berkeley Master of Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.