Google debuts new Android apps for the hard of hearing

Over the past few years, Google has created multiple applications with accessibility in mind. Yet, most of these apps have focused on users who have physical or visual impairments. Examples include last year's Voice Access and Lookout apps. Now, the company wants to provide better support for users who are deaf or hard of hearing as well. As part of this effort, Google is rolling out two new apps -- Live Transcribe and Sound Amplifier -- starting today.

Live Transcribe is fairly self-explanatory. The interface is very simple -- as soon as the app launches, it'll begin transcribing, with large white text on a black background. You can also flip this to black text on a white background by disabling the Dark Theme and the font size can be adjusted too.

There are a few extra features in the app that make it easier for those who are hard-of-hearing to use. For example, there's haptic feedback so that if someone's speaking you from behind, the phone will vibrate to alert you that someone's talking to you. There's also a loudness indicator in the upper right corner, with an inner and outer circle. If the inner circle gets really big, that means you're in a particularly noisy environment, informing the user that the other person will have to speak louder in order for the app to work. It will also let the deaf person know that they'll have to speak louder to be heard.

In the event that the app doesn't work at all, due to how loud the environment is, there's a keyboard shortcut at the bottom that lets you type your message and show the app to the person you're talking to. "I liken it to a communication app," said Brian Kemler, a product manager for Android Accessibility, in an interview with the press. "It's not just a live transcription service." That's why, for example, the app doesn't record or store your transcripts, so you can't really use it as a journalist or a student might.

"The soul and inspiration of this product is for the deaf and hard of hearing," explained Kemler. "So we want to focus on that in the beginning [...] We want to focus on the live, conversational, interactive communication aspect of the product." Additionally, Google's research with Gallaudet university, (a federally chartered private university for the deaf and hard-of-hearing), showed that people get a little self-conscious if they think they're being recorded.

In a demo, I was surprised by how quickly it transcribed the text. It also was smart enough to recognize the context behind certain words. For example, when someone said "whether or not," it initially spelled it "weather," but retroactively changed it to "whether" when it realized the context of the conversation.

Yet, it's not perfect. Sometimes it doesn't quite understand certain words -- "attenuate" was translated as "attend weight" for example. When multiple people are speaking simultaneously, it has a hard time figuring out when one's person speech ends and the other begins, and sometimes doesn't work altogether. It also can't transcribe songs, or even if someone is talking in a sing-songy way. Apparently this is to reduce the possibility of a mistranscription, and thus, miscommunication, but it's also not ideal.

Another potential downside is that Live Transcribe requires an internet connection. Google says it uses the Google Cloud speech API, which has far higher levels of accuracy than on-device speech recognition models. The company says that no audio or text is ever stored, and that the sole process is to send the audio up, and have the text sent down. While that may be the case, it also means you probably can't use Live Transcribe in areas with no internet connection, like an airplane or remote rural areas.

Live Transcribe can understand speech in 70-plus languages and dialects, and if you live in a multilingual household, you can switch between two different ones by selecting a secondary language in the settings menu. Despite that though, it doesn't have the ability to live translate from one language to another, mostly due to technical limitations, but also because the team at Google wanted to focus in on the deaf and hard-of-hearing demographic who might not need this feature as much. Google is, however, working on teaching the algorithm to understand language automatically so that you don't have to keep switching languages.

The second app that Google announced today is Sound Amplifier, which is designed to make conversations in loud environments easier to understand. Think of it as an app-assisted hearing aid, but using only your Android phone and a pair of headphones. The app consists of several sliders that let you reduce environmental sound while boosting quieter sounds like voices to get a better listening experience. Obviously this app is meant for the hard-of-hearing but also just anyone who wants to have a clearer conversation in a noisy environment could make use of this.

I tried this out in a demo as well, and it really did feel like I was wearing a pair of high-end hearing aids. The background noise of the busy conference room faded away ever so slightly, and the voices of those around me seemed amplified. Of course, this might not work in especially noisy environments like a concert, but for everyday situations, it seems like a pretty decent solution.

According to Google, Sound Amplifier uses the new "dynamics processing effect" engine in Android P. Therefore, while Live Transcribe is compatible on phones with Lollipop and up, Sound Amplifier will only be available on phones with Android P onwards. Sound Amplifier doesn't need an internet connection to work -- all the processing is done locally.

Both Live Transcribe and Sound Amplifier will be available in the Play Store starting today, and they will both be pre-installed in all Pixel 3 phones going forward.