Virtual vocal tract developed to help with speech loss

Researchers at the University of California have developed a virtual vocal tract, comprised of a decoder that transforms brain activity into movements, and a synthesiser that converts these movements into an approximation of the patient’s voice, that produces accurate synthetic speech. The researchers noted that this synthetic speech was significantly better than that directly decoded from brain activity without the virtual vocal tract.
The team started by collecting recordings of participants reading sentences and simultaneously logging the corresponding brain activity. Then they reverse engineered the vocal movements necessary to produce the sounds and mapped them to the brain activity associated with those sounds, facilitating the creation of a virtual vocal tract for each participant that their brain activity could control. The neural code for vocal movements overlapped in places, so that one participant’s vocal tract simulation could be altered to respond to the neural instructions recorded from another participant’s brain. This might mean that individuals with speech loss due to neurological impairment could learn to control a speech prosthesis modelled on the voice of someone with intact speech.
Edward Chang, a professor of neurological surgery and member of the UCSF Weill Institute for Neuroscience, said: “For the first time, this study demonstrates that we can generate entire spoken sentences based on an individual’s brain activity. This is an exhilarating proof of principle that with technology that is already within reach, we should be able to build a device that is clinically viable in patients with speech loss.”
Josh Chartier, a bioengineering graduate student in Chang’s lab, said: “We still have a ways to go to perfectly mimic spoken language. We’re quite good at synthesising slower speech sounds like ‘sh’ and ‘z’ as well as maintaining the rhythms and intonations of speech and the speaker’s gender and identity, but some of the more abrupt sounds like ‘b’s and ‘p’s get a bit fuzzy. Still, the levels of accuracy we produced here would be an amazing improvement in real-time communication compared to what’s currently available. People who can’t move their arms and legs have learned to control robotic limbs with their brains. We are hopeful that one day people with speech disabilities will be able to learn to speak again using this brain-controlled artificial vocal tract.”