A novel artificial intelligence (AI) system created by US researchers can convert a person's brain activity while they are listening to or quietly imagine telling a tale into a continuous stream of text.
The system, created by a team at the University of Texas at Austin, is partially powered by a transformer model, like the ones used by Google's Bard and Open AI's ChatGPT.
According to the team that wrote the paper for publication in the journal Nature Neuroscience, it could aid persons who are intellectually cognizant but unable to physically talk, such as those disabled by strokes, to communicate coherently once again.
This language decoding device, known as the semantic decoder, is noninvasive since it does not need people to undergo surgical implants, in contrast to other language decoding systems now under development. Additionally, participants are not required to only use words from the list.
After rigorous training of the decoder, during which the subject spends hours listening to podcasts in the scanner, brain activity is recorded using a functional MRI scanner.
Later, the participant's listening to a new story or imagining telling a story enables the machine to generate corresponding text from brain activity alone, provided that the participant is willing to have their thoughts decoded.
According to Alex Huth, an associate professor of neurology and computer science at UT Austin, “this is a real leap forward compared to what's been done before, which is typically single words or short sentences.”
He said, “We're teaching the model to decode continuous language for long stretches of time with complex thoughts.
The end product is not a verbatim transcript. Instead, researchers created it to poorly capture the essence of what is being said or thought. When the decoder has been trained to keep track of a participant's brain activity, it produces text that roughly half the time matches the original words' intended meanings.
For instance, in studies, a participant's views were interpreted as “She has not even begun to learn to drive yet” when they heard a speaker say, “I don't have my driver's license yet.”
The scientists also addressed concerns around possible technological abuse throughout the trial. The study explains how decoding only worked with interested volunteers who were cooperative throughout decoder training.
Results were incomprehensible for those on whom the decoder had not been taught, and results were equally worthless for people on whom the decoder had been trained but subsequently encountered opposition, such as by thinking different ideas.
“We take very seriously the concerns that it could be used for bad purposes and have worked to avoid that,” said Jerry Tang, a computer science doctorate candidate. We want to ensure that individuals only utilize these technologies when they want to and when they would benefit from doing so.
The researchers also requested participants to view four brief, silent movies while they were being scanned, in addition to having them listen to or reflect on tales. Their brain activity was able to help the semantic decoder properly define certain events from the films.
Due to the system's dependency on the time required on an fMRI machine, it is presently not feasible for application outside of the lab. The functional near-infrared spectroscopy (fNIRS), for instance, might be used to carry out similar studies using more mobile brain imaging equipment, according to the researchers.
