Reconstructing Speech from Brain Signals

The previous post reviewed a science fiction book, The Accord by Keith Brooke, where virtual beings are created based on brain scans. It seems that scanning and decoding brain activity is getting closer to reality: A team of scientists from UC Berkeley, UC San Francisco, University of Maryland, and Johns Hopkins University managed to reconstruct individual words from brain signals of patients listening to recorded speech (B. Pasley et al., PLoS Biology 2012).

Speech reconstruction experiment paradigm

Listening to acoustic waveforms (left top) gives time-resolved signals (bottom right) recorded by probes implanted in the brain (top right). The signals are decoded into a spectrogram (bottom left). Image from original article.

The authors managed to reconstruct individual words by analyzing brain activity data. Reconstructing signals caused by live events is of cause very different from reading out the complete memory of a person as described in the sci-fi story The Accord. In fact, I doubt that memory can be accessed using electrodes. Electrodes require active, electrical signals in the brain while I suppose long-term memory is something more hard-wired. Nevertheless, the research article shows the tremendous progress science and technology is making by the combination of biology and information technology. It will be interesting to see when applications of this technology become available to, e.g., allow disabled persons to communicate better.

Last but not least I would like to thank the authors for publishing their work in an open access journal under a creative commons licence. Otherwise, I would not have been able to read this article and to legally show the picture on this blog.


Book Review: The Accord by Keith Brooke

Book cover of The Accord by Keith Brooke

Cover of the ebook edition.

The Accord by Keith Brooke is a science fiction novel exploring the idea of a virtual heaven. The story is set in the future where climate change and overpopulation cause widespread wars, famine, and mass migration. These problems, however, only serve as the backdrop to a love triangle with Priscilla, her husband Jack, both politicians, and the scientist Noah. Noah is the inventor of the Accord, a virtual heaven.

The story is based on the idea of a future virtualization technology that allows to separate mind from body: brain scans are used to create a snapshot of a human brain, a digitized soul. After death, the snapshot is transferred to the Accord where the soul continues to live on forever in a virtual world. As the inventor of the Accord, Noah creates his own virtual worlds where he tries to get virtual instances of Priscilla to fall in love with him. Jack finds out about this and kills his wife out of jealousy. After their death, Priscilla and Noah meet again in the Accord but Priscilla’s last brain scan is too long ago – she does not remember that she fell in love with Noah. After an assassination, Jack also enters the Accord. He is jealous and starts an ever lasting hunt for Noah, trying to kill his rival. But in the Accord, everything is virtual, even death. People are reborn with full memory of their death.

The plot runs in multiple parallel threads. Keith Brooke does a great job weaving them together, changing narrators often. In addition to the suspens building up by Jack turning into a psychopathic character, the book made me think about the implications of this technology: Is it really desirable to live forever? Can mind be separated from the physical world?  What is real? Are crimes committed in the virtual world bad or do virtual crimes not count? The technology described in The Accord might be far fetched. However, today’s virtual worlds such as gaming communities already let people retreat into virtual lives – virtual love and murder included.

As more and more souls enter the Accord, the computing capacity of the real world cannot keep up with the constantly increasing resource needs of this artificial heaven. The plot moves to quantum space and even toys with (quantum) space exploration. At the end, the story glides into the mystical as the inventor of the Accord, Noah, attracts religious followers – a part that to me did not fit quite naturally into this book. The ending, however, is fascinating and fitting to the quantum nature of the later Accord (spoiler-free review, so you will have to read it yourself).

In conclusion, The Accord has everything a good science fiction book should have: it takes a modern technology – virtualization – and takes it to its extreme, it raises some important questions such as the ethical implications of living in a virtual world, and, most importantly, entertains.

You can find the ebook at Smashwords, DRM-free and for a more than reasonable price. Don’t trust Automatody!? book reviews? – read another review of The Accord at SF site.