论文标题：Real-time decoding of question-and-answer speech dialogue using human cortical activity

期刊：Nature Communications

作者：David A. Moses,Matthew K. Leonard,Joseph G. Makin,Edward F. Chang

发表时间：2019/07/30

数字识别码：10.1038/s41467-019-10994-4

微信链接：https://mp.weixin.qq.com/s/04g7W4O3xXtIgdmw0LmSwA

《自然-通讯》发表的一项研究Real-time decoding of question-and-answer speech dialogue using human cortical activity介绍了一种可以将问答对话相关脑活动转化为文字记录的神经解码器。

图1：问（蓝）和答（红）过程的实时语音解码示意。来源：Moses等

大脑皮层包含不同的区域，其中的神经活动会编码语音感知与生成的表出。研究表明，这种脑活动可以被解码；但是之前的研究侧重于将听说任务分开解码。

美国加州大学旧金山分校的Edward Chang及同事在模拟的问答对话试验中，通过脑活动解码语音感知与生成。被试为三名正在接受癫痫治疗的患者，他们要听一系列的问题并通过一套规定的答案，口头回答问题，作者将被试在这个过程中的脑皮层活动记录了下来，之后这些数据被用于训练语音检测和解码模型。接着，被试再听一系列问题，并大声地用自己所选的答案作答。

仅使用在对话中记录下的神经信号，作者便能够检测被试何时在听，何时在说，并且能够预测所听或所说的是什么。通过解码问题，他们能够利用所得信息提高被解码答案的准确性（因为部分答案只适用于特定问题），而且对于生成语音和感知语音的解码准确性分别提高到了61%和76%。

还需要开展进一步的工作来解码想象的语音回答，这样才能使因为损伤或神经退行性疾病而无法说话的个体可以使用这项技术。

视频：利用来自参与者1的模拟数据实时解码任务。来源：Moses等

摘要：Natural communication often occurs in dialogue, differentially engaging auditory and sensorimotor brain regions during listening and speaking. However, previous attempts to decode speech directly from the human brain typically consider listening or speaking tasks in isolation. Here, human participants listened to questions and responded aloud with answers while we used high-density electrocorticography (ECoG) recordings to detect when they heard or said an utterance and to then decode the utterance’s identity. Because certain answers were only plausible responses to certain questions, we could dynamically update the prior probabilities of each answer using the decoded question likelihoods as context. We decode produced and perceived utterances with accuracy rates as high as 61% and 76%, respectively (chance is 7% and 20%). Contextual integration of decoded question likelihoods significantly improves answer decoding. These results demonstrate real-time decoding of speech in an interactive, conversational setting, which has important implications for patients who are unable to communicate.

（来源：科学网）