Neural Attention: Machine Learning Meets Neuroscience

Neural attention has been applied successfully to a variety of different applications including natural language processing, vision, and memory. An attractive aspect of these neural models is their ability to extract relevant features from data, with minimal feature engineering.   Brian Cheung is a PhD Student at UC Berkeley working with Professor Bruno Olshausen, as well as an Intern at Google Brain. By drawing inspiration from the fields of neuroscience and machine learning, he hopes to create systems which can solve complex vision tasks using attention and memory.        At the Deep Learning Summit in Singapore, Brian will share expertise on the fovea as an emergent property of visual attention, ways we can extend this ability to learning interpretable structural features of the attention window itself, and finding conditions where these emergent properties are amplified or eliminated providing clues to their function. I asked him a few questions ahead of the summit to learn more.How did you start your career is deep learning? What motivated you to join this field in particular?  I simply thought they were an interesting model, the main concepts were fairly easy to grasp. In particular, the emphasis on learning representations rather than engineering them seemed particularly powerful. The fundamental building blocks of deep learning seem far more unified and extendable than other models which existed at the time.   Your research interests seem to lie at the intersection of neuroscience and machine learning. What are the benefits of bringing these two disciplines closer together? Some of my research involves applying machine learning to analyze and encode neuroscience data. Going that direction is tricky because you often have a black box (neural net) predicting another black box (the brain). But once you have a good predictor of brain activity, probing a neural net is still far easier than probing the brain. Furthermore, there is a growing body of work from the community for getting a better understanding of these models. Being in a theoretical neuroscience lab, I also draw some inspiration for ideas in deep learning from our current theories about how the brain works. I think neuroscience can provide a very good reference for the tasks we should be trying to solve. For example, the coupling of action and perception is a very important aspect of intelligence and I think that will play a larger and larger role in deep learning.  How can an understanding of the human brain's capacity for memory and attention be applied in AI? Recurrent networks is an obvious example of the usefulness of memory. Attention mechanisms over the memory of these models has also been shown to be beneficial. For the future, I also think it's important to think of memory and attention in the context of reinforcement learning. I think understanding how memory can be used to train a neural network performing actions in an environment will be very beneficial in the future.   What challenges do you think will be most interesting for researchers & scientists in the next few years? Disclaimer: My answer to this will be inevitably biased towards my own research. But I think integrating the learning process with the inference process in neural networks will be an important next step. Loosening the distinction between the 'training' and 'testing' phase. Again, we see this being very important in action perception tasks which reinforcement learning is trying to solve. A longstanding challenge to this is catastrophic interference. Currently, online learning is very unstable and I expect this issue to gain more attention in the future.   Brian Cheung will be speaking at the Deep Learning Summit in Singapore on 20-21 October. Other speakers include Modar Alaoui, Eyeris; Pradeep Kumar, Lenovo; Vassilios Vonikakis, Advanced Digital Sciences Center; Takashi Miyazaki, Yahoo Japan; and Mun Yew Wong, Asia Genomics.Tickets are limited for this event and previous Deep Learning Summits have sold out. For more information and to book your pass, please visit the website here.