Deep Minds: An Interview with Google's Alex Graves & Koray Kavukcuoglu

At the RE.WORK Deep Learning Summit in London last month, three research scientists from Google DeepMind, Koray Kavukcuoglu, Alex Graves and Sander Dieleman took to the stage to discuss classifying deep neural networks, Neural Turing Machines, reinforcement learning and more. Google DeepMind aims to combine the best techniques from machine learning and systems neuroscience to build powerful general‑purpose learning algorithms. Formerly DeepMind Technologies, Google acquired the company in 2014, and now uses DeepMind algorithms to make its best-known products and services smarter than they were previously. Google's acquisition (rumoured to have cost $400 million) of the company marked the a peak in interest in deep learning that has been building rapidly in recent years.   DeepMind’s area of expertise is reinforcement learning, which involves telling computers to learn about the world from extremely limited feedback. As deep learning expert Yoshua Bengio explains: “Imagine if I only told you what grades you got on a test, but didn’t tell you why, or what the answers were - it’s a difficult problem to know how you could do better.”  However DeepMind has created software that can do just that. They hit headlines when they created an algorithm capable of learning games like Space Invader, where the only instructions the algorithm was given was to maximize the score. Within 30 minutes it was the best Space Invader player in the world, and to date DeepMind's algorithms can able to outperform humans in 31 different video games. This algorithm has been described as the "first significant rung of the ladder" towards proving such a system can work, and a significant step towards use in real-world applications.We caught up with Koray Kavukcuoglu and Alex Graves after their presentations at the Deep Learning Summit to hear more about their work at Google DeepMind.Can you explain your recent work in the Deep QNetwork algorithm?Koray: The research goal behind Deep Q Networks (DQN) is to achieve a general purpose learning agent that can be trained, from raw pixel data to actions and not only for a specific problem or domain, but for wide range of tasks and problems. In order to tackle such a challenge, DQN combines the effectiveness of deep learning models on raw data streams with algorithms from reinforcement learning to train an agent end-to-end. We have developed novel components into the DQN agent to be able to achieve stable training of deep neural networks on a continuous stream of pixel data under very noisy and sparse reward signal. And more recently we have developed a massively parallel version of the DQN algorithm using distributed training to achieve even higher performance in much shorter amount of time.    What are the main areas of application for this progress?K: DQN is a general algorithm that can be applied to many real world tasks where rather than a classification a long term sequential decision making is required. It is a very scalable RL method and we are in the process of applying it on very exciting problems inside Google such as user interactions and recommendations. In general, DQN like algorithms open many interesting possibilities where models with memory and long term decision making are important.    Can you explain your recent work in the neural Turing machines?Alex: The basic idea of the neural Turing machine (NTM) was to combine the fuzzy pattern matching capabilities of neural networks with the algorithmic power of programmable computers. A neural network controller is given read/write access to a memory matrix of floating point numbers, allow it to store and iteratively modify data. As Turing showed, this is sufficient to implement any computable program, as long as you have enough runtime and memory. The key innovation is that all the memory interactions are differentiable, making it possible to optimise the complete system using gradient descent. By learning how to manipulate their memory, Neural Turing Machines can infer algorithms from input and output examples alone. In other words they can learn how to program themselves.    What are the main areas of application for this progress?A: There has been a recent surge in the application of recurrent neural networks — particularly Long Short-Term Memory — to large-scale sequence learning problems. In areas such as speech recognition, language modelling, handwriting recognition and machine translation recurrent networks are already state-of-the-art, and other domains look set to follow. Neural Turing machines may bring advantages to such areas, but they also open the door to problems that require large and persistent memory. One such example would be question answering.      What are the key factors that have enabled recent advancements in deep learning?K: Perhaps the biggest factor has been the huge increase of computational power. This has made it possible to train much larger and deeper architectures, yielding dramatic improvements in performance. More is more when it comes to neural networks. Another catalyst has been the availability of large labelled datasets for tasks such as speech recognition and image classification. At the same time our understanding of how neural networks function has deepened, leading to advances in architectures (rectified linear units, long short-term memory, stochastic latent units), optimisation (rmsProp, Adam, AdaGrad), and regularisation (dropout, variational inference, network compression).   What sectors are most likely to be affected by deep learning?A: All industries where there is a large amount of data and would benefit from recognising and predicting patterns could be improved by Deep Learning.     What advancements excite you most in the field?K: One of the most exciting developments of the last few years has been the introduction of practical network-guided attention. Attention models are now routinely used for tasks as diverse as object recognition, natural language processing and memory selection. Other areas we particularly like are variational autoencoders (especially sequential variants such as DRAW), sequence-to-sequence learning with recurrent networks, neural art, recurrent networks with improved or augmented memory, and stochastic variational inference for network training.  What developments can we expect to see in deep learning research in the next 5 years?K & A: A lot will happen in the next five years. We expect both unsupervised learning and reinforcement learning to become more prominent. We also expect an increase in multimodal learning, and a stronger focus on learning that persists beyond individual datasets.  The next Deep Learning Summit is taking place in San Francisco on 28-29 January, alongside the Virtual Assistant Summit. For more information and to register, please visit the event website here.For further discussions on deep learning, machine intelligence and more, join our groups on Linkedin and Google+.