Deep Learning in Finance: Learning to Trade with Q-RL and DQNs

Price changes in financial products are largely random but can often be supplemented by features, providing additional structure that can be exploited for trading profits. Experienced traders are skilled at identifying these features and deploying profitable exploits, with the use of Q- function based reinforcement learning and DQNs (a neural network based reinforcement learning technique).David Samuel is co-founder of RKR Epsilon & Predictive Machines, where they are developing algorithms for predicting and trading in commercial transactions markets. As a veteran of the financial markets, David has worked in trading and quantitative research for a number of investment banks and proprietary trading firms; as well as lecturing for MSc Mathematical Finance at Oxford University and collaborating on machine learning applications within trading alongside researchers at Cambridge University. At the Deep Learning in Finance Summit in Singapore, David will be sharing expertise on methods using Q- function based reinforcement learning and DQNs trained on simulation models for markets, with data provided by generative models that mimic both the randomness and salient features of actual markets. I asked him a few questions ahead of the summit to learn more.Please tell us more about your work, and a teaser for your session at the upcoming summit.  I’m David Samuel, co-founder of Prediction Machines which is developing IP and conducting research into the fields of commercial transactions and markets, where the prices of products (commodities, assets, financial instruments, including digital/virtual forms) are uncertain at future time horizons and where there is a need to determine optimal trading strategies.  I have worked in the financial markets for over 20 years in roles including quantitative research, trading, risk management and business management.  I have also taught classes for the Masters Degree in Quantitative Finance at Oxford University and presented research at a number of industry events. At the Deep Learning in Finance Summit I shall be presenting some of our latest research into the use of Q-Function Reinforcement Learning (QRL) algorithms for trading financial instruments, where the implementation is via the use of Deep Q-Networks (DQNs).   The starting point of this research is the observation that some experienced traders are able to generate fairly consistent profits from trading the financial markets (accepting that there will be some variability in performance relating to ever changing market opportunities).  Indeed one can specialise this observation to cases where the information available to the trader is limited to the order books, historical price evolution, and trade event history of a small set of products.  The actions of the trader are not random, they are informed decisions based on what is happening in the market.  An interesting question is then is whether these decisions / actions can be learned? Traditional approaches to quantitative trading typically involve the use of well known techniques in machine learning to identify structure (features) within the market data which can then be exploited via an appropriate trading strategy.  Whilst this machine learning approach can offer up some profitable features they are often not the ones spotted by the experienced traders.   Reinforcement Learning provides a potential framework for learning how to trade but traditional methods, when presented with a relatively small amount of noisy market data, are plagued by various complexities that make the approach difficult to tackle.  However, Q-Learning (a form of Reinforcement Learning) applied to specially designed trading simulation games do provide some promising results.  Generating synthetic data that largely mimics the random behaviour of markets but also contains the salient features that will result in exploits presenting themselves is an important part of the process of getting Q-RL to successfully learn how to trade.How did you get into deep learning?  My background is in theoretical physics and I initially joined the financial markets industry as a quantitative researcher / derivatives trader.  I became interested in predictive models of short term price moves in financial instruments using machine learning methods that were growing in popularity in the industry in the early 2000’s, and had some collaborations with researchers at Oxford University and Cambridge University machine learning groups.   A few years later I was fortunate to be able to participate in the Machine Learning Summer School held at Cambridge University (2009) in which one of the lecturers was Professor Geoff Hinton, who presented work on Deep Boltzmann Machines.  This was my first introduction to deep learning methods. At that time most of the financial industry was still focused on the use of the more widely known techniques in machine learning, as these methods were more accessible and easier to implement with financial markets data.   However, over the past three years a combination of the progress made in deep learning and the resources that have been made available for general use (e.g., TensorFlow) have made deep learning methods more accessible and have therefore become a part of our research efforts at Prediction Machines.   Which industries or areas do you feel deep learning will have the most beneficial impact?This is a good question, and a difficult question. Beneficial for whom? A number of very smart people believe that deep learning and its extension into AI will have a significant and disruptive effect on all aspects of global socio-economics.  It is too difficult to predict specific domains of beneficial impact because everything is coupled in a complex manner and no doubt where there are benefits in some areas there will also be negative consequences.  Obviously, one tends to think about advancements and applications in medicine and health care to be the most important benefits for a global society.  However, it is a very complex ecosystem to consider and we are likely to be surprised with some new products or services that don’t yet exist.  It is encouraging to see some smart leaders in this space participating in events such as Asilomar 2017 to help steer the path forward.   What advancements in deep learning would you hope to see in the next 3 years?  It is encouraging to see that the pioneers and leaders in deep learning have adopted an open approach to publishing their work and making specialised resources generally available.  The pace of development in deep learning is currently phenomenal and so extrapolating out three years is challenging.  I hope that whatever advancements take place continue to be open.  I suspect that some of the main technology advancements will be in specialised hardware designed specifically for implementing deep nets.     Will fintech startups continue to influence industry financial services? In areas such as payments, secure transactions, compliance, investment advice, and micro-finance it would appear that there has been a very good start.  There is now an expected crowded field of start-ups ready for some thinning out.  Of course, first-mover advantage doesn’t always hold and there is scope for new entrants with better operating models or technology solutions.   For trading of financial markets it would seem that there is still scope to fully take into account the enormous amount of data that presents itself.  Bringing all this information together in a timely manner and in a way that it can all be combined into optimal decision making processes remains work-in-progresss.David Samuel will be speaking at the Deep Learning in Finance Summit, taking place alongside the Deep Learning Summit in Singapore on 27-28 April. Learn about deep learning applications in the financial sector from algorithms to forecast financial data, to tools used for data mining & pattern recognition in financial time series, to scaling predictive models, to stock market prediction, to using blockchain technology. View further information here.

Tickets are limited for this event. Register your place now.

The Deep Learning in Finance Summit will next be held in London on 15-16 March and New York on 6-7 September. View all upcoming events here.