If you talk to anyone in Neural Nets and ask what function to use, they say you should pick a method where you give an input and compute the derivative of how much money you’re going to make. But that’s not the way people do it. The way people actually do it is to try and get the right answers from their training set, but this isn’t what we actually want, we want it to be correct on the training set. In distillation, we’ve already trained a big set, and now we’re training the new net to generalise the same way as the big net. This is the one case where you actually train the net to generalise.
Geoffrey Hinton, University of Toronto
This week at the Deep Learning Summit in Toronto, RE•WORK brought together leading experts working in AI and Deep Learning for the second Canadian edition of the RE•WORK Global DL Summit Series. Building on the 2017 Montreal event, attendees from over 20 countries came together to learn from industry experts in speech & pattern recognition, neural networks, image analysis and NLP, and explored how deep learning will impact all industries through interactive workshops, interviews, presentations, networking and more.
The two-day summit featured the addition of a new AI for Government Summit, and we welcomed over 600 attendees and 60 global expert speakers, offering 12+ hours of unrivalled networking opportunities, who were welcome to move between sessions from both events.
Throughout the two days, topics from healthcare to finance, and from policy to security were covered. The day one began with attendees being welcomed by a jam-packed schedule which they poured over whilst getting their morning caffeine fix and grabbing a pastry.
I’m interested in the way AI will change. It’s the research that will go a long way, and that’s what I’m excited about.
Florian Werner-Jaeger, infofeld GmbH
Adam Oberman from McGill University and out compére on the deep learning track kicked off by encouraging attendees to introduce themselves to each other, highlighting the importance of collaboration in the field. It was great to see straight away how attendees have come from across the country, the world, or just across the street to join us: ‘I work in San Francisco, I just came down for the conference.’ 'I’m at Shopify in Toronto', ‘I flew in from Germany and am heading to the U.S tomorrow.’ ‘I'm at CBC, the big building across the street!’
David Cox started the morning’s presentations by sharing his work at the MIT-IBM Watson AI Lab, and explained how the collaboration is the first of it’s kind with industry and academia coming together to focus on fundamental research in AI. ‘What’s next in Deep Learning and AI? I want to address the elephant in the room, and that is the phrase ‘Artificial Intelligence’. David explained how they’re looking at Narrow AI (emerging tech), Broad AI (where AI can be disruptive) and General AI (the more revolutionary and crazy far-fetched stuff!). ‘We’ve come so far with deep learning, but there are still challenges - for example in computer vision, when we layer images systems get really confused. That’s not something that would confuse humans, and these systems are supposed to be superhuman. We now need to work on fixing this.’
Next up, and one of the highlights of the morning for many, was Sara Hooker who spoke about frontiers of computer vision beyond accuracy. Sara began by asking ‘what do we mean by interpretability? How do we measure progress?’ Sara gave us a little background on computer vision and went on to explain interpretability and robustness, touching on some challenges in image recognition and computer vision: ‘the HD input space is a problem for machines - a single image has associated with it 1/4 million features. As humans, we’re very robust to these kinds of variations that often derail computers (backgrounds etc.).’ Sara also joined RE•WORK in a filmed interview with Natacha Mainville where they discussed Sara's career, as well as diversity in AI:
Natacha: What can we do to encourage more women into AI?
Sara: I don’t think people need encouragement, I think they need access.
Brendan Frey, an internationally recognized leader in machine learning and genome biology, also joined us in Thursday's sessions. Brendan is Co-Founder & CEO, and Professor at Deep Genomics and University of Toronto, and explained how over the past five years, exponential growth in biomedical datasets has created the perfect opportunity for deep learning to disrupt drug discovery. Throughout the presentation, Brendan shared many examples of AI in genomics, one being the following: ‘65% of your loved ones will be impacted by a serious disease of a genetic basis, so genomes is a big deal. There are 8 million births with genetic diseases per year, and this costs $5million per baby. The scary news is that Pharma is broken. In the 90s the rate of return on research and development was 30%, in 2010 it dropped below the bank rate. It’s expected to go to 0 by 2020. The next approach is AI - that’s how we can fix it. We view AI and DL as a crucial development for the future of drug discovery and medicine. What’s scary is humans aren’t good at reading the genome and never will be. This is a superhuman AI problem we need to solve. ‘
In parallel, the AI for Government Summit explored the opportunities of AI in government and for society. Hon. Peter Bethlenfalvy, President at the Treasury Board Secretariat of Ontario spoke about the recently announced review of government spending, developed by EY Canada, and explained how it focuses on creating a ‘modern, sustainable Ontario Government.’ ‘We ask people to give us their best ideas both inside and outside of government - we received 15,000 surveys and 26,000 ideas about how technology can help transform government. This is exciting for us - we have an opportunity in the digital age to transform government.’ He went on to explain how many industries in the city are benefiting from the transformation of Toronto into a Smart City: ‘Healthcare costs are back-ended so these are the types of things we have to see and improve. In the same hospital, linen is delivered by robots. They’re also using AI to provide predictive characteristics on things like flu season to staff accordingly and manage beds.’
During the coffee break, attendees discussed their mornings and explored the exhibition area. Intuitive AI shared their smart recycling system and explained how they're 'incentivising users in the front end whilst striving to improve recycling. "We're rewarding people for doing their bit, and we're excited to test it out with all the trash items people might have at the summit." - Vivek Ayas
What else did we learn?
PANEL: What are the Key Ethical Considerations of AI Policy & Governance?
Tegan Maharaj: Developers have a responsibility to explain the technologies they are exploring. Policy makers need to understand the technology. These things are explainable and understandable and it is critical for policymakers to not interpret AI as a non-understandable black box and will produce amazing results. They need to understand and there are a lot of nuances in which legal frameworks need to be applied.
Maroussia Levesque, Independent: We need to be building out expertise in two or three domains if we are to have meaningful conversations about policy in the AI space.
Amar Ashar: We now need to think, how do we build and think about emerging and disruptive technologies in the interest of society.
Matt Taylor, BorealisAI: Reinforcement learning is awesome, but sometimes there’s not an environmental reward you can rely on. We, as humans, can provide these sequential examples. The better the human can understand the agent, the more it can teach the agent.
Shane Gu, Google Brain:
In the past 4/5 years, RL’s had success mainly in simulation, where it can get the samples cheaply and fast. We still don’t see lots of applications in the real world. How can we bridge the gap here?
Mohammad Norouzi, Google Brain:
In speech recognition, we're trying to learn to map input audio to output text with lots of labelled data. We look at the number of edits that can be made and still provide the correct output.
Throughout the day’s sessions and coffee breaks, interview and podcast recordings were in full swing with William Brendel from Snap Research discussing the next 12 months of AI and social media and told us: “I’m excited to push boundaries with general AI. We take privacy very seriously at Snap and we must continue to do so. Global policies would be great but infused with best practices applicable to individual cases. If models are created without privacy in mind they can blow up in your face.”
The end of the day saw attendees from both tracks coming together in the plenary session to watch ‘the father of AI’, Geoffrey Hinton share his work and he spoke about a very ambitious project he’d worked on, and why it didn’t work!
Geoffrey introduced the concept of distillation and explained how ‘what we do in distillation, is we train an ensemble, then we get very confident answers. In order to get a softer distribution, we train, then divide the averaged logins from the ensemble by a ‘temperature’ to get a much softer distribution. This reveals much more information about the function on each training case.’ Geoffrey explained that this is very good for transferring knowledge between new models. Geoffrey went on to give us some comprehensive examples of his work and explain why distillation and ensembles may not always provide the best results when combined with other models.
Q: What is something you wish you knew when you were younger?
A: In 1986 I wish I knew this stuff was going to work! There were so many people who said it was nonsense. I wish I could have told them, you wait, but that would have been a stupid thing to say, because it seemed impossible. What I wish I knew now, is whether the brain uses back propagation!
Geoffrey Hinton, Professor, University of Toronto
Highlights & Takeaways
@Renita_Leung: Thank you to the Godfather of Deep Learning/AI Geoffrey Hinton! It was an honour listening to you talk about distillation and ensemble modelling! It was well worth the mad dash from San Francisco yesterday! #worklife #reworkdl #deeplearningsummit #ai #dl #toronto
@synced_global: #SyncedOnTheGo Geoffery Hinton reviews on ensemble learning, distillation, and label smoothing techniques: A better idea preventing big model being too certain - Penalize the entropies of the output distributions if the total entropy for a mini-batch is lower than some threshold.
@yolandalannqist: Geoffrey Hinton @UofT: “What I wish I knew in 1986? That this stuff would work. What I wish I knew now? If the brain uses backpropagation.” #AI #reworkDL
Attendees then came together for networking over wine and antipasti, before heading off to continue the conversations with new connections at Honda Xcelerator’s Networking Mixer at a local bar.
Back again on Friday morning, our AI experts returned bright and early to get stuck into panel discussions, workshops, interviews and more presentations.
“I’ve been recruited to be your master of ceremonies today! We’re very interested in AI because it’s one of the main drivers affecting our work going forwards” -
Chris Calabrese, Compere at the AI for Government Summit
The morning began by exploring how Ontario is leading the way in AI. Joseph Kurian, Sr Advisors, Ministry of Economic Development shared that he was excited to be here, as he’s met a lot of people who are visiting from outside Ontario. "We’re evolving as the global centre of AI - Ontario contributes more than 40% of Canada’s GDP. AI is cross-pollinating across sectors, FinTech, Healthcare, Business, and we’re really proud of this. In Ontario, from the policy side, we have a very vibrant ecosystem for AI. We have uni’s and colleges which have specialist departments in AI. Strong incubators and accelerators are coming together to create a highly collaborative environment. Research excellence and industry come together to excel."
The conversation continued around how we can use AI to create a sustainable future with Farhan Shafiq from Huawei: ‘How do we model a sustainable society? We look at the UN sustainable goals, and it’s leaning towards innovation and technology. There are lots of goals, and we need to map each one to a different layer of a sustainable society model. What do we really mean by a smart city? A lot of digitisation and analytics? A lot of apps? Or a way to create a sustainable environment? A city of the future must use technology to engage citizens to create a sustainable future.’
Over on the Deep Learning track, we were set for a morning of emerging ideas and technologies from leading startups working in AI.
David Julian from NetraDyne spoke about ‘Autonomously Generated HD Maps’, and explored how High-Definition (HD) Maps are a key component for autonomous vehicles. However, cost estimates are $2 Billion to map just the US once using special LIDAR mapping vehicles.: ‘If you look at the autonomous space, they use level 5 maps with very accurate posting of objects to understand the environment. However you really need to understand the changing environment.’
Tzvi Aviv from AgriLogicAI spoke about ‘Geospatial Intelligence for Profitability & Sustainability in the Agri-food Sector’ and explained how they ‘use convolutional neural networks to ingest large amounts of information to get better predictions. We are packaging this into a platform to predict losses in crop insurance.’ Tzvi used an example where they are ingesting data from multiple sources from high-resolution satellite images, historical data and putting it together so that they can provide a plan of a farm to identify financial sustainability of the farm and also to assess environmental risk.
Across both days, we saw attendees joining an extensive range of interactive workshops. Here are some of the highlights:
Building Scalable Machine Learning Architectures - Hosted by CBC
The CBC has the 3rd most traffic on mobile and 8th on web in Canada. In the event of a major news story, in a matter of seconds traffic can spike 10x. During this workshop Jason shared the CBC's unique approach to deploying scalable machine learning models, from the ingestion API, ETL, storage, training of models, to surfacing recommendations to millions of Canadians.
Gosia Loj - Big Innovation Centre: Data & Trust – Open Data and Personal Data Ownership Platforms: Social & Legal Implications (a 2 part workshop across both days)
"We're bringing all stakeholders together to give everyone an even footing in governance. We need to look at how we can link up with global parliaments and international working groups to bring in recommendations where policies should be restructured to adapt to AI. We believe data trust is a key area to start to ensure the access for AI and ML is safe, controlled, and retained within individuals."
Challenges & Opportunities of Investing in AI - VC Panel, Q&A & Networking Session
"The most difficult part of fundraising for a start-up is finding that investor to put the first cheque into it" - Saroop Bharwani
"If there's a data set out there available, somebody will be using it." - Jim Orlando
"If you have an algorithm, you have nothing until you have data" - Christian Lassonde
"Think carefully about contracts of data tracks from your customers.. make sure you have the right to use that data" - Margaret Wu
From Raw Data to Actionable Clinical Insights: High-Throughput Analysis with Microsoft, & Databricks Unified Analytics Platform for Genomics (uap4genomics) - Dr. Helia Mohammadi - National Healthcare, Microsoft Canada
The end-to-end analysis of large-scale genomics data still remains complex and expensive. In this workshop. Microsoft's workshop explained how Microsoft and the Databricks Unified Analytics Platform for Genomics (UAP4Genomics) simplifies the end-to-end process of turning raw sequencing data into actionable insights at population scale. "When you identify what part of the genome is responsible, you can come up with a treatment or prevention."
Attendees: What they say:
Our youngest attendee, Seyone Chithrananda, from The Knowledge Society at just 15 years old had 2 highlights from the summit:
“1. Being able to legitimately learn something new from Geoffrey Hinton as a pioneer in AI. 2. Being able to chat to Brendan Frey after his presentation and ask questions and gain useful insights on genome patterns."
After recording an episode of the Women in AI Podcast, Gosia Loj told us ‘It’s amazing being in a new country and bringing our work to a new group of audience and understanding their work in the Canadian governments and seeing how it’s different from the UK.’
Intuitive AI, who had a booth in the exhibition area, shared why they enjoyed the Summit: ‘You bring together a really interesting mix of academic research papers and industry and government officials. I’ve never seen this before but it really works and we’ve had a great time.’
Darrick Wiebe fromUntether AI thoroughly enjoyed Sara Hooker’s presentation on explainability: ‘It engaged the audience well. We’re working on a processor and I’m on the software side, so I’m most excited for networking and hearing what everyone else is working on, as well as the presentations - the lineup is great.’