Foundations and Challenges of Deep Learning (Yoshua Bengio)

The talks at the Deep Learning School on September 24/25, 2016 were amazing. I clipped out individual talks from the full live streams and provided links to each below in case that's useful for people who want to watch specific talks several times (like I do). Please check out the official website (www.bayareadlschool.org) and full live streams below.
Having read, watched, and presented deep learning material over the past few years, I have to say that this is one of the best collection of introductory deep learning talks I've yet encountered. Here are links to the individual talks and the full live streams for the two days:
1. Foundations of Deep Learning (Hugo Larochelle, Twitter) - • Foundations of Deep Le...
2. Deep Learning for Computer Vision (Andrej Karpathy, OpenAI) - • Deep Learning for Comp...
3. Deep Learning for Natural Language Processing (Richard Socher, Salesforce) - • Deep Learning for Natu...
4. TensorFlow Tutorial (Sherry Moore, Google Brain) - • TensorFlow Tutorial (S...
5. Foundations of Unsupervised Deep Learning (Ruslan Salakhutdinov, CMU) - • Foundations of Unsuper...
6. Nuts and Bolts of Applying Deep Learning (Andrew Ng) - • Nuts and Bolts of Appl...
7. Deep Reinforcement Learning (John Schulman, OpenAI) - • Deep Reinforcement Lea...
8. Theano Tutorial (Pascal Lamblin, MILA) - • Theano Tutorial (Pasca...
9. Deep Learning for Speech Recognition (Adam Coates, Baidu) - • Deep Learning for Spee...
10. Torch Tutorial (Alex Wiltschko, Twitter) - • Torch Tutorial (Alex W...
11. Sequence to Sequence Deep Learning (Quoc Le, Google) - • Sequence to Sequence D...
12. Foundations and Challenges of Deep Learning (Yoshua Bengio) - • Foundations and Challe...
Full Day Live Streams:
Day 1: • Video
Day 2: • Video
Go to www.bayareadlschool.org for more information on the event, speaker bios, slides, etc. Huge thanks to the organizers (Shubho Sengupta et al) for making this event happen.

[@thefunfamilytrees]

"The point I really want to talk about is the fourth one, how do we defeat the curse of dimensionality? In other words, if you don't assume much about the world, it's actually impossible to learn about it."

[@sadesoji]

This is one of my favorite talks on deep learning, period.
The observation that saddle points (not local minima) dominate the loss surfaces of high-D multilayer nets -- and that most local minima are located close to the global minimum -- is one of several convincing reasons why deep neural nets work surprisingly well after reaching convergence. See 25:00 minute mark.

[@kevinurban1016]

Some references from the video for quick reference:
* 1991: Hastad & Goldmann: [On the Power of Small-Depth Threshold Circuits](citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.120.7713&rep=rep1&type=pdf)
* 1994: Bengio et al: [Learning Long-Term Dependencies with Gradient Descent is Difficult](www.comp.hkbu.edu.hk/~markus/teaching/comp7650/tnn-94-gradient.pdf)
* 2007: Bengio & LeCun: [Scaling Learning Algorithms towards AI](www.iro.umontreal.ca/~lisa/bib/pub_subject/language/pointeurs/bengio+lecun-chapter2007.pdf)
* 2009: Bengio: [Learning Deep Architectures for AI](ace.cs.ohio.edu/~razvan/courses/dl6890/papers/ftml.pdf)
* 2011: Bengio & Delalleau: [On the Expressive Power of Deep Architectures](www.iro.umontreal.ca/~lisa/bib/pub_subject/finance/pointeurs/ALT2011.pdf)
* 2011: Delalleau & Bengio: [Shallow vs. Deep Sum-Product Networks](papers.nips.cc/paper/4350-shallow-vs-deep-sum-product-networks)
* 2013: Larochelle & Hinton: [Learning to combine foveal glimpses with a third-order Boltzmann machine](papers.nips.cc/paper/4089-learning-to-combine-foveal-glimpses-with-a-third-order-boltzmann-machine)
* 2013: Pascanu et al: [On the number of response regions of deep feed forward networks with piece-wise linear activations](arxiv.org/abs/1312.6098)
* 2014: Bahdanau et al: [Neural Machine Translation by Jointly Learning to Align and Translate](arxiv.org/abs/1409.0473)
* 2014: Dauphin et al: [Identifying and attacking the saddle point problem in high-dimensional non-convex optimization](papers.nips.cc/paper/5486-identifying-and-attacking-the-saddle-point-problem-in-high-dimensional-non-convex-optimization)
* 2014: Graves et al: [Neural Turing Machines](arxiv.org/abs/1410.5401)
* 2014: Koutnik et al: [A Clockwork RNN](arxiv.org/abs/1402.3511)
* 2014: Montufar & Morton: [When Does a Mixture of Products Contain a Product of Mixtures?](arxiv.org/abs/1206.0387)
* 2014: Montufar et al: [On the Number of Linear Regions of Deep Neural Networks](papers.nips.cc/paper/5422-on-the-number-of-linear-regions-of)
* 2014: Pascanu et al: [On the saddle point problem for non-convex optimization](arxiv.org/abs/1405.4604)
* 2014: Weston et al: [Memory Networks](arxiv.org/abs/1410.3916)
* 2015: Bengio et al: [STDP as presynaptic activity times rate of change of postsynaptic activity](arxiv.org/abs/1509.05936)
* 2015: Bengio et al: [Towards Biologically Plausible Deep Learning](arxiv.org/abs/1502.04156)
* 2015: Choromanska et al: [Open Problem: The landscape of the loss surfaces of multilayer networks](proceedings.mlr.press/v40/Choromanska15.pdf)
* 2015: Choromanska et al: [The Loss Surfaces of Multilayer Networks](proceedings.mlr.press/v38/choromanska15.pdf)
* 2015: Lee et al: [Difference Target Propagation](arxiv.org/abs/1412.7525)
* 2015: Sordoni et al: [A Hierarchical Recurrent Encoder-Decoder for Generative Context-Aware Query Suggestion](arxiv.org/pdf/1507.02221.pdf)
* 2015: Zhou et al: [Object Detectors Emerge in Deep Scene CNNs](arxiv.org/abs/1412.6856)
* 2016: Bahdanau et al: [An Actor-Critic Algorithm for Sequence Prediction](arxiv.org/abs/1607.07086)
* 2016: Bengio & Fischer: [Early Inference in Energy-Based Models Approximates Back-Propagation](arxiv.org/abs/1510.02777)
* 2016: Serban et al: [Building End-To-End Dialogue Systems Using Generative Hierarchical Neural Network Models](www.aaai.org/ocs/index.php/AAAI/AAAI16/paper/viewPaper/11957)
* 2017: Scellier & Bengio: [Equilibrium Propagation: Bridging the Gap between Energy-Based Models and Backpropagation](www.frontiersin.org/articles/10.3389/fncom.2017.00024/full)

[@philwilson1445]

Where can I find these slides?

[@muyunzhao]

media.wix.com/ugd/142eb4_79498997d26e47eaadf5017fa9550e5d.pdf
try this