## Short-cut

- Deep Learning by Andrew Ng (Baidu) (GPU Tech conf, 2015)
- GPU computing를 이용해 Deep Learning 기술을 본 궤도에 올려놓은 Dr. Andrew Ng (Stanford, Baidu)의 최근 강연

- Deep Learning by Yann LeCun, Yoshua Bengio & Geoffrey Hinton (Nature, May 2015)
- 딥러닝을 만든 3명의 대가가 직접 쓴 딥러닝 소개

- Teaching Machines to Understand Us (TechReview, Aug. 2015)
- 딥러닝에 대한 짧은 역사

## Background

### Timelines

**Beginning!**Hinton, Geoffrey E., and Ruslan R. Salakhutdinov.**Reducing the dimensionality of data with neural networks***Science*313.5786 (2006): 504-507.- Hinton introduced deep learning, new methods for training multi-layered (deep) neural networks.
- (Video) The Next Generation of Neural Networks by Geoffrey Hinton (Google TechTalks) (2007)
- (Video) Recent Developments in Deep Learning by Geoffrey Hinton (Google TechTalks) (2010)

**Getting attention!**How Many Computers to Identify a Cat? 16,000 (Andrew Ng, Google) (NYT, 2012.06)- Google’s Artificial Brain Learns to Find Cat Videos (Wired, 2012.06)
- (Paper) Building High-level Features Using Large Scale Unsupervised Learning by Quoc Le et al. and Andrew Ng (Stanford, Google) (ICML, 2012)
- (Paper) Large Scale Distributed Deep Networks by Jeffrey Dean et al. and Andrew Ng (Google) (NIPS, 2012)
- (Video) Machine Learning and AI via Brain Simulations (Deep Learning and Unsupervised Feature Learning) by Andrew Ng (2012)

- Google Hires Brains (Geoffrey Hinton) that Helped Supercharge Machine Learning (Wired, 2013.03)
- The Man Behind the Google Brain: Andrew Ng and the Quest for the New AI (Wired, 2013.05)
**GPU computing!**Now You Can Build Google’s $1M Artificial Brain on the Cheap (Andrew Ng, GPU) (Wired, 2013.06)- Researcher Dreams Up Machines That Learn Without Humans (Yoshua Bengio) (Wired, 2013.06)
- Facebook’s ‘Deep Learning’ Guru (Yann LeCun) Reveals the Future of AI (Wired, 2013.12)
**Deep Learning – 10 Breakthrough technologies in 2013**(TechReview, 2013)

- Meet the Man (Geoffrey Hinton) Google Hired to Make AI a Reality (Wired, 2014.01)
- Man (Andrew Ng) Behind the ‘Google Brain’ Joins Chinese Search Giant Baidu (Wired, 2014.05)
- Microsoft Challenges Google’s Artificial Brain With ‘Project Adam’ (Wired, 2014.07)
- Facebook’s Quest to Build an Artificial Brain Depends on This Guy (Wired, 2014.08)
- The Data Scientist on a Quest to Turn Computers Into Doctors (Wired, 2014.08)
- The ‘Chinese Google’ Is Making Big Bucks Using AI to Target Ads (Wired, 2014.10)
- A Googler’s Quest to Teach Machines How to Understand Emotions (Quoc Le) (Wired, 2014.12)
- New Startup Sets Out to Bring Google-Style AI to the Masses (Wired, 2014.12)

- Facebook Open-Sources a Trove of AI Tools (Wired, 2015.01)
- A Startup’s Neural Network Can Understand Video (TechReview, 2015.02)
- Nvidia’s Powerful New Computer Helps Teach Cars to Drive (Wired, 2015.03)
- Baidu’s Artificial-Intelligence Supercomputer Beats Google at Image Recognition (TechReview, 2015.05)
- Deep Learning Catches On in New Industries, from Fashion to Finance (TechReview, 2015.05)
- Why and How Baidu Cheated an Artificial Intelligence Test (ImageNet Challenges) : Machine learning gets its first cheating scandal (TechReview, 2015.06)
- AI’s Next Frontier: Machines That Understand Language (Wired, 2015.06)
- Google Made a Chatbot That Debates the Meaning of Life (Wired, 2015.06)
- IBM Pushes Deep Learning with a Watson Upgrade (TechReview, 2015.07)
- The Guy Who Taught AI to ‘Remember’ Is Launching a Startup (RNN) (Wired, 2015.07)

### Long stories on Deep Learning

- Internet to Neural Net by Steven Levy
- Deep Mind of Demis Hassabis by Steven Levy
- The Believers – The hidden story behind the code that runs our lives (Feb. 2015)
- Facebook AI Director Yann LeCun on His Quest to Unleash Deep Learning and Make Machines Smarter (IEEE Spectrum, Feb. 2015)
- Teaching Machines to Understand Us (TechReview, Aug. 2015)

### AI and Beyond

- The Three Breakthroughs That Have Finally Unleashed AI on the World (Wired, 2014)
- DeepMind: inside Google’s super-brain (Wired, July 2015)

### Review

**Deep Learning**by Yann LeCun, Yoshua Bengio & Geoffrey Hinton (Nature, May 2015)**Machine learning: Trends, perspectives, and prospects**by M. I. Jordan, T. M. Mitchell (Science, July 2015)

## News & Articles

- How Google Translate squeezes deep learning onto a phone (Google Research Blog, 2015.07)
- Composing Music With Recurrent Neural Networks (2015.08)
- Deep learning for assisting the process of music composition (part 1)
- The neural networks behind Google Voice transcription (Google Research Blog, 2015.08)
- Baidu explains how it mastered Mandarin with deep learning (Baidu, DeepSpeech, Awni Hannun) (2015.08)

## Slides

- Prerequisite : Machine Learning

- Scaling Up Deep Learning by Yoshua Bengio (KDD Tutorial, 2014)
- Deep Learning by Yoshua Bengio (MLSS, 2015)

- Deep Learning by Ruslan Salakhutdinov (KDD Tutorial, 2014)
- Large Scale Deep Learning by Jeff Dean (Google) (CIKM Keynote, 2014)

- Introduction to Deep Learning with GPUs (NVIDIA) (2015)
- Deep Learning for Image Classification (NVIDIA) (2015)

- Deep Learning Tutorial by Yann LeCun (ICML, 2013)
- The Unreasonable Effectiveness of Deep Learning by Yann LeCun (GPUTechConf, 2014)
- What’s wrong with Deep Learning? by Yann LeCun (CVPR Keynote, 2015)

- Deep Learning for Natural Language Processing (without magic) by Richar Socher and Christopher Manning (NAACL, 2013)
- Deep Learning for Natural Language Processing (without magic) by Richar Socher (MLSS, 2014)

- Deep Learning for Natural Language Processing and Related Applications by Xiaodong He, Jianfeng Gao, and Li Deng (Microsoft) (2014)
- Deep Learning for Natural Language Processing: Theory and Practice by Xiaodong He, Jianfeng Gao, Li Deng (Microsoft) (CIKM Tutorial, 2014)
- Deep Learning for Web Search and Natural Language Processing by Jianfeng Gao (Microsoft) (WSDM, 2015)

## Videos (Talks)

- Deep Learning by Geoffrey Hinton (2015.06)
- Deep Learning by Andrew Ng (Baidu) (GPU Tech conf, 2015)
- Large Scale Deep Learning by Jeff Dean (Google) (GPU Tech conf, 2015)
- Deep Learning for Natural Language Processing by Richard Socher (2015)

- GPU Technology conference (2015)
- GPU computing은 지금의 deep learning이 자리잡는데 가장 큰 역할을 한 핵심 기술임.

## Books

### Data Mining, Machine Learning and Big Data Mining

**Pattern Recognition and Machine Learning**by Christopher M. Bishop (2006)**Elements of Statistical Learning**by Trevor Hastie, Robert Tibshirani and Jerome Friedman (2009)**Bayesian Reasoning and Machine Learning**by David Barber (2012)**Mining of Massive Datasets**by Jure Leskovec, Anand Rajaraman, Jeff Ullman (2014)**Probabilistic Programming & Bayesian Methods for Hackers**by Cam Davidson-Pilon (web, Python examples)

**Learning from Data**by Yaser S. Abu-Mostafa, Malik Magdon-Ismail, Hsuan-Tien Lin (2012)**Machine Learning: a Probabilistic Perspective**by Kevin Murphy (2013)

### Neural Networks and Deep Learning

**Deep Learning**by Yoshua Bengio, Ian Goodfellow and Aaron Courville (2015) MIT Press.**Neural Networks and Deep Learning**by Michael Nielsen (2015) web.

## Online Lectures

### Courses

- Prerequisite: Machine Learning

- Neural Networks for Machine Learning by Geoffrey Hinton (2012)
- Deep Learning by Yann LeCun (2014) NYU
- Deep Learning by Yann LeCun (2015) NYU
- Deep Learning by Nando de Freitas (2015) U. Oxford
- Convolutional Neural Networks for Visual Recognition (2015) Stanford
- Deep Learning for Natural Language Processing (2015) Stanford

### Short lectures

## Software

- GPU는 필수. GPU 머신이 없으면 AWS GPU 인스턴스를 이용함.
- Python은 필수. C++은 라이브러리만 쓸 줄 알면 됨. R은 선택 사항.
- 시작은 일단 GPU 머신에 NVIDIA CUDA 드라이브 설치하고, cuDNN도 깔고, Theano나 Caffe 예제 한 번 돌려보는걸로.

### Open sources

- GitHub on Deep Learning
**Caffe**(C++, GPU, Python interface)- DeepLearning4J (Java)
- Torch (Facebook)
**Theano**(Python)- Lasagne (a lightweight library to build and train neural networks in Theano)
- Keras (Theano-based Deep Learning library, convnets, recurrent neural networks, and more)
- Cuda-convnet (Fast convolutional neural networks in C++/CUDA)
- CXXNET (fast, concise, distributed deep learning framework)
- Neon (Nervana’s python based Deep Learning Framework)
- Minerva (a fast and flexible tool for deep learning on multi-GPU)
- NVIDIA cuDNN (GPU Accelerated Deep Learning)
- NVIDIA DIGITS (Interactive Deep Learning GPU Training System)
- convnet
**benchmarks** - word2vec
- ConvNetJS (a Javascript library for training Deep Learning models entirely in your browser)

### commercial software[edit]

- (Dato) Deep Learning: Doubly Easy and Doubly Powerful with
**GraphLab Create** - (Dato) Practical Text Analysis using Deep Learning
- H20 Deep Learning
- How to use R, H2O, and Domino for a Kaggle competition (Sept. 2014)
- (Domino) Faster deep learning with GPUs and Theano (2015.08)

### Experience : Learning from Kaggle[edit]

**Kaggle blog****Winner’s interviews**- Using convolutional neural nets to detect facial keypoints tutorial (GPU, Lasagne)
- Deep Learning Tutorial for Facial Keypoints Detection using
**AWS** - My solution for the Galaxy Zoo challenge (2014)
- Winning the Galaxy Challenge with convnets: Sander Dieleman
- https://github.com/benanne/kaggle-galaxies
- Rotation-invariant convolutional neural networks for galaxy morphology prediction by Sander Dieleman, Kyle W. Willett, Joni Dambre
*Monthly Notices of the Royal Astronomical Society, 450(2), 1441-1459*.

- Classifying plankton with deep neural networks (2015)
- CIFAR-10 Competition Winners: Interviews with Dr. Ben Graham, Phil Culliton, & Zygmunt Zając (2015)
- Detecting diabetic retinopathy in eye images (2015)
**A Full Hardware Guide to Deep Learning**(2015)- Use Google’s Word2Vec for movie reviews (text mining using word2vec)

## Research Papers

### Tutorial / Survey

- Learning Deep Architectures for AI by Yoshua Bengio (FnT, 2009)
- Deep Learning – Methods and Applications by Li Deng and Dong Yu (FnT, 2013)
- Deep Learning of Representations: Looking Forward by Yoshua Bengio (2013)
- Deep Learning in Neural Networks: An Overview by Juergen Schmidhuber (2014)
- A tutorial survey of architectures, algorithms, and applications for deep learning by Li Deng (2014)

### CNN (Convolutional Neural Networks) & ImageNet Classification Challenges

- ImageNet Large Scale Visual Recognition Challenge (2015)
- Why and How Baidu Cheated an Artificial Intelligence Test (ImageNet Challenges) : Machine learning gets its first cheating scandal (TechReview, 2015.06)
- Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification (Microsoft) (2015)
- Very Deep Convolutional Networks for Large-scale Image Recognition (VGG) (ICLR, 2015)
- Going deeper with convolutions (Google) (2014)
- ImageNet Classification with Deep Convolutional Neural Networks (AlexNet) (2012)

### RNN (Recurrent Neural Networks)

### Applications

- Teaching Deep Convolutional Neural Networks to Play
**Go**(바둑) by Christopher Clark, Amos Storke (2014) - Move Evaluation in
**Go**(바둑) Using Deep Convolutional Neural Networks by Chris J. Maddison, Aja Huang, Ilya Sutskever, David Silver (Google) (2014)

### 2015

**Collaborative Deep Learning for Recommender Systems**by Hao Wang et al. (KDD,2015)**Deep Visual-Semantic Alignments for Generating Image Descriptions**by Andrej Karpathy, Li Fei-Fei (CVPR, 2015)**An Empirical Evaluation of Deep Learning on Highway Driving**by Brody Huval et al. and Andrew Ng (2015)**Deep Learning for Acoustic Modeling in Parametric Speech Generation: A systematic review of existing techniques and future trends**by Zhen-Hua Ling, Shi-Yin Kang, Heiga Zen, Andrew Senior, Mike Schuster, Xiao-Jun Qian, Helen Meng, and Li Deng (IEEE SPM, May 2015)**Predicting the sequence specificities of DNA- and RNA-binding proteins by deep learning**by Babak Alipanahi, Andrew Delong, Matthew T Weirauch & Brendan J Frey. Nature Biotechnology 33, 831–838 (2015)**Deep learning for detecting robotic grasps**by Lenz, Ian, Honglak Lee, and Ashutosh Saxena. The International Journal of Robotics Research 34.4-5 (2015): 705-724**EmoNets: Multimodal deep learning approaches for emotion recognition in video**by Kahou et al. and Yoshua Bengio (2015)**Toxicity Prediction using Deep Learning**(2015)

### 2014

- Deep Speech: Scaling up end-to-end speech recognition (Baidu) (2014)
**Project Adam: Building an Efficient and Scalable Deep Learning Training System**by Trishul Chilimbi, Yutaka Suzue, Johnson Apacible, and Karthik Kalyanaraman (Microsoft) (OSDI’14)**Learning a deep convolutional network for image super-resolution**by Dong, Chao, et al. Computer Vision–ECCV 2014.

### 2013

**Deep learning with COTS HPC systems**by Adam Coates et al. and Andrew Ng (ICML, 2013)**Improving deep neural networks for LVCSR using rectified linear units and dropout**by Dahl, George E., Tara N. Sainath, and Geoffrey E. Hinton. Acoustics, Speech and Signal Processing (ICASSP), 2013**Recent Advances in Deep Learning for Speech Research at Microsoft**by Li Deng et al. (2013)**Playing Atari with Deep Reinforcement Learning**(DeepMind) (2013)

### By Authors

- Yoshua Bengio (Google Scholar)
- Geoffrey Hinton (Google Scholar)
- Yann LeCun (Google Scholar)
- Andrew Ng (Google Scholar)