SI151: Optimization and Machine Learning
Description
This course provides a broad introduction to machine learning, statistical learning and deep learning, with particular emphasis on learning models, optimization algorithms and statistical analysis. Topics include: supervised learning (e.g., generative learning, parametric and nonparametric learning, regression, classification, support vector machines, neural networks); unsupervised learning (e.g., clustering, dimensionality reduction, kernel methods, density estimation); statistical learning theory (bias and variance tradeoffs; VC theory; large margins). This course will also introduce optimization methods (e.g., gradient methods, proximal methods, quasiNewton methods, stochastic and randomized algorithms) that are suitable for largescale problems arising in machine learning applications.
Textbooks and Optional References
Textbooks:
Learning from Data, by Yaser S. AbuMostafa, Malik MagdonIsmail, and HsuanTien Lin, AMLBook New York, 2012.
Convex Optimization, by S. Boyd and L. Vandenberghe, Cambridge University Press, 2003.
References:
Pattern Recognition and Machine Learning, by C. M. Bishop, Springer, 2007.
The Elements of Statistical Learning: Data Mining, Inference, and Prediction, by T. Hastie, R. Tibshirani, and J. Friedman, Springer, 2009.
Deep Learning, by I. Goodfellow, Y. Bengio and A. Courville, MIT Press, 2016.
Convex Optimization: Algorithms and Complexity, by S. Bubeck, Foundations and Trends in Machine Learning, 2015.
Firstorder Methods in Optimization, by A. Beck, MOSSIAM Series on Optimization, 2017.
Nonconvex Optimization for Machine Learning, by P. Jain and P. Kark, Foundations and Trends in Machine Learning, 2017.
Lectures
Foundations
The learning problem
Training versus testing
The linear model
Overfitting
Three learning principles
Techniques
Similaritybased methods
Neural networks
Support vector machines
Learning aides
Optimization
Convex and nonconvex optimization
Firstorder optimization algorithms
Secondorder optimization algorithms
Stochastic optimization algorithms
