This course is designed to introduce students to the field of machine learning, an essential toolset for making sense of the vast and complex datasets that have emerged in fields ranging from biology to finance to marketing to astrophysics in the past twenty years.
This class will present a number of important modeling and prediction techniques that are staples in the fields of machine learning, artificial intelligence, and data science. In addition, this course will cover the statistical underpinnings of the methodology. The tentative list of topics includes:
Regression and classification as a predictive task and general model fitting: a review of linear regression, cross-validation, bootstrapping, sparse regression (Ridge, LASSO), tree-based methods.
Neural networks and deep learning: convolutional neural networks, backpropagation, transformer neural networks, language modeling.
Causality, reasoning, inference: potential outcomes, inverse propensity weighting, matching, difference-in-difference.
Unsupervised learning: Principal component analysis, clustering.
Week 1, Sep 6: Introduction
Examples about linear regression, image classification and language modeling (naive Bayes).
Week 2, Sep 10: Linear regression and estimation, Sep 13: Bias-variance tradeoff; K-nearest neighbors
Simple linear regression, multiple linear regression, with some review of linear algebra.
The bias-variance tradeoff in supervised learning, learning polynomial functions, and K-nearest neighbhors.
Week 3, Sep 17: Logistic regression and linear discriminant analysis, Sep 20: LDA and QDA
Logistic function, logistic loss (log-loss), maximum likelihood estimation, and cross-entropy loss. See notes on logistic regression using gradient descent.
Mixture of Gaussians, estimation of linear discriminant analysis (LDA).
Quadratic discriminant analysis, estimation of QDA.
Logisitic regression vs. LDA vs. QDA.
Week 4, Sep 24: Cross validation, bootstrap, and subset selection, Sep 27: Ridge regression and LASSO
Leave-one-out cross validation, k-fold cross validation.
Bootstrap.
Forward subset selection.
Regularization: Ridge regression, LASSO.
Week 5, Oct 1: Decision trees
Regression tree; classification tree.
Bagging.
Week 6, Oct 8: Random forests, boosting, Oct 11: Introduction to neural networks
Cross-validation in bagging.
Random forests.
Gradient boosting.
MNIST.
Artificial neuron (perceptron), activation functions, feedforward neural networks
Week 7, Oct 15: Convolutional neural networks, Oct 18: Implementations of neural networks in PyTorch
Convolution layer, pooling, parameter sharing in convolution.
Implementation of a simple CNN in PyTorch for solving the MNIST dataset.
Forward pass, and PyTorch implementation of SGD.
PyTorch implementation of linear classifiers.
Week 8, Oct 22: Backpropagation, Oct 25: Backpropagation derivation (Vanishing gradients)
Computing the gradient of a two-layer ReLU network.
Deriving the backward pass in a multi-layer linear network in one dimension.
The vanishing gradients problem.
Week 9: Class project proposal presentations.
Week 10, Nov 5: Midterm review
Week 11, Nov 12: Foundation models I, Nov 15: Foundation models II
Introduction to language models and GPT-3.
Week 12, Nov 19: Dimension reduction I, Nov 22: Dimension reduction II
Principal component analysis.
Singular value decomposition.
Clustering methods; contrastive learning.
Week 13, Nov 26: A gentle introduction to cause and effect
The potential outcomes framework.
Average treatment effect, difference-in-means.
Selection bias, randomized experiments, nearest neighbor matching estimators.
Week 14, Dec 3: Introduction to probabilistic diffusion models / Conclusion
The forward and backward process in diffusion models.
Illustration of DDPM, DDIM, and stable diffusion.
Conclusion: Summary of the course.
You are responsible for keeping up with all announcements made in class and for all changes in the schedule that are posted on the Canvas website.
The grade will be based on the following:
Homeworks (5): 40%
Exam (takehome, choose 24 hours): 30%
Course project report (submitted on GitHub): 15%
Course project presentations (one proposal and one final presentation): 15%
Textbooks for reference:
An Introduction to Statistical Learning (ISL). Gareth James, Daniela Witten, Trevor Hastie, and Robert Tibshirani.
Elements of Statistical Learning (ESL). Trevor Hastie, Rob Tibshirani, and Jerome Friedman.